nnimap doing redundant work?

nnimap doing redundant work?

[David Abrahams]

I have been lamenting for some time that using my IMAP server through
GNUs seems to be much slower than working with a dedicated email
program like Mulberry, but recently I have been noticing something
strange.  When I watch my message area, it seems as though nnimap is
usually repeating the same work twice.  For example, I just did `M-G'
in my INBOX summary buffer and my *Messages* buffer contains:

    Expiring articles...done
    nnimap: Updating info for INBOX...done
    Retrieving newsgroup: INBOX...
    nnimap: Updating info for INBOX...done
    Fetching headers for INBOX...done
    Generating summary...done

Note "Updating info for INBOX...done" appears twice.  When I
initially enter GNUs, I often see the message

    nnimap: Mailbox <whatever> modified

twice for several values of <whatever>.

Is this expected?  Is it possible that there's something wrong with
my settings that is causing this behavior?  Lastly, are there any
clues for me about how to speed up IMAP interactions?  Reading news
via NNTP, just moving to the next article still seems way faster than
doing it under IMAP.

Many thanks,
-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap doing redundant work?

[David Abrahams]

David Abrahams <dave@boost-consulting.com> writes:

> When I initially enter GNUs, I often see the message
>
>     nnimap: Mailbox <whatever> modified
>
> twice for several values of <whatever>.

As a matter of fact, it's three times for some of the mailboxes:

    nnimap: Mailbox writing/CUJ modified
    nnimap: Mailbox writing/MPL modified
    nnimap: Mailbox Items modified
    nnimap: Mailbox writing/PyConDC'03 modified
    nnimap: Mailbox writing/MPL modified [2 times]
    nnimap: Mailbox writing/coding-guidelines modified
    nnimap: Mailbox writing/CUJ modified
    nnimap: Mailbox commerce modified
    nnimap: Mailbox tech modified
    nnimap: Mailbox scrapbook modified
    nnimap: Mailbox Sent modified
    nnimap: Mailbox writing/CUJ modified
    nnimap: Mailbox writing modified
    nnimap: Mailbox boost-consulting modified
    nnimap: Mailbox barket modified
    nnimap: Mailbox SpamBox modified
    nnimap: Mailbox Sent Items modified
    nnimap: Mailbox Majordomo modified
    nnimap: Mailbox committee modified
    nnimap: Mailbox Mailers modified
    nnimap: Mailbox license modified
    nnimap: Mailbox INBOX modified
    nnimap: Mailbox tuples modified

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap doing redundant work?

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> When I initially enter GNUs, I often see the message
>>
>>     nnimap: Mailbox <whatever> modified
>>
>> twice for several values of <whatever>.
>
> As a matter of fact, it's three times for some of the mailboxes:
>
>     nnimap: Mailbox writing/CUJ modified
>     nnimap: Mailbox writing/MPL modified
>     nnimap: Mailbox Items modified
>     nnimap: Mailbox writing/PyConDC'03 modified
>     nnimap: Mailbox writing/MPL modified [2 times]
>     nnimap: Mailbox writing/coding-guidelines modified
>     nnimap: Mailbox writing/CUJ modified
>     nnimap: Mailbox commerce modified
>     nnimap: Mailbox tech modified
>     nnimap: Mailbox scrapbook modified
>     nnimap: Mailbox Sent modified
>     nnimap: Mailbox writing/CUJ modified
>     nnimap: Mailbox writing modified
>     nnimap: Mailbox boost-consulting modified
>     nnimap: Mailbox barket modified
>     nnimap: Mailbox SpamBox modified
>     nnimap: Mailbox Sent Items modified
>     nnimap: Mailbox Majordomo modified
>     nnimap: Mailbox committee modified
>     nnimap: Mailbox Mailers modified
>     nnimap: Mailbox license modified
>     nnimap: Mailbox INBOX modified
>     nnimap: Mailbox tuples modified

Weird.  Do you have the same group more than once in the *Group*
buffer if you expand it with L or something?  I can't think of another
explanation.

Perhaps C-k'ing all your nnimap groups, restart emacs and subscribe to
them again will clean things up.

Re: nnimap doing redundant work?

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> Weird.  Do you have the same group more than once in the *Group*
> buffer if you expand it with L or something?  

Nope.

> I can't think of another explanation.
>
> Perhaps C-k'ing all your nnimap groups, restart emacs and subscribe to
> them again will clean things up.

I can try.  Please assure me first that it won't cause the groups to
be deleted from my server.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap doing redundant work?

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

>> I can't think of another explanation.
>>
>> Perhaps C-k'ing all your nnimap groups, restart emacs and subscribe to
>> them again will clean things up.
>
> I can try.  Please assure me first that it won't cause the groups to
> be deleted from my server.

It won't.  If you have group parameters (G p) you may want to rescue
them into gnus-parameters first.  Also, bookmarks are lost.  Take a
backup of .newsrc* end you should be safe.

Re: nnimap doing redundant work?

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> Perhaps C-k'ing all your nnimap groups, restart emacs and subscribe to
> them again will clean things up.

That seems to have cleared the symptoms when entering gnus, but when
I do `M-g' from within my INBOX group, I still get:

    Expiring articles...done
    imap: Connecting to www.stlport.com...
    imap: Opening SSL connection with `openssl s_client -quiet -ssl3 -connect %s:%p'...done
    Waiting for response from www.stlport.com...done
    imap: Plaintext authentication...
    nnimap: Setting marks in INBOX...done
    nnimap: Updating info for INBOX...done
    Retrieving newsgroup: INBOX...
    nnimap: Updating info for INBOX...done
    Fetching headers for INBOX...done
    Generating summary...done

Note that "nnimap: Updating info for INBOX...done" is repeated

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap doing redundant work?

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Perhaps C-k'ing all your nnimap groups, restart emacs and subscribe to
>> them again will clean things up.
>
> That seems to have cleared the symptoms when entering gnus, but when
> I do `M-g' from within my INBOX group, I still get:
>
>     Expiring articles...done
>     imap: Connecting to www.stlport.com...
>     imap: Opening SSL connection with `openssl s_client -quiet -ssl3 -connect %s:%p'...done
>     Waiting for response from www.stlport.com...done
>     imap: Plaintext authentication...
>     nnimap: Setting marks in INBOX...done
>     nnimap: Updating info for INBOX...done
>     Retrieving newsgroup: INBOX...
>     nnimap: Updating info for INBOX...done
>     Fetching headers for INBOX...done
>     Generating summary...done
>
> Note that "nnimap: Updating info for INBOX...done" is repeated

Do you get the same behaviour for M-g on other mailboxes?  Perhaps
your nnimap-split-inbox is INBOX, and splitting is run when M-g is
pressed, so the mailbox is selected twice.  Protocol wise I think this
is optimized away though, frob `imap-log' and look in *imap-log* to
tell for sure.

Re: nnimap doing redundant work?

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>>> Perhaps C-k'ing all your nnimap groups, restart emacs and subscribe to
>>> them again will clean things up.
>>
>> That seems to have cleared the symptoms when entering gnus, but when
>> I do `M-g' from within my INBOX group, I still get:
>>
>>     Expiring articles...done
>>     imap: Connecting to www.stlport.com...
>>     imap: Opening SSL connection with `openssl s_client -quiet -ssl3 -connect %s:%p'...done
>>     Waiting for response from www.stlport.com...done
>>     imap: Plaintext authentication...
>>     nnimap: Setting marks in INBOX...done
>>     nnimap: Updating info for INBOX...done
>>     Retrieving newsgroup: INBOX...
>>     nnimap: Updating info for INBOX...done
>>     Fetching headers for INBOX...done
>>     Generating summary...done
>>
>> Note that "nnimap: Updating info for INBOX...done" is repeated
>
> Do you get the same behaviour for M-g on other mailboxes?  Perhaps
> your nnimap-split-inbox is INBOX, and splitting is run when M-g is
> pressed, so the mailbox is selected twice.  Protocol wise I think this
> is optimized away though, frob `imap-log' and look in *imap-log* to
> tell for sure.

I don't know the answers to this yet, but "it's baaaack...." (the
other behavior):

nnimap: Mailbox writing/CUJ modified
nnimap: Mailbox writing/MPL modified
nnimap: Mailbox Items modified
nnimap: Mailbox writing/MPL modified [2 times]
nnimap: Mailbox CUJ modified
nnimap: Mailbox Drafts modified
nnimap: Mailbox INBOX modified
nnimap: Mailbox Mailers modified
nnimap: Mailbox Majordomo modified
nnimap: Mailbox Sent modified
nnimap: Mailbox Sent Items modified
nnimap: Mailbox SpamBox modified
nnimap: Mailbox barket modified
nnimap: Mailbox boost-consulting modified
nnimap: Mailbox commerce modified
nnimap: Mailbox committee modified
nnimap: Mailbox exceptions modified
nnimap: Mailbox license modified
nnimap: Mailbox scrapbook modified
nnimap: Mailbox snack modified
nnimap: Mailbox tech modified
nnimap: Mailbox tuples modified
nnimap: Mailbox writing modified
nnimap: Mailbox writing/CUJ modified
nnimap: Mailbox writing/PyConDC'03 modified
nnimap: Mailbox writing/coding-guidelines modified
nnimap: Mailbox writing/MPL modified
nnimap: Mailbox writing/CUJ modified

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap doing redundant work?

[David Abrahams]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

Simon Josefsson <jas@extundo.com> writes:

> Do you get the same behaviour for M-g on other mailboxes?  

Here's another example:

nnimap: Setting marks in writing/MPL...done
nnimap: Updating info for writing/MPL...done
Retrieving newsgroup: writing/MPL...
nnimap: Updating info for writing/MPL...done
Fetching headers for writing/MPL...done
Generating summary...done

> Perhaps your nnimap-split-inbox is INBOX, 

It's nil.  

> and splitting is run when M-g is pressed, so the mailbox is selected
> twice.  Protocol wise I think this is optimized away though, frob
> `imap-log' and look in *imap-log* to tell for sure.

In fact, after the "Updating info..." message, I get a
little counter which runs quickly from 0K to 14K, and this happens
for each of the two times the "Updating info..." message appears.

imap-log (enclosed) appears to confirm that work is being repeated, not that I
know how to interpret it...


[-- Attachment #2: imap-log.zip --]
[-- Type: application/zip, Size: 7299 bytes --]

[-- Attachment #3: Type: text/plain, Size: 62 bytes --]



-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap doing redundant work?

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

>> and splitting is run when M-g is pressed, so the mailbox is selected
>> twice.  Protocol wise I think this is optimized away though, frob
>> `imap-log' and look in *imap-log* to tell for sure.
>
> In fact, after the "Updating info..." message, I get a
> little counter which runs quickly from 0K to 14K, and this happens
> for each of the two times the "Updating info..." message appears.

That indicates those groups uses a server definition that isn't
otherwise known to Gnus.  How did you subscribe to the nnimap groups?
What does G E look like on one of these groups?  How do you set up the
nnimap server definition in Gnus (i.e., ~/.gnus or ~/.emacs snippet)?

Re: nnimap doing redundant work?

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>>> and splitting is run when M-g is pressed, so the mailbox is selected
>>> twice.  Protocol wise I think this is optimized away though, frob
>>> `imap-log' and look in *imap-log* to tell for sure.
>>
>> In fact, after the "Updating info..." message, I get a
>> little counter which runs quickly from 0K to 14K, and this happens
>> for each of the two times the "Updating info..." message appears.
>
> That indicates those groups uses a server definition that isn't
> otherwise known to Gnus.  How did you subscribe to the nnimap
> groups?

By entering the server buffer, hitting `u' to remove the K marking on
each of the groups, then hitting `q' to exit the server buffer.

> What does G E look like on one of these groups?  

;;; Editing the group info for `INBOX'.
;; Type `C-c C-c' after you've finished editing.

("INBOX" 3
 ((1 . 11445))
 ((tick 5984 8314 8338 8469 8629 8686 8693 8707 8795 8836 8842 8885 9148 9527 10420 11024 11126 11213 11255 11265 11289)
  (recent 11436 . 11450)
  (reply 8314 8338 8611 8614
         (8618 . 8619)
         8630 8643 8648 8663 8665 8672 8686 8702
         (8713 . 8714)
         8717 8722 8729
         (8737 . 8739)
         8741 8743 8750 8755 8762 8779 8791 8800 8809 8812
         (8820 . 8821)
         8824 8831 8835
         (8858 . 8860)
         8866
         (8884 . 8886)
         8895 8948 8969 8988 9005 9016
         (9021 . 9022)
         9040 9045 9052 9078 9081 9085 9087 9096 9098 9100
         (9120 . 9121)
         9123 9137 9140 9145
         (9148 . 9150)
         (9152 . 9153)
         (9168 . 9169)
         9200
         (9202 . 9203)
         9211 9240 9245 9251 9257 9264 9271 9281 9290
         (9297 . 9298)
         9308 9318 9321 9331 9334 9339 9343 9359 9373 9375 9398 9404 9406 9415 9417 9420
         (9453 . 9454)
         9460 9465 9472 9477 9482
         (9494 . 9495)
         (9500 . 9501)
         9527 9535 9561 9563 9574 9576 9593 9596 9602 9607 9630 9636 9639 9642 9652 9663 9676 9684 9688 9691 9704 9725 9729 9731
         (9744 . 9746)
         9750 9752 9767 9789 9791 9797 9803 9817
         (9826 . 9828)
         9850 9864 9868 9875 9881 9885 9902
         (9905 . 9907)
         9919
         (9921 . 9922)
         9924
         (9934 . 9935)
         9953
         (9984 . 9985)
         9993 9996 9999 10001 10007 10013 10026
         (10031 . 10032)
         10041
         (10051 . 10052)
         10054 10073 10075 10088 10092 10105 10113 10122 10126
         (10131 . 10133)
         10139 10156 10161 10166 10169 10186 10199 10211 10262 10269 10272 10275 10294 10325 10360
         (10371 . 10372)
         (10435 . 10436)
         10440 10445 10452 10457 10468
         (10471 . 10472)
         10526 10549 10568 10597 10607 10620 10625 10629 10636 10639 10645 10659 10662 10670 10674
         (10684 . 10685)
         10690 10695
         (10698 . 10700)
         10704 10719 10733
         (10749 . 10750)
         10752 10764 10789 10811 10818 10822 10827 10850
         (10855 . 10856)
         10865
         (10868 . 10869)
         10875 10877 10889 10894 10904
         (10921 . 10922)
         10937
         (10954 . 10955)
         10960 10965 10975 10981 10987 10992 10994 11007 11009 11011 11019 11026 11040 11043 11049 11056 11064 11067 11072 11076 11082 11091 11093 11097 11103 11109
         (11112 . 11113)
         11115 11153
         (11174 . 11175)
         11185 11198 11233 11255 11271 11273 11288 11304 11313
         (11319 . 11320)
         11325 11336 11338 11340 11342 11345
         (11347 . 11348)
         11367
         (11375 . 11376)
         11386 11403 11407 11411 11421 11445)
  (seen 5984 8314 8338 8469 8629 8686 8693 8707 8795 8825 8836 8842 8885
        (8910 . 8911)
        9148 9527 10249 10420 10965 11024 11202 11225 11237 11255 11258 11265 11289
        (11306 . 11311)
        (11313 . 11317)
        (11319 . 11323)
        (11325 . 11353)
        (11357 . 11360)
        (11362 . 11364)
        (11367 . 11370)
        (11372 . 11380)
        (11382 . 11433)
        (11435 . 11439)
        (11441 . 11445))
  (expire 8910 11315))
 nil
 ((uidvalidity . "255800654")))


> How do you set up the nnimap server definition in Gnus (i.e.,
> ~/.gnus or ~/.emacs snippet)?

(setq gnus-select-method
      '(nnimap "www.stlport.com"
               (nnimap-address "www.stlport.com")
               (nnimap-stream ssl)
               (nnimap-authenticator login)
               ))

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap doing redundant work?

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

>> How do you set up the nnimap server definition in Gnus (i.e.,
>> ~/.gnus or ~/.emacs snippet)?
>
> (setq gnus-select-method
>       '(nnimap "www.stlport.com"
>                (nnimap-address "www.stlport.com")
>                (nnimap-stream ssl)
>                (nnimap-authenticator login)
>                ))

Perhaps this hasn't been tested much.  Could you kill all groups,
restart emacs with this definition instead:

(setq gnus-select-method '(nnnil "")
      gnus-secondary-select-methods '((nnimap "www.stlport.com"
                (nnimap-address "www.stlport.com")
                (nnimap-stream ssl)
                (nnimap-authenticator login)
                )))

If this solves it, it is probably easy to reproduce the problem.  If
you like to debug the lisp to see why this happens, that would be
appreciated.

Re: nnimap doing redundant work?

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> What does G E look like on one of these groups?

Maybe show us the output for a "repeating" group and for a
"non-repeating" group.
-- 
A preposition is not a good thing to end a sentence with.

Re: nnimap doing redundant work?

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>>> How do you set up the nnimap server definition in Gnus (i.e.,
>>> ~/.gnus or ~/.emacs snippet)?
>>
>> (setq gnus-select-method
>>       '(nnimap "www.stlport.com"
>>                (nnimap-address "www.stlport.com")
>>                (nnimap-stream ssl)
>>                (nnimap-authenticator login)
>>                ))
>
> Perhaps this hasn't been tested much.  Could you kill all groups,
> restart emacs with this definition instead:
>
> (setq gnus-select-method '(nnnil "")
>       gnus-secondary-select-methods '((nnimap "www.stlport.com"
>                 (nnimap-address "www.stlport.com")
>                 (nnimap-stream ssl)
>                 (nnimap-authenticator login)
>                 )))

nnimap: Updating info for nnimap+www.stlport.com:INBOX...done
Retrieving newsgroup: nnimap+www.stlport.com:INBOX...
nnimap: Updating info for nnimap+www.stlport.com:INBOX...done
Fetching headers for nnimap+www.stlport.com:INBOX...done
Generating summary...done
No more unread articles


Also, the group buffer was pretty strange after I did this; several
groups appeared multiple times.  I had to kill them and re-check my
subscriptions in order to clean things up.

> If this solves it, it is probably easy to reproduce the problem.  If
> you like to debug the lisp to see why this happens, that would be
> appreciated.

I'm very bad at elisp debugging (not very competent with edebug), but
if there's a specifc thing you want me to look at I'll be happy to do
it.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap doing redundant work?

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> David Abrahams <dave@boost-consulting.com> writes:
>>
>>>> How do you set up the nnimap server definition in Gnus (i.e.,
>>>> ~/.gnus or ~/.emacs snippet)?
>>>
>>> (setq gnus-select-method
>>>       '(nnimap "www.stlport.com"
>>>                (nnimap-address "www.stlport.com")
>>>                (nnimap-stream ssl)
>>>                (nnimap-authenticator login)
>>>                ))
>>
>> Perhaps this hasn't been tested much.  Could you kill all groups,
>> restart emacs with this definition instead:
>>
>> (setq gnus-select-method '(nnnil "")
>>       gnus-secondary-select-methods '((nnimap "www.stlport.com"
>>                 (nnimap-address "www.stlport.com")
>>                 (nnimap-stream ssl)
>>                 (nnimap-authenticator login)
>>                 )))
>
> nnimap: Updating info for nnimap+www.stlport.com:INBOX...done
> Retrieving newsgroup: nnimap+www.stlport.com:INBOX...
> nnimap: Updating info for nnimap+www.stlport.com:INBOX...done
> Fetching headers for nnimap+www.stlport.com:INBOX...done
> Generating summary...done
> No more unread articles
>
>
> Also, the group buffer was pretty strange after I did this; several
> groups appeared multiple times.  I had to kill them and re-check my
> subscriptions in order to clean things up.

Did you kill all groups and then resubscribed to them?  No mention of
the old groups in ~/.newsrc* when you press `s' in Gnus?

Do you have any customization when entering a group?  Or just any kind
of customization that might affect this?

>> If this solves it, it is probably easy to reproduce the problem.  If
>> you like to debug the lisp to see why this happens, that would be
>> appreciated.
>
> I'm very bad at elisp debugging (not very competent with edebug), but
> if there's a specifc thing you want me to look at I'll be happy to do
> it.

Let's try to get things into a working state first, then it is easier
to debug; just do a binary search between the working state and the
buggy state until the cause of the problem is found.  It still takes
some energy though, so perhaps you'll settle with simply having the
problem disappear.  But is useful too, the mailing list archive will
document this, in case someone wants to debug this in the future.

Re: nnimap doing redundant work?

[David Abrahams]

[-- Attachment #1: Type: text/plain, Size: 3081 bytes --]

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>>> David Abrahams <dave@boost-consulting.com> writes:
>>>
>>>>> How do you set up the nnimap server definition in Gnus (i.e.,
>>>>> ~/.gnus or ~/.emacs snippet)?
>>>>
>>>> (setq gnus-select-method
>>>>       '(nnimap "www.stlport.com"
>>>>                (nnimap-address "www.stlport.com")
>>>>                (nnimap-stream ssl)
>>>>                (nnimap-authenticator login)
>>>>                ))
>>>
>>> Perhaps this hasn't been tested much.  Could you kill all groups,
>>> restart emacs with this definition instead:
>>>
>>> (setq gnus-select-method '(nnnil "")
>>>       gnus-secondary-select-methods '((nnimap "www.stlport.com"
>>>                 (nnimap-address "www.stlport.com")
>>>                 (nnimap-stream ssl)
>>>                 (nnimap-authenticator login)
>>>                 )))
>>
>> nnimap: Updating info for nnimap+www.stlport.com:INBOX...done
>> Retrieving newsgroup: nnimap+www.stlport.com:INBOX...
>> nnimap: Updating info for nnimap+www.stlport.com:INBOX...done
>> Fetching headers for nnimap+www.stlport.com:INBOX...done
>> Generating summary...done
>> No more unread articles
>>
>>
>> Also, the group buffer was pretty strange after I did this; several
>> groups appeared multiple times.  I had to kill them and re-check my
>> subscriptions in order to clean things up.
>
> Did you kill all groups and then resubscribed to them?  

Yes.

> No mention of the old groups in ~/.newsrc* when you press `s' in
> Gnus?

In which buffer? `s' in the *Group* buffer doesn't show me
~/.newsrc.  If I go look at it, it says:

    writing/MPL:
    writing/MPL!
    writing/MPL! 1-1082,1084-1093,1095-1375,1377-1381,1383-1389

> Do you have any customization when entering a group?  Or just any kind
> of customization that might affect this?

I'm happy to send you my .gnus (enclosed)...  Since I've been wiping the groups
per your instructions and haven't explicitly customized anything, I
assume there are no group customizations.

>>> If this solves it, it is probably easy to reproduce the problem.  If
>>> you like to debug the lisp to see why this happens, that would be
>>> appreciated.
>>
>> I'm very bad at elisp debugging (not very competent with edebug), but
>> if there's a specifc thing you want me to look at I'll be happy to do
>> it.
>
> Let's try to get things into a working state first, then it is easier
> to debug; just do a binary search between the working state and the
> buggy state until the cause of the problem is found.  It still takes
> some energy though, so perhaps you'll settle with simply having the
> problem disappear.  But is useful too, the mailing list archive will
> document this, in case someone wants to debug this in the future.

Are you sure that I'm seeing some unexpected behavior?  I don't know
how to characterize "working" except that I know it feels slow to use
my IMAP server through gnus.

BTW, should I be updating my gnus installation?  I'm using o0-15.


[-- Attachment #2: .gnus --]
[-- Type: text/plain, Size: 8164 bytes --]

(add-hook 'gnus-select-article-hook 'gnus-agent-fetch-selected-article)

(setq my-inbox-group-params
      '((uidvalidity . "255800654")
        (total-expire . t)
        (spam-process-destination . "SpamBox")
        (expiry-wait . 15)))

(setq my-gnus-parameters
      '(("mail\\..*"  (gnus-show-threads nil)
         (gnus-use-scoring nil)
         (gnus-summary-line-format
          "%U%R%z%I%(%[%d:%ub%-23,23f%]%) %s\n")
         (gcc-self . t)
         (display . all))
        ("mail\\.me" (gnus-use-scoring  t))
        ("list\\..*" (total-expire . t)
         (broken-reply-to . t))))

; Using secondary select method instead per Simon's instructions
(and nil (setq gnus-select-method
               '(nnimap "www.stlport.com"
               (nnimap-address "www.stlport.com")
               (nnimap-stream ssl)
               (nnimap-authenticator login)
               )))

(setq gnus-select-method '(nnnil "")
       gnus-secondary-select-methods
      '(
        ;; Using secondary select method per Simon's instructions
        (nnimap "www.stlport.com"
                (nnimap-address "www.stlport.com")
                (nnimap-stream ssl)
                (nnimap-authenticator login)
                )
        (nntp "news.rcn.com")
        (nntp "news.gmane.org")
        )

      ssl-view-certificate-program-name "notepad"
;     imap-debug t
      gnus-check-new-newsgroups nil
      gnus-local-domain "boost-consulting.com"
      
;      gnus-message-archive-group '("nnimap+www.stlport.com:INBOX"
;                                   ;"nnimap+www.stlport.com:Sent"
;                                   )
      
      gnus-summary-line-format "%O%U%R%z%I%d%(%[%4L: %-20,20n%]%) %s
"
      gnus-summary-ignore-duplicates t
      
;      gnus-total-expirable-newsgroups "INBOX"

;      gnus-cacheable-groups "^nnimap*"
;      gnus-use-cache nil
    gnus-agent-cache t
    gnus-agent t
    gnus-agent-consider-all-articles t
    gnus-agent-expire-days 36500
    ; gnus-agent-mark-unread-after-downloaded nil

     message-subject-trailing-was-query t ;; always strip the trailing old subject in (was: ...) subjects
     message-default-headers "To: dave@boost-consulting.com"
    

my-gnus-summary-highlight

'(((eq mark gnus-canceled-mark)
  . gnus-summary-cancelled-face)
 ((and
   (> score default-high)
   (or
    (eq mark gnus-dormant-mark)
    (eq mark gnus-ticked-mark)))
  . gnus-summary-high-ticked-face)
 ((and
   (< score default-low)
   (or
    (eq mark gnus-dormant-mark)
    (eq mark gnus-ticked-mark)))
  . gnus-summary-low-ticked-face)
 ((or
   (eq mark gnus-dormant-mark)
   (eq mark gnus-ticked-mark))
  . gnus-summary-normal-ticked-face)
 ((and
   (> score default-high)
   (eq mark gnus-ancient-mark))
  . gnus-summary-high-ancient-face)
 ((and
   (< score default-low)
   (eq mark gnus-ancient-mark))
  . gnus-summary-low-ancient-face)
 ((eq mark gnus-ancient-mark)
  . gnus-summary-normal-ancient-face)
 ((and
   (> score default-high)
   (eq mark gnus-unread-mark))
  . gnus-summary-high-unread-face)
 ((and
   (< score default-low)
   (eq mark gnus-unread-mark))
  . gnus-summary-low-unread-face)
 ((eq mark gnus-unread-mark)
  . gnus-summary-normal-unread-face)
 ((> score default-high)
  . gnus-summary-high-read-face)
 ((< score default-low)
  . gnus-summary-low-read-face)
 (t . gnus-summary-normal-read-face))
      )

(if (and (eq system-type 'windows-nt) (< my-emacs-version 21.3))
    (custom-set-faces

     '(gnus-emphasis-bold ((t (:weight bold :height 0.99 :family "tahoma"))))
     '(gnus-emphasis-bold-italic ((t (:slant italic :weight bold :family "arial"))))
     '(gnus-emphasis-italic ((t (:slant italic :family "arial"))))
                                        ; '(gnus-summary-high-ancient-face ((t (:foreground "magenta4" :weight bold :family "tahoma"))))
                                        ; '(gnus-summary-high-ticked-face ((t (:foreground "firebrick" :weight bold :family "arial"))))
                                        ; '(gnus-summary-high-undownloaded-face ((t (:foreground "magenta4" :weight normal))))
                                        ; '(gnus-summary-low-ancient-face ((t (:foreground "magenta4" :slant italic :family "arial"))))
                                        ; '(gnus-summary-low-undownloaded-face ((t (:foreground "magenta4" :slant italic :weight normal :family "arial"))))
                                        ; '(gnus-summary-normal-ancient-face ((((class color) (background light)) (:foreground "magenta4"))))
                                        ; '(gnus-summary-normal-undownloaded-face ((t (:foreground "magenta4" :weight normal))))
     )
)

 (if t
    (progn
      (gnus-add-configuration
       '(article
         (horizontal 1.0
                     (vertical 1.0
                               (group 35)
                               (summary 1.0 point))
                     (vertical .5 (article 1.0)))))


      (gnus-add-configuration
       '(summary
         (horizontal 1.0
                     (vertical 1.0
                               (group 35)
                               (summary 1.0 point))
                     (vertical .5 (article 1.0)))))

      (gnus-add-configuration
       '(message
         (horizontal 1.0
                     (vertical 1.0
                               (group 35)
                               (summary 1.0))
                     (vertical .5
                               (message 1.0 point)))))

      (gnus-add-configuration
       '(reply
         (horizontal 1.0
                     (vertical 1.0
                               (group 35)
                               (summary 1.0))
                     (vertical .5
                               (message 1.0 point)
                               (article .25)))))

      (gnus-add-configuration
       '(reply-yank
         (horizontal 1.0
                     (vertical 1.0
                               (group 35)
                               (summary 1.0))
                     (vertical .5
                               (message 1.0 point)))))
      )
  
  (progn
      (gnus-add-configuration
       '(article
         (horizontal 1.0 
                     (group 60)
                     (summary 1.0 point)
                     (article 80))))


      (gnus-add-configuration
       '(summary
         (horizontal 1.0 
                     (group 60)
                     (summary 1.0 point)
                     (article 80))))

      (gnus-add-configuration
       '(message
         (horizontal 1.0 
                     (group 60)
                     (summary 1.0)
                     (message 80 point))))

      (gnus-add-configuration
       '(reply
         (horizontal 1.0 
                     (group 60)
                     (summary 1.0)
                     (message 80 point))))

      (gnus-add-configuration
       '(reply-yank
         (horizontal 1.0 
                     (group 60)
                     (summary 1.0)
                     (message 80 point))))
    ))


(define-key gnus-summary-mode-map "d" 'gnus-summary-mark-as-expirable)
(define-key gnus-summary-mode-map "$" 'gnus-summary-mark-as-spam)

(add-hook 'gnus-group-mode-hook 'gnus-topic-mode)

(setq message-dont-reply-to-names "\\(dave@boost-consulting\\.com\\)\\|\\(david\\.abrahams@rcn\\.com\\)\\|\\(Undisclosed recipients\\)")

(setq mm-discouraged-alternatives '("text/html" "text/richtext"))

;; delete duplicate messages
(setq nnmail-treat-duplicates 'delete)

;; Hack GNUs' inability to follow references reliably
;; This still doesn't work :(
(defun my-fetch-whole-thread()
   "like `A R' `T o' `A T' in the summary buffer."
   (interactive)
   (gnus-summary-refer-references)
   (gnus-summary-top-thread)
   (gnus-summary-refer-thread))

(define-key gnus-summary-mode-map (kbd "AT") 'my-fetch-whole-thread)

;; leave this at the bottom, on Kai Großjohann's recommendation
(gnus-compile)

[-- Attachment #3: Type: text/plain, Size: 62 bytes --]



-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap doing redundant work?

[Kai Großjohann]

David Abrahams <dave@boost-consulting.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> No mention of the old groups in ~/.newsrc* when you press `s' in
>> Gnus?
>
> In which buffer? `s' in the *Group* buffer doesn't show me
> ~/.newsrc.  If I go look at it, it says:
>
>     writing/MPL:
>     writing/MPL!
>     writing/MPL! 1-1082,1084-1093,1095-1375,1377-1381,1383-1389

Each group should be mentioned only once in ~/.newsrc.

Note that is also another file ~/.newsrc.eld.  Have a look at that,
too.

(Simon typed ~/.newsrc* and the star means all files beginning with
that string.)
-- 
A preposition is not a good thing to end a sentence with.

Re: nnimap doing redundant work?

[David Abrahams]

Sorry, I thought I sent this, but it was just sitting there in my
emacs session :(

Simon Josefsson <jas@extundo.com> writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>>> David Abrahams <dave@boost-consulting.com> writes:
>>>
>>>>> How do you set up the nnimap server definition in Gnus (i.e.,
>>>>> ~/.gnus or ~/.emacs snippet)?
>>>>
>>>> (setq gnus-select-method
>>>>       '(nnimap "www.stlport.com"
>>>>                (nnimap-address "www.stlport.com")
>>>>                (nnimap-stream ssl)
>>>>                (nnimap-authenticator login)
>>>>                ))
>>>
>>> Perhaps this hasn't been tested much.  Could you kill all groups,
>>> restart emacs with this definition instead:
>>>
>>> (setq gnus-select-method '(nnnil "")
>>>       gnus-secondary-select-methods '((nnimap "www.stlport.com"
>>>                 (nnimap-address "www.stlport.com")
>>>                 (nnimap-stream ssl)
>>>                 (nnimap-authenticator login)
>>>                 )))
>>
>> nnimap: Updating info for nnimap+www.stlport.com:INBOX...done
>> Retrieving newsgroup: nnimap+www.stlport.com:INBOX...
>> nnimap: Updating info for nnimap+www.stlport.com:INBOX...done
>> Fetching headers for nnimap+www.stlport.com:INBOX...done
>> Generating summary...done
>> No more unread articles
>>
>>
>> Also, the group buffer was pretty strange after I did this; several
>> groups appeared multiple times.  I had to kill them and re-check my
>> subscriptions in order to clean things up.
>
> Did you kill all groups and then resubscribed to them?  No mention of
> the old groups in ~/.newsrc* when you press `s' in Gnus?

Uh weird:  I did kill all groups (though they kept
reappearing)... but look at my *Group* buffer now:

 U     4: writing/MPL 
 U     *: nnimap+www.stlport.com:CUJ 
 U     *: nnimap+www.stlport.com:Drafts 
*U     0:*nnimap+www.stlport.com:INBOX 
 U     *: nnimap+www.stlport.com:Items 
 U     *: nnimap+www.stlport.com:Mailers 
 U     *: nnimap+www.stlport.com:Majordomo 
 U     *: nnimap+www.stlport.com:SpamBox 
 U     *: nnimap+www.stlport.com:barket 
 U     *: nnimap+www.stlport.com:boost-consulting 
 U     *: nnimap+www.stlport.com:commerce 
 U     *: nnimap+www.stlport.com:committee 
 U     *: nnimap+www.stlport.com:exceptions 
 U     *: nnimap+www.stlport.com:license 
 U     5:*nnimap+www.stlport.com:scrapbook 
 U     *: nnimap+www.stlport.com:tech 
 U     *: nnimap+www.stlport.com:tuples 
 U     *: nnimap+www.stlport.com:writing 
 U     *: nnimap+www.stlport.com:writing/CUJ 
 U     *: nnimap+www.stlport.com:writing/MPL 
 U     *: nnimap+www.stlport.com:writing/PyConDC'03 

(note the first and second-to-last lines)

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap doing redundant work?

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

> Uh weird:  I did kill all groups (though they kept
> reappearing)... but look at my *Group* buffer now:
>
>  U     4: writing/MPL 
>  U     *: nnimap+www.stlport.com:CUJ 
>  U     *: nnimap+www.stlport.com:Drafts 
> *U     0:*nnimap+www.stlport.com:INBOX 
>  U     *: nnimap+www.stlport.com:Items 
>  U     *: nnimap+www.stlport.com:Mailers 
>  U     *: nnimap+www.stlport.com:Majordomo 
>  U     *: nnimap+www.stlport.com:SpamBox 
>  U     *: nnimap+www.stlport.com:barket 
>  U     *: nnimap+www.stlport.com:boost-consulting 
>  U     *: nnimap+www.stlport.com:commerce 
>  U     *: nnimap+www.stlport.com:committee 
>  U     *: nnimap+www.stlport.com:exceptions 
>  U     *: nnimap+www.stlport.com:license 
>  U     5:*nnimap+www.stlport.com:scrapbook 
>  U     *: nnimap+www.stlport.com:tech 
>  U     *: nnimap+www.stlport.com:tuples 
>  U     *: nnimap+www.stlport.com:writing 
>  U     *: nnimap+www.stlport.com:writing/CUJ 
>  U     *: nnimap+www.stlport.com:writing/MPL 
>  U     *: nnimap+www.stlport.com:writing/PyConDC'03 
>
> (note the first and second-to-last lines)

How did you kill them?  C-k is the right way.  Make sure you don't
have a copy of the group hidden in another topic, or something like
that.

Hm, you do quit Gnus properly, don't you?  `q' or `s' saves
~/.newsrc*, which makes sure the groups are really gone.

Re: nnimap doing redundant work?

[David Abrahams]

Simon Josefsson <jas@extundo.com> writes:

> How did you kill them?  C-k is the right way.  

That's what I did.

> Make sure you don't have a copy of the group hidden in another
> topic, or something like that.

I eventually got everything back to "normal.", though performance
doesn't seem to be improved.  In particular, when fetching new
articles with `M-G' in my INBOX it still does the "fetching info;
count up to 14K" dance twice:

    Expiring articles...done
    nnimap: Setting marks in INBOX...done
    nnimap: Setting marks in INBOX...done
    nnimap: Updating info for nnimap+www.stlport.com:INBOX...done
    Retrieving newsgroup: nnimap+www.stlport.com:INBOX...
    nnimap: Updating info for nnimap+www.stlport.com:INBOX...done
    Fetching headers for nnimap+www.stlport.com:INBOX...done
    Generating summary...done

> Hm, you do quit Gnus properly, don't you?  `q' or `s' saves
> ~/.newsrc*, which makes sure the groups are really gone.

Yes, I do use `q'.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap doing redundant work?

[David Abrahams]

David Abrahams <dave@boost-consulting.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> How did you kill them?  C-k is the right way.  
>
> That's what I did.
>
>> Make sure you don't have a copy of the group hidden in another
>> topic, or something like that.
>
> I eventually got everything back to "normal.", though performance
> doesn't seem to be improved.  In particular, when fetching new
> articles with `M-G' in my INBOX it still does the "fetching info;
> count up to 14K" dance twice:
>
>     Expiring articles...done
>     nnimap: Setting marks in INBOX...done
>     nnimap: Setting marks in INBOX...done
>     nnimap: Updating info for nnimap+www.stlport.com:INBOX...done
>     Retrieving newsgroup: nnimap+www.stlport.com:INBOX...
>     nnimap: Updating info for nnimap+www.stlport.com:INBOX...done
>     Fetching headers for nnimap+www.stlport.com:INBOX...done
>     Generating summary...done
>
>> Hm, you do quit Gnus properly, don't you?  `q' or `s' saves
>> ~/.newsrc*, which makes sure the groups are really gone.
>
> Yes, I do use `q'.

More data:

When I subscribe to nnimap groups that are in sub-directories and I'm
using imap as my primary select method, it appears to add each group
to the *Groups* buffer (at least) twice.  That's why you see things
like writing/MPL, writing/CUJ, etc. duplicated in my traces.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap doing redundant work?

[Simon Josefsson]

David Abrahams <dave@boost-consulting.com> writes:

> When I subscribe to nnimap groups that are in sub-directories and I'm
> using imap as my primary select method, it appears to add each group
> to the *Groups* buffer (at least) twice.  That's why you see things
> like writing/MPL, writing/CUJ, etc. duplicated in my traces.

This seems to be the real cause of the problem, then.  Does it only
happen when the server is your primary select method, or does it
happen if you use the suggested nnil primary method and nnimap in the
secondary select method?  Having nnimap as the primary select method
is probably not that common, so perhaps it is buggy.

Re: nnimap doing redundant work?

[David Abrahams]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>>> No mention of the old groups in ~/.newsrc* when you press `s' in
>>> Gnus?
>>
>> In which buffer? `s' in the *Group* buffer doesn't show me
>> ~/.newsrc.  If I go look at it, it says:
>>
>>     writing/MPL:
>>     writing/MPL!
>>     writing/MPL! 1-1082,1084-1093,1095-1375,1377-1381,1383-1389
>
> Each group should be mentioned only once in ~/.newsrc.

It's not, despite the fact that I've deleted the duplicates from my
*Group* buffer.

> Note that is also another file ~/.newsrc.eld.  Have a look at that,
> too.

Yeah, I know about that, but don't know how to interpret it.  The
duplicated groups appear multiple times in there as well.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: nnimap doing redundant work?

[David Abrahams]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> What does G E look like on one of these groups?
>
> Maybe show us the output for a "repeating" group and for a
> "non-repeating" group.

Here's a nice short non-repeater:

--
;;; Editing the group info for `snack'.
;; Type `C-c C-c' after you've finished editing.

("snack" 3 nil nil nil
 ((uidvalidity . "259151356")))
--

And here's a short repeater:

--
;;; Editing the group info for `writing/CUJ'.
;; Type `C-c C-c' after you've finished editing.

("writing/CUJ" 3
 ((1 . 5)
  (10 . 23))
 ((seen
   (6 . 9))
  (reply
   (1 . 2)
   8
   (10 . 11)
   13 15
   (17 . 18)
   20 27))
 nil
 ((uidvalidity . "261144282")))

--

HTH,
-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com