nnimap check for new mail in the background

nnimap check for new mail in the background

[Lars Tobias Børsting]

Hi,

I have used Gnus for a couple of years now, and can't live without
it. I have just switched from getting mail via mbox spool to nnimap,
with CourierIMAP on the server. I have moved my dozens of mail groups
to nnimap groups. I like IMAP very much, except for one thing; it's very
slow to check for new mail.

With the nnmail backend, checking some dozens of groups was instant and
unnoticable. With nnimap, it's painfully slow. That would have been OK
if I could use Gnus in the meantime. However it locks up, showing
"nnimap: updating info for folder xxx...done" until it is done. This is
making me wait for about one minute until I can do something again.

Is it possible to do new mail checking in a separate thread that runs in
the background, so that it doesn't lock up the interface? If not, is it
hard to implement such a feature?

-- 
Regards,

Lars Tobias Børsting

Re: nnimap check for new mail in the background

[Simon Josefsson]

Lars Tobias Børsting <lars.tobias.borsting@ntnu.no> writes:

> Hi,
>
> I have used Gnus for a couple of years now, and can't live without
> it. I have just switched from getting mail via mbox spool to nnimap,
> with CourierIMAP on the server. I have moved my dozens of mail groups
> to nnimap groups. I like IMAP very much, except for one thing; it's very
> slow to check for new mail.
>
> With the nnmail backend, checking some dozens of groups was instant and
> unnoticable. With nnimap, it's painfully slow. That would have been OK
> if I could use Gnus in the meantime. However it locks up, showing
> "nnimap: updating info for folder xxx...done" until it is done. This is
> making me wait for about one minute until I can do something again.

It should not print the "Updating info" messages -- that will be much
slower than regular new mail checking.  It should only say "Quickly
checking mailbox foo", for each nnimap mailbox.

Historically, the reason for the "update info" message was that your
groups' server method (type `G e' on the group in the *Group* buffer)
said (nnimap "foo") rather than "nnimap:foo".  Changing the former
into to the latter would solve the problem.

However, recently (perhaps last month) I have noticed that Gnus CVS
say "Updating info" for me too.  Not always, but sometimes.  So I
think we have another bug.

> Is it possible to do new mail checking in a separate thread that runs in
> the background, so that it doesn't lock up the interface? If not, is it
> hard to implement such a feature?

It should be possible.  Implement something run with the idle timer
and iterate over all groups.  The problem is in informing Gnus about
the update -- there is no clean API to do that.  It could be added,
but I don't know how.

You could also try to fix the bug above, the "Quickly checking" test
is fairly quick.

Re: nnimap check for new mail in the background

[D. Michael McFarland]

Simon Josefsson <jas@extundo.com> writes:

> Historically, the reason for the "update info" message was that your
> groups' server method (type `G e' on the group in the *Group* buffer)
> said (nnimap "foo") rather than "nnimap:foo".  Changing the former
> into to the latter would solve the problem.

Is there a way to do this in .gnus, perhaps in the server definition?

Best regards,
Michael

-- 
D. Michael McFarland
Department of Aerospace Engineering, University of Illinois
http://www.ae.uiuc.edu/~dmmcf/

Re: nnimap check for new mail in the background

[Simon Josefsson]

dmmcf@uiuc.edu (D. Michael McFarland) writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Historically, the reason for the "update info" message was that your
>> groups' server method (type `G e' on the group in the *Group* buffer)
>> said (nnimap "foo") rather than "nnimap:foo".  Changing the former
>> into to the latter would solve the problem.
>
> Is there a way to do this in .gnus, perhaps in the server definition?

Type `G e' on a group and change it.

The server definition in .gnus will likely read (nnimap "foo"
(nnimap-server ...) ...) and that is fine.  It is only the short form
(nnimap "foo") that is used in internal data structures that is
problematic.

Re: nnimap check for new mail in the background

[Lars Tobias Børsting]

Simon Josefsson <jas@extundo.com> writes:

> D. Michael McFarland <dmmcf@uiuc.edu> writes:
>
>> Simon Josefsson <jas@extundo.com> writes:
>>
>>> Historically, the reason for the "update info" message was that your
>>> groups' server method (type `G e' on the group in the *Group* buffer)
>>> said (nnimap "foo") rather than "nnimap:foo".  Changing the former
>>> into to the latter would solve the problem.
>>
>> Is there a way to do this in .gnus, perhaps in the server definition?
>
> Type `G e' on a group and change it.
>
> The server definition in .gnus will likely read (nnimap "foo"
> (nnimap-server ...) ...) and that is fine.  It is only the short form
> (nnimap "foo") that is used in internal data structures that is
> problematic.

I closed Gnus, opened .newsrc.eld and did 'M-%' (search and replace string)
(nnimap "foo") to "nnimap:foo as Simon suggested. That was far quicker
than doing 'G e' on every group.

-- 
Regards,

Lars Tobias

Re: nnimap check for new mail in the background

[Lars Tobias Børsting]

Simon Josefsson <jas@extundo.com> writes:

> Historically, the reason for the "update info" message was that your
> groups' server method (type `G e' on the group in the *Group* buffer)
> said (nnimap "foo") rather than "nnimap:foo".  Changing the former
> into to the latter would solve the problem.

OK, I changed all my groups to "nnimap:foo" now.

> It should not print the "Updating info" messages -- that will be much
> slower than regular new mail checking.  It should only say "Quickly
> checking mailbox foo", for each nnimap mailbox.

It isn't saying "Quickly checking mailbox foo", it says "nnimap:
Checking mailboxes...". It is also approximatelyten thousand times
faster than before, so thanks a lot! :-)

> However, recently (perhaps last month) I have noticed that Gnus CVS
> say "Updating info" for me too.  Not always, but sometimes.  So I
> think we have another bug.

So, it doesn't say "Updating info" anymore after I changed to
"nnimap:foo".

> Lars Tobias Børsting writes:
>
>> Is it possible to do new mail checking in a separate thread that runs
>> in the background, so that it doesn't lock up the interface? If not,
>> is it hard to implement such a feature?
>
> It should be possible.  Implement something run with the idle timer
> and iterate over all groups.  The problem is in informing Gnus about
> the update -- there is no clean API to do that.  It could be added,
> but I don't know how.

Would it be possible to do what gnus-group-get-new-news-this-group does
when updating each modified group?

-- 
Regards,

Lars Tobias

Re: nnimap check for new mail in the background

[Simon Josefsson]

Lars Tobias Børsting <lars.tobias.borsting@ntnu.no> writes:

>> It should not print the "Updating info" messages -- that will be much
>> slower than regular new mail checking.  It should only say "Quickly
>> checking mailbox foo", for each nnimap mailbox.
>
> It isn't saying "Quickly checking mailbox foo", it says "nnimap:
> Checking mailboxes...". It is also approximatelyten thousand times
> faster than before, so thanks a lot! :-)

This might depend on your gnus-verbose setting.

>> However, recently (perhaps last month) I have noticed that Gnus CVS
>> say "Updating info" for me too.  Not always, but sometimes.  So I
>> think we have another bug.
>
> So, it doesn't say "Updating info" anymore after I changed to
> "nnimap:foo".

I haven't seen it either for a while now.  I saw other problems as
well, and when I removed my entire Agent cache, they went away.

>>> Is it possible to do new mail checking in a separate thread that runs
>>> in the background, so that it doesn't lock up the interface? If not,
>>> is it hard to implement such a feature?
>>
>> It should be possible.  Implement something run with the idle timer
>> and iterate over all groups.  The problem is in informing Gnus about
>> the update -- there is no clean API to do that.  It could be added,
>> but I don't know how.
>
> Would it be possible to do what gnus-group-get-new-news-this-group does
> when updating each modified group?

(setq imap-log t) and look in the *imap-log* buffer, if you think of
the IMAP protocol exchange.

Otherwise, I recommend 'make tags' in .../gnus/lisp/ and then use of
edebug stepping.  (setq nnimap-debug t) and looking in the
*nnimap-debug* might be simpler, though.

Re: nnimap check for new mail in the background

[D. Michael McFarland]

Lars Tobias Børsting <lars.tobias.borsting@ntnu.no> writes:

> I closed Gnus, opened .newsrc.eld and did 'M-%' (search and replace
> string) (nnimap "foo") to "nnimap:foo as Simon suggested. That was
> far quicker than doing 'G e' on every group.

"Far quicker," indeed. :-) (Where'd I get 233 IMAP mail groups?)  And
the speed-up in mail checking is remarkable.  Thanks for your
response, and Simon's, of course.  My only remaining question is,
where is this stuff documented?  There's no telling what else I'm
missing....

Best regards,
Michael

-- 
D. Michael McFarland, Research Associate Professor
Department of Aerospace Engineering, University of Illinois at Urbana-Champaign
http://www.ae.uiuc.edu/~dmmcf/

Re: nnimap check for new mail in the background

[Lars Tobias Børsting]

Simon Josefsson <jas@extundo.com> writes:

>>>> Is it possible to do new mail checking in a separate thread that
>>>> runs in the background, so that it doesn't lock up the interface?
>>>> If not, is it hard to implement such a feature?
>>>
>>> It should be possible.  Implement something run with the idle timer
>>> and iterate over all groups.  The problem is in informing Gnus about
>>> the update -- there is no clean API to do that.  It could be added,
>>> but I don't know how.
>>
>> Would it be possible to do what gnus-group-get-new-news-this-group
>> does when updating each modified group?
>
> (setq imap-log t) and look in the *imap-log* buffer, if you think of
> the IMAP protocol exchange.

Actually, I was thinking of how to notify Gnus of the
changes. gnus-group-get-new-news and friends updates the group buffer,
so maybe some of that code is usable.

-- 
Regards,

Lars Tobias

Re: nnimap check for new mail in the background

[Simon Josefsson]

Lars Tobias Børsting <lars.tobias.borsting@ntnu.no> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>>>>> Is it possible to do new mail checking in a separate thread that
>>>>> runs in the background, so that it doesn't lock up the interface?
>>>>> If not, is it hard to implement such a feature?
>>>>
>>>> It should be possible.  Implement something run with the idle timer
>>>> and iterate over all groups.  The problem is in informing Gnus about
>>>> the update -- there is no clean API to do that.  It could be added,
>>>> but I don't know how.
>>>
>>> Would it be possible to do what gnus-group-get-new-news-this-group
>>> does when updating each modified group?
>>
>> (setq imap-log t) and look in the *imap-log* buffer, if you think of
>> the IMAP protocol exchange.
>
> Actually, I was thinking of how to notify Gnus of the
> changes. gnus-group-get-new-news and friends updates the group buffer,
> so maybe some of that code is usable.

That is the tricky question.  You'll probably need to frob the
internal variables directly, and perhaps also ask for a re-fresh of
the group list asynchronously too, there is no complete API you can
invoke.  Looking at how g-g-g-n-n works is probably a good starting
point.

Re: nnimap check for new mail in the background

[Simon Josefsson]

dmmcf@uiuc.edu (D. Michael McFarland) writes:

> Lars Tobias Børsting <lars.tobias.borsting@ntnu.no> writes:
>
>> I closed Gnus, opened .newsrc.eld and did 'M-%' (search and replace
>> string) (nnimap "foo") to "nnimap:foo as Simon suggested. That was
>> far quicker than doing 'G e' on every group.
>
> "Far quicker," indeed. :-) (Where'd I get 233 IMAP mail groups?)  And
> the speed-up in mail checking is remarkable.  Thanks for your
> response, and Simon's, of course.  My only remaining question is,
> where is this stuff documented?  There's no telling what else I'm
> missing....

It is not documented because it is not intended to happen.  When it
does occur, it is a bug.  I don't know how to reproduce the bug, so I
can't fix it...  Perhaps this could go into a FAQ about IMAP slowness,
it seems many IMAP users run into this sooner or later.

If someone wants to debug and fix it, that would be very useful.

Re: nnimap check for new mail in the background

[Jochen Küpper]

Simon Josefsson <jas-2g3DOctPbj9BDgjK7y7TUQ@public.gmane.org> writes:

> It is not documented because it is not intended to happen. When it
> does occur, it is a bug. I don't know how to reproduce the bug, so I
> can't fix it...

Actually a similar/worse scenario is going on here: Once in a while
the "slowness" issue comes up during a Gnus session. If I then C-g my
way out of it, close Gnus and restart it, it is typically good again
(on very,very rare occasions I have to retreat to an old .newsrc.eld).

But I have no idea how it is triggered nor how to "debug" it when it
occurs.

If I think about it, I only remember seeing it at home when working on
over a DSL line which comes and goes -- cannot seeing it when
"directly" connected at work.

Greetings,
Jochen
-- 
Einigkeit und Recht und Freiheit                http://www.Jochen-Kuepper.de
    Liberté, Égalité, Fraternité                GnuPG key: CC1B0B4D
        (Part 3 you find in my messages before fall 2003.)

Re: nnimap check for new mail in the background

[Simon Josefsson]

Jochen Küpper <jochen-+It19tn3Rl9sbm7dSapR3bNAH6kLmebB@public.gmane.org> writes:

> Simon Josefsson <jas-2g3DOctPbj9BDgjK7y7TUQ@public.gmane.org> writes:
>
>> It is not documented because it is not intended to happen. When it
>> does occur, it is a bug. I don't know how to reproduce the bug, so I
>> can't fix it...
>
> Actually a similar/worse scenario is going on here: Once in a while
> the "slowness" issue comes up during a Gnus session. If I then C-g my
> way out of it, close Gnus and restart it, it is typically good again

This matches my experiences too.

> But I have no idea how it is triggered nor how to "debug" it when it
> occurs.

Same here.