sluggish IMAP updating?

sluggish IMAP updating?

[Greg Troxel]

I use gnus (head of cvs) in emacs 21.4 on NetBSD 3.1ish and 4.0ish, with
my mail in IMAP via dovecot (1.0.0).  I use gnome's mail-notification
and sometimes run thunderbird as well.  Generally this works well.

Typically when new mail arrives I'll notice because of mail-notification
(which is using IMAP's idle extension and thus notices new mail
promptly).  I keep a gnus instance running all day, usually sitting in
the *Group* buffer, and I'll type 'g'.  Usually INBOX will show new
messages, and selecting it will let me reaad them as you'd expect.
Fairly often (perhaps 1 in 4?), typing g will result in INBOX showing 0
messages still.  In these cases, entering the group, exiting it, and
then retyping 'g' will show non-zero messages and I can then enter and
read them.  In these cases thunderbird shows new mail just fine.

I know gnus doesn't push read marks back to the server until the group
is exited, and I wonder if there's some other kind of synchroniziation
issue.  I'll enable imap logging and look at this, but I wanted to ask
if anyone else is seeing this, with dovecot, or with any other IMAP
server.

Re: sluggish IMAP updating?

[Gabor Z. Papp]

* Greg Troxel <gdt@work.lexort.com>:

| I know gnus doesn't push read marks back to the server until the group
| is exited, and I wonder if there's some other kind of synchroniziation
| issue.  I'll enable imap logging and look at this, but I wanted to ask
| if anyone else is seeing this, with dovecot, or with any other IMAP
| server.

Yes, I do both on Linux 2.4 and NetBSD-4 with courier-imap 3.0.8 using
GNU Emacs 22.0.97.1 and ngnus 0.6.

I do exactly the same "procedure" except I'm not running any other
imap client paralell (like thunderbird or new mail notify stuff).

Usually I have to exit from gnus and restart it to recognize the new mails.

Re: sluggish IMAP updating?

[Simon Josefsson]

Greg Troxel <gdt@work.lexort.com> writes:

> I use gnus (head of cvs) in emacs 21.4 on NetBSD 3.1ish and 4.0ish, with
> my mail in IMAP via dovecot (1.0.0).  I use gnome's mail-notification
> and sometimes run thunderbird as well.  Generally this works well.
>
> Typically when new mail arrives I'll notice because of mail-notification
> (which is using IMAP's idle extension and thus notices new mail
> promptly).  I keep a gnus instance running all day, usually sitting in
> the *Group* buffer, and I'll type 'g'.  Usually INBOX will show new
> messages, and selecting it will let me reaad them as you'd expect.
> Fairly often (perhaps 1 in 4?), typing g will result in INBOX showing 0
> messages still.  In these cases, entering the group, exiting it, and
> then retyping 'g' will show non-zero messages and I can then enter and
> read them.  In these cases thunderbird shows new mail just fine.
>
> I know gnus doesn't push read marks back to the server until the group
> is exited, and I wonder if there's some other kind of synchroniziation
> issue.  I'll enable imap logging and look at this, but I wanted to ask
> if anyone else is seeing this, with dovecot, or with any other IMAP
> server.

Does toggling nnimap-need-unselect-to-notice-new-mail help?

/Simon

Re: sluggish IMAP updating?

[Greg Troxel]

Simon Josefsson <simon@josefsson.org> writes:

> Greg Troxel <gdt@work.lexort.com> writes:
>
>> [gnus/dovecot doesn't notice new messages right away]
>
> Does toggling nnimap-need-unselect-to-notice-new-mail help?

Yes, that seems to fix the problem.   From *imap-log*, I can see that
STATUS was returning no new messages, but after UNSELECT, it gives the
right status.

So, is dovecot broken, or gnus?  I am not familiar enough with the IMAP
spec to know if dovecot is wrong for failing to report new mail that has
arrived on a selected mailbox since the selection.   With ACID hat on it
seems somewhat reasonable of dovecot conceptually, but not really ("SET
IMAP ISOLATION LEVEL READ COMMITTED" :-).

I realize this may be a deoptimization, but it would seem best for gnus
to UNSELECT a mailbox when the user exits from the group and the summary
buffer is destroyed.  That may let the imap free resources, and seems
like a good idle state.

Re: sluggish IMAP updating?

[Greg Troxel]

Simon Josefsson <simon@josefsson.org> writes:

>
> Does toggling nnimap-need-unselect-to-notice-new-mail help?

The word from dovecot is that STATUS on a selected mailbox is SHOULD NOT
to start with, and MUST NOT to check for new mail.  See 6.3.10 of
RFC3501.

http://dovecot.org/pipermail/dovecot/2007-May/023045.html

Plus, doing lots of STATUS commands seems to be disfavored but I don't
understand IMAP well enough to understand.


Given this, I think it makes sense to set
nnimap-need-unselect-to-notice-new-mail to t by default.

Re: sluggish IMAP updating?

[Simon Josefsson]

Greg Troxel <gdt@work.lexort.com> writes:

> Simon Josefsson <simon@josefsson.org> writes:
>
>> Greg Troxel <gdt@work.lexort.com> writes:
>>
>>> [gnus/dovecot doesn't notice new messages right away]
>>
>> Does toggling nnimap-need-unselect-to-notice-new-mail help?
>
> Yes, that seems to fix the problem.   From *imap-log*, I can see that
> STATUS was returning no new messages, but after UNSELECT, it gives the
> right status.

Right.

>> Does toggling nnimap-need-unselect-to-notice-new-mail help?
>
> The word from dovecot is that STATUS on a selected mailbox is SHOULD NOT
> to start with, and MUST NOT to check for new mail.  See 6.3.10 of
> RFC3501.
>
> http://dovecot.org/pipermail/dovecot/2007-May/023045.html
>
> Plus, doing lots of STATUS commands seems to be disfavored but I don't
> understand IMAP well enough to understand.

nnimap was written when RFC 2060 was current, and we haven't really
updated it for RFC 3501.  I suspect the notes in 6.3.10 was added as a
result of nnimap and other implementations behaviour, but the problem is
that I don't know of a good way to implement this in Gnus without using
STATUS (and, alas, I wouldn't have time to implement anything even if I
had some ideas).  Note that I cannot see that RFC 3501 allows servers to
respond with incorrect data in this situation, which dovecot appears to
be doing.

> Given this, I think it makes sense to set
> nnimap-need-unselect-to-notice-new-mail to t by default.

This will slow things down for other users...  Are you using the latest
Dovecot version?  Maybe we can improve the manual.

I don't have a strong opinion on what the default should be.  If others
think it is safer to let it be t by default, by all means change it.

/Simon

Re: sluggish IMAP updating?

[Greg Troxel]

Simon Josefsson <simon@josefsson.org> writes:

>> The word from dovecot is that STATUS on a selected mailbox is SHOULD NOT
>> to start with, and MUST NOT to check for new mail.  See 6.3.10 of
>> RFC3501.
>>
>> http://dovecot.org/pipermail/dovecot/2007-May/023045.html
>>
>> Plus, doing lots of STATUS commands seems to be disfavored but I don't
>> understand IMAP well enough to understand.
>
> nnimap was written when RFC 2060 was current, and we haven't really
> updated it for RFC 3501.  I suspect the notes in 6.3.10 was added as a
> result of nnimap and other implementations behaviour, but the problem is
> that I don't know of a good way to implement this in Gnus without using
> STATUS (and, alas, I wouldn't have time to implement anything even if I
> had some ideas).  Note that I cannot see that RFC 3501 allows servers to
> respond with incorrect data in this situation, which dovecot appears to
> be doing.

The dovecot people talk about 'synchronizing the mailbox', and
apparently they are in an implementation bind because they aren't
allowed to clear recent flags.

I see your point, but the RFC says "MUST NOT" use STATUS on a selected
mailbox to check for new mail.

I think one is supposed to use SEARCH RECENT, but I really  just barely
understand imap.

> This will slow things down for other users...  Are you using the latest
> Dovecot version?  Maybe we can improve the manual.

Yes, I'm running 1.0.0.

> I don't have a strong opinion on what the default should be.  If others
> think it is safer to let it be t by default, by all means change it.

Someone else was having trouble with courier imap, so this seems not to
just be a dovecot issue.   I'd favor safety over efficiency.   for me
'g' takes a while and does 100 STATUS checks, so an unselect hardly
seems like a big deal.

Re: sluggish IMAP updating?

[Simon Josefsson]

Greg Troxel <gdt@work.lexort.com> writes:

> Simon Josefsson <simon@josefsson.org> writes:
>
>>> The word from dovecot is that STATUS on a selected mailbox is SHOULD NOT
>>> to start with, and MUST NOT to check for new mail.  See 6.3.10 of
>>> RFC3501.
>>>
>>> http://dovecot.org/pipermail/dovecot/2007-May/023045.html
>>>
>>> Plus, doing lots of STATUS commands seems to be disfavored but I don't
>>> understand IMAP well enough to understand.
>>
>> nnimap was written when RFC 2060 was current, and we haven't really
>> updated it for RFC 3501.  I suspect the notes in 6.3.10 was added as a
>> result of nnimap and other implementations behaviour, but the problem is
>> that I don't know of a good way to implement this in Gnus without using
>> STATUS (and, alas, I wouldn't have time to implement anything even if I
>> had some ideas).  Note that I cannot see that RFC 3501 allows servers to
>> respond with incorrect data in this situation, which dovecot appears to
>> be doing.
>
> The dovecot people talk about 'synchronizing the mailbox', and
> apparently they are in an implementation bind because they aren't
> allowed to clear recent flags.
>
> I see your point, but the RFC says "MUST NOT" use STATUS on a selected
> mailbox to check for new mail.

RFC 2060 didn't, and nnimap was written for that RFC.  Ideally, someone
could revise nnimap to make it more modern.

> I think one is supposed to use SEARCH RECENT, but I really  just barely
> understand imap.

Alternatively use something like the IDLE extension.  Or issue a NOOP.

>> I don't have a strong opinion on what the default should be.  If others
>> think it is safer to let it be t by default, by all means change it.
>
> Someone else was having trouble with courier imap, so this seems not to
> just be a dovecot issue.   I'd favor safety over efficiency.   for me
> 'g' takes a while and does 100 STATUS checks, so an unselect hardly
> seems like a big deal.

The unselect happens in a few other situations too, though, which could
trigger other kind of bugs.

Still, let's try it for a while.  I have installed it in CVS.

Heads up: if anyone notices that nnimap has become significantly slower
and/or some new IMAP is showing up that can be traced to this change,
let us know!

/Simon

Re: sluggish IMAP updating?

[Greg Troxel]

Simon Josefsson <simon@josefsson.org> writes:

>> I see your point, but the RFC says "MUST NOT" use STATUS on a selected
>> mailbox to check for new mail.
>
> RFC 2060 didn't, and nnimap was written for that RFC.  Ideally, someone
> could revise nnimap to make it more modern.

sorry - didn't mean to be critical of the code as written, just that
given that the spec has changed it is now not the right behavior.

> Alternatively use something like the IDLE extension.  Or issue a NOOP.

IDLE is probably the coolest thing - that seems to be what most clients
use.

> Heads up: if anyone notices that nnimap has become significantly slower
> and/or some new IMAP is showing up that can be traced to this change,
> let us know!

Thanks for the analysis and making the change - we'll see what happens.

    Greg