nnimap vs dbmail

nnimap vs dbmail

[James Cloos]

I'm looking into converting over to using imap as the backstore for my
mail and am looking at using dbmail for that.

I'm seeing some buglets in nnimap.  

As an example, the unread article counts in the Group buffer are almost
always incorrect.  And errors often get printed to the minibuffer.

One issue is that dbmail gives each article a server-unique UID rather
than a just user-unique or folder-unique.  So they are never contiguous.
nnimap seems to expect contiguous UIDs and goes searching for them,
causing IMAP errors.

As an example, I recorded this imap exchange today via tshark:

,----< 149 SELECT "ROOT" >
| * 4 EXISTS
| * 0 RECENT
| * FLAGS (\Seen \Answered \Deleted \Flagged \Draft)
| * OK [PERMANENTFLAGS (\Seen \Answered \Deleted \Flagged \Draft)]
| * OK [UIDNEXT 3669] Predicted next UID
| * OK [UIDVALIDITY 13] UID value
| 149 OK [READ-WRITE] SELECT completed
`----

OK so far.

,----< 150 UID SEARCH UNSEEN UNDELETED >
| * SEARCH 
| 150 OK SEARCH completed
`----

None unseen and none undeleted.

,----< 151 UID SEARCH SEEN >
| * SEARCH 3662 3664 3666 3668
| 151 OK SEARCH completed
`----

4 seen articles, with UIDs 3662, 3664, 3666 and 3668.

,----< 152 UID SEARCH FLAGGED >
| * SEARCH 
| 152 OK SEARCH completed
`----

None flagged.

,----< 153 UID SEARCH ANSWERED >
| * SEARCH 
| 153 OK SEARCH completed
`----

None answered.

,----< 154 UID SEARCH RECENT >
| * SEARCH 
| 154 OK SEARCH completed
`----

None recent.

,----< 155 UID FETCH 3665,3667 (UID RFC822.SIZE BODY BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref To Newsgroups)]) >
| 155 BAD invalid message range specified
`----

So why fetch 3665 and 3667 if they were not listed in any of the UID searches?

,----< 156 UID FETCH 3665,3667 (UID RFC822.SIZE BODY BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref To Newsgroups)]) >
| 156 BAD invalid message range specified
`----

Twice, even.

,----< 157 UID STORE 3665,3667 +FLAGS (\Seen) >
| 157 BAD invalid message range specified
`----

And dbmail doesn't like storing flags for non-extant UIDs.

,----< 158 STATUS "ROOT" (UNSEEN) >
| * STATUS "ROOT" (UNSEEN 0)
| 158 OK STATUS completed
`----

,----< 159 UNSELECT >
| 159 OK UNSELECT completed
`----

This time the errors did not prevent me from seeing new mail, but that
isn't always the case.  Some new mail is inaccessible in gnus because of
such errors.  A test shows that one of those errors is probably due to
this line in list/imap.el:

,----[ from gnus/lisp/imap.el (imap-parse-body) ]
| (assert (eq (char-after) ?\)) nil "In imap-parse-body")
`----

Are those (assert)s necessary?  They effectively lose mail.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: nnimap vs dbmail

[Simon Josefsson]

James Cloos <cloos+math_uh-ding@jhcloos.com> writes:

> I'm looking into converting over to using imap as the backstore for my
> mail and am looking at using dbmail for that.
>
> I'm seeing some buglets in nnimap.  
>
> As an example, the unread article counts in the Group buffer are almost
> always incorrect.  And errors often get printed to the minibuffer.
>
> One issue is that dbmail gives each article a server-unique UID rather
> than a just user-unique or folder-unique.  So they are never contiguous.
> nnimap seems to expect contiguous UIDs and goes searching for them,
> causing IMAP errors.

Actually, it is Gnus that assumes contiguous article numbers.  When
there are "holes" in the ranges, the Group buffer unread article count
estimate (yes, it is documented as an estimate) will be wrong.  This
happens for nntp, nnml etc too.

> ,----< 155 UID FETCH 3665,3667 (UID RFC822.SIZE BODY BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref To Newsgroups)]) >
> | 155 BAD invalid message range specified
> `----
>
> So why fetch 3665 and 3667 if they were not listed in any of the UID
> searches?

They may be UNSEEN DELETED, but I agree that it should be able to figure
out that these do not exist.  I suspect that the reason nnimap asked
about this is that Gnus asked about those articles.  Running Gnus and
nnimap under edebug will help to understand why information about those
articles are requested.

>
> ,----< 156 UID FETCH 3665,3667 (UID RFC822.SIZE BODY BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref To Newsgroups)]) >
> | 156 BAD invalid message range specified
> `----
>
> Twice, even.

Huh, that's weird.  It might indicate that something non-normal (i.e., a
bug) is happening.

> ,----< 157 UID STORE 3665,3667 +FLAGS (\Seen) >
> | 157 BAD invalid message range specified
> `----
>
> And dbmail doesn't like storing flags for non-extant UIDs.

RFC 3501 says:

6.4.8.  UID Command
...
      A non-existent unique identifier is ignored without any error
      message generated.  Thus, it is possible for a UID FETCH command
      to return an OK without any data or a UID COPY or UID STORE to
      return an OK without performing any operations.

I'd say this is a dbmail bug.

> This time the errors did not prevent me from seeing new mail, but that
> isn't always the case.  Some new mail is inaccessible in gnus because of
> such errors.  A test shows that one of those errors is probably due to
> this line in list/imap.el:
>
> ,----[ from gnus/lisp/imap.el (imap-parse-body) ]
> | (assert (eq (char-after) ?\)) nil "In imap-parse-body")
> `----
>
> Are those (assert)s necessary?  They effectively lose mail.

Can you quote the content of the ' *nnimap* foo' buffer when that
happens?  Alternatively, try '(setq imap-log t)' and quote the contents
from *imap-log* related to that parse assertion.

The reason for the assert is that imap.el cannot parse what the server
is sending.  I think there are only two reasons for that happening:
imap.el is buggy or the server is buggy.  The effects of either will be
that it appears as if a message is missing (or even worse consequences).
I'm not sure what could be done to avoid this.

/Simon

Re: nnimap vs dbmail

[James Cloos]

I have to take some time to write a full reply, but for now I'll just
note that all messages which have a message/rfc822 attachment trigger
the "In imap-parse-body 2" (assert) -- I have a corpus of examples if
anyone needs some -- and if I comment out the (asserts) some of those
emails will work in the *Article* buffer but w/ empty/default data in
the *Summary* buffer and some others do not show up at all.  In every
case, though, mutt has no problems seeing, summarizing and displaying
all of the messages.

I'll followup again with a full replay as soon as I can write it.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: nnimap vs dbmail

[Simon Josefsson]

On 24 maj 2007, at 21.07, James Cloos wrote:

> I have to take some time to write a full reply, but for now I'll just
> note that all messages which have a message/rfc822 attachment trigger
> the "In imap-parse-body 2" (assert) -- I have a corpus of examples if
> anyone needs some -- and if I comment out the (asserts) some of those
> emails will work in the *Article* buffer but w/ empty/default data in
> the *Summary* buffer and some others do not show up at all.  In every
> case, though, mutt has no problems seeing, summarizing and displaying
> all of the messages.
>
> I'll followup again with a full replay as soon as I can write it.

The interesting part for debugging this is the contents of the *imap- 
log* buffer.  The articles themselves probably isn't sufficient to  
reproduce it.

/Simon