nnimap eats all email when IMAP action fails

nnimap eats all email when IMAP action fails

[Gleb Arshinov]

Now that Simon solved my nnimap performance problem I felt inspired to
investigate this other bug.  I can consistently reproduce it at work
running against MS Exchange, and I've seen something similar running
against Linux IMAP servers as well.

The root cause seems that when UID SEARCH UID fails, nnimap ends up
deleting/expiring all articles involed in this UID SEARCH UID command.
UID SEARCH UID seems to fail on large lists of sequences.  In Exchange
case "large" is pretty small :-), but I've seen similar problems on
very large "holey" mailboxes on uw-imapd.

More detailed report is below.

Gleb


Test case:

In 5.10.2 I need to (setq gnus-agent nil) to reproduce this.  I also
notice that nnimap-nov-is-evil is set to true by default.

I have a large ~1200 message nnimap group.  If all articles in the
group are marked !, there is no problems.  If I go into the group and
and mark ~10 non-contiguous articles as E and then enter the group I get:

    Retrieving newsgroup: nnimap+pivia.com:Mail/new/default...
    nnimap: Updating info for nnimap+pivia.com:Mail/new/default...
    nnimap: Updating info for nnimap+pivia.com:Mail/new/default...done
    Fetching headers for nnimap+pivia.com:Mail/new/default...
    nnimap: Retrieving headers... /
    nnimap: Retrieving headers... -
      ...
    nnimap: Retrieving headers... \
    nnimap: Retrieving headers...done
    Fetching headers for nnimap+pivia.com:Mail/new/default...done
    No articles in the group
    nnimap: Setting marks in Mail/new/default...
    nnimap: Setting marks in Mail/new/default...done
    No unread news

If I then C-u RET the group, I see that the artcles marked as E in the
last step are still marked as such, all the other articles (were
marked !) are now O.

Comparing *imap-log* files for no-bug with bug I see that nnimap does:

104 UID SEARCH FLAGGED

    This returns large list of numbers.  I see that articles I marked
    as E are missing from the list, creating more "holes" in the
    sequence.

118 UID SEARCH UID 1:15,17:27,29:105,...   [... is mine]

    It seems that nnimap compressed FLAGGED list into a list of
    sequences and there more sequences involved when some articles are
    marked as E.

118 NO Unspecified error.

    Exchange fails when the list above is large.  In the no-bug case
    it succeds and returns the seemingly the same list as in step 104.
    In no-bug case, this is the end of IMAP activity.

119 UID FETCH 1:15,17:27,29:105,... (UID RFC822.SIZE BODY BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref To Newsgroups)])

    In the bug case, this proceeds to fetch headers from all the
    messages

120 UID STORE 1:15,17:27,29:105... -FLAGS (\Flagged)

* 1 FETCH (FLAGS (\Seen) UID 1)
* 2 FETCH (FLAGS (\Seen) UID 2)
* 3 FETCH (FLAGS (\Seen) UID 3)
....

    We set the flags to equivalent of O    

120 OK STORE completed.
121 STATUS "Mail/new/default" (unseen)
* STATUS Mail/new/default (UNSEEN 0)
121 OK STATUS completed.
122 EXAMINE "Mail/new/default"
* 1210 EXISTS
* 0 RECENT
* FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
* OK [PERMANENTFLAGS ()] Permanent flags
* OK [UIDVALIDITY 34073] UIDVALIDITY value
122 OK [READ-ONLY] EXAMINE completed.
123 CLOSE
123 OK CLOSE completed.

    The end