Re: duplicates with offlineimap+dovecot

[gallego@cri.ensmp.fr (Emilio Jesús Gallego Arias)]

Hi Lars,

Lars Ingebrigtsen <larsi@gnus.org> writes:

> gallego@cri.ensmp.fr (Emilio Jes?FFFAs Gallego Arias) writes:
>
>>> *first press of g*
>>> 21:02:59 [domain] 9020 UID MOVE 4063:4064,4067,4069:4072 "folder1"
>>> * OK [COPYUID 1424475231 4063:4064,4067,4069:4072 26:32] Moved UIDs.
>>> * VANISHED 4062,4066,4068:4072
>>> * 3 RECENT
>>> 9020 OK [HIGHESTMODSEQ 5193] Move completed.
>>
>> I interpret this as "only 4062,4066,4068:4072" got moved.
>
> I can't find anything in the IMAP MOVE standard that says something like
> that.  And it only talks about having to be careful about streaming
> other commands after MOVE as a MOVE may lead to articles being
> renumbered.  And the VANISHED stuff is from unrelated QRESYNC support,
> if I'm reading the QRESYNC RFC correctly.

Note that gnus is currently streaming MOVE commands, however I'm not
sure the RFC means to allow this.

> And, as you see from the VANISHED above there, most of the stuff it says
> is vanished is stuff that we're not even trying to move (4062, 4066).

I'm sorry, I trimmed too much of the log, if you look at my previous
mail you'll see:

> 9019 UID MOVE 4062,4066,4068 "unrelated2"
> 9020 UID MOVE 4063:4064,4067,4069:4072 "folder1"
> * OK [COPYUID 1424475218 4062,4066,4068 376:378] Moved UIDs.
> * OK [COPYUID 1424475231 4063:4064,4067,4069:4072 26:32] Moved UIDs.
> * VANISHED 4062,4066,4068:4072

Thus yeah, we ordered the move of 4062, etc... and the server:

 a) confirms the move for all the messages.
 b) doesn't confirm the expunge from INBOX for all of them!

Thus, it is clear that 4063 was moved but not expunged from the mailbox;
hence the duplicated msg...

> So I don't know what's going on on the server...

I have no clue either but the server is a 100% vanilla Debian Dovecot. I
have this problem with several other servers not administered by me so
if there's a server problem here is not only on my server.

After some more experiments, I believe the bug can be reproduced as
follows: install a Debian stable with Dovecot, then have Gnus to issue
multiple (pipelined) MOVE commands with QRESYNC enabled.

I'd be happy to get in touch with dovecot upstream if we deem it OK.

Regarding the log, note that the RFC says:

   Because a MOVE applies to a set of messages, it might fail partway
   through the set.  Regardless of whether the command is successful in
   moving the entire set, each individual message SHOULD either be moved
   or unaffected.  The server MUST leave each message in a state where
                              ^^^^
   it is in at least one of the source or target mailboxes (no message
   can be lost or orphaned).  The server SHOULD NOT leave any message in
                                         ^^^^^^^^^^
   both mailboxes (it would be bad for a partial failure to result in a
   bunch of duplicate messages).  This is true even if the server
   returns a tagged NO response to the command.

So Dovecot here appears to be violating the "SHOULD NOT"; it effectively
leaves the message in the two mailboxes (even if it notifies us that it
was not VANISHED from INBOX). However:

> RFC6851:

> The server may send EXPUNGE (or VANISHED) responses before the tagged
> response, so the client cannot safely send more commands with message
> sequence number arguments while the server is processing MOVE or UID
> MOVE.

Doesn't that mean that gnus should not stream the two MOVES?

Thanks again for the help!
Regards,
E.