maildir in Gnus vs. clock corrections and PID recycling

[Matthias Andree <ma@dt.e-technik.uni-dortmund.de>]

Hi,

there's been a lot of discussion rolling through MTA lists recently,
Postfix, qmail, Courier namely, about PID reuse by the kernel and
maildir deliveries and maildir file name collisions.

Some background: maildir is a protocol invented by DJB, which is meant
to provide for locking-free mail delivery even across NFS. It delivers
to one mail per file. The process of mail delivery includes writing to a
file tmp/<time>.<unique>.<hostname> which is then linked to
new/<time>.<unique>.<hostname>. Mail reading means moving the file from
new/... to cur/<time>.<unique>.<hostname>,<flags>.

Evidently, when <unique> is not unique or <time> steps back, there may
be a collision between file names in new/ and cur/ sharing the same
"<time>.<unique>.<hostname>" part. This has been shown to cause mail
loss with both mutt and qmail-pop3d, and applications in cooperation
with Courier have shown this problem to be very real, so this is not
theoretical.

What does nnmaildir.el and possibly other maildir code in Gnus do when
it is about to move a mail from new/ to cur/? Does it rely on uniqueness
and will it -- as mutt -- accidentally kill an already-present file in
cur/? Or will it detect the collision and create a new file name for the
cur/ file, to avoid mail loss?



Further background:

The first edition of the maildir protocol documentation, maildir(5) as
shipped with qmail-1.0x, x<=3, was incomplete and used the process ID
for <unique>, which is insufficient because the PID can be recycled by
the kernel within one second.

Mate Wierdl has code to let the PID counter cycle in well below 0.2
seconds on a GHz-class IA32 Linux machine.

Some operating systems randomize their process IDs and may have even
shorter cycles between the same PID being given to a process.

The supposedly later http://cr.yp.to/proto/maildir.html refines the
documentation, and demands that the middle part does not recur within
one second, which is still insufficient given that a computer clock
might be stepped back, for example when ntpd resynchronizes after a
network outage and the clock offset is beyond the slew/step threshold.

It seems that Postfix, Courier and Maildrop will settle on using the
tmp/* file inode number for the <unique> part in new/, so these
applications will rename from tmp/time.pid_count.hostname to
new/time.inode.hostname. Dan J. Bernstein has not mentioned any fix, he
claimed qmail-pop3d was safe, but Mate Wierdl posted counter evidence to
the qmail mailing list. Sendmail 8.12 doesn't to Maildir deliveries. I'm
unsure about Exim and Procmail.

Feel free to forward this mail to the procmail and exim mailing lists
and report about the response.

-- 
Matthias Andree