From mboxrd@z Thu Jan  1 00:00:00 1970
X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/49345
Path: main.gmane.org!not-for-mail
From: Matthias Andree <ma@dt.e-technik.uni-dortmund.de>
Newsgroups: gmane.emacs.gnus.general
Subject: maildir in Gnus vs. clock corrections and PID recycling
Date: Sun, 19 Jan 2003 01:57:11 +0100
Sender: owner-ding@hpc.uh.edu
Message-ID: <m3znpy7y8o.fsf@merlin.emma.line.org>
NNTP-Posting-Host: main.gmane.org
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Trace: main.gmane.org 1042937969 28970 80.91.224.249 (19 Jan 2003 00:59:29 GMT)
X-Complaints-To: usenet@main.gmane.org
NNTP-Posting-Date: Sun, 19 Jan 2003 00:59:29 +0000 (UTC)
Return-path: <owner-ding@hpc.uh.edu>
Original-Received: from malifon.math.uh.edu ([129.7.128.13])
	by main.gmane.org with esmtp (Exim 3.35 #1 (Debian))
	id 18a3o8-0007X7-00
	for <ding-account@gmane.org>; Sun, 19 Jan 2003 01:59:28 +0100
Original-Received: from sina.hpc.uh.edu ([129.7.128.10] ident=lists)
	by malifon.math.uh.edu with esmtp (Exim 3.20 #1)
	id 18a3mL-0004Bn-00; Sat, 18 Jan 2003 18:57:37 -0600
Original-Received: by sina.hpc.uh.edu (TLB v0.09a (1.20 tibbs 1996/10/09 22:03:07)); Sat, 18 Jan 2003 18:58:33 -0600 (CST)
Original-Received: from sclp3.sclp.com (sclp3.sclp.com [66.230.238.2])
	by sina.hpc.uh.edu (8.9.3/8.9.3) with SMTP id SAA19767
	for <ding@hpc.uh.edu>; Sat, 18 Jan 2003 18:58:23 -0600 (CST)
Original-Received: (qmail 20687 invoked by alias); 19 Jan 2003 00:57:20 -0000
Original-Received: (qmail 20682 invoked from network); 19 Jan 2003 00:57:19 -0000
Original-Received: from p50876f63.dip.t-dialin.net (HELO m2a2.dyndns.org) (postfix@80.135.111.99)
  by 66.230.238.6 with SMTP; 19 Jan 2003 00:57:19 -0000
Original-Received: by merlin.emma.line.org (Postfix, from userid 500)
	id 14FFD7514B; Sun, 19 Jan 2003 01:57:12 +0100 (CET)
Original-To: ding@gnus.org
User-Agent: Gnus/5.090013 (Oort Gnus v0.13) Emacs/21.2 (i586-suse-linux)
Precedence: list
X-Majordomo: 1.94.jlt7
Xref: main.gmane.org gmane.emacs.gnus.general:49345
X-Report-Spam: http://spam.gmane.org/gmane.emacs.gnus.general:49345

Hi,

there's been a lot of discussion rolling through MTA lists recently,
Postfix, qmail, Courier namely, about PID reuse by the kernel and
maildir deliveries and maildir file name collisions.

Some background: maildir is a protocol invented by DJB, which is meant
to provide for locking-free mail delivery even across NFS. It delivers
to one mail per file. The process of mail delivery includes writing to a
file tmp/<time>.<unique>.<hostname> which is then linked to
new/<time>.<unique>.<hostname>. Mail reading means moving the file from
new/... to cur/<time>.<unique>.<hostname>,<flags>.

Evidently, when <unique> is not unique or <time> steps back, there may
be a collision between file names in new/ and cur/ sharing the same
"<time>.<unique>.<hostname>" part. This has been shown to cause mail
loss with both mutt and qmail-pop3d, and applications in cooperation
with Courier have shown this problem to be very real, so this is not
theoretical.

What does nnmaildir.el and possibly other maildir code in Gnus do when
it is about to move a mail from new/ to cur/? Does it rely on uniqueness
and will it -- as mutt -- accidentally kill an already-present file in
cur/? Or will it detect the collision and create a new file name for the
cur/ file, to avoid mail loss?



Further background:

The first edition of the maildir protocol documentation, maildir(5) as
shipped with qmail-1.0x, x<=3, was incomplete and used the process ID
for <unique>, which is insufficient because the PID can be recycled by
the kernel within one second.

Mate Wierdl has code to let the PID counter cycle in well below 0.2
seconds on a GHz-class IA32 Linux machine.

Some operating systems randomize their process IDs and may have even
shorter cycles between the same PID being given to a process.

The supposedly later http://cr.yp.to/proto/maildir.html refines the
documentation, and demands that the middle part does not recur within
one second, which is still insufficient given that a computer clock
might be stepped back, for example when ntpd resynchronizes after a
network outage and the clock offset is beyond the slew/step threshold.

It seems that Postfix, Courier and Maildrop will settle on using the
tmp/* file inode number for the <unique> part in new/, so these
applications will rename from tmp/time.pid_count.hostname to
new/time.inode.hostname. Dan J. Bernstein has not mentioned any fix, he
claimed qmail-pop3d was safe, but Mate Wierdl posted counter evidence to
the qmail mailing list. Sendmail 8.12 doesn't to Maildir deliveries. I'm
unsure about Exim and Procmail.

Feel free to forward this mail to the procmail and exim mailing lists
and report about the response.

-- 
Matthias Andree