Request for "maildir" support

[tom@smart.ruhr.de (Thomas Neumann)]

A guy named D.J. Bernstein <djb@koobera.math.uic.edu> is currently
developing "qmail", a new MTA. qmail features a new mailbox
format called "maildir", which is different from std. UNIX mailbox
files, different from MH folders and different from whatever you
care to name. Nevertheless, D.J.'s maildir is rather well designed
and potentially much more reliable than most existing formats, and
I think it would be nice if Red Gnus supports it as a mail format.

I'm appending a manpage (maildir.5) that describes the format, as
well as some general qmail blurb.

I'm sure D.J. would be more than willing to help in case there are
any questions regading maildir semantics.

-t


*** BLURB ***

qmail is a secure, reliable, efficient, simple message transfer agent.
It is meant as a replacement for the entire sendmail-binmail system on
typical Internet-connected UNIX hosts.

Secure: Security isn't just a goal, but an absolute requirement. Mail
delivery is critical for users; it cannot be turned off, so it must be
completely secure. (This is why I started writing qmail: I was sick of
the security holes in sendmail and other MTAs.)

Reliable: qmail's straight-paper-path philosophy guarantees that a
message, once accepted into the system, will never be lost. qmail also
supports maildir, a new, super-reliable user mailbox format. Maildirs,
unlike mbox files and mh folders, won't be corrupted if the system
crashes during delivery. Even better, not only can a user safely read
his mail over NFS, but any number of NFS clients can deliver mail to him
at the same time.

Efficient: On a Pentium under BSD/OS, qmail can easily sustain 200000
local messages per day---that's separate messages injected and delivered
to mailboxes in a real test! Although remote deliveries are inherently
limited by the slowness of DNS and SMTP, qmail overlaps 20 simultaneous
deliveries by default, so it zooms quickly through mailing lists. (This
is why I finished qmail: I had to get a big mailing list set up.)

Simple: qmail is vastly smaller than any other Internet MTA. Some
reasons why: (1) Other MTAs have separate forwarding, aliasing, and
mailing list mechanisms. qmail has one simple forwarding mechanism that
lets users handle their own mailing lists. (2) Other MTAs offer a
spectrum of delivery modes, from fast+unsafe to slow+queued. qmail-send
is instantly triggered by new items in the queue, so the qmail system
has just one delivery mode: fast+queued. (3) Other MTAs include, in
effect, a specialized version of inetd that watches the load average.
qmail's design inherently limits the machine load, so qmail-smtpd can
safely run from your system's inetd.

Replacement for sendmail: qmail supports host and user masquerading,
full host hiding, virtual domains, null clients, list-owner rewriting,
relay control, double-bounce recording, arbitrary RFC 822 address lists,
cross-host mailing list loop detection, per-recipient checkpointing,
etc. In short, it's up to speed on modern MTA features. (If you need
another feature, let me know.) qmail includes a drop-in ``sendmail''
wrapper so that it will be used transparently by your current UAs.


*** maildir.5 ***

.TH maildir 5
.SH "NAME"
maildir \- directory for incoming mail messages
.SH "INTRODUCTION"
.I maildir
is a structure for
directories of incoming mail messages.
It solves the reliability problems that plague
.I mbox
files and
.I mh
folders.
.SH "RELIABILITY ISSUES"
A machine may crash while it is delivering a message.
For both
.I mbox
files and
.I mh
folders this means that the message will be silently truncated.
Even worse: for
.I mbox
format, if the message is truncated in the middle of a line,
it will be silently joined to the next message.
The mail transport agent will try again later to deliver the message,
but it is unacceptable that a corrupted message should show up at all.
In
.IR maildir ,
every message is guaranteed complete upon delivery.

A machine may have two programs simultaneously delivering mail
to the same user.
The
.I mbox
and
.I mh
formats require the programs to update a single central file.
If the programs do not use some locking mechanism,
the central file will be corrupted.
There are several
.I mbox
and
.I mh
locking mechanisms,
none of which work portably and reliably.
In contrast, in
.IR maildir ,
no locks are ever necessary.
Different delivery processes never touch the same file.

A user may try to delete messages from his mailbox at the same
moment that the machine delivers a new message.
For
.I mbox
and
.I mh
formats, the user's mail-reading program must know
what locking mechanism the mail-delivery programs use.
In contrast, in
.IR maildir ,
any delivered message
can be safely updated or deleted by a mail-reading program.

Many sites use Sun's 
.B Network F\fPa\fBil\fPur\fBe System
(NFS),
presumably because the operating system vendor does not offer
anything else.
NFS exacerbates all of the above problems.
Some NFS implementations don't provide
.B any
reliable locking mechanism.
With 
.I mbox
and
.I mh
formats,
if two machines deliver mail to the same user,
or if a user reads mail anywhere except the delivery machine,
the user's mail is at risk.
.I maildir
works without trouble over NFS.
.SH "THE MAILDIR STRUCTURE"
A directory in
.I maildir
format has three subdirectories,
all on the same filesystem:
.BR tmp ,
.BR new ,
and
.BR cur .

Each file in
.B new
is a newly delivered mail message.
The modification time of the file is the delivery date of the message.
The message is delivered
.I without
an extra UUCP-style

.EX
     From djb Mon Jul  3 01:55:29 1995
.EE

line,
.I without
any
.B >From
quoting,
and
.I without
an extra blank line at the end.
The message is probably in RFC 822 format,
but it could contain arbitrary binary data.
It might not even end with a newline.

Files in
.B cur
are just like files in
.BR new .
The big difference is that files in
.B cur
are no longer new mail:
they have been seen by the user's mail-reading program.
.SH "HOW A MESSAGE IS DELIVERED"
The
.B tmp
directory is used to ensure reliable delivery,
as discussed here.

A program delivers a mail message in six steps.
First, it
.B chdir()\fPs
to the
.I maildir
directory.
Second, it 
.B stat()s
the name
.BR tmp/\fItime.pid.host ,
where
.I time
is the number of seconds since the beginning of 1970 GMT,
.I pid
is the program's process ID,
and
.I host
is the host name.
Third, if
.B stat()
returned anything other than ENOENT,
the program sleeps for two seconds, updates
.IR time ,
and tries the
.B stat()
again, a limited number of times.
Fourth, the program
creates
.BR tmp/\fItime.pid.host .
Fifth, the program
.I NFS-writes
the message to the file.
Sixth, the program
.BR link() s
the file to
.BR new/\fItime.pid.host .
At that instant the message has been successfully delivered.

The delivery program is required to start a 24-hour timer before
creating
.BR tmp/\fItime.pid.host ,
and to abort the delivery
if the timer expires.
Upon error, timeout, or normal completion,
the delivery program may attempt to
.B unlink()
.BR tmp/\fItime.pid.host .

.I NFS-writing
means
(1) as usual, checking the number of bytes returned from each
.B write()
call;
(2) calling
.B fsync()
and checking its return value;
(3) calling
.B close()
and checking its return value.
(Standard NFS implementations handle
.B fsync()
incorrectly
but make up for it by abusing
.BR close() .)
.SH "HOW A MESSAGE IS READ"
A mail reader operates as follows.

It looks through the
.B new
directory for new messages.
Say there is a new message,
.BR new/\fItime.pid.host .
The reader may freely display the contents of
.BR new/\fItime.pid.host ,
delete
.BR new/\fItime.pid.host ,
or rename
.B new/\fItime.pid.host
as
.BR cur/\fItime.pid.host:info .
The meaning of
.I info
is up to the mail reader.

The reader is also expected to look through the
.B tmp
directory and to clean up any old files found there.
A file in
.B tmp
may be safely removed if it
has not been accessed in 36 hours.
.SH "ENVIRONMENT VARIABLES"
Mail readers supporting
.I maildir
use the
.B MAILDIR
environment variable
as the name of the user's primary mail directory.
.SH "SEE ALSO"
qmail-alias(8)