Re: Request for "maildir" support

Re: Request for "maildir" support

[eichin]

Oops. I was partly confused, h.b.furuseth had the right of it:

hbf>> No.  Note that the file is written to tmp/.  Only when it's complete is
hbf>> it linked into new/.

In any case, you don't even see the name until the file is
complete. This works with UFS and NFS. (I think it breaks under AFS,
actually, since it can't handle hardlinks across directories, but AFS
is stupid in a lot of other ways too :-)

Re: Request for "maildir" support

[Mike Williams]

Lars> So it would seem that the maildir format is OK.  Making the mail
Lars> backends understand its format should be no problem.  I've added this
Lars> to the Red Gnus todo list.

  More proof that Lars is simply the most progressive author of mail/news
  software on the net. What a can-do guy!

  [ And no, I'm not being sarcastic ...it just sounds like that due to
    lifelong habit. ]

  Thanks for making the world a better place, Lars.

  (setq gush-minor-mode nil)

    ------------------------------------------------------------------
	    Mike Williams, Support & Development, Survey System,
	       Department of Survey & Land Information, NZ.
	     Email: <mikew@dosli.govt.nz>  Fax: +64 4 495 8466

Re: Request for "maildir" support

[Lars Magne Ingebrigtsen]

Hallvard B Furuseth <h.b.furuseth@usit.uio.no> writes:

> >        Fifth, the program _\bN_\bF_\bS_\b-_\bw_\br_\bi_\bt_\be_\bs the mes-
> >        sage to the file. 
> > 
> > If the file is Huge, and Gnus starts reading/moving this file before
> > it has arrived totally, won't this potentially mean that Gnus might
> > get an incomplete version of the file.
> 
> No.  Note that the file is written to tmp/.  Only when it's complete is
> it linked into new/.

Whoops.  My eyes kinda skidded past item six there.  

So it would seem that the maildir format is OK.  Making the mail
backends understand its format should be no problem.  I've added this
to the Red Gnus todo list.

-- 
  "Yes.  The journey through the human heart 
     would have to wait until some other time."

Re: Request for "maildir" support

[eichin]

> So I guess it isn't as simple as the manual says to read the "new"
> directory.  The reader has to check the "cur" directory that the

Umm, if I remember correctly, the file is written to cur and then
renamed into new, so it doesn't appear in new until it's complete...

Re: Request for "maildir" support

[Hallvard B Furuseth]

>        Fifth, the program _\bN_\bF_\bS_\b-_\bw_\br_\bi_\bt_\be_\bs the mes-
>        sage  to the file. 
> 
> If the file is Huge, and Gnus starts reading/moving this file before
> it has arrived totally, won't this potentially mean that Gnus might
> get an incomplete version of the file.

No.  Note that the file is written to tmp/.  Only when it's complete is
it linked into new/.

> So I guess it isn't as simple as the manual says to read the "new"
> directory.  The reader has to check the "cur" directory that the
> corresponding file there has been removed before it attempts to deal
> with a file in the "new" directory.

What corresponding file in cur/?  All messages get different filenames.
Unless you run the machine clock backwards or have two hosts with the
same name.


Regards,

Hallvard

Re: Request for "maildir" support

[Thomas Neumann]

In article  Stainless Steel Rat <ratinox@ccs.neu.edu> writes:

> I would suggest someone inform this person about Eric Allman's v8
> sendmail, 8.7, which is about as secure and reliable as you can get
> these days.  And, when a weakness is found a patch is made available
> immediately.
> 
> [...]

Please note that I just included the text verbatim from the qmail
distribution. I did not write it, and I did not intent to
start a discussion on particular MTA technologie issues on the ding
mailinglist. I included the text as a means of giving some information
of what I was talking about when I said "qmail". Nothing more.
If you want to discuss qmail vs. sendmail-8, I suggest you contact
Mr. Bernstein directly.

-t

Re: Request for "maildir" support

[Lars Magne Ingebrigtsen]

tom@smart.ruhr.de (Thomas Neumann) writes:

> A guy named D.J. Bernstein <djb@koobera.math.uic.edu> is currently
> developing "qmail", a new MTA. qmail features a new mailbox
> format called "maildir", which is different from std. UNIX mailbox
> files, different from MH folders and different from whatever you
> care to name. 

Well, it isn't even an mbox format.  

Reading through the manual I was struck by how absolutely hype-free
and down-to-earth it was.  :-)  He should definitely learn how to
overcome his modesty.  Really.

I think the format he's chosen for the incoming mail is OK, and I
think Gnus should definitely support it if more than 1 site decides to
use qmail.  It'll have to wait until Red Gnus, though.

One thing that puzzles me from the delivery section:

       Fifth, the program _\bN_\bF_\bS_\b-_\bw_\br_\bi_\bt_\be_\bs the mes-
       sage  to the file. 

If the file is Huge, and Gnus starts reading/moving this file before
it has arrived totally, won't this potentially mean that Gnus might
get an incomplete version of the file.

So I guess it isn't as simple as the manual says to read the "new"
directory.  The reader has to check the "cur" directory that the
corresponding file there has been removed before it attempts to deal
with a file in the "new" directory.

-- 
  "Yes.  The journey through the human heart 
     would have to wait until some other time."

Re: Request for "maildir" support

[eichin]

-----BEGIN PGP SIGNED MESSAGE-----

Date: 4 Mar 1996

There are *so* many misconceptions (not just disagreements of opinion,
which I could easily abide, but factual errors) that I feel compelled
to respond, as a qmail beta tester, and a security professional. (I'd
actually like to see maildir support in ding, even though I'll most
likely continue to only use movemail-based POP support -- it would be
nice to have at least one *reliable* non-pop delivery mechanism,
regardless of what mailer implements it.)

> So much for standards :).
When standards are broken, it's long past time to develop new ones :-)

>I would suggest someone inform this person about Eric Allman's v8
>sendmail, 8.7, which is about as secure and reliable as you can get

Yep. Unfortunately, "secure as you can get", with sendmail, is "not
at all." Did you see last weeks "dns spoofing" vulnerability problem?
It's yet another in a long running series of the kind of problem you
have when you *design* a program to have back doors (yes, I mean
that. The sendmail "DEBUG" mode exploited by the Morris Internet worm,
back in 1988, was a *deliberate* inclusion, because Allman couldn't
get root access to the machine he was maintaining sendmail on
originally... and the feature didn't get completely rooted out until
years later...)

> Ahem.  This is bullshit.  NFS is infamous for it's inability to do
> exactly this, and no matter what you do to get around it you will still

Turns out that creating multics-style timestamp-based names which are
unique in and of themselves works fine - you don't have collisions
because you're *never* working on the same file. It *is* possible; you
just have to be more clever than people have in the past...

> a modicum of clues running large mailing lists uses a caching MTA like
> v8 sendmail with bulk_mailer, which I have found to work exceptionally

Well, no. vger.rutgers.edu is running zmailer, because sendmail just
couldn't handle the linux-* mailing lists, even with a multi-cpu
machine with 256M of RAM... David Miller has to work quite closely
with the author of zmailer to get even that to work, but sendmail
isn't even in the running.

> Users handle their own mailing lists != "simple", from my experience
> both as a Majordomo list and server administrator.

Perhaps that's because exsisting techniques are primitive? He really
has come up with some quite clever things -- we may need some more
practice with them, but what he's got does work.

> v8 sendmail does it internally.  All you need to do is configure a
> couple of options in sendmail.cf

That *alone* is a condemnation :-) But it also means that you're stuck
with sendmail's precise scheduling techniques; you don't have nearly
the control that you get with a modern inetd and qmail.

> And then there's simple fact that in a "unified" system if one little
> thing breaks the whole thing goes down, whereas in an "modular" system

You mean the way that if you make a single typo in your sendmail.cf
everything goes down? :-) qmail *is* quite modular. From your
comments, I think you'd be surprised with what you'd find by looking
at it.

> This whole thing reminds me a lot of SMail and how it absolutely failed
> in many ways to "fix" perceived "problems" in sendmail.  Not that I am

Smail didn't even handle *obvious* things like MX records, at least at
first. qmail is written by someone who reads and has written RFC's...

> with an MTA that is written and maintained by someone I know knows to
> write and support a reliable, secure MTA.

I wonder who that could be. Sure isn't Eric Allman :->

If you'd like to know more, you could join the qmail beta test; the
release actually includes a number of detailed explanations of the
issues involved.

-----BEGIN PGP SIGNATURE-----
Version: 2.6.2
Comment: Processed by Mailcrypt 3.4beta, an Emacs/PGP interface

iQCVAwUBMTsgl1y/7Dlmzom3AQHi6gP7BNtPsxZEN9Gw4j90VC/LChBDcg/dJrXF
H6kUl8wbfrgFAGQQixyd9YZpNDTiLZ9M9DP+NceO56kcFnxqeLj1LCbi3v7+5Chf
cyqdRo+M6BI3DcNecGHSt/2U7MhAxME6o89XHcHaxmgU89Bc2xiy5l30iM9akDP5
jGXMwm6ix3w=
=fpfr
-----END PGP SIGNATURE-----

Re: Request for "maildir" support

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "TN" == Thomas Neumann <tom@smart.ruhr.de> writes:

TN> qmail features a new mailbox format called "maildir", which is
TN> different from std. UNIX mailbox files, different from MH folders
TN> and different from whatever you care to name.

So much for standards :).

[...]

TN> Secure: Security isn't just a goal, but an absolute requirement. Mail
TN> delivery is critical for users; it cannot be turned off, so it must be
TN> completely secure. (This is why I started writing qmail: I was sick of
TN> the security holes in sendmail and other MTAs.)

I would suggest someone inform this person about Eric Allman's v8
sendmail, 8.7, which is about as secure and reliable as you can get
these days.  And, when a weakness is found a patch is made available
immediately.

[...]

TN> Even better, not only can a user safely read his mail over NFS, but
TN> any number of NFS clients can deliver mail to him at the same time.

Ahem.  This is bullshit.  NFS is infamous for it's inability to do
exactly this, and no matter what you do to get around it you will still
have file locking problems somewhere if you attempt to read the same
mailbox or file from two different points.

TN> Efficient: On a Pentium under BSD/OS, qmail can easily sustain
TN> 200000 local messages per day---that's separate messages injected
TN> and delivered to mailboxes in a real test!

Now, how many Intel boxes are going to have 1000 users on it?  Or who in
their right mind uses an Intel box for a mail hub?  Besides, anyone with
a modicum of clues running large mailing lists uses a caching MTA like
v8 sendmail with bulk_mailer, which I have found to work exceptionally
well.

[...]

TN> Simple: qmail is vastly smaller than any other Internet MTA. Some
TN> reasons why: (1) Other MTAs have separate forwarding, aliasing, and
TN> mailing list mechanisms. qmail has one simple forwarding mechanism
TN> that lets users handle their own mailing lists.

Users handle their own mailing lists != "simple", from my experience
both as a Majordomo list and server administrator.

TN> (2) Other MTAs offer a spectrum of delivery modes, from fast+unsafe
TN> to slow+queued. qmail-send is instantly triggered by new items in
TN> the queue, so the qmail system has just one delivery mode:
TN> fast+queued.

Besides, v8 sendmail gives you the best of both worlds, allowing fast
and safe "immediate" delivery or queued based on machine load.

TN> (3) Other MTAs include, in effect, a specialized version of inetd
TN> that watches the load average.

v8 sendmail does it internally.  All you need to do is configure a
couple of options in sendmail.cf

And then there's simple fact that in a "unified" system if one little
thing breaks the whole thing goes down, whereas in an "modular" system
if the mailing list manager breaks you can take it down without entirely
disabling mail.

TN> qmail's design inherently limits the machine load, so qmail-smtpd
TN> can safely run from your system's inetd.

Try this when your machine's load is up around 6, you have "fast+unsafe"
as the default, and you get about 200 messages from a batch somewhere,
and honestly tell me that your machine isn't going to throttle itself.

This whole thing reminds me a lot of SMail and how it absolutely failed
in many ways to "fix" perceived "problems" in sendmail.  Not that I am
saying that the format should not be supported, just that I will stick
with an MTA that is written and maintained by someone I know knows to
write and support a reliable, secure MTA.

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3
Charset: noconv

iQCVAwUBMTsKcZ6VRH7BJMxHAQGNkgP/X3dA5pwuvsJnl5p6vsJxwVvlpBQM9igE
lEAXBfhYH1OkvGB8+kzlqy5xic1IGPtioyq2tTu2pLCj+AO5ZiJL+FEYkIsgh8nW
+t7yiH221nc3m+l0REX/5emAeiVWu3fJo0V32L/whjOKKnIC6lGSBQ69OfY4MSIU
rcS3ogfjIEg=
=mHV8
-----END PGP SIGNATURE-----
-- 
Rat <ratinox@ccs.neu.edu>          \ When not in use, Happy Fun Ball should be
PGP Public Key: Ask for one today!  \ returned to its special container and
http://www.ccs.neu.edu/home/ratinox/ \ kept under refrigeration.

Request for "maildir" support

[Thomas Neumann]

A guy named D.J. Bernstein <djb@koobera.math.uic.edu> is currently
developing "qmail", a new MTA. qmail features a new mailbox
format called "maildir", which is different from std. UNIX mailbox
files, different from MH folders and different from whatever you
care to name. Nevertheless, D.J.'s maildir is rather well designed
and potentially much more reliable than most existing formats, and
I think it would be nice if Red Gnus supports it as a mail format.

I'm appending a manpage (maildir.5) that describes the format, as
well as some general qmail blurb.

I'm sure D.J. would be more than willing to help in case there are
any questions regading maildir semantics.

-t


*** BLURB ***

qmail is a secure, reliable, efficient, simple message transfer agent.
It is meant as a replacement for the entire sendmail-binmail system on
typical Internet-connected UNIX hosts.

Secure: Security isn't just a goal, but an absolute requirement. Mail
delivery is critical for users; it cannot be turned off, so it must be
completely secure. (This is why I started writing qmail: I was sick of
the security holes in sendmail and other MTAs.)

Reliable: qmail's straight-paper-path philosophy guarantees that a
message, once accepted into the system, will never be lost. qmail also
supports maildir, a new, super-reliable user mailbox format. Maildirs,
unlike mbox files and mh folders, won't be corrupted if the system
crashes during delivery. Even better, not only can a user safely read
his mail over NFS, but any number of NFS clients can deliver mail to him
at the same time.

Efficient: On a Pentium under BSD/OS, qmail can easily sustain 200000
local messages per day---that's separate messages injected and delivered
to mailboxes in a real test! Although remote deliveries are inherently
limited by the slowness of DNS and SMTP, qmail overlaps 20 simultaneous
deliveries by default, so it zooms quickly through mailing lists. (This
is why I finished qmail: I had to get a big mailing list set up.)

Simple: qmail is vastly smaller than any other Internet MTA. Some
reasons why: (1) Other MTAs have separate forwarding, aliasing, and
mailing list mechanisms. qmail has one simple forwarding mechanism that
lets users handle their own mailing lists. (2) Other MTAs offer a
spectrum of delivery modes, from fast+unsafe to slow+queued. qmail-send
is instantly triggered by new items in the queue, so the qmail system
has just one delivery mode: fast+queued. (3) Other MTAs include, in
effect, a specialized version of inetd that watches the load average.
qmail's design inherently limits the machine load, so qmail-smtpd can
safely run from your system's inetd.

Replacement for sendmail: qmail supports host and user masquerading,
full host hiding, virtual domains, null clients, list-owner rewriting,
relay control, double-bounce recording, arbitrary RFC 822 address lists,
cross-host mailing list loop detection, per-recipient checkpointing,
etc. In short, it's up to speed on modern MTA features. (If you need
another feature, let me know.) qmail includes a drop-in ``sendmail''
wrapper so that it will be used transparently by your current UAs.


*** maildir.5 ***

.TH maildir 5
.SH "NAME"
maildir \- directory for incoming mail messages
.SH "INTRODUCTION"
.I maildir
is a structure for
directories of incoming mail messages.
It solves the reliability problems that plague
.I mbox
files and
.I mh
folders.
.SH "RELIABILITY ISSUES"
A machine may crash while it is delivering a message.
For both
.I mbox
files and
.I mh
folders this means that the message will be silently truncated.
Even worse: for
.I mbox
format, if the message is truncated in the middle of a line,
it will be silently joined to the next message.
The mail transport agent will try again later to deliver the message,
but it is unacceptable that a corrupted message should show up at all.
In
.IR maildir ,
every message is guaranteed complete upon delivery.

A machine may have two programs simultaneously delivering mail
to the same user.
The
.I mbox
and
.I mh
formats require the programs to update a single central file.
If the programs do not use some locking mechanism,
the central file will be corrupted.
There are several
.I mbox
and
.I mh
locking mechanisms,
none of which work portably and reliably.
In contrast, in
.IR maildir ,
no locks are ever necessary.
Different delivery processes never touch the same file.

A user may try to delete messages from his mailbox at the same
moment that the machine delivers a new message.
For
.I mbox
and
.I mh
formats, the user's mail-reading program must know
what locking mechanism the mail-delivery programs use.
In contrast, in
.IR maildir ,
any delivered message
can be safely updated or deleted by a mail-reading program.

Many sites use Sun's 
.B Network F\fPa\fBil\fPur\fBe System
(NFS),
presumably because the operating system vendor does not offer
anything else.
NFS exacerbates all of the above problems.
Some NFS implementations don't provide
.B any
reliable locking mechanism.
With 
.I mbox
and
.I mh
formats,
if two machines deliver mail to the same user,
or if a user reads mail anywhere except the delivery machine,
the user's mail is at risk.
.I maildir
works without trouble over NFS.
.SH "THE MAILDIR STRUCTURE"
A directory in
.I maildir
format has three subdirectories,
all on the same filesystem:
.BR tmp ,
.BR new ,
and
.BR cur .

Each file in
.B new
is a newly delivered mail message.
The modification time of the file is the delivery date of the message.
The message is delivered
.I without
an extra UUCP-style

.EX
     From djb Mon Jul  3 01:55:29 1995
.EE

line,
.I without
any
.B >From
quoting,
and
.I without
an extra blank line at the end.
The message is probably in RFC 822 format,
but it could contain arbitrary binary data.
It might not even end with a newline.

Files in
.B cur
are just like files in
.BR new .
The big difference is that files in
.B cur
are no longer new mail:
they have been seen by the user's mail-reading program.
.SH "HOW A MESSAGE IS DELIVERED"
The
.B tmp
directory is used to ensure reliable delivery,
as discussed here.

A program delivers a mail message in six steps.
First, it
.B chdir()\fPs
to the
.I maildir
directory.
Second, it 
.B stat()s
the name
.BR tmp/\fItime.pid.host ,
where
.I time
is the number of seconds since the beginning of 1970 GMT,
.I pid
is the program's process ID,
and
.I host
is the host name.
Third, if
.B stat()
returned anything other than ENOENT,
the program sleeps for two seconds, updates
.IR time ,
and tries the
.B stat()
again, a limited number of times.
Fourth, the program
creates
.BR tmp/\fItime.pid.host .
Fifth, the program
.I NFS-writes
the message to the file.
Sixth, the program
.BR link() s
the file to
.BR new/\fItime.pid.host .
At that instant the message has been successfully delivered.

The delivery program is required to start a 24-hour timer before
creating
.BR tmp/\fItime.pid.host ,
and to abort the delivery
if the timer expires.
Upon error, timeout, or normal completion,
the delivery program may attempt to
.B unlink()
.BR tmp/\fItime.pid.host .

.I NFS-writing
means
(1) as usual, checking the number of bytes returned from each
.B write()
call;
(2) calling
.B fsync()
and checking its return value;
(3) calling
.B close()
and checking its return value.
(Standard NFS implementations handle
.B fsync()
incorrectly
but make up for it by abusing
.BR close() .)
.SH "HOW A MESSAGE IS READ"
A mail reader operates as follows.

It looks through the
.B new
directory for new messages.
Say there is a new message,
.BR new/\fItime.pid.host .
The reader may freely display the contents of
.BR new/\fItime.pid.host ,
delete
.BR new/\fItime.pid.host ,
or rename
.B new/\fItime.pid.host
as
.BR cur/\fItime.pid.host:info .
The meaning of
.I info
is up to the mail reader.

The reader is also expected to look through the
.B tmp
directory and to clean up any old files found there.
A file in
.B tmp
may be safely removed if it
has not been accessed in 36 hours.
.SH "ENVIRONMENT VARIABLES"
Mail readers supporting
.I maildir
use the
.B MAILDIR
environment variable
as the name of the user's primary mail directory.
.SH "SEE ALSO"
qmail-alias(8)