Re: Gnus and the qmail maildir format

Re: Gnus and the qmail maildir format

[Patrick J. LoPresti]

The following message is a courtesy copy of an article
that has been posted to gnu.emacs.gnus as well.

>>>>> "ratinox" == Rich Pieri <rich.pieri@prescienttech.com> writes:

>>>>> "SK" == Sebastian Kaps <toyland@sauerland.de> writes:

 SK> I'm wondering why Gnus can't handle mails that are stored in a
 SK> qmail maildir.

 ratinox> Because nobody has written a backend to support it.

I admit I know little about Gnus backends, but I bet this would be
very easy to do.

Not only would it be easy, it would provide extremely fast and
reliable mail incorporation.  A maildir is just a directory full of
files with one message per file.  A maildir is almost always stored in
the user's home directory, which is almost always on the same file
system as the user's mail folders.  That means incorporation can just
be a "rename" operation, which is both fast and immune from mail
loss/corruption, even over NFS.

Of course, the backend will need to do something sane (i.e., copy)
when the rename fails, so some care in coding will be necessary.

I suspect it would be much easier for someone familiar with Gnus
innards to learn about maildirs than for someone familiar with
maildirs (e.g., me) to learn about Gnus innards.  A complete
description of maildir format is available at:

  http://www.qmail.org/qmail-manual-html/man5/maildir.html

Anyone want to give it a shot?

 - Pat

Re: Gnus and the qmail maildir format

[Istvan Marko]

patl@curl.com (Patrick J. LoPresti) writes:

> >>>>> "ratinox" == Rich Pieri <rich.pieri@prescienttech.com> writes:
> >>>>> "SK" == Sebastian Kaps <toyland@sauerland.de> writes:
> 
>  SK> I'm wondering why Gnus can't handle mails that are stored in a
>  SK> qmail maildir.
>  ratinox> Because nobody has written a backend to support it.

> I admit I know little about Gnus backends, but I bet this would be
> very easy to do.

Shouldn't the maildir handling stuff be added to movemail instead? The
maildir is just a different kind of INBOX right? So naturally it would
be movemail's (or it's replacement's) job to move the files from
maildir to the crashbox file (or wherever). This modified movemail
then could be used by all Emacs MUAs.

-- 
	Istvan

Re: Gnus and the qmail maildir format

[Patrick J. LoPresti]

>>>>> "imarko" == Istvan Marko <imarko@pacificnet.net> writes:

 imarko> Shouldn't the maildir handling stuff be added to movemail
 imarko> instead? The maildir is just a different kind of INBOX right?

Yes, but with important performance and robustness features that using
movemail would completely defeat.

The original purpose of movemail was to provide system-dependent
locking while accessing a file in the brain-damaged Unix mbox format.
The movemail program is a necessary concession since that brain damage
is so widespread.  Qmail maildirs, on the other hand, are architected
to require no locking at all; they can be accessed simply, directly,
and reliably by Emacs lisp.  (In fact, it is even easier than I first
thought; I just noticed that the "rename-file" function does the Right
Thing whether or not the destination is on a different filesystem than
the source.)

As you probably know, movemail was later modified to support POP.
That was a mistake.  It works by connecting to a server, dragging over
all the messages, and stuffing them into a file in mbox format.  Thus
it inflicts the brain damage of mbox (">From " quoting and *potential
mail loss*) where it isn't necessary.

Abstractions are good.  But abstractions should not be chosen based on
the need to support a stupid and fragile format, then forcing that
format (with its stupidity and fragility) onto everything in the name
of preserving the abstraction.

I agree that maildir handling should be in a library accessible by
other Emacs MUAs.  But it should be written in Emacs Lisp, so that it
will 1) automatically work portably across platforms; 2) preserve the
sanity and robustness of the maildir format; and 3) be faster.

Since nobody else has volunteered to take up this task, I guess I will
have to hack it up myself.

 - Pat

P.S.  Lars, is there any chance of your including such a thing in the
standard distribution?  If not, I do not want to waste time making it
"clean"...

Re: Gnus and the qmail maildir format

[Kai Grossjohann]

patl@curl.com (Patrick J. LoPresti) writes:

  > As you probably know, movemail was later modified to support POP.
  > That was a mistake.  It works by connecting to a server, dragging
  > over all the messages, and stuffing them into a file in mbox
  > format.  Thus it inflicts the brain damage of mbox (">From "
  > quoting and *potential mail loss*) where it isn't necessary.

Minor nitpick: movemail stores its stuff in babyl format if it reads
from a POP server.  Which means that it might be more or less
brain-damaged, depending on your taste: the beginning-of-message and
end-of-message markers are unlikely to occur in actual messages, but
they can't be quoted (I think).

  > I agree that maildir handling should be in a library accessible by
  > other Emacs MUAs.  But it should be written in Emacs Lisp, so that it
  > will 1) automatically work portably across platforms; 2) preserve the
  > sanity and robustness of the maildir format; and 3) be faster.

There's a precedent: pop3.el implements POP3 in Lisp, and I think all
Emacs MUAs have hooks where such a function can be placed.  At least
Gnus has nnmail-movemail-program which can also be a function, despite
its name.

kai
-- 
Really cancel?   [OK]  [Cancel]

Re: Gnus and the qmail maildir format

[Alan Shutko]

>>>>> "R" == Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

R> MH does not (not that it can suck mail
R> from a POP server, anyway)

Right.  Iirc, POP is handled by inc in MH, as are other mail types
such as they are.

-- 
Alan Shutko <ats@acm.org> - By consent of the corrupted
You have a deep appreciation of the arts and music.

Re: Gnus and the qmail maildir format

[Carsten Leonhardt]

[-- Attachment #1.1: Type: text/plain, Size: 391 bytes --]

>>>>> "patl" == Patrick J LoPresti <patl@curl.com> writes:

patl> Since nobody else has volunteered to take up this task, I guess
patl> I will have to hack it up myself.

I hacked(!) up nnmail to support maildir fetching quite some time
ago. I use mv instead of movemail in connection with the patch,
because movemail leaves zero-byte files lying around.

I hope this inspires you :-)

leo


[-- Attachment #1.2: Type: text/plain, Size: 2135 bytes --]

*** nnmail.el.orig	Wed Apr  1 05:43:39 1998
--- nnmail.el	Fri Mar 20 01:57:52 1998
***************
*** 1006,1011 ****
--- 1006,1046 ----
  	(goto-char end)
  	(forward-line 2)))))
  
+ ;; Added by Carsten Leonhardt <leo@arioch.oche.de>.
+ (defun nnmail-process-maildir-mail-format (func artnum-func)
+ ; In qmail's maildir directory, every file contains exactly one mail
+   (let ((case-fold-search t)
+ 	message-id)
+     (goto-char (point-min))
+     ;; Find the end of the head.
+     (narrow-to-region
+      (point-min) 
+      (if (search-forward "\n\n" nil t)
+ 	 (1- (point))
+        ;; This will never happen, but just to be on the safe side --
+        ;; if there is no head-body delimiter, we search a bit manually.
+        (while (and (looking-at "From \\|[^ \t]+:")
+ 		   (not (eobp)))
+ 	 (forward-line 1)
+ 	 (point))))
+     ;; Find the Message-ID header.
+     (goto-char (point-min))
+     (if (re-search-forward "^Message-ID:[ \t]*\\(<[^>]+>\\)" nil t)
+ 	(setq message-id (match-string 1))
+       ;; There is no Message-ID here, so we create one.
+       (save-excursion
+ 	    (when (re-search-backward "^Message-ID[ \t]*:" nil t)
+ 	  (beginning-of-line)
+ 	  (insert "Original-")))
+       (forward-line 1)
+       (insert "Message-ID: " (setq message-id (nnmail-message-id)) "\n"))
+     (run-hooks 'nnmail-prepare-incoming-header-hook)
+     ;; Allow the backend to save the article.
+     (widen)
+     (save-excursion
+ 	(goto-char (point-min))
+ 	(nnmail-check-duplication message-id func artnum-func))))
+ 
  (defun nnmail-split-incoming (incoming func &optional exit-func
  				       group artnum-func)
    "Go through the entire INCOMING file and pick out each individual mail.
***************
*** 1035,1040 ****
--- 1070,1077 ----
  	       (nnmail-process-babyl-mail-format func artnum-func))
  	      ((looking-at "\^A\^A\^A\^A")
  	       (nnmail-process-mmdf-mail-format func artnum-func))
+ 	      ((looking-at "Return-Path:")
+ 	       (nnmail-process-maildir-mail-format func artnum-func))
  	      (t
  	       (nnmail-process-unix-mail-format func artnum-func))))
        (when exit-func

[-- Attachment #2: Type: application/pgp-signature, Size: 460 bytes --]

Re: Gnus and the qmail maildir format

[Robert Bihlmeyer]

Hi,

>>>>> On 23 Apr 1998 17:20:55 -0400
>>>>> Stainless Steel Rat <ratinox@peorth.gweep.net> said:

 ratinox> movemail also ensures that the mailbox is locked before it
 ratinox> attempts to copy it to a staging area. mv does not.

This is a non-issue with maildir, since the MTA never appends to an
already existing mail.

	Robbe

-- 
Robert Bihlmeyer	reads: Deutsch, English, MIME, Latin-1, NO SPAM!
<robbe@orcus.priv.at>	<http://stud2.tuwien.ac.at/~e9426626/sig.html>

Re: Gnus and the qmail maildir format

[Lars Magne Ingebrigtsen]

patl@curl.com (Patrick J. LoPresti) writes:

> P.S.  Lars, is there any chance of your including such a thing in the
> standard distribution?

Sure; qmail support would be welcome.  But the entire
mail-fetching/splitting system is overdue for an overhaul, which means
that whatever work you do on it now might have to be discarded later.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Gnus and the qmail maildir format

[Patrick J. LoPresti]

>>>>> "Robbe" == Robert Bihlmeyer <e9426626@stud2.tuwien.ac.at> writes:

>>>>> On 23 Apr 1998 17:20:55 -0400
>>>>> Stainless Steel Rat <ratinox@peorth.gweep.net> said:

 ratinox> movemail also ensures that the mailbox is locked before it
 ratinox> attempts to copy it to a staging area. mv does not.

 Robbe> This is a non-issue with maildir, since the MTA never appends
 Robbe> to an already existing mail.

Exactly.  Maildir is a lock-free spool format which never mangles
mail, even if you have multiple machines *delivering* to the same
spool simultaneously over NFS.  Maildirs don't quote "From " lines and
they are fully 8-bit clean.  In short, they are the Right Thing and
Gnus should support them.

It is a simple enough format, and the documentation is short:

  http://www.qmail.org/qmail-manual-html/man5/maildir.html

That said, I have looked at nnmail.el and I no longer think this will
be a trivial job.

Gnus's mail incorporation architecture is built around the concept
that a mail spool is a single file whose messages we process by
repeatedly narrowing the spool buffer to a single message of interest.
This basically requires the architecture seen in pop3.el where more
modern mail storage schemes are converted into a single file before
Gnus touches them.

It would not be too hard, I think, to change this model by creating a
Gnus "mail spool" abstraction which represents a set of messages.  The
"mail spool" objects would need to support a certain set of simple
operations, like "iterate over the messages calling this function" or
"delete the spool; I'm done".  (Other actions, like "delete a single
message", might be useful too.  I admit I have not thought through the
details carefully.)

In addition to supporting maildirs easily, such an abstraction would
allow POP support (for example) to be more efficient.

Another nice thing about maildir format is that incorporation itself
can just be a "rename" operation, which is both very fast and very
atomic.  Unfortunately, this won't quite work for us, since Gnus wants
to munge the headers (and possibly the body, depending on certain
hooks) before writing a message to a folder.  Just one of the prices
we pay for nice features, I suppose.

Anyway, I would prefer to see (i.e., convince) someone else (i.e.,
Lars) to redo the mail backend stuff a little bit before I try to add
maildir support.  Comments?

 - Pat

Re: Gnus and the qmail maildir format

[Dan Christensen]

patl@cag.lcs.mit.edu (Patrick J. LoPresti) writes:
> Exactly.  Maildir is a lock-free spool format which never mangles
> mail, even if you have multiple machines *delivering* to the same
> spool simultaneously over NFS.  Maildirs don't quote "From " lines and
> they are fully 8-bit clean.  In short, they are the Right Thing and
> Gnus should support them.

Do you think Gnus should have a maildir backend?  This is orthogonal
to the issue of being able to incorporate mail from a maildir spool,
but there would be some overlap in code.

Dan

-- 
Dan Christensen
jdc@math.jhu.edu

Re: Gnus and the qmail maildir format

[Patrick J. LoPresti]

>>>>> "jdc" == Dan Christensen <jdc@chow.mat.jhu.edu> writes:

 jdc> Do you think Gnus should have a maildir backend?  This is
 jdc> orthogonal to the issue of being able to incorporate mail from a
 jdc> maildir spool, but there would be some overlap in code.

I was talking about incorporating from a maildir spool; sorry for
misusing the phrase "back end" in my last message.

Supporting maildir as a folder format is also an interesting idea, but
probably not as important.  I think nnml (for example) is adequate
since you never have multiple processes manipulating the folder and
you can always regenerate the overview database from scratch.

 - Pat