* Re: Gnus and the qmail maildir format [not found] ` <x7u38e28hc.fsf@gkar.prescienttech.com> @ 1998-04-08 19:28 ` Patrick J. LoPresti 1998-04-08 21:18 ` Istvan Marko 0 siblings, 1 reply; 11+ messages in thread From: Patrick J. LoPresti @ 1998-04-08 19:28 UTC (permalink / raw) Cc: ding The following message is a courtesy copy of an article that has been posted to gnu.emacs.gnus as well. >>>>> "ratinox" == Rich Pieri <rich.pieri@prescienttech.com> writes: >>>>> "SK" == Sebastian Kaps <toyland@sauerland.de> writes: SK> I'm wondering why Gnus can't handle mails that are stored in a SK> qmail maildir. ratinox> Because nobody has written a backend to support it. I admit I know little about Gnus backends, but I bet this would be very easy to do. Not only would it be easy, it would provide extremely fast and reliable mail incorporation. A maildir is just a directory full of files with one message per file. A maildir is almost always stored in the user's home directory, which is almost always on the same file system as the user's mail folders. That means incorporation can just be a "rename" operation, which is both fast and immune from mail loss/corruption, even over NFS. Of course, the backend will need to do something sane (i.e., copy) when the rename fails, so some care in coding will be necessary. I suspect it would be much easier for someone familiar with Gnus innards to learn about maildirs than for someone familiar with maildirs (e.g., me) to learn about Gnus innards. A complete description of maildir format is available at: http://www.qmail.org/qmail-manual-html/man5/maildir.html Anyone want to give it a shot? - Pat ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Gnus and the qmail maildir format 1998-04-08 19:28 ` Gnus and the qmail maildir format Patrick J. LoPresti @ 1998-04-08 21:18 ` Istvan Marko 1998-04-10 17:30 ` Patrick J. LoPresti 0 siblings, 1 reply; 11+ messages in thread From: Istvan Marko @ 1998-04-08 21:18 UTC (permalink / raw) patl@curl.com (Patrick J. LoPresti) writes: > >>>>> "ratinox" == Rich Pieri <rich.pieri@prescienttech.com> writes: > >>>>> "SK" == Sebastian Kaps <toyland@sauerland.de> writes: > > SK> I'm wondering why Gnus can't handle mails that are stored in a > SK> qmail maildir. > ratinox> Because nobody has written a backend to support it. > I admit I know little about Gnus backends, but I bet this would be > very easy to do. Shouldn't the maildir handling stuff be added to movemail instead? The maildir is just a different kind of INBOX right? So naturally it would be movemail's (or it's replacement's) job to move the files from maildir to the crashbox file (or wherever). This modified movemail then could be used by all Emacs MUAs. -- Istvan ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Gnus and the qmail maildir format 1998-04-08 21:18 ` Istvan Marko @ 1998-04-10 17:30 ` Patrick J. LoPresti 1998-04-10 18:39 ` Kai Grossjohann ` (2 more replies) 0 siblings, 3 replies; 11+ messages in thread From: Patrick J. LoPresti @ 1998-04-10 17:30 UTC (permalink / raw) >>>>> "imarko" == Istvan Marko <imarko@pacificnet.net> writes: imarko> Shouldn't the maildir handling stuff be added to movemail imarko> instead? The maildir is just a different kind of INBOX right? Yes, but with important performance and robustness features that using movemail would completely defeat. The original purpose of movemail was to provide system-dependent locking while accessing a file in the brain-damaged Unix mbox format. The movemail program is a necessary concession since that brain damage is so widespread. Qmail maildirs, on the other hand, are architected to require no locking at all; they can be accessed simply, directly, and reliably by Emacs lisp. (In fact, it is even easier than I first thought; I just noticed that the "rename-file" function does the Right Thing whether or not the destination is on a different filesystem than the source.) As you probably know, movemail was later modified to support POP. That was a mistake. It works by connecting to a server, dragging over all the messages, and stuffing them into a file in mbox format. Thus it inflicts the brain damage of mbox (">From " quoting and *potential mail loss*) where it isn't necessary. Abstractions are good. But abstractions should not be chosen based on the need to support a stupid and fragile format, then forcing that format (with its stupidity and fragility) onto everything in the name of preserving the abstraction. I agree that maildir handling should be in a library accessible by other Emacs MUAs. But it should be written in Emacs Lisp, so that it will 1) automatically work portably across platforms; 2) preserve the sanity and robustness of the maildir format; and 3) be faster. Since nobody else has volunteered to take up this task, I guess I will have to hack it up myself. - Pat P.S. Lars, is there any chance of your including such a thing in the standard distribution? If not, I do not want to waste time making it "clean"... ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Gnus and the qmail maildir format 1998-04-10 17:30 ` Patrick J. LoPresti @ 1998-04-10 18:39 ` Kai Grossjohann [not found] ` <x7ra339ho5.fsf@peorth.gweep.net> 1998-04-23 14:23 ` Carsten Leonhardt 1998-04-24 20:08 ` Lars Magne Ingebrigtsen 2 siblings, 1 reply; 11+ messages in thread From: Kai Grossjohann @ 1998-04-10 18:39 UTC (permalink / raw) Cc: ding patl@curl.com (Patrick J. LoPresti) writes: > As you probably know, movemail was later modified to support POP. > That was a mistake. It works by connecting to a server, dragging > over all the messages, and stuffing them into a file in mbox > format. Thus it inflicts the brain damage of mbox (">From " > quoting and *potential mail loss*) where it isn't necessary. Minor nitpick: movemail stores its stuff in babyl format if it reads from a POP server. Which means that it might be more or less brain-damaged, depending on your taste: the beginning-of-message and end-of-message markers are unlikely to occur in actual messages, but they can't be quoted (I think). > I agree that maildir handling should be in a library accessible by > other Emacs MUAs. But it should be written in Emacs Lisp, so that it > will 1) automatically work portably across platforms; 2) preserve the > sanity and robustness of the maildir format; and 3) be faster. There's a precedent: pop3.el implements POP3 in Lisp, and I think all Emacs MUAs have hooks where such a function can be placed. At least Gnus has nnmail-movemail-program which can also be a function, despite its name. kai -- Really cancel? [OK] [Cancel] ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <x7ra339ho5.fsf@peorth.gweep.net>]
* Re: Gnus and the qmail maildir format [not found] ` <x7ra339ho5.fsf@peorth.gweep.net> @ 1998-04-12 16:18 ` Alan Shutko 0 siblings, 0 replies; 11+ messages in thread From: Alan Shutko @ 1998-04-12 16:18 UTC (permalink / raw) >>>>> "R" == Stainless Steel Rat <ratinox@peorth.gweep.net> writes: R> MH does not (not that it can suck mail R> from a POP server, anyway) Right. Iirc, POP is handled by inc in MH, as are other mail types such as they are. -- Alan Shutko <ats@acm.org> - By consent of the corrupted You have a deep appreciation of the arts and music. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Gnus and the qmail maildir format 1998-04-10 17:30 ` Patrick J. LoPresti 1998-04-10 18:39 ` Kai Grossjohann @ 1998-04-23 14:23 ` Carsten Leonhardt [not found] ` <x77m4gnsjs.fsf@peorth.gweep.net> 1998-04-24 20:08 ` Lars Magne Ingebrigtsen 2 siblings, 1 reply; 11+ messages in thread From: Carsten Leonhardt @ 1998-04-23 14:23 UTC (permalink / raw) [-- Attachment #1.1: Type: text/plain, Size: 391 bytes --] >>>>> "patl" == Patrick J LoPresti <patl@curl.com> writes: patl> Since nobody else has volunteered to take up this task, I guess patl> I will have to hack it up myself. I hacked(!) up nnmail to support maildir fetching quite some time ago. I use mv instead of movemail in connection with the patch, because movemail leaves zero-byte files lying around. I hope this inspires you :-) leo [-- Attachment #1.2: Type: text/plain, Size: 2135 bytes --] *** nnmail.el.orig Wed Apr 1 05:43:39 1998 --- nnmail.el Fri Mar 20 01:57:52 1998 *************** *** 1006,1011 **** --- 1006,1046 ---- (goto-char end) (forward-line 2))))) + ;; Added by Carsten Leonhardt <leo@arioch.oche.de>. + (defun nnmail-process-maildir-mail-format (func artnum-func) + ; In qmail's maildir directory, every file contains exactly one mail + (let ((case-fold-search t) + message-id) + (goto-char (point-min)) + ;; Find the end of the head. + (narrow-to-region + (point-min) + (if (search-forward "\n\n" nil t) + (1- (point)) + ;; This will never happen, but just to be on the safe side -- + ;; if there is no head-body delimiter, we search a bit manually. + (while (and (looking-at "From \\|[^ \t]+:") + (not (eobp))) + (forward-line 1) + (point)))) + ;; Find the Message-ID header. + (goto-char (point-min)) + (if (re-search-forward "^Message-ID:[ \t]*\\(<[^>]+>\\)" nil t) + (setq message-id (match-string 1)) + ;; There is no Message-ID here, so we create one. + (save-excursion + (when (re-search-backward "^Message-ID[ \t]*:" nil t) + (beginning-of-line) + (insert "Original-"))) + (forward-line 1) + (insert "Message-ID: " (setq message-id (nnmail-message-id)) "\n")) + (run-hooks 'nnmail-prepare-incoming-header-hook) + ;; Allow the backend to save the article. + (widen) + (save-excursion + (goto-char (point-min)) + (nnmail-check-duplication message-id func artnum-func)))) + (defun nnmail-split-incoming (incoming func &optional exit-func group artnum-func) "Go through the entire INCOMING file and pick out each individual mail. *************** *** 1035,1040 **** --- 1070,1077 ---- (nnmail-process-babyl-mail-format func artnum-func)) ((looking-at "\^A\^A\^A\^A") (nnmail-process-mmdf-mail-format func artnum-func)) + ((looking-at "Return-Path:") + (nnmail-process-maildir-mail-format func artnum-func)) (t (nnmail-process-unix-mail-format func artnum-func)))) (when exit-func [-- Attachment #2: Type: application/pgp-signature, Size: 460 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <x77m4gnsjs.fsf@peorth.gweep.net>]
* Re: Gnus and the qmail maildir format [not found] ` <x77m4gnsjs.fsf@peorth.gweep.net> @ 1998-04-24 8:02 ` Robert Bihlmeyer 1998-04-25 16:04 ` Patrick J. LoPresti 0 siblings, 1 reply; 11+ messages in thread From: Robert Bihlmeyer @ 1998-04-24 8:02 UTC (permalink / raw) Hi, >>>>> On 23 Apr 1998 17:20:55 -0400 >>>>> Stainless Steel Rat <ratinox@peorth.gweep.net> said: ratinox> movemail also ensures that the mailbox is locked before it ratinox> attempts to copy it to a staging area. mv does not. This is a non-issue with maildir, since the MTA never appends to an already existing mail. Robbe -- Robert Bihlmeyer reads: Deutsch, English, MIME, Latin-1, NO SPAM! <robbe@orcus.priv.at> <http://stud2.tuwien.ac.at/~e9426626/sig.html> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Gnus and the qmail maildir format 1998-04-24 8:02 ` Robert Bihlmeyer @ 1998-04-25 16:04 ` Patrick J. LoPresti 1998-04-27 20:48 ` Dan Christensen 0 siblings, 1 reply; 11+ messages in thread From: Patrick J. LoPresti @ 1998-04-25 16:04 UTC (permalink / raw) >>>>> "Robbe" == Robert Bihlmeyer <e9426626@stud2.tuwien.ac.at> writes: >>>>> On 23 Apr 1998 17:20:55 -0400 >>>>> Stainless Steel Rat <ratinox@peorth.gweep.net> said: ratinox> movemail also ensures that the mailbox is locked before it ratinox> attempts to copy it to a staging area. mv does not. Robbe> This is a non-issue with maildir, since the MTA never appends Robbe> to an already existing mail. Exactly. Maildir is a lock-free spool format which never mangles mail, even if you have multiple machines *delivering* to the same spool simultaneously over NFS. Maildirs don't quote "From " lines and they are fully 8-bit clean. In short, they are the Right Thing and Gnus should support them. It is a simple enough format, and the documentation is short: http://www.qmail.org/qmail-manual-html/man5/maildir.html That said, I have looked at nnmail.el and I no longer think this will be a trivial job. Gnus's mail incorporation architecture is built around the concept that a mail spool is a single file whose messages we process by repeatedly narrowing the spool buffer to a single message of interest. This basically requires the architecture seen in pop3.el where more modern mail storage schemes are converted into a single file before Gnus touches them. It would not be too hard, I think, to change this model by creating a Gnus "mail spool" abstraction which represents a set of messages. The "mail spool" objects would need to support a certain set of simple operations, like "iterate over the messages calling this function" or "delete the spool; I'm done". (Other actions, like "delete a single message", might be useful too. I admit I have not thought through the details carefully.) In addition to supporting maildirs easily, such an abstraction would allow POP support (for example) to be more efficient. Another nice thing about maildir format is that incorporation itself can just be a "rename" operation, which is both very fast and very atomic. Unfortunately, this won't quite work for us, since Gnus wants to munge the headers (and possibly the body, depending on certain hooks) before writing a message to a folder. Just one of the prices we pay for nice features, I suppose. Anyway, I would prefer to see (i.e., convince) someone else (i.e., Lars) to redo the mail backend stuff a little bit before I try to add maildir support. Comments? - Pat ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Gnus and the qmail maildir format 1998-04-25 16:04 ` Patrick J. LoPresti @ 1998-04-27 20:48 ` Dan Christensen 1998-04-28 0:59 ` Patrick J. LoPresti 0 siblings, 1 reply; 11+ messages in thread From: Dan Christensen @ 1998-04-27 20:48 UTC (permalink / raw) patl@cag.lcs.mit.edu (Patrick J. LoPresti) writes: > Exactly. Maildir is a lock-free spool format which never mangles > mail, even if you have multiple machines *delivering* to the same > spool simultaneously over NFS. Maildirs don't quote "From " lines and > they are fully 8-bit clean. In short, they are the Right Thing and > Gnus should support them. Do you think Gnus should have a maildir backend? This is orthogonal to the issue of being able to incorporate mail from a maildir spool, but there would be some overlap in code. Dan -- Dan Christensen jdc@math.jhu.edu ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Gnus and the qmail maildir format 1998-04-27 20:48 ` Dan Christensen @ 1998-04-28 0:59 ` Patrick J. LoPresti 0 siblings, 0 replies; 11+ messages in thread From: Patrick J. LoPresti @ 1998-04-28 0:59 UTC (permalink / raw) >>>>> "jdc" == Dan Christensen <jdc@chow.mat.jhu.edu> writes: jdc> Do you think Gnus should have a maildir backend? This is jdc> orthogonal to the issue of being able to incorporate mail from a jdc> maildir spool, but there would be some overlap in code. I was talking about incorporating from a maildir spool; sorry for misusing the phrase "back end" in my last message. Supporting maildir as a folder format is also an interesting idea, but probably not as important. I think nnml (for example) is adequate since you never have multiple processes manipulating the folder and you can always regenerate the overview database from scratch. - Pat ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Gnus and the qmail maildir format 1998-04-10 17:30 ` Patrick J. LoPresti 1998-04-10 18:39 ` Kai Grossjohann 1998-04-23 14:23 ` Carsten Leonhardt @ 1998-04-24 20:08 ` Lars Magne Ingebrigtsen 2 siblings, 0 replies; 11+ messages in thread From: Lars Magne Ingebrigtsen @ 1998-04-24 20:08 UTC (permalink / raw) patl@curl.com (Patrick J. LoPresti) writes: > P.S. Lars, is there any chance of your including such a thing in the > standard distribution? Sure; qmail support would be welcome. But the entire mail-fetching/splitting system is overdue for an overhaul, which means that whatever work you do on it now might have to be discarded later. -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~1998-04-28 0:59 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <m3wwddekkx.fsf@toyland.sauerland.de> [not found] ` <x7u38e28hc.fsf@gkar.prescienttech.com> 1998-04-08 19:28 ` Gnus and the qmail maildir format Patrick J. LoPresti 1998-04-08 21:18 ` Istvan Marko 1998-04-10 17:30 ` Patrick J. LoPresti 1998-04-10 18:39 ` Kai Grossjohann [not found] ` <x7ra339ho5.fsf@peorth.gweep.net> 1998-04-12 16:18 ` Alan Shutko 1998-04-23 14:23 ` Carsten Leonhardt [not found] ` <x77m4gnsjs.fsf@peorth.gweep.net> 1998-04-24 8:02 ` Robert Bihlmeyer 1998-04-25 16:04 ` Patrick J. LoPresti 1998-04-27 20:48 ` Dan Christensen 1998-04-28 0:59 ` Patrick J. LoPresti 1998-04-24 20:08 ` Lars Magne Ingebrigtsen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).