nnmaildir.el + courier IMAP compatibility patch

nnmaildir.el + courier IMAP compatibility patch

[Matt Armstrong]

[-- Attachment #1: Type: text/plain, Size: 1138 bytes --]

Paul and other nnmaildir hackers,

I'm now looking at switching to nnmaildir, primarily because it'll let
me read mail natively via Gnus or via Courier IMAP which also uses
Maildir for its backend.

By default Courier IMAP expects INBOX to be ~/Maildir/ and any
INBOX.subfolder to be ~/Maildir/.subfolder/.  However, if I set up
nnmaildir's create-directory to point at ~/Maildir/, it will create
new folders without the leading dot (~/Maildir/subfolder/) and Courier
IMAP won't find them.  What's more, conflicts would occur if I wanted
to create groups called "cur," "new" or "tmp".

So I added a courier-compatible parameter to the nnmaildir server that
causes it to create new maildirs with a leading . in the dir filename
(while keeping the Gnus group the same).

It also cleans up the logic that turns create-dir into the maildir
name by doing it only once.

It also makes sure not to turn any .nnmaildir directory into a maildir
(by failing to create a "nnmaildir" group when courier-compatible is
set).

Please check out this patch and consider committing it.  I recently
sent Gnus related copyright papers to the FSF.

-- 
matt


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnmaildir.patch --]
[-- Type: text/x-patch, Size: 2456 bytes --]

Index: nnmaildir.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnmaildir.el,v
retrieving revision 6.17
diff -u -r6.17 nnmaildir.el
--- nnmaildir.el	2002/04/22 02:28:17	6.17
+++ nnmaildir.el	2002/05/02 17:42:42
@@ -134,6 +134,7 @@
   (error      nil :type string)         ;; last error message, or nil
   (mtime      nil :type list)           ;; modtime of dir
   (gnm        nil)                      ;; flag: split from mail-sources?
+  (courier    nil)                      ;; flag: create maildirs w/ leading '.'
   (create-dir nil :type string))        ;; group creation directory
 
 (defmacro nnmaildir--nlist-last-num (nlist)
@@ -541,6 +542,10 @@
 	   (car x)
 	   (setf (nnmaildir--srv-gnm server) t)
 	   (require 'nnmail))
+      (and (setq x (assq 'courier-compatible defs))
+	   (setq x (cdr x))
+	   (car x)
+	   (setf (nnmaildir--srv-courier server) t))
       (setq x (assq 'create-directory defs))
       (when x
 	(setq x (cadr x)
@@ -897,7 +902,7 @@
   (nnmaildir--prepare server nil)
   (catch 'return
     (let ((create-dir (nnmaildir--srv-create-dir nnmaildir--cur-server))
-	  srv-dir dir groups)
+	  srv-dir dir groups maildir)
       (when (zerop (length gname))
 	(setf (nnmaildir--srv-error nnmaildir--cur-server)
 	      "Invalid (empty) group name")
@@ -922,13 +927,21 @@
 	(setq dir srv-dir
 	      dir (file-truename dir)
 	      dir (concat dir create-dir)))
-      (setq dir (nnmaildir--subdir (file-name-as-directory dir) gname))
+      (setq dir (file-name-as-directory dir))
+      (setq maildir (concat dir
+			    (if (nnmaildir--srv-courier nnmaildir--cur-server)
+				(concat "." gname)
+			      gname)))
+      (when (equal (file-name-as-directory maildir) (nnmaildir--nndir dir))
+	(setf (nnmaildir--srv-error nnmaildir--cur-server)
+	      (concat "Group name may not be " gname))
+	(throw 'return nil))
+      (setq dir (file-name-as-directory maildir))
       (nnmaildir--mkdir dir)
       (nnmaildir--mkdir (nnmaildir--tmp dir))
       (nnmaildir--mkdir (nnmaildir--new dir))
       (nnmaildir--mkdir (nnmaildir--cur dir))
-      (setq create-dir (file-name-as-directory create-dir))
-      (make-symbolic-link (concat create-dir gname) (concat srv-dir gname))
+      (make-symbolic-link maildir (concat srv-dir gname))
       (nnmaildir-request-scan 'find-new-groups))))
 
 (defun nnmaildir-request-rename-group (gname new-name &optional server)

Re: nnmaildir.el + courier IMAP compatibility patch

[Paul Jarc]

Matt Armstrong <matt@lickey.com> wrote:
> By default Courier IMAP expects INBOX to be ~/Maildir/ and any
> INBOX.subfolder to be ~/Maildir/.subfolder/.  However, if I set up
> nnmaildir's create-directory to point at ~/Maildir/, it will create
> new folders without the leading dot (~/Maildir/subfolder/) and Courier
> IMAP won't find them.

Well, you could create symlinks.  Ah, but there seems to be no
gnus-make-new-group-hook to do that from.

> So I added a courier-compatible parameter to the nnmaildir server that
> causes it to create new maildirs with a leading . in the dir filename
> (while keeping the Gnus group the same).

I think I'd rather have an arbitrarily configurable prefix, which
would default to "" and which in your case could be set to ".", rather
than something unnecessarily tied to a specific other piece of
software.

I've also been meaning to make it so that create-dir could be ".";
i.e., the server directory contains the maildirs directly rather than
symlinks to them.  It seems these features could be related.  I'll
work on it.

I've occasionally thought of writing an IMAP server that would store
marks in an nnmaildir-compatible way.  How does Courier store marks?


paul

Re: nnmaildir.el + courier IMAP compatibility patch

[Lloyd Zusman]

prj@po.cwru.edu (Paul Jarc) writes:

> [ ... ]
>
> I've occasionally thought of writing an IMAP server that would store
> marks in an nnmaildir-compatible way.  How does Courier store marks?

It stores them as part of the message filename.  For example, it appends
the string ",S" to the file names of messages that have the Seen flag
set.


-- 
 Lloyd Zusman
 ljz@asfast.com

Re: nnmaildir.el + courier IMAP compatibility patch

[Matt Armstrong]

prj@po.cwru.edu (Paul Jarc) writes:

> Matt Armstrong <matt@lickey.com> wrote:
>> By default Courier IMAP expects INBOX to be ~/Maildir/ and any
>> INBOX.subfolder to be ~/Maildir/.subfolder/.  However, if I set up
>> nnmaildir's create-directory to point at ~/Maildir/, it will create
>> new folders without the leading dot (~/Maildir/subfolder/) and Courier
>> IMAP won't find them.
>
> Well, you could create symlinks.  Ah, but there seems to be no
> gnus-make-new-group-hook to do that from.

I tried this, but courier seems to like to have the real directory in
~/Maildir

>> So I added a courier-compatible parameter to the nnmaildir server
>> that causes it to create new maildirs with a leading . in the dir
>> filename (while keeping the Gnus group the same).
>
> I think I'd rather have an arbitrarily configurable prefix, which
> would default to "" and which in your case could be set to ".",
> rather than something unnecessarily tied to a specific other piece
> of software.

Sounds like a good idea.


> I've also been meaning to make it so that create-dir could be ".";
> i.e., the server directory contains the maildirs directly rather
> than symlinks to them.  It seems these features could be related.
> I'll work on it.

Right now I'm using the symlinks to remove the leading '.' from the
maildir directory name.  This way Gnus doesn't think the group name is
.list.ding but rather list.ding.


> I've occasionally thought of writing an IMAP server that would store
> marks in an nnmaildir-compatible way.

I've occasionally thought of going the opposite direction and forking
your nnmaildir.el to be less inode intensive by storing marks using
the "experimental suffix" method described in the maildir spec,
storing overview data as nnml does, etc.  I can say with authority
though that I'll never actually do it.  ;-)


> How does Courier store marks?

With the maildir standard ":2,XYZ" suffixes on the filenames and a
'courierimapuidb' in the root of the maildir that maps IMAP unique ids
to maildir files.  E.g.

    1 1020354007 5
    1 1020345973.27446_0.squeaker
    2 1020346023.27461_0.squeaker
    3 1020346249.27609_0.squeaker
    4 1020346675.27758_0.squeaker

If IMAP has a way to set custom marks (which I vaguely think it does)
I don't know how courier stores them.

-- 
matt

Re: nnmaildir.el + courier IMAP compatibility patch

[Matt Armstrong]

Matt Armstrong <matt@lickey.com> writes:

> prj@po.cwru.edu (Paul Jarc) writes:
>
>> I've occasionally thought of writing an IMAP server that would store
>> marks in an nnmaildir-compatible way.
>
> I've occasionally thought of going the opposite direction and

[...]

As a side note, it would be interesting if nnmaildir stored the read,
reply and tick marks in the message filename.  This would allow for
some basic interoperability between Gnus, Mutt and Courier IMAP.

It isn't clear how Gnus could store both the standard maildir marks
(as in ":2,RSF") with custom Gnus marks like dormant.

-- 
matt

Re: nnmaildir.el + courier IMAP compatibility patch

[Simon Josefsson]

Matt Armstrong <matt@lickey.com> writes:

> As a side note, it would be interesting if nnmaildir stored the read,
> reply and tick marks in the message filename.  This would allow for
> some basic interoperability between Gnus, Mutt and Courier IMAP.
>
> It isn't clear how Gnus could store both the standard maildir marks
> (as in ":2,RSF") with custom Gnus marks like dormant.

Does Courier support client-specific marks?  If so, maybe finding out
how it stores them could be useful.

Re: nnmaildir.el + courier IMAP compatibility patch

[Paul Jarc]

Matt Armstrong <matt@lickey.com> wrote:
> prj@po.cwru.edu (Paul Jarc) writes:
>> I've occasionally thought of writing an IMAP server that would store
>> marks in an nnmaildir-compatible way.
>
> I've occasionally thought of going the opposite direction and forking
> your nnmaildir.el to be less inode intensive by storing marks using
> the "experimental suffix" method described in the maildir spec,

That would save you one inode per group, at most.  (In my setup, it
would save one inode total.)

> storing overview data as nnml does, etc.

That would save you one inode per article.  I think it could improve
or hinder performance, depending on your filesystem, but would require
more complex code in any case.  Are you in any danger of running out
of inodes, according to "df -i"?

>> How does Courier store marks?
>
> With the maildir standard ":2,XYZ" suffixes on the filenames

Ok.  There are a few reasons nnmaildir doesn't do that: it can't
accommodate all of Gnus's marks; using two different storage methods
for different sets of marks would be pretty ugly; treating the
filename as constant makes concurrent use much easier.


paul

Re: nnmaildir.el + courier IMAP compatibility patch

[Matt Armstrong]

Simon Josefsson <jas@extundo.com> writes:

> Matt Armstrong <matt@lickey.com> writes:
>
>> As a side note, it would be interesting if nnmaildir stored the read,
>> reply and tick marks in the message filename.  This would allow for
>> some basic interoperability between Gnus, Mutt and Courier IMAP.
>>
>> It isn't clear how Gnus could store both the standard maildir marks
>> (as in ":2,RSF") with custom Gnus marks like dormant.
>
> Does Courier support client-specific marks?  If so, maybe finding
> out how it stores them could be useful.

It doesn't, looks like it issues PERMANENTFLAGS with the list of hard
coded flags it supports.


-- 
Don't send mail to Harmon.Burton@hole.lickey.com
The address is there for spammers to harvest.

Re: nnmaildir.el + courier IMAP compatibility patch

[Matt Armstrong]

prj@po.cwru.edu (Paul Jarc) writes:

> Matt Armstrong <matt@lickey.com> wrote:
>> prj@po.cwru.edu (Paul Jarc) writes:
>>> I've occasionally thought of writing an IMAP server that would store
>>> marks in an nnmaildir-compatible way.
>>
>> I've occasionally thought of going the opposite direction and forking
>> your nnmaildir.el to be less inode intensive by storing marks using
>> the "experimental suffix" method described in the maildir spec,
>
> That would save you one inode per group, at most.  (In my setup, it
> would save one inode total.)

I didn't realize the marks were hard links.  I've got plenty of inodes
to spare.

Anyway, the performance problems I was seeing went away when I set
gnus-fetch-old-headers to nil.  I had it set to 25.  I think
gnus-fetch-old-headers causes a lot of fetches by message-id.  Perhaps
that is slowish with nnmaildir?  I was seeing 30 second group entry
times for a group with 2500 messages, about 25 ticked and none of them
new.  That is gone now and it is pretty zippy.


>>> How does Courier store marks?
>>
>> With the maildir standard ":2,XYZ" suffixes on the filenames
>
> Ok.  There are a few reasons nnmaildir doesn't do that: it can't
> accommodate all of Gnus's marks; using two different storage methods
> for different sets of marks would be pretty ugly; treating the
> filename as constant makes concurrent use much easier.

First, lemme be clear that I think this is a minor thing and that
nnmaildir is great and I'm thankful that you created it.  :-)

But, do you worry about concurrent use by other email clients?  It
seems like nnmaildir can't depend on the :2, suffix remaining constant
anyway.

If that is the case, then the only implementation difficulty is
storing some marks one way and some marks another.  But abstracting
that behind an API doesn't seem too difficult.

One argument against the "maildir way" of storing marks is that it
makes it less efficient to, say, find all the ticked articles.  You've
got to glob the whole cur directory instead of just the ticked marks
dir.

But I still like it as a feature.  Say I go on vacation for two weeks
and read mail via a web mail client that accesses my mail via IMAP.
It'd be great to come back to Gnus and have everything I read actually
marked read in Gnus.  Maybe if I want this I should just read via IMAP
at all times -- I would if it weren't slower.


-- 
Don't send mail to Jan.Holm@hole.lickey.com
The address is there for spammers to harvest.

Re: nnmaildir.el + courier IMAP compatibility patch

[Paul Jarc]

Matt Armstrong <matt@lickey.com> wrote:
> Anyway, the performance problems I was seeing went away when I set
> gnus-fetch-old-headers to nil.  I had it set to 25.  I think
> gnus-fetch-old-headers causes a lot of fetches by message-id.  Perhaps
> that is slowish with nnmaildir?

I wouldn't expect so.  elp is our friend.

> First, lemme be clear that I think this is a minor thing and that
> nnmaildir is great and I'm thankful that you created it.  :-)

:)

> But, do you worry about concurrent use by other email clients?  It
> seems like nnmaildir can't depend on the :2, suffix remaining constant
> anyway.

nnmaildir is not designed to work concurrently with programs that
change the message filenames.  But it's ok to run those programs
non-concurrently, between Gnus sessions.  Marks won't be shared, but
nothing should break.

> One argument against the "maildir way" of storing marks is that it
> makes it less efficient to, say, find all the ticked articles.  You've
> got to glob the whole cur directory instead of just the ticked marks
> dir.

Right.  Also, if you want concurrent access, you can't assume the
filename is constant even within a session, so you may have to rescan
the directory just to find a message, even when you already know the
unique part of the name.

> Say I go on vacation for two weeks and read mail via a web mail
> client that accesses my mail via IMAP.  It'd be great to come back
> to Gnus and have everything I read actually marked read in Gnus.

That makes a case for having a common mark scheme, but not for any one
particular mark scheme. :/  It probably wouldn't be too hard to write
a script to update marks in one direction or ther other, though.


paul