nnmaildir

nnmaildir

[Thomas Schwinge]

Hello!

Convinced by the power of Gnus I want to make it my default mail and
news client.

Regarding news: Converting from slrn to Gnus was no problem at all.

However regarding mail the situation is more complicated:

Since all of my mail folders are in maildir format my backend of choice
would be nnmaildir.

#v+
thomas@server:~ > ls -l Mail/ Mail-nic-nac-project/
Mail-nic-nac-project/:
total 28
drwx------    6 thomas   users         224 Feb  9 20:59 Drafts
drwxr-xr-x    6 thomas   users         224 Feb  9 20:57 INBOX
drwx------    6 thomas   users         224 Feb  9 20:58 Sent
drwxr-xr-x    6 thomas   users         152 Jan 26 18:21 lists.bincimap\.org.binc
drwxr-xr-x    6 thomas   users         152 Jan 26 18:22 lists.bincimap\.org.binc-dev
drwxr-xr-x    6 thomas   users         152 Feb  6 18:23 lists.bincimap\.org.binc-news
[...]
drwxr-xr-x    6 thomas   users         152 Jan 27 16:18 lists.gnus\.org.ding
[...]

Mail/:
total 2404
lrwxrwxrwx    1 thomas   users          30 Feb  9 20:58 Drafts -> ../Mail-nic-nac-project/Drafts
drwx------    6 thomas   users         224 Feb  9 20:55 Elektronik
drwx------    6 thomas   users         224 Feb  9 19:40 INBOX
drwx------    6 thomas   users         224 Feb  9 20:55 INBOX.TODO
lrwxrwxrwx    1 thomas   users          29 Feb  9 20:57 INBOX.nic-nac-project -> ../Mail-nic-nac-project/INBOX
[...]
lrwxrwxrwx    1 thomas   users          28 Feb  9 20:58 Sent -> ../Mail-nic-nac-project/Sent
[...]
#v-

The folders in Mail-nic-nac-project/ are synchronized between several
hosts using maildirsync <URL:http://hacks.dlux.hu/maildirsync/>.

Now you may wonder about the naming scheme, so let me explain this: This
is the IMAPdir layout <URL:http://www.bincimap.org/bincimap-imapdir.html>
used by Binc IMAP <URL:http://www.bincimap.org/> to get the folders
hierarchically displayed in the IMAP client:
#v+
Mail +- INBOX
     |    \- TODO
     +- Drafts
     +- Sent
     +- [...]
     +- lists -+- bincimap.org -+- binc
               |                |- binc-dev
               |                \- binc-news
               +- gnus.org --- ding
               +- [...]
#v-

Parallel to using IMAP I've been using mutt - operating natively on the
maildirs.

I added
#v+
(setq gnus-secondary-select-methods
  '((nnmaildir ""                (directory "~/Mail/"))
    (nnmaildir "nic-nac-project" (directory "~/Mail-nic-nac-project/"))))
#v-
to .gnus.el and the folders showed up in Gnus - although I was not able
to enter the ones whose are containing a '\': The were refused with:
#v+
Retrieving newsgroup: nnmaildir+nic-nac-project:lists.gnus.org.ding...
gnus-select-newsgroup: Couldn't activate group nnmaildir+nic-nac-project:lists.gnus.org.ding: No such group: lists.gnus.org.ding
#v-
The backslash got weeded out somewhere - amazing that this can even
happen in Lisp and not only in shell programming. :-)

The issue remaining to be resolved for me is that I'd like to have the
status flags synchronized between Gnus and the maildirs because it is
very inconvenient to see messages marked 'read' in Gnus flash up again
as 'unread' ones in mutt.
I have to care about this since I'm not able to fully switch over to
Gnus - it won't be installed on all systems I'm working on - and I don't
accept that I won't be able to use two different mail clients on the
same mail sources.

Looking at the documentation and throwing a quick glance at the list's
archives I heard about this synchronisation not being implemented into
nnmaildir at present but that nnmaildirs maintainer, Paul Jarc, would
eventually be willing to implement it.

With nnmaildir the marks are stored under
[maildir]/.nnmaildir/marks/[type of mark]/[message indentifier].

The problem however is that Gnus uses more marks than the current
maildir specification <URL:http://cr.yp.to/proto/maildir.html> provides:
#v+
[file names in maildir/cur are of the form uniq:info]

info is morally equivalent to the Status field used by mbox readers.
It'd be useful to have MUAs agree on the meaning of info, so I'm keeping a list of info semantics.
Here it is.

info starting with "1,": Experimental semantics.

info starting with "2,": Each character after the comma is an independent flag.

    * Flag "P" (passed): the user has resent/forwarded/bounced this message to someone else.
    * Flag "R" (replied): the user has replied to this message.
    * Flag "S" (seen): the user has viewed this message, though perhaps he didn't read all the way through it.
    * Flag "T" (trashed): the user has moved this message to the trash; the trash will be emptied by a later user action.
    * Flag "D" (draft): the user considers this message a draft; toggled at user discretion.
    * Flag "F" (flagged): user-defined flag; toggled at user discretion. 

New flags may be defined later. Flags must be stored in ASCII order: e.g., "2,FRS".
#v-

I was not yet able to figure out which of the possible marks from
<URL:http://www.gnus.org/manual/gnus_64.html> are actually used and
stored on disk by nnmaildir - and btw.: How is this problem handled e.g.
with Gnus' imap backend? - but the ones also being part of maildir's
specification could easily be used and for the others (dormant, ...)
we could either try to convince DJB to add them to the specification
or the 'experimental semantics' described above could be used.

What do you think?

Regards,
 Thomas

Re: nnmaildir

[Paul Jarc]

Thomas Schwinge <schwinge-lists-gnus.org-ding@nic-nac-project.de> wrote:
> although I was not able to enter the ones whose are containing a
> '\': The were refused with:
> #v+
> Retrieving newsgroup: nnmaildir+nic-nac-project:lists.gnus.org.ding...
> gnus-select-newsgroup: Couldn't activate group nnmaildir+nic-nac-project:lists.gnus.org.ding: No such group: lists.gnus.org.ding
> #v-

I'm not sure what's happening here.  It works for me, with a CVS
checkout from 2004-01-25.  What version of Gnus are you using?

You could work around it by making a new directory for nnmaildir to
look at, which would contain symlinks to the actual maildirs.  The
symlink names could omit "\".

> The issue remaining to be resolved for me is that I'd like to have the
> status flags synchronized between Gnus and the maildirs

You might find this useful for now:
http://article.gmane.org/gmane.emacs.gnus.general/55100

> I was not yet able to figure out which of the possible marks from
> <URL:http://www.gnus.org/manual/gnus_64.html> are actually used and
> stored on disk by nnmaildir

The ones I know of are 'dormant, 'expire, 'forward, 'read, 'reply,
'save, and 'tick.  Note that there is no one-to-one correspondence
between the "marks" that appear in the user interface and the "marks"
that are stored persistently (either by Gnus or by a specific
backend).  For example, "r", "R", and "O" are all represented by the
'read mark in the persistent storage.

> for the others (dormant, ...)  we could either try to convince DJB
> to add them to the specification or the 'experimental semantics'
> described above could be used.

The experimental semantics wouldn't be useful; that would mean that
the message's info would look like ":1,..." which other MUAs wouldn't
understand.


paul

Re: nnmaildir

[Thomas Schwinge]

On Wed, Feb 11, 2004 at 12:42:23PM -0500, Paul Jarc wrote:
> Thomas Schwinge <schwinge-lists-gnus.org-ding@nic-nac-project.de> wrote:
> > although I was not able to enter the ones whose are containing a
> > '\': The were refused with:
> > #v+
> > Retrieving newsgroup: nnmaildir+nic-nac-project:lists.gnus.org.ding...
> > gnus-select-newsgroup: Couldn't activate group nnmaildir+nic-nac-project:lists.gnus.org.ding: No such group: lists.gnus.org.ding
> > #v-
> 
> I'm not sure what's happening here.  It works for me, with a CVS
> checkout from 2004-01-25.  What version of Gnus are you using?

I've been using the CVS checkout with release tag v5-10-6.
nnmaildir.el is dated '2003-04-15'.
I'll try the CVS HEAD version tomorrorw.
Is it sufficient to just copy nnmaildir.el?

> You could work around it by making a new directory for nnmaildir to
> look at, which would contain symlinks to the actual maildirs.  The
> symlink names could omit "\".

Yes, I saw that possibility but I didn't want to go that way because I
didn't want to maintain another hierarchy of folders.
But of course I'll do it that way until I know enough of Emacs' Lisp to
implement the needed functionality by myself. :-)

> > The issue remaining to be resolved for me is that I'd like to have the
> > status flags synchronized between Gnus and the maildirs
> 
> You might find this useful for now:
> http://article.gmane.org/gmane.emacs.gnus.general/55100

So, do you have a working version of such a script? I didn't try this
one because of Bijan Soleymani writing in his second email:
#v+
Oops, I just noticed a couple of small errors in the script. Oh
well... Just consider it as a proof of concept thing. Yes such a
simple script can be written :)
#v-

If you're not going to implement the mark synchronisation functionality
into nnmaildir, that will be the way to go.

> > I was not yet able to figure out which of the possible marks from
> > <URL:http://www.gnus.org/manual/gnus_64.html> are actually used and
> > stored on disk by nnmaildir
> 
> The ones I know of are 'dormant, 'expire, 'forward, 'read, 'reply,
> 'save, and 'tick.

Gnus      | Maildir
----------+---------
dormant   |
expire    | T
forward   | P
read      | S
reply     | R
save      |
tick      | F

'dormant' and 'save' are missing. I'm not quite happy with 'expire'
but it should be possible.

What's the purpose of 'save'? I don't quite get this out of
<URL:http://www.gnus.org/manual/gnus_67.html>.

I'd say that 'dormant' is similar to 'tick', so what about that:
If a message which is new to Gnus has 'F' set, it is displayed as
'ticked'. When the user changes the mark to 'dormant' this is recorded
using the current scheme but the 'F' is preserved.
Clients that don't know how to differentiate between these two will see
maildir's 'F' and Gnus is able to tell if a message is marked as
'ticked' or 'dormant'.

> > for the others (dormant, ...)  we could either try to convince DJB
> > to add them to the specification or the 'experimental semantics'
> > described above could be used.
> 
> The experimental semantics wouldn't be useful; that would mean that
> the message's info would look like ":1,..." which other MUAs wouldn't
> understand.

I had not tried that before - I thought that MUAs would leave the flags
alone the didn't understand.
Now I've got disabused (Does one use this word that way? - I'm always
willing to improve my English.)
Mutt does leave them alone - as long as you don't change any flags.
Doing so, unknown ones will get weeded out.
That's no good behaviour as DJB declares that "New flags may be defined
later." in <URL:http://cr.yp.to/proto/maildir.html>.

Regards,
 Thomas

Re: nnmaildir

[Paul Jarc]

Thomas Schwinge <schwinge-lists-gnus.org-ding@nic-nac-project.de> wrote:
> I'll try the CVS HEAD version tomorrorw.
> Is it sufficient to just copy nnmaildir.el?

Probably, but to be sure you can use one whole, consistent checkout.
The backslash removal might not be happening within nnmaildir anyway.

>> You might find this useful for now:
>> http://article.gmane.org/gmane.emacs.gnus.general/55100
>
> So, do you have a working version of such a script?

No.

> If you're not going to implement the mark synchronisation functionality
> into nnmaildir, that will be the way to go.

I'm just not sure when I'll get to it.

> What's the purpose of 'save'?

In a *Summary* buffer, there are some gnus-summary-save-* commands
(type [O C-h] to see a list); if you run one of these commands to save
an article to a file or whatever, then the 'save mark will be set for
that article.

> If a message which is new to Gnus has 'F' set, it is displayed as
> 'ticked'. When the user changes the mark to 'dormant' this is recorded
> using the current scheme but the 'F' is preserved.

I would make F correspond exactly to 'tick, without involving 'dormant
at all.  Otherwise it's too complicated, and the stored information
may be ambiguous.

>> The experimental semantics wouldn't be useful; that would mean that
>> the message's info would look like ":1,..." which other MUAs wouldn't
>> understand.
>
> I had not tried that before - I thought that MUAs would leave the flags
> alone the didn't understand.

Yes, but they also wouldn't be able to interpret them.  ":1,F" does
not necessarily mean the same thing as ":2,F".  ":2,F" should mean the
same thing to all MUAs, but different MUAs may experiment with
different schemes, and might both use ":1,F" to mean different
things.  An MUA shouldn't change something it doesn't understand, but
with ":1," info, different MUAs may think they understand each other
when they really don't.

> Now I've got disabused (Does one use this word that way? - I'm always
> willing to improve my English.)

I would say "Now I have been disabused of that idea".


paul

Re: nnmaildir

[Thomas Schwinge]

On Wed, Feb 11, 2004 at 06:22:47PM -0500, Paul Jarc wrote:
> Thomas Schwinge <schwinge-lists-gnus.org-ding@nic-nac-project.de> wrote:
> > I'll try the CVS HEAD version tomorrorw.
> > Is it sufficient to just copy nnmaildir.el?
> 
> Probably, but to be sure you can use one whole, consistent checkout.
> The backslash removal might not be happening within nnmaildir anyway.

Now I'm running that latest CVS checkout of Gnus.
Starting it gives some (harmless I presume) warnings:
#v+
[...]
Loading easymenu...done
Loading gnus-start...done
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/nntp.el' newer than byte-compiled file
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/nnoo.el' newer than byte-compiled file
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/nndraft.el' newer than byte-compiled file
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/nnmail.el' newer than byte-compiled file
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/mail-source.el' newer than byte-compiled file
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/nnmh.el' newer than byte-compiled file
Reading /home/thomas/.newsrc.eld...
Loading gnus-group...done
Reading active file from news.schwinge.homeip.net via nntp...
Reading active file via nnmaildir...
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/nnmaildir.el' newer than byte-compiled file
Opening nnmaildir server...done
Reading active file from nic-nac-project via nnmaildir...
Opening nnmaildir server on nic-nac-project...done
Loading gnus-cache...done
Loading timezone...done
Opening nnmaildir server...done
Opening nnmaildir server on nic-nac-project...done
No new newsgroups
Checking new news...
Loading gnus-sum...done
Checking new news...done
Loading gnus-topic...done
Retrieving newsgroup: nnmaildir+nic-nac-project:lists.gnus.org.ding...
Loading gnus-ml...
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/mm-view.el' newer than byte-compiled file
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/mm-uu.el' newer than byte-compiled file
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/mml2015.el' newer than byte-compiled file
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/pgg.el' newer than byte-compiled file
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/pgg-def.el' newer than byte-compiled file
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/pgg-parse.el' newer than byte-compiled file
Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/password.el' newer than byte-compiled file
Loading gnus-ml...done
Opening nnmaildir server on nic-nac-project...done
gnus-select-newsgroup: Couldn't activate group nnmaildir+nic-nac-project:lists.gnus.org.ding: No such group: lists.gnus.org.ding
#v-
#v+
thomas@speedy:~ > ls -ld Mail-nic-nac-project/lists.gnus\\.org.ding/
drwxr-xr-x    6 thomas   users         152 Jan 27 16:18 Mail-nic-nac-project/lists.gnus\.org.ding/
thomas@speedy:~ > tail -3 .gnus.el
(setq gnus-secondary-select-methods
  '((nnmaildir ""                (directory "~/Mail/"))
    (nnmaildir "nic-nac-project" (directory "~/Mail-nic-nac-project/"))))
#v-

I checked it again with a new folder
#v+
thomas@speedy:~ > maildirmake Mail/test\\folder
#v-
which got to
#v+
       0: nnmaildir:testfolder
#v-

Shall I post the strace of Gnus (emacs) when entering such a maildir?

> > If you're not going to implement the mark synchronisation functionality
> > into nnmaildir, that will be the way to go.
> 
> I'm just not sure when I'll get to it.

OK.
I'm aware that you have more important things to do that that. :-)
I wrote to Bijan Soleymani and asked him if he has a working version of
the synchronisation script - otherwise I'll write it myself.

> > What's the purpose of 'save'?
> 
> In a *Summary* buffer, there are some gnus-summary-save-* commands
> (type [O C-h] to see a list); if you run one of these commands to save
> an article to a file or whatever, then the 'save mark will be set for
> that article.

I got that; although I don't quite get what I would do with the
information that an article I'll be reading again later has been saved -
since saving is just copying the article to another folger - if I got
that right.

> > If a message which is new to Gnus has 'F' set, it is displayed as
> > 'ticked'. When the user changes the mark to 'dormant' this is recorded
> > using the current scheme but the 'F' is preserved.
> 
> I would make F correspond exactly to 'tick, without involving 'dormant
> at all.  Otherwise it's too complicated, and the stored information
> may be ambiguous.

That would of course be the cleaner solution.
But how should 'dormant and 'save then be reflected?
In what way would my solution be ambiguous?

> >> The experimental semantics wouldn't be useful; that would mean that
> >> the message's info would look like ":1,..." which other MUAs wouldn't
> >> understand.
> >
> > I had not tried that before - I thought that MUAs would leave the flags
> > alone the didn't understand.
> 
> Yes, but they also wouldn't be able to interpret them.

So it should leave them alone.

> ":1,F" does
> not necessarily mean the same thing as ":2,F".  ":2,F" should mean the
> same thing to all MUAs, but different MUAs may experiment with
> different schemes, and might both use ":1,F" to mean different
> things.  An MUA shouldn't change something it doesn't understand, but
> with ":1," info, different MUAs may think they understand each other
> when they really don't.

OK. So using experimental semantics is not the way to go.
But I was quite surprised when I saw mutt just weed out the flags it
didn't understand.
I think I'll write to the maintainer about that (and I'll check other
maildir-using programs I' using).

Regards,
 Thomas

Re: nnmaildir

[Paul Jarc]

Thomas Schwinge <schwinge-lists-gnus.org-ding@nic-nac-project.de> wrote:
> Now I'm running that latest CVS checkout of Gnus.
> Starting it gives some (harmless I presume) warnings:
> #v+
> [...]
> Loading easymenu...done
> Loading gnus-start...done
> Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/nntp.el' newer than byte-compiled file

Hm... I wonder if you're really running the version you think you're
running.  Make sure your .emacs adds the CVS Gnus to your load-path
before you load any Gnus components.  This may require starting emacs
with --no-site-file, since site-start.el is loaded before ~/.emacs,
and site-start.el might load some Gnus components.  (You can have your
.emacs load site-start.el after your load-path changes.)  If Gnus is
properly installed and properly loaded, I don't think these warnings
should appear.

> Shall I post the strace of Gnus (emacs) when entering such a maildir?

No, but a backtrace would be helpful.  Before triggering the error, do
M-x set-variable RET debug-on-error RET t RET.

> I don't quite get what I would do with the information that an
> article I'll be reading again later has been saved - since saving is
> just copying the article to another folger

Someone thought it was useful once, so now Gnus has it.  Such is the
way of Gnus. :)

> But how should 'dormant and 'save then be reflected?

I don't think there is any way to make them visible to other MUAs.
They will have to stay as they are now.

> In what way would my solution be ambiguous?

Suppose a message has the 'tick mark, and the user changes it to
'dormant.  Gnus tells nnmaildir to remove 'tick and add 'dormant.  You
say that nnmaildir should modify its own marks, but leave "F" in
place.  So now, according to the standard maildir flags, the message
is "F"lagged, but according to nnmaildir's separate marks, the message
is not 'tick-ed.

But there is another way we can arrive at this same situation: suppose
a message initially has neither 'tick or 'dormant, and the user adds
'dormant.  Later, another MUA adds "F".  So now, again, the message
has "F" and 'dormant but not 'tick.

So how do we interpret this state of stored information the next time
Gnus starts?  In the first case, we should presumably say that the
message still does not have 'tick.  In the second case, we would like
to say that the message now has 'tick - the "F" mark was added by
another MUA, and we want to pick up that change in Gnus too.  But
since the stored information is the same in these two cases, we can't
distinguish them, and so we can't get them both right.

>>> I had not tried that before - I thought that MUAs would leave the flags
>>> alone the didn't understand.
>>
>> Yes, but they also wouldn't be able to interpret them.
>
> So it should leave them alone.

I agree.

> But I was quite surprised when I saw mutt just weed out the flags it
> didn't understand.

Yes, that is surprising, and seems buggy.


paul

Re: nnmaildir

[Thomas Schwinge]

On Thu, Feb 12, 2004 at 11:35:58AM -0500, Paul Jarc wrote:
> Thomas Schwinge <schwinge-lists-gnus.org-ding@nic-nac-project.de> wrote:
> > Now I'm running that latest CVS checkout of Gnus.
> > Starting it gives some (harmless I presume) warnings:
> > #v+
> > [...]
> > Loading easymenu...done
> > Loading gnus-start...done
> > Source file `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/nntp.el' newer than byte-compiled file
> 
> Hm... I wonder if you're really running the version you think you're
> running.

I'm quite sure:
#v+
thomas@speedy:~ > strace -f -e open emacs -f gnus 2>&1 | grep '/package.*gnus' | cut -d \" -f 2
/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus
/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/info
/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share
/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs
/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/etc
/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp
/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/etc/gnus
/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/etc/smilies
/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/gnus.elc
[...]
/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/nnmaildir.elc
[...]
thomas@speedy:~ > readlink -f /package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus
/fs/packages/mount/package/misc/spf/gnus-2004-02-12-13:01:23-1
thomas@speedy:~ > readlink -f /package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/nnmaildir.elc
/fs/packages/mount/package/misc/spf/gnus-2004-02-12-13:01:23-1/share/emacs/site-lisp/nnmaildir.elc
thomas@speedy:~ > stat /package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/nnmaildir.el*
  File: `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/nnmaildir.el'
  Size: 63235           Blocks: 136        IO Block: 4096   regular file
Device: fe05h/65029d    Inode: 135956      Links: 1    
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2004-02-12 14:18:22.000000000 +0100
Modify: 2004-02-12 14:17:19.000000000 +0100
Change: 2004-02-12 14:18:33.000000000 +0100

  File: `/package/misc/spf/emacs-21.3-1/share/emacs/site-lisp/gnus/share/emacs/site-lisp/nnmaildir.elc'
  Size: 101500          Blocks: 208        IO Block: 4096   regular file
Device: fe05h/65029d    Inode: 135862      Links: 1    
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2004-02-12 22:56:42.000000000 +0100
Modify: 2004-02-12 14:17:18.000000000 +0100
Change: 2004-02-12 14:18:33.000000000 +0100
#v-

So in fact the .elc file is 1 second newer than the .el file.
Strange.
I've been installing it via sp-foreign <http://multivac.cwru.edu./spf/>.

Now I reinstalled it:
#v+
thomas@speedy:~ > stat /fs/packages/mount/package/misc/spf/gnus-2004-02-12-13:01:23-2/share/emacs/site-lisp/nnmaildir.el*
  File: /fs/packages/mount/package/misc/spf/gnus-2004-02-12-13:01:23-2/share/emacs/site-lisp/nnmaildir.el'
  Size: 63235           Blocks: 136        IO Block: 4096   regular file
Device: fe05h/65029d    Inode: 134768      Links: 1
Access: (0644/-rw-r--r--)  Uid: (  500/  thomas)   Gid: (  100/   users)
Access: 2004-02-12 23:08:15.000000000 +0100
Modify: 2004-02-12 23:08:15.000000000 +0100
Change: 2004-02-12 23:08:15.000000000 +0100

  File: /fs/packages/mount/package/misc/spf/gnus-2004-02-12-13:01:23-2/share/emacs/site-lisp/nnmaildir.elc'
  Size: 101468          Blocks: 208        IO Block: 4096   regular file
Device: fe05h/65029d    Inode: 135556      Links: 1
Access: (0644/-rw-r--r--)  Uid: (  500/  thomas)   Gid: (  100/   users)
Access: 2004-02-12 23:08:16.000000000 +0100
Modify: 2004-02-12 23:08:16.000000000 +0100
Change: 2004-02-12 23:08:16.000000000 +0100
#v-

Restarting Gnus, the warnings did not show up again.


> > Shall I post the strace of Gnus (emacs) when entering such a maildir?
> 
> No, but a backtrace would be helpful.  Before triggering the error, do
> M-x set-variable RET debug-on-error RET t RET.

The folder is displayed the wrong way in the group buffer:
#v+
       *: nnmaildir+nic-nac-project:lists.gnus.org.ding
#v-
whereas it should be
#v+
       *: nnmaildir+nic-nac-project:lists.gnus\.org.ding
#v-

So it's quite obvious that Gnus will print the following:
#v+
Debugger entered--Lisp error: (error "Couldn't activate group nnmaildir+nic-nac-project:lists.gnus.org.ding: No such group: lists.gnus.org.ding")
  signal(error ("Couldn't activate group nnmaildir+nic-nac-project:lists.gnus.org.ding: No such group: lists.gnus.org.ding"))
  error("Couldn't activate group %s: %s" "nnmaildir+nic-nac-project:lists.gnus.org.ding" "No such group: lists.gnus.org.ding")
  gnus-select-newsgroup("nnmaildir+nic-nac-project:lists.gnus.org.ding" nil nil)
  gnus-summary-read-group-1("nnmaildir+nic-nac-project:lists.gnus.org.ding" nil t nil nil nil)
  gnus-summary-read-group("nnmaildir+nic-nac-project:lists.gnus.org.ding" nil t nil nil nil nil)
  gnus-group-read-group(nil t)
  gnus-group-select-group(nil)
  gnus-topic-select-group(nil)
  call-interactively(gnus-topic-select-group)
#v-

So the error is likely be somewhere in the code that recognises new
folders and adds them to the group buffer.


> > I don't quite get what I would do with the information that an
> > article I'll be reading again later has been saved - since saving is
> > just copying the article to another folger
> 
> Someone thought it was useful once, so now Gnus has it.  Such is the
> way of Gnus. :)

So it wouldn't be a great loss if 'save would not be reflected in the
maildir?

> > In what way would my solution be ambiguous?
> 
> Suppose a message has the 'tick mark, and the user changes it to
> 'dormant.  Gnus tells nnmaildir to remove 'tick and add 'dormant.  You
> say that nnmaildir should modify its own marks, but leave "F" in
> place.  So now, according to the standard maildir flags, the message
> is "F"lagged, but according to nnmaildir's separate marks, the message
> is not 'tick-ed.
> 
> But there is another way we can arrive at this same situation: suppose
> a message initially has neither 'tick or 'dormant, and the user adds
> 'dormant.
Additionally nnmaildir will set 'F'.
So, the message has 'F' in maildir and 'dormant in Gnus.
The message it flagged in mutt and dormant in Gnus.

Now I unflag it in mutt: 'F' is removed.
I start Gnus, nnmaildir sees 'dormant set but not 'F' and thus removes
dormant - recognized as being old / invalid.
The message is unflagged in mutt and neither ticked nor dormant in Gnus.

Later - using mutt - I decide that the message actually _is_ important:
I flag it, 'F' is set.
Starting Gnus, nnmaildir sees 'F' but neither 'tick nor 'dormant - they
were deleted in the older session -  so the message is recognized as
being newly flagged: 'tick is set.
The message is flagged in mutt and ticked in Gnus.

Now I decide the the message is not actually _that_ important, but I'd
be interested in follow-ups, so I set it to 'dormant.
'tick is removed, 'dormant is set, 'F' is preserved.
The message is flagged in mutt and dormant in Gnus.

Wouldn't that work?

Regards,
 Thomas

Re: nnmaildir

[Paul Jarc]

Thomas Schwinge <schwinge-lists-gnus.org-ding@nic-nac-project.de> wrote:
> So the error is likely be somewhere in the code that recognises new
> folders and adds them to the group buffer.

Ok... try opening the *Server* buffer (type [^] from the *Group*
buffer) and enter the nnmaildir server.  Then switch to the " *nntpd*"
buffer (note the leading space).  See how the group names show up
there - that will at least tell us whether the problem is inside
nnmaildir or outside.

> So it wouldn't be a great loss if 'save would not be reflected in the
> maildir?

Right.  Other MUAs may not have any such concept, and the standard
maildir info doesn't provide any way to represent that information
anyway.

>> But there is another way we can arrive at this same situation: suppose
>> a message initially has neither 'tick or 'dormant, and the user adds
>> 'dormant.
>
> Additionally nnmaildir will set 'F'.

Ok, that would remove the ambiguity.  But even so, I'd rather keep it
simple, and have "F" correspond exactly to 'tick.


paul

Re: nnmaildir

[Thomas Schwinge]

On Thu, Feb 12, 2004 at 05:56:45PM -0500, Paul Jarc wrote:
> Thomas Schwinge <schwinge-lists-gnus.org-ding@nic-nac-project.de> wrote:
> > So the error is likely be somewhere in the code that recognises new
> > folders and adds them to the group buffer.
> 
> Ok... try opening the *Server* buffer (type [^] from the *Group*
> buffer) and enter the nnmaildir server.  Then switch to the " *nntpd*"
> buffer (note the leading space).  See how the group names show up
> there - that will at least tell us whether the problem is inside
> nnmaildir or outside.

#v+
[...]
lists.skarnet\.org.supervision 46 1 y
lists.gnus\.org.ding 38 1 y
INBOX 26 1 y
[...]
#v-

This is correct whereas it's incorrent when browsing the nnmaildir
server:
#v+
[...]
      38: lists.gnus.org.ding
      72: lists.isdn4linux.de.isdn4linux
     131: lists.multivac.cwru.edu.prjware
[...]
#v-


> >> But there is another way we can arrive at this same situation: suppose
> >> a message initially has neither 'tick or 'dormant, and the user adds
> >> 'dormant.
> >
> > Additionally nnmaildir will set 'F'.
> 
> Ok, that would remove the ambiguity.  But even so, I'd rather keep it
> simple, and have "F" correspond exactly to 'tick.

I wouldn't, but of course that's less urgent than getting the whole
thing working at all.

Regards,
 Thomas

Re: nnmaildir

[Paul Jarc]

Thomas Schwinge <schwinge-lists-gnus.org-ding@nic-nac-project.de> wrote:
> On Thu, Feb 12, 2004 at 05:56:45PM -0500, Paul Jarc wrote:
>> Then switch to the " *nntpd*" buffer
...
> lists.skarnet\.org.supervision 46 1 y
> lists.gnus\.org.ding 38 1 y

Ok, so nnmaildir is reporting the names correctly, but then the
backslashes are lost when Gnus parses this buffer.  Or perhaps there's
some undocumented encoding that needs to be applied to the group
names.  Does anyone know?


paul

Re: nnmaildir

[Kai Grossjohann]

On a tangent: does someone have etymology on the "#v+" and "#v-"
markers?

Kai

#v+, #v- (was: nnmaildir)

[lawrence mitchell]

Kai Grossjohann wrote:

> On a tangent: does someone have etymology on the "#v+" and "#v-"
> markers?

It's a slrnism, they're meant to mark verbatim text. e.g.

#v+
Here is some text
(defun foo (bar)
  bar)
it should be considered as "verbatim"
#v-

i.e. anything in between them is (in slrn) fontified in a
different colour, and not wrapped.

I believe they're not supported in other mail/newsreaders
though.

-- 
lawrence mitchell <s0198183+ding@sms.ed.ac.uk>

Re: #v+, #v-

[Kai Grossjohann]

lawrence mitchell <s0198183+ding@sms.ed.ac.uk> writes:

> I believe they're not supported in other mail/newsreaders
> though.

Well, anyone up to doing it for Gnus?  Sounds like a cool feature.  We
*do* already have MIME multiparts, but one feature more, what can it
hurt...

Kai

Re: #v+, #v-

[Karl Pflästerer]

On 14 Feb 2004, Kai Grossjohann <- kai@emptydomain.de wrote:

> lawrence mitchell <s0198183+ding@sms.ed.ac.uk> writes:

>> I believe they're not supported in other mail/newsreaders
>> though.

> Well, anyone up to doing it for Gnus?  Sounds like a cool feature.  We
> *do* already have MIME multiparts, but one feature more, what can it
> hurt...


Well I did a first study to see what maybe had to be done but had quick
some questions which I couldn't answer myself without deeply studying
the sources.

(defun gnus-article-search-verbose ()
  (let (verbose pt)
    (save-match-data
      (save-excursion
	(goto-char (point-min))
	(while (re-search-forward "^#v\\+" nil t)
	  (setq pt (match-beginning 0))
	  (and (re-search-forward "^#v\\(.\\)" nil t)
	       (string-equal (match-string 1) "-")
	       (push (cons pt (match-end 0)) verbose))
	  (unless (eobp) (goto-char (point-at-bol))))))
    (nreverse verbose)))

(defun gnus-article-highlight-verbose () 
  (dolist (region (gnus-article-search-verbose))
    (gnus-overlay-put (gnus-make-overlay (car region) (cdr region))
		      'face 'gnus-header-subject-face)))

(defvar gnus-treat-highlight-verbose t)
(put 'gnus-treat-highlight-verbose 'highlight t)

(setq gnus-treatment-function-alist
	(nconc gnus-treatment-function-alist
	       '((gnus-treat-highlight-verbose gnus-article-highlight-verbose))))


The above code finds the regions of interest and highlights them (I
choose some arbitrary face just to have an effect).

But we wanted more: wrapping (== filling) should be inhibited IIRC.  Can
this be done with overlays?  How should the highlighting be done? Like
here with just one face or do we want different faces for different
tokens?



   KP

-- 
Männer der Wissenschaft! Man sagt ihr viele nach, 
aber die meisten mit Unrecht.  
                             Karl Kraus 'Aphorismen'

db-backed mail back end

[Lars Magne Ingebrigtsen]

Now that there are nice, free SQL-based database servers out there
(PostgreSQL, and, er, PostgreSQL), would it be an idea to have a
db-backed mail back end?  What would you gain by such a beast, and
what would you lose?

1) You would be able to search
2) You could create new groupings on-the-fly -- by altering
   the "splitting" criteria, you would alter which messages
   end up where (perhaps) (or not)
3) erm...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: db-backed mail back end

[Daniel Pittman]

On Wed, 23 Jan 2002, Lars Magne Ingebrigtsen wrote:
> Now that there are nice, free SQL-based database servers out there
> (PostgreSQL, and, er, PostgreSQL), would it be an idea to have a
> db-backed mail back end?  

Oh, yes, it would indeed. IMO, of course, and I never did get off my
backside and work out if it was worth the trouble to migrate to any of:
* SQL
* GDBM, &c.
* Some random Lisp reader format...

> What would you gain by such a beast, and what would you lose?

Gain?  For me, the biggest pain with Gnus right now is that there are
some scalability issues. I got sick of losing information that had been
on mailing lists, so I stopped expiring mail a couple of years ago.

Disk space is sufficiently cheap, even for a laptop, that I don't have
space issues with this. Gnus starts to feel a little pain, sometimes,
though when dealing with groups in the > 75,000 messages range (with
NNML.)

> 1) You would be able to search

...this is one of the places; I go for 'M-x grep' and 'j' to search
messages...

> 2) You could create new groupings on-the-fly -- by altering
>    the "splitting" criteria, you would alter which messages
>    end up where (perhaps) (or not)
> 3) erm...

...hopefully, some improvement in performance. Er, especially with
XEmacs and Postgresql, where there is a native implementation of the
client connection stuff. Mmmm, native C code.

At the very least it should improve the speed of finding files and
returning them, which is the second biggest performance issue.
Especially now that some of my NNML groups are too big for a standard
'ls *' to work on. ;)

        Daniel

-- 
It is quite humbling to realize that the storage occupied by the longest line
from a typical Usenet posting is sufficient to provide a state space so vast
that all the computation power in the world can not conquer it.
        -- Dave Wallace

Re: db-backed mail back end

[Lars Magne Ingebrigtsen]

Daniel Pittman <daniel@rimspace.net> writes:

> Gain?  For me, the biggest pain with Gnus right now is that there are
> some scalability issues. I got sick of losing information that had been
> on mailing lists, so I stopped expiring mail a couple of years ago.

A db doesn't necessarily give you any performance gain.  If you're not
careful with how you set things up, you can get arbitrarily awful db
performance (and I should know, since I've been Oracling a lot these
past years. :-))

> Disk space is sufficiently cheap, even for a laptop, that I don't have
> space issues with this. Gnus starts to feel a little pain, sometimes,
> though when dealing with groups in the > 75,000 messages range (with
> NNML.)

Perhaps the question should be restated as such:

If we were to design a mail back end that's supposed to scale to
several hundred thousands of messages in thousands of groups -- how
would we do that?  Perhaps somebody has pondered this question
before.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: db-backed mail back end

[Russ Allbery]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> If we were to design a mail back end that's supposed to scale to several
> hundred thousands of messages in thousands of groups -- how would we do
> that?  Perhaps somebody has pondered this question before.  :-)

Don't store all the articles in one directory; the file system
implementation will kill you.

Preindex the stuff that you usually want to search for.  .overview is sort
of a basic step in that direction, but for truly vast archives you need
something like Glimpse or htdig or whatnot.  You probably also don't want
just one flat index; you want one index on sender, maybe an index on
recipient, a text index of the message, maybe an index on subject
depending on how you use subject, etc.  And you want to limit searches by
a date range.

-- 
Russ Allbery (rra@stanford.edu)             <http://www.eyrie.org/~eagle/>

Re: db-backed mail back end

[Lars Magne Ingebrigtsen]

Russ Allbery <rra@stanford.edu> writes:

> Don't store all the articles in one directory; the file system
> implementation will kill you.

Yup.  The obvious solution to that would be using some sort of
hierarchy -- using (say) pairs of digits of the article number as
directory names.  Or something.

"d45/d34/d43/4" for the message "4534434".

Or perhaps the other way around:

"45/34/43/a4" for the message "4534434".

> Preindex the stuff that you usually want to search for.  .overview
> is sort of a basic step in that direction, but for truly vast
> archives you need something like Glimpse or htdig or whatnot.

Speaking over .overview files -- putting a 20kline .overview file into
a buffer and then looking through that buffer for the 2 lines we want
is s-l-o-w.  I see two obvious solutions.

1) Split the .overview file into an .overview directory with several
   1kline .overview files in it
2) Implement the basic .overview "find-the-right-line" thing in C. 

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: db-backed mail back end

[Paul Jarc]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> Speaking over .overview files -- putting a 20kline .overview file into
> a buffer and then looking through that buffer for the 2 lines we want
> is s-l-o-w.

nnmaildir stores NOV information in one file per message.  Finding an
article's NOV information is as fast as finding the article itself.


paul

nnmaildir (was: db-backed mail back end)

[Josh Huber]

prj@po.cwru.edu (Paul Jarc) writes:

> nnmaildir stores NOV information in one file per message.  Finding
> an article's NOV information is as fast as finding the article
> itself.

Indeed, I've been meaning to try nnmaildir out.  Would it be very
painful to migrate from nnml to maildir?  Perhaps I'll try it for just
a couple groups to start with...

ttyl,

-- 
Josh Huber

Re: nnmaildir (was: db-backed mail back end)

[Harry Putnam]

Josh Huber <huber@alum.wpi.edu> writes:

> prj@po.cwru.edu (Paul Jarc) writes:
>
>> nnmaildir stores NOV information in one file per message.  Finding
>> an article's NOV information is as fast as finding the article
>> itself.
>
> Indeed, I've been meaning to try nnmaildir out.  Would it be very
> painful to migrate from nnml to maildir?  Perhaps I'll try it for just
> a couple groups to start with...

I'd be interested in hearing your results... I had lots of trouble
trying to run both nnml and nnmaildir, but since then Paul J. has
fixed lots of stuff.  I haven't tried running both since before the
fixes though.  Still using nnmaildir alone on a secondary machine, but
my main mail/news, on the main machine, is still non-nnmaildir and
mostly nnml.

I think, if you have some semi-large to large groups you may notice
some significant slowness in initial startup as nnmaildir processes
everything, but after starting, you may find group entry to be
noticably faster.  I don't hnow how much of that applies since Pauls
new release and other fixes.

One thing attractive about nnmaildir to me is that it doesn't care
about adding and subtracting from outside gnus.  It just finds and
processes what is there (in `new') on entry.  Also plays ball nicely
with procmail (like nnml does) since procmail has a builtin
recognition of maildir setup, and knows how to write to nnmaildir
groups.

Re: nnmaildir (was: db-backed mail back end)

[Paul Jarc]

Harry Putnam <reader@newsguy.com> wrote:
> I had lots of trouble trying to run both nnml and nnmaildir, but
> since then Paul J. has fixed lots of stuff.

I think all your troubles were strictly nnmaildir-related, or at least
not related to running nnml as well.

> I think, if you have some semi-large to large groups you may notice
> some significant slowness in initial startup as nnmaildir processes
> everything, but after starting, you may find group entry to be
> noticably faster.  I don't hnow how much of that applies since Pauls
> new release and other fixes.

I haven't made any changes in that area.


paul

Re: nnmaildir (was: db-backed mail back end)

[Harry Putnam]

prj@po.cwru.edu (Paul Jarc) writes:

> Harry Putnam <reader@newsguy.com> wrote:
>> I had lots of trouble trying to run both nnml and nnmaildir, but
>> since then Paul J. has fixed lots of stuff.
>
> I think all your troubles were strictly nnmaildir-related, or at least
> not related to running nnml as well.
>
>> I think, if you have some semi-large to large groups you may notice
>> some significant slowness in initial startup as nnmaildir processes
>> everything, but after starting, you may find group entry to be
>> noticably faster.  I don't hnow how much of that applies since Pauls
>> new release and other fixes.
>
> I haven't made any changes in that area.

I wasn't sure because I'm using the most recent version, but not
really using it hard like before.  I gave up trying to convert news to
mail that way.  So have no large groups to test it on.  My giving up
was not nnmailder related.  Just seemed like too much complication
using suck, procmail and then gnus to get the results I wanted.  Some
where in that chain of events I was generating lots of duplicates.
Not related to the apps I don't think, but my processing scripts.

There is one small thing that I can think of that might be improved
some how.  Those filenames are pretty ridiculous if you do anything
with those files outside gnus.  Seems like a simple incremental
numbering scheme once the messags are brought from `new' would work.

These are a tad unweildy:
1008622921.3916_0.sol.local.lan:2,
1011293422.3800_0.sol.local.lan:2,

Re: nnmaildir

[Paul Jarc]

Harry Putnam <reader@newsguy.com> wrote:
> There is one small thing that I can think of that might be improved
> some how.  Those filenames are pretty ridiculous if you do anything
> with those files outside gnus.  Seems like a simple incremental
> numbering scheme once the messags are brought from `new' would work.

It works as long as you never have two processes moving mail from new/
to cur/ at the same time.  Otherwise, you still need an uncoordinated
way to ensure filenames don't collide.


paul

Re: nnmaildir

[Paul Jarc]

Josh Huber <huber@alum.wpi.edu> wrote:
> prj@po.cwru.edu (Paul Jarc) writes:
>> nnmaildir stores NOV information in one file per message.  Finding
>> an article's NOV information is as fast as finding the article
>> itself.
>
> Indeed, I've been meaning to try nnmaildir out.

Note that finding the NOV information is also just as *slow* as
finding the article. :)  It's up to you to either keep your groups
small or use a filesystem that can deal with large directories.

> Would it be very painful to migrate from nnml to maildir?

Setting up the server should be pretty straightforward.  Copying the
articles is as easy as B c.  The group parameters for expiry will be
different from what you're used to, but hopefully not too troublesome.


paul