Maildir backend

Maildir backend

[Jérôme Marant]

  Hi,

  I've tried nnmaildir but I think that it is not satisfactory
  since it takes a lot of resources (mostly heavy duplication of
  files) , it is pretty slow (the more files there are the slower
  it starts).

  I would like to know if someone has already thought about
  writing a lighter maildir backend (or modifying the existing
  one) acting like the one in Mutt, i.e. not duplication of files,
  no NOV, and renaming files with respect to their status (read,
  signed, and so on).

  Thanks.

  Cheers,

  PS: I'm sorry I can't really help on this since I have only
  little skill in elisp.

-- 
Jérôme Marant

Re: Maildir backend

[Paul Jarc]

Jérôme Marant <jerome.marant@fr.thalesgroup.com> wrote:
>   I've tried nnmaildir but I think that it is not satisfactory
>   since it takes a lot of resources (mostly heavy duplication of
>   files)

What do you mean?

>   it is pretty slow (the more files there are the slower
>   it starts).

Yes, I've got some ideas to improve that.

>   no NOV,

What's wrong with NOV?

>   and renaming files with respect to their status (read,
>   signed, and so on).

I don't think the standard maildir flags-in-filename scheme is
sufficient to accomodate all of Gnus's marks (and certainly not
user-defined marks).  We could have a different flags-in-filename
scheme that would be more accomodating, but it would be unrecognizable
to other maildir readers.  Would that still be worthwhile for you?
What would be the advantages over nnmaildir's current mark system?


paul

Re: Maildir backend

[Paul Jarc]

Jérôme Marant <jerome.marant@fr.thalesgroup.com> wrote:
>   I've tried nnmaildir but I think that it is not satisfactory
>   since it takes a lot of resources (mostly heavy duplication of
>   files)

What do you mean?

>   it is pretty slow (the more files there are the slower
>   it starts).

Yes, I've got some ideas to improve that.

>   no NOV,

What's wrong with NOV?

>   and renaming files with respect to their status (read,
>   signed, and so on).

I don't think the standard maildir flags-in-filename scheme is
sufficient to accomodate all of Gnus's marks (and certainly not
user-defined marks).  We could have a different flags-in-filename
scheme that would be more accomodating, but it would be unrecognizable
to other maildir readers.  Would that still be worthwhile for you?
What would be the advantages over nnmaildir's current mark system?


paul

Re: Maildir backend

[Jérôme Marant]

On Wed, Mar 06, 2002 at 11:40:27AM -0500, Paul Jarc wrote:
> Jérôme Marant <jerome.marant@fr.thalesgroup.com> wrote:
> >   I've tried nnmaildir but I think that it is not satisfactory
> >   since it takes a lot of resources (mostly heavy duplication of
> >   files)
> 
> What do you mean?

  I mean that it creates a .nnmaildir directory in the mail box
  directory. And it seems that all files from the cur directory
  are duplicated there. I would like the backend to avoid that
  duplication (even if I run a XFS filesystem).

> 
> >   it is pretty slow (the more files there are the slower
> >   it starts).
> 
> Yes, I've got some ideas to improve that.

  Avoid file duplication would be the first improvement I guess.

> 
> >   and renaming files with respect to their status (read,
> >   signed, and so on).
> 
> I don't think the standard maildir flags-in-filename scheme is
> sufficient to accomodate all of Gnus's marks (and certainly not
> user-defined marks).  We could have a different flags-in-filename
> scheme that would be more accomodating, but it would be unrecognizable
> to other maildir readers.  Would that still be worthwhile for you?
> What would be the advantages over nnmaildir's current mark system?

  I'm not that sure that a flags-in-filename scheme would be better.
  I'm not even waiting for any compatibility with other MUA.

  Why aren't directories new, cur and tmp plus a marks file (a kind
  of index file used to store the status of messages) enough to
  handle maildir ?

  Cheers,

-- 
Jérôme Marant

Re: Maildir backend

[Jérôme Marant]

On Wed, Mar 06, 2002 at 11:40:27AM -0500, Paul Jarc wrote:
> Jérôme Marant <jerome.marant@fr.thalesgroup.com> wrote:
> >   I've tried nnmaildir but I think that it is not satisfactory
> >   since it takes a lot of resources (mostly heavy duplication of
> >   files)
> 
> What do you mean?

  I mean that it creates a .nnmaildir directory in the mail box
  directory. And it seems that all files from the cur directory
  are duplicated there. I would like the backend to avoid that
  duplication (even if I run a XFS filesystem).

> 
> >   it is pretty slow (the more files there are the slower
> >   it starts).
> 
> Yes, I've got some ideas to improve that.

  Avoid file duplication would be the first improvement I guess.

> 
> >   and renaming files with respect to their status (read,
> >   signed, and so on).
> 
> I don't think the standard maildir flags-in-filename scheme is
> sufficient to accomodate all of Gnus's marks (and certainly not
> user-defined marks).  We could have a different flags-in-filename
> scheme that would be more accomodating, but it would be unrecognizable
> to other maildir readers.  Would that still be worthwhile for you?
> What would be the advantages over nnmaildir's current mark system?

  I'm not that sure that a flags-in-filename scheme would be better.
  I'm not even waiting for any compatibility with other MUA.

  Why aren't directories new, cur and tmp plus a marks file (a kind
  of index file used to store the status of messages) enough to
  handle maildir ?

  Cheers,

-- 
Jérôme Marant

Re: Maildir backend

[Paul Jarc]

Jérôme Marant <jerome.marant@fr.thalesgroup.com> wrote:
>   I mean that it creates a .nnmaildir directory in the mail box
>   directory.

Right...

>   And it seems that all files from the cur directory
>   are duplicated there.

No.  Different files with similar names are created there.

>   Why aren't directories new, cur and tmp plus a marks file (a kind
>   of index file used to store the status of messages) enough to
>   handle maildir ?

new/, cur/, and tmp/ alone would be enough to provide the necessary
Gnus functionality.  nnmaildir maintains two additional sets of
information: NOV data and marks.  Storing NOV data speeds things up.
Storing marks adds flexibility when working with mailboxes from
outside Gnus.  Storing marks in a single file per group, as is done
with nnml and nnfolder (which I assume is what you were referring to
above), would also add flexibility, but not as much.  If we give up
the extra flexibility, what do we get in return?


paul

Re: Maildir backend

[Paul Jarc]

Jérôme Marant <jerome.marant@fr.thalesgroup.com> wrote:
>   I mean that it creates a .nnmaildir directory in the mail box
>   directory.

Right...

>   And it seems that all files from the cur directory
>   are duplicated there.

No.  Different files with similar names are created there.

>   Why aren't directories new, cur and tmp plus a marks file (a kind
>   of index file used to store the status of messages) enough to
>   handle maildir ?

new/, cur/, and tmp/ alone would be enough to provide the necessary
Gnus functionality.  nnmaildir maintains two additional sets of
information: NOV data and marks.  Storing NOV data speeds things up.
Storing marks adds flexibility when working with mailboxes from
outside Gnus.  Storing marks in a single file per group, as is done
with nnml and nnfolder (which I assume is what you were referring to
above), would also add flexibility, but not as much.  If we give up
the extra flexibility, what do we get in return?


paul

Re: Maildir backend

[Jérôme Marant]

On Thu, Mar 07, 2002 at 01:00:44PM -0500, Paul Jarc wrote:

> >   And it seems that all files from the cur directory
> >   are duplicated there.
> 
> No.  Different files with similar names are created there.

  But they have to be created anyway.

> 
> >   Why aren't directories new, cur and tmp plus a marks file (a kind
> >   of index file used to store the status of messages) enough to
> >   handle maildir ?
> 
> new/, cur/, and tmp/ alone would be enough to provide the necessary
> Gnus functionality.  nnmaildir maintains two additional sets of
> information: NOV data and marks.  Storing NOV data speeds things up.
> Storing marks adds flexibility when working with mailboxes from
> outside Gnus.  Storing marks in a single file per group, as is done
> with nnml and nnfolder (which I assume is what you were referring to
> above), would also add flexibility, but not as much.  If we give up
> the extra flexibility, what do we get in return?

  I'd rather say : if you don't loose anything with adding this
  extra flexibility, then there is no reason for not making it
  possible, unless you're not willing to spend some time on it.

  If it is possible to have new, cur, tmp and a single mark file
  per mailbox (i.e. without the .nnmaildir directory), then I'll
  be happy.

  Thanks in advance.

  Cheers,

-- 
Jérôme Marant

Re: Maildir backend

[Paul Jarc]

Jérôme Marant <jerome.marant@fr.thalesgroup.com> wrote:
> On Thu, Mar 07, 2002 at 01:00:44PM -0500, Paul Jarc wrote:
>>>   And it seems that all files from the cur directory
>>>   are duplicated there.
>>
>> No.  Different files with similar names are created there.
>
>   But they have to be created anyway.

Why is this a problem?


paul

Re: Maildir backend

[Paul Jarc]

Jérôme Marant <jerome.marant@fr.thalesgroup.com> wrote:
> On Thu, Mar 07, 2002 at 01:00:44PM -0500, Paul Jarc wrote:
>>>   And it seems that all files from the cur directory
>>>   are duplicated there.
>>
>> No.  Different files with similar names are created there.
>
>   But they have to be created anyway.

Why is this a problem?


paul

Re: Maildir backend

[Jérôme Marant]

On Fri, Mar 08, 2002 at 11:37:07AM -0500, Paul Jarc wrote:
> Jérôme Marant <jerome.marant@fr.thalesgroup.com> wrote:
> > On Thu, Mar 07, 2002 at 01:00:44PM -0500, Paul Jarc wrote:
> >>>   And it seems that all files from the cur directory
> >>>   are duplicated there.
> >>
> >> No.  Different files with similar names are created there.
> >
> >   But they have to be created anyway.
> 
> Why is this a problem?

  They should not be mandatory and they take too much room
  on my system.

  But, well, it is your software and you can perfectly decide
  not to add this flexibility. I won't blame you for that.

  As I said, I'm interested in a mutt/Wanderlust -like maildir
  backend. Would you mind explaining what in nnmaildir has
  to be modified in order to have, for instance, a filename
  based maildir backend ?

  Thanks.

-- 
Jérôme Marant

Re: Maildir backend

[Paul Jarc]

Jérôme Marant <jerome.marant@fr.thalesgroup.com> wrote:
>   They should not be mandatory and they take too much room
>   on my system.

I don't know how you're measuring the total size, but there's a good
chance the measurement you have is inaccurately high.  The files used
to store marks are actually hard links to a single file, so each
additional mark file doesn't take up any more space except for storing
its name in the directory.  No additional inodes are used.

The NOV information does take up extra inodes and space, possibly
quite a bit depending on what filesystem you use.  It's there only to
speed things up; it doesn't affect the behavior.  So you can do
"rm server-directory/*/.nnmaildir/nov/*" between (or even during) Gnus
sessions to save space if you're willing to accept the performance
hit.  I could also change nnmaildir so that if it's unable to write
the NOV files, it continues on anyway.  Then you could do
"chmod a-w server-directory/*/.nnmaildir/nov" just once and new NOV
files wouldn't be created.  That change shouldn't be too hard.

>   Would you mind explaining what in nnmaildir has to be modified in
>   order to have, for instance, a filename based maildir backend ?

Storing marks in the filenames, you mean?  -request-set-mark and
-request-update-info would have to be changed, at least.  I'm not sure
about anything else.


paul

Re: Maildir backend

[Jérôme Marant]

On Tue, Mar 12, 2002 at 09:30:21AM -0500, Paul Jarc wrote:
> Jérôme Marant <jerome.marant@fr.thalesgroup.com> wrote:
> >   They should not be mandatory and they take too much room
> >   on my system.
> 
> I don't know how you're measuring the total size, but there's a good
> chance the measurement you have is inaccurately high.  The files used
> to store marks are actually hard links to a single file, so each
> additional mark file doesn't take up any more space except for storing
> its name in the directory.  No additional inodes are used.

  Yes, I was probably wrong. I'm trying to understand why it is
  slow on startup.

  Could you please explain how nnmaildir works, mainly the content of
  .nnmaildir/marks? There is also a markfile file that seems to
  be always empty. What is it for?
 
> The NOV information does take up extra inodes and space, possibly
..
> files wouldn't be created.  That change shouldn't be too hard.

NOV can be disable if nov-is-evil is non nil right? 


> >   Would you mind explaining what in nnmaildir has to be modified in
> >   order to have, for instance, a filename based maildir backend ?
> 
> Storing marks in the filenames, you mean?  -request-set-mark and
> -request-update-info would have to be changed, at least.  I'm not sure
> about anything else.

  Well, I'll forget about that.

  Thanks.

  Cheers,

-- 
Jérôme Marant

Re: Maildir backend

[Paul Jarc]

Jérôme Marant <jerome.marant@fr.thalesgroup.com> wrote:
>   I'm trying to understand why it is slow on startup.

I haven't profiled it, but I'm pretty sure it's because it read all
the NOV information at startup.  It uses this information to build a
fast mapping among filenames, article numbers, and Message-IDs, which
speeds things up later when we want to find a message by Message-ID,
for example.

Currently article numbers are only stored in memory, so they change
between Gnus sessions.  This messes with the cache, the agent, and
'seen marks, so I'm going to change it when I have some time.  As part
of that revision, I think I can get rid of the slow startup.  E.g., I
might create a new subdirectory in .nnmaildir/ which contains files
with names like this:
154:<20020313115128.A9018@fr.thalesgroup.com>:1016016979.15575.multivac.cwru.edu
I.e., article-number:message-id:unique-part-of-filename-in-cur.  Then
when we're looking for an article, we can just scan this directory,
looking for whichever identification field we happen to have.  Since
all the information here is contained in the filename, the files can
be empty, and can even be hard links to a single file to avoid
consuming lots of inodes.

>   Could you please explain how nnmaildir works, mainly the content of
>   .nnmaildir/marks? There is also a markfile file that seems to
>   be always empty. What is it for?

Have you read <URL:http://multivac.cwru.edu./nnmaildir/using/#marks>?

> NOV can be disable if nov-is-evil is non nil right?

Not for nnmaildir.  Supporting such an option means more code and more
work.  I think it's better to spend that effort on making the NOV code
work correctly, so no one cares about turning it off.


paul

Re: Maildir backend

[Matt Armstrong]

prj@po.cwru.edu (Paul Jarc) writes:

> Jérôme Marant <jerome.marant@fr.thalesgroup.com> wrote:
>>   I'm trying to understand why it is slow on startup.
>
> I haven't profiled it, but I'm pretty sure it's because it read all
> the NOV information at startup.  It uses this information to build a
> fast mapping among filenames, article numbers, and Message-IDs,
> which speeds things up later when we want to find a message by
> Message-ID, for example.
>
> Currently article numbers are only stored in memory, so they change
> between Gnus sessions.  This messes with the cache, the agent, and
> 'seen marks, so I'm going to change it when I have some time.  As
> part of that revision, I think I can get rid of the slow startup.
> E.g., I might create a new subdirectory in .nnmaildir/ which
> contains files with names like this:
> 154:<20020313115128.A9018@fr.thalesgroup.com>:1016016979.15575.multivac.cwru.edu
> I.e., article-number:message-id:unique-part-of-filename-in-cur.
> Then when we're looking for an article, we can just scan this
> directory, looking for whichever identification field we happen to
> have.  Since all the information here is contained in the filename,
> the files can be empty, and can even be hard links to a single file
> to avoid consuming lots of inodes.

Have you considered placing NOV data in a single .overview file like
nnml?  What is the advantage to keeping nov data in separate files?
My guess is that this will be faster for the common case: when folks
are running on a traditional unix file system.  But it probably boils
down to a philosophical argument.  Neither nnml nor nnmailder are
probably the best tool for storing HUGE mail folders.

-- 
matt

Re: Maildir backend

[Paul Jarc]

Matt Armstrong <matt@lickey.com> wrote:
> Have you considered placing NOV data in a single .overview file like
> nnml?

nnmaildir used to do that.

> What is the advantage to keeping nov data in separate files?

Historically, one of the reasons I made that change was so that you
could remove an individual NOV file to force nnmaildir to reparse the
message itself.  You can still do that, but it's no longer necessary.
The message is now automatically reparsed when either it or
nnmail-extra-headers has changed.

More recently, there were some speed changes to nnml, having to do
with having to load and parse the entire .overview file to find just
one line.  nnmaildir punts to the filesystem; the kernel parses the
directory file to find the entry we're looking for.  The kernel has
the ability to organize the directory in such a way that it won't have
to read the whole thing linearly, so there's an opportunity to beat
nnml there.  (It's too bad the opportunity is so often missed.)  nnml
also has the disadvantage of wading through all the information for
all the articles, rather than just identifiers and pointers to the
identified information.  And it's all in text format.

> Neither nnml nor nnmailder are probably the best tool for storing
> HUGE mail folders.

Probably true at least for traditional filesystems.


paul

Re: Maildir backend

[Jérôme Marant]

On Thu, Mar 07, 2002 at 01:00:44PM -0500, Paul Jarc wrote:

> >   And it seems that all files from the cur directory
> >   are duplicated there.
> 
> No.  Different files with similar names are created there.

  But they have to be created anyway.

> 
> >   Why aren't directories new, cur and tmp plus a marks file (a kind
> >   of index file used to store the status of messages) enough to
> >   handle maildir ?
> 
> new/, cur/, and tmp/ alone would be enough to provide the necessary
> Gnus functionality.  nnmaildir maintains two additional sets of
> information: NOV data and marks.  Storing NOV data speeds things up.
> Storing marks adds flexibility when working with mailboxes from
> outside Gnus.  Storing marks in a single file per group, as is done
> with nnml and nnfolder (which I assume is what you were referring to
> above), would also add flexibility, but not as much.  If we give up
> the extra flexibility, what do we get in return?

  I'd rather say : if you don't loose anything with adding this
  extra flexibility, then there is no reason for not making it
  possible, unless you're not willing to spend some time on it.

  If it is possible to have new, cur, tmp and a single mark file
  per mailbox (i.e. without the .nnmaildir directory), then I'll
  be happy.

  Thanks in advance.

  Cheers,

-- 
Jérôme Marant

Maildir backend

[Jérôme Marant]

  Hi,

  I've tried nnmaildir but I think that it is not satisfactory
  since it takes a lot of resources (mostly heavy duplication of
  files) , it is pretty slow (the more files there are the slower
  it starts).

  I would like to know if someone has already thought about
  writing a lighter maildir backend (or modifying the existing
  one) acting like the one in Mutt, i.e. not duplication of files,
  no NOV, and renaming files with respect to their status (read,
  signed, and so on).

  Thanks.

  Cheers,

  PS: I'm sorry I can't really help on this since I have only
  little skill in elisp.

-- 
Jérôme Marant