improving nnmaildir performance for constant messages (long)?

improving nnmaildir performance for constant messages (long)?

[James Leifer]

I've been doing some experiments with nnmaildir on a test group with
10,000 messages.  Current setup: latest cvs gnus, emacs 21.2.5, 2.5
Ghz CPU, 512MB, Debian woody, ext3.

Everything is working smoothly, though a bit slowly in comparison to
an equivalent 10,000 message nnmh group.  Since adding this maildir it
takes about 17 seconds longer to start gnus, and substantially longer
than nnmh to rescan (M-g) the last 200 messages (5 seconds vs 1
second) and the last 5000 (13 seconds vs 8 seconds) in a group.  (All
my figures are wristwatch calculated, so not so accurate, and repeated
a few times to try to avoid cache vs non-cache distortions.)

In order to try to understand what was going on I ran emacs under
strace.  It seems that nnmaildir is stat-ing a whole lot of files in
order to be really safe.  E.g. when gnus starts up,

* for every uniq (maildir file name) there is a stat of
  foo/.nnmaildir/marks/read/uniq.

* every foo/cur/uniq:2, is stated

* every foo/.nnmaildir/nov/uniq is stated twice and then opened

When gnus enters a nnmaildir group

* for every uniq that needs to be displayed, foo/cur/uniq:2, is stated
  and foo/.nnmaildir/nov/uniq is stated

It seems that to stat many files takes much longer than to simply get
the list of file names (since the latter just reads the directory as a
file while the former has to look up the file's inode I believe).

So why all this stating?  Well I assume that Paul Jarc designed
nnmaildir to be robust in the face of messages or nov data getting
modified.  I'm wondering, therefore, how much faster we could make
nnmaildir if we were willing to adopt the following principle:

* Once a message is added (foo/cur/uniq:2,flags) its contents are
  _constant_, i.e. never modified.

(If one wants to ``modify'' it then one has to rename it to a new
unique name.)

As a result, foo/.nnmaildir/nov/uniq will always be correct as long as
it too is unmodified.

Given that I never knowingly modify any message, I would be happy to
accept this principle in return for faster performance: I believe that
most of the stat-ing could be then eliminated or replaced by simply
listing directories.

Thoughts?

-James

Re: improving nnmaildir performance for constant messages (long)?

[Adam Sjøgren]

On Sat, 06 Dec 2003 19:31:17 +0100, James wrote:

> Thoughts?

Did you try measuring nnml?


  Best regards,

-- 
 "Do not feed the oysters under the clouds"                   Adam Sjøgren
                                                         asjo@koldfront.dk

Re: improving nnmaildir performance for constant messages (long)?

[James Leifer]

spamtrap@koldfront.dk (Adam Sjøgren) writes:
...
> Did you try measuring nnml?

Hi Adam,

I haven't, lacking motivation: I wanted to use nnmaildir to get away
from the nnmh/nnml discipline of integer message numbers so that it
would be easy to synchronize groups that had been separately modified
on my laptop and desktop (see
http://article.gmane.org/gmane.emacs.gnus.general/55107 or
<r77vfourls7.fsf@muscadet.inria.fr>).

Do you think the result would be interesting?

Best wishes,

-James (facing too many things to do!)