Re: Slowdowns with big nnmaildir groups

[Daniel Pittman <daniel@rimspace.net>]

Tassilo Horn <tassilo@member.fsf.org> writes:
> Daniel Pittman <daniel@rimspace.net> writes:
>
>> You probably want to instrument nnmaildir, or possibly nn, as well as
>> gnus.
>
> I did that (with nnmaildir) and the results are in
> <87abmk3k1f.fsf@member.fsf.org>.
>
>> IIRC the nnmaildir code tries to play nice with other Maildir clients,
>> which means reading all the (giant) filenames from disk, which
>> probably works the poor kernel quite hard, especially with such a
>> large directory.
>
> Indeed, the filenames are quite big, like:
>
>   1201848492_1.4393.baldur,U=10600,FMD5=72a62f382d4bdeb4a771c2d3ea493c5d:2,:2,
>
>> Ensuring your Maildir is on a filesystem that handles large
>> directories well is probably the best bet for improving performance.
>
> Hm, it's on a ext3 filesystem.  Maybe I can improve speed with the
> dir_index option...

It certainly won't hurt.  Renewing the directory on a semi-regular
basis, by creating a new one, moving all content to it, then replacing
the original, will also help.

> OfflineIMAP has the possibility to sync an IMAP group to a mbox
> instead of a maildir, but I guess that wouldn't improve the situation.
> Currently the emacs-devel folder is about 140M big, which would exceed
> the buffer size limit on my 30Bit machine.

Yes, it wouldn't do much good if it was bigger than your Emacs can read.

For performance I would suggest one of two paths:

1. Use OfflineIMAP to sync IMAP to IMAP, and run a local IMAP server.
   This doesn't fix any performance problem with nnimap proper, but does
   insulate you from the performance hit of Maildir.
   (I recommend Dovecot for this purpose, if you must.)

2. Write an nnml (better) or nnmh (easier) back-end for OfflineIMAP,
   contribute it upstream, and use that.  nnml with NOV support would be
   the best option, but there may be significant complexity in getting
   this to work coupled with renaming and moving messages between
   folders.

It looks like their code is reasonably well structured and the basics of
adding a new back-end will be easy.

I don't think you will find the improvements you are looking for with
the nnmaildir back-end, though I can't say that with absolute certainty.

Regards,
        Daniel
-- 
Daniel Pittman <daniel@cybersource.com.au>           Phone: 03 9428 6922
1/130-132 Stawell St, Richmond              Web: http://www.cyber.com.au
Cybersource: Australia's Leading Linux and Open Source Solutions Company