Re: Slowdowns with big nnmaildir groups

[Daniel Pittman <daniel@rimspace.net>]

Tassilo Horn <tassilo@member.fsf.org> writes:
> David <de_bb@arcor.de> writes:
>
> Hi David,
>
>> Use the elisp profiler (ELP)
>>
>> M-x elp-instrument-package RET gnus
>> ... perform slow operations ...
>> M-x elp-results
>
> Hey, cool.  I didn't know it's included in emacs.  Before your post I
> downloaded a elp.el from emacswiki but that didn't work with emacs 23.

You probably want to instrument nnmaildir, or possibly nn, as well as gnus.

[...]

> But I guess the relations between function times are correct.  In all
> examples `gnus-retrieve-headers' (called two times) is a huge factor
> in the overall time.  Only with 6000 messages scoring, sorting and
> threading needs more time.  So I think, this is the function that
> causes the slowdown.

IIRC the nnmaildir code tries to play nice with other Maildir clients,
which means reading all the (giant) filenames from disk, which probably
works the poor kernel quite hard, especially with such a large
directory.

Worse, this sort of thing has been known to trigger other bugs.  
I suspect that you will find this is a design feature of the Maildir
system, and that nothing much can be done about it.

(Also, remember that reading messages extends the length of the name, so
expect the directory space to be fairly fragmented as well.)

Ensuring your Maildir is on a filesystem that handles large directories
well is probably the best bet for improving performance.

Regards,
        Daniel
-- 
Daniel Pittman <daniel@cybersource.com.au>           Phone: 03 9428 6922
1/130-132 Stawell St, Richmond              Web: http://www.cyber.com.au
Cybersource: Australia's Leading Linux and Open Source Solutions Company