Gnus slow incorporating vast quantities of mail

Gnus slow incorporating vast quantities of mail

[David Z Maze]

I was travelling on business for a few days last week, and so my
incoming mail spool got sort of big.  (Being on debian-* lists will do
that, somehow... :-)  Looking at the Incoming... file:

{15} Mail% grep -c ^L Incoming-4FNdp 
2475
{16} Mail% ls -l Incoming-4FNdp      
-rw-------   1 dmaze    mit       8785983 Jan 12 14:07 Incoming-4FNdp

Gnus seems to have pulled the mail in just fine, but AFAICT sorting it
into folders was really slow; I didn't get to the group buffer for
about half an hour.  (Hardware is a Sun Fire 280R, two 750 MHz
UltraSPARC-IIIs, 2.0 GB of RAM, I think I was the only serious user of
the machine at the time; CPU load wasn't an issue, but network latency
of getting to the mail store over AFS probably was.)  My suspicion is
that nnmail-split-fancy can be O(n^2) in the size of the incoming mail
spool, meaning that you'll get acceptable performance unless you're
trying to slurp in a truly vast quantity of mail (as I did here).

More configuration details: using nnml (into an AFS directory) fed by
a (Kerberized) POP server, sorting using a moderately complex
nnmail-split-fancy rule set.  Most of the time it works reasonably;
'ls -lSr ~/Mail' says that the new three largest Incoming... files I
have are 2.5, 1.4, and 1.2 MB, respectively, and I haven't noticed
sorting those being too painful.

Any hints as to what might cause this?  Is this an issue in Gnus, or
my splitting rules, or something else?  Is there any easy way to debug
it without letting my mail back up for a week?  :-)  TIA...

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Gnus slow incorporating vast quantities of mail

[Russ Allbery]

David Z Maze <dmaze@MIT.EDU> writes:

> More configuration details: using nnml (into an AFS directory) fed by a
> (Kerberized) POP server, sorting using a moderately complex
> nnmail-split-fancy rule set.  Most of the time it works reasonably; 'ls
> -lSr ~/Mail' says that the new three largest Incoming... files I have
> are 2.5, 1.4, and 1.2 MB, respectively, and I haven't noticed sorting
> those being too painful.

> Any hints as to what might cause this?

Did a lot of that mail sort into the same directory?  I don't trust AFS
file creation speed, and I particularly don't trust AFS with large
directories.

Note also that AFS performance can vary a *lot* depending on network
conditions, file server load, and a lot of other things that are hard to
measure or control.

-- 
Russ Allbery (rra@stanford.edu)             <http://www.eyrie.org/~eagle/>

Re: Gnus slow incorporating vast quantities of mail

[David Z Maze]

Russ Allbery <rra@stanford.edu> writes:
> David Z Maze <dmaze@MIT.EDU> writes:
>> More configuration details: using nnml (into an AFS directory) fed
>> by a (Kerberized) POP server, sorting using a moderately complex
>> nnmail-split-fancy rule set.  Most of the time it works reasonably;
>> 'ls -lSr ~/Mail' says that the new three largest Incoming... files
>> I have are 2.5, 1.4, and 1.2 MB, respectively, and I haven't
>> noticed sorting those being too painful.
> 
>> Any hints as to what might cause this?
> 
> Did a lot of that mail sort into the same directory?  I don't trust
> AFS file creation speed, and I particularly don't trust AFS with
> large directories.

I think there were over a thousand messages to debian-user, for
example.  I don't think they were particularly clustered, though.
Hmm.  (1409 distinct messages in mail.lists.debian.user currently.)

> Note also that AFS performance can vary a *lot* depending on network
> conditions, file server load, and a lot of other things that are
> hard to measure or control.

<nods>

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
...and if you think slurping in 8.5MB of mail is slow, try 'ls -F /afs'.