Re: opening speed for nnml groups

[Russ Allbery <rra@stanford.edu>]

Lars Balker Rasmussen <lbr@mjolner.dk> writes:
> Steve Folkes <smf@rivers.dra.hmg.gb> writes:

>> It may be in the Unix filesystem itself.  I had a problem with our
>> newsfeed a few years ago that put about 10,000 files in one directory;
>> it took over a week to remove them all (using "rm -rf").

> What Unix would that be?

> % mkdir test
> % perl -e 'foreach $a (1..10000) { open FOO, ">test/$a" }'
> % time rm -rf test
> 0.60u 11.18s 1:55.40 10.2%
> % uname -a
> SunOS heimdal 5.5.1 Generic_103640-08 sun4u sparc

A week is a bit long, but when I cut back on the expire time for
control.cancel on our news server from ten days to half a day, it took
fastrm (which is a good bit faster than rm -rf under most circumstances)
the majority of a day to remove the files in the directory, and that was
an UltraSPARC 2 on a RAID array.  The directory itself, not including its
contents, was over 1MB.

> But of course, the Unix filesystem gets slower as more files are in a
> directory, but it's not _that_ bad.

Unless I'm mistaken, Unix stores a directory as a linked list of files.
Each time you open or delete a file, it has to traverse the linked list to
find it, and each time you create a file, it has to traverse the linked
list to find a hole.  It should quickly become clear from that why large
directories cause a slowdown.

(There could also be internal Gnus or emacs performance issues that
overshadow that, such as the .overview handling -- on that score, I have
no idea.)

-- 
Russ Allbery (rra@stanford.edu)         <URL:http://www.eyrie.org/~eagle/>