overview file access when spooling and nnml/nnimap performance

[Hannu Koivisto <azure@iki.fi>]

Greetings,

I'm looking for ways to handle my mail (and news, but that's not
relevant now) faster without getting new hardware.  I have almost
always used nnml because it has scaled better than anything else,
but nowadays I'm not quite satisfied with it either.  Getting new
mail from the spool, even few (5--20) articles, is much slower than
it should, IMHO, be on my PII/266 Linux machine with spools (I
split mail externally with procmail) and nnml folders both on
UW-SCSI disk.  Also entering groups with large amount of articles
is quite slow, but I believe there is really nothing that can be
done for it as Elisp is probably simply too slow for reading the
relevant information from disk and generating the summary buffer.

So, I started to think about what happens when Gnus gets new mail
from the spool.  What seems a real overkill to me is that it seems
to read the whole overview file for each group for which there is
mail to memory, do some magic (that I didn't quite understand and
which probably is the problem here for me thinking this is an
overkill), add new lines and then write it back to disk.  (Please
correct if I'm wrong about this.)  I.e. if I get even just few new
mails to several groups, Gnus ends up reading like 5+Mb data from
disk, fiddling with it and writing it back.  Perhaps this can't be
avoided, in which case, I thought, some sort of database solution
for storing articles would perhaps be the right direction.  This
brings us to nnimap.  How's its performance with some good IMAP
server (does such things exist)?  Somehow I doubt this, but as
someone suggested going for IMAP when someone else was asking for
some sort of database-backend, I'm a bit curious.

-- 
Hannu