Re: overview file access when spooling and nnml/nnimap performance

[Hannu Koivisto <azure@iki.fi>]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

| I think nnimap is even slower than nnml because the NOV parsing done
| by Gnus is so fast.  In fact, nnimap now includes a NOV cache feature
| which fetches overview info from the IMAP server only once.  After
| that, the overview information is stored in a file in ~/News and
| subsequently fetched from there.

Hm, I don't quite follow why this gives a reason why nnimap is even
slower than nnml.  After all, Gnus parses NOV stuff less in
nnimap's case as it doesn't have to touch overview info when mail
is getting spooled to folders -- IMAP server does that.  It has to
handle overview info when entering a group but that's same for nnml
and nnimap, isn't it?  Of course, without caching, that overview
file would be read over network, but after all, I was more
concerned about optimizing the spooling phase which in IMAP's case
is handled on the background, like you say below.  Perhaps there
would be some way to get asynchronous external spooling directly to
nnml folders too, but, unlike with nnmh, that would require locking
here and there and is probably far from easy to do.

| I use a Cyrus server and I do server-side splitting with Sieve.  So

Thanks, I'll check those out.  I hope that splitting doesn't
require anything special and could be done with procmail instead of
that Sieve thingy.  I'd rather avoid converting my 9k .procmailrc
into another system.

| getting new mail is real fast -- O(n) where n is the number of
| subscribed groups.  The splitting happens in the background.  That's a
| nice feature of the Gnus/nnimap combination.

Indeed.

| slow.  Recently, I moved all articles from one 3500 message nnml group
| to an 8000 message group, and that was real slow.  Towards the end,

yeah, and when you have 80000 articles instead of 8000, the real
fun begins :)

| of the last line is found.  And writing new lines could happen by
| appending to the existing file.  Yes, that might be faster for large
| overview files, but what's the size where it starts winning.

Yes, that's what I thought too (and as far as I can see, it should
start winning immediately or something is broken), but the fact
that Gnus possibly removes some lines from those overview files or
whatever (this is the part I didn't quite grok when reading the
source) shoots down the idea of appending.

| As to displaying large summary buffers, I find that I normally get by
| with just displaying few messages.  I switched from total-expire to

So do I, normally, but there are few folders that I don't touch for
a long time and then read all accumulated messages.  This would be
fine (it wouldn't hit me often) if I didn't peek into those folders
for something important every now and then.

| Rather than displaying thousands of messages to search for an old one,
| I use nnir to search for it.  (Should use nnir much more often,
| though.)

Hmm, I'll have to check that out too.  I assume it handles
full-text regexp searches, yes?

-- 
Hannu