Re: db-backed mail back end

[Zlatko Calusic <zlatko.calusic@iskon.hr>]

Karl Kleinpaste <karl@charcoal.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>> NNML scales linearly with the number of messages in the group, does
>> it not?  Gnus definitely doesn't scale linearly, so redesigning
>> something to get support for large mail/news backends should go into
>> Gnus, I think.
>
> I have been wondering about this since this discussion started.  The
> slowness of entering a large group of 10K messages, or 100K, has very
> little to do with getting at either the overview data or the message
> files.  It has to do with threading, scoring, and sorting.  I don't
> know what Gnus' threading algorithm is, but I suspect its performance
> is at least as poor as O(n log n) and may even be as bad as O(n^2).

It is definitely O(n^2), I was measuring it recently. I have an nnml
group (ok, linux-kernel list, why keep it secret :)) which hovers
around the number of 15000 unread mails (12k - 17k typicaly, last few
months). It takes minutes! to enter that group on a quite fast machine
with a gobs of memory. It's no mystery I'm not eager to start reading
that list, and tomorrow things get only worse. :)

My observings are that threading and sorting take a lot of time (not
scoring!). And I found it is exactly O(n^2). If I read, kill,
whatever...  30% of articles in the group, elapsed time during
entering the group cuts in half. I was delibaretely measuring it to
see what is the upper limit of usability. Quite lowish, if I may add.

-- 
Zlatko