Re: Huge memory consumption on accessing large newsgroup

[Daniel Pittman <daniel@rimspace.net>]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Sat, 29 Sep 2007 22:04:18 +0100 Gaute Strokkenes <gs234@srcf.ucam.org> wrote:
>
> GS> I wonder if it would be possible to make Gnus work solely with
> GS> compressed ranges (i.e. lists where dotted pairs are used to
> GS> represent runs of consecutive integers)?
>
> GS> Unless there is some deeper reason why this cannot work, I might
> GS> have a stab at it (eventually).
>
> I think this would be a good idea.  

Ah.

> Consider using inversion lists.  

This is almost certainly unnecessary, not to mention that it would
involve building an entire parallel infrastructure to handle them.

The nnimap code had a similar performance killing "feature" where it
would expand two 'range' lists completely, intersect them, then compress
them again.

This was trivially resolved by using the existing code from
`gnus-range.el' to process this on the compressed versions.

You should be able to find the appropriate bit of history tucked away in
the history of the nnimap.el code via CVS.

I am also happy to try and dig up my memories of the work though,
frankly, they were trivial enough.


The one specific advice I would give you:

Write your code so it is "self testing" and run with that for a long
while. I ended up having the code do the gnus-range based calculations
and compare them to the non-range calculations, then signal an error if
they disagreed.

This cost extra CPU time for the couple of weeks I used it in production
but gave me (and, I think, the rest of the list) a much higher sense of
security that the changes were, in practice, correct.

(You might want to leave that in with a debug option to turn it on so
 that the rest of the Gnus CVS userbase also test this, to catch faults
 that your own use doesn't show.

 I didn't do that and I vaguely regret it now.)

> They don't require pairs of integers; each value represents a flip.
> They have other nice properties too, though it's been a while since I
> looked at them so I can't name them.  

Mmm.  The existing Gnus range code is pretty much as efficient, has
lower computational complexity for the operations Gnus uses and is
already existent and tested.

I don't think that you would see sufficient benefit from introducing the
additional data type to justify spending your time on it -- but since I
am not volunteering to do the work I can't tell you how to do it. ;)

Regards,
        Daniel
-- 
Daniel Pittman <daniel@cybersource.com.au>           Phone: 03 9621 2377
Level 4, 10 Queen St, Melbourne             Web: http://www.cyber.com.au
Cybersource: Australia's Leading Linux and Open Source Solutions Company