Re: Huge memory consumption on accessing large newsgroup

[Ted Zlatanov <tzz@lifelogs.com>]

On Mon, 01 Oct 2007 11:04:54 +1000 Daniel Pittman <daniel@rimspace.net> wrote: 

DP> Ted Zlatanov <tzz@lifelogs.com> writes:

>> Consider using inversion lists.  

>> They don't require pairs of integers; each value represents a flip.
>> They have other nice properties too, though it's been a while since I
>> looked at them so I can't name them.  

DP> Mmm.  The existing Gnus range code is pretty much as efficient, has
DP> lower computational complexity for the operations Gnus uses and is
DP> already existent and tested.

We're talking about reducing memory consumption, not CPU usage.  For CPU
usage the current structure is pretty good.  For memory usage, as you
can see from the subject of the message, it's not ideal.

We could use several data structures as needed to accomodate large data
lists.  Inversion lists are nice in some cases and not others; I think
they are worth at least some consideration since they don't require a
pair of numbers to express a range and thus may save some memory.  I
don't know the Emacs internals intimately, but I think (24 86 88 90)
takes up less memory than ((24 . 86) (88 . 90)).  Am I wrong?

Note Gaute's original message was about compressing pairs of integers,
not general Gnus data structures, so please consider my message in that
context.

DP> I don't think that you would see sufficient benefit from introducing the
DP> additional data type to justify spending your time on it -- but since I
DP> am not volunteering to do the work I can't tell you how to do it. ;)

I am not sure if you are addressing me or Gaute.  I made a suggestion,
and an implementation is already in place (I mentioned Kim Storm
implemented it).  I'm willing to assist Gaute, but he has to be
interested in my suggestion.  I don't have the time for this as a solo
project.

Ted