Re: Huge memory consumption on accessing large newsgroup

[Daniel Pittman <daniel@rimspace.net>]

Ted Zlatanov <tzz@lifelogs.com> writes:
> On Tue, 02 Oct 2007 13:23:56 +1000 Daniel Pittman <daniel@rimspace.net> wrote: 
>
> DP> The issue with large memory consumption, as far as I could see
> DP> from the thread, was that the compressed range data type was
> DP> expanded to a flat list.  This caused, no surprise, huge memory
> DP> use.
>
> DP> The original problem is that the code calls
> DP> `gnus-uncompress-range' on the data *at all* -- and, so, turns a
> DP> nicely brief data structure into a vast bloated million-number
> DP> list.
>
> DP> The *solution* is to rebuild the algorithm to operate on the
> DP> compressed version (regardless of the internal representation),
> DP> not to change the representation.
>
> You're right, I didn't know this.  I thought the memory problems were
> caused by the original list.  

*nod*  I figured as much. :)

[...]

> DP> (And because this has been a stupidly annoying couple of week in other
> DP>  areas, and because this is nice simple and essentially stress-free work
> DP>  I am getting tempted to fix it myself.
>
> DP>  So, maybe inversion lists were the way to get the code fixed after all,
> DP>  if not quite so directly as expected. ;) 
>
> The `gnus-uncompress-range' function is not called in too many places:

[...]

> In gnus-number-of-unseen-articles-in-group for example it's called
> only to find the length of the list, and in gnus-move-group-to-server
> only to see if the list is not nil.  

Mmm.  The not nil version should really be killed, as a guide, and the
length isn't too awful.  The length version can just call
`gnus-range-length' and be done with it...

> There's room for improvement.  Maybe there should be a group API that
> abstracts the data structures, so subordinate code doesn't have to
> know if it's a list or a compressed range?  That would be a good first
> step to cleaning things up.

Maybe.  I think in most cases it is just that the range code wasn't
pushed everywhere it could be because, in most cases, you never /really/
see the pain.

Regards,
        Daniel
-- 
Daniel Pittman <daniel@cybersource.com.au>           Phone: 03 9621 2377
Level 4, 10 Queen St, Melbourne             Web: http://www.cyber.com.au
Cybersource: Australia's Leading Linux and Open Source Solutions Company