Re: Huge memory consumption on accessing large newsgroup

[Ted Zlatanov <tzz@lifelogs.com>]

On Tue, 02 Oct 2007 13:23:56 +1000 Daniel Pittman <daniel@rimspace.net> wrote: 

DP> The issue with large memory consumption, as far as I could see from the
DP> thread, was that the compressed range data type was expanded to a flat
DP> list.  This caused, no surprise, huge memory use.

DP> The original problem is that the code calls `gnus-uncompress-range' on
DP> the data *at all* -- and, so, turns a nicely brief data structure into a
DP> vast bloated million-number list.

DP> The *solution* is to rebuild the algorithm to operate on the compressed
DP> version (regardless of the internal representation), not to change the
DP> representation.

You're right, I didn't know this.  I thought the memory problems were
caused by the original list.  In light of your explanation, I completely
agree that the right way is to find all instances of
`gnus-uncompress-range' and fix their consequences.  Thank you for
explaining.

DP> (And because this has been a stupidly annoying couple of week in other
DP>  areas, and because this is nice simple and essentially stress-free work
DP>  I am getting tempted to fix it myself.

DP>  So, maybe inversion lists were the way to get the code fixed after all,
DP>  if not quite so directly as expected. ;) 

The `gnus-uncompress-range' function is not called in too many places:

gnus-agent.el:7
gnus-draft.el:1
gnus-group.el:2
gnus-move.el:4
gnus-nocem.el:1
gnus-range.el:2
gnus-start.el:2
gnus-sum.el:9
nnimap.el:1
nnmaildir.el:3
nnsoup.el:1
nnvirtual.el:1

In gnus-number-of-unseen-articles-in-group for example it's called only
to find the length of the list, and in gnus-move-group-to-server only to
see if the list is not nil.  There's room for improvement.  Maybe there
should be a group API that abstracts the data structures, so subordinate
code doesn't have to know if it's a list or a compressed range?  That
would be a good first step to cleaning things up.

Ted