memory usage results (ack!)

memory usage results (ack!)

[Wes Hardaker]

Ok, so I upgraded my xemacs to 21.5.4 (the most recent beta) and
played with gnus for a while:

hardaker  1649 10.2 34.3 98716 87628 pts/2   S    16:53   3:32 xemacs
                         ^^^^^

nice nearly 100meg size.

This time, however, I compiled xemacs with memory debugging support.
Quickly tinking to figure out how to use it produces:

        (progn
          (mapcar (function (lambda (z)
        		      (insert (format "%s:\n" (buffer-name z)))
        		      (mapcar (function (lambda (x) (insert (format "  %s %s\n" (symbol-name (car x)) (cdr x))))) (buffer-memory-usage z))
        		      z)) (buffer-list t)))

The above prints memory usage stats for every buffer in xemacs.

Example output:

*scratch*:
  text 8216
  markers 1824
  extents 0
  other 512
  nil nil
  actually-requested 4357
  malloc-overhead 3921
  gap-overhead 2274
  dynarr-overhead 0

So, the text of the scratch buffer is small (8216), and the rest you
can guess it (I won't elaborate here).

So, sorting the entire output by sizes > 1M (using perl (gasp):

% perl -n -e '$x = $_ if (/:/); print $x, $_ if (/ \d{7}/);
 *nntpd*:
  text 34088184
 *nntpd*:
  actually-requested 29622319
 *nntpd*:
  gap-overhead 4365513
 *Gnus agent overview*:
  text 34088184
 *Gnus agent overview*:
  actually-requested 1181163
 *Gnus agent overview*:
  gap-overhead 32805805

Yikes.  Ok, so the nttpd buffer (note all these have a leading space)
is 34megs.  The agent overview is also 34 megs.  Double ouch.  What I
don't get is that the actual size of the buffer is much smaller (92097
of the latter) in, so the text field obviously isn't getting cleaned
up these days in either buffer.  But why?  So, lets be brave, save
this message buffer and kill both these buffers (not that it should
matter from a message sending point of view, but I don't remember how
well gnus deals with important deleted buffers).

Ahhh:
hardaker  1649  9.3 10.0 32332 25668 pts/2   S    16:53   3:53 xemacs
                         ^^^^^

So, the question is (especially for the emacs experts):  What's
happening with these two buffers and how do we fix it?

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."

Agent .overview files not getting truncated (was Re: memory usage results (ack!))

[Wes Hardaker]

>>>>> On Wed, 06 Feb 2002 17:36:33 -0800, Wes Hardaker <wes@hardakers.net> said:

...
Wes> *nntpd*:
Wes> text 34088184
...

Wes> Yikes.  Ok, so the nttpd buffer (note all these have a leading space)
Wes> is 34megs.   The agent overview is also 34 megs.  Double ouch.

Ok, I think I've figured out where the memory has gone:

% ll .overview
-rw-------    1 hardaker wheel    13765613 Feb  7 14:08 .overview
% wc -l .overview
  47372 .overview

So, there are 47k lines in my agent's .overview file.  But there are
only a handfull of messages in the directory itself.  So this quickly
made me check what else is in the .overview file:

% egrep '^10015' .overview | wc -l
     36

Ahh...  So 36 lines are functionally duplicates for a single message.
But why?

So, I move this file aside and reenter the group and bang!  What a
speed difference (and I'm sure a memory difference too, but I probably
need to restart emacs to claim everything).  Now the .overview file:

% egrep '^10015' .overview | wc -l
      1

Yay.  A single line.

keystrokes:  q ENTER

Nada.  No change.  A good thing.

keystrokes:  q 2 g ENTER

% egrep '^10015' .overview | wc -l
      2

Ack.  And so we begin again collecting too much data.

Apparently the new auto-header-fetch code is duplicating stuff?

% ll `find . -name .overview` | awk '{ size=$5 + size } END { print size }'
15750346

15 megs of overview files anyone?  (and this is after removing the two
largest ones which were 13 megs and 18 megs each.

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."

Re: Agent .overview files not getting truncated (was Re: memory usage results (ack!))

[ShengHuo ZHU]

Wes Hardaker <wes@hardakers.net> writes:


[...]

> % egrep '^10015' .overview | wc -l
>      36
>
> Ahh...  So 36 lines are functionally duplicates for a single message.
> But why?
>
> So, I move this file aside and reenter the group and bang!  What a
> speed difference (and I'm sure a memory difference too, but I probably
> need to restart emacs to claim everything).  Now the .overview file:
>
> % egrep '^10015' .overview | wc -l
>       1
>
> Yay.  A single line.
>
> keystrokes:  q ENTER
>
> Nada.  No change.  A good thing.
>
> keystrokes:  q 2 g ENTER
>
> % egrep '^10015' .overview | wc -l
>       2
>
> Ack.  And so we begin again collecting too much data.
>
> Apparently the new auto-header-fetch code is duplicating stuff?

Emm, it looks like a bug. But I didn't see duplication in my .overview
files. Maybe there is something wrong in gnus-agent-braid-nov. Could
you debug it?

ShengHuo

Re: Agent .overview files not getting truncated (was Re: memory usage results (ack!))

[Wes Hardaker]

>>>>> On Thu, 07 Feb 2002 18:24:15 -0500, ShengHuo ZHU <zsh@cs.rochester.edu> said:

>> keystrokes:  q 2 g ENTER
>> 
>> % egrep '^10015' .overview | wc -l
>> 2
>> 
>> Ack.  And so we begin again collecting too much data.
>> 
>> Apparently the new auto-header-fetch code is duplicating stuff?

ZSH> Emm, it looks like a bug. But I didn't see duplication in my
ZSH> .overview files. Maybe there is something wrong in
ZSH> gnus-agent-braid-nov. Could you debug it?

Well, first of in debugging it I've found the following things:

1) it only appears if there is in fact new mail in the folder
   (otherwise gnus-agent-braid-nov doesn't even get called).

2) the problem comes from the fact that the if statement which decides
   whether to insert the contents of the old .overview file into the
   nntp buffer bases it's decision on the last article number in
   the list of articles and the .overview file.  If the list of
   articles contains anything newer, then the *entire* contents of the
   .overview file is inserted.  I think this is where the duplication
   is going on.

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."

Re: Agent .overview files not getting truncated (was Re: memory usage results (ack!))

[ShengHuo ZHU]

Wes Hardaker <wes@hardakers.net> writes:

> Well, first of in debugging it I've found the following things:
>
> 1) it only appears if there is in fact new mail in the folder
>    (otherwise gnus-agent-braid-nov doesn't even get called).
>
> 2) the problem comes from the fact that the if statement which decides
>    whether to insert the contents of the old .overview file into the
>    nntp buffer bases it's decision on the last article number in
>    the list of articles and the .overview file.  If the list of
>    articles contains anything newer, then the *entire* contents of the
>    .overview file is inserted.  I think this is where the duplication
>    is going on.

Thank you for analysis. I guess you've customized
gnus-fetch-old-headers.  Anyway, I fixed the problem.

To fix the .overview files, you can remove all these files, then run
M-x gnus-agent-regenerate.

ShengHuo

Re: Agent .overview files not getting truncated (was Re: memory usage results (ack!))

[Wes Hardaker]

>>>>> On Fri, 08 Feb 2002 01:02:37 -0500, ShengHuo ZHU <zsh@cs.rochester.edu> said:

ZSH> Thank you for analysis. I guess you've customized
ZSH> gnus-fetch-old-headers.

I don't do that new-fangled-graphic-editing stuff.  But yes:

% grep gnus-fetch-old-headers ~/.gnus
(setq gnus-fetch-old-headers 'some)

ZSH> Anyway, I fixed the problem.

Excellent.  Thanks.

ZSH> To fix the .overview files, you can remove all these files, then
ZSH> run M-x gnus-agent-regenerate.

Great!

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."