Agent .overview files not getting truncated (was Re: memory usage results (ack!))

[Wes Hardaker <wes@hardakers.net>]

>>>>> On Wed, 06 Feb 2002 17:36:33 -0800, Wes Hardaker <wes@hardakers.net> said:

...
Wes> *nntpd*:
Wes> text 34088184
...

Wes> Yikes.  Ok, so the nttpd buffer (note all these have a leading space)
Wes> is 34megs.   The agent overview is also 34 megs.  Double ouch.

Ok, I think I've figured out where the memory has gone:

% ll .overview
-rw-------    1 hardaker wheel    13765613 Feb  7 14:08 .overview
% wc -l .overview
  47372 .overview

So, there are 47k lines in my agent's .overview file.  But there are
only a handfull of messages in the directory itself.  So this quickly
made me check what else is in the .overview file:

% egrep '^10015' .overview | wc -l
     36

Ahh...  So 36 lines are functionally duplicates for a single message.
But why?

So, I move this file aside and reenter the group and bang!  What a
speed difference (and I'm sure a memory difference too, but I probably
need to restart emacs to claim everything).  Now the .overview file:

% egrep '^10015' .overview | wc -l
      1

Yay.  A single line.

keystrokes:  q ENTER

Nada.  No change.  A good thing.

keystrokes:  q 2 g ENTER

% egrep '^10015' .overview | wc -l
      2

Ack.  And so we begin again collecting too much data.

Apparently the new auto-header-fetch code is duplicating stuff?

% ll `find . -name .overview` | awk '{ size=$5 + size } END { print size }'
15750346

15 megs of overview files anyone?  (and this is after removing the two
largest ones which were 13 megs and 18 megs each.

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."