large .overview files in agentized nntp groups

large .overview files in agentized nntp groups

[Steinar Bang]

I finally figured out why the emacs process size balooned every time I
had something to do with the newsgroup no.alt.motorsykler:
The ~/News/agent/nntp/news.chello.no/no/alt/motorsykler/.overview file
was 22MB.  At that point Gnus/emacs refused to load it.

I deleted the file, and did a fresh `g', followed by `J s', and
everything was fine, with a 2kB .overview, rather than a 22MB one.

Is there a recommended way to do this kind of housecleaning?  Manually
deleting files in the hierarchies maintained by Gnus feels like a
hack.

Re: large .overview files in agentized nntp groups

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Is there a recommended way to do this kind of housecleaning?  Manually
> deleting files in the hierarchies maintained by Gnus feels like a
> hack.

Doesn't `gnus-agent-expire' delete stuff from the NOV files?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: large .overview files in agentized nntp groups

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Steinar Bang <sb@dod.no> writes:

>> Is there a recommended way to do this kind of housecleaning?
>> Manually deleting files in the hierarchies maintained by Gnus feels
>> like a hack.

> Doesn't `gnus-agent-expire' delete stuff from the NOV files?

I tried running that command just before deleting the NOV file, but it
didn't change the size of the NOV file, so I guess the answer is
"no". :-)

Re: large .overview files in agentized nntp groups

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

>> Doesn't `gnus-agent-expire' delete stuff from the NOV files?
>
> I tried running that command just before deleting the NOV file, but it
> didn't change the size of the NOV file, so I guess the answer is
> "no". :-)

I tried the same just now, and the NOV files shrunk as expected.  So
I'm unable to reproduce this bug.

Looking over the code, I see that it removes all NOV lines that don't
refer to any articles in the directories...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: large .overview files in agentized nntp groups

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Steinar Bang <sb@dod.no> writes:
>>> Doesn't `gnus-agent-expire' delete stuff from the NOV files?
>> 
>> I tried running that command just before deleting the NOV file, but it
>> didn't change the size of the NOV file, so I guess the answer is
>> "no". :-)

> I tried the same just now, and the NOV files shrunk as expected.  So
> I'm unable to reproduce this bug.

22MB is pretty big.  Maybe it's a problem that only occur with very
large NOV files?  There were some 9000-odd cached articles in that
directory after the expiry, and that's more than a week's worth, even
on no.alt.motorsykler.

Maybe agent expiry has a problem with too many articles/too large a
NOV file?

> Looking over the code, I see that it removes all NOV lines that don't
> refer to any articles in the directories...

Hm... wonder why the regenerated NOV file was so small?  Did it only
contain the new articles?

Re: large .overview files in agentized nntp groups

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> 22MB is pretty big.  Maybe it's a problem that only occur with very
> large NOV files?  There were some 9000-odd cached articles in that
> directory after the expiry, and that's more than a week's worth, even
> on no.alt.motorsykler.

I've got agentized groups with more than 15000 articles in them, and
they seem to be cleaned up OK by the expiry function...

Could you try running it again and look at .overview files
before/after running it, to see whether it works at all for you?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: large .overview files in agentized nntp groups

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Steinar Bang <sb@dod.no> writes:

>> 22MB is pretty big.  Maybe it's a problem that only occur with very
>> large NOV files?  There were some 9000-odd cached articles in that
>> directory after the expiry, and that's more than a week's worth,
>> even on no.alt.motorsykler.

> I've got agentized groups with more than 15000 articles in them, and
> they seem to be cleaned up OK by the expiry function...

Something very strange seem to have happened to my no.alt.motorsykler
agent cache on December 30 01:33 through 01:52.

9898 of the currently 10115 articles are dated from 01:33 to 01:52,
with the ones around 01:33 being from December 29 and thereabouts, and
the ones around 01:52 being from May 2000.

> Could you try running it again and look at .overview files
> before/after running it, to see whether it works at all for you?

I checked some groups.

			Before			After

no.alt.motorsykler	Articles: 10115		Articles: 2570
			.overview: 72118 bytes	.overview: 72948 bytes

gnu.emacs.gnus		Articles: 594		Articles: 578
			.overview: 173498 bytes .overview: 81711 bytes

ding.gnus		Articles: 751		Articles: 578
			.overview: 241243 bytes	.overview: 190562

The behaviour seems normal, except for that the .overview file of
no.alt.motorsykler, which has actually increased in size.

I'm guessing that the reason for this is that it has had added the
articles from Dec 30, which hasn't yet expired (remember I just
removed the NOV file at that point).

The reason the NOV file didn't decrease in size on my attempted
expiration, was probably that there was no files to expire at that
point (ie. the oldest articles were the ones from 01:33 on Dec 30).

So I guess the big mystery is what made agent download those 9898
articles.  Maybe the 22MB NOV file held some clues?  Or maybe it was
just to large for my old laptop?  Unfortunately I just deleted it in
irritation instead of gzip'ing it...:-/

Oh well!  Since it isn't reproducible, we'll just have to call it a
fluke. 

BTW what's supposed to happen if you have some old articles ticked in
a group, and then agentize it, and then download articles?

Is agent supposed to download all the intervening articles between the
ticked ones and the current ones?

This is what seemed to happen when I agentized the anonymous
info-cyrus group on cyrus.andrew.cmu.edu.

Or should agent only download the ticked articles?  That's what I
expected (ie. hoped for).

Re: large .overview files in agentized nntp groups

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> BTW what's supposed to happen if you have some old articles ticked in
> a group, and then agentize it, and then download articles?
>
> Is agent supposed to download all the intervening articles between the
> ticked ones and the current ones?

No, it's only supposed to download the ticked and unread articles.

> This is what seemed to happen when I agentized the anonymous
> info-cyrus group on cyrus.andrew.cmu.edu.

Hm...  Is this reproducible?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: large .overview files in agentized nntp groups

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> No, it's only supposed to download the ticked and unread articles.

What about the NOV file?  Will it contain entries for the missing
articles?

>> This is what seemed to happen when I agentized the anonymous
>> info-cyrus group on cyrus.andrew.cmu.edu.

> Hm...  Is this reproducible?

I don't know.  I'd say not to spend too much time on it.  This case,
as the no.alt.motorsykler problem, may just by my little old laptop
croaking on too large NOV files.

In any case, for the record, this is what I did:

 - added
	(nnimap "cyrus.andrew.cmu.edu"
		(nnimap-authenticator anonymous)
		(nnimap-list-pattern "archive.*")
		(nnimap-stream network))
   to gnus-secondary-select-methods

 - started Gnus with `M-x gnus-unplugged RET'

 - entered the server buffer, and pressed SPC on cyrus.andrew.cmu.edu

 - did a `c' on nnimap+cyrus.andrew.cmu.edu:archive.info-cyrus to
   catchup all the 10000-odd unread articles

 - did a `C-u 5000 SPC' on the group, and ticked the some of the
   articles with debian in the Subject fields

 - the next day, when there were 16 new unread articles in the group,
   I went into the server buffer, and put the server under agent
   control 

 - I then started downloading of articles for all groups, using 
   `JjgJsJj'

 - The machine was still working when I went to bed, and the next
   morning, I saw that it had failed somewhere, with the same kind of
   input/output error prompt I got from the 22MB NOV file of
   no.alt.motorsykler.  The prompt message didn't show up in the
   *Message* buffer, so I didn't save it

 - The agent directory for the group had heaps of articles (a lot more
   than 16 or 20 or 24 or whatever), and when I did less on them, they
   were unreleated to what I was.  When I looked at the NOV file, it
   looked bogus.  All articles had an XRef to an nnimap group on a
   different server.  So I deleted the NOV file, the .agentview file,
   and all downloaded articles

So unfortunately I've once more succeeded in erasing all tracks of
what happens. :-)

Re: large .overview files in agentized nntp groups

[Bjørn Mork]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Steinar Bang <sb@dod.no> writes:
>
>> BTW what's supposed to happen if you have some old articles ticked in
>> a group, and then agentize it, and then download articles?
>>
>> Is agent supposed to download all the intervening articles between the
>> ticked ones and the current ones?
>
> No, it's only supposed to download the ticked and unread articles.

It's a long time since this worked for me. Agent now downloads all
articles newer than the latest article in its database, independent of
the marks. This means that if I ever run gnus-agent-expire and this
causes a low traffic group to be completely purged, then agent will
download the entire group next time I do a fetch even if all articles
are marked read. Quite annoying :-(

I read news and mail online most of the time, but (used to) use agent for
short offline periods like the bus ride to work. The current behaviour
makes this usage pattern very inefficient since most articles are
downloaded two times...

I tried to send a bug report a few months ago, but don't think anybody
noticed. I'm still not completely sure that this problem isn't caused
by a configuration error on my end since noone else have reported it.
In that case I would appreciate a hint or two. AFAIK I haven't changed 
anything related to agent since things worked, just done a cvs update
now and then.


Bjørn
-- 
How can you say that Nixon is a selfish beast?  

Re: large .overview files in agentized nntp groups

[Steinar Bang]

>>>>> "Bjørn Mork" <bmork@dod.no>:

> It's a long time since this worked for me. Agent now downloads all
> articles newer than the latest article in its database, independent
> of the marks. This means that if I ever run gnus-agent-expire and
> this causes a low traffic group to be completely purged, then agent
> will download the entire group next time I do a fetch even if all
> articles are marked read. Quite annoying :-(

[snip!]
> I tried to send a bug report a few months ago, but don't think
> anybody noticed.

If this is what I'm seeing too, then things are starting to make
sense: the agent NOV file of no.alt.motorsykler weren't reduced in
size, because no articles were expired.  The reason no, or very few
articles, were expired, was that the agent directory was filled up
with new copies fetched just after the previous time I had run
gnus-agent-expire, which were less than a week old.

The info-cyrus behaviour I observed also seems to indicate the same
thing.

> I'm still not completely sure that this problem isn't caused by a
> configuration error on my end since noone else have reported it.

At least we seem to be two.  If the maintainers can't reproduce the
behaviour, maybe it'll be time to look at both our configs to see if
we have an unusual config setting in common.

Re: large .overview files in agentized nntp groups

[Lars Magne Ingebrigtsen]

"Bjørn Mork" <bmork@dod.no> writes:

> It's a long time since this worked for me. Agent now downloads all
> articles newer than the latest article in its database, independent of
> the marks. This means that if I ever run gnus-agent-expire and this
> causes a low traffic group to be completely purged, then agent will
> download the entire group next time I do a fetch even if all articles
> are marked read. Quite annoying :-(

Hm...

If you eval `(gnus-agent-fetch-headers "some.group")', what list of
articles does it return?  All articles newer than the last article in
its database, or just the unread articles in the group?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: large .overview files in agentized nntp groups

[Bjørn Mork]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> "Bjørn Mork" <bmork@dod.no> writes:
>
>> It's a long time since this worked for me. Agent now downloads all
>> articles newer than the latest article in its database, independent of
>> the marks. This means that if I ever run gnus-agent-expire and this
>> causes a low traffic group to be completely purged, then agent will
>> download the entire group next time I do a fetch even if all articles
>> are marked read. Quite annoying :-(
>
> Hm...
>
> If you eval `(gnus-agent-fetch-headers "some.group")', what list of
> articles does it return?  All articles newer than the last article in
> its database, or just the unread articles in the group?

None of them. I get wrong-type-argument..

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  gnus-agent-enter-history("last-header-fetched-for-session" (("comp.os.os2.networking.tcp-ip" . 34792)) 730869)
  gnus-agent-fetch-headers("comp.os.os2.networking.tcp-ip")
  eval((gnus-agent-fetch-headers "comp.os.os2.networking.tcp-ip"))
  eval-expression((gnus-agent-fetch-headers "comp.os.os2.networking.tcp-ip") nil)
  call-interactively(eval-expression)


Using CVS Oort checked out just a few minutes ago.



Bjørn
-- 
How can you say that all mentally retarded dudes should get Honeywell dps8s ?  

Re: large .overview files in agentized nntp groups

[Lars Magne Ingebrigtsen]

Bjørn Mork <bmork@dod.no> writes:

> None of them. I get wrong-type-argument..

Ah.  That should be:

(gnus-agent-with-fetch
  (gnus-agent-fetch-headers "comp.lang.lisp"))

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: large .overview files in agentized nntp groups

[Bjørn Mork]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Bjørn Mork <bmork@dod.no> writes:
>
>> None of them. I get wrong-type-argument..
>
> Ah.  That should be:
>
> (gnus-agent-with-fetch
>   (gnus-agent-fetch-headers "comp.lang.lisp"))

Hmm, running it on no.alt.config returns a list ranging from 3861 to
5363. But I fetched new articles just a couple of hours ago, so the
last article in the agent database is 5359:

 bmork@rasputin:~$ ls -l News/agent/nntp/news.online.no/no/alt/config/|tail -1
 -rw-r--r--    1 bmork    bmork        2620 Jan 19 19:49 5359

My news server says
 group no.alt.config
 211 439 74 5363 no.alt.config

So it's not a list of all available articles either.


Bjørn
-- 
Don't you realise that most mentally retarded people have dogs ?  

Re: large .overview files in agentized nntp groups

[Lars Magne Ingebrigtsen]

Bjørn Mork <bmork@dod.no> writes:

> Hmm, running it on no.alt.config returns a list ranging from 3861 to
> 5363. But I fetched new articles just a couple of hours ago, so the
> last article in the agent database is 5359:
>
>  bmork@rasputin:~$ ls -l News/agent/nntp/news.online.no/no/alt/config/|tail -1
>  -rw-r--r--    1 bmork    bmork        2620 Jan 19 19:49 5359
>
> My news server says
>  group no.alt.config
>  211 439 74 5363 no.alt.config

The list returned by that function should be the intersection between
`(gnus-list-of-unread-articles "that.group")' and the un-downloaded
articles.  Try edebugging through `gnus-agent-fetch-headers' in
gnus-agent.el and see what goes wrong...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: large .overview files in agentized nntp groups

[Bjørn Mork]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> The list returned by that function should be the intersection between
> `(gnus-list-of-unread-articles "that.group")' and the un-downloaded
> articles.  Try edebugging through `gnus-agent-fetch-headers' in
> gnus-agent.el and see what goes wrong...

See, there you made me learn something new (and probably useful), even
though my plans were a do-nothing-day.

Looks like this piece of code in gnus-agent-fetch-headers adds all
articles with the 'seen' mark to the fetch list:

    ;; Add article with marks to list of article headers we want to fetch.
    (dolist (arts (gnus-info-marks (gnus-get-info group)))
      (setq articles (gnus-range-add articles (cdr arts))))
    (setq articles (sort (gnus-uncompress-sequence articles) '<))



Bjørn
-- 
Let me tell you something, you beast, JFK is white.  

Re: large .overview files in agentized nntp groups

[Simon Josefsson]

Bjørn Mork <bmork@dod.no> writes:

> Looks like this piece of code in gnus-agent-fetch-headers adds 
> all articles with the 'seen' mark to the fetch list:
>
>     ;; Add article with marks to list of article headers we want 
>     to fetch.  (dolist (arts (gnus-info-marks (gnus-get-info 
>     group))) (setq articles (gnus-range-add articles (cdr 
>     arts)))) (setq articles (sort (gnus-uncompress-sequence 
>     articles) '<))

Ouch, yes.  That code was added, because that if the articles are 
not fetched into the agent, when entering the group Gnus will 
notice that certain articles doesn't exists and remove the mark. 
So to kludge around that, the agent fetches all articles with 
marks (which could be useful anyway).  But it surely isn't useful 
with `seen' and `recent'.

I'm not sure what a good solution would be though.

Re: large .overview files in agentized nntp groups

[Lars Magne Ingebrigtsen]

Simon Josefsson <jas@extundo.com> writes:

> But it surely isn't useful with `seen' and `recent'.
>
> I'm not sure what a good solution would be though.

We could just have the code ignore the `seen' and `recent' lists.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: large .overview files in agentized nntp groups

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> But it surely isn't useful with `seen' and `recent'.
>>
>> I'm not sure what a good solution would be though.
>
> We could just have the code ignore the `seen' and `recent' 
  lists.  :-)

It didn't feel like a "good" solution at the time, but I guess it 
works.  If someone feels adventurous, it might be a good idea to 
remove that `dolist' alltogheter to see if this actually still is 
a problem.