Re: Agent downloads too many headers

[kai.grossjohann@uni-duisburg.de (Kai Großjohann)]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> The situation is as follows: in gnus-agent-fetch-headers, if
> gnus-agent-consider-all-articles is non-nil, the list of articles is
> set to the active range of the group.  The active range could be
> something like (1 . 4711), so the list of articles would be (1 2 ...
> 4710 4711).  Then we remove from that list the list of
> already-downloaded articles.  But probably the user has started
> reading the group when the article numbers were greater than 1
> already.  So most probably there are a lot of articles in the
> low-number range which are not in the group at all.
>
> Then the agent fetches the headers from that group for this list of
> articles.
>
> So the agent ends up fetching (almost) all headers for all groups.
>
> What can we do?

The more I think about it, the more confused I get.

My current idea is to store ranges of articles which the agent has
already tried to fetch (it might have failed for nonexisting
articles).  So if the current group starts at article 4712, then the
range will also contain (1 . 4711) because at the first try Gnus will
try to fetch those headers, but they don't exist.

Then at subsequent fetching attempts, we can remove those headers.

But the problem is this: suppose the user uses the agent for a while
in the default setting, then changes gnus-agent-consider-all-articles
to true later on.  This means that lots of articles need to be
fetched where the headers are fetched already.  So at some point
headers from the cache need to be added to the list of articles to
check.  How do I do that?

Another thing is that we probably don't want to iterate over all
those long-fetched headers again to see if we need to fetch the
corresponding article now.  So maybe along with the range of articles
fetched we should store the value of gnus-agent-consider-all-articles
and the predicate that was used.  Then when either one changes we can
consider all those long-fetched headers again and when they stay the
same we only need to check the new headers.

It's getting quite complicated for my feeble mind, and every time I
try to implement something, I get confused.  So it would be really
nice if somebody could help.  Whatever you know, just let me know
about it.  So if you know an algorithm that might work, tell me.  If
you know how to implement a small part of what needs to be done, tell
me.

Thanks,
kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)