Agent downloads too many headers

[Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann)]

The situation is as follows: in gnus-agent-fetch-headers, if
gnus-agent-consider-all-articles is non-nil, the list of articles is
set to the active range of the group.  The active range could be
something like (1 . 4711), so the list of articles would be (1 2 ...
4710 4711).  Then we remove from that list the list of
already-downloaded articles.  But probably the user has started
reading the group when the article numbers were greater than 1
already.  So most probably there are a lot of articles in the
low-number range which are not in the group at all.

Then the agent fetches the headers from that group for this list of
articles.

So the agent ends up fetching (almost) all headers for all groups.

What can we do?

We want to avoid fetching headers for the low-numbered articles where
we already learned yesterday that these articles don't exist.

One possibility would be to tell the agent to never fetch articles
with numbers less than what we've already fetched.  This would be (fairly)
easy to implement, but it would lead to a problem: people who start
using the Agent and download the (unread) message 4711, then decide
they want to download old articles, too.  For them, the agent would
never download articles with numbers lower than 4711 because that's
the lowest number fetched already.  One workaround would be to enter
the group and type `C-u J u' which would fetch even those articles.
But I think it is not nice to require them to do that, they might
have lots of groups.

Another possibility is to keep an "unactive list (of ranges)".  This
would be a range of articles known not to exist.  This would require
storing more data in the agent.  But it would also be precise.  I see
two problems with this, a minor one and a major one.  The minor
problem is that I don't know how to store additional data in the
agent.  The major problem is that I don't know if it will be
efficient: the unactive ranges might grow quite large.  For example,
if you start reading a group starting with article 1 and the agent
fetches every tenth article, then the unactive ranges will be ((2 .
10) (12 . 20) ...) and after a few tens of thousands of articles
there will be information about those long-gone articles 1, 11, 21
that have presumably long been expired from the agent.

What do you think?

Please help!

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)