What is happening in agentized groups?

What is happening in agentized groups?

[Harry Putnam]

I have a default predicate of true and no other categories.
Its been that way for a month or longer.  

Opening gnu.emacs.gnus with C-u prompts for how many, and suggests a
total of 3256.  When the group actually opens I see 2200 of those are
actually those things marked:

   1@  31-Dec [   0: Gnus Agent  ] [Undownloaded article 69363]

Where the number is different for each.  What is the point of this?
Can there really be over 2000 messages on gnu.emacs.help that a
predicate of true has not downloaded?

Far as I know, one doesn't have to explicitly put groups in the
default category, and if the server is mentioned in
News/agent/lib/servers, then it should be downloading them all.
Even if the server is not mentioned and it is the nntp server
mentioned in .gnus, it should have any group in default predicate and
be downloading all messages..

I see thousands of these Undownloaded article summary entries in
several groups.

Re: What is happening in agentized groups?

[ShengHuo ZHU]

Harry Putnam <reader@newsguy.com> writes:

> I have a default predicate of true and no other categories.
> Its been that way for a month or longer.  
>
> Opening gnu.emacs.gnus with C-u prompts for how many, and suggests a
> total of 3256.  When the group actually opens I see 2200 of those are
> actually those things marked:
>
>    1@  31-Dec [   0: Gnus Agent  ] [Undownloaded article 69363]
>
> Where the number is different for each.  What is the point of this?

It is because the NOV of the article doesn't appear in the .overview
file.

> Can there really be over 2000 messages on gnu.emacs.help that a
> predicate of true has not downloaded?

It is not related to predicates.

> Far as I know, one doesn't have to explicitly put groups in the
> default category, and if the server is mentioned in
> News/agent/lib/servers, then it should be downloading them all.
> Even if the server is not mentioned and it is the nntp server
> mentioned in .gnus, it should have any group in default predicate and
> be downloading all messages..
>
> I see thousands of these Undownloaded article summary entries in
> several groups.

ShengHuo

Re: What is happening in agentized groups?

[ShengHuo ZHU]

Harry Putnam <reader@newsguy.com> writes:

> I have a default predicate of true and no other categories.
> Its been that way for a month or longer.  
>
> Opening gnu.emacs.gnus with C-u prompts for how many, and suggests a
> total of 3256.  When the group actually opens I see 2200 of those are
> actually those things marked:
>
>    1@  31-Dec [   0: Gnus Agent  ] [Undownloaded article 69363]
>
> Where the number is different for each.  What is the point of this?

It is because the NOV of the article doesn't appear in the .overview
file.

> Can there really be over 2000 messages on gnu.emacs.help that a
> predicate of true has not downloaded?

It is not related to predicates.

> Far as I know, one doesn't have to explicitly put groups in the
> default category, and if the server is mentioned in
> News/agent/lib/servers, then it should be downloading them all.
> Even if the server is not mentioned and it is the nntp server
> mentioned in .gnus, it should have any group in default predicate and
> be downloading all messages..
>
> I see thousands of these Undownloaded article summary entries in
> several groups.

ShengHuo

Re: What is happening in agentized groups?

[Harry Putnam]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

[...]

>>
>>    1@  31-Dec [   0: Gnus Agent  ] [Undownloaded article 69363]
>>
>> Where the number is different for each.  What is the point of this?
>
> It is because the NOV of the article doesn't appear in the .overview
> file.

Why is it expecting nov information from undownloaded messages?.
If they are undownloaded why is nov a factor.  Why are they
undownloaded?

Is there some point to having thousands of these showing up in my
summary buffer?  Can I turn something off to get rid of them?

>> Can there really be over 2000 messages on gnu.emacs.help that a
>> predicate of true has not downloaded?
>
> It is not related to predicates.

It is, in this way:  A predicate of true should leave *NO*
undownloaded messages, not to mention 2200 of them in a short while.

-- 
People say I'm stupid, but I don't get what they mean.

Re: What is happening in agentized groups?

[Harry Putnam]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

[...]

>>
>>    1@  31-Dec [   0: Gnus Agent  ] [Undownloaded article 69363]
>>
>> Where the number is different for each.  What is the point of this?
>
> It is because the NOV of the article doesn't appear in the .overview
> file.

Why is it expecting nov information from undownloaded messages?.
If they are undownloaded why is nov a factor.  Why are they
undownloaded?

Is there some point to having thousands of these showing up in my
summary buffer?  Can I turn something off to get rid of them?

>> Can there really be over 2000 messages on gnu.emacs.help that a
>> predicate of true has not downloaded?
>
> It is not related to predicates.

It is, in this way:  A predicate of true should leave *NO*
undownloaded messages, not to mention 2200 of them in a short while.

-- 
People say I'm stupid, but I don't get what they mean.

Re: What is happening in agentized groups?

[Simon Josefsson]

Harry Putnam <reader@newsguy.com> writes:

>>> Can there really be over 2000 messages on gnu.emacs.help that a
>>> predicate of true has not downloaded?
>>
>> It is not related to predicates.
>
> It is, in this way:  A predicate of true should leave *NO*
> undownloaded messages, not to mention 2200 of them in a short while.

The Agent has a hard coded behaviour not to download read articles.
There was a patch that added a new predicate `read' and changed the
default predicate from short into (and (not read) short) but I
couldn't find a way to make it backwards compatible so I hesitated
about commiting it.  But I'm not sure if readedness is causing your
problem though.

Re: What is happening in agentized groups?

[Simon Josefsson]

Harry Putnam <reader@newsguy.com> writes:

>>> Can there really be over 2000 messages on gnu.emacs.help that a
>>> predicate of true has not downloaded?
>>
>> It is not related to predicates.
>
> It is, in this way:  A predicate of true should leave *NO*
> undownloaded messages, not to mention 2200 of them in a short while.

The Agent has a hard coded behaviour not to download read articles.
There was a patch that added a new predicate `read' and changed the
default predicate from short into (and (not read) short) but I
couldn't find a way to make it backwards compatible so I hesitated
about commiting it.  But I'm not sure if readedness is causing your
problem though.

Re: What is happening in agentized groups?

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:


> The Agent has a hard coded behaviour not to download read articles.
> There was a patch that added a new predicate `read' and changed the
> default predicate from short into (and (not read) short) but I
> couldn't find a way to make it backwards compatible so I hesitated
> about commiting it.  But I'm not sure if readedness is causing your
> problem though.
>
How can I find out?  I suspect that for some time agent isn't keeping
what it downloads.  I haven't really checked and can't think of a way
to find out immediately either.  I guess I will record the number of
messages in all agentized groups today  and then check them in a week
or so.

My Categories buffer looks like:
 default: 0

The predicate is `true'

About that readedness thing; shouldn't the agent just download
according to predicate and score, not mindfull of readedness.
One might read a message in plugged mode but still want it
downloaded.   In fact, I would think that would be pretty much the
norm.  I'd prefer the agent not pay attention to what I do when plugged.

At the very least, a predicate of `true' should download everthing,
regardless. I would think.

Re: What is happening in agentized groups?

[Simon Josefsson]

Harry Putnam <reader@newsguy.com> writes:

>> The Agent has a hard coded behaviour not to download read articles.
>> There was a patch that added a new predicate `read' and changed the
>> default predicate from short into (and (not read) short) but I
>> couldn't find a way to make it backwards compatible so I hesitated
>> about commiting it.  But I'm not sure if readedness is causing your
>> problem though.
>>
> How can I find out?

Do you read anything before downloading it into the Agent?  Then the
read articles shouldn't be downloaded.  Maybe this has changed, but I
don't think so (but I haven't really used the agent much for a while).

> About that readedness thing; shouldn't the agent just download
> according to predicate and score, not mindfull of readedness.
> One might read a message in plugged mode but still want it
> downloaded.   In fact, I would think that would be pretty much the
> norm.  I'd prefer the agent not pay attention to what I do when plugged.
>
> At the very least, a predicate of `true' should download everthing,
> regardless. I would think.

Yes, I think I agree.  But that intuition might be counter-obvious if
you consider nntp, where you usually begin by catching up thoose
50,000+ message before starting to follow a newsgroup.  You don't want
the agent to download all the old junk then.

Re: What is happening in agentized groups?

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>>> The Agent has a hard coded behaviour not to download read articles.
>>> There was a patch that added a new predicate `read' and changed the
>>> default predicate from short into (and (not read) short) but I
>>> couldn't find a way to make it backwards compatible so I hesitated
>>> about commiting it.  But I'm not sure if readedness is causing your
>>> problem though.
>>>
>> How can I find out?
>
> Do you read anything before downloading it into the Agent?  Then the
> read articles shouldn't be downloaded.  Maybe this has changed, but I
> don't think so (but I haven't really used the agent much for a while).

Oh, yes.  I see.  Read a few plugged, and see if they get downloaded.
My normal routine is to never read when plugged, and almost never go
plugged at all.  I have the downloads run in batch mode by cron.

I don't think reading plugged would be what caused me to have
thousands of those summary marker things.  Only time I read plugged is
by accident, like if I go plugged for some other reason and forget to
go back unplugged. 

>> About that readedness thing; shouldn't the agent just download
>> according to predicate and score, not mindfull of readedness.
>> One might read a message in plugged mode but still want it
>> downloaded.   In fact, I would think that would be pretty much the
>> norm.  I'd prefer the agent not pay attention to what I do when plugged.
>>
>> At the very least, a predicate of `true' should download everthing,
>> regardless. I would think.
>
> Yes, I think I agree.  But that intuition might be counter-obvious if
> you consider nntp, where you usually begin by catching up thoose
> 50,000+ message before starting to follow a newsgroup.  You don't want
> the agent to download all the old junk then.

My usage wouldn't go like that, but it may not be representative.  If
I decide to agentize a group, its usally because I want it on disc for
searching or the like so in most cases, I'd want all the old junk too.

Are there really nntp groups where there could be 50,000 actually on
a server?  The most I've seen was just under 5,000 on comp.os.linux.misc.

At any rate, it couldn't be that hard to have some command that did a
onetime only catchup for the agent.

But back to this business of the undownloaded entries in summary
buffer:  I'm not sure I see what the value of that is.  In my case it
just gives me a huge summary buffer to process if I should happen to
enter the group with C-u.

ShengHuo says its because they are not in .overview, but I wonder why
not. Should they be?  It would seem they should with a predicate of
true and complete turnover on the nntp server.  I'm assuming these
`undownloaded' entries are refering to messages actually on the
server.  Or do they just stay around forever?  If they don't actually
refer to available messages then I really don't see what the value is
with them.

How can I diagnose where they are comming from?  I keep thinking with
a predicate of `true', I shoulld never see them.  At least not after a
month or so when there will have been a turnover on the nntp server.

Re: What is happening in agentized groups?

[ShengHuo ZHU]

Harry Putnam <reader@newsguy.com> writes:

[...]

> But back to this business of the undownloaded entries in summary
> buffer:  I'm not sure I see what the value of that is.  In my case it
> just gives me a huge summary buffer to process if I should happen to
> enter the group with C-u.

Actually, the word "Undownload" in the summary buffer has different
meaning from the one marked as @. Gnus agent grabs articles in three
steps, 1) updating the active file (automatically when you type 'g' in
the group buffer), 2) braiding .overview files (when you type `J s' or
automatically when you enter the group plugged and if gnus-agent-cache
is non-nil), 3) grabbing the articles (when you type 'J s' and if the
predicates return true).

If you enter a group unplugged without step 2, some articles are shown
as "Undownloaded article ???" in the summary buffer, because the NOV
is unknown.  So, it is not related to whether the predicates return
true or false. Before I patched it in January 2002, those articles
would be marked as read though you didn't see them.  It is bad because
you might miss some news or mails.  Also, without the patch, if
gnus-agent-cache is nil and you didn't download the .overview
manually, when you enter the group unplugged, some marks which you
marked when plugged will lose, because nnagent tells Gnus those
articles didn't exist on nnagent, but Gnus interprets it as those
articles didn't exist at all.

Now, the question is what about canceled or expired articles. Ideally,
canceled articles should not be listed, and expired articles on the
server should not either, but agent-expired articles, which are still
on the server, should. However, nnagent knows neither whether they
exist on the server, nor whether they are canceled or expired on the
server (at least not in the current mechanism).  So, for the safety
reason, let's assume those articles exist.

ShengHuo

Re: What is happening in agentized groups?

[Harry Putnam]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
> [...]
>
>> But back to this business of the undownloaded entries in summary
>> buffer:  I'm not sure I see what the value of that is.  In my case it
>> just gives me a huge summary buffer to process if I should happen to
>> enter the group with C-u.
>
> Actually, the word "Undownload" in the summary buffer has different
> meaning from the one marked as @. Gnus agent grabs articles in three
> steps, 1) updating the active file (automatically when you type 'g' in
> the group buffer), 2) braiding .overview files (when you type `J s' or
> automatically when you enter the group plugged and if gnus-agent-cache
> is non-nil), 3) grabbing the articles (when you type 'J s' and if the
> predicates return true).

This is getting confusing...  
   These are marked Undownloaded and have the `@' on them.
   For example, my current gnu.emacs.gnus, if openned with C-u
   while unplugged begins with:

1@  31-Dec [   0: Gnus Agent          ] [Undownloaded article 48094]

   [...] snipped 1200+ just like it with consequtive numbers up to:

1@  31-Dec [   0: Gnus Agent          ] [Undownloaded article 49318]

   Then I have the messages beginning:
20O  30-Jan [  25: A. L. Meyers  ] back to the ... 49319
   up to:
 1R  25-Feb         [  40: Shankar Rao ... 50077

None of the first group are on the server.  The lowest number on
the server is 48323

I don't really follow the formula you listed, and haven't for probably
2 yrs or more.  I do the fetching in batch mode with cron.  Which kind
of follows the formula I guess.  It has worked fine until recently.
Possibly since the patch you mention, not sure.

> If you enter a group unplugged without step 2, some articles are shown
> as "Undownloaded article ???" in the summary buffer, because the NOV

What situation would allow a user to enter without step 2?

Apparently I am missing step 2 in gnus view of things.  But this was
not happening until those `Undownloaded' entries started showing up.

Maybe this new mechanism doesn't work well with my system of batch
downloading, and running unplugged all the time.  Allowing a function
snarfed from Lars long ago to to handle passing the info from the
batch downloads to .newsrc.eld


In the unplugged gnus, I call this:

 (defun pgnus-unplugged ()
   (interactive)
   (setq gnus-plugged nil)
   (gnus))
(setq gnus-read-active-file t)

To update my groups and mail with the latest stuff gathered in the
batch fetchs run by cron.

The batch emacs that does that part, loads this function first:

 (defun lars-fetch-news ()
   (interactive)
   ;(push "/usr/local/pgnus-0.96/lisp" load-path)
   (let ((init-file-user "")
 	(gnus-always-read-dribble-file t))
     (gnus))
   (gnus-group-send-queue)
   (gnus-agent-fetch-session)
   (gnus-group-quit))

Sending and fetching

In the case above, and in my other groups.  If I were to change the 
@ to % none of them will be downloaded because they do not exist on
the server.  Yet thousands of them clutter up my summary buffer, if
entered with C-u.

[...]

> Now, the question is what about canceled or expired articles. Ideally,
> canceled articles should not be listed, and expired articles on the
> server should not either, but agent-expired articles, which are still
> on the server, should. However, nnagent knows neither whether they
> exist on the server, nor whether they are canceled or expired on the
> server (at least not in the current mechanism).  So, for the safety
> reason, let's assume those articles exist.

Isn't there a way for gnus to know if they are not on the server?
In the situation I described above, those messages will NEVER be
braided into .overview, yet I have thousands of lines of summary
buffer devoted to telling me about messages that nothing can be done
about. 

I remember a problem we had going way back before pgnus-0.91 I think.
Kind of the reverse situation, where messages in agentized groups
would disappear from gnus view when they were expunged off the server,
but really still be in the local directory.

Lars fixed that in version 91 I believe (or close) and since then gnus
knows about the agent articles even if not on the server.  Maybe
something in that mechanism will help?

Or even if it is necessary to poll the server, and find out which are
not available, seems it would be better than just continuing to show
the Undownloaded summary lines until hell freezes over.

Re: What is happening in agentized groups?

[ShengHuo ZHU]

Harry Putnam <reader@newsguy.com> writes:

> ShengHuo ZHU <zsh@cs.rochester.edu> writes:

[...]

> This is getting confusing...  
>    These are marked Undownloaded and have the `@' on them.
>    For example, my current gnu.emacs.gnus, if openned with C-u
>    while unplugged begins with:
>
> 1@  31-Dec [   0: Gnus Agent          ] [Undownloaded article 48094]
>
>    [...] snipped 1200+ just like it with consequtive numbers up to:
>
> 1@  31-Dec [   0: Gnus Agent          ] [Undownloaded article 49318]
>
>    Then I have the messages beginning:
> 20O  30-Jan [  25: A. L. Meyers  ] back to the ... 49319
>    up to:
>  1R  25-Feb         [  40: Shankar Rao ... 50077
>
> None of the first group are on the server.  The lowest number on
> the server is 48323
>
> I don't really follow the formula you listed, and haven't for probably
> 2 yrs or more.  I do the fetching in batch mode with cron.  Which kind
> of follows the formula I guess.  It has worked fine until recently.
> Possibly since the patch you mention, not sure.

If 48094 does not come from the server, check whether this number is
from ~/News/cache/active or ~/News/agent/nntp/SERVER/agent.lib/active.


>> If you enter a group unplugged without step 2, some articles are shown
>> as "Undownloaded article ???" in the summary buffer, because the NOV
>
> What situation would allow a user to enter without step 2?

Suppose that the network is down after you type 'g'. The NOVs of some
new articles aren't downloaded. But you still want to read some old
downloaded articles in the group.  Of course, you don't want to mark
those new articles as read because you didn't read them.

> Apparently I am missing step 2 in gnus view of things.  But this was
> not happening until those `Undownloaded' entries started showing up.

They didn't show up, so you may not notice the problem.

[...]

> Lars fixed that in version 91 I believe (or close) and since then gnus
> knows about the agent articles even if not on the server.  Maybe
> something in that mechanism will help?

It is different. In the case you mentioned, nnagent has got the NOV of
those articles. But in the case we are discussing, nnagent didn't get
the NOV of the articles.

> Or even if it is necessary to poll the server, and find out which are
> not available, seems it would be better than just continuing to show
> the Undownloaded summary lines until hell freezes over.

OK. I'll make an assumption that articles with numbers smaller than
the first one Agent knows don't exist on the server.  However, this
assumption doesn't hold for all cases. For example, quimby is covered
by Gnus Agent. Some articles are expired on Agent, but they are still
on quimby.  Anyway, we don't care old articles when unplugged.

The patch is submitted.

ShengHuo

Re: What is happening in agentized groups?

[Harry Putnam]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>> ShengHuo ZHU <zsh@cs.rochester.edu> writes:
>
> [...]
>
>> This is getting confusing...  
>>    These are marked Undownloaded and have the `@' on them.
>>    For example, my current gnu.emacs.gnus, if openned with C-u
>>    while unplugged begins with:
>>
>> 1@  31-Dec [   0: Gnus Agent          ] [Undownloaded article 48094]
>>
>>    [...] snipped 1200+ just like it with consequtive numbers up to:
>>
>> 1@  31-Dec [   0: Gnus Agent          ] [Undownloaded article 49318]
>>
>>    Then I have the messages beginning:
>> 20O  30-Jan [  25: A. L. Meyers  ] back to the ... 49319
>>    up to:
>>  1R  25-Feb         [  40: Shankar Rao ... 50077
>>
>> None of the first group are on the server.  The lowest number on
>> the server is 48323
>>
>> I don't really follow the formula you listed, and haven't for probably
>> 2 yrs or more.  I do the fetching in batch mode with cron.  Which kind
>> of follows the formula I guess.  It has worked fine until recently.
>> Possibly since the patch you mention, not sure.
>
> If 48094 does not come from the server, check whether this number is
> from ~/News/cache/active or ~/News/agent/nntp/SERVER/agent.lib/active.

It comes from this entry in ~/News/agent/nntp/SERVER/agent.lib/active
gnu.emacs.gnus 50082 48094 y  (I guess)

>>> If you enter a group unplugged without step 2, some articles are shown
>>> as "Undownloaded article ???" in the summary buffer, because the NOV
>>
>> What situation would allow a user to enter without step 2?
>
> Suppose that the network is down after you type 'g'. The NOVs of some
> new articles aren't downloaded. But you still want to read some old
> downloaded articles in the group.  Of course, you don't want to mark
> those new articles as read because you didn't read them.

But it seems that if they don't show up in .overview then they should
simply be downloaded on the next fetch.  Why is a display in summary
buffer necessary?

>> Apparently I am missing step 2 in gnus view of things.  But this was
>> not happening until those `Undownloaded' entries started showing up.
>
> They didn't show up, so you may not notice the problem.
>
> [...]

[...]

> OK. I'll make an assumption that articles with numbers smaller than
> the first one Agent knows don't exist on the server.  However, this
> assumption doesn't hold for all cases. For example, quimby is covered
> by Gnus Agent. Some articles are expired on Agent, but they are still
> on quimby.  Anyway, we don't care old articles when unplugged.

Can expiration be made to remove the appropriate message numbers from
the list?

> The patch is submitted.

Thanks... But couldn't gnus just ask the server when `J s' is
happening, what the lowest number is, then remove those below that
number?

In the case of your example of quimby that would be none, but then the
list would get adjusted from expiry.

But I think, I'm not really getting why we need to see this list at
all. Maybe it could be something the user can ask to see, but isn't
shown by default.  I still  don't see the practicle reason for seeing
this list.

Re: What is happening in agentized groups?

[Harry Putnam]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>> ShengHuo ZHU <zsh@cs.rochester.edu> writes:
>
> [...]
>
>> This is getting confusing...  
>>    These are marked Undownloaded and have the `@' on them.
>>    For example, my current gnu.emacs.gnus, if openned with C-u
>>    while unplugged begins with:
>>
>> 1@  31-Dec [   0: Gnus Agent          ] [Undownloaded article 48094]
>>
>>    [...] snipped 1200+ just like it with consequtive numbers up to:
>>
>> 1@  31-Dec [   0: Gnus Agent          ] [Undownloaded article 49318]
>>
>>    Then I have the messages beginning:
>> 20O  30-Jan [  25: A. L. Meyers  ] back to the ... 49319
>>    up to:
>>  1R  25-Feb         [  40: Shankar Rao ... 50077
>>
>> None of the first group are on the server.  The lowest number on
>> the server is 48323
>>
>> I don't really follow the formula you listed, and haven't for probably
>> 2 yrs or more.  I do the fetching in batch mode with cron.  Which kind
>> of follows the formula I guess.  It has worked fine until recently.
>> Possibly since the patch you mention, not sure.
>
> If 48094 does not come from the server, check whether this number is
> from ~/News/cache/active or ~/News/agent/nntp/SERVER/agent.lib/active.

It comes from this entry in ~/News/agent/nntp/SERVER/agent.lib/active
gnu.emacs.gnus 50082 48094 y  (I guess)

>>> If you enter a group unplugged without step 2, some articles are shown
>>> as "Undownloaded article ???" in the summary buffer, because the NOV
>>
>> What situation would allow a user to enter without step 2?
>
> Suppose that the network is down after you type 'g'. The NOVs of some
> new articles aren't downloaded. But you still want to read some old
> downloaded articles in the group.  Of course, you don't want to mark
> those new articles as read because you didn't read them.

But it seems that if they don't show up in .overview then they should
simply be downloaded on the next fetch.  Why is a display in summary
buffer necessary?

>> Apparently I am missing step 2 in gnus view of things.  But this was
>> not happening until those `Undownloaded' entries started showing up.
>
> They didn't show up, so you may not notice the problem.
>
> [...]

[...]

> OK. I'll make an assumption that articles with numbers smaller than
> the first one Agent knows don't exist on the server.  However, this
> assumption doesn't hold for all cases. For example, quimby is covered
> by Gnus Agent. Some articles are expired on Agent, but they are still
> on quimby.  Anyway, we don't care old articles when unplugged.

Can expiration be made to remove the appropriate message numbers from
the list?

> The patch is submitted.

Thanks... But couldn't gnus just ask the server when `J s' is
happening, what the lowest number is, then remove those below that
number?

In the case of your example of quimby that would be none, but then the
list would get adjusted from expiry.

But I think, I'm not really getting why we need to see this list at
all. Maybe it could be something the user can ask to see, but isn't
shown by default.  I still  don't see the practicle reason for seeing
this list.

Re: What is happening in agentized groups?

[Harry Putnam]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

>
> OK. I'll make an assumption that articles with numbers smaller than
> the first one Agent knows don't exist on the server.  However, this
> assumption doesn't hold for all cases. For example, quimby is covered
> by Gnus Agent. Some articles are expired on Agent, but they are still
> on quimby.  Anyway, we don't care old articles when unplugged.
>
> The patch is submitted.
>
> ShengHuo

Just reporting for ShengHuo, that the fix has cleared out all those
thousands of `@Undownloaded' summary entrys ... hope it doesn't break
something else.

Re: What is happening in agentized groups?

[Harry Putnam]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

>
> OK. I'll make an assumption that articles with numbers smaller than
> the first one Agent knows don't exist on the server.  However, this
> assumption doesn't hold for all cases. For example, quimby is covered
> by Gnus Agent. Some articles are expired on Agent, but they are still
> on quimby.  Anyway, we don't care old articles when unplugged.
>
> The patch is submitted.
>
> ShengHuo

Just reporting for ShengHuo, that the fix has cleared out all those
thousands of `@Undownloaded' summary entrys ... hope it doesn't break
something else.

Re: What is happening in agentized groups?

[ShengHuo ZHU]

Harry Putnam <reader@newsguy.com> writes:

> ShengHuo ZHU <zsh@cs.rochester.edu> writes:

[...]

> This is getting confusing...  
>    These are marked Undownloaded and have the `@' on them.
>    For example, my current gnu.emacs.gnus, if openned with C-u
>    while unplugged begins with:
>
> 1@  31-Dec [   0: Gnus Agent          ] [Undownloaded article 48094]
>
>    [...] snipped 1200+ just like it with consequtive numbers up to:
>
> 1@  31-Dec [   0: Gnus Agent          ] [Undownloaded article 49318]
>
>    Then I have the messages beginning:
> 20O  30-Jan [  25: A. L. Meyers  ] back to the ... 49319
>    up to:
>  1R  25-Feb         [  40: Shankar Rao ... 50077
>
> None of the first group are on the server.  The lowest number on
> the server is 48323
>
> I don't really follow the formula you listed, and haven't for probably
> 2 yrs or more.  I do the fetching in batch mode with cron.  Which kind
> of follows the formula I guess.  It has worked fine until recently.
> Possibly since the patch you mention, not sure.

If 48094 does not come from the server, check whether this number is
from ~/News/cache/active or ~/News/agent/nntp/SERVER/agent.lib/active.


>> If you enter a group unplugged without step 2, some articles are shown
>> as "Undownloaded article ???" in the summary buffer, because the NOV
>
> What situation would allow a user to enter without step 2?

Suppose that the network is down after you type 'g'. The NOVs of some
new articles aren't downloaded. But you still want to read some old
downloaded articles in the group.  Of course, you don't want to mark
those new articles as read because you didn't read them.

> Apparently I am missing step 2 in gnus view of things.  But this was
> not happening until those `Undownloaded' entries started showing up.

They didn't show up, so you may not notice the problem.

[...]

> Lars fixed that in version 91 I believe (or close) and since then gnus
> knows about the agent articles even if not on the server.  Maybe
> something in that mechanism will help?

It is different. In the case you mentioned, nnagent has got the NOV of
those articles. But in the case we are discussing, nnagent didn't get
the NOV of the articles.

> Or even if it is necessary to poll the server, and find out which are
> not available, seems it would be better than just continuing to show
> the Undownloaded summary lines until hell freezes over.

OK. I'll make an assumption that articles with numbers smaller than
the first one Agent knows don't exist on the server.  However, this
assumption doesn't hold for all cases. For example, quimby is covered
by Gnus Agent. Some articles are expired on Agent, but they are still
on quimby.  Anyway, we don't care old articles when unplugged.

The patch is submitted.

ShengHuo

Re: What is happening in agentized groups?

[Harry Putnam]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
> [...]
>
>> But back to this business of the undownloaded entries in summary
>> buffer:  I'm not sure I see what the value of that is.  In my case it
>> just gives me a huge summary buffer to process if I should happen to
>> enter the group with C-u.
>
> Actually, the word "Undownload" in the summary buffer has different
> meaning from the one marked as @. Gnus agent grabs articles in three
> steps, 1) updating the active file (automatically when you type 'g' in
> the group buffer), 2) braiding .overview files (when you type `J s' or
> automatically when you enter the group plugged and if gnus-agent-cache
> is non-nil), 3) grabbing the articles (when you type 'J s' and if the
> predicates return true).

This is getting confusing...  
   These are marked Undownloaded and have the `@' on them.
   For example, my current gnu.emacs.gnus, if openned with C-u
   while unplugged begins with:

1@  31-Dec [   0: Gnus Agent          ] [Undownloaded article 48094]

   [...] snipped 1200+ just like it with consequtive numbers up to:

1@  31-Dec [   0: Gnus Agent          ] [Undownloaded article 49318]

   Then I have the messages beginning:
20O  30-Jan [  25: A. L. Meyers  ] back to the ... 49319
   up to:
 1R  25-Feb         [  40: Shankar Rao ... 50077

None of the first group are on the server.  The lowest number on
the server is 48323

I don't really follow the formula you listed, and haven't for probably
2 yrs or more.  I do the fetching in batch mode with cron.  Which kind
of follows the formula I guess.  It has worked fine until recently.
Possibly since the patch you mention, not sure.

> If you enter a group unplugged without step 2, some articles are shown
> as "Undownloaded article ???" in the summary buffer, because the NOV

What situation would allow a user to enter without step 2?

Apparently I am missing step 2 in gnus view of things.  But this was
not happening until those `Undownloaded' entries started showing up.

Maybe this new mechanism doesn't work well with my system of batch
downloading, and running unplugged all the time.  Allowing a function
snarfed from Lars long ago to to handle passing the info from the
batch downloads to .newsrc.eld


In the unplugged gnus, I call this:

 (defun pgnus-unplugged ()
   (interactive)
   (setq gnus-plugged nil)
   (gnus))
(setq gnus-read-active-file t)

To update my groups and mail with the latest stuff gathered in the
batch fetchs run by cron.

The batch emacs that does that part, loads this function first:

 (defun lars-fetch-news ()
   (interactive)
   ;(push "/usr/local/pgnus-0.96/lisp" load-path)
   (let ((init-file-user "")
 	(gnus-always-read-dribble-file t))
     (gnus))
   (gnus-group-send-queue)
   (gnus-agent-fetch-session)
   (gnus-group-quit))

Sending and fetching

In the case above, and in my other groups.  If I were to change the 
@ to % none of them will be downloaded because they do not exist on
the server.  Yet thousands of them clutter up my summary buffer, if
entered with C-u.

[...]

> Now, the question is what about canceled or expired articles. Ideally,
> canceled articles should not be listed, and expired articles on the
> server should not either, but agent-expired articles, which are still
> on the server, should. However, nnagent knows neither whether they
> exist on the server, nor whether they are canceled or expired on the
> server (at least not in the current mechanism).  So, for the safety
> reason, let's assume those articles exist.

Isn't there a way for gnus to know if they are not on the server?
In the situation I described above, those messages will NEVER be
braided into .overview, yet I have thousands of lines of summary
buffer devoted to telling me about messages that nothing can be done
about. 

I remember a problem we had going way back before pgnus-0.91 I think.
Kind of the reverse situation, where messages in agentized groups
would disappear from gnus view when they were expunged off the server,
but really still be in the local directory.

Lars fixed that in version 91 I believe (or close) and since then gnus
knows about the agent articles even if not on the server.  Maybe
something in that mechanism will help?

Or even if it is necessary to poll the server, and find out which are
not available, seems it would be better than just continuing to show
the Undownloaded summary lines until hell freezes over.

Re: What is happening in agentized groups?

[ShengHuo ZHU]

Harry Putnam <reader@newsguy.com> writes:

[...]

> But back to this business of the undownloaded entries in summary
> buffer:  I'm not sure I see what the value of that is.  In my case it
> just gives me a huge summary buffer to process if I should happen to
> enter the group with C-u.

Actually, the word "Undownload" in the summary buffer has different
meaning from the one marked as @. Gnus agent grabs articles in three
steps, 1) updating the active file (automatically when you type 'g' in
the group buffer), 2) braiding .overview files (when you type `J s' or
automatically when you enter the group plugged and if gnus-agent-cache
is non-nil), 3) grabbing the articles (when you type 'J s' and if the
predicates return true).

If you enter a group unplugged without step 2, some articles are shown
as "Undownloaded article ???" in the summary buffer, because the NOV
is unknown.  So, it is not related to whether the predicates return
true or false. Before I patched it in January 2002, those articles
would be marked as read though you didn't see them.  It is bad because
you might miss some news or mails.  Also, without the patch, if
gnus-agent-cache is nil and you didn't download the .overview
manually, when you enter the group unplugged, some marks which you
marked when plugged will lose, because nnagent tells Gnus those
articles didn't exist on nnagent, but Gnus interprets it as those
articles didn't exist at all.

Now, the question is what about canceled or expired articles. Ideally,
canceled articles should not be listed, and expired articles on the
server should not either, but agent-expired articles, which are still
on the server, should. However, nnagent knows neither whether they
exist on the server, nor whether they are canceled or expired on the
server (at least not in the current mechanism).  So, for the safety
reason, let's assume those articles exist.

ShengHuo

Re: What is happening in agentized groups?

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>>> The Agent has a hard coded behaviour not to download read articles.
>>> There was a patch that added a new predicate `read' and changed the
>>> default predicate from short into (and (not read) short) but I
>>> couldn't find a way to make it backwards compatible so I hesitated
>>> about commiting it.  But I'm not sure if readedness is causing your
>>> problem though.
>>>
>> How can I find out?
>
> Do you read anything before downloading it into the Agent?  Then the
> read articles shouldn't be downloaded.  Maybe this has changed, but I
> don't think so (but I haven't really used the agent much for a while).

Oh, yes.  I see.  Read a few plugged, and see if they get downloaded.
My normal routine is to never read when plugged, and almost never go
plugged at all.  I have the downloads run in batch mode by cron.

I don't think reading plugged would be what caused me to have
thousands of those summary marker things.  Only time I read plugged is
by accident, like if I go plugged for some other reason and forget to
go back unplugged. 

>> About that readedness thing; shouldn't the agent just download
>> according to predicate and score, not mindfull of readedness.
>> One might read a message in plugged mode but still want it
>> downloaded.   In fact, I would think that would be pretty much the
>> norm.  I'd prefer the agent not pay attention to what I do when plugged.
>>
>> At the very least, a predicate of `true' should download everthing,
>> regardless. I would think.
>
> Yes, I think I agree.  But that intuition might be counter-obvious if
> you consider nntp, where you usually begin by catching up thoose
> 50,000+ message before starting to follow a newsgroup.  You don't want
> the agent to download all the old junk then.

My usage wouldn't go like that, but it may not be representative.  If
I decide to agentize a group, its usally because I want it on disc for
searching or the like so in most cases, I'd want all the old junk too.

Are there really nntp groups where there could be 50,000 actually on
a server?  The most I've seen was just under 5,000 on comp.os.linux.misc.

At any rate, it couldn't be that hard to have some command that did a
onetime only catchup for the agent.

But back to this business of the undownloaded entries in summary
buffer:  I'm not sure I see what the value of that is.  In my case it
just gives me a huge summary buffer to process if I should happen to
enter the group with C-u.

ShengHuo says its because they are not in .overview, but I wonder why
not. Should they be?  It would seem they should with a predicate of
true and complete turnover on the nntp server.  I'm assuming these
`undownloaded' entries are refering to messages actually on the
server.  Or do they just stay around forever?  If they don't actually
refer to available messages then I really don't see what the value is
with them.

How can I diagnose where they are comming from?  I keep thinking with
a predicate of `true', I shoulld never see them.  At least not after a
month or so when there will have been a turnover on the nntp server.

Re: What is happening in agentized groups?

[Simon Josefsson]

Harry Putnam <reader@newsguy.com> writes:

>> The Agent has a hard coded behaviour not to download read articles.
>> There was a patch that added a new predicate `read' and changed the
>> default predicate from short into (and (not read) short) but I
>> couldn't find a way to make it backwards compatible so I hesitated
>> about commiting it.  But I'm not sure if readedness is causing your
>> problem though.
>>
> How can I find out?

Do you read anything before downloading it into the Agent?  Then the
read articles shouldn't be downloaded.  Maybe this has changed, but I
don't think so (but I haven't really used the agent much for a while).

> About that readedness thing; shouldn't the agent just download
> according to predicate and score, not mindfull of readedness.
> One might read a message in plugged mode but still want it
> downloaded.   In fact, I would think that would be pretty much the
> norm.  I'd prefer the agent not pay attention to what I do when plugged.
>
> At the very least, a predicate of `true' should download everthing,
> regardless. I would think.

Yes, I think I agree.  But that intuition might be counter-obvious if
you consider nntp, where you usually begin by catching up thoose
50,000+ message before starting to follow a newsgroup.  You don't want
the agent to download all the old junk then.

Re: What is happening in agentized groups?

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:


> The Agent has a hard coded behaviour not to download read articles.
> There was a patch that added a new predicate `read' and changed the
> default predicate from short into (and (not read) short) but I
> couldn't find a way to make it backwards compatible so I hesitated
> about commiting it.  But I'm not sure if readedness is causing your
> problem though.
>
How can I find out?  I suspect that for some time agent isn't keeping
what it downloads.  I haven't really checked and can't think of a way
to find out immediately either.  I guess I will record the number of
messages in all agentized groups today  and then check them in a week
or so.

My Categories buffer looks like:
 default: 0

The predicate is `true'

About that readedness thing; shouldn't the agent just download
according to predicate and score, not mindfull of readedness.
One might read a message in plugged mode but still want it
downloaded.   In fact, I would think that would be pretty much the
norm.  I'd prefer the agent not pay attention to what I do when plugged.

At the very least, a predicate of `true' should download everthing,
regardless. I would think.

Re: What is happening in agentized groups?

[Christoph Rohland]

Hi Simon,

On Mon, 25 Feb 2002, Simon Josefsson wrote:
> The Agent has a hard coded behaviour not to download read articles.
> There was a patch that added a new predicate `read' and changed the
> default predicate from short into (and (not read) short) but I
> couldn't find a way to make it backwards compatible so I hesitated
> about commiting it.

I loved that feature (but it corrupted my agent files :-() and I would
still like to see it included.

> Do you read anything before downloading it into the Agent?  Then the
> read articles shouldn't be downloaded.  Maybe this has changed, but
> I don't think so (but I haven't really used the agent much for a
> while).

Actually I would like to be able to have such an article put into the
agent automatically, so I would be able to read it later without
refetching. 

AFAIU this would be against the agents design and that's what the
cache is for. But from a user perspective this distinction is
unintuitive. I would prefer a unified agent/cache where the cached
articles are agentised articles without being expired.

Greetings
		Christoph

Re: What is happening in agentized groups?

[Christoph Rohland]

Hi Simon,

On Mon, 25 Feb 2002, Simon Josefsson wrote:
> The Agent has a hard coded behaviour not to download read articles.
> There was a patch that added a new predicate `read' and changed the
> default predicate from short into (and (not read) short) but I
> couldn't find a way to make it backwards compatible so I hesitated
> about commiting it.

I loved that feature (but it corrupted my agent files :-() and I would
still like to see it included.

> Do you read anything before downloading it into the Agent?  Then the
> read articles shouldn't be downloaded.  Maybe this has changed, but
> I don't think so (but I haven't really used the agent much for a
> while).

Actually I would like to be able to have such an article put into the
agent automatically, so I would be able to read it later without
refetching. 

AFAIU this would be against the agents design and that's what the
cache is for. But from a user perspective this distinction is
unintuitive. I would prefer a unified agent/cache where the cached
articles are agentised articles without being expired.

Greetings
		Christoph

Re: What is happening in agentized groups?

[Steinar Bang]

>>>>> ShengHuo ZHU <zsh@cs.rochester.edu>:

> Harry Putnam <reader@newsguy.com> writes:
>> I have a default predicate of true and no other categories.
>> Its been that way for a month or longer.  

>> Opening gnu.emacs.gnus with C-u prompts for how many, and suggests a
>> total of 3256.  When the group actually opens I see 2200 of those are
>> actually those things marked:

>> 1@  31-Dec [   0: Gnus Agent  ] [Undownloaded article 69363]

>> Where the number is different for each.  What is the point of this?

> It is because the NOV of the article doesn't appear in the .overview
> file.

Just for the record: when I subscribed to
	nnimap+cyrus.andrew.cmu.edu:archive.cyrus-sasl 
I saw real subjects and senders, after I had visited the group while
plugged, and then agent was able to download the articles.

But previous to this, repeated `J %' followed by attempts at download
the articles, didn't make them download.

Is agent supposed to skip undownloaded articles?

Re: What is happening in agentized groups?

[Steinar Bang]

>>>>> ShengHuo ZHU <zsh@cs.rochester.edu>:

> Harry Putnam <reader@newsguy.com> writes:
>> I have a default predicate of true and no other categories.
>> Its been that way for a month or longer.  

>> Opening gnu.emacs.gnus with C-u prompts for how many, and suggests a
>> total of 3256.  When the group actually opens I see 2200 of those are
>> actually those things marked:

>> 1@  31-Dec [   0: Gnus Agent  ] [Undownloaded article 69363]

>> Where the number is different for each.  What is the point of this?

> It is because the NOV of the article doesn't appear in the .overview
> file.

Just for the record: when I subscribed to
	nnimap+cyrus.andrew.cmu.edu:archive.cyrus-sasl 
I saw real subjects and senders, after I had visited the group while
plugged, and then agent was able to download the articles.

But previous to this, repeated `J %' followed by attempts at download
the articles, didn't make them download.

Is agent supposed to skip undownloaded articles?

What is happening in agentized groups?

[Harry Putnam]

I have a default predicate of true and no other categories.
Its been that way for a month or longer.  

Opening gnu.emacs.gnus with C-u prompts for how many, and suggests a
total of 3256.  When the group actually opens I see 2200 of those are
actually those things marked:

   1@  31-Dec [   0: Gnus Agent  ] [Undownloaded article 69363]

Where the number is different for each.  What is the point of this?
Can there really be over 2000 messages on gnu.emacs.help that a
predicate of true has not downloaded?

Far as I know, one doesn't have to explicitly put groups in the
default category, and if the server is mentioned in
News/agent/lib/servers, then it should be downloading them all.
Even if the server is not mentioned and it is the nntp server
mentioned in .gnus, it should have any group in default predicate and
be downloading all messages..

I see thousands of these Undownloaded article summary entries in
several groups.