Agent gone mad?

Agent gone mad?

[Henrik Enberg]

After updating my Gnus the agent wants to download many megs of long
since read ande marked as expirable articles and headers.  Is this an
intended change?  How do I prevent it?

-- 
Booting... /vmemacs.el

Re: Agent gone mad?

[Henrik Enberg]

Henrik Enberg <henrik@enberg.org> writes:

I frrgot to mention, It's really slow too.  It fetches about 50kb per
minute.

-- 
Booting... /vmemacs.el

Re: Agent gone mad?

[Kai Großjohann]

Henrik Enberg <henrik@enberg.org> writes:

> After updating my Gnus the agent wants to download many megs of long
> since read ande marked as expirable articles and headers.  Is this an
> intended change?  How do I prevent it?

Whee.  It's supposed to fetch old mail if
gnus-agent-consider-all-articles is true, but the behavior if that
variable is false should not have changed.  (The default is false.)

What's the value you have?

Hm.  I also made another change, and I notice the same problem.  I
changed it because before, it always downloaded no articles for me,
and that was not right either.

Would you be willing to help out to solve the problem?  I'm having
trouble to understand the code.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Agent gone mad?

[Henrik Enberg]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Henrik Enberg <henrik@enberg.org> writes:
>
>> After updating my Gnus the agent wants to download many megs of long
>> since read ande marked as expirable articles and headers.  Is this an
>> intended change?  How do I prevent it?
>
> Whee.  It's supposed to fetch old mail if
> gnus-agent-consider-all-articles is true, but the behavior if that
> variable is false should not have changed.  (The default is false.)
>
> What's the value you have?

with nil it downloads all headers and messages and then throws away the
messages.  With t it saves the messages.

It does this even if all the messages are downloaded.

> Hm.  I also made another change, and I notice the same problem.  I
> changed it because before, it always downloaded no articles for me,
> and that was not right either.
>
> Would you be willing to help out to solve the problem?  I'm having
> trouble to understand the code.

I don't really understand it either, and I'm rather busy for the next
week or so.  But I'll do my do my best.

-- 
Booting... /vmemacs.el

Re: Agent gone mad?

[Kai Großjohann]

Henrik Enberg <henrik@enberg.org> writes:

> After updating my Gnus the agent wants to download many megs of long
> since read ande marked as expirable articles and headers.  Is this an
> intended change?  How do I prevent it?

I have observed the problem that Gnus tries to fetch nonexisting
articles.  I observed it when gnus-agent-consider-all-articles was
true, I don't know if it also happens when it is false.

Anyhow, I've now changed gnus-agent.el so that it doesn't try to
fetch nonexisting articles anymore.  This speeds up my session fetch
a lot.

But I do still observe that Gnus tries to fetch a lot of headers from
the groups.  I'm not sure where that comes from.  Interestingly, it
seems that Gnus does not fetch the headers from nnimap groups, only
from nntp groups.  Strange.

Simon,

does this ring a bell with you?  Does the agent header caching stuff
do more work for nnimap than for nntp groups?  Or is there additional
header caching going on for nnimap?

Henrik,

is Gnus behaving better now with the change I just made?  How bad is
it still?

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Agent gone mad?

[Henrik Enberg]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> I have observed the problem that Gnus tries to fetch nonexisting
> articles.  I observed it when gnus-agent-consider-all-articles was
> true, I don't know if it also happens when it is false.

It does, see my other message.

> Anyhow, I've now changed gnus-agent.el so that it doesn't try to
> fetch nonexisting articles anymore.  This speeds up my session fetch
> a lot.

Yup, it sees to be as fast as it used to be.

> is Gnus behaving better now with the change I just made?  How bad is
> it still?

Unusably bad.  Everytime I do 'J s', It'll download about 100 megs.

-- 
Booting... /vmemacs.el

Re: Agent gone mad?

[Kai Großjohann]

Henrik Enberg <henrik@enberg.org> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>
>> Anyhow, I've now changed gnus-agent.el so that it doesn't try to
>> fetch nonexisting articles anymore.  This speeds up my session fetch
>> a lot.
>
> Yup, it sees to be as fast as it used to be.
>
>> is Gnus behaving better now with the change I just made?  How bad is
>> it still?
>
> Unusably bad.  Everytime I do 'J s', It'll download about 100 megs.

Hm.  So it is as fast as before, which is unusably bad :-)

Okay.  What's in the 100 megs that it is still downloading?  Is it
headers or articles or both?  I might need to meditate a lot more
about that code...

Btw, I've now reinstated another change of mine which abstains from
fetching already-fetched messages.  Maybe that helps some more.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Agent gone mad?

[Henrik Enberg]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Henrik Enberg <henrik@enberg.org> writes:
>
>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>>
>>> Anyhow, I've now changed gnus-agent.el so that it doesn't try to
>>> fetch nonexisting articles anymore.  This speeds up my session fetch
>>> a lot.
>>
>> Yup, it sees to be as fast as it used to be.
>>
>>> is Gnus behaving better now with the change I just made?  How bad is
>>> it still?
>>
>> Unusably bad.  Everytime I do 'J s', It'll download about 100 megs.
>
> Hm.  So it is as fast as before, which is unusably bad :-)

Yes, because while it's fast, it also tries to download every article
my server knows about.

> Okay.  What's in the 100 megs that it is still downloading?  Is it
> headers or articles or both?  I might need to meditate a lot more
> about that code...

It is both headers and articles.  Even those already downloaded.  It
seems to just throw away what it downloads.

> Btw, I've now reinstated another change of mine which abstains from
> fetching already-fetched messages.  Maybe that helps some more.

There are some changes in the behaviour.  With
`gnus-agent-consider-all-articles' set to nil, it still does what I
described above.  If I set it to t, Gnus just locks up until I hit C-g.

I've been staring at the code for quite some time, but I don't
understand why it doesn't work.  It is clearly conditioned on
gnus-agent-consider-all-articles.  It really _ought_ to work.

-- 
Booting... /vmemacs.el

Re: Agent gone mad?

[Kai Großjohann]

Henrik Enberg <henrik@enberg.org> writes:

> Yes, because while it's fast, it also tries to download every article
> my server knows about.

Whee.  Hm.  This does not happen for me.  Otherwise things would be
even slower still.

>> Okay.  What's in the 100 megs that it is still downloading?  Is it
>> headers or articles or both?  I might need to meditate a lot more
>> about that code...
>
> It is both headers and articles.  Even those already downloaded.  It
> seems to just throw away what it downloads.

Really really strange.  How could that happen?  Can you trace the
code (with edebug?) to see what happens?

>> Btw, I've now reinstated another change of mine which abstains from
>> fetching already-fetched messages.  Maybe that helps some more.
>
> There are some changes in the behaviour.  With
> `gnus-agent-consider-all-articles' set to nil, it still does what I
> described above.  If I set it to t, Gnus just locks up until I hit C-g.
>
> I've been staring at the code for quite some time, but I don't
> understand why it doesn't work.  It is clearly conditioned on
> gnus-agent-consider-all-articles.  It really _ought_ to work.

Well, the code that's conditioned on gnus-agent-consider-all-articles
only influences the header fetching.  Hm.

There is one spot which I changed:

    ;; Remove known articles.
    (when (gnus-agent-load-alist group)
      ;; Remove articles marked as downloaded.
      (setq articles
	    (gnus-sorted-difference
	     articles
	     (delq nil
		   (mapcar (lambda (x) (when (cdr x) (car x)))
			   gnus-agent-article-alist))))
      (let ((low (1+ (caar (last gnus-agent-article-alist))))
	    (high (cdr (gnus-active group))))
	;; I suspect a deeper problem here and I suspect that low
	;; should never be greater than high.  But for the time being
	;; we just work around the problem and abstain from frobbing
	;; the article list in that case.  If anyone knows how to
	;; properly deal with it, please holler.  -- kai
	(when (<= low high)
	  (setq articles (gnus-list-range-intersection
			  articles (list (cons low high)))))))

The last `when' was not there originally.  For me, low was greater
than high and therefore gnus-list-range-intersection returned nil.

But maybe for you low is also greater than high but
gnus-list-range-intersection does not return nil?

How about you try to remove that (<= low high) condition and see what
happens?

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Agent gone mad?

[Henrik Enberg]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Henrik Enberg <henrik@enberg.org> writes:

[...]

>> It is both headers and articles.  Even those already downloaded.  It
>> seems to just throw away what it downloads.
>
> Really really strange.  How could that happen?  Can you trace the
> code (with edebug?) to see what happens?

Yes,  I'll do that tomorrow night.  I have an assigment that needs to be
in tomorrow morning :(

> There is one spot which I changed:
>
> 	(when (<= low high)
> 	  (setq articles (gnus-list-range-intersection
> 			  articles (list (cons low high)))))))
>
> The last `when' was not there originally.  For me, low was greater
> than high and therefore gnus-list-range-intersection returned nil.
>
> But maybe for you low is also greater than high but
> gnus-list-range-intersection does not return nil?
>
> How about you try to remove that (<= low high) condition and see what
> happens?

Whee, it works when I remove that condition.  Both with
`gnus-agent-consider-all-articles' set to nil and t.  `low' is indeed
higher than `high' for my groups. 

-- 
Booting... /vmemacs.el

Re: Agent gone mad?

[Kai Großjohann]

Henrik Enberg <henrik@enberg.org> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>
>> There is one spot which I changed:
>>
>> 	(when (<= low high)
>> 	  (setq articles (gnus-list-range-intersection
>> 			  articles (list (cons low high)))))))
>>
>> The last `when' was not there originally.  For me, low was greater
>> than high and therefore gnus-list-range-intersection returned nil.
>>
>> But maybe for you low is also greater than high but
>> gnus-list-range-intersection does not return nil?
>>
>> How about you try to remove that (<= low high) condition and see what
>> happens?
>
> Whee, it works when I remove that condition.  Both with
> `gnus-agent-consider-all-articles' set to nil and t.  `low' is indeed
> higher than `high' for my groups. 

Fascinating.  What are those values, more precisely?  I found that
(gnus-list-range-intersection '(1 2 3 4 5) '((5 . 4))) evals to nil,
for instance.  So when low > high, no articles were downloaded in my
case.  What happens when you eval that expression?  If you get
non-nil, then something is fishy in Gnus.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Agent gone mad?

[Henrik Enberg]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Henrik Enberg <henrik@enberg.org> writes:
>
>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>>
>>> 	(when (<= low high)
>>> 	  (setq articles (gnus-list-range-intersection
>>> 			  articles (list (cons low high)))))))
>>>
>> Whee, it works when I remove that condition.  Both with
>> `gnus-agent-consider-all-articles' set to nil and t.  `low' is indeed
>> higher than `high' for my groups. 
>
> Fascinating.  What are those values, more precisely?  

Running the below function to mimic the behaviour in the above code, I
always get "high" as 1 less than "low", so the when clause will never
be executed.

(defun high/low ()
  (interactive)
  (let ((low (1+ (caar (last gnus-agent-article-alist))))
	(high (cdr (gnus-active gnus-newsgroup-name))))
    (message "high = %s, low = %s" high low)))

> I found that (gnus-list-range-intersection '(1 2 3 4 5) '((5 . 4)))
> evals to nil, for instance.  So when low > high, no articles were
> downloaded in my case.  What happens when you eval that expression?
> If you get non-nil, then something is fishy in Gnus.

I also get nil when I eval that.

-- 
Booting... /vmemacs.el

Re: Agent gone mad?

[Kai Großjohann]

Henrik Enberg <henrik@enberg.org> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>
>> Henrik Enberg <henrik@enberg.org> writes:
>>
>>> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>>>
>>>> 	(when (<= low high)
>>>> 	  (setq articles (gnus-list-range-intersection
>>>> 			  articles (list (cons low high)))))))
>>>>
>>> Whee, it works when I remove that condition.  Both with
>>> `gnus-agent-consider-all-articles' set to nil and t.  `low' is indeed
>>> higher than `high' for my groups. 
>>
>> Fascinating.  What are those values, more precisely?  
>
> Running the below function to mimic the behaviour in the above code, I
> always get "high" as 1 less than "low", so the when clause will never
> be executed.
>
> (defun high/low ()
>   (interactive)
>   (let ((low (1+ (caar (last gnus-agent-article-alist))))
> 	(high (cdr (gnus-active gnus-newsgroup-name))))
>     (message "high = %s, low = %s" high low)))

Gah?

>> I found that (gnus-list-range-intersection '(1 2 3 4 5) '((5 . 4)))
>> evals to nil, for instance.  So when low > high, no articles were
>> downloaded in my case.  What happens when you eval that expression?
>> If you get non-nil, then something is fishy in Gnus.
>
> I also get nil when I eval that.

Okay, so to summarize: when you remove the when condition, Gnus is
quick again.

But that's because it retrieves no articles, because
gnus-list-range-intersection returns nil.

So I'm not sure it's a good idea to remove the condition.


But I don't believe you were happy with the previous agent behavior
if it never retrieved any articles.  So how about you go back to the
gnus-agent version before my messing around with it and see if that
retrieves any articles at all, and also examine the situation in the
code we're talking about.

Maybe I did something wrong when changing the code.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Agent gone mad?

[Henrik Enberg]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Henrik Enberg <henrik@enberg.org> writes:
>
>> Running the below function to mimic the behaviour in the above code, I
>> always get "high" as 1 less than "low", so the when clause will never
>> be executed.
>>
>> (defun high/low ()
>>   (interactive)
>>   (let ((low (1+ (caar (last gnus-agent-article-alist))))
>> 	(high (cdr (gnus-active gnus-newsgroup-name))))
>>     (message "high = %s, low = %s" high low)))
>
> Gah?

Hmm, what do you get?

[...]

> Okay, so to summarize: when you remove the when condition, Gnus is
> quick again.
>
> But that's because it retrieves no articles, because
> gnus-list-range-intersection returns nil.
>
> So I'm not sure it's a good idea to remove the condition.

No,  Gnus _does_ retrive articles when I remove the condition.  It even
works as I think you intended it.  Old style behaviour when
gnus-agent-consider-all-articles is nil, and all articles when it is t.

> But I don't believe you were happy with the previous agent behavior
> if it never retrieved any articles.  So how about you go back to the
> gnus-agent version before my messing around with it and see if that
> retrieves any articles at all, and also examine the situation in the
> code we're talking about.

Ok, I tried to check out the version from the 17th.  It works just like
it always has, retrieving only new articles and headers.

> Maybe I did something wrong when changing the code.

It just might me my Gnus that is screwed up, but I tried to subscribe
to a new group I've never subscribed to before.  And it gave me the
same behaviour as all my other groups.

A final note.  I may have to turn in my geek badge, but I can't work
out how to edebug a Gnus function.  Say I instrument
gnus-agent-retrieve-headers, how do I step through it?

-- 
Booting... /vmemacs.el

Re: Agent gone mad?

[Kai Großjohann]

Henrik Enberg <henrik@enberg.org> writes:

> kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
>
>> Henrik Enberg <henrik@enberg.org> writes:
>>
>>> Running the below function to mimic the behaviour in the above code, I
>>> always get "high" as 1 less than "low", so the when clause will never
>>> be executed.
>>>
>>> (defun high/low ()
>>>   (interactive)
>>>   (let ((low (1+ (caar (last gnus-agent-article-alist))))
>>> 	(high (cdr (gnus-active gnus-newsgroup-name))))
>>>     (message "high = %s, low = %s" high low)))
>>
>> Gah?
>
> Hmm, what do you get?
>
> [...]

I didn't try the function, but I also got low = high + 1 when I used
the debugger.

The "gah" meant that I don't know why the original code works for
you.  It should fail!

>> Okay, so to summarize: when you remove the when condition, Gnus is
>> quick again.
>>
>> But that's because it retrieves no articles, because
>> gnus-list-range-intersection returns nil.
>>
>> So I'm not sure it's a good idea to remove the condition.
>
> No,  Gnus _does_ retrive articles when I remove the condition.  It even
> works as I think you intended it.  Old style behaviour when
> gnus-agent-consider-all-articles is nil, and all articles when it is t.

Strange.

>> But I don't believe you were happy with the previous agent behavior
>> if it never retrieved any articles.  So how about you go back to the
>> gnus-agent version before my messing around with it and see if that
>> retrieves any articles at all, and also examine the situation in the
>> code we're talking about.
>
> Ok, I tried to check out the version from the 17th.  It works just like
> it always has, retrieving only new articles and headers.
>
>> Maybe I did something wrong when changing the code.
>
> It just might me my Gnus that is screwed up, but I tried to subscribe
> to a new group I've never subscribed to before.  And it gave me the
> same behaviour as all my other groups.

Okay, now we need to verify:

Usually, low is greater than high.  You verified this already,
check.  When this happens, gnus-list-range-intersection returns nil.
Right?  So then `articles' will be nil.  Right?

So Gnus only fetches articles which are explicitly marked for
download.  Right?

Since we have now reached a contradiction, one of the checks above
must fail.  But which one?

> A final note.  I may have to turn in my geek badge, but I can't work
> out how to edebug a Gnus function.  Say I instrument
> gnus-agent-retrieve-headers, how do I step through it?

With point inside the function, hit M-x edebug-defun RET.  Then you
should either arrange for the buffer to be well visible in a frame,
or you should bury the buffer (then edebug will display it in the
current frame).  Then you just do something that invokes the
function.  Then you can use SPC to step through it and e to eval Lisp
expressions (and the menu).

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: Agent gone mad?

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Simon,
>
> does this ring a bell with you?  Does the agent header caching stuff
> do more work for nnimap than for nntp groups?  Or is there additional
> header caching going on for nnimap?

Nnimap used to do its own NOV caching which was (usually) more
efficient than the Agent caching (specifically -- nnimap only retrieve
NOV headers in blocks, leaving no gaps in the .overview file, this
means it sometimes does too much work, but it can pay this back by
needing less round trips and probably loading the server less).  But
this is now probably disabled:

(defvoo nnimap-nov-is-evil gnus-agent

Although this should really be backend dependent, and only be disabled
on agentized backends.  I remember trying to optimize the nntp.el code
too, but the NNTP command trace doesn't display the received data so
it was a little difficult to debug.

Re: Agent gone mad?

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>
>> Simon,
>>
>> does this ring a bell with you?  Does the agent header caching stuff
>> do more work for nnimap than for nntp groups?  Or is there additional
>> header caching going on for nnimap?
>
> Nnimap used to do its own NOV caching which was (usually) more
> efficient than the Agent caching (specifically -- nnimap only retrieve
> NOV headers in blocks, leaving no gaps in the .overview file, this
> means it sometimes does too much work, but it can pay this back by
> needing less round trips and probably loading the server less).

I now think that things will be much more efficient if we fetch each
header just once and not every time.  This is done already.

But now the code still fetches headers for nonexisting articles,
which is silly.  How to solve that?  See my other posting about this.

Thanks for your info, however.  I didn't know that.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)