double download of attachments?

double download of attachments?

[Wes Hardaker]

Currently (at least with nnimap) attachments seem to be downloaded
twice.  Is there a way we can get around that (without a major code
rewrite)?

Specifically, send yourself something near a meg or so and then click
on the article.  Watch the downloading size counter and you'll see it
reset and start again from 0.  I assume that this is done twice once
for display and once for the agent cache...

It's a bit of a problem when I'm over a slower link (on my 100Mb
direct connection to the imap server, it's not a problem.  Over a
upstream DSL line (128k) it's much more annoying).

-- 
"The trouble with having an open mind, of course, is that people will
 insist on coming along and trying to put things in it."   -- Terry Pratchett

Re: double download of attachments?

[Simon Josefsson]

Wes Hardaker <wes@hardakers.net> writes:

> Currently (at least with nnimap) attachments seem to be downloaded
> twice.  Is there a way we can get around that (without a major code
> rewrite)?
>
> Specifically, send yourself something near a meg or so and then click
> on the article.  Watch the downloading size counter and you'll see it
> reset and start again from 0.  I assume that this is done twice once
> for display and once for the agent cache...
>
> It's a bit of a problem when I'm over a slower link (on my 100Mb
> direct connection to the imap server, it's not a problem.  Over a
> upstream DSL line (128k) it's much more annoying).

Does it help to use the backlog?  I.e., (setq gnus-keep-backlog 10)

Re: double download of attachments?

[Wes Hardaker]

>>>>> On Wed, 13 Nov 2002 10:15:47 +0100, Simon Josefsson <jas@extundo.com> said:

>> Currently (at least with nnimap) attachments seem to be downloaded
>> twice.  Is there a way we can get around that (without a major code
>> rewrite)?
>> 
>> Specifically, send yourself something near a meg or so and then click
>> on the article.  Watch the downloading size counter and you'll see it
>> reset and start again from 0.  I assume that this is done twice once
>> for display and once for the agent cache...
>> 
>> It's a bit of a problem when I'm over a slower link (on my 100Mb
>> direct connection to the imap server, it's not a problem.  Over a
>> upstream DSL line (128k) it's much more annoying).

Simon> Does it help to use the backlog?  I.e., (setq gnus-keep-backlog 10)

Yes, it does.  Though this still should be fixed (IMHO) if possible.

-- 
"The trouble with having an open mind, of course, is that people will
 insist on coming along and trying to put things in it."   -- Terry Pratchett

Re: double download of attachments?

[Simon Josefsson]

Wes Hardaker <wes@hardakers.net> writes:

>>>>>> On Wed, 13 Nov 2002 10:15:47 +0100, Simon Josefsson <jas@extundo.com> said:
>
>>> Currently (at least with nnimap) attachments seem to be downloaded
>>> twice.  Is there a way we can get around that (without a major code
>>> rewrite)?
>>> 
>>> Specifically, send yourself something near a meg or so and then click
>>> on the article.  Watch the downloading size counter and you'll see it
>>> reset and start again from 0.  I assume that this is done twice once
>>> for display and once for the agent cache...
>>> 
>>> It's a bit of a problem when I'm over a slower link (on my 100Mb
>>> direct connection to the imap server, it's not a problem.  Over a
>>> upstream DSL line (128k) it's much more annoying).
>
> Simon> Does it help to use the backlog?  I.e., (setq gnus-keep-backlog 10)
>
> Yes, it does.  Though this still should be fixed (IMHO) if possible.

Maybe we can "fix" it by enabling the backlog by default?

Re: double download of attachments?

[Niklas Morberg]

Simon Josefsson <jas@extundo.com> writes:

> Maybe we can "fix" it by enabling the backlog by default?

Does the backlog work without the agent and cache?

Niklas

Re: double download of attachments?

[Simon Josefsson]

Niklas Morberg <niklas.morberg@axis.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Maybe we can "fix" it by enabling the backlog by default?
>
> Does the backlog work without the agent and cache?

Yes, it is a independent in-memory session cache.

Re: double download of attachments?

[Wes Hardaker]

>>>>> On Thu, 14 Nov 2002 08:24:48 +0100, Simon Josefsson <jas@extundo.com> said:

Simon> Does it help to use the backlog?  I.e., (setq gnus-keep-backlog 10)
>> 
>> Yes, it does.  Though this still should be fixed (IMHO) if possible.

Simon> Maybe we can "fix" it by enabling the backlog by default?

I guess a default setting of 1 or 2 seems reasonable and should fix it?

-- 
"The trouble with having an open mind, of course, is that people will
 insist on coming along and trying to put things in it."   -- Terry Pratchett

Re: double download of attachments?

[Simon Josefsson]

Wes Hardaker <wes@hardakers.net> writes:

>>>>>> On Thu, 14 Nov 2002 08:24:48 +0100, Simon Josefsson <jas@extundo.com> said:
>
> Simon> Does it help to use the backlog?  I.e., (setq gnus-keep-backlog 10)
>>> 
>>> Yes, it does.  Though this still should be fixed (IMHO) if possible.
>
> Simon> Maybe we can "fix" it by enabling the backlog by default?
>
> I guess a default setting of 1 or 2 seems reasonable and should fix it?

OTOH sometimes when I enter groups with large messages, the summary
buffer doesn't match the article buffer contents sometimes, and I have
been suspecting the backlog and/or asynchronous prefetching for that
(but never been able to track down exactly what is happening).
<optimist>Maybe if we enable the backlog by default, people will see
the bug and fix it.</optimist>

Re: double download of attachments?

[dme]

* jas@extundo.com [2002-11-15 15:30:41]
> OTOH sometimes when I enter groups with large messages, the summary
> buffer doesn't match the article buffer contents sometimes, and I have
> been suspecting the backlog and/or asynchronous prefetching for that
> (but never been able to track down exactly what is happening).
> <optimist>Maybe if we enable the backlog by default, people will see
> the bug and fix it.</optimist>

I've seen this.  I disabled the backlog, and it didn't go away.
After spending some time watching the async prefetch buffer and
monitoring gnus-async-article-alist, entries in
gnus-async-article-alist were occasionally noticed to have bad
markers.

They were bad in the sense that they pointed to the wrong place
(i.e. if there are 10 articles in the prefetch buffer, 9 of the
entries in gnus-async-article-alist had markers pointing to the same
article).  The remainder of the entry was fine, just the markers were
bad.

I tuned the prefetch down to 3 articles (sufficient to make imap
reading bearable over a slowish link) and I've not seen the problem
again, but it's probably still lurking there.

Re: double download of attachments?

[Danny Siu]

Simon Josefsson writes:

  Simon> OTOH sometimes when I enter groups with large messages, the summary
  Simon> buffer doesn't match the article buffer contents sometimes, and I
  Simon> have been suspecting the backlog and/or asynchronous prefetching
  Simon> for that (but never been able to track down exactly what is
  Simon> happening).  <optimist>Maybe if we enable the backlog by default,
  Simon> people will see the bug and fix it.</optimist>

I have been observing the same problem (summary lines mis-matched with
articles) and neither can't find a backlog setting that would get around it.
As far as I can recall, the problem started when I switched the agent on
sometimes ago after the IMAP headers/caching work was committed.

-- 
Danny Siu

Re: double download of attachments?

[Simon Josefsson]

Danny Siu <dsiu@adobe.com> writes:

> Simon Josefsson writes:
>
>   Simon> OTOH sometimes when I enter groups with large messages, the summary
>   Simon> buffer doesn't match the article buffer contents sometimes, and I
>   Simon> have been suspecting the backlog and/or asynchronous prefetching
>   Simon> for that (but never been able to track down exactly what is
>   Simon> happening).  <optimist>Maybe if we enable the backlog by default,
>   Simon> people will see the bug and fix it.</optimist>
>
> I have been observing the same problem (summary lines mis-matched with
> articles) and neither can't find a backlog setting that would get around it.
> As far as I can recall, the problem started when I switched the agent on
> sometimes ago after the IMAP headers/caching work was committed.

Do you use asynchronous prefetching?  `gnus-asynchronous'.

Re: double download of attachments?

[Simon Josefsson]

dme@dme.org writes:

> * jas@extundo.com [2002-11-15 15:30:41]
>> OTOH sometimes when I enter groups with large messages, the summary
>> buffer doesn't match the article buffer contents sometimes, and I have
>> been suspecting the backlog and/or asynchronous prefetching for that
>> (but never been able to track down exactly what is happening).
>> <optimist>Maybe if we enable the backlog by default, people will see
>> the bug and fix it.</optimist>
>
> I've seen this.  I disabled the backlog, and it didn't go away.

I'm now able to reproduce it (it is easy to reproduce if you have some
large articles in the group that is prefetched early on). The cache,
agent or backlog is not the cause, it seems to be caused by the asynch
prefetch alone.  The agent, cache and backlog only indirectly worsen
the bug, since it makes the errorenous article stay around after the
asynch prefetch received the incorrect article.  When the agent, cache
and backlog is disabled, the asynch article is incorrect when you
first select it, but it is removed from the asynch cache when it is
read so if you select the article again the correct article is
downloaded.

I'll try to track it down further...

In any case, I think we should enable the backlog by default.

Re: double download of attachments?

[Danny Siu]

Simon Josefsson writes:

  Simon> Do you use asynchronous prefetching?  `gnus-asynchronous'.

yes.  does disabling asynchronous fetching help?
-- 
Danny Siu

Re: double download of attachments?

[Simon Josefsson]

Danny Siu <dsiu@adobe.com> writes:

> Simon Josefsson writes:
>
>   Simon> Do you use asynchronous prefetching?  `gnus-asynchronous'.
>
> yes.  does disabling asynchronous fetching help?

Yes.  I believe this was a nnimap bug which should be fixed in CVS now.

Did anyone ever see this bug in nntp groups too?

2002-11-16  Simon Josefsson  <jas@extundo.com>

	* nnimap.el (nnimap-callback-callback-function):
	(nnimap-callback-buffer): Removed, these cannot be global but must
	be embedded into the callback.
	(nnimap-make-callback): New.  Embedd article number, callback and
	buffer in function.
	(nnimap-callback, nnimap-request-article-part): Update.

--- nnimap.el.~6.52~	2002-10-27 11:47:24.000000000 +0100
+++ nnimap.el	2002-11-16 14:01:48.000000000 +0100
@@ -392,10 +392,6 @@
 (defvar nnimap-progress-chars '(?| ?/ ?- ?\\))
 (defvar nnimap-progress-how-often 20)
 (defvar nnimap-counter)
-(defvar nnimap-callback-callback-function nil
-  "Gnus callback the nnimap asynchronous callback should call.")
-(defvar nnimap-callback-buffer nil
-  "Which buffer the asynchronous article prefetch callback should work in.")
 (defvar nnimap-server-buffer-alist nil)	;; Map server name to buffers.
 (defvar nnimap-current-server nil) ;; Current server
 (defvar nnimap-server-buffer nil) ;; Current servers' buffer
@@ -784,19 +780,26 @@
 	     'identity)
 	   (or string "")))
 
-(defun nnimap-callback ()
-  (remove-hook 'imap-fetch-data-hook 'nnimap-callback)
-  (with-current-buffer nnimap-callback-buffer
+(defun nnimap-make-callback (article gnus-callback buffer)
+  "Return a callback function."
+  `(lambda () 
+     (nnimap-callback ,article ,gnus-callback ,buffer)))
+
+(defun nnimap-callback (article gnus-callback buffer)
+  (when (eq article (imap-current-message))
+    (remove-hook 'imap-fetch-data-hook
+		 (nnimap-make-callback article gnus-callback buffer))
+    (with-current-buffer buffer
     (insert
      (with-current-buffer nnimap-server-buffer
        (nnimap-demule
 	(if (imap-capability 'IMAP4rev1)
 	    ;; xxx don't just use car? alist doesn't contain
 	    ;; anything else now, but it might...
-	    (nth 2 (car (imap-message-get (imap-current-message) 'BODYDETAIL)))
-	  (imap-message-get (imap-current-message) 'RFC822)))))
+	      (nth 2 (car (imap-message-get article 'BODYDETAIL)))
+	    (imap-message-get article 'RFC822)))))
     (nnheader-ms-strip-cr)
-    (funcall nnimap-callback-callback-function t)))
+      (funcall gnus-callback t))))
 
 (defun nnimap-request-article-part (article part prop &optional
 					    group server to-buffer detail)
@@ -828,9 +831,10 @@
 					       gnus-newsgroup-name)
 				   (imap-error-text nnimap-server-buffer))
 		(cons group article)))
-	  (add-hook 'imap-fetch-data-hook 'nnimap-callback)
-	  (setq nnimap-callback-callback-function nnheader-callback-function
-		nnimap-callback-buffer nntp-server-buffer)
+	  (add-hook 'imap-fetch-data-hook
+		    (nnimap-make-callback article 
+					  nnheader-callback-function 
+					  nntp-server-buffer))
 	  (imap-fetch-asynch article part nil nnimap-server-buffer)
 	  (cons group article))))))
 

Re: double download of attachments?

[Lars Magne Ingebrigtsen]

Simon Josefsson <jas@extundo.com> writes:

>> Simon> Does it help to use the backlog?  I.e., (setq gnus-keep-backlog 10)
>>
>> Yes, it does.  Though this still should be fixed (IMHO) if possible.
>
> Maybe we can "fix" it by enabling the backlog by default?

I've now done so, and set it to 20.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: double download of attachments?

[Wes Hardaker]

>>>>> On Sat, 28 Dec 2002 22:25:44 +0100, Lars Magne Ingebrigtsen <larsi@gnus.org> said:

Lars> Simon Josefsson <jas@extundo.com> writes:
Simon> Does it help to use the backlog?  I.e., (setq gnus-keep-backlog 10)
>>> 
>>> Yes, it does.  Though this still should be fixed (IMHO) if possible.
>> 
>> Maybe we can "fix" it by enabling the backlog by default?

Lars> I've now done so, and set it to 20.

20 seems large.  Well, not for me and you certainly (with lots of
memory) but it seems large for smaller machines.  Again, not that I care.

-- 
"The trouble with having an open mind, of course, is that people will
 insist on coming along and trying to put things in it."   -- Terry Pratchett

Re: double download of attachments?

[Lars Magne Ingebrigtsen]

Wes Hardaker <wes@hardakers.net> writes:

> 20 seems large.  Well, not for me and you certainly (with lots of
> memory) but it seems large for smaller machines.

10, then?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen