Fetching all messages into the Agent

Fetching all messages into the Agent

[Richard Riley]

When firing off J s in the group buffer is there a way, or could there
please, to give some sort of progress feedback? As it is gnus just "locks
up" (ie fetches with no indication its doing anything) , sometimes for a
long time, with no indication its doing anything! Possibly just message
the group currently being fetched.

Re: Fetching all messages into the Agent

[Robert Pluim]

Richard Riley <rileyrg@gmail.com> writes:

> When firing off J s in the group buffer is there a way, or could there
> please, to give some sort of progress feedback? As it is gnus just "locks
> up" (ie fetches with no indication its doing anything) , sometimes for a
> long time, with no indication its doing anything! Possibly just message
> the group currently being fetched.

When I used the agent, I used the following. I hereby place it in the
public domain (although I doubt it passes the 'originality' test :) )

diff --git a/lisp/gnus-agent.el b/lisp/gnus-agent.el
index 3597037..530ebd6 100644
--- a/lisp/gnus-agent.el
+++ b/lisp/gnus-agent.el
@@ -1570,10 +1570,12 @@ downloaded into the agent."
                (date (time-to-days (current-time)))
                (case-fold-search t)
                pos crosses id
-	       (file-name-coding-system nnmail-pathname-coding-system))
+	       (file-name-coding-system nnmail-pathname-coding-system)
+	       (n 1) total)
 
           (setcar selected-sets (nreverse (car selected-sets)))
           (setq selected-sets (nreverse selected-sets))
+	  (setq total (reduce '+ (mapcar 'length selected-sets)))
 
           (gnus-make-directory dir)
 	  (gnus-message 7 "Fetching articles for %s..."
@@ -1587,9 +1589,10 @@ downloaded into the agent."
                   (with-temp-buffer
                     (let (article)
                       (while (setq article (pop articles))
-                        (gnus-message 10 "Fetching article %s for %s..."
-				      article
+                        (gnus-message 7 "Fetching article %d of %d for %s..."
+				      n total
 				      (gnus-agent-decoded-group-name group))
+			(incf n)
                         (when (or
                                (gnus-backlog-request-article group article
                                                              nntp-server-buffer)

Re: Fetching all messages into the Agent

[Eric S Fraga]

Richard Riley <rileyrg@gmail.com> writes:

> When firing off J s in the group buffer is there a way, or could there
> please, to give some sort of progress feedback? As it is gnus just "locks
> up" (ie fetches with no indication its doing anything) , sometimes for a
> long time, with no indication its doing anything! Possibly just message
> the group currently being fetched.

I would indeed appreciate something like this, especially lately as it
seems that the agent support has changed and many more articles are
being downloaded (but this could be my imagination).

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.1.50.1 + Ma Gnus v0.6

Re: Fetching all messages into the Agent

[Tassilo Horn]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

>> When firing off J s in the group buffer is there a way, or could
>> there please, to give some sort of progress feedback? As it is gnus
>> just "locks up" (ie fetches with no indication its doing anything) ,
>> sometimes for a long time, with no indication its doing anything!
>> Possibly just message the group currently being fetched.
>
> I would indeed appreciate something like this, especially lately as it
> seems that the agent support has changed and many more articles are
> being downloaded (but this could be my imagination).

See the other thread "Excessive nntp reads since today" starting with
Message-ID: <877gvcd91f.fsf@thinkpad.tsdh.de>.

Bye,
Tassilo

Re: Fetching all messages into the Agent

[Wolfgang Jenkner]

On Thu, Jun 14 2012, Eric S Fraga wrote:

> lately
Lately?

> as it seems that the agent support has changed and many more articles
> are being downloaded (but this could be my imagination).

Could you please find out if `gnus-agent-retrieve-headers' is called
with a non-nil last (`fetch-old') argument while headers are being
downloaded.  If so, do you know why?

If I do trace-function-background on this function and then J s in the
*Group* buffer I get a *trace-output* buffer with stanzas like

======================================================================
1 -> gnus-agent-retrieve-headers: articles=(289666 289667 289668 289669 289670 289671 289672 289673 289674 289675 289676 289677 289678 289679 289680 289681 289682 289683 289684 289685 289686 289687 289688 289689 289690 289691 289692 289693 289694 289695 289696 289697 289698 289699 289700 289701 289702 289703 289704 289705 289706 289707 289708 289709 289710 289711 289712 289713 289714 289715 289716) group="gmane.os.freebsd.questions" fetch-old=nil
1 <- gnus-agent-retrieve-headers: nov

All of them with fetch-old=nil.

Wolfgang

Re: Fetching all messages into the Agent

[Eric S Fraga]

Wolfgang Jenkner <wjenkner@inode.at> writes:

> On Thu, Jun 14 2012, Eric S Fraga wrote:
>
>> lately
> Lately?

well, lately for me because I stopped using the agent 4-5 months ago: my
situation changed so that I seldom was offline compared with my previous
case of 3 hours commute per day...  However, I started using the agent
again this past week as my situation has changed yet again!

>> as it seems that the agent support has changed and many more articles
>> are being downloaded (but this could be my imagination).
>
> Could you please find out if `gnus-agent-retrieve-headers' is called
> with a non-nil last (`fetch-old') argument while headers are being
> downloaded.  If so, do you know why?
>
> If I do trace-function-background on this function and then J s in the
> *Group* buffer I get a *trace-output* buffer with stanzas like

I will try to do this when next online.  Will this help me answer your
question regarding the argument to gnus-agent-retrieve-headers?

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.1.50.1 + Ma Gnus v0.6

Re: Fetching all messages into the Agent

[Wolfgang Jenkner]

On Thu, Jun 14 2012, Eric S Fraga wrote:

>> If I do trace-function-background on this function and then J s in the
>> *Group* buffer I get a *trace-output* buffer with stanzas like
>
> I will try to do this when next online.  Will this help me answer your
> question regarding the argument to gnus-agent-retrieve-headers?

Yes, of course.  I would just like to know if your observation is
related to my patch and the other mailing list thread Tassilo referred
you to.

Wolfgang

Re: Fetching all messages into the Agent

[Tassilo Horn]

Wolfgang Jenkner <wjenkner@inode.at> writes:

>> as it seems that the agent support has changed and many more articles
>> are being downloaded (but this could be my imagination).
>
> Could you please find out if `gnus-agent-retrieve-headers' is called
> with a non-nil last (`fetch-old') argument while headers are being
> downloaded.  If so, do you know why?
>
> If I do trace-function-background on this function and then J s in the
> *Group* buffer I get a *trace-output* buffer with stanzas like
>
> ======================================================================
> 1 -> gnus-agent-retrieve-headers: articles=(289666 289667 289668 289669 289670
> 289671 289672 289673 289674 289675 289676 289677 289678 289679 289680 289681
> 289682 289683 289684 289685 289686 289687 289688 289689 289690 289691 289692
> 289693 289694 289695 289696 289697 289698 289699 289700 289701 289702 289703
> 289704 289705 289706 289707 289708 289709 289710 289711 289712 289713 289714
> 289715 289716) group="gmane.os.freebsd.questions" fetch-old=nil
> 1 <- gnus-agent-retrieve-headers: nov
>
> All of them with fetch-old=nil.

I bet 5 virtual dollars on Eric using `gnus-fetch-old-headers' set to t
or 'some. ;-)

Bye,
Tassilo

Re: Fetching all messages into the Agent

[Eric S Fraga]

Tassilo Horn <tassilo@member.fsf.org> writes:

> Wolfgang Jenkner <wjenkner@inode.at> writes:
>
>>> as it seems that the agent support has changed and many more articles
>>> are being downloaded (but this could be my imagination).
>>
>> Could you please find out if `gnus-agent-retrieve-headers' is called
>> with a non-nil last (`fetch-old') argument while headers are being
>> downloaded.  If so, do you know why?
>>
>> If I do trace-function-background on this function and then J s in the
>> *Group* buffer I get a *trace-output* buffer with stanzas like
>>
>> ======================================================================
>> 1 -> gnus-agent-retrieve-headers: articles=(289666 289667 289668 289669 289670
>> 289671 289672 289673 289674 289675 289676 289677 289678 289679 289680 289681
>> 289682 289683 289684 289685 289686 289687 289688 289689 289690 289691 289692
>> 289693 289694 289695 289696 289697 289698 289699 289700 289701 289702 289703
>> 289704 289705 289706 289707 289708 289709 289710 289711 289712 289713 289714
>> 289715 289716) group="gmane.os.freebsd.questions" fetch-old=nil
>> 1 <- gnus-agent-retrieve-headers: nov
>>
>> All of them with fetch-old=nil.
>
> I bet 5 virtual dollars on Eric using `gnus-fetch-old-headers' set to t
> or 'some. ;-)

Sorry but you've lost the bet ;-)

I did use to have that set to 'some but turned it off a little while ago
(maybe a few weeks ago?) due to the sudden slowness, as you have
diagnosed in another thread.  I now have it set to nil.  I would prefer
to have the old behaviour that I used to get with 'some but it's not
critical and I understand the change.  A T does the job for me well
enough when I need the full thread.

However, this setting doesn't seem to make any difference to the agent.

I still haven't tried the tracing suggested by Wolfgang (too busy
reading emails ;-).

thanks,
eric

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.1.50.1 + Ma Gnus v0.6

Re: Fetching all messages into the Agent

[Eric S Fraga]

Wolfgang Jenkner <wjenkner@inode.at> writes:

> On Thu, Jun 14 2012, Eric S Fraga wrote:
>
>> lately
> Lately?
>
>> as it seems that the agent support has changed and many more articles
>> are being downloaded (but this could be my imagination).
>
> Could you please find out if `gnus-agent-retrieve-headers' is called
> with a non-nil last (`fetch-old') argument while headers are being
> downloaded.  If so, do you know why?

No, it looks like fetch-old is set to nil:

  1 -> gnus-agent-retrieve-headers:
       articles=(1490 2723 3828 3832 3833 3834 3838 3839 3840 3844 3846
       3849 3850)
       group="nnimap+ucl:lists.gnus" fetch-old=nil
  1 <- gnus-agent-retrieve-headers: nov

and this all looks quite reasonable actually.  The slowness must have
been my imagination.  It would still be useful to have some progress
reporting during agent download, especially for large groups.

This is with gnus up to date from git as of a few minutes ago, just in
case.

thanks,
eric

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.1.50.1 + Ma Gnus v0.6

Re: Fetching all messages into the Agent

[Lars Magne Ingebrigtsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> It would still be useful to have some progress reporting during agent
> download, especially for large groups.

Yes.  But flashing a gazillion messages at people isn't nice, either.

During the Emacs 24.2 cycle I hope to implement `maybe-message' (or
something) that will allow us a convenient way to message stuff if
things take too long.

(maybe-message 1.0 "Fetching group foo...")

If Emacs hasn't returned to the idle loop before one second has passed,
then the message is displayed.  That way we'll only show a message if it
turns out that the function took a long while to complete.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Fetching all messages into the Agent

[Eric S Fraga]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> It would still be useful to have some progress reporting during agent
>> download, especially for large groups.
>
> Yes.  But flashing a gazillion messages at people isn't nice, either.
>
> During the Emacs 24.2 cycle I hope to implement `maybe-message' (or
> something) that will allow us a convenient way to message stuff if
> things take too long.
>
> (maybe-message 1.0 "Fetching group foo...")
>
> If Emacs hasn't returned to the idle loop before one second has passed,
> then the message is displayed.  That way we'll only show a message if it
> turns out that the function took a long while to complete.

That would be great.

I agree that flashing tons of messages is not a good thing but, at the
moment, I am working with a fragile connection and I cannot be sure that
the delay (seconds to minutes sometimes for downloading to the agent) is
due to gnus working away or Emacs having problems with the
connection.  Getting a periodic indication from gnus that something is
actually happening would be quite helpful.

thanks,
eric

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.1.50.1 + Ma Gnus v0.6

Re: Fetching all messages into the Agent

[Richard Riley]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> It would still be useful to have some progress reporting during agent
>> download, especially for large groups.
>
> Yes.  But flashing a gazillion messages at people isn't nice, either.

No one suggested a gazillion. Just something. As it is it locks up for
ages with no indication anything is happening.

Re: Fetching all messages into the Agent

[Vagn Johansen]

Richard Riley <rileyrg@gmail.com> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>
>>> It would still be useful to have some progress reporting during agent
>>> download, especially for large groups.
>>
>> Yes.  But flashing a gazillion messages at people isn't nice, either.
>
> No one suggested a gazillion. Just something. As it is it locks up for
> ages with no indication anything is happening.

It can be difficult to predict if something on occasion happends a
gazillion times, so maybe-message idea is really cool.

-- 
Vagn Johansen

Re: Fetching all messages into the Agent

[Ted Zlatanov]

On Wed, 20 Jun 2012 20:33:57 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> During the Emacs 24.2 cycle I hope to implement `maybe-message' (or
LMI> something) that will allow us a convenient way to message stuff if
LMI> things take too long.

LMI> (maybe-message 1.0 "Fetching group foo...")

LMI> If Emacs hasn't returned to the idle loop before one second has passed,
LMI> then the message is displayed.  That way we'll only show a message if it
LMI> turns out that the function took a long while to complete.

May I suggest a macro like (with-timeout-message TIMEOUT MESSAGE BODY)?

That's much more useful because you can time any body or function.  We
already have `with-timeout' in timer.el but that actually aborts the
code.  Still, maybe you can use its ideas.

Thanks
Ted

Re: Fetching all messages into the Agent

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> May I suggest a macro like (with-timeout-message TIMEOUT MESSAGE BODY)?
>
> That's much more useful because you can time any body or function.  We
> already have `with-timeout' in timer.el but that actually aborts the
> code.  Still, maybe you can use its ideas.

I'd like to keep the code un-general and specifically useful.  :-)

It would be kinda nice if it didn't evaluate anything until after the
timeout, but that may be kinda...  icky...

That is, it would be nice if one were able to write stuff like

(maybe-message 1 "This sure is a long list: %d" (compute-length-of-list list))

with impunity, without having to worry about the message taking a long
time to create, because it'll only be displayed in the rare instances
where things really took a long time.

But I think that would be 1) weird, 2) unpredictable and 3) difficult to
implement.

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Lars Magne Ingebrigtsen

Re: Fetching all messages into the Agent

[Lars Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> May I suggest a macro like (with-timeout-message TIMEOUT MESSAGE BODY)?

I just had a new idea.  What about

(with-timeout-forms
   (progn
     (bla)
     (bla))
  (1 (message "This is sure taking a lot of time..."))
  (10 (message "This is sure taking forever!")))

A la condition-case and stuff, although here we have forms that are
run if the main form takes too long.  And we can have several of them,
like "Waiting for Gmail to finish; type C-g to skip" in the second
form.

And stuff.  This almost sounds generally useful.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  http://lars.ingebrigtsen.no  *  Lars Magne Ingebrigtsen