Getting articles asynchronously.....

Getting articles asynchronously.....

[Patrick Audley]

	Since it is possible to fetch the next article asynchronously
while reading an article, is there a way to make gnus fetch all
articles in the background.  This way, one could select all the
binaries in, say, comp.binaries.os2 and hit "X U" then go on one's
merry way, emacsing as usual while Gnus fetches and uudecodes all my
spiffy new utilities ;)

-- 
... Putting your best foot forward at least keeps it out of your mouth.
			  -Morris Mandel in _The Jewish Press_
/*----------------------------------------------------------------.
| The Crystal Wind is The Storm,      Patrick Audley              |
|    The Storm is The Data,     ______/\/\/\/\/\/\/\_______       |
|      The Data is Life.        InterNet: paudley@portal.ca       |
`---[OS/2]--[Anime]-[Trance]-[C++]-------------------------------*/

Re: Getting articles asynchronously.....

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "PA" == Patrick Audley <paudley@portal.ca> writes:

PA> Since it is possible to fetch the next article asynchronously while
PA> reading an article, is there a way to make gnus fetch all articles
PA> in the background.

That is an exceedingly bad plan in large newsgroups or groups with large
articles, such as source or binaries newsgroups.  Emacs will run out of
memory before it's finished pre-fetching.

PA> as usual while Gnus fetches and uudecodes all my spiffy new
PA> utilities ;)

That would require a multithreaded Emacs kernel, something that a) does
not exist and, b) likely won't exist for quite some time.

-----BEGIN PGP SIGNATURE-----
Version: 2.6.2
Comment: Processed by Mailcrypt 3.4, an Emacs/PGP interface

iQCVAwUBMOAELJ6VRH7BJMxHAQEE8QQAlfHC26AEaoxXyVk6hShByOcL+1NXS8P1
lTRJt5+56fq6UKN8bIzrZjTOvMZM/md5mFWGvaX5GFWObB27L4aIiUPJHW4BxJGj
WfSjiLjT0/GzvaGhhHONeO/tgEVioaTzADcRu64gVJB8SJCPvNQjJjafe1Xly/OQ
KV56VDXtEkQ=
=hdl1
-----END PGP SIGNATURE-----
-- 
Rat <ratinox@ccs.neu.edu>          \ Warning: pregnant women, the elderly, and
PGP Public Key: Ask for one today!  \ children under 10 should avoid prolonged
http://www.ccs.neu.edu/home/ratinox/ \ exposure to Happy Fun Ball.

Re: Getting articles asynchronously.....

[Patrick Audley]

>>>>> "Rat" == Stainless Steel Rat <ratinox@ccs.neu.edu> writes:
>>>>> "PA" == Patrick Audley <paudley@portal.ca> writes:

Rat> That is an exceedingly bad plan in large newsgroups or groups
Rat> with large articles, such as source or binaries newsgroups.
Rat> Emacs will run out of memory before it's finished pre-fetching.

	No, No...  I didn't mean that Gnus should fetch articles
arbitrarily.  Here's an example...  Imagine you're on a slow
connection (say.. 28.8) and you're browsing you're favorite binaries
group.  You see a great program that you just have to have, so you
mark all the uuencoded parts and hit "XU".  And wait...  And
Wait... And Wait.

	Since Gnus can already fetch articles in the background (I
assume this is what gnus-asynchronous toggles), is there not a way to
do all fetching in the background.  The TCP process that connects to
the NNTP server (at least under OS/2) is running as a separate
process, could it not spool all the information into a buffer and
notify Gnus when it was done?

Rat> That would require a multithreaded Emacs kernel, something that
Rat> a) does not exist and, b) likely won't exist for quite some time.

	I know that at least some of this is possible now, for
instance, start up a (shell) and execute some _LONG_ task, say zipping
20M of text or something.  Switch to another window, and the shell
keeps feeding output to the *Shell* buffer, while you're doing other
things.

-- 
... They laughed at Edison and Einstein, but somehow I still feel uncomfortable
   when they laugh at me.  -Ashleigh Brilliant (_Pot Shots_)
/*----------------------------------------------------------------.
| The Crystal Wind is The Storm,      Patrick Audley              |
|    The Storm is The Data,     ______/\/\/\/\/\/\/\_______       |
|      The Data is Life.        InterNet: paudley@portal.ca       |
`---[OS/2]--[Anime]-[Trance]-[C++]-------------------------------*/

Re: Getting articles asynchronously.....

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "PA" == Patrick Audley <paudley@portal.ca> writes:

PA> 	Since Gnus can already fetch articles in the background

Actually, it doesn't.  The asynchronous article fetch occours when Emacs
isn't doing anything else.  Ie, it is multitasking, not multithreading,
just like an inferior shell process.  Inferior shells are the same: the
inferior process (shell, TCP/IP connection, whatever) is chugging along
independant of Emacs, spitting output through whatever (interrupt driver
like) process handler has been set up.

What you are asking is to have Emacs do two things at once: process
marked articles and read more news.  Neither of these are handled in
inferior processes.  In short, Emacs just can't do it because it isn't
threaded.

-----BEGIN PGP SIGNATURE-----
Version: 2.6.2
Comment: Processed by Mailcrypt 3.4, an Emacs/PGP interface

iQCVAwUBMOCGZp6VRH7BJMxHAQHOfAP/dEd/BxsGYYe1nwszW/ID55rGjwh41Htm
GaAbuyuJNl9n/luI1iuArf22esMx4QhVE6bo4+esje+6VL0TPcvpobESmRrbyqma
7nm6o5EAusg9ZPpPMQWD48nscQiZX+iebDGzT9J+mzCgx1IIzL1fogk9GyhJOs3R
5GcCZBr5u24=
=aRCT
-----END PGP SIGNATURE-----
-- 
Rat <ratinox@ccs.neu.edu>          \ Do not use Happy Fun Ball on concrete.
PGP Public Key: Ask for one today!  \ 
http://www.ccs.neu.edu/home/ratinox/ \ 

Re: Getting articles asynchronously.....

[Felix Lee]

> That is an exceedingly bad plan in large newsgroups or groups with large
> articles, such as source or binaries newsgroups.  Emacs will run out of
> memory before it's finished pre-fetching.

well, you pre-fetch to disk, rather than to memory.

> That would require a multithreaded Emacs kernel, something that a) does
> not exist and, b) likely won't exist for quite some time.

you can do most of the job by running subprocesses.  a caching nntp
proxy could take care of asynchronous article fetching.  once you have
that, gnus just needs to add a "maybe-it-would-be-a-good-idea-
to-prefetch-this" entry point to its backend interface.

(async article fetching is on my wishlist-that-I-might-actually-do-
something-about, because "Free Agent" (a Windoze newsreader my
boyfriend uses) is actually much nicer than gnus over a ppp link.  and
there's no way I'm going to run Windoze to use a newsreader.)
--

Re: Getting articles asynchronously.....

[Lars Magne Ingebrigtsen]

Felix Lee <flee@teleport.com> writes:

> > That would require a multithreaded Emacs kernel, something that a) does
> > not exist and, b) likely won't exist for quite some time.
> 
> you can do most of the job by running subprocesses.  a caching nntp
> proxy could take care of asynchronous article fetching.  once you have
> that, gnus just needs to add a "maybe-it-would-be-a-good-idea-
> to-prefetch-this" entry point to its backend interface.

Howzabout an "asynchronous cache"?  I mean, you press a magic key on
(say) a group, and then Gnus starts fetching the (unread) articles in
the group asynchronously and puts them all in the cache as they
arrive.  The user would typically do something else than wait for the
cache to fill up -- read mail, or whatver.  (If you have a slow disk,
this will mean that Emacs would become more sluggish since most of the
time would be spent in `write-region' calls in the "asynchronous" part
of the code, which would basically be filters that sniffs the output
from the tcp process.)

Then, uhm, when you enter the group all the articles are read from
disk, as usual when using the cache.

... perhaps one could have "postponed commands"?  Process-mark a few
articles, press a magic "postpone" key and then `X U', and Gnus would
do the asynchronous cache thing on the process-marked articles, and
the execute the `X U' when all the articles have arrived in the cache.

Or whatever.

I don't think this would be extremely much work, actually.  It would
be pretty localized and would rely on the already established cache
mechanisms. 

-- 
   Lars Magne Ingebrigtsen * larsi@ifi.uio.no
      (a red leaf that falls from the purple tree)

Re: Getting articles asynchronously.....

[Mark W. Eichin]

> That would require a multithreaded Emacs kernel, something that a) does
> not exist and, b) likely won't exist for quite some time.

Actually, I've heard that some progress has been made integrating
pthreads into guile, and an emacs that uses guile (either with or
instead of elisp) is a likely next step, so "quite some time" may be a
matter of months...
					_Mark_ <eichin@cygnus.com>
					Cygnus Support, Eastern USA

Re: Getting articles asynchronously.....

[Greg Stark]

Actually, I was just in the process of writing a back end, and it occurred to
me I was taking an essentially asynchronous operation (interacting with a
subprocess) and serializing it.

This is an enormous lose, an existing interface allows users to start an
operation and go work on other things; it does this by putting most of the
work in the filter function.

The basic design to make Gnus do this would be to split each of the backend
interface functions into two steps.  Gnus would call a normal looking backend
function, but would pass a couple extra arguments.  One of the extra arguments
would be a function the backend function would call when it finshed its
processing.  I think it would also be useful to pass a nonce along as well and
pass it back to Gnus.

I would have said this would be very difficult to get right, but I also would
have said that about a lot of existing Gnus features.  And in my opinion this
is the key feature that would make Gnus a lot more useful.  I do a lot of
things in my Emacs and it frustrates me when it freezes for more than a few
seconds.

In fact, I think the least robust part of my backend is going to be the
joining of the synchronous backend interface and the asynchronous sub-process.
It would be easier to implement robustly if both were asynchronous.

greg




In case I wasn't clear here's an example of what I mean:

(defun nnchoke-retrieve-headers (articles call-back nonce &optional g s f)
  (start-process foo ...)
  (set-process-filter foo 'nnchoke-retrieve-headers-do-work)
  'in-progress)

(defun nnchoke-retrieve-headers-do-work (p s)
  ...
  process a lot of data
  but without blocking Emacs
  ...
  (funcall call-back nonce 'nov))


obviously some parts of the api need to be fixed to handle this, 
The nntp-server-buffer must either be passed down or dynamically bound.
The concept of a ``current group'' should probably be given up.
things like that, but nothing terribly major.
(except that i don't think the above example works because of the scoping of
call-back and nonce, hmm)

greg

Re: Getting articles asynchronously.....

[Greg Stark]

Hmm, having thought about this a bit more it seems to me that the right design
is a little subtler than I described.  Instead specific functions could be
made optionally asynchronous in a backwards compatible manner by having a
valid return value be something of the form '(in-progess nonce) where nonce
would be something returned by gnus-make-nonce.

Then those specific functions would have corresponding call-back functions
that the backend would be expected to call when they completed or in some
cases when even some of the data was ready.  So for example if gnus called
nnchoke-retrieve-headers then nnchoke would return (list ('in-progress
(gnus-make-nonce))) and set the filter function on its process or network
stream to something that would call gnus-callback-retrieve-headers with the
resultant data and the returned value (which would include the nonce so gnus
would know what operation had just completed).  

A few notes, some things like the use of a single buffer for backends to put
the resulting data in would have to be fixed, since as long as there's a
pending operation on one buffer gnus would have to be careful not to reuse
that buffer.  Also, it would be nice if Gnus also provided some callbacks for
processing partial results, instead of having to fetch all the data then
process it all.  There's a lot of idle time when fetching data that the
processor could be busy processing the data it has already.

There's a lot of real gains to be made here.  Gnus would be a lot more useful
if it didn't lock up Emacs while doing long operations like listing groups,
checking for bogus groups, or entering a group (once took me over 30 minutes).
And it can be faster if it didn't inneffiently wait for all the data to
arrive, perhaps over a slow connection, before trying to process any of it.


greg


>   Actually, I was just in the process of writing a back end, and it occurred
>   to me I was taking an essentially asynchronous operation (interacting with 
>   a subprocess) and serializing it.
>
>   This is an enormous lose, an existing interface allows users to start an
>   operation and go work on other things; it does this by putting most of the
>   work in the filter function.
>
>   The basic design to make Gnus do this would be to split each of the backend
>   interface functions into two steps.  Gnus would call a normal looking backend
>   function, but would pass a couple extra arguments.  One of the extra arguments
>   would be a function the backend function would call when it finshed its
>   processing.  I think it would also be useful to pass a nonce along as well and
>   pass it back to Gnus.
>
>   I would have said this would be very difficult to get right, but I also would
>   have said that about a lot of existing Gnus features.  And in my opinion this
>   is the key feature that would make Gnus a lot more useful.  I do a lot of
>   things in my Emacs and it frustrates me when it freezes for more than a few
>   seconds.
>
>   In fact, I think the least robust part of my backend is going to be the
>   joining of the synchronous backend interface and the asynchronous sub-process.
>   It would be easier to implement robustly if both were asynchronous.
>
>   greg
>
>
>
>
>   In case I wasn't clear here's an example of what I mean:
>
>   (defun nnchoke-retrieve-headers (articles call-back nonce &optional g s f)
>     (start-process foo ...)
>     (set-process-filter foo 'nnchoke-retrieve-headers-do-work)
>     'in-progress)
>
>   (defun nnchoke-retrieve-headers-do-work (p s)
>     ...
>     process a lot of data
>     but without blocking Emacs
>     ...
>     (funcall call-back nonce 'nov))
>
>
>   obviously some parts of the api need to be fixed to handle this, 
>   The nntp-server-buffer must either be passed down or dynamically bound.
>   The concept of a ``current group'' should probably be given up.
>   things like that, but nothing terribly major.
>   (except that i don't think the above example works because of the scoping of
>   call-back and nonce, hmm)
>
>   greg
>

Re: Getting articles asynchronously.....

[Steven L Baur]

>>>>> "Greg" == Greg Stark <gsstark@MIT.EDU> writes:

Greg> There's a lot of real gains to be made here.  Gnus would be a
Greg> lot more useful if it didn't lock up Emacs while doing long
Greg> operations like listing groups, checking for bogus groups, or
Greg> entering a group (once took me over 30 minutes).

It's making less and less sense to me to have Gnus wait while reading
the NNTP active file at startup, with mail to be read and such.

Could the active file be read by a subprocess similar to the exmh-bg
mailchecker in exmh?

-- 
steve@miranova.com baur

Re: Getting articles asynchronously.....

[Ketil Z Malde]

>>>>> "Greg" == Greg Stark <gsstark@MIT.EDU> writes:

    Greg> There's a lot of real gains to be made here.  Gnus would be
    Greg> a lot more useful if it didn't lock up Emacs while doing
    Greg> long operations like listing groups, checking for bogus
    Greg> groups, or entering a group (once took me over 30 minutes).

>>>>> "Steven" == Steven L Baur <steve@miranova.com> writes:

  Steven> It's making less and less sense to me to have Gnus wait
  Steven> while reading the NNTP active file at startup, with mail
  Steven> to be read and such.

Not to mention a couple of nnkiboze processes that would be sooo nice
to run in the background.  I assume a multithreaded emacs is
non-trivial? 

-kzm

Re: Getting articles asynchronously.....

[Greg Stark]

   >>>>> "Steven" == Steven L Baur <steve@miranova.com> writes:
     Steven> It's making less and less sense to me to have Gnus wait
     Steven> while reading the NNTP active file at startup, with mail
     Steven> to be read and such.
   Not to mention a couple of nnkiboze processes that would be sooo nice
   to run in the background.  I assume a multithreaded emacs is
   non-trivial? 
   -kzm

The point I'm trying to make is that you don't need a multithreaded emacs.
The interface to network streams and subprocesses is already asynchronous, 
we just need to make the interface to Gnus backends asynchronous as well. 

Some existing interfaces already support this.  
I've seen asynchonous mh, discuss, and rmail interfaces.
(well I haven't actually seen the last one, but i'm told it almost exists.)

greg

Re: Getting articles asynchronously.....

[Massimo Campostrini]

In article <ycqsphgiemc.fsf@fierce-bad-rabbit.MIT.EDU> gsstark@MIT.EDU (Greg Stark) writes:

> The point I'm trying to make is that you don't need a multithreaded emacs.
> The interface to network streams and subprocesses is already asynchronous, 
> we just need to make the interface to Gnus backends asynchronous as well. 
> 
> Some existing interfaces already support this.  
> I've seen asynchonous mh, discuss, and rmail interfaces.
> (well I haven't actually seen the last one, but i'm told it almost exists.)

The standard `manual-entry' command interface has been asynchonous for
a long time (at least since emacs 18).  Cfr. `man.el'.

        Massimo Campostrini, 
Istituto Nazionale di Fisica Nucleare, Sezione di Pisa.
WWW home page: http://www.difi.unipi.it/~campo/

Re: Getting articles asynchronously.....

[Lars Magne Ingebrigtsen]

Massimo Campostrini <campo@sunthpi3.difi.unipi.it> writes:

> The standard `manual-entry' command interface has been asynchonous for
> a long time (at least since emacs 18).  Cfr. `man.el'.

Yes, but man.el simply starts a man/sed/awk combo that runs in the
background.  There are no sequencing or data munging problems (and
it's buffer-oriented to boot), so it's a completely different issue
than making Gnus asynchronous. 

-- 
Home is where the cat is.

Re: Getting articles asynchronously.....

[Lars Magne Ingebrigtsen]

Steven L Baur <steve@miranova.com> writes:

> Could the active file be read by a subprocess similar to the exmh-bg
> mailchecker in exmh?

No.  But the thing that takes time is transferring the file from the
server to Emacs, which is something that one doesn't have to wait for,
really.  I've got some ideas.  We'll see whether they are doable.

-- 
Home is where the cat is.