url-retrieve parallelism

url-retrieve parallelism

[Lars Magne Ingebrigtsen]

shr (and gnus-html, I guess) fire off a call to `url-retrieve' for every
<img> it finds.  If a HTML message has 1000 <img>s, then Emacs is going
to do a DOS of the poor image web server.

We obviously want to have more than a single `url-retrieve' call going
at once, but we want to rate-limit this somewhat.  To perhaps 10 at a
time?  So we need some kind of easy callback-ey interface, I think...
But I'm wondering whether to make it a totally general library, that
would be, like:

(defun concurrent (concurrency function callback callback-arguments)
  ...)

So FUNCTION would be required to return a process object (and have a
parameter list based on `url-retrieve', which seems quite sensible), and
CONCURRENT would just maintain a queue of processes, and fire off a new
one (if any) when something returns, and so on...

Would this be useful, or is my master's in Over Engineering showing
again?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url-retrieve parallelism

[Philipp Haselwarter]

As discussed here http://www.emacswiki.org/emacs/ConcurrentEmacs ?
That would be bloody ingenious.
Like, you could use this and I could have gnus refresh and get my mail
without emacs locking up? I'd definitely pay you a beer for that :)


-- 
Philipp Haselwarter

Re: url-retrieve parallelism

[Lars Magne Ingebrigtsen]

Philipp Haselwarter <philipp.haselwarter@gmx.de> writes:

> Like, you could use this and I could have gnus refresh and get my mail
> without emacs locking up?

No, not really.  This is just about scheduling external
processes/sockets.  Emacs doesn't multitask.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url-retrieve parallelism

[Ted Zlatanov]

On Sun, 19 Dec 2010 16:38:55 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Philipp Haselwarter <philipp.haselwarter@gmx.de> writes:
>> Like, you could use this and I could have gnus refresh and get my mail
>> without emacs locking up?

LMI> No, not really.  This is just about scheduling external
LMI> processes/sockets.  Emacs doesn't multitask.

Tom Tromey just committed some prep code for the concurrent branch.  It
may arrive this decade.

Ted

Re: url-retrieve parallelism

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> shr (and gnus-html, I guess) fire off a call to `url-retrieve' for every
> <img> it finds.  If a HTML message has 1000 <img>s, then Emacs is going
> to do a DOS of the poor image web server.

> We obviously want to have more than a single `url-retrieve' call going
> at once,

If they are all going to the same server, I think the network friendly
thing is to pipeline them in a single HTTP connection.  Ie. fire off all
GET requests for the images without waiting for a response, and handle
them as they arrive:
  http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1.2.2

That means that there needs to be something that listens to the incoming
byte stream and identifies what is a response header and routes it to
its handlers.

The old w3c libwww did that kind of thing, and curl does as well AFAIK.
I have no idea what (url-retrieve) does.  Hm... it has a callback
argument...?  No meaningful google matches on "url-retrieve pipeline"
though... 

Re: callbacks, the way libwww did it was to let the object representing
the request live in the system, and when a response corresponding to the
request returned, the request header obejct was linked to the response
header object.  The MIME type of the response was used to select a
handler.  Ie. it wasn't the request that determined what handler should
handle the response.  But the request object provided context, typically
so that the handler would know where to put its handled results.

I think where I'm going with this, is that just providing a callback
sounds too simple.  It's OK if you get what you want, but not if what
you get back is something other than what you asked for.

Re: url-retrieve parallelism

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> The old w3c libwww did that kind of thing, 

Of course, modeling a lisp program on the way a C library did things, is
probably not the right thing.

But I think the way it operated was sound: a request was fired off, and
the caller basically forgot about it.  When responses arrived, a handler
was created and the handled results were put in their right place, or
errors where created, and displayed in an "error place" (eg. a log or an
error window or the minibuffer or whatever).

But handling pipelinging means being more intimate with the TCP
connection than you probably plan to be...?

Re: url-retrieve parallelism

[Steinar Bang]

One thing you could do, if curl is installed on the system:
 Start curl in a separate process, giving it all the URLs you want to
 download, and then route the curl output into an emacs buffer and parse
 it for progress information and error messages
	http://curl.haxx.se/docs/manpage.html

curl will do things the network friendly way, and pipeline if it can,
AFAIK.

Re: url-retrieve parallelism

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> One thing you could do, if curl is installed on the system:

I'd rather not rely on external programs.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url-retrieve parallelism

[David Engster]

Lars Magne Ingebrigtsen writes:
> But I'm wondering whether to make it a totally general library, that
> would be, like:
>
> (defun concurrent (concurrency function callback callback-arguments)
>   ...)

Maybe the transaction queue library (tq.el, ships with Emacs) could help
you with this?  It's very small and simple; EMMS uses it to
asynchronously communicate with the music player daemon (mpd), and it
works quite well (debugging becomes a nightmare, though).

-David

Re: url-retrieve parallelism

[Lars Magne Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> Maybe the transaction queue library (tq.el, ships with Emacs) could help
> you with this?  It's very small and simple; EMMS uses it to
> asynchronously communicate with the music player daemon (mpd), and it
> works quite well (debugging becomes a nightmare, though).

tq seems to be about queueing chatter to an external process.  That's
doesn't quite apply to what `url-retrieve' does.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url-retrieve parallelism

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 320 bytes --]

On Sun, Dec 19 2010, Lars Magne Ingebrigtsen wrote:

> Would this be useful, or is my master's in Over Engineering showing
> again?

Sounds useful and a bit Over Engineering/paranoiac too, but I do not see
any other use-case than url-retrieve. Would be there?

-- 
Julien Danjou
❱ http://julien.danjou.info

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: url-retrieve parallelism

[Lars Magne Ingebrigtsen]

Julien Danjou <julien@danjou.info> writes:

> Sounds useful and a bit Over Engineering/paranoiac too, but I do not see
> any other use-case than url-retrieve. Would be there?

That's what I'm wondering...  xargs har -P max-procs, so somebody is
doing something in parallel, but I can't really think of a use case.

So perhaps this should just go into url.el.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url-retrieve parallelism

[Katsumi Yamaoka]

Lars Magne Ingebrigtsen wrote:
> So perhaps this should just go into url.el.

Please note that XEmacs' url.el is very old and won't be updated.
Cf. http://article.gmane.org/gmane.emacs.gnus.general/72704

Currently it's not used but I get:

Compiling gnus/lisp/gravatar.el...
While compiling the end of the data in file gnus/lisp/gravatar.el:
  ** the function url-retrieve-synchronously is not known to be defined.
Wrote gnus/lisp/gravatar.elc

Re: url-retrieve parallelism

[Lars Magne Ingebrigtsen]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Please note that XEmacs' url.el is very old and won't be updated.

Oh, darn.  I had forgotten about that...

I've been reading the XEmacs mailing list, and it seems like they're
making progress with their conversion to GPLv3, so you'd think they'd be
gearing up to just import stuff like url.el from Emacs now?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: url-retrieve parallelism

[Robert Pluim]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Katsumi Yamaoka <yamaoka@jpl.org> writes:
>
>> Please note that XEmacs' url.el is very old and won't be updated.
>
> Oh, darn.  I had forgotten about that...
>
> I've been reading the XEmacs mailing list, and it seems like they're
> making progress with their conversion to GPLv3, so you'd think they'd be
> gearing up to just import stuff like url.el from Emacs now?

Yes. There's various bits of code for tls and xml access written by some
larsi guy that are prime candidates for snarfing as well.

Robert

Re: url-retrieve parallelism

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> So perhaps this should just go into url.el.

Thinking about this a bit more, I think it might make sense to try to
make this be a general run-in-parallel package, even though url fetching
is the only use case at present.  Especially since this has to be
supported across many {X,}Emacs versions.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen