Re: url-retrieve parallelism

[Steinar Bang <sb@dod.no>]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> shr (and gnus-html, I guess) fire off a call to `url-retrieve' for every
> <img> it finds.  If a HTML message has 1000 <img>s, then Emacs is going
> to do a DOS of the poor image web server.

> We obviously want to have more than a single `url-retrieve' call going
> at once,

If they are all going to the same server, I think the network friendly
thing is to pipeline them in a single HTTP connection.  Ie. fire off all
GET requests for the images without waiting for a response, and handle
them as they arrive:
  http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1.2.2

That means that there needs to be something that listens to the incoming
byte stream and identifies what is a response header and routes it to
its handlers.

The old w3c libwww did that kind of thing, and curl does as well AFAIK.
I have no idea what (url-retrieve) does.  Hm... it has a callback
argument...?  No meaningful google matches on "url-retrieve pipeline"
though... 

Re: callbacks, the way libwww did it was to let the object representing
the request live in the system, and when a response corresponding to the
request returned, the request header obejct was linked to the response
header object.  The MIME type of the response was used to select a
handler.  Ie. it wasn't the request that determined what handler should
handle the response.  But the request object provided context, typically
so that the handler would know where to put its handled results.

I think where I'm going with this, is that just providing a callback
sounds too simple.  It's OK if you get what you want, but not if what
you get back is something other than what you asked for.