url-retrieve parallelism

[Lars Magne Ingebrigtsen <larsi@gnus.org>]

shr (and gnus-html, I guess) fire off a call to `url-retrieve' for every
<img> it finds.  If a HTML message has 1000 <img>s, then Emacs is going
to do a DOS of the poor image web server.

We obviously want to have more than a single `url-retrieve' call going
at once, but we want to rate-limit this somewhat.  To perhaps 10 at a
time?  So we need some kind of easy callback-ey interface, I think...
But I'm wondering whether to make it a totally general library, that
would be, like:

(defun concurrent (concurrency function callback callback-arguments)
  ...)

So FUNCTION would be required to return a process object (and have a
parameter list based on `url-retrieve', which seems quite sensible), and
CONCURRENT would just maintain a queue of processes, and fire off a new
one (if any) when something returns, and so on...

Would this be useful, or is my master's in Over Engineering showing
again?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen