Re: Asynchroneous image retrieval in HTML rendering

[Ted Zlatanov <tzz@lifelogs.com>]

On Mon, 02 May 2011 16:30:09 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Antoine Levitt <antoine.levitt@gmail.com> writes:
>> That doesn't change much for my test cases, so I guess the DNS lookup is
>> indeed the bottleneck.

LMI> Have you verified that the DNS lookup is really the problem?  Because I
LMI> have more theories.  :-) (Or, as scientists call it, "guesses".)

LMI> One thing about the async image retrieval is that it's massively
LMI> parallel.  If there's 500 images in the article, Gnus will call
LMI> `url-retrieve' 500 times in rapid succession, which will then fire off
LMI> 500 HTTP connections (in parallel, more or less).

LMI> I suspect that Emacs doesn't really like having this number of async
LMI> socket sentinels going on at the same time, which means that the main
LMI> Emacs thread gets no CPU time, which gives us interactive pauses.

LMI> (And it's really rude towards the server.  It's basically a one-Emacs
LMI> DoS attack.)

LMI> So I'm going to rewrite this by adding a new library, called, er,
LMI> url-queue.el, which has the same interface as `url-retrieve', but which
LMI> manages the parallelism in a more sensible way, by not having more than
LMI> (for instance) four connections going at the same time.

It may be nice at this point to include libcurl in Emacs.

I would start by testing an article with 500 local images vs. one with
500 remote images.  That will eliminate the network variable.

Ted