Re: Asynchroneous image retrieval in HTML rendering

[David Engster <deng@randomsample.de>]

Antoine Levitt writes:
> 05/05/11 12:48, Ted Zlatanov
>> On Mon, 02 May 2011 19:40:43 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 
>>
>> LMI> The next thing to tackle is doing the DNS resolving asynchronously.
>
>> ...
>> LMI> If only somebody could implement an async C-level resolver.  :-)
>>
>> On Thu, 05 May 2011 10:25:15 +0200 Julien Danjou <julien@danjou.info> wrote: 
>>
>> JD>     http://www.corpit.ru/mjt/udns.html
>>
>> On Wed, 04 May 2011 20:40:58 +0200 Simon Josefsson <simon@josefsson.org> wrote: 
>>
>> SJ> http://www.chiark.greenend.org.uk/~ian/adns/
>>
>> Has anyone tested to see if DNS lookups are actually a problem?  Test an
>> entirely local page vs. a page with lots of remote image URLs.  I don't
>> see a difference on my machine.
>
> Maybe your connexion or DNS servers are better than mine, or some other
> parameter is different, but I definitely experienced freezes that were
> DNS-related. Lars also pointed out explicitely resolving the DNS in
> /etc/hosts sped things up immensely for gwene.com.wordpress.terrytao, so
> you could try this if you're not convinced by my admitedly
> non-repeatable arguments.

I highly doubt this has anything to do with DNS. Usually there's some
kind of cache involved and you shouldn't notice any delay after the
first request. Just run tcpdump and look at the traffic. I see exactly
one DNS request, but I still see a massive delay when retrieving
articles from the mentioned group.

I also see an insane amount of connections the Emacs url package is
opening. There seems to be no upper limit. Just use

(setq url-debug t)

and look in the *URL DEBUG* buffer, or use netstat to see the amount of
connections. It's more or less a little DoS on the server (I've seen
over 200 connections easily), and most servers are configured to
throttle stuff like this. You then have over 200 sentinels waiting to
parse the HTTP headers. No wonder this is slow. The right thing to do is
to open a moderate amount of connections (the max-connections-per-server
in Firefox is 15) and then reuse those.

> I don't know why, but DNS results don't seem to be cached. I don't know
> much about DNS queries so I'm not sure if that's supposed to be the case
> or not.

Usually your local server will cache them for you. Some OS also have
some kind of caching server running, which usually caches a bunch of
services at once (DNS, LDAP, etc.).

-David