[9fans] small dns improvements

[erik quanstrom <quanstro@quanstro.net>]

it must be that time of year.  dns is driving folks bats.  :-)

i've been spending some time looking at why ndb/dns fails.  as is well known,
there are very long-standing locking problems.  in the past, i've gotten hung up on
those and not made any progress.  while imho, the long-term strategy should be
to replace ndb/dns with an easier-to-maintain structure, i only have a few weeks
to fix as much as possible.  so i decided to see if there were simple things we could
do to improve things.

geoff has made a few big improvements.  some sites which were broken for a long
time are now working.  tomshardware.com is one that i've used as a test, and it
finally works.  (although the results don't seem worth the effort.  ☺)

but there are a number of other lookups that are still broken for me, and it
there seem to be some straightforward reasons that i think i've fixed:

1.  we're sending the RD (recursion desired) bit when we ourselves are acting as
a recursive server.  this looks okay by the standard, but many servers return Srvfail
(code 2, Rserver in the dns code) rather than ignoring this bit.  turning this off
helps alot (example: ocsp.netsolssl.com).

2.  we're ignoring status codes that we should be treating as fatal (like Srvfail)

3.  we're not using edns0.  this is kind of a sticky bit.  some servers insist on sending
enormous answers but don't answer via tcp.  on the other hand, some servers insist
on sending enormous answers, but return nasty errors when given edns0 queries.
what seems to work best is to send udp/no edns0, udp/edns0 and finally tcp.

4.  we get confused attaching the name servers to an answer for an out-of-baliwick
cname record.  (this is largely a problem with logging, but has the potential to
corrupt the database.)

if anyone would like to try a 386 executable (amd64 available on request),
i've put a copy at
	http://ftp.quanstro.net/other/^(dns dnsdebug)

i'd be happy to hear of any dns lookup problems.  please let me know
which version of dns you're using.

thanks,

- erik