dnscache endless resolving loop

dnscache endless resolving loop

[Alex Efros]

[-- Attachment #1: Type: text/plain, Size: 2113 bytes --]

Hi!

Probably this isn't a right maillist to ask, but I think this will be
interesting to people here.

On one of my server I permanently have issue with dnscache, which looks
like this:

  PID  PPID    TIME+  %CPU %MEM  PR  NI S  VIRT SWAP  RES  UID COMMAND         
 3301   507  49:42.59 16.6  0.4  15   0 R  4148 2200 1948 1000 dnscache         
11519   507 602:44.67  3.3  0.2  15   0 R  2860 2052  808 1001 svlogd           
15610 26870   0:00.20  0.3  0.3  15   0 R  3588 1884 1704    0 top              

i.e. dnscache permanently use ~15% CPU (TIME+ column show less time for
dnscache than for it svlogd just because I restarted dnscache some time
ago). This server isn't very powerful (Celeron 2GHz, 512RAM), so this
issue slowdown server significantly. Of course I can disable svlogd for
dnscache (because writing huge amount of logd to disk also slowdown it),
but I need logs to find out what's wrong.

Restarting dnscache solve this issue for about a hour, and then it arise
again (probably it initiated by some spammer who connect to my qmail from
this IP, and tcpserver trying to resolve it).

According to logs, dnscache again and again trying to resolve PTR
124.240.91.30, and every time got negative result (which it doesn't cache,
AFAIK).

I spend enough time trying to find out WHO is sending these PTR requests
to dnscache (is anybody knows how to find out process which own UDP port?
things like netstat can't answer to question "who is asking my DNS
server")... dnscache run on 127.0.0.1 so it isn't remote attack.

After all I got an Idea: maybe it's dnscache is the one who asking
dnscache? :) I did strace, and, yes, it looks like dnscache send PTR
requests to itself in endless loop. Look:

http://powerman.name/tmp/dnscache_bug/log (1MB)
http://powerman.name/tmp/dnscache_bug/strace (6.5MB)
http://powerman.name/tmp/dnscache_bug/strace.filtered (2MB, only network I/O)

About djbdns patches. I use Gentoo's net-dns/djbdns-1.05-r21 package.
It apply several patches, but they all looks 100% harmless, and no one
touch C code (except errno problem) - I'll attach them.

-- 
			WBR, Alex.

[-- Attachment #2: 1.05-errno.patch --]
[-- Type: application/x-patch, Size: 249 bytes --]

[-- Attachment #3: dnsroots.patch --]
[-- Type: application/x-patch, Size: 367 bytes --]

[-- Attachment #4: dnstracesort.patch --]
[-- Type: application/x-patch, Size: 338 bytes --]

[-- Attachment #5: headtail.patch --]
[-- Type: application/x-patch, Size: 1848 bytes --]

Re: dnscache endless resolving loop

[Jose Celestino]

Words by Alex Efros [Thu, Feb 28, 2008 at 05:22:54PM +0200]:
> Hi!
> 
> Probably this isn't a right maillist to ask, but I think this will be
> interesting to people here.
> 

No, the right list would be dns@list.cr.yp.to.

> On one of my server I permanently have issue with dnscache, which looks
> like this:
> 
>   PID  PPID    TIME+  %CPU %MEM  PR  NI S  VIRT SWAP  RES  UID COMMAND         
>  3301   507  49:42.59 16.6  0.4  15   0 R  4148 2200 1948 1000 dnscache         
> 11519   507 602:44.67  3.3  0.2  15   0 R  2860 2052  808 1001 svlogd           
> 15610 26870   0:00.20  0.3  0.3  15   0 R  3588 1884 1704    0 top              
> 
> i.e. dnscache permanently use ~15% CPU (TIME+ column show less time for
> dnscache than for it svlogd just because I restarted dnscache some time
> ago). This server isn't very powerful (Celeron 2GHz, 512RAM), so this
> issue slowdown server significantly. Of course I can disable svlogd for
> dnscache (because writing huge amount of logd to disk also slowdown it),
> but I need logs to find out what's wrong.
>

Yes, "/dev/null"ing the logs is a good idea for extremely busy servers.

> 
> Restarting dnscache solve this issue for about a hour, and then it arise
> again (probably it initiated by some spammer who connect to my qmail from
> this IP, and tcpserver trying to resolve it).
> 

You have the ip that does the dns queries on the logs:

2008-02-28_14:28:42.67086 query 1260583 7f000001:a399:3ad9 12 30.91.240.124.in-addr.arpa.

7f000001 is the ip (hex)

>
> According to logs, dnscache again and again trying to resolve PTR
> 124.240.91.30, and every time got negative result (which it doesn't cache,
> AFAIK).
> 

It didn't got a negative result, it got a timeout (and that's why it
didn't cache it).

>
> I spend enough time trying to find out WHO is sending these PTR requests
> to dnscache (is anybody knows how to find out process which own UDP port?
> things like netstat can't answer to question "who is asking my DNS
> server")... dnscache run on 127.0.0.1 so it isn't remote attack.
> 
> After all I got an Idea: maybe it's dnscache is the one who asking
> dnscache? :) I did strace, and, yes, it looks like dnscache send PTR
> requests to itself in endless loop. Look:
> 

Yes. Because the authoritative for 91.240.124.in-addr.arpa. is
remove.this.nserver.to.enable.zone.at.apnic.net and this is 127.0.0.1:

$ dnsqr ns 91.240.124.in-addr.arpa.
2 91.240.124.in-addr.arpa:
118 bytes, 1+1+0+1 records, response, noerror
query: 2 91.240.124.in-addr.arpa
answer: 91.240.124.in-addr.arpa 80944 NS remove.this.nserver.to.enable.zone.at.apnic.net
additional: remove.this.nserver.to.enable.zone.at.apnic.net 1871 A 127.0.0.1

Anyway, send any further questions to the above mentioned list, this is OT
here.

-- 
Jose Celestino
----------------------------------------------------------------
http://www.msversus.org/     ; http://techp.org/petition/show/1
http://www.vinc17.org/noswpat.en.html
----------------------------------------------------------------
"If you would have your slaves remain docile, teach them hymns."
    -- Ed Weathers ("The Empty Box")

Re: dnscache endless resolving loop

[Charlie Brady]

On Thu, 28 Feb 2008, Alex Efros wrote:

> server")... dnscache run on 127.0.0.1 so it isn't remote attack.
>
> After all I got an Idea: maybe it's dnscache is the one who asking
> dnscache? :) I did strace, and, yes, it looks like dnscache send PTR
> requests to itself in endless loop.

Move your dnscache to, say, 127.0.0.9, and you will escape this loop.
Or add an override delegation for that reverse zone, in ./root/servers.

Re: dnscache endless resolving loop

[Alex Efros]

Hi!

On Thu, Feb 28, 2008 at 11:16:45AM -0500, Charlie Brady wrote:
>> server")... dnscache run on 127.0.0.1 so it isn't remote attack.
>>
>> After all I got an Idea: maybe it's dnscache is the one who asking
>> dnscache? :) I did strace, and, yes, it looks like dnscache send PTR
>> requests to itself in endless loop.
>
> Move your dnscache to, say, 127.0.0.9, and you will escape this loop.
> Or add an override delegation for that reverse zone, in ./root/servers.

... and repeat this each time something like this happens? :(

Actually, all of this in fact is remote DoS attack (but not intentional,
of course) to dnscache running on 127.0.0.1. WOW! :)

-- 
			WBR, Alex.

Re: dnscache endless resolving loop

[Alex Efros]

Hi!

On Thu, Feb 28, 2008 at 03:38:21PM +0000, Jose Celestino wrote:
> No, the right list would be dns@list.cr.yp.to.
...
> Anyway, send any further questions to the above mentioned list, this is OT
> here.

Ohh, my intuition was right when suggest me to avoid subscribing to that
maillist. After I subscribe, and send my question, I've received 7 (!)
emails from Amazon, 1 from PayPal and 1 from djb's qsecretary - all from
automatic autoresponders and all 'regarding my email to them' (looks like
Amazon & PayPal subscribed their autoresponders to this maillist!).
And while I can understand needs of qsecretary, all others are just SPAM.
And I afraid more spam is on it way to me... 'regarding my email'... shit.

-- 
			WBR, Alex.