[musl] DNS resolver fails prematurely when server reports failure?

[musl] DNS resolver fails prematurely when server reports failure?

[Mark Hills]

With multiple DNS servers in /etc/resolv.conf, the docs [1] are clear:

  "musl's resolver queries them all in parallel and accepts whichever 
   response arrives first."

So dual configuration is expected to give greater resiliancy:

  nameserver 213.186.33.99  # OVH
  nameserver 1.1.1.1        # Cloudflare

However, 1.1.1.1 appears quite prone to some kind of internal SERVFAIL 
(may be internal load shedding; though we are not making excessive DNS 
queries)

With glibc's cascading behaviour (or perhaps another OS) this may be dealt 
with by the client.

But if the wiki is read literally, the first response received is "this 
server has failed" then a good response from another server is ignored?

And indeed this seems to be the behaviour we experience, as removing 
1.1.1.1 restored reliability.

I tried to confirm this in the source [2] but found I'd need more time to 
understand this code.

Also, diagnosis was made more difficult by a colleage diligently following 
the resolv.conf(5) man page on the host (installed via man-pages on Alpine 
Linux) but this documents glibc. Perhaps musl could/should provide its 
own, but I expect there is a policy for this and similar issues.

Thanks

[1] https://wiki.musl-libc.org/functional-differences-from-glibc.html
[2] https://git.musl-libc.org/cgit/musl/tree/src/network/lookup_name.c#n296

-- 
Mark

Re: [musl] DNS resolver fails prematurely when server reports failure?

[Rich Felker]

On Wed, Dec 01, 2021 at 12:49:07PM +0000, Mark Hills wrote:
> With multiple DNS servers in /etc/resolv.conf, the docs [1] are clear:
> 
>   "musl's resolver queries them all in parallel and accepts whichever 
>    response arrives first."
> 
> So dual configuration is expected to give greater resiliancy:
> 
>   nameserver 213.186.33.99  # OVH
>   nameserver 1.1.1.1        # Cloudflare
> 
> However, 1.1.1.1 appears quite prone to some kind of internal SERVFAIL 
> (may be internal load shedding; though we are not making excessive DNS 
> queries)
> 
> With glibc's cascading behaviour (or perhaps another OS) this may be dealt 
> with by the client.
> 
> But if the wiki is read literally, the first response received is "this 
> server has failed" then a good response from another server is ignored?

No. ServFail is an inconclusive response, treated basically the same
as if no packet had arrived at all. (Slight difference: it triggers
immediate retry up to a limited number of times.)

> And indeed this seems to be the behaviour we experience, as removing 
> 1.1.1.1 restored reliability.

Have you looked at a packet capture of what's happening? Likely
1.1.1.1 was returning a false conclusive result (NxDomain or NODATA)
rather than ServFail.

Rich

Re: [musl] DNS resolver fails prematurely when server reports failure?

[Mark Hills]

On Wed, 1 Dec 2021, Rich Felker wrote:

> On Wed, Dec 01, 2021 at 12:49:07PM +0000, Mark Hills wrote:
> > With multiple DNS servers in /etc/resolv.conf, the docs [1] are clear:
> > 
> >   "musl's resolver queries them all in parallel and accepts whichever 
> >    response arrives first."
> > 
> > So dual configuration is expected to give greater resiliancy:
> > 
> >   nameserver 213.186.33.99  # OVH
> >   nameserver 1.1.1.1        # Cloudflare
> > 
> > However, 1.1.1.1 appears quite prone to some kind of internal SERVFAIL 
> > (may be internal load shedding; though we are not making excessive DNS 
> > queries)
> > 
> > With glibc's cascading behaviour (or perhaps another OS) this may be dealt 
> > with by the client.
> > 
> > But if the wiki is read literally, the first response received is "this 
> > server has failed" then a good response from another server is ignored?
> 
> No. ServFail is an inconclusive response, treated basically the same
> as if no packet had arrived at all. (Slight difference: it triggers
> immediate retry up to a limited number of times.)

Ok, thanks. That sounds correct, and I realise now that the real process 
of the query is in this source file [1] which is why the code looked so 
opaque.

Could it be better to make a small change to the wiki text? Perhaps 
"conclusive answer" instead of "response":

  accepts whichever conclusive answer arrives first

> > And indeed this seems to be the behaviour we experience, as removing 
> > 1.1.1.1 restored reliability.
> 
> Have you looked at a packet capture of what's happening? Likely
> 1.1.1.1 was returning a false conclusive result (NxDomain or NODATA)
> rather than ServFail.

We caught the problem with a tcpdump (which is first how we realised the 
differing behaviour between the man page and musl), and reproduced it with 
"dig", however it doesn't seem to be reproducable now. My recollection is 
that was an instant response and where I first encountered "ServFail" but 
I'll see if we have logged the actual run. I'm _fairly_ sure I'd have 
noticed a false but conclusive response.

I'm re-adding the "backup" DNS on a test system to see if we can get back 
to reproducing the problem.

[1] https://git.musl-libc.org/cgit/musl/tree/src/network/res_msend.c#n30

Thanks

-- 
Mark