resolv.conf ordering

resolv.conf ordering

[Jameel Al-Aziz]

[-- Attachment #1: Type: text/plain, Size: 1386 bytes --]

I'm sure this has been brought up before, but just thought I'd reach out
see if there's a solution.

I use musl on Alpine via Docker. I encountered issues today where DNS
wasn't resolving the way we expect in our images. I finally managed to
trace it down to musl's resolver (
http://wiki.musl-libc.org/wiki/Functional_differences_from_glibc#Name_Resolver_.2F_DNS
).

We configure resolv.conf with three DNS servers: Consul DNS, AWS VPC DNS,
Google DNS. It turns out that the AWS VPC DNS is the fastest to respond and
therefore causes results to fail even though they can be served via Consul
DNS. Putting aside that the musl resolver logic breaks convention (which
many people rely on), it seems that in this case it is more unpredictable
than simply following the order.

The host DNS is Consul, and while we could just setup Consul with
recursors, we run the risk of failing to resolve anything if Consul fails.
Setting up a local caching DNS is also overkill (we're in Docker
containers).

Is there no way to force musl to follow the order of nameservers in
resolv.conf? Or even if not, to allow musl to accept the first successful
response instead of failing on the first response? It seems to me that we
have to give up reliability for predictability, which is not what this
feature was intended to do from my understanding.

Any help on this matter would be greatly appreciated!

[-- Attachment #2: Type: text/html, Size: 1681 bytes --]

Re: resolv.conf ordering

[Rich Felker]

On Tue, Sep 15, 2015 at 03:25:20AM +0000, Jameel Al-Aziz wrote:
> I'm sure this has been brought up before, but just thought I'd reach out
> see if there's a solution.
> 
> I use musl on Alpine via Docker. I encountered issues today where DNS
> wasn't resolving the way we expect in our images. I finally managed to
> trace it down to musl's resolver (
> http://wiki.musl-libc.org/wiki/Functional_differences_from_glibc#Name_Resolver_.2F_DNS
> ).
> 
> We configure resolv.conf with three DNS servers: Consul DNS, AWS VPC DNS,
> Google DNS. It turns out that the AWS VPC DNS is the fastest to respond and
> therefore causes results to fail even though they can be served via Consul
> DNS. Putting aside that the musl resolver logic breaks convention (which
> many people rely on), it seems that in this case it is more unpredictable
> than simply following the order.
> 
> The host DNS is Consul, and while we could just setup Consul with
> recursors, we run the risk of failing to resolve anything if Consul fails.
> Setting up a local caching DNS is also overkill (we're in Docker
> containers).
> 
> Is there no way to force musl to follow the order of nameservers in
> resolv.conf? Or even if not, to allow musl to accept the first successful
> response instead of failing on the first response? It seems to me that we
> have to give up reliability for predictability, which is not what this
> feature was intended to do from my understanding.
> 
> Any help on this matter would be greatly appreciated!

Someone else raised this question on our IRC channel a week or two
ago, and in short, the answer is no. Basically this setup does not
make sense, even if you do have a resolver (glibc) that does do
ordered fallback:

- If you expect to sometimes need the second or third nameserver for
  queries the first cannot answer, then you're going to have terrible
  performance (multi-second delay before falling back to the second
  one).

- Unless all the nameservers agree on the records they're serving (in
  which case you wouldn't care about order), your query results will
  be unstable/inconsistent when the first server fails to respond. The
  typical result is that you will wrongly get NxDomain instead of a
  failed/timed-out query.

The second issue is really the motivation for what musl is doing: musl
is assuming that all the nameservers have consistent records, because
if they didn't, actual positive/negative results would be affected by
transient failures rather than transient failures being reported to
the calling program. This is a serious class of robustness (and
possibly security, since DoS can translate into false results)
failure.

If you really need to union inconsistent records from multiple
nameservers, the right way to do this is with a dns proxy/cache.
Assuming no _conflicting_ positive responses, it would need to do
something like forward positive responses as soon as it has at least
one positive response from upstream, but only forward negative
responses once it has a negative response from _all_ upstream sources.
Of course these are the constraints to do it "right"/robustly. If all
you want is something that works at least as well as glibc is working
for you now, dnsmasq is probably sufficient.

The conversation about all this on IRC was actually quite interesting.
We have a no-public-logging policy so there are not logs posted
anywhere, but if you're interested in more of what was discussed I
could try to summarize it or see if the people involved would be ok
with sharing a log excerpt.

Rich

Re: resolv.conf ordering

[Jameel Al-Aziz]

[-- Attachment #1: Type: text/plain, Size: 6420 bytes --]

Thanks for the response!

I would love to know more about the conversation on IRC.

I almost feel like there are valid arguments on both sides. In a
distributed environment, where machines and services come and go, it's
pretty difficult to guarantee consistent records both reliably and quickly.

While I was able to semi-solve my problem by enabling recursors through
Consul DNS, I realized that I have a chicken and egg problem. The caveat
here is this is particular to docker and some of the decisions they've made.

The basic issue is that I have some containers that need to be run with
"--net=host" and some that do not. In the "--net=host" containers
effectively copy over the host's resolv.conf. In order to make sure
everything can be resolved, I need to guarantee that Consul is setup as
early as possible. However, in the case that the setup process needs DNS,
you run into a problem. I could do some clever hackery to use the default
host DNS and overwrite the host's /etc/resolv.conf after setting up Consul
DNS, but that's not the greatest solution. This problem can also occur with
bridged-networking containers if you choose to specify the "dynamic" DNS
server as a default dns option to the docker daemon.

Put in more simple terms, we need normal DNS resolution while
bootstrapping, then as services register themselves, we need dependent
services to be able to look up the newly registered entries. Effectively,
the consistency is delayed at best.

The other issue here is that having recursion enabled just feels wrong and
insecure. Sure, this is all behind a VPC, but I like to err on the side of
caution.

I am probably wrong here, but it seems that the musl logic is only valid
when all nameservers are consistent. However, with dynamic service
registration, that consistency comes at the cost of speed.

The behavior we would ideally want is as you mentioned:
"Assuming no _conflicting_ positive responses, it would need to do
something like forward positive responses as soon as it has at least
one positive response from upstream, but only forward negative
responses once it has a negative response from _all_ upstream sources."

I'm almost certain we can accomplish what we want by having dnsmasq or some
other dns proxy/cache try Consul DNS first and then fallback upstream for
non-authoritative domains. The proxy has to be available very early on,
which is entirely doable in our scenario. However, it does add another
layer of indirection, which is just another potential failure point.

All that being said, I definitely understand why the decision was made,
just would be nice to have an option to enable the "robust" logic! :)

On Mon, Sep 14, 2015 at 9:43 PM Rich Felker <dalias@libc.org> wrote:

> On Tue, Sep 15, 2015 at 03:25:20AM +0000, Jameel Al-Aziz wrote:
> > I'm sure this has been brought up before, but just thought I'd reach out
> > see if there's a solution.
> >
> > I use musl on Alpine via Docker. I encountered issues today where DNS
> > wasn't resolving the way we expect in our images. I finally managed to
> > trace it down to musl's resolver (
> >
> http://wiki.musl-libc.org/wiki/Functional_differences_from_glibc#Name_Resolver_.2F_DNS
> > ).
> >
> > We configure resolv.conf with three DNS servers: Consul DNS, AWS VPC DNS,
> > Google DNS. It turns out that the AWS VPC DNS is the fastest to respond
> and
> > therefore causes results to fail even though they can be served via
> Consul
> > DNS. Putting aside that the musl resolver logic breaks convention (which
> > many people rely on), it seems that in this case it is more unpredictable
> > than simply following the order.
> >
> > The host DNS is Consul, and while we could just setup Consul with
> > recursors, we run the risk of failing to resolve anything if Consul
> fails.
> > Setting up a local caching DNS is also overkill (we're in Docker
> > containers).
> >
> > Is there no way to force musl to follow the order of nameservers in
> > resolv.conf? Or even if not, to allow musl to accept the first successful
> > response instead of failing on the first response? It seems to me that we
> > have to give up reliability for predictability, which is not what this
> > feature was intended to do from my understanding.
> >
> > Any help on this matter would be greatly appreciated!
>
> Someone else raised this question on our IRC channel a week or two
> ago, and in short, the answer is no. Basically this setup does not
> make sense, even if you do have a resolver (glibc) that does do
> ordered fallback:
>
> - If you expect to sometimes need the second or third nameserver for
>   queries the first cannot answer, then you're going to have terrible
>   performance (multi-second delay before falling back to the second
>   one).
>
> - Unless all the nameservers agree on the records they're serving (in
>   which case you wouldn't care about order), your query results will
>   be unstable/inconsistent when the first server fails to respond. The
>   typical result is that you will wrongly get NxDomain instead of a
>   failed/timed-out query.
>
> The second issue is really the motivation for what musl is doing: musl
> is assuming that all the nameservers have consistent records, because
> if they didn't, actual positive/negative results would be affected by
> transient failures rather than transient failures being reported to
> the calling program. This is a serious class of robustness (and
> possibly security, since DoS can translate into false results)
> failure.
>
> If you really need to union inconsistent records from multiple
> nameservers, the right way to do this is with a dns proxy/cache.
> Assuming no _conflicting_ positive responses, it would need to do
> something like forward positive responses as soon as it has at least
> one positive response from upstream, but only forward negative
> responses once it has a negative response from _all_ upstream sources.
> Of course these are the constraints to do it "right"/robustly. If all
> you want is something that works at least as well as glibc is working
> for you now, dnsmasq is probably sufficient.
>
> The conversation about all this on IRC was actually quite interesting.
> We have a no-public-logging policy so there are not logs posted
> anywhere, but if you're interested in more of what was discussed I
> could try to summarize it or see if the people involved would be ok
> with sharing a log excerpt.
>
> Rich
>

[-- Attachment #2: Type: text/html, Size: 8175 bytes --]

Re: resolv.conf ordering

[Andy Shinn]

[-- Attachment #1: Type: text/plain, Size: 7329 bytes --]

Hi Jameel,

Also on the same subject (since you have specifically pointed out Consul)
is a thread I started at http://www.openwall.com/lists/musl/2015/09/04/3
which may interest you. I actually maintain the official Docker Alpine
Linux image at https://github.com/gliderlabs/docker-alpine and there is a
similar thread we are tracking information at
https://github.com/gliderlabs/docker-alpine/issues/8.

I'd also be interested in the conversation that took place on IRC (I'm in
the channel but must have missed it). Rich, are you able to give dates /
times that I might be able to go back in my own IRC client to check out
what was discussed?

-Andy

On Tue, Sep 15, 2015 at 5:56 AM, Jameel Al-Aziz <me@jalaziz.net> wrote:

> Thanks for the response!
>
> I would love to know more about the conversation on IRC.
>
> I almost feel like there are valid arguments on both sides. In a
> distributed environment, where machines and services come and go, it's
> pretty difficult to guarantee consistent records both reliably and quickly.
>
> While I was able to semi-solve my problem by enabling recursors through
> Consul DNS, I realized that I have a chicken and egg problem. The caveat
> here is this is particular to docker and some of the decisions they've made.
>
> The basic issue is that I have some containers that need to be run with
> "--net=host" and some that do not. In the "--net=host" containers
> effectively copy over the host's resolv.conf. In order to make sure
> everything can be resolved, I need to guarantee that Consul is setup as
> early as possible. However, in the case that the setup process needs DNS,
> you run into a problem. I could do some clever hackery to use the default
> host DNS and overwrite the host's /etc/resolv.conf after setting up Consul
> DNS, but that's not the greatest solution. This problem can also occur with
> bridged-networking containers if you choose to specify the "dynamic" DNS
> server as a default dns option to the docker daemon.
>
> Put in more simple terms, we need normal DNS resolution while
> bootstrapping, then as services register themselves, we need dependent
> services to be able to look up the newly registered entries. Effectively,
> the consistency is delayed at best.
>
> The other issue here is that having recursion enabled just feels wrong and
> insecure. Sure, this is all behind a VPC, but I like to err on the side of
> caution.
>
> I am probably wrong here, but it seems that the musl logic is only valid
> when all nameservers are consistent. However, with dynamic service
> registration, that consistency comes at the cost of speed.
>
> The behavior we would ideally want is as you mentioned:
> "Assuming no _conflicting_ positive responses, it would need to do
> something like forward positive responses as soon as it has at least
> one positive response from upstream, but only forward negative
> responses once it has a negative response from _all_ upstream sources."
>
> I'm almost certain we can accomplish what we want by having dnsmasq or
> some other dns proxy/cache try Consul DNS first and then fallback upstream
> for non-authoritative domains. The proxy has to be available very early on,
> which is entirely doable in our scenario. However, it does add another
> layer of indirection, which is just another potential failure point.
>
> All that being said, I definitely understand why the decision was made,
> just would be nice to have an option to enable the "robust" logic! :)
>
> On Mon, Sep 14, 2015 at 9:43 PM Rich Felker <dalias@libc.org> wrote:
>
>> On Tue, Sep 15, 2015 at 03:25:20AM +0000, Jameel Al-Aziz wrote:
>> > I'm sure this has been brought up before, but just thought I'd reach out
>> > see if there's a solution.
>> >
>> > I use musl on Alpine via Docker. I encountered issues today where DNS
>> > wasn't resolving the way we expect in our images. I finally managed to
>> > trace it down to musl's resolver (
>> >
>> http://wiki.musl-libc.org/wiki/Functional_differences_from_glibc#Name_Resolver_.2F_DNS
>> > ).
>> >
>> > We configure resolv.conf with three DNS servers: Consul DNS, AWS VPC
>> DNS,
>> > Google DNS. It turns out that the AWS VPC DNS is the fastest to respond
>> and
>> > therefore causes results to fail even though they can be served via
>> Consul
>> > DNS. Putting aside that the musl resolver logic breaks convention (which
>> > many people rely on), it seems that in this case it is more
>> unpredictable
>> > than simply following the order.
>> >
>> > The host DNS is Consul, and while we could just setup Consul with
>> > recursors, we run the risk of failing to resolve anything if Consul
>> fails.
>> > Setting up a local caching DNS is also overkill (we're in Docker
>> > containers).
>> >
>> > Is there no way to force musl to follow the order of nameservers in
>> > resolv.conf? Or even if not, to allow musl to accept the first
>> successful
>> > response instead of failing on the first response? It seems to me that
>> we
>> > have to give up reliability for predictability, which is not what this
>> > feature was intended to do from my understanding.
>> >
>> > Any help on this matter would be greatly appreciated!
>>
>> Someone else raised this question on our IRC channel a week or two
>> ago, and in short, the answer is no. Basically this setup does not
>> make sense, even if you do have a resolver (glibc) that does do
>> ordered fallback:
>>
>> - If you expect to sometimes need the second or third nameserver for
>>   queries the first cannot answer, then you're going to have terrible
>>   performance (multi-second delay before falling back to the second
>>   one).
>>
>> - Unless all the nameservers agree on the records they're serving (in
>>   which case you wouldn't care about order), your query results will
>>   be unstable/inconsistent when the first server fails to respond. The
>>   typical result is that you will wrongly get NxDomain instead of a
>>   failed/timed-out query.
>>
>> The second issue is really the motivation for what musl is doing: musl
>> is assuming that all the nameservers have consistent records, because
>> if they didn't, actual positive/negative results would be affected by
>> transient failures rather than transient failures being reported to
>> the calling program. This is a serious class of robustness (and
>> possibly security, since DoS can translate into false results)
>> failure.
>>
>> If you really need to union inconsistent records from multiple
>> nameservers, the right way to do this is with a dns proxy/cache.
>> Assuming no _conflicting_ positive responses, it would need to do
>> something like forward positive responses as soon as it has at least
>> one positive response from upstream, but only forward negative
>> responses once it has a negative response from _all_ upstream sources.
>> Of course these are the constraints to do it "right"/robustly. If all
>> you want is something that works at least as well as glibc is working
>> for you now, dnsmasq is probably sufficient.
>>
>> The conversation about all this on IRC was actually quite interesting.
>> We have a no-public-logging policy so there are not logs posted
>> anywhere, but if you're interested in more of what was discussed I
>> could try to summarize it or see if the people involved would be ok
>> with sharing a log excerpt.
>>
>> Rich
>>
>

[-- Attachment #2: Type: text/html, Size: 9640 bytes --]

Re: resolv.conf ordering

[Szabolcs Nagy]

* Andy Shinn <andys@andyshinn.as> [2015-09-15 09:23:44 -0700]:
> I'd also be interested in the conversation that took place on IRC (I'm in
> the channel but must have missed it). Rich, are you able to give dates /
> times that I might be able to go back in my own IRC client to check out
> what was discussed?

i think it was between
2015-08-28 22:40 UTC
and
2015-08-28 23:10 UTC

Re: resolv.conf ordering

[Rich Felker]

On Tue, Sep 15, 2015 at 12:56:41PM +0000, Jameel Al-Aziz wrote:
> Thanks for the response!
> 
> I would love to know more about the conversation on IRC.
> 
> I almost feel like there are valid arguments on both sides. In a
> distributed environment, where machines and services come and go, it's
> pretty difficult to guarantee consistent records both reliably and quickly.
> 
> While I was able to semi-solve my problem by enabling recursors through
> Consul DNS, I realized that I have a chicken and egg problem. The caveat
> here is this is particular to docker and some of the decisions they've made.
> 
> The basic issue is that I have some containers that need to be run with
> "--net=host" and some that do not. In the "--net=host" containers
> effectively copy over the host's resolv.conf. In order to make sure
> everything can be resolved, I need to guarantee that Consul is setup as
> early as possible. However, in the case that the setup process needs DNS,
> you run into a problem. I could do some clever hackery to use the default
> host DNS and overwrite the host's /etc/resolv.conf after setting up Consul
> DNS, but that's not the greatest solution. This problem can also occur with
> bridged-networking containers if you choose to specify the "dynamic" DNS
> server as a default dns option to the docker daemon.
> 
> Put in more simple terms, we need normal DNS resolution while
> bootstrapping, then as services register themselves, we need dependent
> services to be able to look up the newly registered entries. Effectively,
> the consistency is delayed at best.

Thanks for explaining.

> The other issue here is that having recursion enabled just feels wrong and
> insecure. Sure, this is all behind a VPC, but I like to err on the side of
> caution.

It doesn't need to be recursion (generally recursion is slow unless
you have a huge cache), just forwarding/caching. This should not be a
serious security risk as long as you restrict access to localhost (or
even more fine-grained via iptables marking of local packet origins)
and properly privilege-isolate the proxy nameserver daemon.

> I am probably wrong here, but it seems that the musl logic is only valid
> when all nameservers are consistent. However, with dynamic service
> registration, that consistency comes at the cost of speed.
> 
> The behavior we would ideally want is as you mentioned:
> "Assuming no _conflicting_ positive responses, it would need to do
> something like forward positive responses as soon as it has at least
> one positive response from upstream, but only forward negative
> responses once it has a negative response from _all_ upstream sources."
> 
> I'm almost certain we can accomplish what we want by having dnsmasq or some
> other dns proxy/cache try Consul DNS first and then fallback upstream for
> non-authoritative domains. The proxy has to be available very early on,
> which is entirely doable in our scenario. However, it does add another
> layer of indirection, which is just another potential failure point.
> 
> All that being said, I definitely understand why the decision was made,
> just would be nice to have an option to enable the "robust" logic! :)

In theory it might be possible to give the behavior you want,
_without_ resorting to serialized lookups, by having an option to
defer acceptance of negative answers from nameserver #N until negative
results or refusal-to-answer replies have also been obtained from
nameservers 0...N-1. I'm not sure how complicated the logic for this
would be, and it would need an option in resolv.conf since making it
default would penalize "normal" use of multiple nameservers for mere
redundancy/performance, but it seems like an option we could explore.

Still however this would give much worse performance if a server is
down, compared to just having a proper caching nameserver on
localhost.

Rich