Re: resolv.conf ordering

[Rich Felker <dalias@libc.org>]

On Tue, Sep 15, 2015 at 12:56:41PM +0000, Jameel Al-Aziz wrote:
> Thanks for the response!
> 
> I would love to know more about the conversation on IRC.
> 
> I almost feel like there are valid arguments on both sides. In a
> distributed environment, where machines and services come and go, it's
> pretty difficult to guarantee consistent records both reliably and quickly.
> 
> While I was able to semi-solve my problem by enabling recursors through
> Consul DNS, I realized that I have a chicken and egg problem. The caveat
> here is this is particular to docker and some of the decisions they've made.
> 
> The basic issue is that I have some containers that need to be run with
> "--net=host" and some that do not. In the "--net=host" containers
> effectively copy over the host's resolv.conf. In order to make sure
> everything can be resolved, I need to guarantee that Consul is setup as
> early as possible. However, in the case that the setup process needs DNS,
> you run into a problem. I could do some clever hackery to use the default
> host DNS and overwrite the host's /etc/resolv.conf after setting up Consul
> DNS, but that's not the greatest solution. This problem can also occur with
> bridged-networking containers if you choose to specify the "dynamic" DNS
> server as a default dns option to the docker daemon.
> 
> Put in more simple terms, we need normal DNS resolution while
> bootstrapping, then as services register themselves, we need dependent
> services to be able to look up the newly registered entries. Effectively,
> the consistency is delayed at best.

Thanks for explaining.

> The other issue here is that having recursion enabled just feels wrong and
> insecure. Sure, this is all behind a VPC, but I like to err on the side of
> caution.

It doesn't need to be recursion (generally recursion is slow unless
you have a huge cache), just forwarding/caching. This should not be a
serious security risk as long as you restrict access to localhost (or
even more fine-grained via iptables marking of local packet origins)
and properly privilege-isolate the proxy nameserver daemon.

> I am probably wrong here, but it seems that the musl logic is only valid
> when all nameservers are consistent. However, with dynamic service
> registration, that consistency comes at the cost of speed.
> 
> The behavior we would ideally want is as you mentioned:
> "Assuming no _conflicting_ positive responses, it would need to do
> something like forward positive responses as soon as it has at least
> one positive response from upstream, but only forward negative
> responses once it has a negative response from _all_ upstream sources."
> 
> I'm almost certain we can accomplish what we want by having dnsmasq or some
> other dns proxy/cache try Consul DNS first and then fallback upstream for
> non-authoritative domains. The proxy has to be available very early on,
> which is entirely doable in our scenario. However, it does add another
> layer of indirection, which is just another potential failure point.
> 
> All that being said, I definitely understand why the decision was made,
> just would be nice to have an option to enable the "robust" logic! :)

In theory it might be possible to give the behavior you want,
_without_ resorting to serialized lookups, by having an option to
defer acceptance of negative answers from nameserver #N until negative
results or refusal-to-answer replies have also been obtained from
nameservers 0...N-1. I'm not sure how complicated the logic for this
would be, and it would need an option in resolv.conf since making it
default would penalize "normal" use of multiple nameservers for mere
redundancy/performance, but it seems like an option we could explore.

Still however this would give much worse performance if a server is
down, compared to just having a proper caching nameserver on
localhost.

Rich