Re: Re: Would love to see reconsideration for domain and search

[Tim Hockin <thockin@google.com>]

On Thu, Oct 22, 2015 at 9:27 PM, Rich Felker <dalias@libc.org> wrote:
> On Thu, Oct 22, 2015 at 04:37:18PM -0700, Tim Hockin wrote:
>> On Thu, Oct 22, 2015 at 4:00 PM, Josiah Worcester <josiahw@gmail.com> wrote:
>> > On Thu, Oct 22, 2015 at 3:37 PM Tim Hockin <thockin@google.com> wrote:
>> >>
>> >> On Thu, Oct 22, 2015 at 2:56 PM, Rich Felker <dalias@libc.org> wrote:
>> >> > On Thu, Oct 22, 2015 at 02:24:11PM -0700, Tim Hockin wrote:
>> >> >> Hi all,
>> >> >>
>> >> >> I saw this thread on the web archive but am not sure how to respond to
>> >> >> the thread directly as a new joinee of the ML.  I hope this finds its
>> >> >> way...
>> >> >
>> >> > No problem; just starting a new thread like this and quoting the old
>> >> > one is fine.
>> >> >
>> >> >> I am one of the developers of Kubernetes and I own the DNS portion, in
>> >> >> particular.  I desperately want to use Alpine Linux (based on musl)
>> >> >> but for now I have to warn people NOT to use it because of this issue.
>> >> >>
>> >> >> On Fri, Sep 04, 2015 at 02:04:29PM -0400, Rich Felker wrote:
>> >> >> > On Fri, Sep 04, 2015 at 12:11:36PM -0500, Andy Shinn wrote:
>> >> >> >> I'm writing the wonderful musl project today to open discussion
>> >> >> >> about the future possibility of DNS search and domain keyword
>> >> >> >> support. We've been using musl libc (by way of Alpine Linux) for
>> >> >> >> new development of applications as containers that discover each
>> >> >> >> other through DNS and other software defined networking.
>> >> >> >>
>> >> >> >> In particular, we are starting to use applications like SkyDNS,
>> >> >> >> Consul, and Kubernetes, all of which rely on local name
>> >> >> >> resolution in some way using search paths. Many users of the
>> >> >> >> Alpine Linux container image have also expressed their desire for
>> >> >> >> this feature at
>> >> >> >> https://github.com/gliderlabs/docker-alpine/issues/8.
>> >> >> >>
>> >> >> >> On the functional differences between glibc page, the domain and
>> >> >> >> search keyword "Support may be added in the future if there is
>> >> >> >> demand". So please consider this request an addition to whatever
>> >> >> >> demand for the feature already exists.
>> >> >> >>
>> >> >> >> Thank you for your time and great work on the musl libc project!
>> >> >> >
>> >> >> > I think this is a reasonable request. I'll look into it more.
>> >> >> >
>> >> >> > One property I do not want to break is deterministic results, so
>> >> >> > when a search is performed, if any step of the search ends with
>> >> >> > an error rather than a positive or negative result, the whole
>> >> >> > lookup needs to stop and report the error rather than falling
>> >> >> > back. Falling back is not safe and creates a situation where DoS
>> >> >> > can be used to control which results are returned.
>> >> >>
>> >> >> I understand your point, though the world at large tends to disagree.
>> >> >> Everyone has a primary and secondary `nameserver` record (or should).
>> >> >> If the first one times out, try the second.  Most resolver libs seem
>> >> >> to accept a SERVFAIL response or a timeout as a signal to try the next
>> >> >> server, and I would encourage you to do the same.
>> >> >
>> >> > musl intentionally does not do this because it yields abysmal
>> >> > performance. If the first nameserver is overloaded or the packet is
>> >> > lost, you suffer several-second lookup latency.
>> >>
>> >> But at least it works eventually.  You're faced with a choice.  Wait 2
>> >> seconds for ns1 to timeout and then fail in a way that most apps don't
>> >> handle well or wait for 2 seconds and then (usually) get a fast
>> >> response from ns2.
>> >>
>> >> It seems better in every way to eventually succeed, though I agree
>> >> it's a bit less visible.
>> >
>> >
>> > With musl's current design, you get a request to ns1 and ns2, and the first
>> > authoritative response wins. So, if ns1 fails then all is well and
>> > performance isn't even notably impacted. What you are describing appears to
>> > be how you would *have* to implement it if you decide against considering
>> > all servers equal, but instead try and serve the union of their responses
>> > (that is, wait for timeout and then fail).
>>
>> The authoritative-ness is a dimension I had not considered.  I could
>> believe that the first authoritative answer wins, but what if you only
>> get a non-authoritative answer? from ns1 and ns2 never responds?
>
> I don't think Josiah's use of "authoritative" here is consistent with
> what musl actually does, and it's likely not a useful dimension. A
> stub resolver is generally not intended to communicate with
> authoritative nameservers. The current behavior is that a positive or
> negative (nxdomain) result is treated as concluding the query, and
> other results (servfail, refused, etc.) are treated as inconclusive
> and allow the query to continue until it times out. (Also, servfail
> results in a limited number of immediate retries; this behavior was
> arrived at based on the behavior of nameservers that fail with
> servfail.)
>
>> > Consider what would happen if ns1 and ns2 have different responses, but ns1
>> > for whatever reason times out (potentially an attacker). Then you get the
>> > results for ns2, even though ns1 is intended to override it.
>>
>> I agree in theory.  And yet this is how most resolvers work today.
>> Are they all broken?
>
> No, the resolvers are not broken. The configuration is, at least the
> way I see it. The intent of the resolvers is that they're
> communicating with redundant servers, not a sequence of overlaid
> hostname databases. If that assumption is satisfied, there's no
> problem.

The way you see it is not how everyone sees it, obviously.  And
there's not a standard or spec here that I know of.  That said, as
before, We can probably live with this assumption.

> BTW I think there are other strong reasons to move to a model based on
> a local nameserver that does the unioning, not just performance. The
> most compelling is DNSSEC, which requires a trusted channel between
> the nameserver and the stub resolver in order for results to be
> meaningful/trusted. In the future everybody should be running a
> nameserver on localhost to do DNSSEC signature validation. In that
> scheme, resolv.conf would just contain 127.0.0.1 (or could be omitted
> entirely since that's the default, at least on musl).

I can see a local nameserver doing resolution, but doing search
expansion seems like a stretch (and superfluous since it is local).