Re: Would love to see reconsideration for domain and search

Re: Would love to see reconsideration for domain and search

[Tim Hockin]

Hi all,

I saw this thread on the web archive but am not sure how to respond to
the thread directly as a new joinee of the ML.  I hope this finds its
way...

I am one of the developers of Kubernetes and I own the DNS portion, in
particular.  I desperately want to use Alpine Linux (based on musl)
but for now I have to warn people NOT to use it because of this issue.

On Fri, Sep 04, 2015 at 02:04:29PM -0400, Rich Felker wrote:
> On Fri, Sep 04, 2015 at 12:11:36PM -0500, Andy Shinn wrote:
>> I'm writing the wonderful musl project today to open discussion
>> about the future possibility of DNS search and domain keyword
>> support. We've been using musl libc (by way of Alpine Linux) for
>> new development of applications as containers that discover each
>> other through DNS and other software defined networking.
>>
>> In particular, we are starting to use applications like SkyDNS,
>> Consul, and Kubernetes, all of which rely on local name
>> resolution in some way using search paths. Many users of the
>> Alpine Linux container image have also expressed their desire for
>> this feature at
>> https://github.com/gliderlabs/docker-alpine/issues/8.
>>
>> On the functional differences between glibc page, the domain and
>> search keyword "Support may be added in the future if there is
>> demand". So please consider this request an addition to whatever
>> demand for the feature already exists.
>>
>> Thank you for your time and great work on the musl libc project!
>
> I think this is a reasonable request. I'll look into it more.
>
> One property I do not want to break is deterministic results, so
> when a search is performed, if any step of the search ends with
> an error rather than a positive or negative result, the whole
> lookup needs to stop and report the error rather than falling
> back. Falling back is not safe and creates a situation where DoS
> can be used to control which results are returned.

I understand your point, though the world at large tends to disagree.
Everyone has a primary and secondary `nameserver` record (or should).
If the first one times out, try the second.  Most resolver libs seem
to accept a SERVFAIL response or a timeout as a signal to try the next
server, and I would encourage you to do the same.

Stopping on positive response or NXDOMAIN seems to be commonly
accepted with a caveat.  You can't query all nameservers and just take
the first NXDOMAIN to respond.  You can only accept NXDOMAIN if all of
the higher-priority (listed first in resolv.conf) nameservers have
timed out or SERVFAIL'ed.  You can issue queries in parallel, but you
must process responses in order, which is what you describe below.

> While it would be possible to parallelize the search while
> serializing the results (i.e. waiting to accept a result from the
> second query until the first query finished with a negative
> result), I think the consensus during the last round of
> discussion of this topic was that the complexity cost is too
> great and the benefit too small. Ideally, the first query should
> always succeed, anyway.

The real world is not ideal.  Not all nameservers are identically
scoped - you MUST respect the ordering in resolv.conf - to do
otherwise is semantically broken.  If implementation simplicity means
literally doing queries in serial, then that is what you should do.

Similarly, you can't just search all search domains in parallel and
take the first response.  The ordering is meaningful.

> I also have a few questions:
>
> 1. Do you need multiple search items, or just a single domain?
>    Any setup with multiple searches necessarily has suboptimal
>    performance because ndots is not sufficient to make the right
>    initial choice of query. If you do need this functionality, a
>    unioning proxy dns server may be a better option than resolv.conf
>    domain search; it would give much better performance.

We use multiple search paths and ndots > 1.  I'm not sure what you
mean by "unioning" here.  Search path ordering is as meaningful as
nameserver ordering.  You can't avoid making the query for each search
suffix in the worst case, and it has the same restriction as
nameserver - the search order must be respected.

There does seem to be some different implementations that search for
the "naked" query first vs last, though.  I think the semantically
correct (but pessimal performance) is to search for that last.

> 2. For your intended applications, is there a need to support
>    ndots>1?  Such configurations are generally not friendly to
>    applications that expect to be able to resolve normal internet
>    domain lookups, and performance for such lookups will be very bad
>    (because the search domains first have to fail).

DNS is a very lightweight protocol.  We have not measured any
practical detriment for having 6 search domains and ndots=5.  In the
normal case it fails very quickly.  That aside, it should be my
business if I want to (mis)configure my system that way :)

> 3. The glibc behavior is just to swap the order of search when
>    the query string has >=ndots dots in it, but would it be
>    acceptable never to try the search domains at all in this case?
>    That would yield much better performance for nxdomain results and
>    avoid unexpected positive results due to weird subdomains
>    existing in your search domain (e.g. a wildcard for
>    *.us.example.com would cause *.us to wrongly resolve for
>    non-existant .us domains).

I think that would be correct.  If I have 3 dots and ndots=2, search
paths should be ignored.

Tim

Re: Re: Would love to see reconsideration for domain and search

[Rich Felker]

On Thu, Oct 22, 2015 at 02:24:11PM -0700, Tim Hockin wrote:
> Hi all,
> 
> I saw this thread on the web archive but am not sure how to respond to
> the thread directly as a new joinee of the ML.  I hope this finds its
> way...

No problem; just starting a new thread like this and quoting the old
one is fine.

> I am one of the developers of Kubernetes and I own the DNS portion, in
> particular.  I desperately want to use Alpine Linux (based on musl)
> but for now I have to warn people NOT to use it because of this issue.
> 
> On Fri, Sep 04, 2015 at 02:04:29PM -0400, Rich Felker wrote:
> > On Fri, Sep 04, 2015 at 12:11:36PM -0500, Andy Shinn wrote:
> >> I'm writing the wonderful musl project today to open discussion
> >> about the future possibility of DNS search and domain keyword
> >> support. We've been using musl libc (by way of Alpine Linux) for
> >> new development of applications as containers that discover each
> >> other through DNS and other software defined networking.
> >>
> >> In particular, we are starting to use applications like SkyDNS,
> >> Consul, and Kubernetes, all of which rely on local name
> >> resolution in some way using search paths. Many users of the
> >> Alpine Linux container image have also expressed their desire for
> >> this feature at
> >> https://github.com/gliderlabs/docker-alpine/issues/8.
> >>
> >> On the functional differences between glibc page, the domain and
> >> search keyword "Support may be added in the future if there is
> >> demand". So please consider this request an addition to whatever
> >> demand for the feature already exists.
> >>
> >> Thank you for your time and great work on the musl libc project!
> >
> > I think this is a reasonable request. I'll look into it more.
> >
> > One property I do not want to break is deterministic results, so
> > when a search is performed, if any step of the search ends with
> > an error rather than a positive or negative result, the whole
> > lookup needs to stop and report the error rather than falling
> > back. Falling back is not safe and creates a situation where DoS
> > can be used to control which results are returned.
> 
> I understand your point, though the world at large tends to disagree.
> Everyone has a primary and secondary `nameserver` record (or should).
> If the first one times out, try the second.  Most resolver libs seem
> to accept a SERVFAIL response or a timeout as a signal to try the next
> server, and I would encourage you to do the same.

musl intentionally does not do this because it yields abysmal
performance. If the first nameserver is overloaded or the packet is
lost, you suffer several-second lookup latency.

> Stopping on positive response or NXDOMAIN seems to be commonly
> accepted with a caveat.  You can't query all nameservers and just take
> the first NXDOMAIN to respond.  You can only accept NXDOMAIN if all of
> the higher-priority (listed first in resolv.conf) nameservers have
> timed out or SERVFAIL'ed.  You can issue queries in parallel, but you
> must process responses in order, which is what you describe below.

Timeout or servfail is not sufficient to accept an nxdomain from a
lower-priority server. To preserve consistency of results under
transient failure, you actually have to wait for the nxdomain from the
higher-priority server. Either way, this very much pessimizes usage
cases like running "netstat" with huge numbers of connections where
many of the ip addresses fail to reverse. Being able to return
immediately as soon as any one of the nameservers responds with
nxdomain makes the difference between a <1s netstat run and a 5-10s
netstat run.

Thus, if we add extensions to support the kind of result unioning you
want across multiple nameservers, I think they should be configurable
and off-by-default. A simple option in resolv.conf could turn them on.
And there could be options for requiring nxdomain from all servers
(true union) or just for highest-priority when accepting negative
results.

> > While it would be possible to parallelize the search while
> > serializing the results (i.e. waiting to accept a result from the
> > second query until the first query finished with a negative
> > result), I think the consensus during the last round of
> > discussion of this topic was that the complexity cost is too
> > great and the benefit too small. Ideally, the first query should
> > always succeed, anyway.
> 
> The real world is not ideal.  Not all nameservers are identically
> scoped - you MUST respect the ordering in resolv.conf - to do
> otherwise is semantically broken.  If implementation simplicity means
> literally doing queries in serial, then that is what you should do.

Even legacy resolvers had the option to rotate the nameservers for
load-balancing, so I think it's a stretch to say the ordering is
supposed to be semantic. My view has always been that multiple
nameservers in resolv.conf are for redundancy, not for serving
conflicting records.

> Similarly, you can't just search all search domains in parallel and
> take the first response.  The ordering is meaningful.

Indeed, search domains are like that, because they inherently produce
ambiguity/overlapping namespaces with different definitions. This is
why myself and others who weighed in on the original question of
supporting them were against, but left the option open to revisit the
topic if users who need them show up.

> > I also have a few questions:
> >
> > 1. Do you need multiple search items, or just a single domain?
> >    Any setup with multiple searches necessarily has suboptimal
> >    performance because ndots is not sufficient to make the right
> >    initial choice of query. If you do need this functionality, a
> >    unioning proxy dns server may be a better option than resolv.conf
> >    domain search; it would give much better performance.
> 
> We use multiple search paths and ndots > 1.  I'm not sure what you
> mean by "unioning" here.  Search path ordering is as meaningful as
> nameserver ordering.  You can't avoid making the query for each search
> suffix in the worst case, and it has the same restriction as
> nameserver - the search order must be respected.
> 
> There does seem to be some different implementations that search for
> the "naked" query first vs last, though.  I think the semantically
> correct (but pessimal performance) is to search for that last.

The traditional behavior is to do the naked query first if the query
string has at least 'ndots' dots, and to do the search domains first
otherwise. Also I believe a final dot always suppresses search.

My point was that with ndots=1 (default) and only a single search
domain, the _expected_ result is that the first query succeed. But if
you have ndots>1 or multiple search domains, you expect a portion of
your queries to fall back at least once. This adds significant
latency.

In such a situation, you can avoid the additional latency (except on
the first query of a given record) by running a local caching
nameserver that does the search and unioning for you, rather than
having the stub resolver in libc do it. Then subsequent queries
succeed immediately using the cache. The reason I asked about usage
case (ndots=1 vs ndots>1, single vs multiple search) is that, in the
multi-fallback case, it might make more sense (from a performance and
clean design standpoint) to implement this with a caching nameserver
on localhost rather than in musl.

> > 2. For your intended applications, is there a need to support
> >    ndots>1?  Such configurations are generally not friendly to
> >    applications that expect to be able to resolve normal internet
> >    domain lookups, and performance for such lookups will be very bad
> >    (because the search domains first have to fail).
> 
> DNS is a very lightweight protocol.  We have not measured any
> practical detriment for having 6 search domains and ndots=5.  In the
> normal case it fails very quickly.  That aside, it should be my
> business if I want to (mis)configure my system that way :)

I suspect we have different definitions of quick... :)

> > 3. The glibc behavior is just to swap the order of search when
> >    the query string has >=ndots dots in it, but would it be
> >    acceptable never to try the search domains at all in this case?
> >    That would yield much better performance for nxdomain results and
> >    avoid unexpected positive results due to weird subdomains
> >    existing in your search domain (e.g. a wildcard for
> >    *.us.example.com would cause *.us to wrongly resolve for
> >    non-existant .us domains).
> 
> I think that would be correct.  If I have 3 dots and ndots=2, search
> paths should be ignored.

Glad we agree on this.

I hope you feel like this conversation is productive. I don't want to
rule out anything/"say no" right away, but rather try to get a better
understanding of your requirements first and figure out what makes the
most sense to do on musl's side.

Rich

Re: Re: Would love to see reconsideration for domain and search

[Tim Hockin]

On Thu, Oct 22, 2015 at 2:56 PM, Rich Felker <dalias@libc.org> wrote:
> On Thu, Oct 22, 2015 at 02:24:11PM -0700, Tim Hockin wrote:
>> Hi all,
>>
>> I saw this thread on the web archive but am not sure how to respond to
>> the thread directly as a new joinee of the ML.  I hope this finds its
>> way...
>
> No problem; just starting a new thread like this and quoting the old
> one is fine.
>
>> I am one of the developers of Kubernetes and I own the DNS portion, in
>> particular.  I desperately want to use Alpine Linux (based on musl)
>> but for now I have to warn people NOT to use it because of this issue.
>>
>> On Fri, Sep 04, 2015 at 02:04:29PM -0400, Rich Felker wrote:
>> > On Fri, Sep 04, 2015 at 12:11:36PM -0500, Andy Shinn wrote:
>> >> I'm writing the wonderful musl project today to open discussion
>> >> about the future possibility of DNS search and domain keyword
>> >> support. We've been using musl libc (by way of Alpine Linux) for
>> >> new development of applications as containers that discover each
>> >> other through DNS and other software defined networking.
>> >>
>> >> In particular, we are starting to use applications like SkyDNS,
>> >> Consul, and Kubernetes, all of which rely on local name
>> >> resolution in some way using search paths. Many users of the
>> >> Alpine Linux container image have also expressed their desire for
>> >> this feature at
>> >> https://github.com/gliderlabs/docker-alpine/issues/8.
>> >>
>> >> On the functional differences between glibc page, the domain and
>> >> search keyword "Support may be added in the future if there is
>> >> demand". So please consider this request an addition to whatever
>> >> demand for the feature already exists.
>> >>
>> >> Thank you for your time and great work on the musl libc project!
>> >
>> > I think this is a reasonable request. I'll look into it more.
>> >
>> > One property I do not want to break is deterministic results, so
>> > when a search is performed, if any step of the search ends with
>> > an error rather than a positive or negative result, the whole
>> > lookup needs to stop and report the error rather than falling
>> > back. Falling back is not safe and creates a situation where DoS
>> > can be used to control which results are returned.
>>
>> I understand your point, though the world at large tends to disagree.
>> Everyone has a primary and secondary `nameserver` record (or should).
>> If the first one times out, try the second.  Most resolver libs seem
>> to accept a SERVFAIL response or a timeout as a signal to try the next
>> server, and I would encourage you to do the same.
>
> musl intentionally does not do this because it yields abysmal
> performance. If the first nameserver is overloaded or the packet is
> lost, you suffer several-second lookup latency.

But at least it works eventually.  You're faced with a choice.  Wait 2
seconds for ns1 to timeout and then fail in a way that most apps don't
handle well or wait for 2 seconds and then (usually) get a fast
response from ns2.

It seems better in every way to eventually succeed, though I agree
it's a bit less visible.

>> Stopping on positive response or NXDOMAIN seems to be commonly
>> accepted with a caveat.  You can't query all nameservers and just take
>> the first NXDOMAIN to respond.  You can only accept NXDOMAIN if all of
>> the higher-priority (listed first in resolv.conf) nameservers have
>> timed out or SERVFAIL'ed.  You can issue queries in parallel, but you
>> must process responses in order, which is what you describe below.
>
> Timeout or servfail is not sufficient to accept an nxdomain from a
> lower-priority server. To preserve consistency of results under
> transient failure, you actually have to wait for the nxdomain from the

I have to disagree.  Some non-forwarding DNS servers use SERVFAIL to
indicate "I am not serving for that domain" specifically to make the
client move to their next nameserver.  if ns1 returns SERVFAIL, try
ns2.  If ns1 times out, try ns2.  Otherwise what good is ns2?

> higher-priority server. Either way, this very much pessimizes usage
> cases like running "netstat" with huge numbers of connections where
> many of the ip addresses fail to reverse. Being able to return
> immediately as soon as any one of the nameservers responds with
> nxdomain makes the difference between a <1s netstat run and a 5-10s
> netstat run.

Sure it's faster but it's WRONG.  Returning a random number would be
faster, too, but it is equally wrong.  This is why netstat (and myriad
other tools) has a `-n` flag.

> Thus, if we add extensions to support the kind of result unioning you
> want across multiple nameservers, I think they should be configurable
> and off-by-default. A simple option in resolv.conf could turn them on.
> And there could be options for requiring nxdomain from all servers
> (true union) or just for highest-priority when accepting negative
> results.

I can't agree with this.  It's reasonable to make options for these,
but I think the non-standard behaviors should be off by default.
Consider this from the point of view of a system like Docker or
Kubernetes, which generate resolv.conf for you - they have no idea
what libc your apps are using, so it's unreasonable to ask them to
turn off libc-specific flags.  However, the end user knows, and it is
perfectly sane to ask them to explicitly opt-in to non-standard
optimized behaviors.

>> > While it would be possible to parallelize the search while
>> > serializing the results (i.e. waiting to accept a result from the
>> > second query until the first query finished with a negative
>> > result), I think the consensus during the last round of
>> > discussion of this topic was that the complexity cost is too
>> > great and the benefit too small. Ideally, the first query should
>> > always succeed, anyway.
>>
>> The real world is not ideal.  Not all nameservers are identically
>> scoped - you MUST respect the ordering in resolv.conf - to do
>> otherwise is semantically broken.  If implementation simplicity means
>> literally doing queries in serial, then that is what you should do.
>
> Even legacy resolvers had the option to rotate the nameservers for
> load-balancing, so I think it's a stretch to say the ordering is
> supposed to be semantic. My view has always been that multiple
> nameservers in resolv.conf are for redundancy, not for serving
> conflicting records.

You argued above that you should not try a secondary server in case of
timeout or SERVFAIL.  Obviously you would not try it on success nor
NXDOMAIN.  When do you see a secondary being used at all?

As for rotate, note that it is an option and OFF by default in every
mainstream resolver implementation.

But this point is sort of academic for us - we're moving to a
forwarding nameserver so really there is only the primary nameserver.
 We just need you to ask the first nameserver first.

>> Similarly, you can't just search all search domains in parallel and
>> take the first response.  The ordering is meaningful.
>
> Indeed, search domains are like that, because they inherently produce
> ambiguity/overlapping namespaces with different definitions. This is
> why myself and others who weighed in on the original question of
> supporting them were against, but left the option open to revisit the
> topic if users who need them show up.

Yeah, I scanned the related threads.  I understand the issue in
theory, but in practice these are things configured by admins.  If
there's a conflict or ambiguity, you should solve that, not jettison
powerful functionality.

>> > I also have a few questions:
>> >
>> > 1. Do you need multiple search items, or just a single domain?
>> >    Any setup with multiple searches necessarily has suboptimal
>> >    performance because ndots is not sufficient to make the right
>> >    initial choice of query. If you do need this functionality, a
>> >    unioning proxy dns server may be a better option than resolv.conf
>> >    domain search; it would give much better performance.
>>
>> We use multiple search paths and ndots > 1.  I'm not sure what you
>> mean by "unioning" here.  Search path ordering is as meaningful as
>> nameserver ordering.  You can't avoid making the query for each search
>> suffix in the worst case, and it has the same restriction as
>> nameserver - the search order must be respected.
>>
>> There does seem to be some different implementations that search for
>> the "naked" query first vs last, though.  I think the semantically
>> correct (but pessimal performance) is to search for that last.
>
> The traditional behavior is to do the naked query first if the query
> string has at least 'ndots' dots, and to do the search domains first
> otherwise. Also I believe a final dot always suppresses search.
>
> My point was that with ndots=1 (default) and only a single search
> domain, the _expected_ result is that the first query succeed. But if
> you have ndots>1 or multiple search domains, you expect a portion of
> your queries to fall back at least once. This adds significant
> latency.

It adds latency, but the magnitude is very much determined by the
installation.  In our case it is negligible and well worth the cost.
I fear you're optimizing without data - it should be the site-admin's
problem to configure things in an acceptable way.  libc doesn't get to
decide what "acceptable" means.

> In such a situation, you can avoid the additional latency (except on
> the first query of a given record) by running a local caching
> nameserver that does the search and unioning for you, rather than
> having the stub resolver in libc do it. Then subsequent queries
> succeed immediately using the cache. The reason I asked about usage
> case (ndots=1 vs ndots>1, single vs multiple search) is that, in the
> multi-fallback case, it might make more sense (from a performance and
> clean design standpoint) to implement this with a caching nameserver
> on localhost rather than in musl.

We might be moving to a per-machine local DNS agent, which would cache
as you describe.  HOWEVER, there's a pretty important piece that I
guess I left out.  Docker and Kubernetes and similar systems run many
containers per machine.  Each container has a potentially different
search path.  I might run 100 or more containers on a single machine -
I can't run 100 DNS caches, and I can't put that back on users.

So from our perspective the search paths MUST come from the containers
themselves, even if we run a machine local cache to mitigate latency
and SPOF.

>> > 2. For your intended applications, is there a need to support
>> >    ndots>1?  Such configurations are generally not friendly to
>> >    applications that expect to be able to resolve normal internet
>> >    domain lookups, and performance for such lookups will be very bad
>> >    (because the search domains first have to fail).
>>
>> DNS is a very lightweight protocol.  We have not measured any
>> practical detriment for having 6 search domains and ndots=5.  In the
>> normal case it fails very quickly.  That aside, it should be my
>> business if I want to (mis)configure my system that way :)
>
> I suspect we have different definitions of quick... :)

Quick is situational.  In a cloud-based mostly-webapp stack, 50ms to
do a name lookup ain't so bad, given the relative infrequency of that
operation.  Also most names actually DO resolve on the first or second
search path.

>> > 3. The glibc behavior is just to swap the order of search when
>> >    the query string has >=ndots dots in it, but would it be
>> >    acceptable never to try the search domains at all in this case?
>> >    That would yield much better performance for nxdomain results and
>> >    avoid unexpected positive results due to weird subdomains
>> >    existing in your search domain (e.g. a wildcard for
>> >    *.us.example.com would cause *.us to wrongly resolve for
>> >    non-existant .us domains).
>>
>> I think that would be correct.  If I have 3 dots and ndots=2, search
>> paths should be ignored.
>
> Glad we agree on this.
>
> I hope you feel like this conversation is productive. I don't want to
> rule out anything/"say no" right away, but rather try to get a better
> understanding of your requirements first and figure out what makes the
> most sense to do on musl's side.

Absolutely.  I'm happy to engage.  Obviously our use case is a bit
outside of what musl was really aiming for, but it offers a really
nice base for very efficient containers.  A lot of people want to use
Alpine and it breaks my heart to tell them they can't.

Re: Re: Would love to see reconsideration for domain and search

[Josiah Worcester]

[-- Attachment #1: Type: text/plain, Size: 14778 bytes --]

On Thu, Oct 22, 2015 at 3:37 PM Tim Hockin <thockin@google.com> wrote:

> On Thu, Oct 22, 2015 at 2:56 PM, Rich Felker <dalias@libc.org> wrote:
> > On Thu, Oct 22, 2015 at 02:24:11PM -0700, Tim Hockin wrote:
> >> Hi all,
> >>
> >> I saw this thread on the web archive but am not sure how to respond to
> >> the thread directly as a new joinee of the ML.  I hope this finds its
> >> way...
> >
> > No problem; just starting a new thread like this and quoting the old
> > one is fine.
> >
> >> I am one of the developers of Kubernetes and I own the DNS portion, in
> >> particular.  I desperately want to use Alpine Linux (based on musl)
> >> but for now I have to warn people NOT to use it because of this issue.
> >>
> >> On Fri, Sep 04, 2015 at 02:04:29PM -0400, Rich Felker wrote:
> >> > On Fri, Sep 04, 2015 at 12:11:36PM -0500, Andy Shinn wrote:
> >> >> I'm writing the wonderful musl project today to open discussion
> >> >> about the future possibility of DNS search and domain keyword
> >> >> support. We've been using musl libc (by way of Alpine Linux) for
> >> >> new development of applications as containers that discover each
> >> >> other through DNS and other software defined networking.
> >> >>
> >> >> In particular, we are starting to use applications like SkyDNS,
> >> >> Consul, and Kubernetes, all of which rely on local name
> >> >> resolution in some way using search paths. Many users of the
> >> >> Alpine Linux container image have also expressed their desire for
> >> >> this feature at
> >> >> https://github.com/gliderlabs/docker-alpine/issues/8.
> >> >>
> >> >> On the functional differences between glibc page, the domain and
> >> >> search keyword "Support may be added in the future if there is
> >> >> demand". So please consider this request an addition to whatever
> >> >> demand for the feature already exists.
> >> >>
> >> >> Thank you for your time and great work on the musl libc project!
> >> >
> >> > I think this is a reasonable request. I'll look into it more.
> >> >
> >> > One property I do not want to break is deterministic results, so
> >> > when a search is performed, if any step of the search ends with
> >> > an error rather than a positive or negative result, the whole
> >> > lookup needs to stop and report the error rather than falling
> >> > back. Falling back is not safe and creates a situation where DoS
> >> > can be used to control which results are returned.
> >>
> >> I understand your point, though the world at large tends to disagree.
> >> Everyone has a primary and secondary `nameserver` record (or should).
> >> If the first one times out, try the second.  Most resolver libs seem
> >> to accept a SERVFAIL response or a timeout as a signal to try the next
> >> server, and I would encourage you to do the same.
> >
> > musl intentionally does not do this because it yields abysmal
> > performance. If the first nameserver is overloaded or the packet is
> > lost, you suffer several-second lookup latency.
>
> But at least it works eventually.  You're faced with a choice.  Wait 2
> seconds for ns1 to timeout and then fail in a way that most apps don't
> handle well or wait for 2 seconds and then (usually) get a fast
> response from ns2.
>
> It seems better in every way to eventually succeed, though I agree
> it's a bit less visible.
>

With musl's current design, you get a request to ns1 and ns2, and the first
authoritative response wins. So, if ns1 fails then all is well and
performance isn't even notably impacted. What you are describing appears to
be how you would *have* to implement it if you decide against considering
all servers equal, but instead try and serve the union of their responses
(that is, wait for timeout and then fail).

Consider what would happen if ns1 and ns2 have different responses, but ns1
for whatever reason times out (potentially an attacker). Then you get the
results for ns2, even though ns1 is intended to override it.


> >> Stopping on positive response or NXDOMAIN seems to be commonly
> >> accepted with a caveat.  You can't query all nameservers and just take
> >> the first NXDOMAIN to respond.  You can only accept NXDOMAIN if all of
> >> the higher-priority (listed first in resolv.conf) nameservers have
> >> timed out or SERVFAIL'ed.  You can issue queries in parallel, but you
> >> must process responses in order, which is what you describe below.
> >
> > Timeout or servfail is not sufficient to accept an nxdomain from a
> > lower-priority server. To preserve consistency of results under
> > transient failure, you actually have to wait for the nxdomain from the
>
> I have to disagree.  Some non-forwarding DNS servers use SERVFAIL to
> indicate "I am not serving for that domain" specifically to make the
> client move to their next nameserver.  if ns1 returns SERVFAIL, try
> ns2.  If ns1 times out, try ns2.  Otherwise what good is ns2?
>

Note that this means that any condition where ns1 can't be accessed changes
what DNS resolves to. If you wish to prevent unexpected behavior when ns1
can't be accessed, you simply have to get a response from ns1 first, and
only ever query ns2 when the response from ns1 indicates you may.


>
> > higher-priority server. Either way, this very much pessimizes usage
> > cases like running "netstat" with huge numbers of connections where
> > many of the ip addresses fail to reverse. Being able to return
> > immediately as soon as any one of the nameservers responds with
> > nxdomain makes the difference between a <1s netstat run and a 5-10s
> > netstat run.
>
> Sure it's faster but it's WRONG.  Returning a random number would be
> faster, too, but it is equally wrong.  This is why netstat (and myriad
> other tools) has a `-n` flag.


 Again, musl's design assumes that all nameservers are hosting the same
space, and thus a single nxdomain is authoritative. If this is the case,
then this is perfectly correct. If it's not the case, then it's wrong (and
you need to wait for literally everything to nxdomain, and if there's a
timeout from a single server you *need* to report that).


> > Thus, if we add extensions to support the kind of result unioning you
> > want across multiple nameservers, I think they should be configurable
> > and off-by-default. A simple option in resolv.conf could turn them on.
> > And there could be options for requiring nxdomain from all servers
> > (true union) or just for highest-priority when accepting negative
> > results.
>
> I can't agree with this.  It's reasonable to make options for these,
> but I think the non-standard behaviors should be off by default.
> Consider this from the point of view of a system like Docker or
> Kubernetes, which generate resolv.conf for you - they have no idea
> what libc your apps are using, so it's unreasonable to ask them to
> turn off libc-specific flags.  However, the end user knows, and it is
> perfectly sane to ask them to explicitly opt-in to non-standard
> optimized behaviors.
>
> >> > While it would be possible to parallelize the search while
> >> > serializing the results (i.e. waiting to accept a result from the
> >> > second query until the first query finished with a negative
> >> > result), I think the consensus during the last round of
> >> > discussion of this topic was that the complexity cost is too
> >> > great and the benefit too small. Ideally, the first query should
> >> > always succeed, anyway.
> >>
> >> The real world is not ideal.  Not all nameservers are identically
> >> scoped - you MUST respect the ordering in resolv.conf - to do
> >> otherwise is semantically broken.  If implementation simplicity means
> >> literally doing queries in serial, then that is what you should do.
> >
> > Even legacy resolvers had the option to rotate the nameservers for
> > load-balancing, so I think it's a stretch to say the ordering is
> > supposed to be semantic. My view has always been that multiple
> > nameservers in resolv.conf are for redundancy, not for serving
> > conflicting records.
>
> You argued above that you should not try a secondary server in case of
> timeout or SERVFAIL.  Obviously you would not try it on success nor
> NXDOMAIN.  When do you see a secondary being used at all?
>

If I understand correctly, what you are expecting is this:
Try resolving from ns1.
If that domain does not exist on ns1, try ns2.
If that domain does not exist on ns2, report failure.

The only way to do this in a consistent fashion is actually to fall through
to the secondary if and only if you get an nxdomain from ns1.


>
> As for rotate, note that it is an option and OFF by default in every
> mainstream resolver implementation.
>
> But this point is sort of academic for us - we're moving to a
> forwarding nameserver so really there is only the primary nameserver.
>  We just need you to ask the first nameserver first.
>
> >> Similarly, you can't just search all search domains in parallel and
> >> take the first response.  The ordering is meaningful.
> >
> > Indeed, search domains are like that, because they inherently produce
> > ambiguity/overlapping namespaces with different definitions. This is
> > why myself and others who weighed in on the original question of
> > supporting them were against, but left the option open to revisit the
> > topic if users who need them show up.
>
> Yeah, I scanned the related threads.  I understand the issue in
> theory, but in practice these are things configured by admins.  If
> there's a conflict or ambiguity, you should solve that, not jettison
> powerful functionality.
>
> >> > I also have a few questions:
> >> >
> >> > 1. Do you need multiple search items, or just a single domain?
> >> >    Any setup with multiple searches necessarily has suboptimal
> >> >    performance because ndots is not sufficient to make the right
> >> >    initial choice of query. If you do need this functionality, a
> >> >    unioning proxy dns server may be a better option than resolv.conf
> >> >    domain search; it would give much better performance.
> >>
> >> We use multiple search paths and ndots > 1.  I'm not sure what you
> >> mean by "unioning" here.  Search path ordering is as meaningful as
> >> nameserver ordering.  You can't avoid making the query for each search
> >> suffix in the worst case, and it has the same restriction as
> >> nameserver - the search order must be respected.
> >>
> >> There does seem to be some different implementations that search for
> >> the "naked" query first vs last, though.  I think the semantically
> >> correct (but pessimal performance) is to search for that last.
> >
> > The traditional behavior is to do the naked query first if the query
> > string has at least 'ndots' dots, and to do the search domains first
> > otherwise. Also I believe a final dot always suppresses search.
> >
> > My point was that with ndots=1 (default) and only a single search
> > domain, the _expected_ result is that the first query succeed. But if
> > you have ndots>1 or multiple search domains, you expect a portion of
> > your queries to fall back at least once. This adds significant
> > latency.
>
> It adds latency, but the magnitude is very much determined by the
> installation.  In our case it is negligible and well worth the cost.
> I fear you're optimizing without data - it should be the site-admin's
> problem to configure things in an acceptable way.  libc doesn't get to
> decide what "acceptable" means.
>
> > In such a situation, you can avoid the additional latency (except on
> > the first query of a given record) by running a local caching
> > nameserver that does the search and unioning for you, rather than
> > having the stub resolver in libc do it. Then subsequent queries
> > succeed immediately using the cache. The reason I asked about usage
> > case (ndots=1 vs ndots>1, single vs multiple search) is that, in the
> > multi-fallback case, it might make more sense (from a performance and
> > clean design standpoint) to implement this with a caching nameserver
> > on localhost rather than in musl.
>
> We might be moving to a per-machine local DNS agent, which would cache
> as you describe.  HOWEVER, there's a pretty important piece that I
> guess I left out.  Docker and Kubernetes and similar systems run many
> containers per machine.  Each container has a potentially different
> search path.  I might run 100 or more containers on a single machine -
> I can't run 100 DNS caches, and I can't put that back on users.
>

Why not? 100 DNS caches shouldn't be particularly expensive.


>
> So from our perspective the search paths MUST come from the containers
> themselves, even if we run a machine local cache to mitigate latency
> and SPOF.
>
> >> > 2. For your intended applications, is there a need to support
> >> >    ndots>1?  Such configurations are generally not friendly to
> >> >    applications that expect to be able to resolve normal internet
> >> >    domain lookups, and performance for such lookups will be very bad
> >> >    (because the search domains first have to fail).
> >>
> >> DNS is a very lightweight protocol.  We have not measured any
> >> practical detriment for having 6 search domains and ndots=5.  In the
> >> normal case it fails very quickly.  That aside, it should be my
> >> business if I want to (mis)configure my system that way :)
> >
> > I suspect we have different definitions of quick... :)
>
> Quick is situational.  In a cloud-based mostly-webapp stack, 50ms to
> do a name lookup ain't so bad, given the relative infrequency of that
> operation.  Also most names actually DO resolve on the first or second
> search path.
>
> >> > 3. The glibc behavior is just to swap the order of search when
> >> >    the query string has >=ndots dots in it, but would it be
> >> >    acceptable never to try the search domains at all in this case?
> >> >    That would yield much better performance for nxdomain results and
> >> >    avoid unexpected positive results due to weird subdomains
> >> >    existing in your search domain (e.g. a wildcard for
> >> >    *.us.example.com would cause *.us to wrongly resolve for
> >> >    non-existant .us domains).
> >>
> >> I think that would be correct.  If I have 3 dots and ndots=2, search
> >> paths should be ignored.
> >
> > Glad we agree on this.
> >
> > I hope you feel like this conversation is productive. I don't want to
> > rule out anything/"say no" right away, but rather try to get a better
> > understanding of your requirements first and figure out what makes the
> > most sense to do on musl's side.
>
> Absolutely.  I'm happy to engage.  Obviously our use case is a bit
> outside of what musl was really aiming for, but it offers a really
> nice base for very efficient containers.  A lot of people want to use
> Alpine and it breaks my heart to tell them they can't.
>

[-- Attachment #2: Type: text/html, Size: 18192 bytes --]

Re: Re: Would love to see reconsideration for domain and search

[Tim Hockin]

On Thu, Oct 22, 2015 at 4:00 PM, Josiah Worcester <josiahw@gmail.com> wrote:
> On Thu, Oct 22, 2015 at 3:37 PM Tim Hockin <thockin@google.com> wrote:
>>
>> On Thu, Oct 22, 2015 at 2:56 PM, Rich Felker <dalias@libc.org> wrote:
>> > On Thu, Oct 22, 2015 at 02:24:11PM -0700, Tim Hockin wrote:
>> >> Hi all,
>> >>
>> >> I saw this thread on the web archive but am not sure how to respond to
>> >> the thread directly as a new joinee of the ML.  I hope this finds its
>> >> way...
>> >
>> > No problem; just starting a new thread like this and quoting the old
>> > one is fine.
>> >
>> >> I am one of the developers of Kubernetes and I own the DNS portion, in
>> >> particular.  I desperately want to use Alpine Linux (based on musl)
>> >> but for now I have to warn people NOT to use it because of this issue.
>> >>
>> >> On Fri, Sep 04, 2015 at 02:04:29PM -0400, Rich Felker wrote:
>> >> > On Fri, Sep 04, 2015 at 12:11:36PM -0500, Andy Shinn wrote:
>> >> >> I'm writing the wonderful musl project today to open discussion
>> >> >> about the future possibility of DNS search and domain keyword
>> >> >> support. We've been using musl libc (by way of Alpine Linux) for
>> >> >> new development of applications as containers that discover each
>> >> >> other through DNS and other software defined networking.
>> >> >>
>> >> >> In particular, we are starting to use applications like SkyDNS,
>> >> >> Consul, and Kubernetes, all of which rely on local name
>> >> >> resolution in some way using search paths. Many users of the
>> >> >> Alpine Linux container image have also expressed their desire for
>> >> >> this feature at
>> >> >> https://github.com/gliderlabs/docker-alpine/issues/8.
>> >> >>
>> >> >> On the functional differences between glibc page, the domain and
>> >> >> search keyword "Support may be added in the future if there is
>> >> >> demand". So please consider this request an addition to whatever
>> >> >> demand for the feature already exists.
>> >> >>
>> >> >> Thank you for your time and great work on the musl libc project!
>> >> >
>> >> > I think this is a reasonable request. I'll look into it more.
>> >> >
>> >> > One property I do not want to break is deterministic results, so
>> >> > when a search is performed, if any step of the search ends with
>> >> > an error rather than a positive or negative result, the whole
>> >> > lookup needs to stop and report the error rather than falling
>> >> > back. Falling back is not safe and creates a situation where DoS
>> >> > can be used to control which results are returned.
>> >>
>> >> I understand your point, though the world at large tends to disagree.
>> >> Everyone has a primary and secondary `nameserver` record (or should).
>> >> If the first one times out, try the second.  Most resolver libs seem
>> >> to accept a SERVFAIL response or a timeout as a signal to try the next
>> >> server, and I would encourage you to do the same.
>> >
>> > musl intentionally does not do this because it yields abysmal
>> > performance. If the first nameserver is overloaded or the packet is
>> > lost, you suffer several-second lookup latency.
>>
>> But at least it works eventually.  You're faced with a choice.  Wait 2
>> seconds for ns1 to timeout and then fail in a way that most apps don't
>> handle well or wait for 2 seconds and then (usually) get a fast
>> response from ns2.
>>
>> It seems better in every way to eventually succeed, though I agree
>> it's a bit less visible.
>
>
> With musl's current design, you get a request to ns1 and ns2, and the first
> authoritative response wins. So, if ns1 fails then all is well and
> performance isn't even notably impacted. What you are describing appears to
> be how you would *have* to implement it if you decide against considering
> all servers equal, but instead try and serve the union of their responses
> (that is, wait for timeout and then fail).

The authoritative-ness is a dimension I had not considered.  I could
believe that the first authoritative answer wins, but what if you only
get a non-authoritative answer? from ns1 and ns2 never responds?

> Consider what would happen if ns1 and ns2 have different responses, but ns1
> for whatever reason times out (potentially an attacker). Then you get the
> results for ns2, even though ns1 is intended to override it.

I agree in theory.  And yet this is how most resolvers work today.
Are they all broken?

>> >> Stopping on positive response or NXDOMAIN seems to be commonly
>> >> accepted with a caveat.  You can't query all nameservers and just take
>> >> the first NXDOMAIN to respond.  You can only accept NXDOMAIN if all of
>> >> the higher-priority (listed first in resolv.conf) nameservers have
>> >> timed out or SERVFAIL'ed.  You can issue queries in parallel, but you
>> >> must process responses in order, which is what you describe below.
>> >
>> > Timeout or servfail is not sufficient to accept an nxdomain from a
>> > lower-priority server. To preserve consistency of results under
>> > transient failure, you actually have to wait for the nxdomain from the
>>
>> I have to disagree.  Some non-forwarding DNS servers use SERVFAIL to
>> indicate "I am not serving for that domain" specifically to make the
>> client move to their next nameserver.  if ns1 returns SERVFAIL, try
>> ns2.  If ns1 times out, try ns2.  Otherwise what good is ns2?
>
>
> Note that this means that any condition where ns1 can't be accessed changes
> what DNS resolves to. If you wish to prevent unexpected behavior when ns1
> can't be accessed, you simply have to get a response from ns1 first, and
> only ever query ns2 when the response from ns1 indicates you may.

If you get a response from ns1, why would you ever go to ns2?

>> > higher-priority server. Either way, this very much pessimizes usage
>> > cases like running "netstat" with huge numbers of connections where
>> > many of the ip addresses fail to reverse. Being able to return
>> > immediately as soon as any one of the nameservers responds with
>> > nxdomain makes the difference between a <1s netstat run and a 5-10s
>> > netstat run.
>>
>> Sure it's faster but it's WRONG.  Returning a random number would be
>> faster, too, but it is equally wrong.  This is why netstat (and myriad
>> other tools) has a `-n` flag.
>
>
>  Again, musl's design assumes that all nameservers are hosting the same
> space, and thus a single nxdomain is authoritative. If this is the case,
> then this is perfectly correct. If it's not the case, then it's wrong (and

Agree

> you need to wait for literally everything to nxdomain, and if there's a
> timeout from a single server you *need* to report that).

I don't buy the "everything" part.  If we assume that nameserver
ordering is priority ordering, the first NXDOMAIN is sufficient.   But
I can see how this is maybe not the normal case, and we're probably
moving to a model more consistent with musl's view (but we still need
search :)

>> > Thus, if we add extensions to support the kind of result unioning you
>> > want across multiple nameservers, I think they should be configurable
>> > and off-by-default. A simple option in resolv.conf could turn them on.
>> > And there could be options for requiring nxdomain from all servers
>> > (true union) or just for highest-priority when accepting negative
>> > results.
>>
>> I can't agree with this.  It's reasonable to make options for these,
>> but I think the non-standard behaviors should be off by default.
>> Consider this from the point of view of a system like Docker or
>> Kubernetes, which generate resolv.conf for you - they have no idea
>> what libc your apps are using, so it's unreasonable to ask them to
>> turn off libc-specific flags.  However, the end user knows, and it is
>> perfectly sane to ask them to explicitly opt-in to non-standard
>> optimized behaviors.
>>
>> >> > While it would be possible to parallelize the search while
>> >> > serializing the results (i.e. waiting to accept a result from the
>> >> > second query until the first query finished with a negative
>> >> > result), I think the consensus during the last round of
>> >> > discussion of this topic was that the complexity cost is too
>> >> > great and the benefit too small. Ideally, the first query should
>> >> > always succeed, anyway.
>> >>
>> >> The real world is not ideal.  Not all nameservers are identically
>> >> scoped - you MUST respect the ordering in resolv.conf - to do
>> >> otherwise is semantically broken.  If implementation simplicity means
>> >> literally doing queries in serial, then that is what you should do.
>> >
>> > Even legacy resolvers had the option to rotate the nameservers for
>> > load-balancing, so I think it's a stretch to say the ordering is
>> > supposed to be semantic. My view has always been that multiple
>> > nameservers in resolv.conf are for redundancy, not for serving
>> > conflicting records.
>>
>> You argued above that you should not try a secondary server in case of
>> timeout or SERVFAIL.  Obviously you would not try it on success nor
>> NXDOMAIN.  When do you see a secondary being used at all?
>
>
> If I understand correctly, what you are expecting is this:
> Try resolving from ns1.
> If that domain does not exist on ns1, try ns2.
> If that domain does not exist on ns2, report failure.
>
> The only way to do this in a consistent fashion is actually to fall through
> to the secondary if and only if you get an nxdomain from ns1.

not nxdomain - servfail.  If you get nxdomain the server is saying "I
know that this domain does not exist".  This seems to work on glibc
and busybox and every other resolver we have tried except musl.  But
again, we're not even using this functionality and probably moving to
a more "standard" model.

>> As for rotate, note that it is an option and OFF by default in every
>> mainstream resolver implementation.
>>
>> But this point is sort of academic for us - we're moving to a
>> forwarding nameserver so really there is only the primary nameserver.
>>  We just need you to ask the first nameserver first.
>>
>> >> Similarly, you can't just search all search domains in parallel and
>> >> take the first response.  The ordering is meaningful.
>> >
>> > Indeed, search domains are like that, because they inherently produce
>> > ambiguity/overlapping namespaces with different definitions. This is
>> > why myself and others who weighed in on the original question of
>> > supporting them were against, but left the option open to revisit the
>> > topic if users who need them show up.
>>
>> Yeah, I scanned the related threads.  I understand the issue in
>> theory, but in practice these are things configured by admins.  If
>> there's a conflict or ambiguity, you should solve that, not jettison
>> powerful functionality.
>>
>> >> > I also have a few questions:
>> >> >
>> >> > 1. Do you need multiple search items, or just a single domain?
>> >> >    Any setup with multiple searches necessarily has suboptimal
>> >> >    performance because ndots is not sufficient to make the right
>> >> >    initial choice of query. If you do need this functionality, a
>> >> >    unioning proxy dns server may be a better option than resolv.conf
>> >> >    domain search; it would give much better performance.
>> >>
>> >> We use multiple search paths and ndots > 1.  I'm not sure what you
>> >> mean by "unioning" here.  Search path ordering is as meaningful as
>> >> nameserver ordering.  You can't avoid making the query for each search
>> >> suffix in the worst case, and it has the same restriction as
>> >> nameserver - the search order must be respected.
>> >>
>> >> There does seem to be some different implementations that search for
>> >> the "naked" query first vs last, though.  I think the semantically
>> >> correct (but pessimal performance) is to search for that last.
>> >
>> > The traditional behavior is to do the naked query first if the query
>> > string has at least 'ndots' dots, and to do the search domains first
>> > otherwise. Also I believe a final dot always suppresses search.
>> >
>> > My point was that with ndots=1 (default) and only a single search
>> > domain, the _expected_ result is that the first query succeed. But if
>> > you have ndots>1 or multiple search domains, you expect a portion of
>> > your queries to fall back at least once. This adds significant
>> > latency.
>>
>> It adds latency, but the magnitude is very much determined by the
>> installation.  In our case it is negligible and well worth the cost.
>> I fear you're optimizing without data - it should be the site-admin's
>> problem to configure things in an acceptable way.  libc doesn't get to
>> decide what "acceptable" means.
>>
>> > In such a situation, you can avoid the additional latency (except on
>> > the first query of a given record) by running a local caching
>> > nameserver that does the search and unioning for you, rather than
>> > having the stub resolver in libc do it. Then subsequent queries
>> > succeed immediately using the cache. The reason I asked about usage
>> > case (ndots=1 vs ndots>1, single vs multiple search) is that, in the
>> > multi-fallback case, it might make more sense (from a performance and
>> > clean design standpoint) to implement this with a caching nameserver
>> > on localhost rather than in musl.
>>
>> We might be moving to a per-machine local DNS agent, which would cache
>> as you describe.  HOWEVER, there's a pretty important piece that I
>> guess I left out.  Docker and Kubernetes and similar systems run many
>> containers per machine.  Each container has a potentially different
>> search path.  I might run 100 or more containers on a single machine -
>> I can't run 100 DNS caches, and I can't put that back on users.
>
>
> Why not? 100 DNS caches shouldn't be particularly expensive.

I only have one port 53 to use on the host, and I can't force this to
run inside user containers.  I guess I could write a split-horizon
proxy that expands search paths and caches per-client.  That's
exceedingly silly given that ONLY musl doesn't work :)  It's also
somewhat wrong since I will return an address for a name the client
did not actually ask for.

>> So from our perspective the search paths MUST come from the containers
>> themselves, even if we run a machine local cache to mitigate latency
>> and SPOF.
>>
>> >> > 2. For your intended applications, is there a need to support
>> >> >    ndots>1?  Such configurations are generally not friendly to
>> >> >    applications that expect to be able to resolve normal internet
>> >> >    domain lookups, and performance for such lookups will be very bad
>> >> >    (because the search domains first have to fail).
>> >>
>> >> DNS is a very lightweight protocol.  We have not measured any
>> >> practical detriment for having 6 search domains and ndots=5.  In the
>> >> normal case it fails very quickly.  That aside, it should be my
>> >> business if I want to (mis)configure my system that way :)
>> >
>> > I suspect we have different definitions of quick... :)
>>
>> Quick is situational.  In a cloud-based mostly-webapp stack, 50ms to
>> do a name lookup ain't so bad, given the relative infrequency of that
>> operation.  Also most names actually DO resolve on the first or second
>> search path.
>>
>> >> > 3. The glibc behavior is just to swap the order of search when
>> >> >    the query string has >=ndots dots in it, but would it be
>> >> >    acceptable never to try the search domains at all in this case?
>> >> >    That would yield much better performance for nxdomain results and
>> >> >    avoid unexpected positive results due to weird subdomains
>> >> >    existing in your search domain (e.g. a wildcard for
>> >> >    *.us.example.com would cause *.us to wrongly resolve for
>> >> >    non-existant .us domains).
>> >>
>> >> I think that would be correct.  If I have 3 dots and ndots=2, search
>> >> paths should be ignored.
>> >
>> > Glad we agree on this.
>> >
>> > I hope you feel like this conversation is productive. I don't want to
>> > rule out anything/"say no" right away, but rather try to get a better
>> > understanding of your requirements first and figure out what makes the
>> > most sense to do on musl's side.
>>
>> Absolutely.  I'm happy to engage.  Obviously our use case is a bit
>> outside of what musl was really aiming for, but it offers a really
>> nice base for very efficient containers.  A lot of people want to use
>> Alpine and it breaks my heart to tell them they can't.

Re: Re: Would love to see reconsideration for domain and search

[Rich Felker]

On Thu, Oct 22, 2015 at 04:37:18PM -0700, Tim Hockin wrote:
> On Thu, Oct 22, 2015 at 4:00 PM, Josiah Worcester <josiahw@gmail.com> wrote:
> > On Thu, Oct 22, 2015 at 3:37 PM Tim Hockin <thockin@google.com> wrote:
> >>
> >> On Thu, Oct 22, 2015 at 2:56 PM, Rich Felker <dalias@libc.org> wrote:
> >> > On Thu, Oct 22, 2015 at 02:24:11PM -0700, Tim Hockin wrote:
> >> >> Hi all,
> >> >>
> >> >> I saw this thread on the web archive but am not sure how to respond to
> >> >> the thread directly as a new joinee of the ML.  I hope this finds its
> >> >> way...
> >> >
> >> > No problem; just starting a new thread like this and quoting the old
> >> > one is fine.
> >> >
> >> >> I am one of the developers of Kubernetes and I own the DNS portion, in
> >> >> particular.  I desperately want to use Alpine Linux (based on musl)
> >> >> but for now I have to warn people NOT to use it because of this issue.
> >> >>
> >> >> On Fri, Sep 04, 2015 at 02:04:29PM -0400, Rich Felker wrote:
> >> >> > On Fri, Sep 04, 2015 at 12:11:36PM -0500, Andy Shinn wrote:
> >> >> >> I'm writing the wonderful musl project today to open discussion
> >> >> >> about the future possibility of DNS search and domain keyword
> >> >> >> support. We've been using musl libc (by way of Alpine Linux) for
> >> >> >> new development of applications as containers that discover each
> >> >> >> other through DNS and other software defined networking.
> >> >> >>
> >> >> >> In particular, we are starting to use applications like SkyDNS,
> >> >> >> Consul, and Kubernetes, all of which rely on local name
> >> >> >> resolution in some way using search paths. Many users of the
> >> >> >> Alpine Linux container image have also expressed their desire for
> >> >> >> this feature at
> >> >> >> https://github.com/gliderlabs/docker-alpine/issues/8.
> >> >> >>
> >> >> >> On the functional differences between glibc page, the domain and
> >> >> >> search keyword "Support may be added in the future if there is
> >> >> >> demand". So please consider this request an addition to whatever
> >> >> >> demand for the feature already exists.
> >> >> >>
> >> >> >> Thank you for your time and great work on the musl libc project!
> >> >> >
> >> >> > I think this is a reasonable request. I'll look into it more.
> >> >> >
> >> >> > One property I do not want to break is deterministic results, so
> >> >> > when a search is performed, if any step of the search ends with
> >> >> > an error rather than a positive or negative result, the whole
> >> >> > lookup needs to stop and report the error rather than falling
> >> >> > back. Falling back is not safe and creates a situation where DoS
> >> >> > can be used to control which results are returned.
> >> >>
> >> >> I understand your point, though the world at large tends to disagree.
> >> >> Everyone has a primary and secondary `nameserver` record (or should).
> >> >> If the first one times out, try the second.  Most resolver libs seem
> >> >> to accept a SERVFAIL response or a timeout as a signal to try the next
> >> >> server, and I would encourage you to do the same.
> >> >
> >> > musl intentionally does not do this because it yields abysmal
> >> > performance. If the first nameserver is overloaded or the packet is
> >> > lost, you suffer several-second lookup latency.
> >>
> >> But at least it works eventually.  You're faced with a choice.  Wait 2
> >> seconds for ns1 to timeout and then fail in a way that most apps don't
> >> handle well or wait for 2 seconds and then (usually) get a fast
> >> response from ns2.
> >>
> >> It seems better in every way to eventually succeed, though I agree
> >> it's a bit less visible.
> >
> >
> > With musl's current design, you get a request to ns1 and ns2, and the first
> > authoritative response wins. So, if ns1 fails then all is well and
> > performance isn't even notably impacted. What you are describing appears to
> > be how you would *have* to implement it if you decide against considering
> > all servers equal, but instead try and serve the union of their responses
> > (that is, wait for timeout and then fail).
> 
> The authoritative-ness is a dimension I had not considered.  I could
> believe that the first authoritative answer wins, but what if you only
> get a non-authoritative answer? from ns1 and ns2 never responds?

I don't think Josiah's use of "authoritative" here is consistent with
what musl actually does, and it's likely not a useful dimension. A
stub resolver is generally not intended to communicate with
authoritative nameservers. The current behavior is that a positive or
negative (nxdomain) result is treated as concluding the query, and
other results (servfail, refused, etc.) are treated as inconclusive
and allow the query to continue until it times out. (Also, servfail
results in a limited number of immediate retries; this behavior was
arrived at based on the behavior of nameservers that fail with
servfail.)

> > Consider what would happen if ns1 and ns2 have different responses, but ns1
> > for whatever reason times out (potentially an attacker). Then you get the
> > results for ns2, even though ns1 is intended to override it.
> 
> I agree in theory.  And yet this is how most resolvers work today.
> Are they all broken?

No, the resolvers are not broken. The configuration is, at least the
way I see it. The intent of the resolvers is that they're
communicating with redundant servers, not a sequence of overlaid
hostname databases. If that assumption is satisfied, there's no
problem.

BTW I think there are other strong reasons to move to a model based on
a local nameserver that does the unioning, not just performance. The
most compelling is DNSSEC, which requires a trusted channel between
the nameserver and the stub resolver in order for results to be
meaningful/trusted. In the future everybody should be running a
nameserver on localhost to do DNSSEC signature validation. In that
scheme, resolv.conf would just contain 127.0.0.1 (or could be omitted
entirely since that's the default, at least on musl).

Rich

Re: Re: Would love to see reconsideration for domain and search

[Tim Hockin]

On Thu, Oct 22, 2015 at 9:27 PM, Rich Felker <dalias@libc.org> wrote:
> On Thu, Oct 22, 2015 at 04:37:18PM -0700, Tim Hockin wrote:
>> On Thu, Oct 22, 2015 at 4:00 PM, Josiah Worcester <josiahw@gmail.com> wrote:
>> > On Thu, Oct 22, 2015 at 3:37 PM Tim Hockin <thockin@google.com> wrote:
>> >>
>> >> On Thu, Oct 22, 2015 at 2:56 PM, Rich Felker <dalias@libc.org> wrote:
>> >> > On Thu, Oct 22, 2015 at 02:24:11PM -0700, Tim Hockin wrote:
>> >> >> Hi all,
>> >> >>
>> >> >> I saw this thread on the web archive but am not sure how to respond to
>> >> >> the thread directly as a new joinee of the ML.  I hope this finds its
>> >> >> way...
>> >> >
>> >> > No problem; just starting a new thread like this and quoting the old
>> >> > one is fine.
>> >> >
>> >> >> I am one of the developers of Kubernetes and I own the DNS portion, in
>> >> >> particular.  I desperately want to use Alpine Linux (based on musl)
>> >> >> but for now I have to warn people NOT to use it because of this issue.
>> >> >>
>> >> >> On Fri, Sep 04, 2015 at 02:04:29PM -0400, Rich Felker wrote:
>> >> >> > On Fri, Sep 04, 2015 at 12:11:36PM -0500, Andy Shinn wrote:
>> >> >> >> I'm writing the wonderful musl project today to open discussion
>> >> >> >> about the future possibility of DNS search and domain keyword
>> >> >> >> support. We've been using musl libc (by way of Alpine Linux) for
>> >> >> >> new development of applications as containers that discover each
>> >> >> >> other through DNS and other software defined networking.
>> >> >> >>
>> >> >> >> In particular, we are starting to use applications like SkyDNS,
>> >> >> >> Consul, and Kubernetes, all of which rely on local name
>> >> >> >> resolution in some way using search paths. Many users of the
>> >> >> >> Alpine Linux container image have also expressed their desire for
>> >> >> >> this feature at
>> >> >> >> https://github.com/gliderlabs/docker-alpine/issues/8.
>> >> >> >>
>> >> >> >> On the functional differences between glibc page, the domain and
>> >> >> >> search keyword "Support may be added in the future if there is
>> >> >> >> demand". So please consider this request an addition to whatever
>> >> >> >> demand for the feature already exists.
>> >> >> >>
>> >> >> >> Thank you for your time and great work on the musl libc project!
>> >> >> >
>> >> >> > I think this is a reasonable request. I'll look into it more.
>> >> >> >
>> >> >> > One property I do not want to break is deterministic results, so
>> >> >> > when a search is performed, if any step of the search ends with
>> >> >> > an error rather than a positive or negative result, the whole
>> >> >> > lookup needs to stop and report the error rather than falling
>> >> >> > back. Falling back is not safe and creates a situation where DoS
>> >> >> > can be used to control which results are returned.
>> >> >>
>> >> >> I understand your point, though the world at large tends to disagree.
>> >> >> Everyone has a primary and secondary `nameserver` record (or should).
>> >> >> If the first one times out, try the second.  Most resolver libs seem
>> >> >> to accept a SERVFAIL response or a timeout as a signal to try the next
>> >> >> server, and I would encourage you to do the same.
>> >> >
>> >> > musl intentionally does not do this because it yields abysmal
>> >> > performance. If the first nameserver is overloaded or the packet is
>> >> > lost, you suffer several-second lookup latency.
>> >>
>> >> But at least it works eventually.  You're faced with a choice.  Wait 2
>> >> seconds for ns1 to timeout and then fail in a way that most apps don't
>> >> handle well or wait for 2 seconds and then (usually) get a fast
>> >> response from ns2.
>> >>
>> >> It seems better in every way to eventually succeed, though I agree
>> >> it's a bit less visible.
>> >
>> >
>> > With musl's current design, you get a request to ns1 and ns2, and the first
>> > authoritative response wins. So, if ns1 fails then all is well and
>> > performance isn't even notably impacted. What you are describing appears to
>> > be how you would *have* to implement it if you decide against considering
>> > all servers equal, but instead try and serve the union of their responses
>> > (that is, wait for timeout and then fail).
>>
>> The authoritative-ness is a dimension I had not considered.  I could
>> believe that the first authoritative answer wins, but what if you only
>> get a non-authoritative answer? from ns1 and ns2 never responds?
>
> I don't think Josiah's use of "authoritative" here is consistent with
> what musl actually does, and it's likely not a useful dimension. A
> stub resolver is generally not intended to communicate with
> authoritative nameservers. The current behavior is that a positive or
> negative (nxdomain) result is treated as concluding the query, and
> other results (servfail, refused, etc.) are treated as inconclusive
> and allow the query to continue until it times out. (Also, servfail
> results in a limited number of immediate retries; this behavior was
> arrived at based on the behavior of nameservers that fail with
> servfail.)
>
>> > Consider what would happen if ns1 and ns2 have different responses, but ns1
>> > for whatever reason times out (potentially an attacker). Then you get the
>> > results for ns2, even though ns1 is intended to override it.
>>
>> I agree in theory.  And yet this is how most resolvers work today.
>> Are they all broken?
>
> No, the resolvers are not broken. The configuration is, at least the
> way I see it. The intent of the resolvers is that they're
> communicating with redundant servers, not a sequence of overlaid
> hostname databases. If that assumption is satisfied, there's no
> problem.

The way you see it is not how everyone sees it, obviously.  And
there's not a standard or spec here that I know of.  That said, as
before, We can probably live with this assumption.

> BTW I think there are other strong reasons to move to a model based on
> a local nameserver that does the unioning, not just performance. The
> most compelling is DNSSEC, which requires a trusted channel between
> the nameserver and the stub resolver in order for results to be
> meaningful/trusted. In the future everybody should be running a
> nameserver on localhost to do DNSSEC signature validation. In that
> scheme, resolv.conf would just contain 127.0.0.1 (or could be omitted
> entirely since that's the default, at least on musl).

I can see a local nameserver doing resolution, but doing search
expansion seems like a stretch (and superfluous since it is local).

Re: Re: Would love to see reconsideration for domain and search

[Kurt H Maier]

On Thu, Oct 22, 2015 at 02:24:11PM -0700, Tim Hockin wrote:
> 
> I understand your point, though the world at large tends to disagree.

The world at large uses bad software.  Please don't use this sort of
reasoning as a justification for and embrace-extend operation on actual
standards.

> The real world is not ideal.  Not all nameservers are identically
> scoped - you MUST respect the ordering in resolv.conf - to do
> otherwise is semantically broken.  If implementation simplicity means
> literally doing queries in serial, then that is what you should do.

You absolutely cannot respect the ordering in resolv.conf; at least not
if you're relying on someone else's resolver.  If the orchestration
software depends on specific results being returned in particular
orders, the orchestration software should provide a mechanism to
generate them. 

> Similarly, you can't just search all search domains in parallel and
> take the first response.  The ordering is meaningful.

It should not be, and more to the point will not reliably be,
meaningful.

You are arguing for introducing performance penalties into musl that do
not affect you but do very much affect lots of other users.  I hope they
do not happen -- musl is not the right place to fix your problem.

khm

Re: Re: Would love to see reconsideration for domain and search

[Rich Felker]

On Thu, Oct 22, 2015 at 10:13:18PM -0700, Tim Hockin wrote:
> >> > Consider what would happen if ns1 and ns2 have different responses, but ns1
> >> > for whatever reason times out (potentially an attacker). Then you get the
> >> > results for ns2, even though ns1 is intended to override it.
> >>
> >> I agree in theory.  And yet this is how most resolvers work today.
> >> Are they all broken?
> >
> > No, the resolvers are not broken. The configuration is, at least the
> > way I see it. The intent of the resolvers is that they're
> > communicating with redundant servers, not a sequence of overlaid
> > hostname databases. If that assumption is satisfied, there's no
> > problem.
> 
> The way you see it is not how everyone sees it, obviously.  And
> there's not a standard or spec here that I know of.  That said, as
> before, We can probably live with this assumption.

Indeed. But if you want to consider the overlaid model as the proper
one, then as we already noted, all existing implementations have an
exploitable bug where an attacker who can cause transient failures can
change the result of queries (rather than just making queries fail).
That seems rather undesirable.

> > BTW I think there are other strong reasons to move to a model based on
> > a local nameserver that does the unioning, not just performance. The
> > most compelling is DNSSEC, which requires a trusted channel between
> > the nameserver and the stub resolver in order for results to be
> > meaningful/trusted. In the future everybody should be running a
> > nameserver on localhost to do DNSSEC signature validation. In that
> > scheme, resolv.conf would just contain 127.0.0.1 (or could be omitted
> > entirely since that's the default, at least on musl).
> 
> I can see a local nameserver doing resolution, but doing search
> expansion seems like a stretch (and superfluous since it is local).

Search would also get a lot of performance benefit from doing in the
caching nameserver, but I agree with your assessment that it's a
separate issue and that there's no _need_ to do it at that level to
ensure correctness. So for now let's focus on a plan for adding
suitable search domain support in musl.

I believe search only affects DNS queries, not hosts file lookups,
right? So it should be at the name_from_dns stage in lookup_name.c.
The simplest implementation approach is probably to wrap name_from_dns
with a name_from_dns_search function that reads the search domains and
repeatedly calls name_from_dns until it gets success.

One oddity/ugliness of search that needs to be considered is that
querying different address families may lead to differently-sourced
results. For example if you query example.us with ndots=2, a search
domain of example.com, and the following records present:

	example.us.example.com A
	example.us AAAA

then an AF_UNSPEC or AF_INET query yields a v4-only result for
example.us.example.com while an AF_INET6 query "sees through" to the
example.us record because the search of example.com fails. This seems
quite ugly and counter-intuitive, but I don't see any way to do better
that makes sense.

Also, are we agreed that queries with >=ndots dots, or a final dot (as
in example.com.) should never be searched (rather than searched after
first trying them in the root)?

Rich

Re: Re: Would love to see reconsideration for domain and search

[Tim Hockin]

On Thu, Oct 22, 2015 at 10:31 PM, Rich Felker <dalias@libc.org> wrote:
>> > BTW I think there are other strong reasons to move to a model based on
>> > a local nameserver that does the unioning, not just performance. The
>> > most compelling is DNSSEC, which requires a trusted channel between
>> > the nameserver and the stub resolver in order for results to be
>> > meaningful/trusted. In the future everybody should be running a
>> > nameserver on localhost to do DNSSEC signature validation. In that
>> > scheme, resolv.conf would just contain 127.0.0.1 (or could be omitted
>> > entirely since that's the default, at least on musl).
>>
>> I can see a local nameserver doing resolution, but doing search
>> expansion seems like a stretch (and superfluous since it is local).
>
> Search would also get a lot of performance benefit from doing in the
> caching nameserver, but I agree with your assessment that it's a
> separate issue and that there's no _need_ to do it at that level to
> ensure correctness. So for now let's focus on a plan for adding
> suitable search domain support in musl.

Sounds right.

> I believe search only affects DNS queries, not hosts file lookups,

Also my understanding

> right? So it should be at the name_from_dns stage in lookup_name.c.
> The simplest implementation approach is probably to wrap name_from_dns
> with a name_from_dns_search function that reads the search domains and
> repeatedly calls name_from_dns until it gets success.
>
> One oddity/ugliness of search that needs to be considered is that
> querying different address families may lead to differently-sourced
> results. For example if you query example.us with ndots=2, a search
> domain of example.com, and the following records present:
>
>         example.us.example.com A
>         example.us AAAA
>
> then an AF_UNSPEC or AF_INET query yields a v4-only result for
> example.us.example.com while an AF_INET6 query "sees through" to the
> example.us record because the search of example.com fails. This seems
> quite ugly and counter-intuitive, but I don't see any way to do better
> that makes sense.

If that is how the records are laid out, I think you have to allow it.

> Also, are we agreed that queries with >=ndots dots, or a final dot (as
> in example.com.) should never be searched (rather than searched after
> first trying them in the root)?

That sounds right.

I saw from a different thread that musl doesn't or didn't do TCP
fallbacks - is that still the case?  I know we need that for things
like large multi-SRV sets (which I do not expect libc to support), and
we have some people who have large A sets (which I do expect libc to
support).

Re: Re: Would love to see reconsideration for domain and search

[Rich Felker]

On Thu, Oct 22, 2015 at 10:37:53PM -0700, Tim Hockin wrote:
> I saw from a different thread that musl doesn't or didn't do TCP
> fallbacks - is that still the case?  I know we need that for things
> like large multi-SRV sets (which I do not expect libc to support), and
> we have some people who have large A sets (which I do expect libc to
> support).

Indeed. The only way you can overflow the UDP size limit with the
records the stub resolver uses is with a max-length CNAME pointing to
a max or near-max length record with little or no overlap to allow for
compression. Of course you might run out of space for all the address
results in other cases, but the truncated packet will still have
usable results. While I'm not aware of any official document to this
effect, for practical purposes you just have to avoid making names
that long. There are too many nameservers that don't do TCP at all, as
well as locked-down networks that don't allow TCP except on a few
specific ports, to be able to rely on doing DNS over TCP.

Naturally other non-stub-resolver things like zone transfers may need
TCP, but that's outside the domain of the stub resolver. Note that the
libc res_*/dn_*/ns_* APIs should be capable of working with longer
messages over TCP as long as you setup the socket and do the send/recv
yourself.

Rich

Re: Re: Would love to see reconsideration for domain and search

[Tim Hockin]

On Thu, Oct 22, 2015 at 11:00 PM, Rich Felker <dalias@libc.org> wrote:
> On Thu, Oct 22, 2015 at 10:37:53PM -0700, Tim Hockin wrote:
>> I saw from a different thread that musl doesn't or didn't do TCP
>> fallbacks - is that still the case?  I know we need that for things
>> like large multi-SRV sets (which I do not expect libc to support), and
>> we have some people who have large A sets (which I do expect libc to
>> support).
>
> Indeed. The only way you can overflow the UDP size limit with the
> records the stub resolver uses is with a max-length CNAME pointing to
> a max or near-max length record with little or no overlap to allow for
> compression. Of course you might run out of space for all the address
> results in other cases, but the truncated packet will still have
> usable results. While I'm not aware of any official document to this
> effect, for practical purposes you just have to avoid making names
> that long. There are too many nameservers that don't do TCP at all, as
> well as locked-down networks that don't allow TCP except on a few
> specific ports, to be able to rely on doing DNS over TCP.

Our case is exposing sets of fungible backends as a DNS name with
multiple A records.  Truncating the set will cause incorrect results
for clients who need to discover the whole set.  We can cross that
bridge when we get there.

> Naturally other non-stub-resolver things like zone transfers may need
> TCP, but that's outside the domain of the stub resolver. Note that the
> libc res_*/dn_*/ns_* APIs should be capable of working with longer
> messages over TCP as long as you setup the socket and do the send/recv
> yourself.
>
> Rich

Re: Re: Would love to see reconsideration for domain and search

[u-uy74]

On Thu, Oct 22, 2015 at 11:00:32PM +0000, Josiah Worcester wrote:
> > On Thu, Oct 22, 2015 at 02:24:11PM -0700, Tim Hockin wrote:
> > But at least it works eventually.  You're faced with a choice.  Wait 2
> > seconds for ns1 to timeout and then fail in a way that most apps don't
> > handle well or wait for 2 seconds and then (usually) get a fast
> > response from ns2.
> >
> > It seems better in every way to eventually succeed, though I agree
> > it's a bit less visible.

> Consider what would happen if ns1 and ns2 have different responses, but ns1
> for whatever reason times out (potentially an attacker). Then you get the
> results for ns2, even though ns1 is intended to override it.

> > I have to disagree.  Some non-forwarding DNS servers use SERVFAIL to
> > indicate "I am not serving for that domain" specifically to make the
> > client move to their next nameserver.  if ns1 returns SERVFAIL, try
> > ns2.  If ns1 times out, try ns2.  Otherwise what good is ns2?

> Note that this means that any condition where ns1 can't be accessed changes
> what DNS resolves to. If you wish to prevent unexpected behavior when ns1
> can't be accessed, you simply have to get a response from ns1 first, and
> only ever query ns2 when the response from ns1 indicates you may.

> > Sure it's faster but it's WRONG.  Returning a random number would be
> > faster, too, but it is equally wrong.  This is why netstat (and myriad
> > other tools) has a `-n` flag.
> 
>  Again, musl's design assumes that all nameservers are hosting the same
> space, and thus a single nxdomain is authoritative. If this is the case,
> then this is perfectly correct. If it's not the case, then it's wrong (and
> you need to wait for literally everything to nxdomain, and if there's a
> timeout from a single server you *need* to report that).

> > search path.  I might run 100 or more containers on a single machine -
> > I can't run 100 DNS caches, and I can't put that back on users.
> 
> Why not? 100 DNS caches shouldn't be particularly expensive.

+1 to the level 1 comments

+1 to solving the particular problem of a certain deployment
   (the massive reliance on search paths)
   inside the deployment, not in a general purpose library

It looks unrealistic to both have redundancy and differing/conflicting
data on different servers in a consistent fashion, by the mere mechanisms
of resolv.conf.

Nice that musl chose consistency and redundancy.

Good if there is an option suitable for systems which need differing
functionality, but please not at an extra cost for other parties.

A DNS cache looks of course like the right option for fine tuning the
desired resolution behaviour.

Rune

Re: Re: Would love to see reconsideration for domain and search

[Laurent Bercot]

On 23/10/2015 10:12, u-uy74@aetey.se wrote:
> +1 to the level 1 comments
>
> +1 to solving the particular problem of a certain deployment
>     (the massive reliance on search paths)
>     inside the deployment, not in a general purpose library
>
> It looks unrealistic to both have redundancy and differing/conflicting
> data on different servers in a consistent fashion, by the mere mechanisms
> of resolv.conf.
>
> Nice that musl chose consistency and redundancy.
>
> Good if there is an option suitable for systems which need differing
> functionality, but please not at an extra cost for other parties.
>
> A DNS cache looks of course like the right option for fine tuning the
> desired resolution behaviour.

  +1 to all this.
  "search" in /etc/resolv.conf was *never* meant to list resolvers
serving different results. It was always meant to list backups
serving consistent results. Any other use of "search" lines is
abusing the mechanism; if it works with glibc, it is a testimony
that glibc doesn't care about consistency, not an argument that it
is a valid use case. The right way to proceed is to implement an
ad-hoc resolver that suits the specific wanted usage, not to
modify a libc that implements the proper behaviour.

-- 
  Laurent

Re: Re: Would love to see reconsideration for domain and search

[Laurent Bercot]

On 23/10/2015 11:35, Laurent Bercot wrote:
>   "search" in /etc/resolv.conf was *never* meant to list resolvers
> serving different results.

  I meant "nameserver", of course. :)

  ("search" is also abused, but that's another topic.)

-- 
  Laurent

Re: Re: Would love to see reconsideration for domain and search

[Tim Hockin]

On Fri, Oct 23, 2015 at 2:35 AM, Laurent Bercot
<ska-dietlibc@skarnet.org> wrote:
> On 23/10/2015 10:12, u-uy74@aetey.se wrote:
>>
>> +1 to the level 1 comments
>>
>> +1 to solving the particular problem of a certain deployment
>>     (the massive reliance on search paths)
>>     inside the deployment, not in a general purpose library
>>
>> It looks unrealistic to both have redundancy and differing/conflicting
>> data on different servers in a consistent fashion, by the mere mechanisms
>> of resolv.conf.
>>
>> Nice that musl chose consistency and redundancy.
>>
>> Good if there is an option suitable for systems which need differing
>> functionality, but please not at an extra cost for other parties.
>>
>> A DNS cache looks of course like the right option for fine tuning the
>> desired resolution behaviour.
>
>
>  +1 to all this.
>  "search" in /etc/resolv.conf was *never* meant to list resolvers
> serving different results. It was always meant to list backups

citation needed

Re: Re: Would love to see reconsideration for domain and search

[Tim Hockin]

[-- Attachment #1: Type: text/plain, Size: 1666 bytes --]

On Oct 24, 2015 12:20 PM, "Kurt H Maier" <khm@sdf.org> wrote:
>
> On Thu, Oct 22, 2015 at 02:24:11PM -0700, Tim Hockin wrote:
> >
> > I understand your point, though the world at large tends to disagree.
>
> The world at large uses bad software.  Please don't use this sort of
> reasoning as a justification for and embrace-extend operation on actual
> standards.

Where is the standard that defines ordering semantics in resolv.conf?

> > The real world is not ideal.  Not all nameservers are identically
> > scoped - you MUST respect the ordering in resolv.conf - to do
> > otherwise is semantically broken.  If implementation simplicity means
> > literally doing queries in serial, then that is what you should do.
>
> You absolutely cannot respect the ordering in resolv.conf; at least not
> if you're relying on someone else's resolver.  If the orchestration
> software depends on specific results being returned in particular
> orders, the orchestration software should provide a mechanism to
> generate them.
>
> > Similarly, you can't just search all search domains in parallel and
> > take the first response.  The ordering is meaningful.
>
> It should not be, and more to the point will not reliably be,
> meaningful.

Search has to be ordered.  You can not possibly argue otherwise?

> You are arguing for introducing performance penalties into musl that do
> not affect you but do very much affect lots of other users.  I hope they
> do not happen -- musl is not the right place to fix your problem.

I am arguing for adding a very standard feature (search) to open musl to a
whole new space of users. Nobody is forcing you to use search paths or
ndots.

[-- Attachment #2: Type: text/html, Size: 2083 bytes --]

Re: Re: Would love to see reconsideration for domain and search

[Kurt H Maier]

On Sat, Oct 24, 2015 at 02:33:31PM -0700, Tim Hockin wrote:
> Where is the standard that defines ordering semantics in resolv.conf?

There isn't one, but that doesn't mean you get to ensconce the behavior
you like.  That was my point.

> Search has to be ordered.  You can not possibly argue otherwise?

I can and do.  

> I am arguing for adding a very standard feature (search) to open musl to a
> whole new space of users.

You're not arguing for simple inclusion of search; you're arguing for
specific behavior that is impossible to implement in a sane manner,
because that's how glibc does it.  I am not against domain and search, 
but I don't just want functionality poorly added just to check some box 
on a random other project's wishlist. 

> Nobody is forcing you to use search paths or ndots.

Not sure how this is relevant?  Nobody's forcing anyone to do anything.
What is your point?

khm

Re: Re: Would love to see reconsideration for domain and search

[Rich Felker]

On Sat, Oct 24, 2015 at 02:33:31PM -0700, Tim Hockin wrote:
> On Oct 24, 2015 12:20 PM, "Kurt H Maier" <khm@sdf.org> wrote:
> >
> > On Thu, Oct 22, 2015 at 02:24:11PM -0700, Tim Hockin wrote:
> > >
> > > I understand your point, though the world at large tends to disagree.
> >
> > The world at large uses bad software.  Please don't use this sort of
> > reasoning as a justification for and embrace-extend operation on actual
> > standards.
> 
> Where is the standard that defines ordering semantics in resolv.conf?

I don't think it's useful to argue about intent unless someone wants
to dig up history and find out what the original implementors
intended, and even then it's rather arbitrary whether people would
care about that intent since it doesn't seem to have been documented
explicitly. My view is that it's more useful to consider the
consequences of both interpretations and draw a conclusion that one
should be preferred from the bad consequences of the other.

> > > The real world is not ideal.  Not all nameservers are identically
> > > scoped - you MUST respect the ordering in resolv.conf - to do
> > > otherwise is semantically broken.  If implementation simplicity means
> > > literally doing queries in serial, then that is what you should do.
> >
> > You absolutely cannot respect the ordering in resolv.conf; at least not
> > if you're relying on someone else's resolver.  If the orchestration
> > software depends on specific results being returned in particular
> > orders, the orchestration software should provide a mechanism to
> > generate them.
> >
> > > Similarly, you can't just search all search domains in parallel and
> > > take the first response.  The ordering is meaningful.
> >
> > It should not be, and more to the point will not reliably be,
> > meaningful.
> 
> Search has to be ordered.  You can not possibly argue otherwise?

Indeed, search certainly has to be ordered. Otherwise results are most
definitely non-deterministic. The trivial example would be looking up
"www" with 2 or more search domains.

In any case, it was discussed way back that, while parallel search
could be implemented as long as a result from search domain N is not
accepted until negative results from domains 1 to N-1 are received,
the implementation complexity cost was too high relative to the value
of such a feature.

> > You are arguing for introducing performance penalties into musl that do
> > not affect you but do very much affect lots of other users.  I hope they
> > do not happen -- musl is not the right place to fix your problem.
> 
> I am arguing for adding a very standard feature (search) to open musl to a
> whole new space of users. Nobody is forcing you to use search paths or
> ndots.

The only place adding search support might negatively impact existing
musl users is by causing hostnames with no dots to be queried with the
(often useless and unwanted) default domain set by dhcp before
failing. My preference would probably be having musl default to
ndots=0 rather than ndots=1 so that search has to be enabled
explicitly. Are there any reasons this would be undesirable?

Rich

Re: Re: Would love to see reconsideration for domain and search

[Tim Hockin]

[-- Attachment #1: Type: text/plain, Size: 3763 bytes --]

On Oct 24, 2015 3:02 PM, "Rich Felker" <dalias@libc.org> wrote:
>
> On Sat, Oct 24, 2015 at 02:33:31PM -0700, Tim Hockin wrote:
> > On Oct 24, 2015 12:20 PM, "Kurt H Maier" <khm@sdf.org> wrote:
> > >
> > > On Thu, Oct 22, 2015 at 02:24:11PM -0700, Tim Hockin wrote:
> > > >
> > > > I understand your point, though the world at large tends to
disagree.
> > >
> > > The world at large uses bad software.  Please don't use this sort of
> > > reasoning as a justification for and embrace-extend operation on
actual
> > > standards.
> >
> > Where is the standard that defines ordering semantics in resolv.conf?
>
> I don't think it's useful to argue about intent unless someone wants
> to dig up history and find out what the original implementors

My point was only that you can't call this "embrace and extend" of a
"standard" when no such standard exists AND the most widely used software
doesn't conform to your supposed standard.

I can see both sides of the argument and have already conceded this as the
least important part of the proposal, for our use case.

> intended, and even then it's rather arbitrary whether people would
> care about that intent since it doesn't seem to have been documented
> explicitly. My view is that it's more useful to consider the
> consequences of both interpretations and draw a conclusion that one
> should be preferred from the bad consequences of the other.
>
> > > > The real world is not ideal.  Not all nameservers are identically
> > > > scoped - you MUST respect the ordering in resolv.conf - to do
> > > > otherwise is semantically broken.  If implementation simplicity
means
> > > > literally doing queries in serial, then that is what you should do.
> > >
> > > You absolutely cannot respect the ordering in resolv.conf; at least
not
> > > if you're relying on someone else's resolver.  If the orchestration
> > > software depends on specific results being returned in particular
> > > orders, the orchestration software should provide a mechanism to
> > > generate them.
> > >
> > > > Similarly, you can't just search all search domains in parallel and
> > > > take the first response.  The ordering is meaningful.
> > >
> > > It should not be, and more to the point will not reliably be,
> > > meaningful.
> >
> > Search has to be ordered.  You can not possibly argue otherwise?
>
> Indeed, search certainly has to be ordered. Otherwise results are most
> definitely non-deterministic. The trivial example would be looking up
> "www" with 2 or more search domains.

OK, I thought for a minute you went off the deep end :)

> In any case, it was discussed way back that, while parallel search
> could be implemented as long as a result from search domain N is not
> accepted until negative results from domains 1 to N-1 are received,
> the implementation complexity cost was too high relative to the value
> of such a feature.

Agree

> > > You are arguing for introducing performance penalties into musl that
do
> > > not affect you but do very much affect lots of other users.  I hope
they
> > > do not happen -- musl is not the right place to fix your problem.
> >
> > I am arguing for adding a very standard feature (search) to open musl
to a
> > whole new space of users. Nobody is forcing you to use search paths or
> > ndots.
>
> The only place adding search support might negatively impact existing
> musl users is by causing hostnames with no dots to be queried with the
> (often useless and unwanted) default domain set by dhcp before
> failing. My preference would probably be having musl default to
> ndots=0 rather than ndots=1 so that search has to be enabled
> explicitly. Are there any reasons this would be undesirable?

So you want "naked" names to not use search at all?  Is that really useful?

[-- Attachment #2: Type: text/html, Size: 4825 bytes --]

Re: Re: Would love to see reconsideration for domain and search

[Rich Felker]

On Sat, Oct 24, 2015 at 05:57:10PM -0400, Kurt H Maier wrote:
> On Sat, Oct 24, 2015 at 02:33:31PM -0700, Tim Hockin wrote:
> > Where is the standard that defines ordering semantics in resolv.conf?
> 
> There isn't one, but that doesn't mean you get to ensconce the behavior
> you like.  That was my point.
> 
> > Search has to be ordered.  You can not possibly argue otherwise?
> 
> I can and do.  
> 
> > I am arguing for adding a very standard feature (search) to open musl to a
> > whole new space of users.
> 
> You're not arguing for simple inclusion of search;

At this point that actually does seem to be all that Tim is asking
for. I think we're in agreement that there are multiple problems with
trying to use ordered nameservers to overlay inconsistent data, and
that even under the way glibc treats this setup, there are subtle
problems.

> you're arguing for
> specific behavior that is impossible to implement in a sane manner,
> because that's how glibc does it.  I am not against domain and search, 
> but I don't just want functionality poorly added just to check some box 
> on a random other project's wishlist. 
>
> > Nobody is forcing you to use search paths or ndots.
> 
> Not sure how this is relevant?  Nobody's forcing anyone to do anything.
> What is your point?

Could you try to hold off on the hostility? I don't think there's any
actual disagreement left here. Support for search domains was tabled
but left open for future consideration back when the last phase of dns
overhaul was done, pending actual requests/usage-cases needing them.
It seems like they can be added in an inexpensive and ubobtrusive way,
so I don't think it should really be controversial.

Note that, as I said before, search really does have to be ordered.
Otherwise you have inconsistent/non-deterministic results that can be
controlled by inducing intermittent failures (DoS). The proper way to
handle search is the same way the current fallback sequence (ip
literal, hosts file, dns) is done: a positive result at any step ends
the query, a negative result causes fallback to the next search
element, and any failure at any stage causes the whole query to fail
with an appropriate error.

Rich

Re: Re: Would love to see reconsideration for domain and search

[u-uy74]

On Sat, Oct 24, 2015 at 06:02:15PM -0400, Rich Felker wrote:
> The only place adding search support might negatively impact existing
> musl users is by causing hostnames with no dots to be queried with the
> (often useless and unwanted) default domain set by dhcp before
> failing. My preference would probably be having musl default to
> ndots=0 rather than ndots=1 so that search has to be enabled
> explicitly.

+1

> Are there any reasons this would be undesirable?

I don't think so.

On the other side, the "design" of resolv.conf is far from well thought
out or consequent (given that there is no standard, to begin with).
So whatever you do, there of course will be people expecting/wanting a
different behaviour. Sigh.

Rune

Re: Re: Would love to see reconsideration for domain and search

[Jan Broer]

[-- Attachment #1: Type: text/plain, Size: 4594 bytes --]

> The only place adding search support might negatively impact existing
> musl users is by causing hostnames with no dots to be queried with the
> (often useless and unwanted) default domain set by dhcp before
> failing. My preference would probably be having musl default to
> ndots=0 rather than ndots=1 so that search has to be enabled
> explicitly. Are there any reasons this would be undesirable?


I don't think it is a good idea do default to ndots=0. This would
essentially break search for systems where resolv.conf values are managed
by the DHCP server. DHCP expects search to work when there is at least one
entry in the domain-search option returned by the DHCP server. There is no
DHCP option for configuring ndots (see
http://linux.die.net/man/5/dhcp-options) and therefore search would not
work in these configurations when ndots defaults to 0.

Also, i don't think your argument for setting ndots to 0 by default is
valid:

Search always to be enabled explicitly even when the default for ndots=1:
There is no search unless the user explicitely writes search paths in his
resolv.conf or configures his DHCP client to get the domain-search option
via DHCP.
So the resolvers search functionality is already something the user has to
enable explicitely.


On Sun, Oct 25, 2015 at 12:02 AM, Rich Felker <dalias@libc.org> wrote:

> On Sat, Oct 24, 2015 at 02:33:31PM -0700, Tim Hockin wrote:
> > On Oct 24, 2015 12:20 PM, "Kurt H Maier" <khm@sdf.org> wrote:
> > >
> > > On Thu, Oct 22, 2015 at 02:24:11PM -0700, Tim Hockin wrote:
> > > >
> > > > I understand your point, though the world at large tends to disagree.
> > >
> > > The world at large uses bad software.  Please don't use this sort of
> > > reasoning as a justification for and embrace-extend operation on actual
> > > standards.
> >
> > Where is the standard that defines ordering semantics in resolv.conf?
>
> I don't think it's useful to argue about intent unless someone wants
> to dig up history and find out what the original implementors
> intended, and even then it's rather arbitrary whether people would
> care about that intent since it doesn't seem to have been documented
> explicitly. My view is that it's more useful to consider the
> consequences of both interpretations and draw a conclusion that one
> should be preferred from the bad consequences of the other.
>
> > > > The real world is not ideal.  Not all nameservers are identically
> > > > scoped - you MUST respect the ordering in resolv.conf - to do
> > > > otherwise is semantically broken.  If implementation simplicity means
> > > > literally doing queries in serial, then that is what you should do.
> > >
> > > You absolutely cannot respect the ordering in resolv.conf; at least not
> > > if you're relying on someone else's resolver.  If the orchestration
> > > software depends on specific results being returned in particular
> > > orders, the orchestration software should provide a mechanism to
> > > generate them.
> > >
> > > > Similarly, you can't just search all search domains in parallel and
> > > > take the first response.  The ordering is meaningful.
> > >
> > > It should not be, and more to the point will not reliably be,
> > > meaningful.
> >
> > Search has to be ordered.  You can not possibly argue otherwise?
>
> Indeed, search certainly has to be ordered. Otherwise results are most
> definitely non-deterministic. The trivial example would be looking up
> "www" with 2 or more search domains.
>
> In any case, it was discussed way back that, while parallel search
> could be implemented as long as a result from search domain N is not
> accepted until negative results from domains 1 to N-1 are received,
> the implementation complexity cost was too high relative to the value
> of such a feature.
>
> > > You are arguing for introducing performance penalties into musl that do
> > > not affect you but do very much affect lots of other users.  I hope
> they
> > > do not happen -- musl is not the right place to fix your problem.
> >
> > I am arguing for adding a very standard feature (search) to open musl to
> a
> > whole new space of users. Nobody is forcing you to use search paths or
> > ndots.
>
> The only place adding search support might negatively impact existing
> musl users is by causing hostnames with no dots to be queried with the
> (often useless and unwanted) default domain set by dhcp before
> failing. My preference would probably be having musl default to
> ndots=0 rather than ndots=1 so that search has to be enabled
> explicitly. Are there any reasons this would be undesirable?
>
> Rich
>

[-- Attachment #2: Type: text/html, Size: 5763 bytes --]

Re: Re: Would love to see reconsideration for domain and search

[u-uy74]

On Sun, Oct 25, 2015 at 02:06:29PM +0100, Jan Broer wrote:
> I don't think it is a good idea do default to ndots=0. This would
> essentially break search for systems where resolv.conf values are managed
> by the DHCP server. DHCP expects search to work when there is at least one
> entry in the domain-search option returned by the DHCP server. There is no
> DHCP option for configuring ndots (see
> http://linux.die.net/man/5/dhcp-options) and therefore search would not
> work in these configurations when ndots defaults to 0.

To be fair, it is not the dhcp server who generates the resolv.conf but
the tools on the computer itself (the dhcp client implementation).
IOW it is the administrator of the computer who decides what resolv.conf
shall look like, even if she/he possibly uses certain data from dhcp.

Rune

Re: Re: Would love to see reconsideration for domain and search

[Jan Broer]

[-- Attachment #1: Type: text/plain, Size: 1236 bytes --]

I don't think we disagree there. The user decides whether search is used by
either setting the search keyword in resolv.conf or by configuring his DHCP
client (dhclient.conf) to fetch the domain-search option from the DHCP
server.
Thats why there really is no need to misuse ndots option as "on/off switch"
for searching.

On Sun, Oct 25, 2015 at 2:19 PM, <u-uy74@aetey.se> wrote:

> On Sun, Oct 25, 2015 at 02:06:29PM +0100, Jan Broer wrote:
> > I don't think it is a good idea do default to ndots=0. This would
> > essentially break search for systems where resolv.conf values are managed
> > by the DHCP server. DHCP expects search to work when there is at least
> one
> > entry in the domain-search option returned by the DHCP server. There is
> no
> > DHCP option for configuring ndots (see
> > http://linux.die.net/man/5/dhcp-options) and therefore search would not
> > work in these configurations when ndots defaults to 0.
>
> To be fair, it is not the dhcp server who generates the resolv.conf but
> the tools on the computer itself (the dhcp client implementation).
> IOW it is the administrator of the computer who decides what resolv.conf
> shall look like, even if she/he possibly uses certain data from dhcp.
>
> Rune
>
>

[-- Attachment #2: Type: text/html, Size: 1739 bytes --]

Re: Re: Would love to see reconsideration for domain and search

[u-uy74]

On Sun, Oct 25, 2015 at 02:39:36PM +0100, Jan Broer wrote:
> I don't think we disagree there. The user decides whether search is used by
> either setting the search keyword in resolv.conf or by configuring his DHCP
> client (dhclient.conf) to fetch the domain-search option from the DHCP
> server.

You seem to assume that the user can not influence the ndots option in the
resolv.conf if the corresponding host is to use some other data delivered
by dhcp (i.e. for building resolv.conf).

For certain setups this can be true, probably meaning that they [rely
on a non-existent standard and] do not bother having proper control
over resolv.conf?

Otherwise, in a general case, we disagree.

Isn't it reasonable that people wishing to use extra and
"somewhat/potentially expensive" features of resolv.conf have to put
effort into actually activating them?

Regards,
Rune

Re: Re: Would love to see reconsideration for domain and search

[Rich Felker]

On Sun, Oct 25, 2015 at 02:06:29PM +0100, Jan Broer wrote:
> > The only place adding search support might negatively impact existing
> > musl users is by causing hostnames with no dots to be queried with the
> > (often useless and unwanted) default domain set by dhcp before
> > failing. My preference would probably be having musl default to
> > ndots=0 rather than ndots=1 so that search has to be enabled
> > explicitly. Are there any reasons this would be undesirable?
> 
> 
> I don't think it is a good idea do default to ndots=0. This would
> essentially break search for systems where resolv.conf values are managed
> by the DHCP server. DHCP expects search to work when there is at least one
> entry in the domain-search option returned by the DHCP server. There is no
> DHCP option for configuring ndots (see
> http://linux.die.net/man/5/dhcp-options) and therefore search would not
> work in these configurations when ndots defaults to 0.

What I've found is that almost all dhcp setups include a search
domain, and dhcp clients set it in resolv.conf, but it's rare that the
user actually wants to search this domain. For example do Comcast
(random ISP to pick on) customers actually want random bare (no
domain) queries like "foo" to be tried as "foo.comcast.com"? Almost
surely not.

Of course, as noted before, only bare queries are affected if ndots=1,
so the only behavioral change is that some such queries may
unexpectedly succeed (if they exist in the search domain) or take
longer to fail (since two or more queries have to fail).

> Also, i don't think your argument for setting ndots to 0 by default is
> valid:
> 
> Search always to be enabled explicitly even when the default for ndots=1:
> There is no search unless the user explicitely writes search paths in his
> resolv.conf or configures his DHCP client to get the domain-search option
> via DHCP.
> 
> So the resolvers search functionality is already something the user has to
> enable explicitely.

My experience is that dhcp clients do this by default. Of course
there's a good argument to be made that fixing this should be done at
the dhcp client level (or resolv.conf updating system level) rather
than having libc try to 'fix' it. There's certainly value to having
consistent behavior between programs using different libcs on the same
system, so perhaps sticking with ndots=1 is better.

Rich

Re: Re: Would love to see reconsideration for domain and search

[Isaac Dunham]

On Sat, Oct 24, 2015 at 06:02:15PM -0400, Rich Felker wrote:
> The only place adding search support might negatively impact existing
> musl users is by causing hostnames with no dots to be queried with the
> (often useless and unwanted) default domain set by dhcp before
> failing. My preference would probably be having musl default to
> ndots=0 rather than ndots=1 so that search has to be enabled
> explicitly. Are there any reasons this would be undesirable?

Could you explain what this all would mean to someone who has only a
general understanding of how DNS works, a home network, and a desire
to set up a local DNS server?

I have a couple use-cases in mind, which I think involve either the
"search" or "domain" keywords in resolv.conf; I'll describe them in case
they pertain.

1: the university I attended had a bunch of resources which were available
as sub-domains. The way DHCP/DNS/resolv.conf was set up, local sites
(for example, http://myweb.csuchico.edu/) were available using only the
bare subdomain (in the same example, http://myweb/); I forget whether
they used "domain" or "search" for this. This was something I
appreciated.

2: On my home network (configured via DHCP, no DNS server yet), I have a
network printer that advertises itself using a name in the general format
of MFC0000DEADBEEF (where 0000DEADBEEF is the MAC address, stripped of
separators). It uses this hostname for DHCP and avahi.
However, this name is only available for avahi clients like cups; I
want to be able to access it by the same name from non-avahi programs
like 'ping', 'links', etcetera, since it's rather annoying to have a
'magic' name that usually works in your printing daemon, but no way
to map it to an IP for any troupleshooting/configuration tools.

Currently, I've got it set up so the printer has a static lease and I
thus know the IP, but I want to use DNS because that's the only way all
tools on all computers on the network will automatically know that
MFC0000DEADBEEF is (for example) IP 192.168.255.255.
It would be rather annoying to have some tools where I can use that name,
and some where I need to specify "mfc0000deadbeef.local" instead.

Thanks,
Isaac Dunham

Re: Re: Would not love to see reconsideration for domain and search

[John Levine]

>BTW I think there are other strong reasons to move to a model based on
>a local nameserver that does the unioning, not just performance. The
>most compelling is DNSSEC, which requires a trusted channel between
>the nameserver and the stub resolver in order for results to be
>meaningful/trusted. ...

Yes, definitely.

DNS search lists seemed like a good idea back in the 1980s.  Then in
1990 they added .CS for Czechoslovakia to the DNS root, and in
Computer Science departments all over the world, addresses like
joe@frodo.cs stopped working, since the search list that used to turn
it into joe@frodo.cs.stateu.edu didn't do that any more.

ICANN has added about 600 new top level domains in the past two years,
There's still nearly a thousand more in the pipeline, and they're
talking about another round that will add thousands more.  I went to a
two day meeting about name collisions after the London ICANN meeting,
and a great deal of the discussion was about how to flush out old
search list queries before they started resolving wrong.

If you want to have a local namespace overlaid on the DNS, it is not
hard to configure bind or unbound to do that so, e.g. names in
whatever.blah resolve locally.  You can even configure in local DNSSEC
anchors for .blah if you want.  In that case if there's ever a global
.blah TLD, your local users won't be able to see it, but your local
applications will keep working.

I'd strongly suggest that the lack of DNS search lists is a feature,
and not to change it.

R's,
John

Re: Re: Would not love to see reconsideration for domain and search

[Tim Hockin]

Maybe I am confused..

On Sun, Oct 25, 2015 at 7:14 PM, John Levine <johnl@iecc.com> wrote:

> DNS search lists seemed like a good idea back in the 1980s.  Then in
> 1990 they added .CS for Czechoslovakia to the DNS root, and in
> Computer Science departments all over the world, addresses like
> joe@frodo.cs stopped working, since the search list that used to turn
> it into joe@frodo.cs.stateu.edu didn't do that any more.

If I have a search path of "stateu.edu" and ndots=2, and I resolve
"frodo.cs", wouldn't that FIRST try frodo.cs.stateu.edu. BEFORE
frodo.cs. ?

Re: Re: Would love to see reconsideration for domain and search

[Rich Felker]

On Sun, Oct 25, 2015 at 06:26:43PM -0700, Isaac Dunham wrote:
> On Sat, Oct 24, 2015 at 06:02:15PM -0400, Rich Felker wrote:
> > The only place adding search support might negatively impact existing
> > musl users is by causing hostnames with no dots to be queried with the
> > (often useless and unwanted) default domain set by dhcp before
> > failing. My preference would probably be having musl default to
> > ndots=0 rather than ndots=1 so that search has to be enabled
> > explicitly. Are there any reasons this would be undesirable?
> 
> Could you explain what this all would mean to someone who has only a
> general understanding of how DNS works, a home network, and a desire
> to set up a local DNS server?
> 
> I have a couple use-cases in mind, which I think involve either the
> "search" or "domain" keywords in resolv.conf; I'll describe them in case
> they pertain.
> 
> 1: the university I attended had a bunch of resources which were available
> as sub-domains. The way DHCP/DNS/resolv.conf was set up, local sites
> (for example, http://myweb.csuchico.edu/) were available using only the
> bare subdomain (in the same example, http://myweb/); I forget whether
> they used "domain" or "search" for this. This was something I
> appreciated.

Yes, this is what adding search domain support would allow. Note that
with ndots=1, "myweb" would work, but "myweb.math" would not.

> 2: On my home network (configured via DHCP, no DNS server yet), I have a
> network printer that advertises itself using a name in the general format
> of MFC0000DEADBEEF (where 0000DEADBEEF is the MAC address, stripped of
> separators). It uses this hostname for DHCP and avahi.
> However, this name is only available for avahi clients like cups; I
> want to be able to access it by the same name from non-avahi programs
> like 'ping', 'links', etcetera, since it's rather annoying to have a
> 'magic' name that usually works in your printing daemon, but no way
> to map it to an IP for any troupleshooting/configuration tools.
> 
> Currently, I've got it set up so the printer has a static lease and I
> thus know the IP, but I want to use DNS because that's the only way all
> tools on all computers on the network will automatically know that
> MFC0000DEADBEEF is (for example) IP 192.168.255.255.
> It would be rather annoying to have some tools where I can use that name,
> and some where I need to specify "mfc0000deadbeef.local" instead.

Doing this requires dunning a local nameserver that joins the avahi
(mdns, I think) results under some domain you control with the global
dns namespace. I suspect dnsmasq or one of the other dns servers made
for home network use can do this but I haven't checked. In principle
it's even possible with BIND and some scripting to query mdns and use
the results to update dynamic dns entries but that's rather ugly.

Rich

Re: Re: Would not love to see reconsideration for domain and search

[Rich Felker]

On Sun, Oct 25, 2015 at 10:14:42PM -0700, Tim Hockin wrote:
> Maybe I am confused..
> 
> On Sun, Oct 25, 2015 at 7:14 PM, John Levine <johnl@iecc.com> wrote:
> 
> > DNS search lists seemed like a good idea back in the 1980s.  Then in
> > 1990 they added .CS for Czechoslovakia to the DNS root, and in
> > Computer Science departments all over the world, addresses like
> > joe@frodo.cs stopped working, since the search list that used to turn
> > it into joe@frodo.cs.stateu.edu didn't do that any more.
> 
> If I have a search path of "stateu.edu" and ndots=2, and I resolve
> "frodo.cs", wouldn't that FIRST try frodo.cs.stateu.edu. BEFORE
> frodo.cs. ?

I suspect John was using a configuration with ndots=1, where the
legacy behavior for queries with dots>=ndots was to first try the
global scope, then fallback to search domains. And this example is
(aside from nasty performance hit, which also matters) precisely why I
think supporting this fallback is a bad idea: changes to the global
dns scope, outside of your control, can change the results you get for
things that you thought were under your control.

Without this fallback, you would need ndots=2 to get the "desired"
behavior, and addition of the .cs TLD would not have broken it. Of
course you'd be blocking access to a number of real global-scope .cs
domains, which would be bad policy, but at least stable.

Note that ndots=1 search is rather harmless as long as ICANN has a
prohibition on top-level domains resolving to an address.

Rich

Re: Re: Would not love to see reconsideration for domain and search

[John Levine]

>Note that ndots=1 search is rather harmless as long as ICANN has a
>prohibition on top-level domains resolving to an address.

That horse left the barn over 15 years ago:

https://www.rfc-editor.org/info/rfc7085

ICANN currently has a rule against it for generic TLDs, but they have
no control over two-letter country codes, and as our RFC notes, a lot
of ccTLDs have had A and MX records.

I say currently because Google asked for an exception to put an A
record to make http://search/ work, and it took some discussion before
ICANN said no.  The no was as much about anti-competitive reasons, the
default would be to Google's search engine, as the technical issues.
If someeone else asked, they'd probably say no, but it's not cast in
stone.

In response to another question about search order, the default value
of ndots is 1, so any domain name with at least one dot, such as
frodo.cs, is looked up directly before it tries a search list.  You
can set ndots to anything you want, but I expect that your users would
not be happy if gmail.com and yahoo.com could be shadowed by local host
names.

R's,
John

PS: Just for the record, the world's shortest e-mail address is n@ai.
I'm j@jl.ly, but that's almost twice as long.

Re: Re: Would not love to see reconsideration for domain and search

[Rich Felker]

On Mon, Oct 26, 2015 at 05:41:48PM -0000, John Levine wrote:
> >Note that ndots=1 search is rather harmless as long as ICANN has a
> >prohibition on top-level domains resolving to an address.
> 
> That horse left the barn over 15 years ago:
> 
> https://www.rfc-editor.org/info/rfc7085
> 
> ICANN currently has a rule against it for generic TLDs, but they have
> no control over two-letter country codes, and as our RFC notes, a lot
> of ccTLDs have had A and MX records.

This is probably an argument for a default of ndots=0, to avoid a
regression looking up such names.

> I say currently because Google asked for an exception to put an A
> record to make http://search/ work, and it took some discussion before
> ICANN said no.  The no was as much about anti-competitive reasons, the
> default would be to Google's search engine, as the technical issues.
> If someeone else asked, they'd probably say no, but it's not cast in
> stone.
> 
> In response to another question about search order, the default value
> of ndots is 1, so any domain name with at least one dot, such as
> frodo.cs, is looked up directly before it tries a search list.  You

This is the behavior on glibc and most/all other legacy
implementations. The proposed musl behavior would not search at all in
the dots>=ndots case.

> can set ndots to anything you want, but I expect that your users would
> not be happy if gmail.com and yahoo.com could be shadowed by local host
> names.

Indeed, users setting up search domains and ndots>1 need to be careful
that they control the contents of those domains and that they don't
shadow anything needed from the global scope. This was a lot easier
before the ridiculous arbitrary-string TLDs were added. IMO use of
search domains is pretty much an outdated idea (that poses serious
risks of future breakage) but without the global-to-search fallback
that legacy implementions have (i.e. with the proposed musl
implementation), at least the risk of breakage is isolated to
inability to resolve _new_ domains rather than failure to resolve
existing ones that previously worked.

Rich

Re: Re: Would love to see reconsideration for domain and search

[Rich Felker]

On Fri, Oct 23, 2015 at 01:31:09AM -0400, Rich Felker wrote:
> > > BTW I think there are other strong reasons to move to a model based on
> > > a local nameserver that does the unioning, not just performance. The
> > > most compelling is DNSSEC, which requires a trusted channel between
> > > the nameserver and the stub resolver in order for results to be
> > > meaningful/trusted. In the future everybody should be running a
> > > nameserver on localhost to do DNSSEC signature validation. In that
> > > scheme, resolv.conf would just contain 127.0.0.1 (or could be omitted
> > > entirely since that's the default, at least on musl).
> > 
> > I can see a local nameserver doing resolution, but doing search
> > expansion seems like a stretch (and superfluous since it is local).
> 
> Search would also get a lot of performance benefit from doing in the
> caching nameserver, but I agree with your assessment that it's a
> separate issue and that there's no _need_ to do it at that level to
> ensure correctness. So for now let's focus on a plan for adding
> suitable search domain support in musl.
> 
> I believe search only affects DNS queries, not hosts file lookups,
> right? So it should be at the name_from_dns stage in lookup_name.c.
> The simplest implementation approach is probably to wrap name_from_dns
> with a name_from_dns_search function that reads the search domains and
> repeatedly calls name_from_dns until it gets success.

I noticed in the process of trying to draft code to do this that there
will be a lot of code duplication with the resolv.conf parsing in
res_msend.c, and that this code has some stupid bugs (for example it
stops parsing after it gets 3 nameservers, so it might miss options
later in the file), so I think I'll take a look at factoring it into a
new function to gather all the interesting information from
resolv.conf that can be used in both places.

A couple additional things I noticed from resolv.conf(5):

1. The default domain used by glibc is not the dns root but rather the
   domain portion of the local hostname determined by gethostname().
   Is there any value in duplicating this? Does anyone want/need it?

2. It's not clear from the documentation of "search" whether its
   presence overrides/suppresses the "domain" (default or set by
   resolv.conf) or adds additional searches before or after it. Which
   should it do?

While glibc/legacy behavior is worth looking at, I don't think we need
to look at things from a standpoint of exactly duplicating that.
Meeting real-world modern application needs while avoiding
inconveniencing users with stupid/unwanted behavior should be the
primary goal.

Rich

Re: Re: Would love to see reconsideration for domain and search

[Tim Hockin]

wrt 2) my understanding is that you get at-most-one-of `search` or `domain`.

On Mon, Oct 26, 2015 at 5:30 PM, Rich Felker <dalias@libc.org> wrote:
> On Fri, Oct 23, 2015 at 01:31:09AM -0400, Rich Felker wrote:
>> > > BTW I think there are other strong reasons to move to a model based on
>> > > a local nameserver that does the unioning, not just performance. The
>> > > most compelling is DNSSEC, which requires a trusted channel between
>> > > the nameserver and the stub resolver in order for results to be
>> > > meaningful/trusted. In the future everybody should be running a
>> > > nameserver on localhost to do DNSSEC signature validation. In that
>> > > scheme, resolv.conf would just contain 127.0.0.1 (or could be omitted
>> > > entirely since that's the default, at least on musl).
>> >
>> > I can see a local nameserver doing resolution, but doing search
>> > expansion seems like a stretch (and superfluous since it is local).
>>
>> Search would also get a lot of performance benefit from doing in the
>> caching nameserver, but I agree with your assessment that it's a
>> separate issue and that there's no _need_ to do it at that level to
>> ensure correctness. So for now let's focus on a plan for adding
>> suitable search domain support in musl.
>>
>> I believe search only affects DNS queries, not hosts file lookups,
>> right? So it should be at the name_from_dns stage in lookup_name.c.
>> The simplest implementation approach is probably to wrap name_from_dns
>> with a name_from_dns_search function that reads the search domains and
>> repeatedly calls name_from_dns until it gets success.
>
> I noticed in the process of trying to draft code to do this that there
> will be a lot of code duplication with the resolv.conf parsing in
> res_msend.c, and that this code has some stupid bugs (for example it
> stops parsing after it gets 3 nameservers, so it might miss options
> later in the file), so I think I'll take a look at factoring it into a
> new function to gather all the interesting information from
> resolv.conf that can be used in both places.
>
> A couple additional things I noticed from resolv.conf(5):
>
> 1. The default domain used by glibc is not the dns root but rather the
>    domain portion of the local hostname determined by gethostname().
>    Is there any value in duplicating this? Does anyone want/need it?
>
> 2. It's not clear from the documentation of "search" whether its
>    presence overrides/suppresses the "domain" (default or set by
>    resolv.conf) or adds additional searches before or after it. Which
>    should it do?
>
> While glibc/legacy behavior is worth looking at, I don't think we need
> to look at things from a standpoint of exactly duplicating that.
> Meeting real-world modern application needs while avoiding
> inconveniencing users with stupid/unwanted behavior should be the
> primary goal.
>
> Rich

Re: Re: Would love to see reconsideration for domain and search

[Rich Felker]

On Mon, Oct 26, 2015 at 05:37:20PM -0700, Tim Hockin wrote:
> wrt 2) my understanding is that you get at-most-one-of `search` or `domain`.

Ah, yes, it's lower in the man page after the list of options:

       The domain and search keywords are mutually exclusive.  If more than
       one instance of these keywords is present, the last instance wins.

I see no reason to do differently than glibc (and legacy resolvers)
does here; that would just be gratuitous difference for applications
to deal with.

Rich

Re: Re: Would love to see reconsideration for domain and search

[u-uy74]

On Mon, Oct 26, 2015 at 08:30:21PM -0400, Rich Felker wrote:
> 1. The default domain used by glibc is not the dns root but rather the
>    domain portion of the local hostname determined by gethostname().
>    Is there any value in duplicating this? Does anyone want/need it?

This is an annoying design misfeature - annoying because it
reflects and perpetuates confusion between DNS as a global name space
(for _interfaces_) and the "nodename" (for the _host_), which came from
UUCP with a very different name semantics.

It just makes no sense, besides reflecting the many existing "well
misunderstood" setups and contributing to creation of new ones.

If you ask me, don't ever rely on a certain choice of the nodename to be
the same as some record in DNS or resemble it. Too bad, a lot of programs
have been written with assumptions "a single network interface [with a
single DNS name?] and the host administrator certainly having reused
the nodename to be set to one of the dns names of the single interface;
let's report anything else as a misconfiguration".

The purpose of DNS was hardly understood when it was conceived, dealing
with "hosts" while actually it concerns "services" (the port part of the
adressing was thus left out of DNS until many years later, having been added
with SRV). The misunderstanding are now built in into the traditions and
among others in interpreting resolv.conf.

Using DNS names as if they'd refer to hardware / OS-instances ("hosts") is
in fact a misunderstanding, internet is about interfaces, not hosts,
but a way too many people fall for it.

IOW, please skip this reliance on gethostname() whenever possible at all.

Rune

Re: Re: Would love to see reconsideration for domain and search

[Jan Broer]

[-- Attachment #1: Type: text/plain, Size: 1473 bytes --]

Hi Rich, i was wondering about the progress of implementing support for
'search' in resolv.conf. Is this on the road map?

Thanks!

On Tue, Oct 27, 2015 at 1:30 AM, Rich Felker <dalias@libc.org> wrote:

>
> I noticed in the process of trying to draft code to do this that there
> will be a lot of code duplication with the resolv.conf parsing in
> res_msend.c, and that this code has some stupid bugs (for example it
> stops parsing after it gets 3 nameservers, so it might miss options
> later in the file), so I think I'll take a look at factoring it into a
> new function to gather all the interesting information from
> resolv.conf that can be used in both places.
>
> A couple additional things I noticed from resolv.conf(5):
>
> 1. The default domain used by glibc is not the dns root but rather the
>    domain portion of the local hostname determined by gethostname().
>    Is there any value in duplicating this? Does anyone want/need it?
>
> 2. It's not clear from the documentation of "search" whether its
>    presence overrides/suppresses the "domain" (default or set by
>    resolv.conf) or adds additional searches before or after it. Which
>    should it do?
>
> While glibc/legacy behavior is worth looking at, I don't think we need
> to look at things from a standpoint of exactly duplicating that.
> Meeting real-world modern application needs while avoiding
> inconveniencing users with stupid/unwanted behavior should be the
> primary goal.
>
> Rich
>

[-- Attachment #2: Type: text/html, Size: 1994 bytes --]

Re: Re: Would love to see reconsideration for domain and search

[Rich Felker]

On Sat, Nov 28, 2015 at 11:48:09PM +0100, Jan Broer wrote:
> Hi Rich, i was wondering about the progress of implementing support for
> 'search' in resolv.conf. Is this on the road map?

I started work on implementing it when this discussion wrapped up and
realized that it made sense to do a little bit of refactoring first
and fix some (non-serious, but possibly annoying) bugs in resolv.conf
parsing at the same time. I added it to the roadmap for this release
cycle and hope to get to it soon.

Rich

Re: Re: Would love to see reconsideration for domain and search

[Jan Broer]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

That sounds great and is much appreciated! :-)

On Sun, Nov 29, 2015 at 12:20 AM, Rich Felker <dalias@libc.org> wrote:

> On Sat, Nov 28, 2015 at 11:48:09PM +0100, Jan Broer wrote:
> > Hi Rich, i was wondering about the progress of implementing support for
> > 'search' in resolv.conf. Is this on the road map?
>
> I started work on implementing it when this discussion wrapped up and
> realized that it made sense to do a little bit of refactoring first
> and fix some (non-serious, but possibly annoying) bugs in resolv.conf
> parsing at the same time. I added it to the roadmap for this release
> cycle and hope to get to it soon.
>
> Rich
>

[-- Attachment #2: Type: text/html, Size: 1061 bytes --]

Re: Re: Would love to see reconsideration for domain and search

[Rich Felker]

On Thu, Oct 22, 2015 at 10:37:53PM -0700, Tim Hockin wrote:
> > right? So it should be at the name_from_dns stage in lookup_name.c.
> > The simplest implementation approach is probably to wrap name_from_dns
> > with a name_from_dns_search function that reads the search domains and
> > repeatedly calls name_from_dns until it gets success.
> >
> > One oddity/ugliness of search that needs to be considered is that
> > querying different address families may lead to differently-sourced
> > results. For example if you query example.us with ndots=2, a search
> > domain of example.com, and the following records present:
> >
> >         example.us.example.com A
> >         example.us AAAA
> >
> > then an AF_UNSPEC or AF_INET query yields a v4-only result for
> > example.us.example.com while an AF_INET6 query "sees through" to the
> > example.us record because the search of example.com fails. This seems
> > quite ugly and counter-intuitive, but I don't see any way to do better
> > that makes sense.
> 
> If that is how the records are laid out, I think you have to allow it.

It turns out this is a non-issue. If a name lacks the requested A or
AAAA record, but has some records, then we get rcode==0 rather than
rcode==3 (success rather than NxDomain) with 0 results. In that case
we can stop the search rather than continuing and there is no
inconsistency. Making this work right required a small fix in musl's
current rcode handling but it was easy.

Rich

Re: Re: Would love to see reconsideration for domain and search

[Rich Felker]

On Sat, Nov 28, 2015 at 11:48:09PM +0100, Jan Broer wrote:
> Hi Rich, i was wondering about the progress of implementing support for
> 'search' in resolv.conf. Is this on the road map?
> 
> Thanks!

I'm working on it now and plan to have it committed soon.

Rich

Re: Re: Would love to see reconsideration for domain and search

[Jan Broer]

[-- Attachment #1: Type: text/plain, Size: 2066 bytes --]

A very interesting discussion going on here.

> You can't query all nameservers and just take
> the first NXDOMAIN to respond.  You can only accept NXDOMAIN if all of
> the higher-priority (listed first in resolv.conf) nameservers have
> timed out or SERVFAIL'ed.  You can issue queries in parallel, but you
> must process responses in order, which is what you describe below.

GNU libc resolver gives priority to the primary nameserver because it
assumes that this is the one with the lowest RTT / highest availability.
This priorization was never understood as meaning that the primary's
servers records are more "authorative" than those of secondary nameservers.

> My view has always been that multiple
> nameservers in resolv.conf are for redundancy, not for serving
> conflicting records.

I agree.

>I have to disagree.  Some non-forwarding DNS servers use SERVFAIL to
>indicate "I am not serving for that domain" specifically to make the
>client move to their next nameserver.  if ns1 returns SERVFAIL, try
>ns2.  If ns1 times out, try ns2.  Otherwise what good is ns2?

This kind of result unioning you want from musl-libs isn't even supported
by GNU libc resolver. Yes it works for you - but this is because your DNS
server is breaking DNS RFC specs: SERVFAIL is not a technically legal
response for  "I am not serving for that domain". It's just WRONG. When an
authorative server (which the Kubernetes DNS server is) doesn't serve a
specific zone it has to respond with  NXDOMAIN. Of course if your server
would follow the rules than that type of result unioning doesn't work
because GNU libc resolver does not failover to secondary servers when the
response is NXDOMAIN.
It's just a weak argument to essentially say: "The GNU libc resolver does
exactly what i want when i point it to my broken DNS server, why can't
musl-libc?".

The bottom line:

Leave nameserver querying logic as it is: Check!
Implement search paths: Check!
Consider ndots option in resolv.conf: Check!

query name <= ndots: search first
query name > ndots : never search

LGTM!

Jan

[-- Attachment #2: Type: text/html, Size: 2621 bytes --]