> Yes, this works. Here's the problem:
> 
> > execve("./gai3a", ["./gai3a", "google.com"], [/* 15 vars */]) = 0 
> > [...] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 
> > IPPROTO_IP) = 3 bind(3, {sa_family=AF_INET, sin_port=htons(0), 
> > sin_addr=inet_addr("0.0.0.0")}, 16) = 0 sendto(3, 
> > "\271\377\1\0\0\1\0\0\0\0\0\0\6google\3com\7default\3"..., 54, 
> > MSG_NOSIGNAL, {sa_family=AF_INET, sin_port=htons(53), 
> > sin_addr=inet_addr("10.43.0.10")}, 16) = 54 sendto(3, 
> > "\341\262\1\0\0\1\0\0\0\0\0\0\6google\3com\7default\3"..., 54, 
> > MSG_NOSIGNAL, {sa_family=AF_INET, sin_port=htons(53), 
> > sin_addr=inet_addr("10.43.0.10")}, 16) = 54 poll([{fd=3, 
> > events=POLLIN}], 1, 2500)  = 1 ([{fd=3, revents=POLLIN}]) recvfrom(3, 
> > "\271\377\201\203\0\1\0\0\0\1\0\0\6google\3com\7default\3"..., 512, 0, 
> > {sa_family=AF_INET, sin_port=htons(53), 
> > sin_addr=inet_addr("10.43.0.10")}, [16]) = 147 recvfrom(3, 
> > "\341\262\201\203\0\1\0\0\0\1\0\0\6google\3com\7default\3"..., 512, 0, 
> > {sa_family=AF_INET, sin_port=htons(53), 
> > sin_addr=inet_addr("10.43.0.10")}, [16]) = 147
>                         ^^^^^^^
> 
> This correctly returns NxDomain (the \203 byte). But:
> 
> > [...]
> > socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3 
> > bind(3, {sa_family=AF_INET, sin_port=htons(0), 
> > sin_addr=inet_addr("0.0.0.0")}, 16) = 0 sendto(3, 
> > "H\244\1\0\0\1\0\0\0\0\0\0\6google\3com\7kubelet\n"..., 64, 
> > MSG_NOSIGNAL, {sa_family=AF_INET, sin_port=htons(53), 
> > sin_addr=inet_addr("10.43.0.10")}, 16) = 64 sendto(3, 
> > "K\266\1\0\0\1\0\0\0\0\0\0\6google\3com\7kubelet\n"..., 64, 
> > MSG_NOSIGNAL, {sa_family=AF_INET, sin_port=htons(53), 
> > sin_addr=inet_addr("10.43.0.10")}, 16) = 64 poll([{fd=3, 
> > events=POLLIN}], 1, 2500)  = 1 ([{fd=3, revents=POLLIN}]) recvfrom(3, 
> > "H\244\205\200\0\1\0\0\0\1\0\0\6google\3com\7kubelet\n"..., 512, 0, 
> > {sa_family=AF_INET, sin_port=htons(53), 
> > sin_addr=inet_addr("10.43.0.10")}, [16]) = 100 recvfrom(3, 
> > 0x7fff9c3f83c0, 512, 0, 0x7fff9c3f7e70, [16]) = -1 EAGAIN (Resource 
> > temporarily unavailable) poll([{fd=3, events=POLLIN}], 1, 2498)  = 1 
> > ([{fd=3, revents=POLLIN}]) recvfrom(3, 
> > "K\266\205\200\0\1\0\0\0\1\0\0\6google\3com\7kubelet\n"..., 512, 0, 
> > {sa_family=AF_INET, sin_port=htons(53), 
> > sin_addr=inet_addr("10.43.0.10")}, [16]) = 100
>                     ^^^^^^^^
> 
> Here we see your nameserver returning a RCODE=0 (Success, the \200
> byte) reply for google.com.kubelet[...] rather than NxDomain. Sorry I don't have the full name; you need to pass a larger -s to strace to get it not to truncate strings. You need to figure out why the nameserver is claiming this exists; it might be some sort of wildcard record or just a buggy nameserver (probably some component of kubernetes).
> 
> Rich

By preventing the trace from being truncated (see attached) it appears that this only occurs when querying names for which Rancher's DNS is authoritative, and is not happening for names for which Kubernetes' DNS is authoritative.

FYI, this is how `search` was defined in `resolv.conf`:

```
search default.svc.cluster.local svc.cluster.local cluster.local kubelet.kubernetes.rancher.internal kubernetes.rancher.internal rancher.internal
```

Where `default.svc.cluster.local`, `svc.cluster.local` and `cluster.local` are for service discovery in Kubernetes and `kubelet.kubernetes.rancher.internal`, `kubernetes.rancher.internal` and `rancher.internal` are (I believe) something to do with rancher-dns.

This would explain why only some people have continued having problems since Alpine 3.4 was released with the `search` and `domain` support contained within musl libc.

Raising a bug with Rancher now. Thanks so much for your help here!
LEGAL DISCLAIMER. The contents of this electronic communication
and any attached documents are strictly confidential and they may not
be used or disclosed by someone who is not a named recipient.
If you have received this electronic communication in error please notify
the sender by replying to this electronic communication inserting the
word "misdirected" as the subject and delete this communication from
your system.