* [musl] [BUG] Non-FQDN domain resolving failure on musl-1.2.x
@ 2022-06-24 10:28 Markus Geiger
2022-06-24 14:59 ` Rich Felker
0 siblings, 1 reply; 8+ messages in thread
From: Markus Geiger @ 2022-06-24 10:28 UTC (permalink / raw)
To: musl
[-- Attachment #1: Type: text/plain, Size: 14262 bytes --]
Hej!
First, I love MUSL (and alpine linux). Great project!
We encountered a bug in our CI pipeline using alpine images in conjunction
with AWS DNS servers - and it seems to be related to MUSL:
$ curl -fsSL https://slack.com
curl: (6) Could not resolve host: slack.com
Usually that should return some HTML. It seems to affect only non-FQDN
domains. As a workaround we use now full FQDN api.slack.com. But there is a
bug in resolvement! It seems if an AAAA domain is queried over an IPV4
IP/DNS and doesn’t not return a record the overall resolvement of the
domain fails.
*DEBUG LOG*
We try several alpine images and musl libs on an EC2 host with docker and
AWS DNS exclusivly:
-
alpine 3.12 with musl-1.1.24-r10 is last known to work
-
alpine 3.13 with musl-1.2.2-r1 starts failing (something introduced in
musl-1.2 ?)
-
current alpine 3.16 with current musl-1.2.3-r0 still fails
alpine 3.12 with musl-1.1.24-r10 is last known to work (see string
“success”)
docker run -it --rm --dns=10.204.109.209 alpine:3.12 ash -c 'apk add
curl bind-tools;set -x;curl -fsSL https://slack.com 1>/dev/null &&
echo success;host -4 -AAAA slack.com;apk list | grep musl' ✓
ns-watch-attribution-nonprod 12:13
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/21) Installing fstrm (0.6.0-r1)
(2/21) Installing krb5-conf (1.0-r2)
(3/21) Installing libcom_err (1.45.6-r0)
(4/21) Installing keyutils-libs (1.6.1-r1)
(5/21) Installing libverto (0.3.1-r1)
(6/21) Installing krb5-libs (1.18.5-r0)
(7/21) Installing json-c (0.14-r1)
(8/21) Installing libgcc (9.3.0-r2)
(9/21) Installing libstdc++ (9.3.0-r2)
(10/21) Installing libprotobuf (3.12.2-r0)
(11/21) Installing libprotoc (3.12.2-r0)
(12/21) Installing protobuf-c (1.3.3-r1)
(13/21) Installing libuv (1.38.1-r0)
(14/21) Installing xz-libs (5.2.5-r1)
(15/21) Installing libxml2 (2.9.14-r0)
(16/21) Installing bind-libs (9.16.27-r1)
(17/21) Installing bind-tools (9.16.27-r1)
(18/21) Installing ca-certificates (20211220-r0)
(19/21) Installing nghttp2-libs (1.41.0-r0)
(20/21) Installing libcurl (7.79.1-r1)
(21/21) Installing curl (7.79.1-r1)
Executing busybox-1.31.1-r22.trigger
Executing ca-certificates-20211220-r0.trigger
OK: 20 MiB in 35 packages
+ curl -fsSL https://slack.com
+ echo success
success
+ host -4 -AAAA slack.com
Trying "slack.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55308
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 13, ADDITIONAL: 8
;; QUESTION SECTION:
;slack.com. IN ANY
;; ANSWER SECTION:
slack.com. 38 IN A 34.203.97.10
slack.com. 38 IN A 34.193.255.5
slack.com. 38 IN A 54.147.59.169
slack.com. 38 IN A 34.231.24.224
slack.com. 38 IN A 34.225.62.185
slack.com. 38 IN A 3.95.117.96
slack.com. 38 IN A 54.163.235.119
slack.com. 38 IN A 34.204.109.226
slack.com. 38 IN A 54.92.199.186
;; AUTHORITY SECTION:
. 208510 IN NS j.root-servers.net.
. 208510 IN NS a.root-servers.net.
. 208510 IN NS b.root-servers.net.
. 208510 IN NS g.root-servers.net.
. 208510 IN NS l.root-servers.net.
. 208510 IN NS d.root-servers.net.
. 208510 IN NS i.root-servers.net.
. 208510 IN NS h.root-servers.net.
. 208510 IN NS k.root-servers.net.
. 208510 IN NS f.root-servers.net.
. 208510 IN NS m.root-servers.net.
. 208510 IN NS e.root-servers.net.
. 208510 IN NS c.root-servers.net.
;; ADDITIONAL SECTION:
b.root-servers.net. 415251 IN A 199.9.14.201
l.root-servers.net. 357447 IN A 199.7.83.42
h.root-servers.net. 417397 IN A 198.97.190.53
g.root-servers.net. 403642 IN A 192.112.36.4
j.root-servers.net. 365003 IN A 192.58.128.30
i.root-servers.net. 376774 IN A 192.36.148.17
e.root-servers.net. 415445 IN A 192.203.230.10
f.root-servers.net. 406658 IN A 192.5.5.241
Received 510 bytes from 10.204.109.209#53 in 286 ms
+ apk list
+ grep musl
musl-dbg-1.1.24-r10 x86_64 {musl} (MIT)
musl-1.1.24-r10 x86_64 {musl} (MIT) [installed]
musl-locales-lang-0_git20200319-r0 x86_64 {musl-locales} (MIT)
musl-obstack-1.1-r1 x86_64 {musl-obstack} (GPL-2.0-or-later)
libc6-compat-1.1.24-r10 x86_64 {musl} (MIT)
musl-utils-1.1.24-r10 x86_64 {musl} (MIT BSD GPL2+) [installed]
musl-nscd-dev-1.0.3-r0 x86_64 {musl-nscd} (MIT)
musl-dev-1.1.24-r10 x86_64 {musl} (MIT)
musl-nscd-1.0.3-r0 x86_64 {musl-nscd} (MIT)
musl-locales-0_git20200319-r0 x86_64 {musl-locales} (LGPL-3.0-only)
musl-nscd-doc-1.0.3-r0 x86_64 {musl-nscd} (MIT)
musl-obstack-dev-1.1-r1 x86_64 {musl-obstack} (GPL-2.0-or-later)
musl-libintl-1.1.24-r10 x86_64 {musl} (MIT)
Things change with alpine 3.12 and musl-1.2.2-r1 (now it spits ou “curl:
(6) Could not resolve host: slack.com”)
docker run -it --rm --dns=10.204.109.209 alpine:3.13 ash -c 'apk add
curl bind-tools;set -x;curl -fsSL https://slack.com 1>/dev/null &&
echo success;host -4 -AAAA slack.com;apk list | grep musl' ✓
ns-watch-attribution-nonprod 12:14
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
(1/22) Installing fstrm (0.6.0-r1)
(2/22) Installing krb5-conf (1.0-r2)
(3/22) Installing libcom_err (1.45.7-r0)
(4/22) Installing keyutils-libs (1.6.3-r0)
(5/22) Installing libverto (0.3.1-r1)
(6/22) Installing krb5-libs (1.18.5-r0)
(7/22) Installing json-c (0.15-r1)
(8/22) Installing libgcc (10.2.1_pre1-r3)
(9/22) Installing libstdc++ (10.2.1_pre1-r3)
(10/22) Installing libprotobuf (3.13.0-r2)
(11/22) Installing libprotoc (3.13.0-r2)
(12/22) Installing protobuf-c (1.3.3-r4)
(13/22) Installing libuv (1.40.0-r0)
(14/22) Installing xz-libs (5.2.5-r1)
(15/22) Installing libxml2 (2.9.14-r0)
(16/22) Installing bind-libs (9.16.27-r0)
(17/22) Installing bind-tools (9.16.27-r0)
(18/22) Installing ca-certificates (20211220-r0)
(19/22) Installing brotli-libs (1.0.9-r3)
(20/22) Installing nghttp2-libs (1.42.0-r1)
(21/22) Installing libcurl (7.79.1-r1)
(22/22) Installing curl (7.79.1-r1)
Executing busybox-1.32.1-r8.trigger
Executing ca-certificates-20211220-r0.trigger
OK: 21 MiB in 36 packages
+ curl -fsSL https://slack.com
curl: (6) Could not resolve host: slack.com
+ host -4 -AAAA slack.com
Trying "slack.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55471
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 13, ADDITIONAL: 8
;; QUESTION SECTION:
;slack.com. IN ANY
;; ANSWER SECTION:
slack.com. 23 IN A 34.231.24.224
slack.com. 23 IN A 54.163.235.119
slack.com. 23 IN A 34.225.62.185
slack.com. 23 IN A 34.203.97.10
slack.com. 23 IN A 3.95.117.96
slack.com. 23 IN A 34.193.255.5
slack.com. 23 IN A 34.204.109.226
slack.com. 23 IN A 54.92.199.186
slack.com. 23 IN A 54.147.59.169
;; AUTHORITY SECTION:
. 208436 IN NS j.root-servers.net.
. 208436 IN NS a.root-servers.net.
. 208436 IN NS h.root-servers.net.
. 208436 IN NS m.root-servers.net.
. 208436 IN NS k.root-servers.net.
. 208436 IN NS l.root-servers.net.
. 208436 IN NS g.root-servers.net.
. 208436 IN NS f.root-servers.net.
. 208436 IN NS c.root-servers.net.
. 208436 IN NS b.root-servers.net.
. 208436 IN NS e.root-servers.net.
. 208436 IN NS d.root-servers.net.
. 208436 IN NS i.root-servers.net.
;; ADDITIONAL SECTION:
b.root-servers.net. 415177 IN A 199.9.14.201
l.root-servers.net. 357373 IN A 199.7.83.42
h.root-servers.net. 417323 IN A 198.97.190.53
g.root-servers.net. 403568 IN A 192.112.36.4
j.root-servers.net. 364929 IN A 192.58.128.30
i.root-servers.net. 376700 IN A 192.36.148.17
e.root-servers.net. 415371 IN A 192.203.230.10
f.root-servers.net. 406584 IN A 192.5.5.241
Received 510 bytes from 10.204.109.209#53 in 381 ms
+ apk list
+ grep musl
musl-dbg-1.2.2-r1 x86_64 {musl} (MIT)
musl-1.2.2-r1 x86_64 {musl} (MIT) [installed]
musl-locales-lang-0_git20200319-r1 x86_64 {musl-locales} (MIT)
musl-obstack-1.1-r1 x86_64 {musl-obstack} (GPL-2.0-or-later)
libc6-compat-1.2.2-r1 x86_64 {musl} (MIT)
musl-utils-1.2.2-r1 x86_64 {musl} (MIT BSD GPL2+) [installed]
musl-nscd-dev-1.0.3-r1 x86_64 {musl-nscd} (MIT)
musl-dev-1.2.2-r1 x86_64 {musl} (MIT)
musl-nscd-1.0.3-r1 x86_64 {musl-nscd} (MIT)
musl-locales-0_git20200319-r1 x86_64 {musl-locales} (LGPL-3.0-only)
musl-nscd-doc-1.0.3-r1 x86_64 {musl-nscd} (MIT)
musl-obstack-dev-1.1-r1 x86_64 {musl-obstack} (GPL-2.0-or-later)
musl-libintl-1.2.2-r1 x86_64 {musl} (MIT)
Still alpine 3.16 with musl-1.2.3-r0 fails:
docker run -it --rm --dns=10.204.109.209 alpine:3.16 ash -c 'apk add
curl bind-tools;set -x;curl -fsSL https://slack.com 1>/dev/null &&
echo success;host -4 -AAAA slack.com;apk list | grep musl' ✓
ns-watch-attribution-nonprod 12:15
Unable to find image 'alpine:3.16' locally
3.16: Pulling from library/alpine
Digest: sha256:686d8c9dfa6f3ccfc8230bc3178d23f84eeaf7e457f36f271ab1acc53015037c
Status: Downloaded newer image for alpine:3.16
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
(1/18) Installing fstrm (0.6.1-r0)
(2/18) Installing krb5-conf (1.0-r2)
(3/18) Installing libcom_err (1.46.5-r0)
(4/18) Installing keyutils-libs (1.6.3-r1)
(5/18) Installing libverto (0.3.2-r0)
(6/18) Installing krb5-libs (1.19.3-r0)
(7/18) Installing json-c (0.16-r0)
(8/18) Installing protobuf-c (1.4.0-r0)
(9/18) Installing libuv (1.44.1-r0)
(10/18) Installing xz-libs (5.2.5-r1)
(11/18) Installing libxml2 (2.9.14-r0)
(12/18) Installing bind-libs (9.16.29-r0)
(13/18) Installing bind-tools (9.16.29-r0)
(14/18) Installing ca-certificates (20211220-r0)
(15/18) Installing brotli-libs (1.0.9-r6)
(16/18) Installing nghttp2-libs (1.47.0-r0)
(17/18) Installing libcurl (7.83.1-r1)
(18/18) Installing curl (7.83.1-r1)
Executing busybox-1.35.0-r13.trigger
Executing ca-certificates-20211220-r0.trigger
OK: 15 MiB in 32 packages
+ curl -fsSL https://slack.com
curl: (6) Could not resolve host: slack.com
+ host -4 -AAAA slack.com
Trying "slack.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60635
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 13, ADDITIONAL: 8
;; QUESTION SECTION:
;slack.com. IN ANY
;; ANSWER SECTION:
slack.com. 5 IN A 3.95.117.96
slack.com. 5 IN A 54.92.199.186
slack.com. 5 IN A 34.225.62.185
slack.com. 5 IN A 34.203.97.10
slack.com. 5 IN A 34.193.255.5
slack.com. 5 IN A 54.163.235.119
slack.com. 5 IN A 34.204.109.226
slack.com. 5 IN A 34.231.24.224
slack.com. 5 IN A 54.147.59.169
;; AUTHORITY SECTION:
. 208128 IN NS m.root-servers.net.
. 208128 IN NS a.root-servers.net.
. 208128 IN NS j.root-servers.net.
. 208128 IN NS f.root-servers.net.
. 208128 IN NS d.root-servers.net.
. 208128 IN NS h.root-servers.net.
. 208128 IN NS k.root-servers.net.
. 208128 IN NS b.root-servers.net.
. 208128 IN NS g.root-servers.net.
. 208128 IN NS i.root-servers.net.
. 208128 IN NS c.root-servers.net.
. 208128 IN NS l.root-servers.net.
. 208128 IN NS e.root-servers.net.
;; ADDITIONAL SECTION:
b.root-servers.net. 414869 IN A 199.9.14.201
l.root-servers.net. 357065 IN A 199.7.83.42
h.root-servers.net. 417015 IN A 198.97.190.53
g.root-servers.net. 403260 IN A 192.112.36.4
j.root-servers.net. 364621 IN A 192.58.128.30
i.root-servers.net. 376392 IN A 192.36.148.17
e.root-servers.net. 415063 IN A 192.203.230.10
f.root-servers.net. 406276 IN A 192.5.5.241
Received 510 bytes from 10.204.109.209#53 in 407 ms
+ apk list
+ grep musl
musl-dbg-1.2.3-r0 x86_64 {musl} (MIT)
musl-1.2.3-r0 x86_64 {musl} (MIT) [installed]
musl-locales-lang-0.1.0-r0 x86_64 {musl-locales} (MIT)
musl-obstack-1.2.3-r0 x86_64 {musl-obstack} (LGPL-2.1-or-later)
libc6-compat-1.2.3-r0 x86_64 {musl} (MIT)
musl-utils-1.2.3-r0 x86_64 {musl} (MIT BSD GPL2+) [installed]
musl-nscd-dev-1.1.1-r0 x86_64 {musl-nscd} (MIT)
musl-dev-1.2.3-r0 x86_64 {musl} (MIT)
musl-nscd-1.1.1-r0 x86_64 {musl-nscd} (MIT)
musl-locales-0.1.0-r0 x86_64 {musl-locales} (LGPL-3.0-only)
musl-nscd-doc-1.1.1-r0 x86_64 {musl-nscd} (MIT)
musl-legacy-error-0.5-r0 x86_64 {musl-legacy-error} (BSD-2-Clause)
musl-obstack-dev-1.2.3-r0 x86_64 {musl-obstack} (LGPL-2.1-or-later)
musl-libintl-1.2.3-r0 x86_64 {musl} (MIT)
Greetings,
Markus
--
Markus Geiger
Protean Linux | Cloud | DevOps Engineer
RefinedLabs - A Nielsen Company
www.nielsen.com
<https://global.nielsen.com/>
Refined Labs GmbH - A Nielsen Company
Herzog-Wilhelm-Straße 26, 80331 München
Geschäftsführer: Dietmar Krauss, Robert Moor
Sitz München, Amtsgericht München HRB 166589
*This e-mail may contain confidential and/or privileged information. *
*If you are not the intended recipient or have received this e-mail in
error please be kind and notify the sender immediately and delete this mail
and all its attachments subsequently. Please improve communication in the
workplace by not using emails at all since they are not secure,
anti-social, bring confusion, often destroy focus and lack transparency.
Use encrypted group chats, social media or directly talk to people and use
an agile task board for your daily planning. By reading this you agree to
stop agreeing to useless disclaimers and learn about security and securing
your communication.*
[-- Attachment #2: Type: text/html, Size: 104478 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [musl] [BUG] Non-FQDN domain resolving failure on musl-1.2.x
2022-06-24 10:28 [musl] [BUG] Non-FQDN domain resolving failure on musl-1.2.x Markus Geiger
@ 2022-06-24 14:59 ` Rich Felker
2022-06-24 15:10 ` Jeffrey Walton
2022-06-24 17:14 ` Markus Geiger
0 siblings, 2 replies; 8+ messages in thread
From: Rich Felker @ 2022-06-24 14:59 UTC (permalink / raw)
To: Markus Geiger; +Cc: musl
On Fri, Jun 24, 2022 at 12:28:24PM +0200, Markus Geiger wrote:
> Hej!
>
> First, I love MUSL (and alpine linux). Great project!
>
> We encountered a bug in our CI pipeline using alpine images in conjunction
> with AWS DNS servers - and it seems to be related to MUSL:
>
> $ curl -fsSL https://slack.com
> curl: (6) Could not resolve host: slack.com
>
> Usually that should return some HTML. It seems to affect only non-FQDN
> domains. As a workaround we use now full FQDN api.slack.com. But there is a
> bug in resolvement! It seems if an AAAA domain is queried over an IPV4
> IP/DNS and doesn’t not return a record the overall resolvement of the
> domain fails.
That's not non-FQDN. Non-FQDN would be "api" as short for
api.slack.com. slack.com is just the apex of a zone, but there's
nothing special about that for resolving; it's likely just a
difference in the records for it vs api, or something fishy the
recursive nameserver you're using is doing...
> *DEBUG LOG*
>
> We try several alpine images and musl libs on an EC2 host with docker and
> AWS DNS exclusivly:
>
> -
>
> alpine 3.12 with musl-1.1.24-r10 is last known to work
> -
>
> alpine 3.13 with musl-1.2.2-r1 starts failing (something introduced in
> musl-1.2 ?)
> -
>
> current alpine 3.16 with current musl-1.2.3-r0 still fails
>
> alpine 3.12 with musl-1.1.24-r10 is last known to work (see string
> “success”)
>
> docker run -it --rm --dns=10.204.109.209 alpine:3.12 ash -c 'apk add
> curl bind-tools;set -x;curl -fsSL https://slack.com 1>/dev/null &&
> echo success;host -4 -AAAA slack.com;apk list | grep musl' ✓
> ns-watch-attribution-nonprod 12:13
> fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
> fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
> (1/21) Installing fstrm (0.6.0-r1)
> (2/21) Installing krb5-conf (1.0-r2)
> (3/21) Installing libcom_err (1.45.6-r0)
> (4/21) Installing keyutils-libs (1.6.1-r1)
> (5/21) Installing libverto (0.3.1-r1)
> (6/21) Installing krb5-libs (1.18.5-r0)
> (7/21) Installing json-c (0.14-r1)
> (8/21) Installing libgcc (9.3.0-r2)
> (9/21) Installing libstdc++ (9.3.0-r2)
> (10/21) Installing libprotobuf (3.12.2-r0)
> (11/21) Installing libprotoc (3.12.2-r0)
> (12/21) Installing protobuf-c (1.3.3-r1)
> (13/21) Installing libuv (1.38.1-r0)
> (14/21) Installing xz-libs (5.2.5-r1)
> (15/21) Installing libxml2 (2.9.14-r0)
> (16/21) Installing bind-libs (9.16.27-r1)
> (17/21) Installing bind-tools (9.16.27-r1)
> (18/21) Installing ca-certificates (20211220-r0)
> (19/21) Installing nghttp2-libs (1.41.0-r0)
> (20/21) Installing libcurl (7.79.1-r1)
> (21/21) Installing curl (7.79.1-r1)
> Executing busybox-1.31.1-r22.trigger
> Executing ca-certificates-20211220-r0.trigger
> OK: 20 MiB in 35 packages
> + curl -fsSL https://slack.com
> + echo success
> success
> + host -4 -AAAA slack.com
^^^^
This does not request AAAA. It (-A repeated redundantly 4 times)
request ANY, which is deprecated. So the output is not terribly
helpful in figuring out what's going on.
Can you provide tcpdump of port 53 traffic when curl makes the query,
and/or full strace of the curl execution? This would show what wrong
responses the nameserver is giving that's causing curl to fail.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [musl] [BUG] Non-FQDN domain resolving failure on musl-1.2.x
2022-06-24 14:59 ` Rich Felker
@ 2022-06-24 15:10 ` Jeffrey Walton
2022-06-24 15:15 ` Rich Felker
2022-06-24 17:14 ` Markus Geiger
1 sibling, 1 reply; 8+ messages in thread
From: Jeffrey Walton @ 2022-06-24 15:10 UTC (permalink / raw)
To: musl; +Cc: Markus Geiger
On Fri, Jun 24, 2022 at 10:59 AM Rich Felker <dalias@libc.org> wrote:
>
> On Fri, Jun 24, 2022 at 12:28:24PM +0200, Markus Geiger wrote:
> > Hej!
> >
> > First, I love MUSL (and alpine linux). Great project!
> >
> > We encountered a bug in our CI pipeline using alpine images in conjunction
> > with AWS DNS servers - and it seems to be related to MUSL:
> >
> > $ curl -fsSL https://slack.com
> > curl: (6) Could not resolve host: slack.com
> >
> > Usually that should return some HTML. It seems to affect only non-FQDN
> > domains. As a workaround we use now full FQDN api.slack.com. But there is a
> > bug in resolvement! It seems if an AAAA domain is queried over an IPV4
> > IP/DNS and doesn’t not return a record the overall resolvement of the
> > domain fails.
>
> That's not non-FQDN. Non-FQDN would be "api" as short for
> api.slack.com. slack.com is just the apex of a zone, but there's
> nothing special about that for resolving; it's likely just a
> difference in the records for it vs api, or something fishy the
> recursive nameserver you're using is doing...
+1.
A FQDN ends in '.' (dot). The dot specifies the root of the DNS tree.
'slack.com.' is fully qualified, but 'slack.com' is not. If you are
configured to search with domain suffixes, 'slack.com' could resolve
to 'slack.com.home.pvt' because it is not fully qualified.
Jeff
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [musl] [BUG] Non-FQDN domain resolving failure on musl-1.2.x
2022-06-24 15:10 ` Jeffrey Walton
@ 2022-06-24 15:15 ` Rich Felker
0 siblings, 0 replies; 8+ messages in thread
From: Rich Felker @ 2022-06-24 15:15 UTC (permalink / raw)
To: Jeffrey Walton; +Cc: musl, Markus Geiger
On Fri, Jun 24, 2022 at 11:10:37AM -0400, Jeffrey Walton wrote:
> On Fri, Jun 24, 2022 at 10:59 AM Rich Felker <dalias@libc.org> wrote:
> >
> > On Fri, Jun 24, 2022 at 12:28:24PM +0200, Markus Geiger wrote:
> > > Hej!
> > >
> > > First, I love MUSL (and alpine linux). Great project!
> > >
> > > We encountered a bug in our CI pipeline using alpine images in conjunction
> > > with AWS DNS servers - and it seems to be related to MUSL:
> > >
> > > $ curl -fsSL https://slack.com
> > > curl: (6) Could not resolve host: slack.com
> > >
> > > Usually that should return some HTML. It seems to affect only non-FQDN
> > > domains. As a workaround we use now full FQDN api.slack.com. But there is a
> > > bug in resolvement! It seems if an AAAA domain is queried over an IPV4
> > > IP/DNS and doesn’t not return a record the overall resolvement of the
> > > domain fails.
> >
> > That's not non-FQDN. Non-FQDN would be "api" as short for
> > api.slack.com. slack.com is just the apex of a zone, but there's
> > nothing special about that for resolving; it's likely just a
> > difference in the records for it vs api, or something fishy the
> > recursive nameserver you're using is doing...
>
> +1.
>
> A FQDN ends in '.' (dot). The dot specifies the root of the DNS tree.
> 'slack.com.' is fully qualified, but 'slack.com' is not. If you are
> configured to search with domain suffixes, 'slack.com' could resolve
> to 'slack.com.home.pvt' because it is not fully qualified.
While this is pedantically correct in some usage, it's not really the
issue at hand here. In ordinary usage, most folks call a domain you
just *intend* to be interpreted from the DNS root a FQDN, regardless
of whether it has a dot to express that. But in any case, the point
was that the issue is not a matter of FQDNs but of something wrong
Amazon's nameservers are apparently doing (again?? *headdesk*). Let's
see if we can figure out what and how to get them to fix it...
Rich
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [musl] [BUG] Non-FQDN domain resolving failure on musl-1.2.x
2022-06-24 14:59 ` Rich Felker
2022-06-24 15:10 ` Jeffrey Walton
@ 2022-06-24 17:14 ` Markus Geiger
2022-06-25 1:56 ` Rich Felker
1 sibling, 1 reply; 8+ messages in thread
From: Markus Geiger @ 2022-06-24 17:14 UTC (permalink / raw)
To: musl
[-- Attachment #1: Type: text/plain, Size: 6580 bytes --]
Sorry: not Amazon DNS – 10.204.109.209 is a BIND server in our network
we've setup to work with our global VPN/DNS.
BUT the strange thing is that the domain lookup works with musl-1.1.24
while with some musl-1.2.x just quits with an error.
a comparison with the docker runs and `sudo tcpdump -v -i docker0 udp port
53 or tcp port 53` did not bring up any diffs except the list of A records
returned is in a different order (which i think is completely normal). the
order of requests is the same
tcpdump from working version:
> bind-us-east-1a.XXXXXXXXXXXXXX.domain > 172.17.0.3.45501: 18685 9/13/8
slack.com. A 3.95.117.96, slack.com. A 34.231.24.224, slack.com. A
54.163.235.119, slack.com. A 54.147.59.169, slack.com. A 34.193.255.5,
slack.com. A 34.204.109.226, slack.com. A 34.225.62.185, slack.com. A
34.203.97.10, slack.com. A 54.92.199.186 (510)
tcpdump from non-working version:
> bind-us-east-1a.XXXXXXXXXXXXXX.domain > 172.17.0.3.59951: 49211 9/13/8
slack.com. A 34.225.62.185, slack.com. A 54.163.235.119, slack.com. A
34.231.24.224, slack.com. A 54.147.59.169, slack.com. A 34.193.255.5,
slack.com. A 34.204.109.226, slack.com. A 54.92.199.186, slack.com. A
3.95.117.96, slack.com. A 34.203.97.10 (510)
Complete log:
172.17.0.3.59951 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
49211+ A? slack.com. (27)
18:56:19.990087 IP (tos 0x0, ttl 64, id 10210, offset 0, flags [DF], proto
UDP (17), length 55)
172.17.0.3.59951 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
49334+ AAAA? slack.com. (27)
18:56:20.154990 IP (tos 0x0, ttl 250, id 17825, offset 0, flags [none],
proto UDP (17), length 538)
bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.59951:
49211 9/13/8 slack.com. A 34.225.62.185, slack.com. A 54.163.235.119,
slack.com. A 34.231.24.224, slack.com. A 54.147.59.169, slack.com. A
34.193.255.5, slack.com. A 34.204.109.226, slack.com. A 54.92.199.186,
slack.com. A 3.95.117.96, slack.com. A 34.203.97.10 (510)
18:56:20.241377 IP (tos 0x0, ttl 250, id 17846, offset 0, flags [none],
proto UDP (17), length 55)
bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.59951:
49334 ServFail 0/0/0 (27)
18:56:20.241501 IP (tos 0x0, ttl 64, id 10233, offset 0, flags [DF], proto
UDP (17), length 55)
172.17.0.3.59951 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
49334+ AAAA? slack.com. (27)
18:56:20.401765 IP (tos 0x0, ttl 250, id 17879, offset 0, flags [none],
proto UDP (17), length 55)
bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.59951:
49334 ServFail 0/0/0 (27)
18:56:20.401831 IP (tos 0x0, ttl 64, id 10247, offset 0, flags [DF], proto
UDP (17), length 55)
172.17.0.3.59951 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
49334+ AAAA? slack.com. (27)
18:56:20.563915 IP (tos 0x0, ttl 250, id 17906, offset 0, flags [none],
proto UDP (17), length 55)
bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.59951:
49334 ServFail 0/0/0 (27)
18:56:20.563946 IP (tos 0x0, ttl 64, id 10300, offset 0, flags [DF], proto
UDP (17), length 55)
172.17.0.3.59951 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
49334+ AAAA? slack.com. (27)
18:56:20.801072 IP (tos 0x0, ttl 250, id 17915, offset 0, flags [none],
proto UDP (17), length 55)
bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.59951:
49334 ServFail 0/0/0 (27)
18:56:20.801187 IP (tos 0x0, ttl 64, id 10392, offset 0, flags [DF], proto
UDP (17), length 55)
172.17.0.3.59951 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
49334+ AAAA? slack.com. (27)
18:56:20.961903 IP (tos 0x0, ttl 250, id 17948, offset 0, flags [none],
proto UDP (17), length 55)
bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.59951:
49334 ServFail 0/0/0 (27)
18:56:22.492127 IP (tos 0x0, ttl 64, id 10410, offset 0, flags [DF], proto
UDP (17), length 55)
172.17.0.3.59951 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
49334+ AAAA? slack.com. (27)
18:56:22.745884 IP (tos 0x0, ttl 250, id 18135, offset 0, flags [none],
proto UDP (17), length 55)
bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.59951:
49334 ServFail 0/0/0 (27)
18:56:22.745930 IP (tos 0x0, ttl 64, id 10492, offset 0, flags [DF], proto
UDP (17), length 55)
172.17.0.3.59951 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
49334+ AAAA? slack.com. (27)
18:56:22.906904 IP (tos 0x0, ttl 250, id 18171, offset 0, flags [none],
proto UDP (17), length 55)
bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.59951:
49334 ServFail 0/0/0 (27)
18:56:22.906941 IP (tos 0x0, ttl 64, id 10548, offset 0, flags [DF], proto
UDP (17), length 55)
172.17.0.3.59951 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
49334+ AAAA? slack.com. (27)
18:56:23.067742 IP (tos 0x0, ttl 250, id 18209, offset 0, flags [none],
proto UDP (17), length 55)
bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.59951:
49334 ServFail 0/0/0 (27)
18:56:23.067773 IP (tos 0x0, ttl 64, id 10559, offset 0, flags [DF], proto
UDP (17), length 55)
172.17.0.3.59951 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
49334+ AAAA? slack.com. (27)
18:56:23.228046 IP (tos 0x0, ttl 250, id 18244, offset 0, flags [none],
proto UDP (17), length 55)
bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.59951:
49334 ServFail 0/0/0 (27)
18:56:23.228083 IP (tos 0x0, ttl 64, id 10598, offset 0, flags [DF], proto
UDP (17), length 55)
172.17.0.3.59951 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
49334+ AAAA? slack.com. (27)
18:56:23.388598 IP (tos 0x0, ttl 250, id 18274, offset 0, flags [none],
proto UDP (17), length 55)
bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.59951:
49334 ServFail 0/0/0 (27)
18:56:24.998168 IP (tos 0x0, ttl 64, id 10759, offset 0, flags [none],
proto UDP (17), length 55)
172.17.0.3.55720 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
2282+ ANY? slack.com. (27)
18:56:25.199844 IP (tos 0x0, ttl 250, id 18505, offset 0, flags [none],
proto UDP (17), length 538)
bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.55720:
2282 9/13/8 slack.com. A 34.231.24.224, slack.com. A 34.193.255.5, slack.com.
A 54.163.235.119, slack.com. A 54.92.199.186, slack.com. A 54.147.59.169,
slack.com. A 3.95.117.96, slack.com. A 34.225.62.185, slack.com. A
34.204.109.226, slack.com. A 34.203.97.10 (510)
Thanks for looking into!
Greetings,
Markus
[-- Attachment #2: Type: text/html, Size: 8499 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [musl] [BUG] Non-FQDN domain resolving failure on musl-1.2.x
2022-06-24 17:14 ` Markus Geiger
@ 2022-06-25 1:56 ` Rich Felker
2022-06-27 11:35 ` Markus Geiger
0 siblings, 1 reply; 8+ messages in thread
From: Rich Felker @ 2022-06-25 1:56 UTC (permalink / raw)
To: Markus Geiger; +Cc: musl
On Fri, Jun 24, 2022 at 07:14:10PM +0200, Markus Geiger wrote:
> Sorry: not Amazon DNS – 10.204.109.209 is a BIND server in our network
> we've setup to work with our global VPN/DNS.
>
> BUT the strange thing is that the domain lookup works with musl-1.1.24
> while with some musl-1.2.x just quits with an error.
>
> a comparison with the docker runs and `sudo tcpdump -v -i docker0 udp port
> 53 or tcp port 53` did not bring up any diffs except the list of A records
> returned is in a different order (which i think is completely normal). the
> order of requests is the same
>
> tcpdump from working version:
> > bind-us-east-1a.XXXXXXXXXXXXXX.domain > 172.17.0.3.45501: 18685 9/13/8
> slack.com. A 3.95.117.96, slack.com. A 34.231.24.224, slack.com. A
> 54.163.235.119, slack.com. A 54.147.59.169, slack.com. A 34.193.255.5,
> slack.com. A 34.204.109.226, slack.com. A 34.225.62.185, slack.com. A
> 34.203.97.10, slack.com. A 54.92.199.186 (510)
>
> tcpdump from non-working version:
> > bind-us-east-1a.XXXXXXXXXXXXXX.domain > 172.17.0.3.59951: 49211 9/13/8
> slack.com. A 34.225.62.185, slack.com. A 54.163.235.119, slack.com. A
> 34.231.24.224, slack.com. A 54.147.59.169, slack.com. A 34.193.255.5,
> slack.com. A 34.204.109.226, slack.com. A 54.92.199.186, slack.com. A
> 3.95.117.96, slack.com. A 34.203.97.10 (510)
>
> Complete log:
>
> 172.17.0.3.59951 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
> 49211+ A? slack.com. (27)
> 18:56:19.990087 IP (tos 0x0, ttl 64, id 10210, offset 0, flags [DF], proto
> UDP (17), length 55)
> 172.17.0.3.59951 > bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain:
> 49334+ AAAA? slack.com. (27)
> 18:56:20.154990 IP (tos 0x0, ttl 250, id 17825, offset 0, flags [none],
> proto UDP (17), length 538)
> bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.59951:
> 49211 9/13/8 slack.com. A 34.225.62.185, slack.com. A 54.163.235.119,
> slack.com. A 34.231.24.224, slack.com. A 54.147.59.169, slack.com. A
> 34.193.255.5, slack.com. A 34.204.109.226, slack.com. A 54.92.199.186,
> slack.com. A 3.95.117.96, slack.com. A 34.203.97.10 (510)
> 18:56:20.241377 IP (tos 0x0, ttl 250, id 17846, offset 0, flags [none],
> proto UDP (17), length 55)
> bind-us-east-1a.XXXXXXXXXXXXXXXXXXXXXXXXXx.domain > 172.17.0.3.59951:
> 49334 ServFail 0/0/0 (27)
> 18:56:20.241501 IP (tos 0x0, ttl 64, id 10233, offset 0, flags [DF], proto
> UDP (17), length 55)
Here's your problem -- the server is returning ServFail rather than an
answer for some of the queries. This makes musl's resolver continue
retrying for an answer. In an old version, there may have been a bug
whereby, after the retries timed out, the fact that one query failed
was sometimes overlooked. This logic was improved between the versions
you tested as part of ensuring DNSSEC integrity. In any case, you just
need to find the cause of the ServFail (maybe a hack someone put in
place to try to suppress use of IPv6?) and fix it.
Rich
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [musl] [BUG] Non-FQDN domain resolving failure on musl-1.2.x
2022-06-25 1:56 ` Rich Felker
@ 2022-06-27 11:35 ` Markus Geiger
2022-06-27 14:06 ` Rich Felker
0 siblings, 1 reply; 8+ messages in thread
From: Markus Geiger @ 2022-06-27 11:35 UTC (permalink / raw)
Cc: musl
[-- Attachment #1: Type: text/plain, Size: 4282 bytes --]
Hi,
Thanks for the info. I'm not a everyday gdb user (once a year?) – but
I debugged the issue.
In musl 1.2.2 we run into EAI_AGAIN in name_from_dns on
network/lookup_name.c:166
In musl 1.1.24 we don't.
Just that I'm not quite sure if that behavior is correct – if you can
verify that it is correct behaviour treat this issue as resolved. And
thX for looking into it.
gdb backtrace -----------------------------------------------------------------------
#0 name_from_dns (buf=buf@entry=0x7f5d11872490,
canon=canon@entry=0x7f5d11872390 "slack.com",
name=name@entry=0x7f5d118e1c10 "slack.com",
family=family@entry=0, conf=conf@entry=0x7f5d1187219c) at
src/network/lookup_name.c:166
#1 0x00007f5d11cd8ac2 in name_from_dns_search (family=0,
name=0x7f5d118e1c10 "slack.com", canon=0x7f5d11872390 "slack.com",
buf=0x7f5d11872490)
at src/network/lookup_name.c:219
#2 __lookup_name (buf=buf@entry=0x7f5d11872490,
canon=canon@entry=0x7f5d11872390 "slack.com",
name=name@entry=0x7f5d118e1c10 "slack.com",
family=family@entry=0, flags=flags@entry=0) at src/network/lookup_name.c:322
#3 0x00007f5d11cd5fdd in getaddrinfo (host=0x7f5d118e1c10
"slack.com", serv=serv@entry=0x7f5d11872a7c "443",
hint=hint@entry=0x7f5d1188d320,
res=res@entry=0x7f5d11872a20) at src/network/getaddrinfo.c:91
--Type <RET> for more, q to quit, c to continue without paging--
#4 0x00007f5d11c36ed8 in Curl_getaddrinfo_ex (nodename=<optimized
out>, servname=servname@entry=0x7f5d11872a7c "443",
hints=hints@entry=0x7f5d1188d320, result=result@entry=0x7f5d1188d318)
at curl_addrinfo.c:124
#5 0x00007f5d11c317ce in getaddrinfo_thread
(arg=arg@entry=0x7f5d1188d2e8) at asyn-thread.c:305
#6 0x00007f5d11c39548 in curl_thread_create_thunk (arg=<optimized
out>) at curl_threads.c:57
#7 0x00007f5d11cf1221 in start (p=0x7f5d11872b00) at
src/thread/pthread_create.c:203
#8 0x00007f5d11cf33e0 in __clone () at src/thread/x86_64/clone.s:22
in we run into the line with return EAI_AGAIN with following locals
-----------------------------------------------------------------------
qbuf = {
":s\001\000\000\001\000\000\000\000\000\000\005slack\003com\000\000\001\000\001",
'\000' <repeats 13 times>, "\061\067\062.17.0.3\tee6c51ce088c\n\000st
ip6-loopback\n", '\000' <repeats 103 times>,
"\006\037\207\021]\177\000\000\350\034\207\021]\177\000\000\000\000\000\000\000\000\000\000"...,
";\237\001\000\000\001\000\000\000\000\000\000\005slack\003com\000\000\034\000\001\000\000\000\000\000|*\207\021]\177\000\000\210\035\207\021]\177\000\000\n\000\000\000\000\000\000\000|*\207\021]\177\000\000H\034\207\021]\177\000\000\006\037\207\021]\177\000\000\335\332\316\021]\177\000\000\000\000\000\000\000\000\000\000\377\377\377\377\377\377\377\377",
'\000' <repeats 16 times>,
"\177*\207\021]\177\000\000\377\377\377\377\377\377\377\377", '\000'
<repeats 32 times>, "\377\377\377\377\377\377\377\377", '\000'
<repeats 24 times>...}
abuf = {
"\n\000\000\000\000\000\000\000\314\000\000\000\000\000\000\000m\000\000\000\000\000\000\000\321\000\000\000\000\000\000\000t
ip6-lo\234!\207\021]\177\000\000\373\036\207\021]\177\000\000\234!\207\021]\177\000\000\001\000\000\000\000\000\000\000\351X\316\021]\177\000\000\217\035\207\021]\177\000\000\000\000\000\000\000\000\000\000\370\037\207\021]\177\000\000\370\000\000\000\000\000\000\000\b\036\207\021]\177\000\000\b\036\207\021]\177\000\000\360\036\207\021]\177\000\000R[\316\021]\177\000\000uters\n172.17.0.s\360\036\207\021]\177\000\000\247b\316\021]\177\000\000\000\000\000\000\000\000\000\000\"
\207\021]\177\000\000\b\036\207\021]\177\000\000"...,
"nameserver 10.204.109.209\000\000\000\000\000\000\000\020\034\216\021]\177\000\000\377\377\377\377\377\377\377\377",
'\000' <repeats 64 times>, "\020\034\216\021]\177", '\000' <repeats 90
times>...}
qp = {0x7f5d11871ac0 ":s\001", 0x7f5d11871bd8 ";\237\001"}
ap = {0x7f5d11871cf0 "\n", 0x7f5d11871ef0 "nameserver 10.204.109.209"}
qlens = {27, 27}
alens = {0, 0}
i = 0
nq = 2
ctx = {addrs = 0x7f5d11872490, canon = 0x7f5d11872390 "slack.com", cnt = 0}
For diff please see attachment since Google Mail Plain Text mode just
.... you know.
I guess it has something to do with RR_AAAA.
Greetings,
Markus Geiger
[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 157076 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [musl] [BUG] Non-FQDN domain resolving failure on musl-1.2.x
2022-06-27 11:35 ` Markus Geiger
@ 2022-06-27 14:06 ` Rich Felker
0 siblings, 0 replies; 8+ messages in thread
From: Rich Felker @ 2022-06-27 14:06 UTC (permalink / raw)
To: Markus Geiger; +Cc: musl
On Mon, Jun 27, 2022 at 01:35:50PM +0200, Markus Geiger wrote:
> Hi,
>
> Thanks for the info. I'm not a everyday gdb user (once a year?) – but
> I debugged the issue.
>
> In musl 1.2.2 we run into EAI_AGAIN in name_from_dns on
> network/lookup_name.c:166
> In musl 1.1.24 we don't.
>
> Just that I'm not quite sure if that behavior is correct – if you can
> verify that it is correct behaviour treat this issue as resolved. And
> thX for looking into it.
Yes, that behavior is correct. The responses were inconclusive and
returning success as if they were conclusive is a DNSSEC-bypass bug
that was fixed.
Rich
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2022-06-27 14:06 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-24 10:28 [musl] [BUG] Non-FQDN domain resolving failure on musl-1.2.x Markus Geiger
2022-06-24 14:59 ` Rich Felker
2022-06-24 15:10 ` Jeffrey Walton
2022-06-24 15:15 ` Rich Felker
2022-06-24 17:14 ` Markus Geiger
2022-06-25 1:56 ` Rich Felker
2022-06-27 11:35 ` Markus Geiger
2022-06-27 14:06 ` Rich Felker
Code repositories for project(s) associated with this public inbox
https://git.vuxu.org/mirror/musl/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).