Hej!

First, I love MUSL (and alpine linux). Great project!

We encountered a bug in our CI pipeline using alpine images in conjunction with AWS DNS servers - and it seems to be related to MUSL:

$ curl -fsSL https://slack.com
curl: (6) Could not resolve host: slack.com

Usually that should return some HTML. It seems to affect only non-FQDN domains. As a workaround we use now full FQDN api.slack.com. But there is a bug in resolvement! It seems if an AAAA domain is queried over an IPV4 IP/DNS and doesn’t not return a record the overall resolvement of the domain fails.

DEBUG LOG

We try several alpine images and musl libs on an EC2 host with docker and AWS DNS exclusivly:

alpine 3.12 with musl-1.1.24-r10 is last known to work (see string “success”)

docker run -it --rm --dns=10.204.109.209 alpine:3.12 ash -c 'apk add curl bind-tools;set -x;curl -fsSL https://slack.com 1>/dev/null && echo success;host -4 -AAAA slack.com;apk list | grep musl'       ✓ ns-watch-attribution-nonprod 12:13
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/21) Installing fstrm (0.6.0-r1)
(2/21) Installing krb5-conf (1.0-r2)
(3/21) Installing libcom_err (1.45.6-r0)
(4/21) Installing keyutils-libs (1.6.1-r1)
(5/21) Installing libverto (0.3.1-r1)
(6/21) Installing krb5-libs (1.18.5-r0)
(7/21) Installing json-c (0.14-r1)
(8/21) Installing libgcc (9.3.0-r2)
(9/21) Installing libstdc++ (9.3.0-r2)
(10/21) Installing libprotobuf (3.12.2-r0)
(11/21) Installing libprotoc (3.12.2-r0)
(12/21) Installing protobuf-c (1.3.3-r1)
(13/21) Installing libuv (1.38.1-r0)
(14/21) Installing xz-libs (5.2.5-r1)
(15/21) Installing libxml2 (2.9.14-r0)
(16/21) Installing bind-libs (9.16.27-r1)
(17/21) Installing bind-tools (9.16.27-r1)
(18/21) Installing ca-certificates (20211220-r0)
(19/21) Installing nghttp2-libs (1.41.0-r0)
(20/21) Installing libcurl (7.79.1-r1)
(21/21) Installing curl (7.79.1-r1)
Executing busybox-1.31.1-r22.trigger
Executing ca-certificates-20211220-r0.trigger
OK: 20 MiB in 35 packages
+ curl -fsSL https://slack.com
+ echo success
success
+ host -4 -AAAA slack.com
Trying "slack.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55308
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 13, ADDITIONAL: 8

;; QUESTION SECTION:
;slack.com.         IN  ANY

;; ANSWER SECTION:
slack.com.      38  IN  A   34.203.97.10
slack.com.      38  IN  A   34.193.255.5
slack.com.      38  IN  A   54.147.59.169
slack.com.      38  IN  A   34.231.24.224
slack.com.      38  IN  A   34.225.62.185
slack.com.      38  IN  A   3.95.117.96
slack.com.      38  IN  A   54.163.235.119
slack.com.      38  IN  A   34.204.109.226
slack.com.      38  IN  A   54.92.199.186

;; AUTHORITY SECTION:
.           208510  IN  NS  j.root-servers.net.
.           208510  IN  NS  a.root-servers.net.
.           208510  IN  NS  b.root-servers.net.
.           208510  IN  NS  g.root-servers.net.
.           208510  IN  NS  l.root-servers.net.
.           208510  IN  NS  d.root-servers.net.
.           208510  IN  NS  i.root-servers.net.
.           208510  IN  NS  h.root-servers.net.
.           208510  IN  NS  k.root-servers.net.
.           208510  IN  NS  f.root-servers.net.
.           208510  IN  NS  m.root-servers.net.
.           208510  IN  NS  e.root-servers.net.
.           208510  IN  NS  c.root-servers.net.

;; ADDITIONAL SECTION:
b.root-servers.net. 415251  IN  A   199.9.14.201
l.root-servers.net. 357447  IN  A   199.7.83.42
h.root-servers.net. 417397  IN  A   198.97.190.53
g.root-servers.net. 403642  IN  A   192.112.36.4
j.root-servers.net. 365003  IN  A   192.58.128.30
i.root-servers.net. 376774  IN  A   192.36.148.17
e.root-servers.net. 415445  IN  A   192.203.230.10
f.root-servers.net. 406658  IN  A   192.5.5.241

Received 510 bytes from 10.204.109.209#53 in 286 ms
+ apk list
+ grep musl
musl-dbg-1.1.24-r10 x86_64 {musl} (MIT)
musl-1.1.24-r10 x86_64 {musl} (MIT) [installed]
musl-locales-lang-0_git20200319-r0 x86_64 {musl-locales} (MIT)
musl-obstack-1.1-r1 x86_64 {musl-obstack} (GPL-2.0-or-later)
libc6-compat-1.1.24-r10 x86_64 {musl} (MIT)
musl-utils-1.1.24-r10 x86_64 {musl} (MIT BSD GPL2+) [installed]
musl-nscd-dev-1.0.3-r0 x86_64 {musl-nscd} (MIT)
musl-dev-1.1.24-r10 x86_64 {musl} (MIT)
musl-nscd-1.0.3-r0 x86_64 {musl-nscd} (MIT)
musl-locales-0_git20200319-r0 x86_64 {musl-locales} (LGPL-3.0-only)
musl-nscd-doc-1.0.3-r0 x86_64 {musl-nscd} (MIT)
musl-obstack-dev-1.1-r1 x86_64 {musl-obstack} (GPL-2.0-or-later)
musl-libintl-1.1.24-r10 x86_64 {musl} (MIT)

Things change with alpine 3.12 and musl-1.2.2-r1 (now it spits ou “curl: (6) Could not resolve host: slack.com”)

docker run -it --rm --dns=10.204.109.209 alpine:3.13 ash -c 'apk add curl bind-tools;set -x;curl -fsSL https://slack.com 1>/dev/null && echo success;host -4 -AAAA slack.com;apk list | grep musl'       ✓ ns-watch-attribution-nonprod 12:14
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
(1/22) Installing fstrm (0.6.0-r1)
(2/22) Installing krb5-conf (1.0-r2)
(3/22) Installing libcom_err (1.45.7-r0)
(4/22) Installing keyutils-libs (1.6.3-r0)
(5/22) Installing libverto (0.3.1-r1)
(6/22) Installing krb5-libs (1.18.5-r0)
(7/22) Installing json-c (0.15-r1)
(8/22) Installing libgcc (10.2.1_pre1-r3)
(9/22) Installing libstdc++ (10.2.1_pre1-r3)
(10/22) Installing libprotobuf (3.13.0-r2)
(11/22) Installing libprotoc (3.13.0-r2)
(12/22) Installing protobuf-c (1.3.3-r4)
(13/22) Installing libuv (1.40.0-r0)
(14/22) Installing xz-libs (5.2.5-r1)
(15/22) Installing libxml2 (2.9.14-r0)
(16/22) Installing bind-libs (9.16.27-r0)
(17/22) Installing bind-tools (9.16.27-r0)
(18/22) Installing ca-certificates (20211220-r0)
(19/22) Installing brotli-libs (1.0.9-r3)
(20/22) Installing nghttp2-libs (1.42.0-r1)
(21/22) Installing libcurl (7.79.1-r1)
(22/22) Installing curl (7.79.1-r1)
Executing busybox-1.32.1-r8.trigger
Executing ca-certificates-20211220-r0.trigger
OK: 21 MiB in 36 packages
+ curl -fsSL https://slack.com
curl: (6) Could not resolve host: slack.com
+ host -4 -AAAA slack.com
Trying "slack.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55471
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 13, ADDITIONAL: 8

;; QUESTION SECTION:
;slack.com.         IN  ANY

;; ANSWER SECTION:
slack.com.      23  IN  A   34.231.24.224
slack.com.      23  IN  A   54.163.235.119
slack.com.      23  IN  A   34.225.62.185
slack.com.      23  IN  A   34.203.97.10
slack.com.      23  IN  A   3.95.117.96
slack.com.      23  IN  A   34.193.255.5
slack.com.      23  IN  A   34.204.109.226
slack.com.      23  IN  A   54.92.199.186
slack.com.      23  IN  A   54.147.59.169

;; AUTHORITY SECTION:
.           208436  IN  NS  j.root-servers.net.
.           208436  IN  NS  a.root-servers.net.
.           208436  IN  NS  h.root-servers.net.
.           208436  IN  NS  m.root-servers.net.
.           208436  IN  NS  k.root-servers.net.
.           208436  IN  NS  l.root-servers.net.
.           208436  IN  NS  g.root-servers.net.
.           208436  IN  NS  f.root-servers.net.
.           208436  IN  NS  c.root-servers.net.
.           208436  IN  NS  b.root-servers.net.
.           208436  IN  NS  e.root-servers.net.
.           208436  IN  NS  d.root-servers.net.
.           208436  IN  NS  i.root-servers.net.

;; ADDITIONAL SECTION:
b.root-servers.net. 415177  IN  A   199.9.14.201
l.root-servers.net. 357373  IN  A   199.7.83.42
h.root-servers.net. 417323  IN  A   198.97.190.53
g.root-servers.net. 403568  IN  A   192.112.36.4
j.root-servers.net. 364929  IN  A   192.58.128.30
i.root-servers.net. 376700  IN  A   192.36.148.17
e.root-servers.net. 415371  IN  A   192.203.230.10
f.root-servers.net. 406584  IN  A   192.5.5.241

Received 510 bytes from 10.204.109.209#53 in 381 ms
+ apk list
+ grep musl
musl-dbg-1.2.2-r1 x86_64 {musl} (MIT)
musl-1.2.2-r1 x86_64 {musl} (MIT) [installed]
musl-locales-lang-0_git20200319-r1 x86_64 {musl-locales} (MIT)
musl-obstack-1.1-r1 x86_64 {musl-obstack} (GPL-2.0-or-later)
libc6-compat-1.2.2-r1 x86_64 {musl} (MIT)
musl-utils-1.2.2-r1 x86_64 {musl} (MIT BSD GPL2+) [installed]
musl-nscd-dev-1.0.3-r1 x86_64 {musl-nscd} (MIT)
musl-dev-1.2.2-r1 x86_64 {musl} (MIT)
musl-nscd-1.0.3-r1 x86_64 {musl-nscd} (MIT)
musl-locales-0_git20200319-r1 x86_64 {musl-locales} (LGPL-3.0-only)
musl-nscd-doc-1.0.3-r1 x86_64 {musl-nscd} (MIT)
musl-obstack-dev-1.1-r1 x86_64 {musl-obstack} (GPL-2.0-or-later)
musl-libintl-1.2.2-r1 x86_64 {musl} (MIT)

Still alpine 3.16 with musl-1.2.3-r0 fails:

docker run -it --rm --dns=10.204.109.209 alpine:3.16 ash -c 'apk add curl bind-tools;set -x;curl -fsSL https://slack.com 1>/dev/null && echo success;host -4 -AAAA slack.com;apk list | grep musl'       ✓ ns-watch-attribution-nonprod 12:15
Unable to find image 'alpine:3.16' locally
3.16: Pulling from library/alpine
Digest: sha256:686d8c9dfa6f3ccfc8230bc3178d23f84eeaf7e457f36f271ab1acc53015037c
Status: Downloaded newer image for alpine:3.16
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
(1/18) Installing fstrm (0.6.1-r0)
(2/18) Installing krb5-conf (1.0-r2)
(3/18) Installing libcom_err (1.46.5-r0)
(4/18) Installing keyutils-libs (1.6.3-r1)
(5/18) Installing libverto (0.3.2-r0)
(6/18) Installing krb5-libs (1.19.3-r0)
(7/18) Installing json-c (0.16-r0)
(8/18) Installing protobuf-c (1.4.0-r0)
(9/18) Installing libuv (1.44.1-r0)
(10/18) Installing xz-libs (5.2.5-r1)
(11/18) Installing libxml2 (2.9.14-r0)
(12/18) Installing bind-libs (9.16.29-r0)
(13/18) Installing bind-tools (9.16.29-r0)
(14/18) Installing ca-certificates (20211220-r0)
(15/18) Installing brotli-libs (1.0.9-r6)
(16/18) Installing nghttp2-libs (1.47.0-r0)
(17/18) Installing libcurl (7.83.1-r1)
(18/18) Installing curl (7.83.1-r1)
Executing busybox-1.35.0-r13.trigger
Executing ca-certificates-20211220-r0.trigger
OK: 15 MiB in 32 packages
+ curl -fsSL https://slack.com
curl: (6) Could not resolve host: slack.com
+ host -4 -AAAA slack.com
Trying "slack.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60635
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 13, ADDITIONAL: 8

;; QUESTION SECTION:
;slack.com.         IN  ANY

;; ANSWER SECTION:
slack.com.      5   IN  A   3.95.117.96
slack.com.      5   IN  A   54.92.199.186
slack.com.      5   IN  A   34.225.62.185
slack.com.      5   IN  A   34.203.97.10
slack.com.      5   IN  A   34.193.255.5
slack.com.      5   IN  A   54.163.235.119
slack.com.      5   IN  A   34.204.109.226
slack.com.      5   IN  A   34.231.24.224
slack.com.      5   IN  A   54.147.59.169

;; AUTHORITY SECTION:
.           208128  IN  NS  m.root-servers.net.
.           208128  IN  NS  a.root-servers.net.
.           208128  IN  NS  j.root-servers.net.
.           208128  IN  NS  f.root-servers.net.
.           208128  IN  NS  d.root-servers.net.
.           208128  IN  NS  h.root-servers.net.
.           208128  IN  NS  k.root-servers.net.
.           208128  IN  NS  b.root-servers.net.
.           208128  IN  NS  g.root-servers.net.
.           208128  IN  NS  i.root-servers.net.
.           208128  IN  NS  c.root-servers.net.
.           208128  IN  NS  l.root-servers.net.
.           208128  IN  NS  e.root-servers.net.

;; ADDITIONAL SECTION:
b.root-servers.net. 414869  IN  A   199.9.14.201
l.root-servers.net. 357065  IN  A   199.7.83.42
h.root-servers.net. 417015  IN  A   198.97.190.53
g.root-servers.net. 403260  IN  A   192.112.36.4
j.root-servers.net. 364621  IN  A   192.58.128.30
i.root-servers.net. 376392  IN  A   192.36.148.17
e.root-servers.net. 415063  IN  A   192.203.230.10
f.root-servers.net. 406276  IN  A   192.5.5.241

Received 510 bytes from 10.204.109.209#53 in 407 ms
+ apk list
+ grep musl
musl-dbg-1.2.3-r0 x86_64 {musl} (MIT)
musl-1.2.3-r0 x86_64 {musl} (MIT) [installed]
musl-locales-lang-0.1.0-r0 x86_64 {musl-locales} (MIT)
musl-obstack-1.2.3-r0 x86_64 {musl-obstack} (LGPL-2.1-or-later)
libc6-compat-1.2.3-r0 x86_64 {musl} (MIT)
musl-utils-1.2.3-r0 x86_64 {musl} (MIT BSD GPL2+) [installed]
musl-nscd-dev-1.1.1-r0 x86_64 {musl-nscd} (MIT)
musl-dev-1.2.3-r0 x86_64 {musl} (MIT)
musl-nscd-1.1.1-r0 x86_64 {musl-nscd} (MIT)
musl-locales-0.1.0-r0 x86_64 {musl-locales} (LGPL-3.0-only)
musl-nscd-doc-1.1.1-r0 x86_64 {musl-nscd} (MIT)
musl-legacy-error-0.5-r0 x86_64 {musl-legacy-error} (BSD-2-Clause)
musl-obstack-dev-1.2.3-r0 x86_64 {musl-obstack} (LGPL-2.1-or-later)
musl-libintl-1.2.3-r0 x86_64 {musl} (MIT)


Greetings,
Markus

--

Markus Geiger

Protean Linux | Cloud | DevOps Engineer

RefinedLabs - A Nielsen Company

www.nielsen.com



Refined Labs GmbH - A Nielsen Company

Herzog-Wilhelm-Straße 26, 80331 München

Geschäftsführer: Dietmar Krauss, Robert Moor

Sitz München, Amtsgericht München HRB 166589  


This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please be kind and notify the sender immediately and delete this mail and all its attachments subsequently. Please improve communication in the workplace by not using emails at all since they are not secure, anti-social, bring confusion, often destroy focus and lack transparency. Use encrypted group chats, social media or directly talk to people and use an agile task board for your daily planning. By reading this you agree to stop agreeing to useless disclaimers and learn about security and securing your communication.