Hej! First, I love MUSL (and alpine linux). Great project! We encountered a bug in our CI pipeline using alpine images in conjunction with AWS DNS servers - and it seems to be related to MUSL: $ curl -fsSL https://slack.com curl: (6) Could not resolve host: slack.com Usually that should return some HTML. It seems to affect only non-FQDN domains. As a workaround we use now full FQDN api.slack.com. But there is a bug in resolvement! It seems if an AAAA domain is queried over an IPV4 IP/DNS and doesn’t not return a record the overall resolvement of the domain fails. *DEBUG LOG* We try several alpine images and musl libs on an EC2 host with docker and AWS DNS exclusivly: - alpine 3.12 with musl-1.1.24-r10 is last known to work - alpine 3.13 with musl-1.2.2-r1 starts failing (something introduced in musl-1.2 ?) - current alpine 3.16 with current musl-1.2.3-r0 still fails alpine 3.12 with musl-1.1.24-r10 is last known to work (see string “success”) docker run -it --rm --dns=10.204.109.209 alpine:3.12 ash -c 'apk add curl bind-tools;set -x;curl -fsSL https://slack.com 1>/dev/null && echo success;host -4 -AAAA slack.com;apk list | grep musl' ✓ ns-watch-attribution-nonprod 12:13 fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz (1/21) Installing fstrm (0.6.0-r1) (2/21) Installing krb5-conf (1.0-r2) (3/21) Installing libcom_err (1.45.6-r0) (4/21) Installing keyutils-libs (1.6.1-r1) (5/21) Installing libverto (0.3.1-r1) (6/21) Installing krb5-libs (1.18.5-r0) (7/21) Installing json-c (0.14-r1) (8/21) Installing libgcc (9.3.0-r2) (9/21) Installing libstdc++ (9.3.0-r2) (10/21) Installing libprotobuf (3.12.2-r0) (11/21) Installing libprotoc (3.12.2-r0) (12/21) Installing protobuf-c (1.3.3-r1) (13/21) Installing libuv (1.38.1-r0) (14/21) Installing xz-libs (5.2.5-r1) (15/21) Installing libxml2 (2.9.14-r0) (16/21) Installing bind-libs (9.16.27-r1) (17/21) Installing bind-tools (9.16.27-r1) (18/21) Installing ca-certificates (20211220-r0) (19/21) Installing nghttp2-libs (1.41.0-r0) (20/21) Installing libcurl (7.79.1-r1) (21/21) Installing curl (7.79.1-r1) Executing busybox-1.31.1-r22.trigger Executing ca-certificates-20211220-r0.trigger OK: 20 MiB in 35 packages + curl -fsSL https://slack.com + echo success success + host -4 -AAAA slack.com Trying "slack.com" ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55308 ;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 13, ADDITIONAL: 8 ;; QUESTION SECTION: ;slack.com. IN ANY ;; ANSWER SECTION: slack.com. 38 IN A 34.203.97.10 slack.com. 38 IN A 34.193.255.5 slack.com. 38 IN A 54.147.59.169 slack.com. 38 IN A 34.231.24.224 slack.com. 38 IN A 34.225.62.185 slack.com. 38 IN A 3.95.117.96 slack.com. 38 IN A 54.163.235.119 slack.com. 38 IN A 34.204.109.226 slack.com. 38 IN A 54.92.199.186 ;; AUTHORITY SECTION: . 208510 IN NS j.root-servers.net. . 208510 IN NS a.root-servers.net. . 208510 IN NS b.root-servers.net. . 208510 IN NS g.root-servers.net. . 208510 IN NS l.root-servers.net. . 208510 IN NS d.root-servers.net. . 208510 IN NS i.root-servers.net. . 208510 IN NS h.root-servers.net. . 208510 IN NS k.root-servers.net. . 208510 IN NS f.root-servers.net. . 208510 IN NS m.root-servers.net. . 208510 IN NS e.root-servers.net. . 208510 IN NS c.root-servers.net. ;; ADDITIONAL SECTION: b.root-servers.net. 415251 IN A 199.9.14.201 l.root-servers.net. 357447 IN A 199.7.83.42 h.root-servers.net. 417397 IN A 198.97.190.53 g.root-servers.net. 403642 IN A 192.112.36.4 j.root-servers.net. 365003 IN A 192.58.128.30 i.root-servers.net. 376774 IN A 192.36.148.17 e.root-servers.net. 415445 IN A 192.203.230.10 f.root-servers.net. 406658 IN A 192.5.5.241 Received 510 bytes from 10.204.109.209#53 in 286 ms + apk list + grep musl musl-dbg-1.1.24-r10 x86_64 {musl} (MIT) musl-1.1.24-r10 x86_64 {musl} (MIT) [installed] musl-locales-lang-0_git20200319-r0 x86_64 {musl-locales} (MIT) musl-obstack-1.1-r1 x86_64 {musl-obstack} (GPL-2.0-or-later) libc6-compat-1.1.24-r10 x86_64 {musl} (MIT) musl-utils-1.1.24-r10 x86_64 {musl} (MIT BSD GPL2+) [installed] musl-nscd-dev-1.0.3-r0 x86_64 {musl-nscd} (MIT) musl-dev-1.1.24-r10 x86_64 {musl} (MIT) musl-nscd-1.0.3-r0 x86_64 {musl-nscd} (MIT) musl-locales-0_git20200319-r0 x86_64 {musl-locales} (LGPL-3.0-only) musl-nscd-doc-1.0.3-r0 x86_64 {musl-nscd} (MIT) musl-obstack-dev-1.1-r1 x86_64 {musl-obstack} (GPL-2.0-or-later) musl-libintl-1.1.24-r10 x86_64 {musl} (MIT) Things change with alpine 3.12 and musl-1.2.2-r1 (now it spits ou “curl: (6) Could not resolve host: slack.com”) docker run -it --rm --dns=10.204.109.209 alpine:3.13 ash -c 'apk add curl bind-tools;set -x;curl -fsSL https://slack.com 1>/dev/null && echo success;host -4 -AAAA slack.com;apk list | grep musl' ✓ ns-watch-attribution-nonprod 12:14 fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz (1/22) Installing fstrm (0.6.0-r1) (2/22) Installing krb5-conf (1.0-r2) (3/22) Installing libcom_err (1.45.7-r0) (4/22) Installing keyutils-libs (1.6.3-r0) (5/22) Installing libverto (0.3.1-r1) (6/22) Installing krb5-libs (1.18.5-r0) (7/22) Installing json-c (0.15-r1) (8/22) Installing libgcc (10.2.1_pre1-r3) (9/22) Installing libstdc++ (10.2.1_pre1-r3) (10/22) Installing libprotobuf (3.13.0-r2) (11/22) Installing libprotoc (3.13.0-r2) (12/22) Installing protobuf-c (1.3.3-r4) (13/22) Installing libuv (1.40.0-r0) (14/22) Installing xz-libs (5.2.5-r1) (15/22) Installing libxml2 (2.9.14-r0) (16/22) Installing bind-libs (9.16.27-r0) (17/22) Installing bind-tools (9.16.27-r0) (18/22) Installing ca-certificates (20211220-r0) (19/22) Installing brotli-libs (1.0.9-r3) (20/22) Installing nghttp2-libs (1.42.0-r1) (21/22) Installing libcurl (7.79.1-r1) (22/22) Installing curl (7.79.1-r1) Executing busybox-1.32.1-r8.trigger Executing ca-certificates-20211220-r0.trigger OK: 21 MiB in 36 packages + curl -fsSL https://slack.com curl: (6) Could not resolve host: slack.com + host -4 -AAAA slack.com Trying "slack.com" ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55471 ;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 13, ADDITIONAL: 8 ;; QUESTION SECTION: ;slack.com. IN ANY ;; ANSWER SECTION: slack.com. 23 IN A 34.231.24.224 slack.com. 23 IN A 54.163.235.119 slack.com. 23 IN A 34.225.62.185 slack.com. 23 IN A 34.203.97.10 slack.com. 23 IN A 3.95.117.96 slack.com. 23 IN A 34.193.255.5 slack.com. 23 IN A 34.204.109.226 slack.com. 23 IN A 54.92.199.186 slack.com. 23 IN A 54.147.59.169 ;; AUTHORITY SECTION: . 208436 IN NS j.root-servers.net. . 208436 IN NS a.root-servers.net. . 208436 IN NS h.root-servers.net. . 208436 IN NS m.root-servers.net. . 208436 IN NS k.root-servers.net. . 208436 IN NS l.root-servers.net. . 208436 IN NS g.root-servers.net. . 208436 IN NS f.root-servers.net. . 208436 IN NS c.root-servers.net. . 208436 IN NS b.root-servers.net. . 208436 IN NS e.root-servers.net. . 208436 IN NS d.root-servers.net. . 208436 IN NS i.root-servers.net. ;; ADDITIONAL SECTION: b.root-servers.net. 415177 IN A 199.9.14.201 l.root-servers.net. 357373 IN A 199.7.83.42 h.root-servers.net. 417323 IN A 198.97.190.53 g.root-servers.net. 403568 IN A 192.112.36.4 j.root-servers.net. 364929 IN A 192.58.128.30 i.root-servers.net. 376700 IN A 192.36.148.17 e.root-servers.net. 415371 IN A 192.203.230.10 f.root-servers.net. 406584 IN A 192.5.5.241 Received 510 bytes from 10.204.109.209#53 in 381 ms + apk list + grep musl musl-dbg-1.2.2-r1 x86_64 {musl} (MIT) musl-1.2.2-r1 x86_64 {musl} (MIT) [installed] musl-locales-lang-0_git20200319-r1 x86_64 {musl-locales} (MIT) musl-obstack-1.1-r1 x86_64 {musl-obstack} (GPL-2.0-or-later) libc6-compat-1.2.2-r1 x86_64 {musl} (MIT) musl-utils-1.2.2-r1 x86_64 {musl} (MIT BSD GPL2+) [installed] musl-nscd-dev-1.0.3-r1 x86_64 {musl-nscd} (MIT) musl-dev-1.2.2-r1 x86_64 {musl} (MIT) musl-nscd-1.0.3-r1 x86_64 {musl-nscd} (MIT) musl-locales-0_git20200319-r1 x86_64 {musl-locales} (LGPL-3.0-only) musl-nscd-doc-1.0.3-r1 x86_64 {musl-nscd} (MIT) musl-obstack-dev-1.1-r1 x86_64 {musl-obstack} (GPL-2.0-or-later) musl-libintl-1.2.2-r1 x86_64 {musl} (MIT) Still alpine 3.16 with musl-1.2.3-r0 fails: docker run -it --rm --dns=10.204.109.209 alpine:3.16 ash -c 'apk add curl bind-tools;set -x;curl -fsSL https://slack.com 1>/dev/null && echo success;host -4 -AAAA slack.com;apk list | grep musl' ✓ ns-watch-attribution-nonprod 12:15 Unable to find image 'alpine:3.16' locally 3.16: Pulling from library/alpine Digest: sha256:686d8c9dfa6f3ccfc8230bc3178d23f84eeaf7e457f36f271ab1acc53015037c Status: Downloaded newer image for alpine:3.16 fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz (1/18) Installing fstrm (0.6.1-r0) (2/18) Installing krb5-conf (1.0-r2) (3/18) Installing libcom_err (1.46.5-r0) (4/18) Installing keyutils-libs (1.6.3-r1) (5/18) Installing libverto (0.3.2-r0) (6/18) Installing krb5-libs (1.19.3-r0) (7/18) Installing json-c (0.16-r0) (8/18) Installing protobuf-c (1.4.0-r0) (9/18) Installing libuv (1.44.1-r0) (10/18) Installing xz-libs (5.2.5-r1) (11/18) Installing libxml2 (2.9.14-r0) (12/18) Installing bind-libs (9.16.29-r0) (13/18) Installing bind-tools (9.16.29-r0) (14/18) Installing ca-certificates (20211220-r0) (15/18) Installing brotli-libs (1.0.9-r6) (16/18) Installing nghttp2-libs (1.47.0-r0) (17/18) Installing libcurl (7.83.1-r1) (18/18) Installing curl (7.83.1-r1) Executing busybox-1.35.0-r13.trigger Executing ca-certificates-20211220-r0.trigger OK: 15 MiB in 32 packages + curl -fsSL https://slack.com curl: (6) Could not resolve host: slack.com + host -4 -AAAA slack.com Trying "slack.com" ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60635 ;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 13, ADDITIONAL: 8 ;; QUESTION SECTION: ;slack.com. IN ANY ;; ANSWER SECTION: slack.com. 5 IN A 3.95.117.96 slack.com. 5 IN A 54.92.199.186 slack.com. 5 IN A 34.225.62.185 slack.com. 5 IN A 34.203.97.10 slack.com. 5 IN A 34.193.255.5 slack.com. 5 IN A 54.163.235.119 slack.com. 5 IN A 34.204.109.226 slack.com. 5 IN A 34.231.24.224 slack.com. 5 IN A 54.147.59.169 ;; AUTHORITY SECTION: . 208128 IN NS m.root-servers.net. . 208128 IN NS a.root-servers.net. . 208128 IN NS j.root-servers.net. . 208128 IN NS f.root-servers.net. . 208128 IN NS d.root-servers.net. . 208128 IN NS h.root-servers.net. . 208128 IN NS k.root-servers.net. . 208128 IN NS b.root-servers.net. . 208128 IN NS g.root-servers.net. . 208128 IN NS i.root-servers.net. . 208128 IN NS c.root-servers.net. . 208128 IN NS l.root-servers.net. . 208128 IN NS e.root-servers.net. ;; ADDITIONAL SECTION: b.root-servers.net. 414869 IN A 199.9.14.201 l.root-servers.net. 357065 IN A 199.7.83.42 h.root-servers.net. 417015 IN A 198.97.190.53 g.root-servers.net. 403260 IN A 192.112.36.4 j.root-servers.net. 364621 IN A 192.58.128.30 i.root-servers.net. 376392 IN A 192.36.148.17 e.root-servers.net. 415063 IN A 192.203.230.10 f.root-servers.net. 406276 IN A 192.5.5.241 Received 510 bytes from 10.204.109.209#53 in 407 ms + apk list + grep musl musl-dbg-1.2.3-r0 x86_64 {musl} (MIT) musl-1.2.3-r0 x86_64 {musl} (MIT) [installed] musl-locales-lang-0.1.0-r0 x86_64 {musl-locales} (MIT) musl-obstack-1.2.3-r0 x86_64 {musl-obstack} (LGPL-2.1-or-later) libc6-compat-1.2.3-r0 x86_64 {musl} (MIT) musl-utils-1.2.3-r0 x86_64 {musl} (MIT BSD GPL2+) [installed] musl-nscd-dev-1.1.1-r0 x86_64 {musl-nscd} (MIT) musl-dev-1.2.3-r0 x86_64 {musl} (MIT) musl-nscd-1.1.1-r0 x86_64 {musl-nscd} (MIT) musl-locales-0.1.0-r0 x86_64 {musl-locales} (LGPL-3.0-only) musl-nscd-doc-1.1.1-r0 x86_64 {musl-nscd} (MIT) musl-legacy-error-0.5-r0 x86_64 {musl-legacy-error} (BSD-2-Clause) musl-obstack-dev-1.2.3-r0 x86_64 {musl-obstack} (LGPL-2.1-or-later) musl-libintl-1.2.3-r0 x86_64 {musl} (MIT) Greetings, Markus -- Markus Geiger Protean Linux | Cloud | DevOps Engineer RefinedLabs - A Nielsen Company www.nielsen.com Refined Labs GmbH - A Nielsen Company Herzog-Wilhelm-Straße 26, 80331 München Geschäftsführer: Dietmar Krauss, Robert Moor Sitz München, Amtsgericht München HRB 166589 *This e-mail may contain confidential and/or privileged information. * *If you are not the intended recipient or have received this e-mail in error please be kind and notify the sender immediately and delete this mail and all its attachments subsequently. Please improve communication in the workplace by not using emails at all since they are not secure, anti-social, bring confusion, often destroy focus and lack transparency. Use encrypted group chats, social media or directly talk to people and use an agile task board for your daily planning. By reading this you agree to stop agreeing to useless disclaimers and learn about security and securing your communication.*