Re: [musl] Re: musl getaddr info breakage on older kernels

[Satadru Pramanik <satadru@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 7694 bytes --]

Here are illustrative logs of output and strace logs.

Note that while the musl toolchain is built in a container on a much more
powerful machine, this "musl_getaddrinfo_test" app is built locally on the
machine with the 3.8 kernel.

I ran the following to get the output on the smaller i686 machine
immediately after the app is built.
Apologies for the ruby code wrapping the shell commands.

    @musl_ver = `#{CREW_MUSL_PREFIX}/lib/libc.so 2>&1 >/dev/null | head -2
| tail -1 | awk '{print $2}'`.chomp
    puts 'Testing the musl resolver to see if it can resolve google.com:
'.lightblue
    system "./musl_getaddrinfo_test google.com set_ai_family 2>&1 |tee -a
/tmp/musl_#{@musl_ver}_getaddrinfo_test_google.com_set_ai_family.txt "
    system "./musl_getaddrinfo_test google.com 2>&1 |tee -a
/tmp/musl_#{@musl_ver}_getaddrinfo_test_google.com.txt"
    system "strace -o
/tmp/musl_#{@musl_ver}_getaddrinfo_test_google.com_set_ai_family_STRACE.txt
./musl_getaddrinfo_test google.com set_ai_family"
    system "strace -o
/tmp/musl_#{@musl_ver}_getaddrinfo_test_google.com_STRACE.txt
./musl_getaddrinfo_test google.com"

And here is the output for each run before running again via strace. Note
how IPv6 addresses show up sporadically, and for 1.2.2 nothing at all shows
up, but everything works fine according to the strace logs. (Strace is
built against glibc 2.23.)

==>
musl_1.2.0-git-17-g33338ebc_getaddrinfo_test_google.com_set_ai_family.txt
<==
AF_INET: 142.251.40.110

==> musl_1.2.0-git-17-g33338ebc_getaddrinfo_test_google.com.txt <==
AF_INET: 142.251.40.110

==>
musl_1.2.0-git-39-g5cf1ac24_getaddrinfo_test_google.com_set_ai_family.txt
<==
AF_INET: 142.251.40.142

==> musl_1.2.0-git-39-g5cf1ac24_getaddrinfo_test_google.com.txt <==
getaddrinfo: Try again

==>
musl_1.2.0-git-40-g1b4e84c5_getaddrinfo_test_google.com_set_ai_family.txt
<==
AF_INET: 142.251.40.206

==> musl_1.2.0-git-40-g1b4e84c5_getaddrinfo_test_google.com.txt <==
AF_INET6: 2607:f8b0:4006:81f::200e
AF_INET: 142.251.40.206

==>
musl_1.2.0-git-6-g2f2348c9_getaddrinfo_test_google.com_set_ai_family.txt <==
AF_INET: 142.250.65.206

==> musl_1.2.0-git-6-g2f2348c9_getaddrinfo_test_google.com.txt <==
AF_INET: 142.250.65.206

==> musl_1.2.1_getaddrinfo_test_google.com_set_ai_family.txt <==
AF_INET: 142.251.40.110

==> musl_1.2.1_getaddrinfo_test_google.com.txt <==
getaddrinfo: Try again

==> musl_1.2.2_getaddrinfo_test_google.com_set_ai_family.txt <==
getaddrinfo: Try again

==> musl_1.2.2_getaddrinfo_test_google.com.txt <==
getaddrinfo: Try again

Regards,

Satadru


On Sun, Feb 6, 2022 at 6:44 PM Rich Felker <dalias@aerifal.cx> wrote:

> On Sun, Feb 06, 2022 at 06:25:34PM -0500, Satadru Pramanik wrote:
> > So here's the funny thing. These test binaries always work when run
> through
> > strace, sometimes even working once the binary been run after it has been
> > run with strace, but tends to fail when musl is installed, and then run
> for
> > the first time after being built against musl. (It isn't just these test
> > case binaries which have this issue. I've seen the same issue with the
> test
> > binaries from c_ares.)
> >
> > Does that ring a bell at all? Does it make sense for this to run
> > correctly when run with strace (built against glibc) but have problems
> when
> > run w/o strace?
>
> Can you include that strace, even if it's for a run that worked? It
> might include evidence of the cause. My best guess based on this
> symptom is a buggy Docker version blocking time64 syscalls with EPERM.
>
> > Happy to provide binaries as a well as instructions for a container setup
> > to build musl like I'm doing, which can generate such binaries.
> >
> > Of course this could be a buggy nameserver issue. My test setup uses an
> > OpenWRT dnsmasq name server forwarding to 9.9.9.9, 8.8.8.8, and 1.1.1.1
> >
> > Satadru
> >
> > On Sun, Feb 6, 2022 at 4:30 PM Rich Felker <dalias@aerifal.cx> wrote:
> >
> > > On Sun, Feb 06, 2022 at 03:32:40PM -0500, Satadru Pramanik wrote:
> > > > Hello,
> > > >
> > > > I'm a dev for the Chromebrew linux distribution, and we've been
> trying to
> > > > support older EOL i686 chromebooks stuck on older kernels.
> > > >
> > > > We have noticed that running newer versions of musl on such machines
> > > breaks
> > > > getaddrinfo in musl. This is a problem as we are trying to build
> > > > musl-static builds of curl and git which will work on these older
> > > machines.
> > > > (This is really irritating since these binaries work fine in i686
> > > > containers on newer machines running newer kernels, but then fail
> when
> > > run
> > > > on the hardware which has the older kernels.)
> > >
> > > The problem is almost surely not the kernel or musl but either buggy
> > > nameservers or buggy Docker. Can you post the strace log of the
> > > failing test case? That would quickly determine which it is.
> > >
> > > > I have been attempting to bisect where this happens, and have
> > > > determined that the first commit which breaks DNS resolution on musl
> for
> > > > these older machines is
> > > >
> > >
> https://git.musl-libc.org/cgit/musl/commit/?id=2f2348c9588d61680123bbe438db38acf5dfea4c
> > >
> > > This commit does not change anything in musl, which does not use those
> > > macros at all internally, just the public headers. So the bisect
> > > result must be wrong.
> > >
> > > > (I emailed you since you are the person who made that commit.)
> > >
> > > Please mail the list. I've CC'd it in my reply here.
> > >
> > > > Just applying a reverse of that commit allows musl networking to work
> > > > properly up to the (2020-03-14)
> 2b2c8aafce9d80f9d58652643538f4d58e82b856
> > > > commit. I'm trying newer commits with a
> > > > 2f2348c9588d61680123bbe438db38acf5dfea4c reversal patch to see if
> where
> > > > networking breaks again, as later commits such as the (2020-05-18)
> > > > fd7ec068efd590c0393a612599a4fab9bb0a8633 commit have broken ipv6 DNS
> > > > resolving.
> > > >
> > > > I'd like to be able to get this issue fixed, as we would like to use
> a
> > > > newer version of musl not susceptible to CVE-2020-28928.
> > > >
> > > > The test case I'm using is the code at
> > > > https://www.openwall.com/lists/musl/2021/07/19/1
> > > >
> > > > Our implementation is here:
> > > >
> > >
> https://github.com/skycocker/chromebrew/blob/master/packages/musl_getaddrinfo_test.rb
> > > >
> > > > Once I find other commits which are breaking this, I'd like to be
> able to
> > > > get a patch integrated into musl so that we don't have to maintain a
> > > > patchset for older machines.
> > > >
> > > > Would that be feasible? And my apologies if this should have gone to
> one
> > > of
> > > > the mailing lists first. I figured I should ask you since you might
> have
> > > > the most insight into this breaking installs.
> > > >
> > > > Here is the current reversal patch we are testing with:
> > > >
> > > > --- b/include/arpa/inet.h
> > > > +++ a/include/arpa/inet.h
> > > > @@ -24,6 +24,11 @@
> > > >  in_addr_t inet_lnaof(struct in_addr);
> > > >  in_addr_t inet_netof(struct in_addr);
> > > >
> > > > +#undef INET_ADDRSTRLEN
> > > > +#undef INET6_ADDRSTRLEN
> > > > +#define INET_ADDRSTRLEN  16
> > > > +#define INET6_ADDRSTRLEN 46
> > > > +
> > > >  #ifdef __cplusplus
> > > >  }
> > > >  #endif
> > > > --- b/include/netinet/in.h
> > > > +++ a/include/netinet/in.h
> > > > @@ -60,6 +60,8 @@
> > > >
> > > >  extern const struct in6_addr in6addr_any, in6addr_loopback;
> > > >
> > > > +#undef INET_ADDRSTRLEN
> > > > +#undef INET6_ADDRSTRLEN
> > > >  #define INET_ADDRSTRLEN  16
> > > >  #define INET6_ADDRSTRLEN 46
> > > >
> > > > Thanks,
> > > >
> > > > Satadru Pramanik
> > > > Dev Team
> > > > Chromebrew
> > >
>

[-- Attachment #1.2: Type: text/html, Size: 10278 bytes --]

[-- Attachment #2: musl_getaddrinfo_logs_i686.tgz --]
[-- Type: application/gzip, Size: 3783 bytes --]