From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 28759 invoked from network); 16 Feb 2022 21:53:57 -0000 Received: from mother.openwall.net (195.42.179.200) by inbox.vuxu.org with ESMTPUTF8; 16 Feb 2022 21:53:57 -0000 Received: (qmail 5971 invoked by uid 550); 16 Feb 2022 21:53:55 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 5935 invoked from network); 16 Feb 2022 21:53:54 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=CRqoOkUCwKec3x/HGo4I19P9niApMrFLncpIAK/eq+I=; b=SOSgLcQ1+KDwXC26lmMYhKX2WkAZTr9fojRJNmKLeytqfEEgFVYd1+D7WyL0JwkVsj KSh3jB5z2si0ovs3exvH1bDkshVm0TLC99J4nPqVwK+0JLZh2BqLuzuGaS7LDd5oSyIk YsO0thpzetFyFGCV2NAb8mRUuWxVGvcYdw+om+kKpdzUC7rcFj49y8X69Og8rV8NggmD /FYwtF0Q0KeX9BWt6yv3eqQO55H01qX39yxh8VMijpMmNpeu7JC+5zMX07HpZnDwpFNP M6aSlbEi8kcOfIApisV/ckk3IpLSeUoLqyjr52R2BGiZepAOXNQbZJzCWW0yx4T+8fNR bV2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=CRqoOkUCwKec3x/HGo4I19P9niApMrFLncpIAK/eq+I=; b=JTmQDOytJzT3MH8zY7k5Obo6AeaGkt1iruxLccU6e+7x0eERPuhoCD6HCX1ZT506VX IOdFQP1LMkdX8rDCqrBZXD5w3yGlbsxMpqazqzHzPHXUGAdNdnTp/Dr0qSAe9iv+xO8E 57dth7sf1Lf+WQwBD47fCmFEClX/8uwVVyMym4FtdK3Ht0lqk7JbQPxmIyo+oWVXzUwD 1NyV0iR99a7xw+JQym9ZQ5Glmw3wmZkeh9AovAWY7vwsVHqTwbTiurPB0yp8dP7A79Et cFTArwIe+0MlXI8loTr87Y8AvSdVEWrUTTtX4N8iPubYB4SpAzsgx1YdtDBHLbxdXw3Q w4Jw== X-Gm-Message-State: AOAM532fwRWVJBIJ48h6LHG9jAiPBHr5sXIP5cwQvSs5wM+qCy8+ZAfd 23qrVPe6BH3krgVaxCpUR2Jql5WCrxyoLZp2efv4Ug/2 X-Google-Smtp-Source: ABdhPJwMtDpCTHEZp8uToFKYquvoWsrr9Z+HK++i20z/4+Xb6vI34rBFFOSaLAd3Lyz6s4aqNifa0gwKPuQ9he5gEZY= X-Received: by 2002:a05:651c:12c6:b0:244:c116:3b75 with SMTP id 6-20020a05651c12c600b00244c1163b75mr21018lje.193.1645048422853; Wed, 16 Feb 2022 13:53:42 -0800 (PST) MIME-Version: 1.0 References: <20220214182952.GI7074@brightrain.aerifal.cx> <20220214220043.GK7074@brightrain.aerifal.cx> <20220215174420.GL7074@brightrain.aerifal.cx> <20220216014153.GM7074@brightrain.aerifal.cx> <20220216213335.GO7074@brightrain.aerifal.cx> In-Reply-To: <20220216213335.GO7074@brightrain.aerifal.cx> From: Satadru Pramanik Date: Wed, 16 Feb 2022 16:53:31 -0500 Message-ID: To: Rich Felker Cc: musl@lists.openwall.com Content-Type: multipart/alternative; boundary="000000000000e1f05205d829ae88" Subject: Re: [musl] Re: musl getaddr info breakage on older kernels --000000000000e1f05205d829ae88 Content-Type: text/plain; charset="UTF-8" I was looking at that commit too. I've started a build with that reverted and should be able to check back on that tomorrow. On Wed, Feb 16, 2022 at 4:33 PM Rich Felker wrote: > On Wed, Feb 16, 2022 at 01:44:35PM -0500, Satadru Pramanik wrote: > > The only change to socket.c I'm seeing is use __socketcall to simplify > > socket() > > < > https://git.musl-libc.org/cgit/musl/commit/?id=7063c459e7dbd63c2c94e04413743abab5272001 > >, > > so maybe it would make sense for me to try building with that reversed? > > That should not be a functional change, but you may be overlooking > commit c2feda4e2ea61f4da73f2f38b2be5e327a7d1a91, which was: using the > new (added in 4.3) individual socket syscalls instead of the legacy > multiplexed SYS_socketcall. It's supposed to fall back to using the > old ones, but perhaps something goes wrong on your kernel that's > preventing it. I'm not sure what the mechanism by which it works when > straced/single-stepped could be, though, but if it's a weird kernel > bug anything is possible. > > Reverting that commit should be entirely safe, if it turns out to be > what's triggering your problem, but I'd like to get to the root cause > and see if there's anything we can do to ensure this doesn't come up > again. > > > > On Wed, Feb 16, 2022 at 1:37 PM Satadru Pramanik > wrote: > > > > > > > >> > > >> - Whether any network traffic occurs when it fails (in the real > > >> environment not a replicated one elsewhere). > > >> > > >> > > > There is no network traffic in the real environment. > > > > > > > > >> - Whether it fails or succeeds under strace (in the real > > >> environment not a replicated one elsewhere). > > >> > > >> It succeeds in strace (in the real environment) > > > > > > > > > > > >> - Whether the real environment involves Docker or not. > > >> > > >> The real environment does not involve docker. > > > > > > > > > > > >> - What's in resolv.conf (in the real environment not a replicated one > > >> elsewhere) and what nameserver software (if known) is running on the > > >> nameserver(s) listed in there. > > >> > > >> The nameserver is picked up from dhcp. The contents of the file are as > > > follows: > > > nameserver 192.168.0.1 > > > search lan. > > > options single-request timeout:1 attempts:5 > > > > > > > > >> - Anything else that might be relevant. > > >> > > >> DNS server is dnsmasq running on a current OpenWRT device. > > > > > > > > >> It's really hard to offer any productive advice when the problem is > > >> unclear. > > >> > > >> Apologies for the confusion. > > > I'm really just trying to debug this getaddrinfo breakage on this older > > > hardware. The docker containers setup is something we use to build > packages > > > for this hardware, and our frustration is that the software works > perfectly > > > fine in the docker containers, but not on the hardware. > > > > > > > Any other suggestions on how to track down this issue? > > >> > > >> Rather than stepping through, I would put a single breakpoint at a > > >> place you want to see whether execution reaches before running the > > >> test program, then start it and see if the breakpoint fires or not. > > >> Then remove the breakpoint, add a different one, and repeat. For > > >> example, see if __res_msend is ever called, and if so, whether > > >> particular lines of it are reached (or just put breakpoints on some of > > >> the functions it calls, like socket, bind, recvfrom, poll, etc. to see > > >> if they're called). > > >> > > >> It might also be useful to put a breakpoint on clock_gettime and then > > >> 'finish' to see what it returns (in case the problem is something > > >> time64-related). > > >> > > >> > > > The only breakpoint which fixed the execution was for line 20 (which > > > invokes getaddrinfo). Stepping through the __kernel_vsyscall and then > > > continuing is the only way it does not result in failure. > > > > > > Any later breakpoints fail. > > > > > > I went though the other breakpoints as requested. > > > clock_gettime did not fire. > > > > > > Breakpoint 1 at 0x5c2f7: file > ../src_musl/compat/time32/clock_gettime32.c, > > > line 9. > > > __res_msend, setsockopt also did not fire. > > > The ones that did fire were: socket, bind, recvfrom, poll, > __res_msend_rc, > > > memset, sendto, __get_resolv_conf, pthread_setcancelstate, > > > __pthread_setcancelstate, __lookup_serv, __lookup_name, memcpy > > > > > > When breaking on socket, stepping through the __kernel_vsyscall call > after > > > socket and then continuing succeeds. > > > > > > Is it possible that the socket is not waiting long enough for a > response > > > from __kernel_vsyscall? Has that changed? > > > Breaking, stepping, and continuing on every other function above fails. > > > > > > The gdb log is attached. > > > > > > Regards, > > > > > > Satadru > > > > > > > --000000000000e1f05205d829ae88 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I was looking at that commit too. I've started a build= with that reverted and should be able to check back on that tomorrow.
On We= d, Feb 16, 2022 at 4:33 PM Rich Felker <dalias@aerifal.cx> wrote:
On Wed, Feb 16, 2022 at 01:44:35PM -0500, Satadru Pra= manik wrote:
> The only change to socket.c I'm seeing is use __socketcall to simp= lify
> socket()
> <ht= tps://git.musl-libc.org/cgit/musl/commit/?id=3D7063c459e7dbd63c2c94e0441374= 3abab5272001>,
> so maybe it would make sense for me to try building with that reversed= ?

That should not be a functional change, but you may be overlooking
commit c2feda4e2ea61f4da73f2f38b2be5e327a7d1a91, which was: using the
new (added in 4.3) individual socket syscalls instead of the legacy
multiplexed SYS_socketcall. It's supposed to fall back to using the
old ones, but perhaps something goes wrong on your kernel that's
preventing it. I'm not sure what the mechanism by which it works when straced/single-stepped could be, though, but if it's a weird kernel
bug anything is possible.

Reverting that commit should be entirely safe, if it turns out to be
what's triggering your problem, but I'd like to get to the root cau= se
and see if there's anything we can do to ensure this doesn't come u= p
again.


> On Wed, Feb 16, 2022 at 1:37 PM Satadru Pramanik <satadru@gmail.com> wrote:
>
> >
> >>
> >> - Whether any network traffic occurs when it fails (in the re= al
> >>=C2=A0 =C2=A0environment not a replicated one elsewhere).
> >>
> >>
> > There is no network traffic in the real environment.
> >
> >
> >> - Whether it fails or succeeds under strace (in the real
> >>=C2=A0 =C2=A0environment not a replicated one elsewhere).
> >>
> >> It succeeds in strace (in the real environment)
> >
> >
> >
> >> - Whether the real environment involves Docker or not.
> >>
> >> The real environment does not involve docker.
> >
> >
> >
> >> - What's in resolv.conf (in the real environment not a re= plicated one
> >>=C2=A0 =C2=A0elsewhere) and what nameserver software (if known= ) is running on the
> >>=C2=A0 =C2=A0nameserver(s) listed in there.
> >>
> >> The nameserver is picked up from dhcp. The contents of the fi= le are as
> > follows:
> > nameserver 192.168.0.1
> > search lan.
> > options single-request timeout:1 attempts:5
> >
> >
> >> - Anything else that might be relevant.
> >>
> >> DNS server is dnsmasq running on a current OpenWRT device. > >
> >
> >> It's really hard to offer any productive advice when the = problem is
> >> unclear.
> >>
> >> Apologies for the confusion.
> > I'm really just trying to debug this getaddrinfo breakage on = this older
> > hardware. The docker containers setup is something we use to buil= d packages
> > for this hardware, and our frustration is that the software works= perfectly
> > fine in the docker containers, but not on the hardware.
> >
> > > Any other suggestions on how to track down this issue?
> >>
> >> Rather than stepping through, I would put a single breakpoint= at a
> >> place you want to see whether execution reaches before runnin= g the
> >> test program, then start it and see if the breakpoint fires o= r not.
> >> Then remove the breakpoint, add a different one, and repeat. = For
> >> example, see if __res_msend is ever called, and if so, whethe= r
> >> particular lines of it are reached (or just put breakpoints o= n some of
> >> the functions it calls, like socket, bind, recvfrom, poll, et= c. to see
> >> if they're called).
> >>
> >> It might also be useful to put a breakpoint on clock_gettime = and then
> >> 'finish' to see what it returns (in case the problem = is something
> >> time64-related).
> >>
> >>
> > The only breakpoint which fixed the execution was for line 20 (wh= ich
> > invokes getaddrinfo). Stepping through the __kernel_vsyscall and = then
> > continuing is the only way it does not result in failure.
> >
> > Any later breakpoints fail.
> >
> > I went though the other breakpoints as requested.
> > clock_gettime did not fire.
> >
> > Breakpoint 1 at 0x5c2f7: file ../src_musl/compat/time32/clock_get= time32.c,
> > line 9.
> > __res_msend, setsockopt also did not fire.
> > The ones that did fire were: socket, bind, recvfrom, poll, __res_= msend_rc,
> > memset, sendto, __get_resolv_conf, pthread_setcancelstate,
> > __pthread_setcancelstate, __lookup_serv, __lookup_name, memcpy > >
> > When breaking on socket, stepping through the __kernel_vsyscall c= all after
> > socket and then continuing succeeds.
> >
> > Is it possible that the socket is not waiting long enough for a r= esponse
> > from __kernel_vsyscall? Has that changed?
> > Breaking, stepping, and continuing on every other function above = fails.
> >
> > The gdb log is attached.
> >
> > Regards,
> >
> > Satadru
> >
> >
--000000000000e1f05205d829ae88--