From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-8.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HTML_MESSAGE,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 26586 invoked from network); 30 Sep 2022 23:04:18 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 30 Sep 2022 23:04:18 -0000 Received: (qmail 28046 invoked by uid 550); 30 Sep 2022 23:04:15 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 28010 invoked from network); 30 Sep 2022 23:04:14 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=oXaHb0PD4zYa63Yt03TdLj3hUIluuVxSWGf0BfsSiu8=; b=E9A8DROBjvZKX37KZmRyBgU3KmwcAITMJffOEFlp+aqEFbdlVPikLmd1qnfCkG+7Sv eTanXVSgVkbOrKSZlpawLN3uuofZqlwPvDDCs0DoPZIqHa4+/d1k0VMQaFokJbOIaTnw RnMUCX1iHILi3Mwh8O2vDnu7RGWP8zAk+U3WnovVxjHYeqKy7iEsRslCmxAlRklHLx8E iMtyg1ZSSfFMBMrp3GKMT2EeEK4tlzlbNWDBdBE8fOf59WpSEXMpBUiyvKikQ72NYQG7 rjLiNX1caMSbmaLnFyWBsSjnDf9RbZ+6NNZwFwPTzpoftlNHKKYW+B5rSGWFvY27Pt+/ 4otQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=oXaHb0PD4zYa63Yt03TdLj3hUIluuVxSWGf0BfsSiu8=; b=lJHYdXxR+3GIB0Gh8yzlbeVpfbyBOtceAw4ngbq5yifltVSjg1ahmbglYq9OHYkzv0 ja9paYr6Xz5Avr4C1Wa5IcBErZMiBhEWMrR0Vle/h1MR9PUu0Ma/Bi/Gx46JwUKoOC3V vRw7JhufljGb4cCNFzlELyf8iKB9SNVhGNfuk9NHRxDdJyFE3GzQk7+aYLu5FyKDeIOG CpH/PqXCItPZJcHxfS1BYkD1tndHJhcvdNwZJcW23FBS2RouFX+HYDcPhO/Iitp8WnwI 1oRL+yjYo8G3OEgE7kiGCL8HpCa+dZAmahaN/CmIiHTVaryy07YpojkcuZm8N6Y0kIn8 I3UQ== X-Gm-Message-State: ACrzQf1Y8NQHfI0tb5dMRcf2qsu0hTJDHBV9ocR4xrd7IgX1f4cSl724 LGfW13MDgGXSVfkSeJ+fX1z5bf5Y2FdTV8mvuQhdPA== X-Google-Smtp-Source: AMsMyM6cDrhooq1ab/NWrw6+wLf/N5+u5HTOLmpcI64WgvTCv/YOPDREquVcr+Ef4vjzqgP7co8oIv2GtElwq7dIsEw= X-Received: by 2002:a05:6512:1084:b0:4a2:eb2:9d9e with SMTP id j4-20020a056512108400b004a20eb29d9emr2900564lfg.553.1664579042461; Fri, 30 Sep 2022 16:04:02 -0700 (PDT) MIME-Version: 1.0 References: <20220926220449.GE9709@brightrain.aerifal.cx> <20220927122005.GG9709@brightrain.aerifal.cx> <20220927190357.GH9709@brightrain.aerifal.cx> <20220927190853.GI9709@brightrain.aerifal.cx> <20220929230707.GA29905@brightrain.aerifal.cx> <20220930024447.GC2645@voyager> <20220930125754.GB29905@brightrain.aerifal.cx> <20220930192617.GC29905@brightrain.aerifal.cx> In-Reply-To: <20220930192617.GC29905@brightrain.aerifal.cx> From: enh Date: Fri, 30 Sep 2022 16:03:50 -0700 Message-ID: To: Rich Felker Cc: musl@lists.openwall.com, Markus Wichmann Content-Type: multipart/alternative; boundary="000000000000876b7c05e9ed0286" Subject: Re: [musl] Revisiting LFS64 removal --000000000000876b7c05e9ed0286 Content-Type: text/plain; charset="UTF-8" On Fri, Sep 30, 2022 at 12:26 PM Rich Felker wrote: > On Fri, Sep 30, 2022 at 11:13:28AM -0700, enh wrote: > > On Fri, Sep 30, 2022 at 10:36 AM Colin Cross wrote: > > > > > On Fri, Sep 30, 2022 at 5:58 AM Rich Felker wrote: > > > > > > > > On Fri, Sep 30, 2022 at 04:44:47AM +0200, Markus Wichmann wrote: > > > > > On Thu, Sep 29, 2022 at 07:07:08PM -0400, Rich Felker wrote: > > > > > > As an alternative, maybe we should consider leaving these but > only > > > > > > under explict _LARGEFILE64_SOURCE rather than implicitly via > > > > > > _GNU_SOURCE for at least one release cycle. This would allow > > > makeshift > > > > > > fixing of any builds that break by just adding > -D_LARGEFILE64_SOURCE > > > > > > until a proper fix can be applied. > > > > > > > > > > > > Any preference here? > > > > > > > > > > > > Rich > > > > > > > > > > Given that nothing lasts as long as a temporary measure, I'd say > it is > > > > > > > > That's not a given for musl, quite the opposite. I would expect it to > > > > last at most a release cycle, possibly even to disappear before then > > > > if distros backport the changes to their development branches early > > > > and find and fix everything. > > > > > > > > > better to rip the band-aid off in one go rather than two. Besides, > any > > > > > breakage ought to be able to be dealt with by a simple replacement, > > > > > right? > > > > > > > > It's a simple fix, but the question is how many such simple fixes a > > > > distro might need to make in their packages, and that I don't know. > > > > > > > > If they have a trivial mechanical fix of "add something to CFLAGS" > > > > they can do at first, that lets them build a list of affected > packages > > > > while quickly getting them all building again, then work out the > right > > > > fixes one at a time according to usual triage rather than being > > > > swamped with these taking priority over issues with more depth. > > > > > > > > Rich > > > > > > I experimented with building all the host code in the Android tree > > > with these two patches just to measure the damage. The first patch is > > > mostly fine, but causes link failures in rust modules. I think that's > > > due to the upstream rust libc assuming the presence of fstat64, etc.: > > > > > > > https://cs.android.com/android/platform/superproject/+/master:external/rust/crates/libc/src/unix/linux_like/mod.rs;l=1665;drc=b38fde0ab980c7d79f0a55aec1b7121022a38257 > > > > > > The second patch causes 680 unique errors. Many of those are in > > > Android-specific code that uses the *64 functions directly, which I > > > think is especially common due to early bionic's poor LFS64 support. > > > > > > > ....and the fact that we can't flip the switch globally for the OS build > > without risking subtle ABI changes. but maybe these days we could > recognize > > such things (via the .lsdump stuff)? maybe it's time to try? > > > > (we have similar issues with code that's also built for macOS, which > never > > had the *64 functions or off64_t, so there might be some low-hanging > fruit > > by extending those `#if __APPLE__` hacks -- that are basically just > > `#define off64_t off_t` etc -- to apply to musl too...) > > The intended usage as I understand it is that you would probe for > sizeof(off_t)>=8 and only try to use these hacks if that's not > satisfied, not hard-code combinatoric platform knowledge. yeah, the problem was that we had (ten years ago) the following three situations, and code that needed to build with all three: 1. bionic at the time had a 32-bit off_t and didn't have _FILE_OFFSET_BITS, but did have off64_t and [some of] the *64 functions. 2. glibc had _FILE_OFFSET_BITS, but we weren't using it (or were using it set to 32?). 3. apple's libc didn't have _FILE_OFFSET_BITS, didn't have off64_t or the *64 functions, but everything was 64-bit anyway. so what's happened is that we've got some source (stuff that needs to build for device and host) where we have heavy use of the *64 functions, and then #define hacks in the case where it also needs to build for the mac. luckily, in the meantime we took over bionic and fixed that, though we _didn't_ change the default for `_FILE_OFFSET_BITS` for the ABI reasons i mentioned previously. what we _did_ do though was make the *host* build use `-D_FILE_OFFSET_BITS=64`. so given that musl is only for the host, replacing glibc, and it seems unlikely that much if any of this code actually needs to compile for ancient versions of bionic, i think we can just "remove the 64s". ccross: happy to shard that work if you want to go that route! > Something > like: > > checking if sizeof(off_t)>=8... > [if no] checking if -D_FILE_OFFSET_BITS=64 makes sizeof(off_t)>=8... > [if no] checking if -D_LARGEFILE64_SOURCE exposes off64_t interfaces... > ... > > Rich > --000000000000876b7c05e9ed0286 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Fri, Sep 30, 2022 at 12:26 PM Rich= Felker <dalias@libc.org> wrot= e:
On Fri, Sep 3= 0, 2022 at 11:13:28AM -0700, enh wrote:
> On Fri, Sep 30, 2022 at 10:36 AM Colin Cross <ccross@google.com> wrote:
>
> > On Fri, Sep 30, 2022 at 5:58 AM Rich Felker <dalias@libc.org> wrote:
> > >
> > > On Fri, Sep 30, 2022 at 04:44:47AM +0200, Markus Wichmann wr= ote:
> > > > On Thu, Sep 29, 2022 at 07:07:08PM -0400, Rich Felker w= rote:
> > > > > As an alternative, maybe we should consider leavin= g these but only
> > > > > under explict _LARGEFILE64_SOURCE rather than impl= icitly via
> > > > > _GNU_SOURCE for at least one release cycle. This w= ould allow
> > makeshift
> > > > > fixing of any builds that break by just adding -D_= LARGEFILE64_SOURCE
> > > > > until a proper fix can be applied.
> > > > >
> > > > > Any preference here?
> > > > >
> > > > > Rich
> > > >
> > > > Given that nothing lasts as long as a temporary measure= , I'd say it is
> > >
> > > That's not a given for musl, quite the opposite. I would= expect it to
> > > last at most a release cycle, possibly even to disappear bef= ore then
> > > if distros backport the changes to their development branche= s early
> > > and find and fix everything.
> > >
> > > > better to rip the band-aid off in one go rather than tw= o. Besides, any
> > > > breakage ought to be able to be dealt with by a simple = replacement,
> > > > right?
> > >
> > > It's a simple fix, but the question is how many such sim= ple fixes a
> > > distro might need to make in their packages, and that I don&= #39;t know.
> > >
> > > If they have a trivial mechanical fix of "add something= to CFLAGS"
> > > they can do at first, that lets them build a list of affecte= d packages
> > > while quickly getting them all building again, then work out= the right
> > > fixes one at a time according to usual triage rather than be= ing
> > > swamped with these taking priority over issues with more dep= th.
> > >
> > > Rich
> >
> > I experimented with building all the host code in the Android tre= e
> > with these two patches just to measure the damage.=C2=A0 The firs= t patch is
> > mostly fine, but causes link failures in rust modules.=C2=A0 I th= ink that's
> > due to the upstream rust libc assuming the presence of fstat64, e= tc.:
> >
> > https://cs.android.com/android/platform/superproject/+/master:extern= al/rust/crates/libc/src/unix/linux_like/mod.rs;l=3D1665;drc=3Db38fde0ab980c= 7d79f0a55aec1b7121022a38257
> >
> > The second patch causes 680 unique errors.=C2=A0 Many of those ar= e in
> > Android-specific code that uses the *64 functions directly, which= I
> > think is especially common due to early bionic's poor LFS64 s= upport.
> >
>
> ....and the fact that we can't flip the switch globally for the OS= build
> without risking subtle ABI changes. but maybe these days we could reco= gnize
> such things (via the .lsdump stuff)? maybe it's time to try?
>
> (we have similar issues with code that's also built for macOS, whi= ch never
> had the *64 functions or off64_t, so there might be some low-hanging f= ruit
> by extending those `#if __APPLE__` hacks -- that are basically just > `#define off64_t off_t` etc -- to apply to musl too...)

The intended usage as I understand it is that you would probe for
sizeof(off_t)>=3D8 and only try to use these hacks if that's not
satisfied, not hard-code combinatoric platform knowledge.

yeah, the problem was that we had (ten years ago) the follo= wing three situations, and code that needed to build with all three:
<= div>
1. bionic at the time had a 32-bit off_t and didn't = have _FILE_OFFSET_BITS, but did have off64_t and [some of] the *64 function= s.
2. glibc had _FILE_OFFSET_BITS, but we weren't using it (o= r were using it set to 32?).
3. apple's libc didn't have = _FILE_OFFSET_BITS, didn't have off64_t or the *64 functions, but everyt= hing was 64-bit anyway.

so what's happened is = that we've got some source (stuff that needs to build for device and ho= st) where we have heavy use of the *64 functions, and then #define hacks in= the case where it also needs to build for the mac.

luckily, in the meantime we took over bionic and fixed that, though we _d= idn't_ change the default for `_FILE_OFFSET_BITS` for the ABI reasons i= mentioned previously. what we _did_ do though was make the *host* build us= e `-D_FILE_OFFSET_BITS=3D64`. so given that musl is only for the host, repl= acing glibc, and it seems unlikely that much if any of this code actually n= eeds to compile for ancient versions of bionic, i think we can just "r= emove the 64s".

ccross: happy to shard that w= ork if you want to go that route!
=C2=A0
Something
like:

checking if sizeof(off_t)>=3D8...
[if no] checking if -D_FILE_OFFSET_BITS=3D64 makes sizeof(off_t)>=3D8...=
[if no] checking if -D_LARGEFILE64_SOURCE exposes off64_t interfaces...
...

Rich
--000000000000876b7c05e9ed0286--