From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FROM,HTML_MESSAGE,MAILING_LIST_MULTI, RCVD_IN_MSPIKE_H2 autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 18298 invoked from network); 28 Jul 2022 20:39:25 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 28 Jul 2022 20:39:25 -0000 Received: (qmail 20184 invoked by uid 550); 28 Jul 2022 20:39:22 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 20148 invoked from network); 28 Jul 2022 20:39:21 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mac.com; s=1a1hai; t=1659040749; bh=bFYqUhZQDw58ECVSActaDYG1SjMOBqA0WNY5OvdPAgA=; h=From:Content-Type:Mime-Version:Subject:Date:To:Message-Id; b=OKiI/BnYGZN/O0khIy6juy0vkXWP5TPqtuNret/NNtBMJ4nOUgQxGVN7WFbn5ENjx f4FDL3yhc07YUL4y20W1AQ7PD0vn9mYvGqdSwOnb+1jxBs4+RuLV2LFvGdBk+lPcms YzB8pVubh92W3zTsgHqhEl0gyzs9XV1CKpT9TxKnfv3HoT9zua0ZvO4BlEVVwRDaE2 B04f4euOEb7vJlSfGuyALTvcgwhPaPUnHkKxO5VGw6uAj6xRkD3c+/MZeh4m8vMPTE Y2VxCpOwrp8CNP5rwdMqTtJZvsJOL0P4oTlO8hqAhhEGW7XjuOMdS1ilCd6FCq+Q8o FpnjkDHOb/JcA== From: Christopher Sean Morrison Content-Type: multipart/alternative; boundary="Apple-Mail=_397097C9-F54C-4FF2-AFCD-B404D80E2A8B" Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Date: Thu, 28 Jul 2022 16:39:07 -0400 References: <43e9ad2b-d53c-5fd5-0211-766f946c90f9@landley.net> <48546934-5ccd-0b1e-bf2b-306133101c6b@landley.net> <03A11083-A7D8-409E-BA70-AC42F52FF7B2@mac.com> <20220728091827.GD1320090@port70.net> To: musl@lists.openwall.com In-Reply-To: <20220728091827.GD1320090@port70.net> Message-Id: X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Proofpoint-GUID: Ns-BHULw4qkoroSsC0yYKN0K5TA5eCSi X-Proofpoint-ORIG-GUID: Ns-BHULw4qkoroSsC0yYKN0K5TA5eCSi X-Proofpoint-Virus-Version: =?UTF-8?Q?vendor=3Dfsecure_engine=3D1.1.170-22c6f66c430a71ce266a39bfe25bc?= =?UTF-8?Q?2903e8d5c8f:6.0.138,18.0.790,17.11.62.513.0000000_definitions?= =?UTF-8?Q?=3D2022-01-12=5F03:2020-02-14=5F02,2022-01-12=5F03,2021-12-02?= =?UTF-8?Q?=5F01_signatures=3D0?= X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0 clxscore=1015 spamscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2206140000 definitions=main-2207280094 Subject: Re: [musl] dynamic linker is capturing "reserved" library names erroneously --Apple-Mail=_397097C9-F54C-4FF2-AFCD-B404D80E2A8B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jul 28, 2022, at 5:18 AM, Szabolcs Nagy wrote: >=20 > * Christopher Sean Morrison [2022-07-27 19:06:24 = -0400]: >> First consideration, the code seems to take a position that those = library names are somehow universally reserved and I believe that to be = incorrect. >=20 > the way libraries are looked up is not specified in detail and > the standard gives the permission to implementations to decide. This is my understanding as well. That=E2=80=99s why I characterized it = as incorrect with respect to the existing implementation identifying = them as =E2=80=9Creserved=E2=80=9D, as well as with regard to the = simplified example provided. Posix does not reserve them, the = compiler+linker behaves as expected, but it=E2=80=99s musl=E2=80=99s = dynamic linker implementation that currently does not behave as I would = intuitively expect. > some library names are special, but in conforming environment > this is only observable via the c99 tool so that's where this > is specified, the language does not know anything about libraries. It's observable when an application is run, and posix certainly = describes a multitude of aspects of runtime behavior. The 2017 update = and precursors could have called out such reservations, and they do not. = By that same logic, I could just as well create a =E2=80=98librt=E2=80=99= library using fort77. That=E2=80=99s still in posix, but its man page = only calls out undefined behavior on the =E2=80=98libf.a=E2=80=99 = library. In that case, the runtime behavior would still be incorrect=E2=80= =A6 Those pages really only referred to just those specific tools, = their behavior, and really has no bearing on resolving symbols at = runtime =E2=80=94 it=E2=80=99s entirely musl=E2=80=99s dynamic linker = prerogative. > i believe the reason musl has to special case the names in ld.so > is to be able to load libraries linked against glibc. another > reason is to support looking up symbols in librt etc via dlsym. That is why I proposed a solution of deferring the capture until after = searching, so it will both satisfy that desire to resolve symbols for = when -lm, -lr, -lpthread, etc are specified while also supporting the = general case where such libraries actually do exist and are intended to = be used. =20 >> For background context, I maintain BRL-CAD, a large open source CAD = system that has been in development for over 40 years. >> BRL-CAD=E2=80=99s flagship API with thousands of integrations around = the world is the =E2=80=9Clibrt=E2=80=9D ray tracing library. >=20 > this would not work eg on older glibc if one of your dependencies > used clock_gettime. there is a reason why the c99 tool spec says > that the behaviour is unspecified if you have another librt in the > lookup path, this is a fragile setup. With all due respect, our experience in this regard does not reflect = your statement. We=E2=80=99ve been able to robustly support compilation = and deployment across old versions of glibc without issue or = fragility=E2=80=A6 for decades. If anything, the glibc environments = have always been some of the most robust and reliable resolution = behavior (SGI IRIX and SYSV Solaris on the other hand=E2=80=A6). Most = issues are mitigated by utilizing an isolated installation root (e.g., = /usr/X11R6). > i think if you want to avoid porting work then you should fix this > name collision on your side. Thank you for your advice, but I'd reject that on principle alone. That = advice also seems to be discounting the significant aforementioned = costs. It=E2=80=99d be a great way to lose customers and 3rd party = integrations. Porting is trivial in comparison and has been managed = just fine for a very long time. We=E2=80=99re good. I suspect you=E2=80=99d be similarly disagreeable if you=E2=80=99d used = =E2=80=98nsz@=E2=80=99 for 10+ years and were suggested to =E2=80=98fix=E2= =80=99 it, i.e., change yours just because an external (non-critically = impacting) hosting provider (e.g., Yahoo) felt 4 chars was plenty or = decided to reserve =E2=80=99nsz=E2=80=99 for their own convenience, even = if some notion of a standard were involved. Technically trivial, but = changing would affect friends, family, banking, social media accounts, = and (for sake of comparative argument) 2000 other sites you use = regularly, including many government agencies that require paper-mailing = any changes. It just wouldn=E2=80=99t be worth the effort and you might = even feel like you shouldn=E2=80=99t have to change it. Of course, = ymmv. > of course musl can also be fixed to be more friendly to such usage, > but it's not obvious how to tell if a librt.so is a user library or > part of libc (or glibc in case glibc abi compat is used). I=E2=80=99ll note that the compiler has no problem sorting it out = consistently. In fact it did so, or there would have been missing = symbol errors during library or executable linking. It=E2=80=99s really = the dynamic linker using different logic that is causing this issue. I = suspect it=E2=80=99d work just fine to eliminate the reserved code block = altogether for binaries compiled on a given system (not sure about = cross-compiled). That=E2=80=99s why I suggested the path of "if = there=E2=80=99s a library listed in the ldd table that exists in the = prescribed rpath directory or in the system's linker search path, that = should be used". Then it can fall back or not. ld-musl is already = deafult loaded, so it will typically just be missing symbols that = weren't loaded.=20 Cheers! Sean --Apple-Mail=_397097C9-F54C-4FF2-AFCD-B404D80E2A8B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On = Jul 28, 2022, at 5:18 AM, Szabolcs Nagy <nsz@port70.net> = wrote:

* Christopher Sean Morrison <brlcad@mac.com> = [2022-07-27 19:06:24 -0400]:
First consideration, the code seems to take a position that = those library names are somehow universally reserved and I believe that = to be incorrect.

the way = libraries are looked up is not specified in detail and
the = standard gives the permission to implementations to decide.

This is my understanding as well. =  That=E2=80=99s why I characterized it as incorrect with respect to = the existing implementation identifying them as =E2=80=9Creserved=E2=80=9D= , as well as with regard to the simplified example provided.  Posix = does not reserve them, the compiler+linker behaves as expected, but = it=E2=80=99s musl=E2=80=99s dynamic linker implementation that currently = does not behave as I would intuitively expect.

some library names are special, but in = conforming environment
this is only observable via the c99 = tool so that's where this
is specified, the language does = not know anything about libraries.

It's = observable when an application is run, and posix certainly describes a = multitude of aspects of runtime behavior.  The 2017 update and = precursors could have called out such reservations, and they do not. =  By that same logic, I could just as well create a =E2=80=98librt=E2=80= =99 library using fort77.  That=E2=80=99s still in posix, but its = man page only calls out undefined behavior on the =E2=80=98libf.a=E2=80=99= library.  In that case, the runtime behavior would still be = incorrect=E2=80=A6  Those pages really only referred to just those = specific tools, their behavior, and really has no bearing on resolving = symbols at runtime =E2=80=94 it=E2=80=99s entirely musl=E2=80=99s = dynamic linker prerogative.

i believe the = reason musl has to special case the names in = ld.so
is to be able to load = libraries linked against glibc. another
reason is to = support looking up symbols in librt etc via dlsym.

That is = why I proposed a solution of deferring the capture until after = searching, so it will both satisfy that desire to resolve symbols for = when -lm, -lr, -lpthread, etc are specified while also supporting the = general case where such libraries actually do exist and are intended to = be used.
 
For background context, I maintain BRL-CAD, a large open = source CAD system that has been in development for over 40 years.
BRL-CAD=E2=80=99s flagship API with thousands of integrations = around the world is the =E2=80=9Clibrt=E2=80=9D ray tracing library.

this would not work eg on older = glibc if one of your dependencies
used clock_gettime. = there is a reason why the c99 tool spec says
that the = behaviour is unspecified if you have another librt in the
lookup path, this is a fragile setup.

With = all due respect, our experience in this regard does not reflect your = statement.  We=E2=80=99ve been able to robustly support compilation = and deployment across old versions of glibc without issue or = fragility=E2=80=A6 for decades.  If anything, the glibc = environments have always been some of the most robust and reliable = resolution behavior (SGI IRIX and SYSV Solaris on the other hand=E2=80=A6)= .  Most issues are mitigated by utilizing an isolated installation = root (e.g., /usr/X11R6).

i think if you = want to avoid porting work then you should fix this
name = collision on your side.

Thank you for your advice, but I'd reject that on = principle alone.  That advice also seems to be discounting the = significant aforementioned costs.  It=E2=80=99d be a great way to = lose customers and 3rd party integrations.  Porting is trivial in = comparison and has been managed just fine for a very long time. =  We=E2=80=99re good.

I suspect you=E2=80=99d be similarly = disagreeable if you=E2=80=99d used =E2=80=98nsz@=E2=80=99 for 10+ years = and were suggested to =E2=80=98fix=E2=80=99 it, i.e., change yours = just because an external (non-critically impacting) hosting provider = (e.g., Yahoo) felt 4 chars was plenty or decided to reserve =E2=80=99nsz=E2= =80=99 for their own convenience, even if some notion of a standard were = involved.  Technically trivial, but changing would affect friends, = family, banking, social media accounts, and (for sake of comparative = argument) 2000 other sites you use regularly, including many government = agencies that require paper-mailing any changes.  It just = wouldn=E2=80=99t be worth the effort and you might even feel like you = shouldn=E2=80=99t have to change it.  Of course, = ymmv.

of course musl can also be = fixed to be more friendly to such usage,
but it's not = obvious how to tell if a librt.so is a user library or
part = of libc (or glibc in case glibc abi compat is used).

I=E2=80=99ll note that the compiler has no problem sorting it = out consistently.  In fact it did so, or there would have been = missing symbol errors during library or executable linking.  It=E2=80= =99s really the dynamic linker using different logic that is causing = this issue.  I suspect it=E2=80=99d work just fine to eliminate the = reserved code block altogether for binaries compiled on a given system = (not sure about cross-compiled).  That=E2=80=99s why I suggested = the path of "if there=E2=80=99s a library listed in the ldd table that = exists in the prescribed rpath directory or in the system's linker = search path, that should be used".  Then it can fall back or not. =  ld-musl is already deafult loaded, so it will typically just be = missing symbols that weren't loaded. 

Cheers!
Sean

= --Apple-Mail=_397097C9-F54C-4FF2-AFCD-B404D80E2A8B--