From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.1 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 17852 invoked from network); 12 Jan 2022 14:52:57 -0000 Received: from mother.openwall.net (195.42.179.200) by inbox.vuxu.org with ESMTPUTF8; 12 Jan 2022 14:52:57 -0000 Received: (qmail 9649 invoked by uid 550); 12 Jan 2022 14:52:54 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 9609 invoked from network); 12 Jan 2022 14:52:53 -0000 Date: Wed, 12 Jan 2022 14:52:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=harmenstoppels.nl; s=protonmail; t=1641999161; bh=L1nWj/LrUy07umw1vsoSRs9F7YRset7eigBCF/Ah3ks=; h=Date:To:From:Cc:Reply-To:Subject:Message-ID:In-Reply-To: References:From:To:Cc; b=gl2reDdK3BbFbVwYZUavFYPiKHIQSv2cf1LQ2jgLqgJ9WJxNxPrO6M9mfK4QOH2tK wEViXUCoE+XHxsXYZtxYv8KCauPtdKJzjjtCSo2QFFO7slEAWF2p27ZNt2OcyFHyKm QM0v29etnwE7e1o87DdI2t37Xn4/iSZz8alf1F/g= To: Rich Felker From: Harmen Stoppels Cc: musl@lists.openwall.com Message-ID: In-Reply-To: <20220111222206.GW7074@brightrain.aerifal.cx> References: <20220111175541.GB2046@voyager> <20220111222206.GW7074@brightrain.aerifal.cx> MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pgp-signature"; micalg=pgp-sha256; boundary="------87700670429fbe13b72ec2b38167cc1bf49b49779124cdbf8d936ef20f2fa53f"; charset=utf-8 Subject: Re: [musl] Cannot dlopen() an already loaded shared library by its SONAME name This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------87700670429fbe13b72ec2b38167cc1bf49b49779124cdbf8d936ef20f2fa53f Content-Type: multipart/mixed;boundary=---------------------84457556664a9174320cb140216879b2 -----------------------84457556664a9174320cb140216879b2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain;charset=utf-8 > > > > 1. Julia [1] splits binary dependencies into separate packages, so wh= en > > liba.so depends on libb.so, they live in a different dir, where > > the absolute and relative paths are only known when the julia > > interpreter has started, so neither rpaths or LD_LIBRARY_PATH can > > be used. > > So they dlopen libb.so, and then dlopen liba.so in that > > order, and then assume liba.so does not have to locate libb.so > > again, because its soname is already seens before. > > The proposed workaround was: don't list libb.so in the > > DT_NEEDED of liba.so (that is, if you're already doing the work of > > the linker, you might as well not use the linker at all for locati= ng > > libs). However, being able to run executables shipped with julia > > packages would still be nice (e.g. a subprocess with LD_LIBRARY_PA= TH > > set properly) > > > > 2. The Nix / Guix / Spack people are trying to reduce startup time of > > executables with many shared libraries (as well as fixing library > > paths once and for all to keep executables run deterministically). > > In Guix there's a blog post where they call this the "stat storm" = [2], > > and they solve it in a glibc patch: using context dependent ld.so.= cache, > > that is, a reverse mapping soname =3D> library path. > > In Nix the proposal to fix the "stat storm" is to replace DT_NEEDE= D > > in executables with absolute paths of all required libs (also > > transient ones). This works fine, except on musl, where a dlopen b= y > > soname will still do a search. > > This could be solved much better by making an application-specific > directory full of symlinks to the libraries it uses and putting that > directory as the first thing in the program binary's rpath. So the proposal is basically to replicate an application-specific ld.so.cache in the filesystem? Create a dir per ELF file, put its path in the RPATH, fill the dir with symlinks from soname =3D> library. One problem with this is that $ORIGIN starts behaving differently compared to ld.so.cache. $ORIGIN is now relative to the directory of the symlink, not to the realpath of library. So if a library of a dependent package dlopen's a library in its own prefix by soname, relying on an rpath of say $ORIGIN/plugins, this will fail if the symlink dir is just a flat file list. So it means you'd effectively need to merge the prefixes, and this won't fly in Nix / Spack. Also it won't be a manageable solution for Julia, cause they require immut= ability of each prefix (and if they knew the relative path ahead of time, they wouldn't need this at all). > but having this happen on libraries without any SONAME is really an anti= -feature. What do you mean? I think we're talking only about libraries that have a S= ONAME, do you mean dlopen-by-soname? So, do I understand correctly that loading a lib by path & putting its son= ame in a dict, so that future libs opened by soname can early exit is fine? But loading a lib by soname first, and then by path, and upon opening it happens to have a soname seen before, but is a different file (st_dev/st_i= no), then it should continue with this lib, not early exit? -----------------------84457556664a9174320cb140216879b2 Content-Type: application/pgp-keys; filename="publickey - me@harmenstoppels.nl - 0xFD537C88.asc"; name="publickey - me@harmenstoppels.nl - 0xFD537C88.asc" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="publickey - me@harmenstoppels.nl - 0xFD537C88.asc"; name="publickey - me@harmenstoppels.nl - 0xFD537C88.asc" LS0tLS1CRUdJTiBQR1AgUFVCTElDIEtFWSBCTE9DSy0tLS0tDQpWZXJzaW9uOiBPcGVuUEdQLmpz IHY0LjEwLjEwDQpDb21tZW50OiBodHRwczovL29wZW5wZ3Bqcy5vcmcNCg0KeHNCTkJGblJhMUlC Q0FET3JaRXJVVGNDakphbTNkekcrUzBhdGNkS2FGcDJ0cm9HNXRoS3RaaUt1b0dEDQpUZllpT1VF RER1d3VGWS8xYkF2Zi9YeWYxcVhlNXI2ZFM0Vk1QYlhvaWptNUxaZmNmdTNlV0tzKytnVzMNCnY3 SDQxTlVsNEVramNqVzZ6NURZeUpGQnRqaEVWYzh3dk1jV0ZDcnA1ZjRyeXFtaGk1MlVZMnFvS0Ry VQ0KSmQ2bDc4c256a0diUmw1VzNJVG5FTHdNN1dTV1R5NG44ZWl0ZEQ2Q2xEVk94akR4ZUhBd1dX b3pNb3R6DQpHZ0N3ZWJaaDFXaG1VcjQ1UkM0RC9DSk0yZHBrc3Ezb241cHBIRTR4Vk5FUmkvWk9V dmxDTUZvLzA4WDUNCkRPamZOaHc1OWRkUnJQc2xTbExRTS91bFpNV0FKckcwaDJTQjd3djk4ODFD bnZ1MVdUM28xSTV0QUJFQg0KQUFITksyMWxRR2hoY20xbGJuTjBiM0J3Wld4ekxtNXNJRHh0WlVC b1lYSnRaVzV6ZEc5d2NHVnNjeTV1DQpiRDdDd0g4RUVBRUlBQ2tGQWxuUmExUUdDd2tIQ0FNQ0NS Q05IT0Z2WmZtRWdnUVZDQW9DQXhZQ0FRSVoNCkFRSWJBd0llQVFBS0NSQ05IT0Z2WmZtRWduZTZD QUMzdjBzNUdiSmhYRFJmQWJrV1ZMSUNSb2hFYlZSbw0KMjZBZUErL1hMeUpmYU1lOGFaQUxGUFdp bktVU3B0RjU3YlhCdCtUYnZ0QmZyOHdaTURkTkcycjRTQTd3DQpOcVp0L0R2SzZDbGU3R3JuZmVE WXY2OWpPVCt5cnhoOERzUjNXenpBdllXUVdVTUxEVUQ2OUNIQkgyNWQNCjNsQVViUU5aODBDT1dh UmI4MWtVbVNSY3d6M25vV08vK3Q5L0l1TkxHdG9PbnBqaCthZW1nNUFtWjlYNQ0KdE54UGZxNzhD N3ptQ1c0Ynp1RXZBczc3S09ld2dkT0VaYWs5QldMMWlBSXJLWk5LU3NXN3ZXSTFSMXY1DQptTUZu STY1OFVadW1hRjVJaThLNmxTTStNRDZrVCt4aEFkRDNoTmJGbTd2cDdtZ1pGMHpadmR1c0FMYUoN Ci9Gam9GUHpBa3BkQ3pzQk5CRm5SYTFJQkNBRFhEQ0RmY2FXTndGRnA3K01LNExVWVM5RkxUNG4r akpFdw0KM0o2c2VVY2JDM25MMjJwQ3lXL0xkN1RXOXlsTDlpczgzbExVOGwwbVJvc1lnaG9aS1kx akREUk1Da1kvDQprKyt6S3h1QkpmSHdCZktjZm9nNWZCdTgvaGNSbVlBUUgrZ2ZnK0Z1T3JNeHBV OTQ2bGxBWnVCQnlJeHUNCkM0WnArSVJOOUxuWjFwVE9TRWo2ZlhZc2xXNmxGcmpFbXcxUWwxTFgw TUI5d2lHWGs3aHhSTEJpUzluaQ0KRGtRMFFtU0YyOE82R3V0ZCt6VlJhcmFJeTJQeFovVjQ0cjV6 NVI0NU0wUHlRdVQxVTN1QVd6NitqUEMzDQpPY0FrTlBTL0xDWGR2TXpOa3FBeFJCYnVxdlR3QmJN aE5hNmR0L0V5ZStjQXpYQUs2YkFOeFlqL2s2TWYNCnduSmh1WWVqQUJFQkFBSEN3R2tFR0FFSUFC TUZBbG5SYTFVSkVJMGM0VzlsK1lTQ0Foc01BQW9KRUkwYw0KNFc5bCtZU0NPd0lJQUpQYTdCN25s N1pGY0xiRXVaOU1ZeGNpaCthU0U5Qkt2S3VZaFFYajdkdllEUEFaDQp6MUI4bEJNRUZsNy9PS3hM OUpzQ0QwcngvZWFRaldBTUhFQTFJUlhDSzRrYUppVk54SU9vQVRzeVpLVXINCnE4S2s5Z3pwZWtM WXl5T08zUlRYUkxYUW1EMHp1YjRsS01GdmpvRzBTbGtMNUFubVBtYXZiVXhqcnorMg0KUUhwZFB1 QTZ6VndXME5yMXZ2V1RVdmtpYkhUUktwUTlzdEtsV3JTOWpZaTU0MDVJaXR2c2N0MEQ5Y2ZwDQpp ZVdJckk0aHd3NjZQQmJ2OE1MQWNKZi9pamVPWURQY0FxYzAzMTZOVkhBSTdpYm5MVWNnYUhFNHAx VU4NCmZEVzEzd29vRWR1bktkbkE5c1RuTHhtZmwwWnJJc0FHenRTa0RDT3NlWDg1ZHViVHBvc1M2 MFk9DQo9T0x1ZQ0KLS0tLS1FTkQgUEdQIFBVQkxJQyBLRVkgQkxPQ0stLS0tLQ0K -----------------------84457556664a9174320cb140216879b2-- --------87700670429fbe13b72ec2b38167cc1bf49b49779124cdbf8d936ef20f2fa53f Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: ProtonMail wsBzBAEBCAAGBQJh3usnACEJEI0c4W9l+YSCFiEE/VN8iJjfG7NRPUUojRzh b2X5hIKLnQgAtjIuU5CPWeRCuc3n0idlLxKA79Hgy4Ff47I3zkClHuUcqr8U DTEmtfpnJuXLMJh+MmQ/wpfyHr/uNgcQbpsKcQy7nV0dBotUqUNtgBC372Ga /NFx+d8dGHOvLckXZtbLdWuS47xkUu+r+qrO5uGxneJlPRW1WUy5H9dH5JxN OW1/ApqSQJHvtBWWcxRFuosee2nH1NG4uCI4fsk4VcNpJ1WvMVhTUWv6uwAT wQygHsvK+jkI9dgHaPymc6/gsBbf1LKWtpNewBaYj2M8ZcGIyMAcD9yG6PPa vzH8TaHtTuojxYHtNdi006UHJyJYF440BYOFHys1FsiUcyxGT51HVg== =IDw7 -----END PGP SIGNATURE----- --------87700670429fbe13b72ec2b38167cc1bf49b49779124cdbf8d936ef20f2fa53f--