Github messages for voidlinux
 help / color / mirror / Atom feed
* [ISSUE] tracking issue: deprecating -lib32 multilib system
@ 2020-12-22  0:50 q66
  2020-12-22  1:17 ` [RFC/tracking issue] " ahesford
                   ` (29 more replies)
  0 siblings, 30 replies; 31+ messages in thread
From: q66 @ 2020-12-22  0:50 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 10235 bytes --]

New issue by q66 on void-packages repository

https://github.com/void-linux/void-packages/issues/27337

Description:
So, right now we have a system in place that works like this:

- When building `i686` packages, and `i686` specifically and only, libraries are taken and special `-32bit` packages are also generated for `x86_64` in the special `multilib/` repository, whether it's just libraries or full package is then controlled via the mode and so on
- In order to ensure that these libraries can be installed together with native `x86_64` ones and not conflict, `i686` packages are usually configured to use `/usr/lib32` and all other packages are configured to use `/usr/lib`; through hooks, we make sure everything ends up in `/usr/lib` in the end, except the `-lib32` packages which are populated into `lib32` (this is also needed in case programs, e.g. mesa, dynamically load library plugins from somewhere else)
- This is pretty clunky and hacky, and only works for `i686` on glibc, and only works with `x86_64` on glibc
- It is also potentially problematic when migrating off buildbot, as it means the `x86_64` and `i686` targets cannot be separated
- But mainly it's a problem for other architectures and musl
- It also used to be a problem that the `lib32`/`lib64` symlinks did not exist in unconfigured chroots, so toolchains had to be configured to always use dynamic linkers from `lib` and that was terrible and ABI breaking, but has since been mitigated and will be fully put into place with gcc10 upgrade; we could fix this because `/usr/lib64` is now a symlink to `/usr/lib` on 64-bit systems, and `/usr/lib32` is a symlink to `/usr/lib` on 32-bit systems, and these symlinks are a part of `base-files`, always

I'd like to finally deal with this; some initial steps have already been taken care of:

- 00da0a22326293560a0a043b4b583ca2dcec06fa (fixes dynamic linkers and makes toolchains default to `lib32` and `lib64` as needed)
- 6283b0d2098746cf5e111fd8144da6dcc9a22954 (allows build systems to use `lib64` in addition to `lib32` while ensuring that in the end all packages end up with `lib`)

# Proposed new system

Therefore, I am proposing a new system. The goals of this system should be:

- It should not need any special handling in xbps-src
- It should not need any special treatment by xbps itself
- It should not need generation of any special multilib repos/packages
- It should work on any architecture, any libc with a 32-bit counterpart

There are distributions, mainly Debian, that solve this by having a complete multiarch system in place. That involves having support for handling multiple architectures at once in their package manager, and having everything built in a way that `/usr/lib` is never used directly, but rather libraries use a special prefix that contains the target triplet, so there are never conflicts. While I considered implementing such system in Void, it quickly turned out to be too large of a can of worms, without that much benefit, which would also require a considerable amount of effort put everywhere. So, instead I am proposing what I call *multilib prefixes*, which is a simplified, generalized system that only handles the common case of running 32-bit processes on 64-bit systems (or alternatively, 64-bit processes on systems running 64-bit kernel with 32-bit userland).

# How do multilib prefixes work?

Before explaining how this is going to work, I'm going to describe the steps needed to get there, it will make it easier.

- First, we should fix our toolchains to use proper dynamic linker paths again (**done**)
- Then, we should update the toolchains to use `lib32` on 32-bit systems and `lib64` on 64-bit systems, unconditionally (**done**; this is actually not truly important, as the real libdir is handled separately by build systems, but still, for correctness)
- Then, we should fix `xbps-src` to allow `lib64` paths in addition to `lib32`, this is **done**, by fixing the `xbps-src` hooks
- Then, we should fix `glibc` and other related base bits (crosstoolchains and so on) to use `lib32` and `lib64` instead of `lib`
- Similarly, all `build-style` need to be fixed to follow this (use `lib${XBPS_TARGET_WORDSIZE}`), and all templates that explicitly set `lib32` on `i686` need to be generalized (e.g. `mesa`, `pulseaudio` and so on; these generally load plugins at runtime, so the path is important there; for most programs, it's not, since library paths are not written in executables/shared libs, only names are, and the lookup paths are controlled by dynamic linker)
- We will also need to find every program/library that does dynamic loading like above and the search path is hardcoded and meaningful there; these templates need to have their revision bumped to rebuild, probably after `gcc` is updated to 10 and so on
- For `musl`, we will need to ship custom `/etc/ld-musl-ARCHITECTURE.path` which will override the default dynlinker search paths to use the appropriate `lib32` or `lib64` ones - `musl` by default always uses `lib`
- For `glibc`, this can be (and already is, though a bit by accident) handled by `ld.so.conf.d`
- Then we will need to introduce handling for the prefixes themselves. More about that later.

Now, to get to how it works, and also why it works.

For one, the `/usr/lib` path we currently use will be preserved. This is because `xbps-src` is already updated to intercept anything installing into `lib32` or `lib64` and properly migrating it to `lib` using symlinks. This is important.

On 64-bit systems, `base-files` contains symlinks so that e.g. `/usr/lib64` points to `/usr/lib`. Similar thing happens for `lib32` on 32-bit systems. However, since the build systems and so on will automatically default to `lib32` or `lib64`, all library lookups will happen through those symlinks, and not through the actual path. What does this mean?

In order to deal with the case of 32-bit binaries on 64-bit systems, all we have to do is simply create a new Void "root". Let's say this root will be `/usr/i686-linux-gnu`. This root will contain all the usual files. So your 32-bit libraries will be located in `/usr/i686-linux-gnu/usr/lib`.

How to install things into this prefix? That is easy; you can just use `xbps-install`. For example, `XBPS_ARCH=i686 xbps-install -r /usr/i686-linux-gnu ...`. That means you will be able to make use of the entire standard 32-bit repo without any special handling. This is in fact how `xbps-src` already manages `makedepends` when cross-compiling.

What is needed now is to hook this prefix into your system. Since 64-bit systems will use `lib64` and 32-bit ones will use `lib32` for lookups, they are non-conflicting. Therefore, what you can do is make your `/usr/lib32` a symlink to `/usr/i686-linux-gnu/usr/lib`. And that's it; if you run any 32-bit process in your 64-bit system, it will look up its libraries through `/usr/lib32`, which will point to your custom prefix, containing all the dependencies.

If you think about it, that's not very different from how the current 32bit libs are handled. It's just generalized, and does not use any special tooling.

Of course, this is actually not going to work as is, out of box. There is also the dynamic linker to take care of. This dynamic linker does not always respect `lib32` paths or whatever, so it's going to be slightly more involved. Notably on `musl`, dynamic linker always comes from `/lib`. E.g. `/lib/ld-musl-i686.so.1`.

However, the dynamic linkers are the one thing where we are sure there will never be file conflicts. Therefore, you can just symlink the dynamic linker from your multilib prefix to the path where it's expected to be, and everything will just work.

And that is basically the gist of it; there is nothing much more complicated about it conceptually. Of course, there are issues we need to work out, because in practice it's not as smooth.

# Problems

- If we choose `/usr/<target-triplet>` as the default multilib prefix, it will conflict with cross-toolchains. What we could do is integrate this with managing of cross-toolchains on your host system and make them a lot more useful. However, cross-toolchains also provide the libc and other packages that would otherwise be installed. In `xbps-src`, `cross-vpkg-dummy` is installed.
- We will need tooling for managing these multilib prefixes. Using just `xbps` as it is is fairly clunky, because by default these prefixes don't have repos set up in them and so on. Especially initial setup is relatively involved. Another problem is making sure the symlinks are properly set up, for `lib32` (or `lib64`) as well as the dynamic linker symlinks if necessary. The hypothetical management tool could deal with this automatically, besides providing functionality for installing the packages into the prefix.
- Right now, `/usr/lib32` is actually not a symlink on 64-bit systems, but a directory. This is because `/usr/lib32/locale` is a symlink, to your native locale data. We would need to replace that.
- There might be packages that require wordsize-specific data files in `/usr/share`. These need to be found, and such files need to go to `/usr/lib(32|64)`. This is the right thing to do either way, since they're no different from binary files, `/usr/share` should only contain architecture-independent data.
- There might be templates where it would be useful to split the libraries manually into a subpackage (`foo-libs`, for example). This is not a hard requirement, but since multilib prefixes should generally only contain libraries and data files are useless, it might reduce how much disk space we waste.
- There are some "full" 32-bit packages which also contain executable programs. Those would need to be handled specially. Perhaps the hypothetical "prefix management tool" could also take care of installing appropriate symlinks in `/usr/bin`, probably upon user request.

There might be more issues to work out, these are just ones I can think of right now. I think all of them are solvable, so we should be able to make a concept like this into reality sometime in near future, and deprecate/remove the current multilib repos.

@void-linux/pkg-committers 

^ permalink raw reply	[flat|nested] 31+ messages in thread
end of thread, other threads:[~2022-05-01  5:45 UTC | newest]

Thread overview: 31+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-22  0:50 [ISSUE] tracking issue: deprecating -lib32 multilib system q66
2020-12-22  1:17 ` [RFC/tracking issue] " ahesford
2020-12-22  1:18 ` q66
2020-12-22  1:22 ` Skirmisher
2020-12-22  1:58 ` q66
2020-12-22  2:00 ` q66
2020-12-22  2:05 ` ericonr
2020-12-22  4:31 ` ericonr
2020-12-22  4:31 ` ericonr
2020-12-22  5:11 ` ericonr
2020-12-22  5:26 ` ericonr
2020-12-22 20:59 ` q66
2020-12-22 20:59 ` q66
2020-12-22 20:59 ` q66
2020-12-22 20:59 ` q66
2020-12-22 20:59 ` q66
2020-12-22 20:59 ` q66
2020-12-22 21:22 ` q66
2020-12-22 21:34 ` q66
2020-12-22 21:58 ` q66
2020-12-22 22:16 ` q66
2020-12-23  0:26 ` q66
2020-12-23  0:26 ` q66
2020-12-23  0:27 ` q66
2020-12-23  0:39 ` q66
2020-12-27 20:07 ` ericonr
2020-12-27 20:26 ` ericonr
2020-12-28  1:35 ` ericonr
2021-01-07 20:31 ` ahesford
2022-05-01  2:14 ` github-actions
2022-05-01  5:45 ` the-maldridge

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).