Adapting binaries easily to musl, or database with binaries (musl)

Adapting binaries easily to musl, or database with binaries (musl)

[Brian Peregrine]

I'm looking for an easy way on how people can either:
* adapt their downloaded binaries to be able to run on musl
* or simply download prepared musl binaries from a database

I doubt that second option exists, as such a database would need to
cover a large amounts of programs, for various cpu classes, and be
often updated with new versions of those programs.

I'm particularly looking for the most recent (stable, musl) version of
chromium browser and/or firefox, I'm running gentoo linux (musl), on a
32-bit (i686) machine.

For firefox, glibc binaries are available from
https://www.mozilla.org/en-US/firefox/linux/
For chromium I couldn't really find a glibc binary;
there's https://www.chromium.org/getting-involved/download-chromium
but that redirects to an unstable version
(https://download-chromium.appspot.com ) and to
http://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html
(but those are old binaries).

Anyway, can someone describe how the binaries can be easily altered to
run on a musl distro , and tell me a download link for chromium for
the latest stable release ?
With the binaries, I want to avoid to need to compile the gentoo
package (this can be problematic as live distro's are not updated that
often).

Brian

Re: Adapting binaries easily to musl, or database with binaries (musl)

[Rich Felker]

On Fri, Jun 07, 2019 at 01:53:50PM +0000, Brian Peregrine wrote:
> I'm looking for an easy way on how people can either:
> * adapt their downloaded binaries to be able to run on musl
> * or simply download prepared musl binaries from a database
> 
> I doubt that second option exists, as such a database would need to
> cover a large amounts of programs, for various cpu classes, and be
> often updated with new versions of those programs.

The best way is to use binaries packaged by your distribution rather
than by the software authors. This is true whether you're using a
musl-based or glibc-based distribution. There's usually a better chain
of trust/responsibility, and you know they're built to work with the
library ecosystem you'll have rather than one that's just
"sufficiently similar" (often not).

With that said, there's also an *intent* that projects that want to
ship binaries should be able to ship *static-linked* musl binaries
that run on any system, glibc or musl or something else, possibly not
even Linux, as long as it supports the Linux syscall ABI. However,
this is currently difficult for graphics applications due to the
current OpenGL architecture of dlopen'ing a userspace driver (linked
against a particular libc/library ecosystem) into your program.
Solving this problem is a major long-term goal.

> I'm particularly looking for the most recent (stable, musl) version of
> chromium browser and/or firefox, I'm running gentoo linux (musl), on a
> 32-bit (i686) machine.

Did you roll your own "distro" from scratch, or are you using one of
the musl-based distros?

> For firefox, glibc binaries are available from
> https://www.mozilla.org/en-US/firefox/linux/
> For chromium I couldn't really find a glibc binary;
> there's https://www.chromium.org/getting-involved/download-chromium
> but that redirects to an unstable version
> (https://download-chromium.appspot.com ) and to
> http://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html
> (but those are old binaries).
> 
> Anyway, can someone describe how the binaries can be easily altered to
> run on a musl distro , and tell me a download link for chromium for
> the latest stable release ?
> With the binaries, I want to avoid to need to compile the gentoo
> package (this can be problematic as live distro's are not updated that
> often).

Isn't building from source kinda the point of Gentoo? I can understand
not wanting to do that for everything, but that's the reason I prefer
other distros. If Bedrock Linux is still around, it might make it easy
to use packages from other musl-based distros like Alpine, Void, or
Adelie, but I think they'd use their own library ecosystems rather
than the main host one, which would be less efficient and kinda defeat
the purpose. I may be wrong about that, though. There should also be
ways to manually extract/integrate their packages into your Gentoo
host.

Rich

Re: Adapting binaries easily to musl, or database with binaries (musl)

[Brian Peregrine]

Hey Rich,

Replied in-line:
Did you roll your own "distro" from scratch, or are you using one of
the musl-based distros?
We rolled our own (and by "we", I mean VCT Labs did). The distro is
called "TAZ" and we'll release the first beta within the next months
(we're currently still in alpha-stage, no official release yet).

For the (glibc) binaries: yes using binaries posted on the website of
the distribution (in our case, Gentoo) is best, but as far as I know,
Gentoo doesn't release binaries, only ebuilds. It refers to binhosts
for binaries but there is no list of those, and even if there was, I
wouldn't trust them. For firefox, not a problem as there are binaries
from the official website, for chromium it does seem to be a bit of an
issue (but I might be wrong here).

Most Gentoo users indeed prefer building from source, but we are
compiling it for specific machines (e.g. i686) so we are taking
advantage of the compilation-features gentoo has to offer.

I looked at Bedrock Linux, Alpine, Void, Adelie, ... but I don't see
them having binaries downloads either. Only package downloads are
avilable at Alpine and Void (see https://pkgs.alpinelinux.org/packages
, https://voidlinux.org/packages/?arch=x86_64&q=firefox ), Adelie and
Bedrock don't offer any downloads from the website at all.

Let me know how the glibc binaries can be directly edited (i.e. using
an IDE editor as Geany) to work on musl, and if someone finds the
download URL for the latest stable chromium binary, that would also be
very welcome.


On 6/7/19, Rich Felker <dalias@libc.org> wrote:
> On Fri, Jun 07, 2019 at 01:53:50PM +0000, Brian Peregrine wrote:
>> I'm looking for an easy way on how people can either:
>> * adapt their downloaded binaries to be able to run on musl
>> * or simply download prepared musl binaries from a database
>>
>> I doubt that second option exists, as such a database would need to
>> cover a large amounts of programs, for various cpu classes, and be
>> often updated with new versions of those programs.
>
> The best way is to use binaries packaged by your distribution rather
> than by the software authors. This is true whether you're using a
> musl-based or glibc-based distribution. There's usually a better chain
> of trust/responsibility, and you know they're built to work with the
> library ecosystem you'll have rather than one that's just
> "sufficiently similar" (often not).
>
> With that said, there's also an *intent* that projects that want to
> ship binaries should be able to ship *static-linked* musl binaries
> that run on any system, glibc or musl or something else, possibly not
> even Linux, as long as it supports the Linux syscall ABI. However,
> this is currently difficult for graphics applications due to the
> current OpenGL architecture of dlopen'ing a userspace driver (linked
> against a particular libc/library ecosystem) into your program.
> Solving this problem is a major long-term goal.
>
>> I'm particularly looking for the most recent (stable, musl) version of
>> chromium browser and/or firefox, I'm running gentoo linux (musl), on a
>> 32-bit (i686) machine.
>
> Did you roll your own "distro" from scratch, or are you using one of
> the musl-based distros?
>
>> For firefox, glibc binaries are available from
>> https://www.mozilla.org/en-US/firefox/linux/
>> For chromium I couldn't really find a glibc binary;
>> there's https://www.chromium.org/getting-involved/download-chromium
>> but that redirects to an unstable version
>> (https://download-chromium.appspot.com ) and to
>> http://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html
>> (but those are old binaries).
>>
>> Anyway, can someone describe how the binaries can be easily altered to
>> run on a musl distro , and tell me a download link for chromium for
>> the latest stable release ?
>> With the binaries, I want to avoid to need to compile the gentoo
>> package (this can be problematic as live distro's are not updated that
>> often).
>
> Isn't building from source kinda the point of Gentoo? I can understand
> not wanting to do that for everything, but that's the reason I prefer
> other distros. If Bedrock Linux is still around, it might make it easy
> to use packages from other musl-based distros like Alpine, Void, or
> Adelie, but I think they'd use their own library ecosystems rather
> than the main host one, which would be less efficient and kinda defeat
> the purpose. I may be wrong about that, though. There should also be
> ways to manually extract/integrate their packages into your Gentoo
> host.
>
> Rich
>

Re: Adapting binaries easily to musl, or database with binaries (musl)

[A. Wilcox]

[-- Attachment #1.1: Type: text/plain, Size: 1566 bytes --]

On 06/07/19 11:02, Brian Peregrine wrote:
> Most Gentoo users indeed prefer building from source, but we are
> compiling it for specific machines (e.g. i686) so we are taking
> advantage of the compilation-features gentoo has to offer.


Adélie is built for 586+MMX, which is very close to 686.


> I looked at Bedrock Linux, Alpine, Void, Adelie, ... but I don't see
> them having binaries downloads either. Only package downloads are
> avilable at Alpine and Void (see https://pkgs.alpinelinux.org/packages
> , https://voidlinux.org/packages/?arch=x86_64&q=firefox ), Adelie and
> Bedrock don't offer any downloads from the website at all.


We don't have a package site yet, but
https://distfiles.adelielinux.org/adelie/1.0-beta3/user/pmmx/ is going
to have every package for the 586.

Our Firefox package is
https://distfiles.adelielinux.org/adelie/1.0-beta3/user/pmmx/firefox-esr-52.9.0-r4.apk
for instance.


> Let me know how the glibc binaries can be directly edited (i.e. using
> an IDE editor as Geany) to work on musl, and if someone finds the
> download URL for the latest stable chromium binary, that would also be
> very welcome.


1) It's not possible to "edit" a glibc binary to use musl like that,
unfortunately.

2) Chromium does not work on musl.  There are (bad quality) patches
floating around, but they're beyond worthless.  Expect random crashes
and chasing upstream constantly.  Don't even bother.


Best,
--arw

-- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Adapting binaries easily to musl, or database with binaries (musl)

[Alexis Boulva]

[-- Attachment #1: Type: text/plain, Size: 2828 bytes --]

Hi Brian,

> On Jun 7, 2019, at 14:07, A. Wilcox <awilfox@adelielinux.org> wrote:
> 
>> On 06/07/19 11:02, Brian Peregrine wrote:
>> Most Gentoo users indeed prefer building from source, but we are
>> compiling it for specific machines (e.g. i686) so we are taking
>> advantage of the compilation-features gentoo has to offer.

In case you're in need of a cross-compiling method for Firefox on gentoo i686 musl, once I've finished building/testing the result, I'll be uploading some experimental build environment configuration files/patches for gentoo x86_64 musl to GitHub, which I use to cross-compile from glibc to musl (why I began with a glibc chroot is a long story) and can easily be ported to other architectures (x86_64 glibc => aarch64 musl, for example). I used to have it working with older software versions and am just updating the work to current versions.

Another thing: some work on cross-compiling from x86_64 musl to other architectures, if that's more what you would prefer, has also been done by gentoo devs ( hint: https://github.com/smaeul/portage-overlay/tree/master/dev-lang/rust ), yet isn't in their musl or main ebuild repos yet. My recommendation would be to ask in #gentoo-hardened on Freenode IRC if you can, and see what your best approach would be after discussion and code review.

If you can't go on IRC, I'll let you know when I'm finished with testing here.

Regards,
Alexis

> 
> Adélie is built for 586+MMX, which is very close to 686.
> 
>> I looked at Bedrock Linux, Alpine, Void, Adelie, ... but I don't see
>> them having binaries downloads either. Only package downloads are
>> avilable at Alpine and Void (see https://pkgs.alpinelinux.org/packages
>> , https://voidlinux.org/packages/?arch=x86_64&q=firefox ), Adelie and
>> Bedrock don't offer any downloads from the website at all.
> 
> 
> We don't have a package site yet, but
> https://distfiles.adelielinux.org/adelie/1.0-beta3/user/pmmx/ is going
> to have every package for the 586.
> 
> Our Firefox package is
> https://distfiles.adelielinux.org/adelie/1.0-beta3/user/pmmx/firefox-esr-52.9.0-r4.apk
> for instance.
> 
> 
>> Let me know how the glibc binaries can be directly edited (i.e. using
>> an IDE editor as Geany) to work on musl, and if someone finds the
>> download URL for the latest stable chromium binary, that would also be
>> very welcome.
> 
> 
> 1) It's not possible to "edit" a glibc binary to use musl like that,
> unfortunately.
> 
> 2) Chromium does not work on musl.  There are (bad quality) patches
> floating around, but they're beyond worthless.  Expect random crashes
> and chasing upstream constantly.  Don't even bother.
> 
> 
> Best,
> --arw
> 
> -- 
> A. Wilcox (awilfox)
> Project Lead, Adélie Linux
> https://www.adelielinux.org
> 

[-- Attachment #2: Type: text/html, Size: 5469 bytes --]

Re: Adapting binaries easily to musl, or database with binaries (musl)

[Brian Peregrine]

Hey Alex, awilfox,

I looked at https://distfiles.adelielinux.org/adelie/1.0-beta3/user/pmmx/
but those are apk packages. I actually want something that avoids
users needing to compile non-standard programs themselves and have
these technical (compiling) skills to begin with.

I'm not familiar with apk packages; can these simply be unzipped (ie
with xarchiver) and can the unzipped files then run directly in gentoo
(same as with a binary) ? If so, then it would be suitable
(installation with APK Tools wouldn't be suitable as we have ). I
don't care too much with these on whether it's for i586 and not i686
as long as it runs and people can download the file quickly and run
without too much fuss (the extraction step doesn't add that much time
compared to compiling it, ...), mainly as it's just a 2nd browser that
would not be used often, except in special cases.

For TAZ, VCT Labs compiles the standard gentoo packages and makes sure
it runs under musl, in TAZ. So technically, there's not reason why we
can't do this for firefox, and simply include that program (firefox)
in our distro. Why haven't we done, you might ask. Reason is that we
already have a browser (Palemoon) and this has been made to run under
musl. However, for only certain websites (github for instance) it
doesn't work, even when using (recent firefox or chrome user agent
strings). I don't want to add another browser in our distro since that
would be (most of the time) redundant and I want TAZ users to use
Palemoon (as that focuses on security and simplicity more).

We also have git-scm but that doesn't allow to do all functions github
has (for instance, changing the profile photo, modifying git issues,
...) and I think that it's also annoying to need to use git-scm when
you need to change just a few lines on your repo (which in the
browser, would take far less time to do then with git-scm).