Re: Adapting binaries easily to musl, or database with binaries (musl)

Re: Adapting binaries easily to musl, or database with binaries (musl)

[Brian Peregrine]

Hey Alex, awilfox,

I looked at https://distfiles.adelielinux.org/adelie/1.0-beta3/user/pmmx/
but those are apk packages. I actually want something that avoids
users needing to compile non-standard programs themselves and have
these technical (compiling) skills to begin with.

I'm not familiar with apk packages; can these simply be unzipped (ie
with xarchiver) and can the unzipped files then run directly in gentoo
(same as with a binary) ? If so, then it would be suitable
(installation with APK Tools wouldn't be suitable as we have ). I
don't care too much with these on whether it's for i586 and not i686
as long as it runs and people can download the file quickly and run
without too much fuss (the extraction step doesn't add that much time
compared to compiling it, ...), mainly as it's just a 2nd browser that
would not be used often, except in special cases.

For TAZ, VCT Labs compiles the standard gentoo packages and makes sure
it runs under musl, in TAZ. So technically, there's not reason why we
can't do this for firefox, and simply include that program (firefox)
in our distro. Why haven't we done, you might ask. Reason is that we
already have a browser (Palemoon) and this has been made to run under
musl. However, for only certain websites (github for instance) it
doesn't work, even when using (recent firefox or chrome user agent
strings). I don't want to add another browser in our distro since that
would be (most of the time) redundant and I want TAZ users to use
Palemoon (as that focuses on security and simplicity more).

We also have git-scm but that doesn't allow to do all functions github
has (for instance, changing the profile photo, modifying git issues,
...) and I think that it's also annoying to need to use git-scm when
you need to change just a few lines on your repo (which in the
browser, would take far less time to do then with git-scm).

Re: Re: Adapting binaries easily to musl, or database with binaries (musl)

[Szabolcs Nagy]

* Brian Peregrine <peregrinebrian@gmail.com> [2019-06-08 09:33:32 +0000]:
> I looked at https://distfiles.adelielinux.org/adelie/1.0-beta3/user/pmmx/
> but those are apk packages. I actually want something that avoids
> users needing to compile non-standard programs themselves and have
> these technical (compiling) skills to begin with.

you can bundle the apks into a self extracting script.

> 
> I'm not familiar with apk packages; can these simply be unzipped (ie
> with xarchiver) and can the unzipped files then run directly in gentoo
> (same as with a binary) ? If so, then it would be suitable
> (installation with APK Tools wouldn't be suitable as we have ). I

$ file foo.apk
foo.apk: gzip compressed data
$ gunzip <foo.apk |file -
/dev/stdin: POSIX tar archive

it's a tar.gz with the files to install and some metadata.

a packaging tool can manage dependencies, so you can
install/uninstall/update components easily, if you do
it with a third party script, it will be unclear what
deps come from the system and what deps from the script
and how to manage security updates for the latter.
and if the abi of a system component changes after an
update then the third party app may stop working.

extracting is likely not enough, you want to put the
binaries under a separate directory and set up paths
for that so lib/exe/etc is found by the app.

> don't care too much with these on whether it's for i586 and not i686
> as long as it runs and people can download the file quickly and run
> without too much fuss (the extraction step doesn't add that much time
> compared to compiling it, ...), mainly as it's just a 2nd browser that
> would not be used often, except in special cases.
> 
> For TAZ, VCT Labs compiles the standard gentoo packages and makes sure
> it runs under musl, in TAZ. So technically, there's not reason why we
> can't do this for firefox, and simply include that program (firefox)
> in our distro. Why haven't we done, you might ask. Reason is that we
> already have a browser (Palemoon) and this has been made to run under
> musl. However, for only certain websites (github for instance) it
> doesn't work, even when using (recent firefox or chrome user agent
> strings). I don't want to add another browser in our distro since that
> would be (most of the time) redundant and I want TAZ users to use
> Palemoon (as that focuses on security and simplicity more).
> 
> We also have git-scm but that doesn't allow to do all functions github
> has (for instance, changing the profile photo, modifying git issues,
> ...) and I think that it's also annoying to need to use git-scm when
> you need to change just a few lines on your repo (which in the
> browser, would take far less time to do then with git-scm).

Re: Re: Adapting binaries easily to musl, or database with binaries (musl)

[Brian Peregrine]

I've been trying to get firefox musl (from
https://distfiles.adelielinux.org/adelie/1.0-beta3/user/pmmx/ )
started (haven't gotten much further then a XPCOMGlueLoad error so
far). Regardless of this, I did notice that the firefox version is
52.9.0-r4 whereas we're currently at version 67, so not sure whether
that's recent enough for github.

I'm also thinking whether this is really the best approach or not.

One alternative could be to just download the ebuild from portage, and
then compile it for musl i686 to a usb stick (main reason for putting
the files to a usb stick is that TAZ is a livecd so you can only store
files for a limited duration -any files not stored on removable media
are erased at reboot). After saving the files to the usb stick, the
user would start the program from the usb stick (so not copying the
files manually to the system; reason is that it's easier to not copy
the files every time and the system itself is read-only, no copying to
the system is permitted)

I don't know however whether
* ebuilds from portage can be compiled under musl at all
* whether the files saved this way can actually be started from usb
stick (I doubt the files thus obtained by this method qualify as a
"binary")
* whether compiling programs using portage is leaner (less cpu/ram
intensive) then compiling the programs from (github) source (assuming
there is a difference at all). This is important as since it's a live
cd, it is already ram-intensive and we also focus on low-spec machines
(starting from i686)

Another possible way is to use anbox on gentoo and run the android
versions of the programs (which can be found easily at the app store)
directly (so not through installation with APK Tools). I don't know
however whether this would run at all under gentoo musl and it's
probably also overkill (as anbox, adb, ... need to be installed and
the android programs might also not be stable even if they do run)


On 6/8/19, Szabolcs Nagy <nsz@port70.net> wrote:
> * Brian Peregrine <peregrinebrian@gmail.com> [2019-06-08 09:33:32 +0000]:
>> I looked at https://distfiles.adelielinux.org/adelie/1.0-beta3/user/pmmx/
>> but those are apk packages. I actually want something that avoids
>> users needing to compile non-standard programs themselves and have
>> these technical (compiling) skills to begin with.
>
> you can bundle the apks into a self extracting script.
>
>>
>> I'm not familiar with apk packages; can these simply be unzipped (ie
>> with xarchiver) and can the unzipped files then run directly in gentoo
>> (same as with a binary) ? If so, then it would be suitable
>> (installation with APK Tools wouldn't be suitable as we have ). I
>
> $ file foo.apk
> foo.apk: gzip compressed data
> $ gunzip <foo.apk |file -
> /dev/stdin: POSIX tar archive
>
> it's a tar.gz with the files to install and some metadata.
>
> a packaging tool can manage dependencies, so you can
> install/uninstall/update components easily, if you do
> it with a third party script, it will be unclear what
> deps come from the system and what deps from the script
> and how to manage security updates for the latter.
> and if the abi of a system component changes after an
> update then the third party app may stop working.
>
> extracting is likely not enough, you want to put the
> binaries under a separate directory and set up paths
> for that so lib/exe/etc is found by the app.
>
>> don't care too much with these on whether it's for i586 and not i686
>> as long as it runs and people can download the file quickly and run
>> without too much fuss (the extraction step doesn't add that much time
>> compared to compiling it, ...), mainly as it's just a 2nd browser that
>> would not be used often, except in special cases.
>>
>> For TAZ, VCT Labs compiles the standard gentoo packages and makes sure
>> it runs under musl, in TAZ. So technically, there's not reason why we
>> can't do this for firefox, and simply include that program (firefox)
>> in our distro. Why haven't we done, you might ask. Reason is that we
>> already have a browser (Palemoon) and this has been made to run under
>> musl. However, for only certain websites (github for instance) it
>> doesn't work, even when using (recent firefox or chrome user agent
>> strings). I don't want to add another browser in our distro since that
>> would be (most of the time) redundant and I want TAZ users to use
>> Palemoon (as that focuses on security and simplicity more).
>>
>> We also have git-scm but that doesn't allow to do all functions github
>> has (for instance, changing the profile photo, modifying git issues,
>> ...) and I think that it's also annoying to need to use git-scm when
>> you need to change just a few lines on your repo (which in the
>> browser, would take far less time to do then with git-scm).
>

Re: Re: Adapting binaries easily to musl, or database with binaries (musl)

[A. Wilcox]

[-- Attachment #1.1: Type: text/plain, Size: 4085 bytes --]

On 06/09/19 09:10, Brian Peregrine wrote:
> I've been trying to get firefox musl (from
> https://distfiles.adelielinux.org/adelie/1.0-beta3/user/pmmx/ )
> started (haven't gotten much further then a XPCOMGlueLoad error so
> far). Regardless of this, I did notice that the firefox version is
> 52.9.0-r4 whereas we're currently at version 67, so not sure whether
> that's recent enough for github.


It's still recent enough for GitHub so far.  We won't be bumping to 60
due to CPU incompatibilities ( https://bugzilla.mozilla.org/1499121 ).

We hope to have 68 available soon after it is released.


> I'm also thinking whether this is really the best approach or not.
> 
> One alternative could be to just download the ebuild from portage, and
> then compile it for musl i686 to a usb stick (main reason for putting
> the files to a usb stick is that TAZ is a livecd so you can only store
> files for a limited duration -any files not stored on removable media
> are erased at reboot). After saving the files to the usb stick, the
> user would start the program from the usb stick (so not copying the
> files manually to the system; reason is that it's easier to not copy
> the files every time and the system itself is read-only, no copying to
> the system is permitted)
> 
> I don't know however whether
> * ebuilds from portage can be compiled under musl at all


Adélie started life four years ago as a Gentoo fork with binpkgs.  There
are a few overlays:

https://gitweb.gentoo.org/proj/musl.git/

https://github.com/smaeul/portage-overlay/

The second one actually has a pretty good Chromium patchset, much higher
quality than the others I've seen.  (I still wouldn't advise running
Chromium on low-end hardware, or anywhere that security or privacy
matter, but at least it won't wreck itself over musl with these patches.)


> * whether the files saved this way can actually be started from usb
> stick (I doubt the files thus obtained by this method qualify as a
> "binary")


As long as the compilation flags are generic enough, it is exactly how
binaries are generated in any other project.


> * whether compiling programs using portage is leaner (less cpu/ram
> intensive) then compiling the programs from (github) source (assuming
> there is a difference at all). This is important as since it's a live
> cd, it is already ram-intensive and we also focus on low-spec machines
> (starting from i686)


There isn't a large difference, but using the musl overlays and the
Gentoo tree, you at least have a semi-decent chance of building without
any errors, as you will have the patches already put in.

Do note that Firefox requires at least 4 GB RAM with "lots of swap", and
at least 6 GB disk space:

https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Linux_Prerequisites

Now that it uses Rust (since version 58, IIRC), if you are on a source
distribution like Gentoo you will have to build Rust as well.  Rust is
very hard to build on musl (you *need* to use the overlays above), and
needs a lot of time and RAM/CPU.  For a specific example, the Adélie
builder for x86 is a 12-core Xeon system with 24 GB RAM.  It takes about
three hours to build Rust.


> Another possible way is to use anbox on gentoo and run the android
> versions of the programs (which can be found easily at the app store)
> directly (so not through installation with APK Tools). I don't know
> however whether this would run at all under gentoo musl and it's
> probably also overkill (as anbox, adb, ... need to be installed and
> the android programs might also not be stable even if they do run)


In addition to those issues, Firefox for Android takes a lot more RAM
than desktop Firefox (which is already a large amount; it barely runs in
512 MB here).  I wouldn't recommend that, though if you wanted to try
porting anbox to musl, I'm sure that'd be appreciated by the community.

-- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]