A running list of questions from "porting" Slackware to musl

A running list of questions from "porting" Slackware to musl

[Weldon Goree]

Hi all,

I added the quotation marks of shame because it's not a "port" in a real 
sense. But still: I've had this side project[1] for a while of porting 
Slackware to use Musl and it's Nearly There (tm), but I was hoping for 
some advice on some persistent irritations I have. (Sorry for the length.)

1. error.h. I've worked around most uses of this (where I didn't decide 
to simply delete the "else" branch) with a stub liberror (another list 
user suggested an actually functional musl-friendly liberror which I 
should probably integrate at some point). Still, I'd always like to hear 
other ideas. Kbd in particular remains problematic, though it at least 
compiles (which means it ships!).

2. NLS. Yeah, NLS. I realize this is still currently a moving target 
(and a very impressive one so far) on musl, so I've been adding 
--disable-nls or --with-gettext=no or whatever to the build scripts as a 
matter of course, but I'd like to hear what progress other musl users 
have made on this front. Can someone who "gets" this explain to a 
simpleton why gettext linked against musl doesn't present a libintl.so 
that behaves like client applications expect? (In particular the symbol 
'libintl_gettext' seems to not exist. Bonus points: why does libintl.h 
belong to the C library rather than gettext?)

2a. ${LIBDIR}/charset.alias. GNU software seems to love to make this 
file, and it only seems to be made when gettext is unhappy (though 
gettext devs seem to blame this on iconv, which I don't even use). It's 
similar to /usr/share/info/dir, in that subsequent packages are supposed 
to append to it, but appending is the bane of package management. I've 
altered my makepkg script to just delete it when it's found, but I'm 
curious if anybody knows a way to tell GNU tools they really don't have 
to make it. (This isn't really about musl, but other musl users have 
probably run into this.)

3. Is there a set of configure options for musl that would let me have 
libc.so in libdir, ld-musl-${ARCH}.so.1 in syslibdir, and *crt* in 
$(something else)? I would ideally like libc.so in /lib, the loader in 
/lib, and the crt files in /usr/lib, but a general solution would be 
appreciated. (Particularly one that "talks to" musl-gcc.specs, though I 
don't use that for this project). (This is literally just about my 
laziness in packaging gcc, so I won't be hurt if nobody suggests 
anything...)

4. Is Pth a lost cause?

5. Not meaning to start another round of spam against the gnulib team: I 
seem to have had good luck with findutils by simply ripping out their 
gnulib dir and updating it from the beta find. Does anybody have any 
caveats from having done this? (Bonus points: what in particular about 
fseeko makes it so difficult?) The only other option seems to be 
sabotage's "we will shoot gnulib files in the face" method, which I 
admit is emotionally satisfying. (../../../gnulibfix lib/ > cflags FTW)

6. Stack protection. This one really puzzles me. Stack protection is as 
alien to glibc as it is to musl, but I keep running into this. 90% of 
the problems can be avoided with adding -fno-stack-protector 
appropriately, but libtool is very "helpful" on matters like this and 
seems to find a way to put it back. I've actually not found an 
unworkable problem yet (though several very annoying ones); I guess I'm 
just curious what the real state of ssp on musl is (I'm not a fan of the 
concept, personally, but I know a lot of people are), and whether 
there's a general solution to just telling software to trust the ****ing 
stack.

7. Dynamic linking. In assembling muslack I've been leaning a lot on the 
shoulders of the giants who came before me. But in all that I keep 
running into static linking. Snowflake does some dynamic linking, and 
Sabotage submits to it when necessary (perl, etc.) but I don't know of a 
musl-based distro that dynamically links like "normal" people do. Does 
anybody know of one I can shamelessly steal from?

8. Finally: thanks to everybody on this project and on this list; I've 
really enjoyed the year since I read about musl on a random reddit comment.

Any thoughts would be appreciated,
Weldon

[1] http://muslack.org

Re: A running list of questions from "porting" Slackware to musl

[stephen Turner]

[-- Attachment #1: Type: text/plain, Size: 4928 bytes --]

Weldon, i actually started building my own lfs with musl and i am currently
contemplating my own distribution centered around my experiences in the IT
field. Do you have a good source for building out a complete system capable
of compiling? Im currently working on a minimal build environment using
Busybox, Bash, Linux-Headers, Musl, Make, Binutils, Glibc (with dependent
math libs)  And i am capable of building a working system however when you
test it the compiled files (a.out) are not working even after verifying
they are executable (chmod +x)

any thoughts?

On Tue, Sep 30, 2014 at 8:43 AM, Weldon Goree <weldon@langurwallah.org>
wrote:

> Hi all,
>
> I added the quotation marks of shame because it's not a "port" in a real
> sense. But still: I've had this side project[1] for a while of porting
> Slackware to use Musl and it's Nearly There (tm), but I was hoping for some
> advice on some persistent irritations I have. (Sorry for the length.)
>
> 1. error.h. I've worked around most uses of this (where I didn't decide to
> simply delete the "else" branch) with a stub liberror (another list user
> suggested an actually functional musl-friendly liberror which I should
> probably integrate at some point). Still, I'd always like to hear other
> ideas. Kbd in particular remains problematic, though it at least compiles
> (which means it ships!).
>
> 2. NLS. Yeah, NLS. I realize this is still currently a moving target (and
> a very impressive one so far) on musl, so I've been adding --disable-nls or
> --with-gettext=no or whatever to the build scripts as a matter of course,
> but I'd like to hear what progress other musl users have made on this
> front. Can someone who "gets" this explain to a simpleton why gettext
> linked against musl doesn't present a libintl.so that behaves like client
> applications expect? (In particular the symbol 'libintl_gettext' seems to
> not exist. Bonus points: why does libintl.h belong to the C library rather
> than gettext?)
>
> 2a. ${LIBDIR}/charset.alias. GNU software seems to love to make this file,
> and it only seems to be made when gettext is unhappy (though gettext devs
> seem to blame this on iconv, which I don't even use). It's similar to
> /usr/share/info/dir, in that subsequent packages are supposed to append to
> it, but appending is the bane of package management. I've altered my
> makepkg script to just delete it when it's found, but I'm curious if
> anybody knows a way to tell GNU tools they really don't have to make it.
> (This isn't really about musl, but other musl users have probably run into
> this.)
>
> 3. Is there a set of configure options for musl that would let me have
> libc.so in libdir, ld-musl-${ARCH}.so.1 in syslibdir, and *crt* in
> $(something else)? I would ideally like libc.so in /lib, the loader in
> /lib, and the crt files in /usr/lib, but a general solution would be
> appreciated. (Particularly one that "talks to" musl-gcc.specs, though I
> don't use that for this project). (This is literally just about my laziness
> in packaging gcc, so I won't be hurt if nobody suggests anything...)
>
> 4. Is Pth a lost cause?
>
> 5. Not meaning to start another round of spam against the gnulib team: I
> seem to have had good luck with findutils by simply ripping out their
> gnulib dir and updating it from the beta find. Does anybody have any
> caveats from having done this? (Bonus points: what in particular about
> fseeko makes it so difficult?) The only other option seems to be sabotage's
> "we will shoot gnulib files in the face" method, which I admit is
> emotionally satisfying. (../../../gnulibfix lib/ > cflags FTW)
>
> 6. Stack protection. This one really puzzles me. Stack protection is as
> alien to glibc as it is to musl, but I keep running into this. 90% of the
> problems can be avoided with adding -fno-stack-protector appropriately, but
> libtool is very "helpful" on matters like this and seems to find a way to
> put it back. I've actually not found an unworkable problem yet (though
> several very annoying ones); I guess I'm just curious what the real state
> of ssp on musl is (I'm not a fan of the concept, personally, but I know a
> lot of people are), and whether there's a general solution to just telling
> software to trust the ****ing stack.
>
> 7. Dynamic linking. In assembling muslack I've been leaning a lot on the
> shoulders of the giants who came before me. But in all that I keep running
> into static linking. Snowflake does some dynamic linking, and Sabotage
> submits to it when necessary (perl, etc.) but I don't know of a musl-based
> distro that dynamically links like "normal" people do. Does anybody know of
> one I can shamelessly steal from?
>
> 8. Finally: thanks to everybody on this project and on this list; I've
> really enjoyed the year since I read about musl on a random reddit comment.
>
> Any thoughts would be appreciated,
> Weldon
>
> [1] http://muslack.org
>

[-- Attachment #2: Type: text/html, Size: 5473 bytes --]

Re: A running list of questions from "porting" Slackware to musl

[Isaac Dunham]

On Tue, Sep 30, 2014 at 06:13:37PM +0530, Weldon Goree wrote:
> Hi all,
> 
> I added the quotation marks of shame because it's not a "port" in a real
> sense. But still: I've had this side project[1] for a while of porting
> Slackware to use Musl and it's Nearly There (tm), but I was hoping for some
> advice on some persistent irritations I have. (Sorry for the length.)
<snip> 
> 6. Stack protection. This one really puzzles me. Stack protection is as
> alien to glibc as it is to musl, but I keep running into this. 90% of the
> problems can be avoided with adding -fno-stack-protector appropriately, but
> libtool is very "helpful" on matters like this and seems to find a way to
> put it back. I've actually not found an unworkable problem yet (though
> several very annoying ones); I guess I'm just curious what the real state of
> ssp on musl is (I'm not a fan of the concept, personally, but I know a lot
> of people are), and whether there's a general solution to just telling
> software to trust the ****ing stack.

You need a "libssp_nonshared.a" containing a function named
__stack_chk_fail_local, which need only call __stack_chk_fail.
No idea why, but this cannot be in a shared libary.

> 7. Dynamic linking. In assembling muslack I've been leaning a lot on the
> shoulders of the giants who came before me. But in all that I keep running
> into static linking. Snowflake does some dynamic linking, and Sabotage
> submits to it when necessary (perl, etc.) but I don't know of a musl-based
> distro that dynamically links like "normal" people do. Does anybody know of
> one I can shamelessly steal from?

Alpine Linux, alpinelinux.org
> 8. Finally: thanks to everybody on this project and on this list; I've
> really enjoyed the year since I read about musl on a random reddit comment.

You're welcome.
> Any thoughts would be appreciated,
> Weldon

Hope this helps,
Isaac Dunham
> 
> [1] http://muslack.org

Re: A running list of questions from "porting" Slackware to musl

[Rich Felker]

On Tue, Sep 30, 2014 at 06:13:37PM +0530, Weldon Goree wrote:
> Hi all,
> 
> I added the quotation marks of shame because it's not a "port" in a
> real sense. But still: I've had this side project[1] for a while of
> porting Slackware to use Musl and it's Nearly There (tm), but I was
> hoping for some advice on some persistent irritations I have. (Sorry
> for the length.)

A good source of information, in general, is the patches from Sabotage
Linux and Alpine Linux.

> 1. error.h. I've worked around most uses of this (where I didn't
> decide to simply delete the "else" branch) with a stub liberror
> (another list user suggested an actually functional musl-friendly
> liberror which I should probably integrate at some point). Still,
> I'd always like to hear other ideas. Kbd in particular remains
> problematic, though it at least compiles (which means it ships!).

Yes, error.h has been an ongoing topic that wasn't really resolved.

> 2. NLS. Yeah, NLS. I realize this is still currently a moving target
> (and a very impressive one so far) on musl, so I've been adding
> --disable-nls or --with-gettext=no or whatever to the build scripts
> as a matter of course, but I'd like to hear what progress other musl
> users have made on this front. Can someone who "gets" this explain
> to a simpleton why gettext linked against musl doesn't present a
> libintl.so that behaves like client applications expect? (In
> particular the symbol 'libintl_gettext' seems to not exist. Bonus
> points: why does libintl.h belong to the C library rather than
> gettext?)

Like glibc, musl supports the gettext API natively. There is no need
for libintl and it should not be installed. libintl.h is just the
header for the gettext API.

I'm not sure what's happening with symbol names in your libintl. I
suspect it was either built incorrectly or has some invalid glibc
assumptions, but I know other users (even Alpine Linux) have used it
successfully with musl in the past, before musl had gettext.

> 2a. ${LIBDIR}/charset.alias. GNU software seems to love to make this
> file, and it only seems to be made when gettext is unhappy (though
> gettext devs seem to blame this on iconv, which I don't even use).
> It's similar to /usr/share/info/dir, in that subsequent packages are
> supposed to append to it, but appending is the bane of package
> management. I've altered my makepkg script to just delete it when
> it's found, but I'm curious if anybody knows a way to tell GNU tools
> they really don't have to make it. (This isn't really about musl,
> but other musl users have probably run into this.)

I've seen this but I just ignore it; I never really looked into the
cause.

> 3. Is there a set of configure options for musl that would let me
> have libc.so in libdir, ld-musl-${ARCH}.so.1 in syslibdir, and *crt*
> in $(something else)? I would ideally like libc.so in /lib, the
> loader in /lib, and the crt files in /usr/lib, but a general
> solution would be appreciated.

I don't think gcc/ld search /lib when linking, so putting libc.so
there (and not having it in /usr/lib) would probably leave things
broken. If the core issue is that you want the actual file on /, you
could swap the direction of the symlink (make ld-musl the real file)
or make both ld-musl and /usr/lib/libc.so symlinks to /lib/libc.so or
similar.

> (Particularly one that "talks to"
> musl-gcc.specs, though I don't use that for this project). (This is
> literally just about my laziness in packaging gcc, so I won't be
> hurt if nobody suggests anything...)

If you're "porting" the whole system to build against musl, you
shouldn't need the musl-gcc wrapper, and in fact you're going to have
a hard time using it as soon as you try to build C++ software. You
really want a native compiler toolchain that supports musl. See the
musl-cross project for the necessary patches to GCC.

> 4. Is Pth a lost cause?

Yes. But there's a new portable pth (sorry, couldn't resist) in the
works that several projects (iirc gnupg) are eagar to switch to, since
pth is so bad and has no future (it's incompatible with programs that
use real threads, and likely with most/all future systems). It's
API-compatible but based on pthread so it works anywhere. No idea what
the status is, though.

> 5. Not meaning to start another round of spam against the gnulib
> team: I seem to have had good luck with findutils by simply ripping
> out their gnulib dir and updating it from the beta find. Does
> anybody have any caveats from having done this? (Bonus points: what
> in particular about fseeko makes it so difficult?) The only other
> option seems to be sabotage's "we will shoot gnulib files in the
> face" method, which I admit is emotionally satisfying.
> (../../../gnulibfix lib/ > cflags FTW)

Yes, updating gnulib is a good idea. Actually gnulib documents a way
to update in-tree without "ripping out" the old one first, but I don't
recall the details.

> 6. Stack protection. This one really puzzles me. Stack protection is
> as alien to glibc as it is to musl, but I keep running into this.
> 90% of the problems can be avoided with adding -fno-stack-protector
> appropriately, but libtool is very "helpful" on matters like this
> and seems to find a way to put it back. I've actually not found an
> unworkable problem yet (though several very annoying ones); I guess
> I'm just curious what the real state of ssp on musl is (I'm not a
> fan of the concept, personally, but I know a lot of people are), and
> whether there's a general solution to just telling software to trust
> the ****ing stack.

I'd need more context to know what, if anything, you're doing wrong,
or if the packages you're compiling just insist on using stack
protector. In any case having it on should not break anything, but you
may need libssp_nonshared.a (from a gcc natively targeting musl) for
ssp to link right in shared libraries. libssp.so should not exist, and
libssp.a should be an empty .a file (ar rc libssp.a to make it).

> 7. Dynamic linking. In assembling muslack I've been leaning a lot on
> the shoulders of the giants who came before me. But in all that I
> keep running into static linking. Snowflake does some dynamic
> linking, and Sabotage submits to it when necessary (perl, etc.) but
> I don't know of a musl-based distro that dynamically links like
> "normal" people do. Does anybody know of one I can shamelessly steal
> from?

Alpine. But there shouldn't be any particular difficulty to dynamic
linking; in fact it's done by default by GCC when possible.

Rich

Re: A running list of questions from "porting" Slackware to musl

[Rich Felker]

On Tue, Sep 30, 2014 at 08:32:16AM -0700, Isaac Dunham wrote:
> On Tue, Sep 30, 2014 at 06:13:37PM +0530, Weldon Goree wrote:
> > Hi all,
> > 
> > I added the quotation marks of shame because it's not a "port" in a real
> > sense. But still: I've had this side project[1] for a while of porting
> > Slackware to use Musl and it's Nearly There (tm), but I was hoping for some
> > advice on some persistent irritations I have. (Sorry for the length.)
> <snip> 
> > 6. Stack protection. This one really puzzles me. Stack protection is as
> > alien to glibc as it is to musl, but I keep running into this. 90% of the
> > problems can be avoided with adding -fno-stack-protector appropriately, but
> > libtool is very "helpful" on matters like this and seems to find a way to
> > put it back. I've actually not found an unworkable problem yet (though
> > several very annoying ones); I guess I'm just curious what the real state of
> > ssp on musl is (I'm not a fan of the concept, personally, but I know a lot
> > of people are), and whether there's a general solution to just telling
> > software to trust the ****ing stack.
> 
> You need a "libssp_nonshared.a" containing a function named
> __stack_chk_fail_local, which need only call __stack_chk_fail.
> No idea why, but this cannot be in a shared libary.

When gcc generates the canary-check code, on failure it normally
calls/jumps to __stack_chk_fail. But for shared libraries, that call
would go to a thunk in the library's PLT, which depends on the GOT
register being initialized (actually this varies by arch; x86_64
doesn't need it). In order to avoid (expensive) loading of the GOT
register in every function just as a contingency in case
__stack_chk_fail needs to be called, for position-independent code GCC
generates a call to __stack_chk_fail_local instead. This is a hidden
function (and necessarily exists within the same .so) so the call
doesn't have to go through the PLT; it's just a straight relative
call/jump instruction. __stack_chk_fail_local is then responsible for
loading the GOT register and calling __stack_chk_fail.

Rich

Re: A running list of questions from "porting" Slackware to musl

[Weldon Goree]

On 09/30/2014 09:16 PM, Rich Felker wrote:
>
> Like glibc, musl supports the gettext API natively. There is no need
> for libintl and it should not be installed. libintl.h is just the
> header for the gettext API.
>
> I'm not sure what's happening with symbol names in your libintl. I
> suspect it was either built incorrectly or has some invalid glibc
> assumptions, but I know other users (even Alpine Linux) have used it
> successfully with musl in the past, before musl had gettext.
>

Nor I, but I've tried it with all the permutations of older gettext 
versions and no gettext versions, and several GNU programs fail to link 
if they aren't told to --disable-nls. I'm not a programmer, just a 
sysadmin, and I'm interested in musl because it might make my life 
easier in the future, but I'm in no place (yet) to figure out what I've 
done wrong.

> I don't think gcc/ld search /lib when linking, so putting libc.so
> there (and not having it in /usr/lib) would probably leave things
> broken. If the core issue is that you want the actual file on /, you
> could swap the direction of the symlink (make ld-musl the real file)
> or make both ld-musl and /usr/lib/libc.so symlinks to /lib/libc.so or
> similar.
>

When I cross-compile a native gcc it "wants" the linker to be in /lib 
(or I should say, Gregor's patches tell it to look for the symlink in 
/lib, and my being a sysadmin tells me that whatever /lib/ld-foo links 
to should not be in /usr because I do have to boot my server occasionally).

>
> If you're "porting" the whole system to build against musl, you
> shouldn't need the musl-gcc wrapper, and in fact you're going to have
> a hard time using it as soon as you try to build C++ software. You
> really want a native compiler toolchain that supports musl. See the
> musl-cross project for the necessary patches to GCC.

Granted: my slackware port doesn't use musl-gcc, but I'm maintaining the 
community port of musl for slackware-gnu users too.

>
>> 4. Is Pth a lost cause?
>
> Yes. But there's a new portable pth (sorry, couldn't resist) in the
> works that several projects (iirc gnupg) are eagar to switch to, since
> pth is so bad and has no future (it's incompatible with programs that
> use real threads, and likely with most/all future systems). It's
> API-compatible but based on pthread so it works anywhere. No idea what
> the status is, though.
>

I'll look into it. xorg was happy with the stub lib they wrote years ago 
so I haven't looked past that because it did what I needed; that was 
more out of curiosity.

>
> I'd need more context to know what, if anything, you're doing wrong,
> or if the packages you're compiling just insist on using stack
> protector. In any case having it on should not break anything, but you
> may need libssp_nonshared.a (from a gcc natively targeting musl) for
> ssp to link right in shared libraries. libssp.so should not exist, and
> libssp.a should be an empty .a file (ar rc libssp.a to make it).
>

RPM is the candidate this time; I only need it to build rpm2cpio to get 
the linuxdoc sources (I found a workaround), but it also took some 
finagling to get Perl to stop "helping" me by adding -f-stack-protector. 
This was again a point of curiosity.

Cheers,
Weldon

Re: A running list of questions from "porting" Slackware to musl

[Weldon Goree]

On 09/30/2014 08:29 PM, stephen Turner wrote:
> Weldon, i actually started building my own lfs with musl and i am
> currently contemplating my own distribution centered around my
> experiences in the IT field. Do you have a good source for building out
> a complete system capable of compiling? Im currently working on a
> minimal build environment using Busybox, Bash, Linux-Headers, Musl,
> Make, Binutils, Glibc (with dependent math libs)  And i am capable of
> building a working system however when you test it the compiled files
> (a.out) are not working even after verifying they are executable (chmod +x)
>
> any thoughts?
>

Well, for this latest project I've been using Slackware's ARM port 
documentation (http://alien.slackbook.org/blog/armport/), though 
obviously the particulars of $TARGET and $HOST are different (but pretty 
easily substituded). Cross Linux From Scratch (http://cross-lfs.org/) 
helped too, though some of the things they do are frankly psychotic, 
particularly with gcc's spec file (seriously: ignore all of that and 
just use CFLAGS and LDFLAGS).

I've also got a side-side project going on to do a radically simplified 
Linux userland (as few code paths as possible), and that's pretty 
straightforward from Sabotage (even if sabotage isn't doing what you're 
doing, it's worth reading what they're doing: 
https://github.com/sabotage-linux/sabotage) as well as the good people 
at suckless (http://suckless.org).

Cheers,
Weldon

Re: A running list of questions from "porting" Slackware to musl

[Andy Lutomirski]

On 09/30/2014 08:50 AM, Rich Felker wrote:
> On Tue, Sep 30, 2014 at 08:32:16AM -0700, Isaac Dunham wrote:
>> On Tue, Sep 30, 2014 at 06:13:37PM +0530, Weldon Goree wrote:
>>> Hi all,
>>>
>>> I added the quotation marks of shame because it's not a "port" in a real
>>> sense. But still: I've had this side project[1] for a while of porting
>>> Slackware to use Musl and it's Nearly There (tm), but I was hoping for some
>>> advice on some persistent irritations I have. (Sorry for the length.)
>> <snip> 
>>> 6. Stack protection. This one really puzzles me. Stack protection is as
>>> alien to glibc as it is to musl, but I keep running into this. 90% of the
>>> problems can be avoided with adding -fno-stack-protector appropriately, but
>>> libtool is very "helpful" on matters like this and seems to find a way to
>>> put it back. I've actually not found an unworkable problem yet (though
>>> several very annoying ones); I guess I'm just curious what the real state of
>>> ssp on musl is (I'm not a fan of the concept, personally, but I know a lot
>>> of people are), and whether there's a general solution to just telling
>>> software to trust the ****ing stack.
>>
>> You need a "libssp_nonshared.a" containing a function named
>> __stack_chk_fail_local, which need only call __stack_chk_fail.
>> No idea why, but this cannot be in a shared libary.
> 
> When gcc generates the canary-check code, on failure it normally
> calls/jumps to __stack_chk_fail. But for shared libraries, that call
> would go to a thunk in the library's PLT, which depends on the GOT
> register being initialized (actually this varies by arch; x86_64
> doesn't need it). In order to avoid (expensive) loading of the GOT
> register in every function just as a contingency in case
> __stack_chk_fail needs to be called, for position-independent code GCC
> generates a call to __stack_chk_fail_local instead. This is a hidden
> function (and necessarily exists within the same .so) so the call
> doesn't have to go through the PLT; it's just a straight relative
> call/jump instruction. __stack_chk_fail_local is then responsible for
> loading the GOT register and calling __stack_chk_fail.

[slightly off topic]

Does GCC even know how to call through the GOT instead of the PLT?
Windows (at least 32-bit Windows) has done for decades, at least if
dllimport is set.

On x86_64, this would be call *whatever@gotoff(%rip) instead of call
whatever@plt.

(Even better: the loader could patch the PLT with a direct jump.  Could
musl do this?  At least in the case where the symbol is within 2G of the
PLT entry, this should be straightforward if no threads have been
started yet.  If musl did this, it could advertise a nice speedup over
glibc...)

--Andy

> 
> Rich
> 

Re: Re: A running list of questions from "porting" Slackware to musl

[Rich Felker]

On Tue, Sep 30, 2014 at 04:50:28PM -0700, Andy Lutomirski wrote:
> > When gcc generates the canary-check code, on failure it normally
> > calls/jumps to __stack_chk_fail. But for shared libraries, that call
> > would go to a thunk in the library's PLT, which depends on the GOT
> > register being initialized (actually this varies by arch; x86_64
> > doesn't need it). In order to avoid (expensive) loading of the GOT
> > register in every function just as a contingency in case
> > __stack_chk_fail needs to be called, for position-independent code GCC
> > generates a call to __stack_chk_fail_local instead. This is a hidden
> > function (and necessarily exists within the same .so) so the call
> > doesn't have to go through the PLT; it's just a straight relative
> > call/jump instruction. __stack_chk_fail_local is then responsible for
> > loading the GOT register and calling __stack_chk_fail.
> 
> [slightly off topic]
> 
> Does GCC even know how to call through the GOT instead of the PLT?
> Windows (at least 32-bit Windows) has done for decades, at least if
> dllimport is set.
> 
> On x86_64, this would be call *whatever@gotoff(%rip) instead of call
> whatever@plt.

This precludes optimizing out the indirection at link time (or at
least it requires more complex transformation in the linker). I'm not
sure if there are cases where GCC generates this kind of code or not.
It's also not practical on many ISAs.

> (Even better: the loader could patch the PLT with a direct jump.  Could
> musl do this?  At least in the case where the symbol is within 2G of the
> PLT entry,

This is really not a good idea. The old PowerPC ABI did this, and musl
does not support it (it requires the new "secure-plt" mode). Hardened
kernels have various restrictions on modifying executable pages, up to
and including completely forbidding this kind of usage. And even if
it's not forbidden, it's going to use more memory due to an additional
page (or more) per shared library that's not going to be sharable.
Also it requires complex per-arch code (minimal machine code
generation, instruction cache flushing/barriers, etc.).

> this should be straightforward if no threads have been
> started yet.

Threads having been started or not are not relevant. The newly loaded
code is not visible until dlopen returns, so nothing can race with
modifications to it.

> If musl did this, it could advertise a nice speedup over
> glibc...)

I think the performance gain would be mostly theoretical. Do you have
any timings that show otherwise?

Rich

Re: A running list of questions from "porting" Slackware to musl

[Andy Lutomirski]

On 09/30/2014 05:05 PM, Rich Felker wrote:
> On Tue, Sep 30, 2014 at 04:50:28PM -0700, Andy Lutomirski wrote:
>>> When gcc generates the canary-check code, on failure it normally
>>> calls/jumps to __stack_chk_fail. But for shared libraries, that call
>>> would go to a thunk in the library's PLT, which depends on the GOT
>>> register being initialized (actually this varies by arch; x86_64
>>> doesn't need it). In order to avoid (expensive) loading of the GOT
>>> register in every function just as a contingency in case
>>> __stack_chk_fail needs to be called, for position-independent code GCC
>>> generates a call to __stack_chk_fail_local instead. This is a hidden
>>> function (and necessarily exists within the same .so) so the call
>>> doesn't have to go through the PLT; it's just a straight relative
>>> call/jump instruction. __stack_chk_fail_local is then responsible for
>>> loading the GOT register and calling __stack_chk_fail.
>>
>> [slightly off topic]
>>
>> Does GCC even know how to call through the GOT instead of the PLT?
>> Windows (at least 32-bit Windows) has done for decades, at least if
>> dllimport is set.
>>
>> On x86_64, this would be call *whatever@gotoff(%rip) instead of call
>> whatever@plt.
>
> This precludes optimizing out the indirection at link time (or at
> least it requires more complex transformation in the linker). I'm not
> sure if there are cases where GCC generates this kind of code or not.
> It's also not practical on many ISAs.

I think I filed a bug asking for this (among other things) in GCC once. 
  Basically, I want __attribute__((visibility("imported"))) or something 
that.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=56527

>
>> (Even better: the loader could patch the PLT with a direct jump.  Could
>> musl do this?  At least in the case where the symbol is within 2G of the
>> PLT entry,
>
> This is really not a good idea. The old PowerPC ABI did this, and musl
> does not support it (it requires the new "secure-plt" mode). Hardened
> kernels have various restrictions on modifying executable pages, up to
> and including completely forbidding this kind of usage. And even if
> it's not forbidden, it's going to use more memory due to an additional
> page (or more) per shared library that's not going to be sharable.
> Also it requires complex per-arch code (minimal machine code
> generation, instruction cache flushing/barriers, etc.).

That extra page might not be needed if the linker could end up removing 
a bunch of GOT entries for functions that don't have their addresses 
taken.  (Or, on x86_64, where unaligned access is cheap, the GOT could 
actually overlap the PLT in memory, but only if DT_BIND_NOW or whatever 
it's called is on.  Hmm.  I bet that the linker could do this in a way 
that doesn't require loader support at all as long as textrel is allowed.)

>
>> this should be straightforward if no threads have been
>> started yet.
>
> Threads having been started or not are not relevant. The newly loaded
> code is not visible until dlopen returns, so nothing can race with
> modifications to it.

True, at least when lazy binding is off.

>
>> If musl did this, it could advertise a nice speedup over
>> glibc...)
>
> I think the performance gain would be mostly theoretical. Do you have
> any timings that show otherwise?

No.  It would reduce pressure on whatever presumably limited resources 
the CPU has for predicting indirect jumps, and it would reduce the 
number of cache lines needed for a call through the PLT.

Doing it cleanly would also probably require a new dynamic entry and a 
new relocation type.

Also, it might be a lost cause when selinux is being used.  I *hate* 
execmem, execmod, etc -- it really should be possible to do this and to 
write a sensible JIT without requiring special selinux permissions.  I 
think that what's needed is a syscall to make a writeable alias of an 
executable mapping.

Anyway, probably not worth it.

--Andy

Re: A running list of questions from "porting" Slackware to musl

[Timo Teras]

Hi,

I'm writing on behalf of Alpine Linux, as I did most of the integration
work for musl there.

On Tue, 30 Sep 2014 18:13:37 +0530
Weldon Goree <weldon@langurwallah.org> wrote:

> 1. error.h. I've worked around most uses of this (where I didn't
> decide to simply delete the "else" branch) with a stub liberror
> (another list user suggested an actually functional musl-friendly
> liberror which I should probably integrate at some point). Still, I'd
> always like to hear other ideas. Kbd in particular remains
> problematic, though it at least compiles (which means it ships!).

We generally have patched these out. Would probably be nice to have this
in musl, or as an add-on library. Then again, some projects seem to
have stopped using error.h because of portability issues to BSD and
other OSes.

> 2. NLS. Yeah, NLS. I realize this is still currently a moving target 
> (and a very impressive one so far) on musl, so I've been adding 
> --disable-nls or --with-gettext=no or whatever to the build scripts
> as a matter of course, but I'd like to hear what progress other musl
> users have made on this front. Can someone who "gets" this explain to
> a simpleton why gettext linked against musl doesn't present a
> libintl.so that behaves like client applications expect? (In
> particular the symbol 'libintl_gettext' seems to not exist. Bonus
> points: why does libintl.h belong to the C library rather than
> gettext?)

gettext can be compiled in two modes: "use libc gettext" or
"ship libintl gettext".

In musl 1.0 we removed gettext from musl. Currently we just remove
libintl.h from musl-dev and use libintl's version when needed. We build
only specific set against libintl-dev; so most packages in Alpine do not
get nls enabled due to missing libintl.h.

We plan to start using musl's libintl.h and remove gettext's libintl
later.

> 2a. ${LIBDIR}/charset.alias. GNU software seems to love to make this 
> file, and it only seems to be made when gettext is unhappy (though 
> gettext devs seem to blame this on iconv, which I don't even use).
> It's similar to /usr/share/info/dir, in that subsequent packages are
> supposed to append to it, but appending is the bane of package
> management. I've altered my makepkg script to just delete it when
> it's found, but I'm curious if anybody knows a way to tell GNU tools
> they really don't have to make it. (This isn't really about musl, but
> other musl users have probably run into this.)

This is due to gnulib. It does this if it does not find glibc. We did
not bother to fix gnulib, but are just removing it manually before
packaging. Would probably be good to fix gnulib.

> 3. Is there a set of configure options for musl that would let me
> have libc.so in libdir, ld-musl-${ARCH}.so.1 in syslibdir, and *crt*
> in $(something else)? I would ideally like libc.so in /lib, the
> loader in /lib, and the crt files in /usr/lib, but a general solution
> would be appreciated. (Particularly one that "talks to"
> musl-gcc.specs, though I don't use that for this project). (This is
> literally just about my laziness in packaging gcc, so I won't be hurt
> if nobody suggests anything...)

Like Rich mentioned the option in other mail, it is done in Alpine
Linux. The direction of the libc.so and symlink is changed. ld-musl is
the real file, and libc.so is the symlink.

IIRC, the primary reason was to keep libc.so and libc.a both
in /usr/lib, and have ld-musl in /lib. If the .so and .a are in
different paths we got trouble with gcc. I think glibc this works
because the libc.so in /lib is versioned (as-in libc.so.1.2.3) and
libc.so symlink to that is in /usr/lib or similar.

> 4. Is Pth a lost cause?

Yes. It's evil, don't spend time on it. Only gpg 2.0.x uses that, and
they remove it for gpg 2.1.x. It'll use npth. npth is slightly ugly too
but at least that works.

> 5. Not meaning to start another round of spam against the gnulib
> team: I seem to have had good luck with findutils by simply ripping
> out their gnulib dir and updating it from the beta find. Does anybody
> have any caveats from having done this? (Bonus points: what in
> particular about fseeko makes it so difficult?) The only other option
> seems to be sabotage's "we will shoot gnulib files in the face"
> method, which I admit is emotionally satisfying. (../../../gnulibfix
> lib/ > cflags FTW)

It's because findutils has ancient gnulib. Seems gnulib is causing more
trouble then fixing with musl. The most unintrusive patch we came with
is at:
http://git.alpinelinux.org/cgit/aports/tree/main/findutils/fix-gnulib-freadahead.patch

Seems upgrading gnulib is tedious. Though, we had to do it one package
(or possibly two) and the diffs are huge.

> 6. Stack protection. This one really puzzles me. Stack protection is
> as alien to glibc as it is to musl, but I keep running into this. 90%
> of the problems can be avoided with adding -fno-stack-protector 
> appropriately, but libtool is very "helpful" on matters like this and 
tmu> seems to find a way to put it back. I've actually not found an 
> unworkable problem yet (though several very annoying ones); I guess
> I'm just curious what the real state of ssp on musl is (I'm not a fan
> of the concept, personally, but I know a lot of people are), and
> whether there's a general solution to just telling software to trust
> the ****ing stack.

Alpine ships libssp_nonshared.a:
http://git.alpinelinux.org/cgit/aports/tree/main/musl/APKBUILD#n66

Doing it like this means you also need to compile gcc with libssp
disabled. See:
http://git.alpinelinux.org/cgit/aports/tree/main/gcc/APKBUILD#n272

IIRC, Gcc's libssp is supposed to provide that normally, but it breaks
horrible because it does not properly detect libssp in musl and has
glibc specific hacks. As musl has full libssp functionality (in terms
what gcc code generation requires) with the exception of this
libssp_nonshared.a, it felt simplest to add it to Alpine's musl
packaging for the time being.

IIRC, this is architecture dependant if it's really needed or not. We
had long discussion and the end result was that gcc should really ship
that function in linkonce section and not depend on external library.

> 7. Dynamic linking. In assembling muslack I've been leaning a lot on
> the shoulders of the giants who came before me. But in all that I
> keep running into static linking. Snowflake does some dynamic
> linking, and Sabotage submits to it when necessary (perl, etc.) but I
> don't know of a musl-based distro that dynamically links like
> "normal" people do. Does anybody know of one I can shamelessly steal
> from?

As mentioned before, we build almost all in Alpine dynamically linked.
Feel free to look at what we do.


/Timo

Re: Re: A running list of questions from "porting" Slackware to musl

[Szabolcs Nagy]

* Andy Lutomirski <luto@amacapital.net> [2014-09-30 16:50:28 -0700]:
> On 09/30/2014 08:50 AM, Rich Felker wrote:
> > When gcc generates the canary-check code, on failure it normally
> > calls/jumps to __stack_chk_fail. But for shared libraries, that call
> > would go to a thunk in the library's PLT, which depends on the GOT
> > register being initialized (actually this varies by arch; x86_64
...
> 
> On x86_64, this would be call *whatever@gotoff(%rip) instead of call
> whatever@plt.
> 
> (Even better: the loader could patch the PLT with a direct jump.  Could
...

hm this seems to be a lot of complication just to crash

if gcc had a -fcrash-on-ssp-chk-fail flag that simply generated
a crash instruction that would be simpler/smaller/more secure

(actually i think that should be the default behaviour)

Re: Re: A running list of questions from "porting" Slackware to musl

[u-wsnj]

On Wed, Oct 01, 2014 at 09:48:17AM +0200, Szabolcs Nagy wrote:
> if gcc had a -fcrash-on-ssp-chk-fail flag that simply generated
> a crash instruction that would be simpler/smaller/more secure
> 
> (actually i think that should be the default behaviour)

+1

Rune

Re: Re: A running list of questions from "porting" Slackware to musl

[Rich Felker]

On Tue, Sep 30, 2014 at 10:49:15PM -0700, Andy Lutomirski wrote:
> >>(Even better: the loader could patch the PLT with a direct jump.  Could
> >>musl do this?  At least in the case where the symbol is within 2G of the
> >>PLT entry,
> >
> >This is really not a good idea. The old PowerPC ABI did this, and musl
> >does not support it (it requires the new "secure-plt" mode). Hardened
> >kernels have various restrictions on modifying executable pages, up to
> >and including completely forbidding this kind of usage. And even if
> >it's not forbidden, it's going to use more memory due to an additional
> >page (or more) per shared library that's not going to be sharable.
> >Also it requires complex per-arch code (minimal machine code
> >generation, instruction cache flushing/barriers, etc.).
> 
> That extra page might not be needed if the linker could end up
> removing a bunch of GOT entries for functions that don't have their
> addresses taken.  (Or, on x86_64, where unaligned access is cheap,
> the GOT could actually overlap the PLT in memory, but only if

This is not an option. It would require the page containing the GOT
(and a lot of data) to be executable. Not only is this a huge security
risk (makes exploiting other vulnerabilities a lot easier); it's also
physically impossible to do on hardened kernels which simply lack
"rwx" permission for mappings.

> >>this should be straightforward if no threads have been
> >>started yet.
> >
> >Threads having been started or not are not relevant. The newly loaded
> >code is not visible until dlopen returns, so nothing can race with
> >modifications to it.
> 
> True, at least when lazy binding is off.

musl does not do lazy binding at all, and won't. There's been some
demand for it in terms of allowing loading hacks where the symbols
needed by the loaded library are not provided until later, but that
can be satisfied by emulating it (basically, keeping a list of
unsatisfied relocs and retrying them after each dlopen) rather than
actually doing lazy binding at call time.

Rich

Re: Re: A running list of questions from "porting" Slackware to musl

[Rich Felker]

On Wed, Oct 01, 2014 at 09:48:17AM +0200, Szabolcs Nagy wrote:
> * Andy Lutomirski <luto@amacapital.net> [2014-09-30 16:50:28 -0700]:
> > On 09/30/2014 08:50 AM, Rich Felker wrote:
> > > When gcc generates the canary-check code, on failure it normally
> > > calls/jumps to __stack_chk_fail. But for shared libraries, that call
> > > would go to a thunk in the library's PLT, which depends on the GOT
> > > register being initialized (actually this varies by arch; x86_64
> ....
> > 
> > On x86_64, this would be call *whatever@gotoff(%rip) instead of call
> > whatever@plt.
> > 
> > (Even better: the loader could patch the PLT with a direct jump.  Could
> ....
> 
> hm this seems to be a lot of complication just to crash
> 
> if gcc had a -fcrash-on-ssp-chk-fail flag that simply generated
> a crash instruction that would be simpler/smaller/more secure
> 
> (actually i think that should be the default behaviour)

Yes, it really should. Perhaps we could post a GCC bug report
requesting this with a link to one or more of the articles on
exploiting the introspective debug code in gcc/glibc crash handlers.

Rich

Re: A running list of questions from "porting" Slackware to musl

[Andy Lutomirski]

On 10/01/2014 06:29 AM, Rich Felker wrote:
> On Tue, Sep 30, 2014 at 10:49:15PM -0700, Andy Lutomirski wrote:
>>>> (Even better: the loader could patch the PLT with a direct jump.  Could
>>>> musl do this?  At least in the case where the symbol is within 2G of the
>>>> PLT entry,
>>>
>>> This is really not a good idea. The old PowerPC ABI did this, and musl
>>> does not support it (it requires the new "secure-plt" mode). Hardened
>>> kernels have various restrictions on modifying executable pages, up to
>>> and including completely forbidding this kind of usage. And even if
>>> it's not forbidden, it's going to use more memory due to an additional
>>> page (or more) per shared library that's not going to be sharable.
>>> Also it requires complex per-arch code (minimal machine code
>>> generation, instruction cache flushing/barriers, etc.).
>>
>> That extra page might not be needed if the linker could end up
>> removing a bunch of GOT entries for functions that don't have their
>> addresses taken.  (Or, on x86_64, where unaligned access is cheap,
>> the GOT could actually overlap the PLT in memory, but only if
>
> This is not an option. It would require the page containing the GOT
> (and a lot of data) to be executable. Not only is this a huge security
> risk (makes exploiting other vulnerabilities a lot easier); it's also
> physically impossible to do on hardened kernels which simply lack
> "rwx" permission for mappings.

No rwx mapping would ever be needed.  The GOT would be shoved *into* the 
text segment, and the loader would switch it to rw, write in the 
relocations, and switch it back to rx.

Some hardened kernels disallow this.  I think that this particular 
hardening option has long since served its purpose (distros no longer 
ship things that are dangerous like this), but any exploit that can 
convince a buggy program to do:

  - mprotect to rw (or just find an rw mapping to begin with)
  - attacker-controlled shellcode write
  - mprotect to rx
  - attacker-contolled jump

on a non-stack address has already won (they could just as easily have 
written a ROP or SROP payload, or they could have written an environment 
block and called execve instead of mprotect).  Disallowing the mprotect 
to rx is IMO utterly pointless and merely annoys everyone who tries to 
write a JIT compiler.

At least new enough kernels probably allow reliable JITting using 
memfd_create and two mmap calls.

(It wouldn't work anyway as I've described it because x86_64 has no 
64-bit absolute jump.)

Anyway, this is off-topic enough that I'll shut up now.

>
>>>> this should be straightforward if no threads have been
>>>> started yet.
>>>
>>> Threads having been started or not are not relevant. The newly loaded
>>> code is not visible until dlopen returns, so nothing can race with
>>> modifications to it.
>>
>> True, at least when lazy binding is off.
>
> musl does not do lazy binding at all, and won't. There's been some
> demand for it in terms of allowing loading hacks where the symbols
> needed by the loaded library are not provided until later, but that
> can be satisfied by emulating it (basically, keeping a list of
> unsatisfied relocs and retrying them after each dlopen) rather than
> actually doing lazy binding at call time.
>
> Rich
>