libc.so segfaults on mips

libc.so segfaults on mips

[Рысь]

Hi list!

Not a really blocking bug but I was wondering why cross compiled musl libc.so for embedded wrt system is segfaults with this:

% ./strace ./libc.so 
execve("./libc.so", ["./libc.so"], [/* 10 vars */]) = 0
--- {si_signo=SIGSEGV, si_code=SI_USER, si_errno=EPERM, si_addr=0x4} (Segmentation fault) ---
+++ killed by SIGSEGV +++
Segmentation fault

A strange segfault with EPERM.

The target is mips2 bigendian router, specifically it is ar9331 SoC.
This is how file reports on libc.so: ELF 32-bit MSB shared object, MIPS, MIPS-I version 1 (SYSV), dynamically linked, not stripped

Such a trouble I had once when system enforced mmap_min_addr, however there it has default value.

The toolchain is somewhat old: gcc 4.2.1, binutils 2.17.50.0.17 but confirmed working and stable.

Musl is 1.1.4.

What else can cause this?

Re: libc.so segfaults on mips

[Szabolcs Nagy]

* ???????? <lynx@sibserver.ru> [2014-09-04 16:56:40 +0800]:
> 4 ???????????????? 2014 ??. 15:35:45 KRAT, Rich Felker <dalias@libc.org> ??????????:
> >On Thu, Sep 04, 2014 at 03:07:57PM +0800, ???????? wrote:
> >> Not a really blocking bug but I was wondering why cross compiled
> >> musl libc.so for embedded wrt system is segfaults with this:

OpenWRT does not use standard mips ABI but its own thing:
you need mips-sf (softfloat) toolchain

> >> % ./strace ./libc.so 
> >> execve("./libc.so", ["./libc.so"], [/* 10 vars */]) = 0
> >> --- {si_signo=SIGSEGV, si_code=SI_USER, si_errno=EPERM, si_addr=0x4}
> >(Segmentation fault) ---
> >> +++ killed by SIGSEGV +++
> >> Segmentation fault
> >> 
> >> A strange segfault with EPERM.
> >> 
> >> The target is mips2 bigendian router, specifically it is ar9331 SoC.
> >> This is how file reports on libc.so: ELF 32-bit MSB shared object,
> >> MIPS, MIPS-I version 1 (SYSV), dynamically linked, not stripped

> >Could you attach the output of readelf -a libc.so? I'm guessing the
> The output of readelf attached.

i don't see the softfloat symbols of libgcc there
(eg __addsf3)

(musl malloc uses float arithmetics so ./libc.so will fail
immediately)

(openwrt kernel can be built with fpu emulation which is
the default on normal mips kernels)

Re: libc.so segfaults on mips

[Рысь]

4 сентября 2014 г. 20:32:09 KRAT, Szabolcs Nagy <nsz@port70.net> пишет:
>* ???????? <lynx@sibserver.ru> [2014-09-04 16:56:40 +0800]:
>> 4 ???????????????? 2014 ??. 15:35:45 KRAT, Rich Felker
><dalias@libc.org> ??????????:
>> >On Thu, Sep 04, 2014 at 03:07:57PM +0800, ???????? wrote:
>> >> Not a really blocking bug but I was wondering why cross compiled
>> >> musl libc.so for embedded wrt system is segfaults with this:
>
>OpenWRT does not use standard mips ABI but its own thing:
>you need mips-sf (softfloat) toolchain
>
>> >> % ./strace ./libc.so 
>> >> execve("./libc.so", ["./libc.so"], [/* 10 vars */]) = 0
>> >> --- {si_signo=SIGSEGV, si_code=SI_USER, si_errno=EPERM,
>si_addr=0x4}
>> >(Segmentation fault) ---
>> >> +++ killed by SIGSEGV +++
>> >> Segmentation fault
>> >> 
>> >> A strange segfault with EPERM.
>> >> 
>> >> The target is mips2 bigendian router, specifically it is ar9331
>SoC.
>> >> This is how file reports on libc.so: ELF 32-bit MSB shared object,
>> >> MIPS, MIPS-I version 1 (SYSV), dynamically linked, not stripped
>
>> >Could you attach the output of readelf -a libc.so? I'm guessing the
>> The output of readelf attached.
>
>i don't see the softfloat symbols of libgcc there
>(eg __addsf3)
>
>(musl malloc uses float arithmetics so ./libc.so will fail
>immediately)
>
>(openwrt kernel can be built with fpu emulation which is
>the default on normal mips kernels)

Strange, I have already fpu emulator enabled (Algorithmics/MIPS FPU emulator 1.5 that comes with oprnwrt kernel). Btw without it even static userspace faults.
Landley toolchain "broken" here too, but I don't know is it softfloat or not.
Static userspace now works flawlessly, so I will investigate my toolchain.

Re: libc.so segfaults on mips

[Szabolcs Nagy]

* ???????? <lynx@sibserver.ru> [2014-09-04 22:03:58 +0800]:
> 4 ???????????????? 2014 ??. 20:32:09 KRAT, Szabolcs Nagy <nsz@port70.net> ??????????:
> >(openwrt kernel can be built with fpu emulation which is
> >the default on normal mips kernels)
> 
> Strange, I have already fpu emulator enabled (Algorithmics/MIPS FPU emulator 1.5 that comes with oprnwrt kernel). Btw without it even static userspace faults.
> Landley toolchain "broken" here too, but I don't know is it softfloat or not.
> Static userspace now works flawlessly, so I will investigate my toolchain.

if you already enabled fpu emu then it is not the softfloat issue

btw you can tell if your toolchain is sf by
gcc -dM -E - </dev/null |grep __mips_soft_float

Re: libc.so segfaults on mips

[Рысь]

4 сентября 2014 г. 22:45:43 KRAT, Szabolcs Nagy <nsz@port70.net> пишет:
>* ???????? <lynx@sibserver.ru> [2014-09-04 22:03:58 +0800]:
>> 4 ???????????????? 2014 ??. 20:32:09 KRAT, Szabolcs Nagy
><nsz@port70.net> ??????????:
>> >(openwrt kernel can be built with fpu emulation which is
>> >the default on normal mips kernels)
>> 
>> Strange, I have already fpu emulator enabled (Algorithmics/MIPS FPU
>emulator 1.5 that comes with oprnwrt kernel). Btw without it even
>static userspace faults.
>> Landley toolchain "broken" here too, but I don't know is it softfloat
>or not.
>> Static userspace now works flawlessly, so I will investigate my
>toolchain.
>
>if you already enabled fpu emu then it is not the softfloat issue
>
>btw you can tell if your toolchain is sf by
>gcc -dM -E - </dev/null |grep __mips_soft_float

#define __mips_soft_float 1

So it seems that it has sf (libgcc.a also has __addsf3 and such)

Re: libc.so segfaults on mips

[Rich Felker]

On Thu, Sep 04, 2014 at 04:56:40PM +0800, Рысь wrote:
> 
> 
> 4 сентября 2014 г. 15:35:45 KRAT, Rich Felker <dalias@libc.org> пишет:
> >On Thu, Sep 04, 2014 at 03:07:57PM +0800, Рысь wrote:
> >> Hi list!
> >> 
> >> Not a really blocking bug but I was wondering why cross compiled
> >> musl libc.so for embedded wrt system is segfaults with this:
> >> 
> >> % ./strace ./libc.so 
> >> execve("./libc.so", ["./libc.so"], [/* 10 vars */]) = 0
> >> --- {si_signo=SIGSEGV, si_code=SI_USER, si_errno=EPERM, si_addr=0x4}
> >(Segmentation fault) ---
> >> +++ killed by SIGSEGV +++
> >> Segmentation fault
> >> 
> >> A strange segfault with EPERM.
> >> 
> >> The target is mips2 bigendian router, specifically it is ar9331 SoC.
> >> This is how file reports on libc.so: ELF 32-bit MSB shared object,
> >> MIPS, MIPS-I version 1 (SYSV), dynamically linked, not stripped
> >> 
> >> Such a trouble I had once when system enforced mmap_min_addr,
> >> however there it has default value.
> >> 
> >> The toolchain is somewhat old: gcc 4.2.1, binutils 2.17.50.0.17 but
> >> confirmed working and stable.
> >> 
> >> Musl is 1.1.4.
> >> 
> >> What else can cause this?
> >
> >Could you attach the output of readelf -a libc.so? I'm guessing the
> >problem is a broken linker that's accepting the option but not
> >actually honoring -Bsymbolic-functions, or another toolchain related
> >issue.
> >
> >Rich
> 
> Ugh, some things changed because of removal of local patches, but issue still same. Segfault line for unmodified musl:
> 
> --- {si_signo=SIGSEGV, si_code=SI_USER, si_errno=EPERM, si_addr=0x7b2a0} (Segmentation fault) ---
> 
> The output of readelf attached.

This output looks incomplete. MIPS readelf -a output should end with a
large section that starts with:

Primary GOT:
 Canonical gp value: 000b2f90

followed by

 Local entries:
  ...

 Global entries:
  ...

This is where I would be able to tell if something is wrong. Do you
perhaps have an ancient version of readelf? If you're using the native
one that came with your mips toolchain, try the host readelf instead
from a normal system.

Rich

Re: libc.so segfaults on mips

[Рысь]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]



4 сентября 2014 г. 22:57:01 KRAT, Rich Felker <dalias@libc.org> пишет:

>This output looks incomplete. MIPS readelf -a output should end with a
>large section that starts with:
>
>Primary GOT:
> Canonical gp value: 000b2f90
>
>followed by
>
> Local entries:
>  ...
>
> Global entries:
>  ...
>
>This is where I would be able to tell if something is wrong. Do you
>perhaps have an ancient version of readelf? If you're using the native
>one that came with your mips toolchain, try the host readelf instead
>from a normal system.
>
>Rich

Yes, it is. The host one is same and generates same output. Had to download file to another machine with binutils 2.21, it is attached. (Or if you want to play with it, here is a compiled libc.so: http://lynxlynx.tk/lynx/libc.so)

[-- Attachment #2: libc.so.relf.gz --]
[-- Type: application/x-gzip, Size: 71736 bytes --]

Re: libc.so segfaults on mips

[Rich Felker]

On Thu, Sep 04, 2014 at 11:20:32PM +0800, Рысь wrote:
> 
> 
> 4 сентября 2014 г. 22:57:01 KRAT, Rich Felker <dalias@libc.org> пишет:
> 
> >This output looks incomplete. MIPS readelf -a output should end with a
> >large section that starts with:
> >
> >Primary GOT:
> > Canonical gp value: 000b2f90
> >
> >followed by
> >
> > Local entries:
> >  ...
> >
> > Global entries:
> >  ...
> >
> >This is where I would be able to tell if something is wrong. Do you
> >perhaps have an ancient version of readelf? If you're using the native
> >one that came with your mips toolchain, try the host readelf instead
> >from a normal system.
> >
> >Rich
> 
> Yes, it is. The host one is same and generates same output. Had to
> download file to another machine with binutils 2.21, it is attached.
> (Or if you want to play with it, here is a compiled libc.so:
> http://lynxlynx.tk/lynx/libc.so)

I can tell you from reading that readelf output that
-Bsymbolic-functions was ignored by the linker (or it was patched out
of your musl build entirely?). For a correctly linked libc.so, the
local entries list should be huge and the global entries list should
be very small, referring only to global data symbols, not functions.

In any case, that's your problem. You need a linker that respects
-Bsymbolic-functions to build musl. The configure script tests for
accepting the option, but I don't have a way to test whether it's
broken at this time.

Rich

Re: libc.so segfaults on mips

[Rob Landley]

On Thu, Sep 4, 2014 at 10:58 AM, Rich Felker <dalias@libc.org> wrote:
> On Thu, Sep 04, 2014 at 11:20:32PM +0800, Рысь wrote:
>>
>>
>> 4 сентября 2014 г. 22:57:01 KRAT, Rich Felker <dalias@libc.org> пишет:
>>
>> >This output looks incomplete. MIPS readelf -a output should end with a
>> >large section that starts with:
>> >
>> >Primary GOT:
>> > Canonical gp value: 000b2f90
>> >
>> >followed by
>> >
>> > Local entries:
>> >  ...
>> >
>> > Global entries:
>> >  ...
>> >
>> >This is where I would be able to tell if something is wrong. Do you
>> >perhaps have an ancient version of readelf? If you're using the native
>> >one that came with your mips toolchain, try the host readelf instead
>> >from a normal system.
>> >
>> >Rich
>>
>> Yes, it is. The host one is same and generates same output. Had to
>> download file to another machine with binutils 2.21, it is attached.
>> (Or if you want to play with it, here is a compiled libc.so:
>> http://lynxlynx.tk/lynx/libc.so)
>
> I can tell you from reading that readelf output that
> -Bsymbolic-functions was ignored by the linker (or it was patched out
> of your musl build entirely?). For a correctly linked libc.so, the
> local entries list should be huge and the global entries list should
> be very small, referring only to global data symbols, not functions.

I have the same problem in my aboriginal linux toolchain, which is
also the last gplv2 releases of gcc and binutils. (It's not exaclty
being ignored, but it's not working right either.)

Fixing it is on my todo list, but at the moment I'm trying to get a
release out with the new wrapper and uClibc (and linux 3.15) to just
fix part of the problem and have the release version less behind.
(Fixing this shouldn't be all that _hard_, it's just out of scope for
this release which is _already_ hugely overdue...)

Rob

Re: libc.so segfaults on mips

[Laurent Bercot]

On 04/09/2014 18:17, Rob Landley wrote:
> at the moment I'm trying to get a release out with the new wrapper and uClibc

  Rob, would you have a very vague ETA for an Aboriginal Linux release
with musl ? (I'm mostly interested in the toolchains, but I guess you
won't release them without the rest of the project.)

  If porting it to musl is hard, can you share what the main points of
contention are, what still needs work, and if there are things that
can be done on the musl side to help ?

-- 
  Laurent

Re: libc.so segfaults on mips

[Rich Felker]

On Thu, Sep 04, 2014 at 06:38:10PM +0100, Laurent Bercot wrote:
> On 04/09/2014 18:17, Rob Landley wrote:
> >at the moment I'm trying to get a release out with the new wrapper and uClibc
> 
>  Rob, would you have a very vague ETA for an Aboriginal Linux release
> with musl ? (I'm mostly interested in the toolchains, but I guess you
> won't release them without the rest of the project.)

As I understand, the main problem right now is that the toolchains are
insufficient for compiling musl, at least on some archs (see: broken
-Bsymbolic-functions).

Rich

Re: libc.so segfaults on mips

[Рысь]

5 сентября 2014 г. 1:17:57 KRAT, Rob Landley <rob@landley.net> пишет:

>I have the same problem in my aboriginal linux toolchain, which is
>also the last gplv2 releases of gcc and binutils. (It's not exaclty
>being ignored, but it's not working right either.)
>
>Fixing it is on my todo list, but at the moment I'm trying to get a
>release out with the new wrapper and uClibc (and linux 3.15) to just
>fix part of the problem and have the release version less behind.
>(Fixing this shouldn't be all that _hard_, it's just out of scope for
>this release which is _already_ hugely overdue...)
>
>Rob

Ok then, I will pickup your fix when it will be ready, not a problem from now.

Thanks to anyone replied!

Re: libc.so segfaults on mips

[Isaac Dunham]

On Thu, Sep 04, 2014 at 01:41:54PM -0400, Rich Felker wrote:
> On Thu, Sep 04, 2014 at 06:38:10PM +0100, Laurent Bercot wrote:
> > On 04/09/2014 18:17, Rob Landley wrote:
> > >at the moment I'm trying to get a release out with the new wrapper and uClibc
> > 
> >  Rob, would you have a very vague ETA for an Aboriginal Linux release
> > with musl ? (I'm mostly interested in the toolchains, but I guess you
> > won't release them without the rest of the project.)
> 
> As I understand, the main problem right now is that the toolchains are
> insufficient for compiling musl, at least on some archs (see: broken
> -Bsymbolic-functions).
> 

FYI, I've been using binutils at the same version as Rob and a mix of
gcc 3.4 and gcc 4.2 with some patches from Aboriginal on x86.
The resulting libc works; I can provide the readelf output if you're
interested.

Thanks,
Isaac Dunham

Re: libc.so segfaults on mips

[Rob Landley]

On Thu, Sep 4, 2014 at 12:41 PM, Rich Felker <dalias@libc.org> wrote:
> On Thu, Sep 04, 2014 at 06:38:10PM +0100, Laurent Bercot wrote:
>> On 04/09/2014 18:17, Rob Landley wrote:
>> >at the moment I'm trying to get a release out with the new wrapper and uClibc
>>
>>  Rob, would you have a very vague ETA for an Aboriginal Linux release
>> with musl ? (I'm mostly interested in the toolchains, but I guess you
>> won't release them without the rest of the project.)
>
> As I understand, the main problem right now is that the toolchains are
> insufficient for compiling musl, at least on some archs (see: broken
> -Bsymbolic-functions).
>
> Rich

That's the big one, but there are a few others. Huh, I haven't rsynced
http://landley.net/notes.html in a while, have I? [does that, yay
phone tethering]

September 3, august 25, august 17, august 16, all talk about me trying
to fix stuff in aboriginal. Not necessarily directly musl related, but
mostly fallout from switching build infrastructure over to
musl-friendly versions. The ccwrap rewrite was _fiddly_.

Yeah, it's going slowly. Attack of life. My $DAYJOB has been the most
time consuming part (nothing to do with anything I do in open source).
But also since my last release: I gave myself heatstroke at texas
linuxfest (did you know you can wind up sick for a month after that?),
my house got flooded (followed by professional disaster mitigation
people and construction guys replacing all the floors and baseboards).
The house's air conditioner stopped working twice (more or less
overwhelmed by humidity from the flooding and running constantly to
offset the fans and giant dehumidifiers the disaster recovery people
brought in), the car's air conditioner essentially exploded
(unrelated! but expensive), the kitchen sink broke and mildly flooded
the kitchen (different flooding!) and is now leaking again for yet
another unrelated reason (I hate garbage disposals), we got a new
kitten named Zabina (she attacks feet at 3am), and the dog died (out
of the blue, previously unknown tumor ruptured suddenly) and so my
wife got a new dog (which is why I got 5 hours of sleep last night).

On top of all that, the vast majority of my open source programming
time has been going into toybox (ala http://landley.net/hg/toybox), so
aboriginal bubbles to the top of the todo list when I need a test
harness for things that only run as root (such as mount and useradd)
because I dowanna test them directly on my development system. (Also
things like "rm -rf", testing _that_ was done under aboriginal, not on
the host.)

Rob

P.S. the http://landley.net/notes-2014.html#14-08-2014 entry is kind
of entertaining from a musl perspective. But then the fact the new
toybox mount wouldn't compile against uClibc because they're missing
all the new mount symbols since MS_BIND which went into the tree in
2.5.0.4 circa 2001 (_eleven_years_ before their last-ever release in
2012)... That was sad. I'm tempted to post to the uClibc list and ask
them to just stop, because it's not helping anymore.

(This mount command built against musl out of the box, of course,
using the musl wrapper on the host toolchain. I do regression test
that a bit as I go. It doesn't _work_ right yet (see "testing",
above), but it _compiled_...)