debugging problem with musl ld and qemu-ppc

debugging problem with musl ld and qemu-ppc

[Felix Janda]

Hello list,

can you maybe help me debugging the following problem with qemu-ppc?
It gives an invalid instruction error after doing:

tar -xf crossx86-powerpc-linux-musl-0.9.9.tar.xz
# (I get something similar for other versions of musl or gcc)
cat > a.s <<EOF
b	_GLOBAL_OFFSET_TABLE_@local
EOF
cat > b.c <<EOF
int main(void) { return 0; }
EOF
./powerpc-linux-musl/bin/powerpc-linux-musl-gcc -o good b.c
./powerpc-linux-musl/bin/powerpc-linux-musl-gcc -o bad a.s b.c
# set up symlinks to make 'qemu-ppc good' work
qemu-ppc bad

After musl ld has done its work it jumps to libgcc's _init and
very soon tries to execute some data.

When executing in some chroot the 'qemu-ppc -d in_asm' for
good and bad is exactly the same up to two instructions in
libgcc until it diverges (and bad crashes).

Do you have any ideas what could be the problem?

Thanks,
Felix

Re: debugging problem with musl ld and qemu-ppc

[Rich Felker]

On Thu, Oct 16, 2014 at 08:09:01AM +0200, Felix Janda wrote:
> Hello list,
> 
> can you maybe help me debugging the following problem with qemu-ppc?
> It gives an invalid instruction error after doing:
> 
> tar -xf crossx86-powerpc-linux-musl-0.9.9.tar.xz
> # (I get something similar for other versions of musl or gcc)
> cat > a.s <<EOF
> b	_GLOBAL_OFFSET_TABLE_@local
> EOF
> cat > b.c <<EOF
> int main(void) { return 0; }
> EOF
> ../powerpc-linux-musl/bin/powerpc-linux-musl-gcc -o good b.c
> ../powerpc-linux-musl/bin/powerpc-linux-musl-gcc -o bad a.s b.c
> # set up symlinks to make 'qemu-ppc good' work
> qemu-ppc bad
> 
> After musl ld has done its work it jumps to libgcc's _init and
> very soon tries to execute some data.
> 
> When executing in some chroot the 'qemu-ppc -d in_asm' for
> good and bad is exactly the same up to two instructions in
> libgcc until it diverges (and bad crashes).
> 
> Do you have any ideas what could be the problem?

Using the cross compiler I had lying around, I get a warning while
linking "bad":

/opt/powerpc-linux-musl/bin/../lib/gcc/powerpc-linux-musl/4.7.2/../../../../powerpc-linux-musl/bin/ld: bss-plt forced due to /tmp/ccfPiE1t.o

I suspect this is happening for you too (possibly without a warning
being printed?) and that this is the cause of the problem: something
about your asm file is forcing the linker to use the old "bss-plt"
dynamic linking model, which musl does not support, rather than the
"secure-plt" model.

The "bss-plt" model requires the dynamic linker to generate direct
call instructions inline in a writable (mode rwx!) page rather than
having the PLT thunks load their actual function addresses from
pointers filled into the GOT by the dynamic linker, so it's a major
risk from a security standpoint, and also a lot more work to implement
in the dynamic linker. As such, when powerpc support was added we
opted to omit this model and push for use of the more secure model
that works like every other arch.

What I'm not clear about is the cause for why the linker is forcing
you back to the bss-plt model. It might be a matter of the strange
relocation type you put in a.s:

00000000 <.text>:
   0:   48 00 00 00     b       0x0
                           0: R_PPC_LOCAL24PC      _GLOBAL_OFFSET_TABLE_

Let's see if anyone else has ideas.

Rich

Re: debugging problem with musl ld and qemu-ppc

[Felix Janda]

Rich Felker wrote:
> On Thu, Oct 16, 2014 at 08:09:01AM +0200, Felix Janda wrote:
> > Hello list,
> > 
> > can you maybe help me debugging the following problem with qemu-ppc?
> > It gives an invalid instruction error after doing:
> > 
> > tar -xf crossx86-powerpc-linux-musl-0.9.9.tar.xz
> > # (I get something similar for other versions of musl or gcc)
> > cat > a.s <<EOF
> > b	_GLOBAL_OFFSET_TABLE_@local
> > EOF
> > cat > b.c <<EOF
> > int main(void) { return 0; }
> > EOF
> > ../powerpc-linux-musl/bin/powerpc-linux-musl-gcc -o good b.c
> > ../powerpc-linux-musl/bin/powerpc-linux-musl-gcc -o bad a.s b.c
> > # set up symlinks to make 'qemu-ppc good' work
> > qemu-ppc bad
> > 
> > After musl ld has done its work it jumps to libgcc's _init and
> > very soon tries to execute some data.
> > 
> > When executing in some chroot the 'qemu-ppc -d in_asm' for
> > good and bad is exactly the same up to two instructions in
> > libgcc until it diverges (and bad crashes).
> > 
> > Do you have any ideas what could be the problem?
> 
> Using the cross compiler I had lying around, I get a warning while
> linking "bad":
> 
> /opt/powerpc-linux-musl/bin/../lib/gcc/powerpc-linux-musl/4.7.2/../../../../powerpc-linux-musl/bin/ld: bss-plt forced due to /tmp/ccfPiE1t.o
> 
> I suspect this is happening for you too (possibly without a warning
> being printed?) and that this is the cause of the problem: something
> about your asm file is forcing the linker to use the old "bss-plt"
> dynamic linking model, which musl does not support, rather than the
> "secure-plt" model.

Thanks for reproducing the issue and explaining what is bss-plt.
Depending on the compiler I also got this warning. Sorry for forgetting
to mention this.

> The "bss-plt" model requires the dynamic linker to generate direct
> call instructions inline in a writable (mode rwx!) page rather than
> having the PLT thunks load their actual function addresses from
> pointers filled into the GOT by the dynamic linker, so it's a major
> risk from a security standpoint, and also a lot more work to implement
> in the dynamic linker. As such, when powerpc support was added we
> opted to omit this model and push for use of the more secure model
> that works like every other arch.
> 
> What I'm not clear about is the cause for why the linker is forcing
> you back to the bss-plt model. It might be a matter of the strange
> relocation type you put in a.s:
> 
> 00000000 <.text>:
>    0:   48 00 00 00     b       0x0
>                            0: R_PPC_LOCAL24PC      _GLOBAL_OFFSET_TABLE_

The problem caused a self-compiled native gcc to crash because the file
"mpn/powerpc32/elf.m4" (used to generate some assembly) in the gmp tree 
contains a similar instruction.

I've found the part in binutils emitting the warning and will try to see
why it forces bss-plt.

Thanks,
Felix

Re: debugging problem with musl ld and qemu-ppc

[Felix Janda]

Felix Janda wrote:
> Rich Felker wrote:
[..]
> > What I'm not clear about is the cause for why the linker is forcing
> > you back to the bss-plt model. It might be a matter of the strange
> > relocation type you put in a.s:
> > 
> > 00000000 <.text>:
> >    0:   48 00 00 00     b       0x0
> >                            0: R_PPC_LOCAL24PC      _GLOBAL_OFFSET_TABLE_
> 
> The problem caused a self-compiled native gcc to crash because the file
> "mpn/powerpc32/elf.m4" (used to generate some assembly) in the gmp tree 
> contains a similar instruction.
> 
> I've found the part in binutils emitting the warning and will try to see
> why it forces bss-plt.

This seems to be caused by the part starting from lines 4267 in [1]

	  /* This refers only to functions defined in the shared library.  */
	case R_PPC_LOCAL24PC:
	  if (h != NULL && h == htab->elf.hgot && htab->plt_type == PLT_UNSET)
	    {
	      htab->plt_type = PLT_OLD;
	      htab->old_bfd = abfd;
	    }

I think it was added to be helpful and detect the construction

bl _GLOBAL_OFFSET_TABLE_@local-4
mflr r30

intended to load a pointer to the got into r30, which no longer works
with secure-plt. See [2].

Felix

[1]: http://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob;f=bfd/elf32-ppc.c;h=04c2d6ad60b4122eb1727e932cfcda035feeac68;hb=HEAD
[2]: http://www.sourceware.org/ml/binutils/2005-05/txt00011.txt

Re: debugging problem with musl ld and qemu-ppc

[Rich Felker]

On Sun, Oct 19, 2014 at 10:29:35PM +0200, Felix Janda wrote:
> Felix Janda wrote:
> > Rich Felker wrote:
> [..]
> > > What I'm not clear about is the cause for why the linker is forcing
> > > you back to the bss-plt model. It might be a matter of the strange
> > > relocation type you put in a.s:
> > > 
> > > 00000000 <.text>:
> > >    0:   48 00 00 00     b       0x0
> > >                            0: R_PPC_LOCAL24PC      _GLOBAL_OFFSET_TABLE_
> > 
> > The problem caused a self-compiled native gcc to crash because the file
> > "mpn/powerpc32/elf.m4" (used to generate some assembly) in the gmp tree 
> > contains a similar instruction.
> > 
> > I've found the part in binutils emitting the warning and will try to see
> > why it forces bss-plt.
> 
> This seems to be caused by the part starting from lines 4267 in [1]
> 
> 	  /* This refers only to functions defined in the shared library.  */
> 	case R_PPC_LOCAL24PC:
> 	  if (h != NULL && h == htab->elf.hgot && htab->plt_type == PLT_UNSET)
> 	    {
> 	      htab->plt_type = PLT_OLD;
> 	      htab->old_bfd = abfd;
> 	    }
> 
> I think it was added to be helpful and detect the construction
> 
> bl _GLOBAL_OFFSET_TABLE_@local-4
> mflr r30
> 
> intended to load a pointer to the got into r30, which no longer works
> with secure-plt. See [2].

Nice job tracking this down. If there's not a reason you need this
specific construct, I would just avoid it, but really the above hack
should be removed from binutils. There are all sorts of asm constructs
that are specific to one ABI, and the presence of something that seems
to be specific to one ABI is not a justification for ignoring the
user's choice of link options.

Rich

Re: debugging problem with musl ld and qemu-ppc

[Felix Janda]

Rich Felker wrote:
> On Sun, Oct 19, 2014 at 10:29:35PM +0200, Felix Janda wrote:
[..]
> > This seems to be caused by the part starting from lines 4267 in [1]
> > 
> > 	  /* This refers only to functions defined in the shared library.  */
> > 	case R_PPC_LOCAL24PC:
> > 	  if (h != NULL && h == htab->elf.hgot && htab->plt_type == PLT_UNSET)
> > 	    {
> > 	      htab->plt_type = PLT_OLD;
> > 	      htab->old_bfd = abfd;
> > 	    }
> > 
> > I think it was added to be helpful and detect the construction
> > 
> > bl _GLOBAL_OFFSET_TABLE_@local-4
> > mflr r30
> > 
> > intended to load a pointer to the got into r30, which no longer works
> > with secure-plt. See [2].
> 
> Nice job tracking this down. If there's not a reason you need this
> specific construct, I would just avoid it, but really the above hack
> should be removed from binutils. There are all sorts of asm constructs
> that are specific to one ABI, and the presence of something that seems
> to be specific to one ABI is not a justification for ignoring the
> user's choice of link options.

The construct is used in a file in gmp and this caused my home-built gcc
to crash. I'm wondering why other people have not hit this issue. (I
guess it is because they didn't use dynamic linking.)

Using debian code search, the construct still seems to be used in
libffi (and its copies in many projects). Looking closer, the code is
behind an #ifdef PROF and PROF isn't defined anywhere in the codebase.
That should be the reason why I had no problems with python.

Felix

Re: debugging problem with musl ld and qemu-ppc

[Rich Felker]

On Sat, Nov 01, 2014 at 10:41:28PM +0100, Felix Janda wrote:
> Rich Felker wrote:
> > On Sun, Oct 19, 2014 at 10:29:35PM +0200, Felix Janda wrote:
> [..]
> > > This seems to be caused by the part starting from lines 4267 in [1]
> > > 
> > > 	  /* This refers only to functions defined in the shared library.  */
> > > 	case R_PPC_LOCAL24PC:
> > > 	  if (h != NULL && h == htab->elf.hgot && htab->plt_type == PLT_UNSET)
> > > 	    {
> > > 	      htab->plt_type = PLT_OLD;
> > > 	      htab->old_bfd = abfd;
> > > 	    }
> > > 
> > > I think it was added to be helpful and detect the construction
> > > 
> > > bl _GLOBAL_OFFSET_TABLE_@local-4
> > > mflr r30
> > > 
> > > intended to load a pointer to the got into r30, which no longer works
> > > with secure-plt. See [2].
> > 
> > Nice job tracking this down. If there's not a reason you need this
> > specific construct, I would just avoid it, but really the above hack
> > should be removed from binutils. There are all sorts of asm constructs
> > that are specific to one ABI, and the presence of something that seems
> > to be specific to one ABI is not a justification for ignoring the
> > user's choice of link options.
> 
> The construct is used in a file in gmp and this caused my home-built gcc
> to crash. I'm wondering why other people have not hit this issue. (I
> guess it is because they didn't use dynamic linking.)

Could you give a link to the code where it's used? It would be nice to
see if this is actually something broken that's assuming the
insecure-plt ABI or whether it's just binutils misinterpreting it and
making trouble for us.

Rich

Re: debugging problem with musl ld and qemu-ppc

[Felix Janda]

Rich Felker wrote:
> On Sat, Nov 01, 2014 at 10:41:28PM +0100, Felix Janda wrote:
[..]
> > The construct is used in a file in gmp and this caused my home-built gcc
> > to crash. I'm wondering why other people have not hit this issue. (I
> > guess it is because they didn't use dynamic linking.)
> 
> Could you give a link to the code where it's used? It would be nice to
> see if this is actually something broken that's assuming the
> insecure-plt ABI or whether it's just binutils misinterpreting it and
> making trouble for us.

See

https://gmplib.org/repo/gmp/file/bfc8adc7cdfe/mpn/powerpc32/elf.m4
https://gmplib.org/list-archives/gmp-bugs/2014-November/003568.html

Felix

Re: debugging problem with musl ld and qemu-ppc

[stephen Turner]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

Would this issue affect all programs not just the rebuilding of gcc? I have
been having a lot of problems with one lib or another failing when trying
to rebuild code on my static musl gcc system (building both dynamic and
static)

thanks,
stephen

On Sat, Nov 1, 2014 at 6:01 PM, Felix Janda <felix.janda@posteo.de> wrote:

> Rich Felker wrote:
> > On Sat, Nov 01, 2014 at 10:41:28PM +0100, Felix Janda wrote:
> [..]
> > > The construct is used in a file in gmp and this caused my home-built
> gcc
> > > to crash. I'm wondering why other people have not hit this issue. (I
> > > guess it is because they didn't use dynamic linking.)
> >
> > Could you give a link to the code where it's used? It would be nice to
> > see if this is actually something broken that's assuming the
> > insecure-plt ABI or whether it's just binutils misinterpreting it and
> > making trouble for us.
>
> See
>
> https://gmplib.org/repo/gmp/file/bfc8adc7cdfe/mpn/powerpc32/elf.m4
> https://gmplib.org/list-archives/gmp-bugs/2014-November/003568.html
>
> Felix
>

[-- Attachment #2: Type: text/html, Size: 1776 bytes --]

Re: debugging problem with musl ld and qemu-ppc

[Felix Janda]

stephen Turner wrote:
> Would this issue affect all programs not just the rebuilding of gcc? I have
> been having a lot of problems with one lib or another failing when trying
> to rebuild code on my static musl gcc system (building both dynamic and
> static)

My particular example only applies to programs linked dynamically
against gmp (configured with --enable-assembly). More generally it
affects all programs dynamically linked to libraries containing
relocations which binutils thinks of as requiring bss-plt. (The
programs could also contain the relocation themselves.) I have not
encountered this for anything other than gmp, yet.

Felix

Re: debugging problem with musl ld and qemu-ppc

[Felix Janda]

Rich Felker wrote:
[..]
> 
> What I'm not clear about is the cause for why the linker is forcing
> you back to the bss-plt model. It might be a matter of the strange
> relocation type you put in a.s:
> 
> 00000000 <.text>:
>    0:   48 00 00 00     b       0x0
>                            0: R_PPC_LOCAL24PC      _GLOBAL_OFFSET_TABLE_

It also seems to be possible to get this kind of relocation by doing

cat > a.c <<EOF
extern int __attribute__ ((__visibility__ ("hidden"))) f(void);
int main(void) { return f(); }
EOF
gcc -c a.c

Felix

Re: debugging problem with musl ld and qemu-ppc

[Rich Felker]

On Fri, Oct 17, 2014 at 09:30:39PM +0200, Felix Janda wrote:
> Rich Felker wrote:
> [..]
> > 
> > What I'm not clear about is the cause for why the linker is forcing
> > you back to the bss-plt model. It might be a matter of the strange
> > relocation type you put in a.s:
> > 
> > 00000000 <.text>:
> >    0:   48 00 00 00     b       0x0
> >                            0: R_PPC_LOCAL24PC      _GLOBAL_OFFSET_TABLE_
> 
> It also seems to be possible to get this kind of relocation by doing
> 
> cat > a.c <<EOF
> extern int __attribute__ ((__visibility__ ("hidden"))) f(void);
> int main(void) { return f(); }
> EOF
> gcc -c a.c

This should not happen if gcc is built to use secure-plt mode. gcc
should use whatever relocation types work for it. If it does, it's a
bug in some part of the toolchain. Really I don't get why ld rejects
the relocation. It seems perfectly valid to me.

Rich

Re: debugging problem with musl ld and qemu-ppc

[Felix Janda]

Rich Felker wrote:
> On Fri, Oct 17, 2014 at 09:30:39PM +0200, Felix Janda wrote:
> > Rich Felker wrote:
> > [..]
> > > 
> > > What I'm not clear about is the cause for why the linker is forcing
> > > you back to the bss-plt model. It might be a matter of the strange
> > > relocation type you put in a.s:
> > > 
> > > 00000000 <.text>:
> > >    0:   48 00 00 00     b       0x0
> > >                            0: R_PPC_LOCAL24PC      _GLOBAL_OFFSET_TABLE_
> > 
> > It also seems to be possible to get this kind of relocation by doing
> > 
> > cat > a.c <<EOF
> > extern int __attribute__ ((__visibility__ ("hidden"))) f(void);
> > int main(void) { return f(); }
> > EOF
> > gcc -c a.c
> 
> This should not happen if gcc is built to use secure-plt mode. gcc
> should use whatever relocation types work for it. If it does, it's a
> bug in some part of the toolchain. Really I don't get why ld rejects
> the relocation. It seems perfectly valid to me.

Here indeed everything is ok. While a.o will have a R_PPC_LOCAL24PC
relocation it is resolved without falling back to bss-plt.

Felix