Re: possible bug in setjmp implementation for ppc64

[David Edelsohn <dje.gcc@gmail.com>]

On Tue, Aug 1, 2017 at 1:10 AM, Bobby Bingham <koorogi@koorogi.info> wrote:
> On Mon, Jul 31, 2017 at 04:30:07PM -0400, Rich Felker wrote:
>> On Mon, Jul 31, 2017 at 10:06:51PM +0200, felix.winkelmann@bevuta.com wrote:
>> > Hi!
>> >
>> > I think I may have come across a bug in musl on PPC64(le), and the folks
>> > on the #musl IRC channel directed me here. I'm not totally sure whether
>> > the problem is caused by a my misunderstanding of C library functions or whether
>> > it is a plain bug in the musl implementation of setjmp(3).
>> >
>> > In out project[1] we use setjmp to establish a global trampoline
>> > and allocate small objects on the stack using alloca (see [2] for
>> > more information about the compiliation strategy used). I was able to reduce
>> > the code that crashes to the following:
>> >
>> > ---
>> > #include <stdio.h>
>> > #include <alloca.h>
>> > #include <setjmp.h>
>> > #include <string.h>
>> > #include <stdlib.h>
>> >
>> > jmp_buf jb;
>> >
>> > int foo = 99;
>> > int c = 0;
>> >
>> > void bar()
>> > {
>> >   c++;
>> >   longjmp(jb, 1);
>> > }
>> >
>> > int main()
>> > {
>> >   setjmp(jb);
>> >   char *p = alloca(256);
>> >   memset(p, 0, 256);
>> >   printf("%d\n", foo);
>> >
>> >   if(c < 10) bar();
>> >
>> >   exit(0);
>> > }
>> > ---
>> >
>> > When executing the longjmp, the code that restores $r2 (TOC) after the call
>> > to setjmp reads invalid data, because the memset apparently clobbered
>> > the stack frame - i.e. the pointer returned be alloca points into a part
>> > of the stack frame that is still in use.
>> >
>> > I tried this on arm, x86_64 and ppc64 with glibc and it seems to work fine,
>> > but crashes when linked with musl (running Alpine Linux on a VM)
>> >
>> > If you need more information, please feel free to ask. You can also keep
>> > me CC'd, since I'd be interested in knowing more about the details.
>>
>> It looks to me like we have a bug here, but it's one where I or
>> someone else needs to read and understand the PPC64 ELFv2 ABI document
>> to fully understand what's going on and make a fix. I'll try to get to
>> it soon, or I'm happy if someone else wants to. I don't just want to
>> cargo-cult whatever glibc is doing, though; a fix should be
>> accompanied by an understanding of why it's right.
>
> I think I can explain what's happening.
>
> The TOC pointer is constant within a given dynamic module (the main
> executable or a library), but needs to be adjusted at cross-module
> calls.  Each function has two entry points in the ELFv2 ABI.  The entry
> point for intra-module calls can assume r2 is already set up correctly.
> The entry point for inter-module calls starts two instructions earlier
> and adjusts r2 before falling through to the intra-module entry point.
>
> Normally, r2 is supposed to be preserved across calls.  For intra-module
> calls, there's no problem.  For inter-module calls, the PLT stub saves
> the caller's r2 value to a slot in the caller's stack frame that's
> required to be reserved for it, at r1+24.  The linker then inserts code
> in the caller to restore the value from the stack immediately after the
> call.
>
> So what's happening here is that the value of r2 that setjmp saves and
> that longjmp restores is the TOC pointer for libc, as set up by the PLT
> stub.  It's not the value of r2 that the caller had.  But that's
> normally fine -- after the second return from setjmp, the caller will
> restore its TOC pointer from the stack where it had been saved by the
> PLT stub when it originally called setjmp.  But in this example, gcc
> decides to allocate the 256 bytes overtop the part of the stack where
> the setjmp PLT stub had saved the TOC pointer, so it gets clobbered.
>
> The problem is that static linking and dynamic linking need to work
> differently.  With dynamic linking, we can fix this by changing setjmp
> to read the caller's TOC pointer from the reserved slot in the caller's
> stack frame, and longjmp to restore it to the stack instead of to r2.
>
> But with static linking, there's no PLT stub or code added by the linker
> to restore the TOC pointer from the stack, so we need to save/restore
> from/to r2, not the TOC slot in the caller's stack from.
>
> I think this either requires having different versions of setjmp/longjmp
> for static and dynamic libc, or to increase the size of jmpbuf so we can
> always save/restore both r2 and the value on the stack, but this would
> be an ABI change.

The analysis is correct.  Quoting my colleague:

"If glibc is built as a static library, the contents of r2 are saved
in the jmp_buf; but if glibc is built as a dynamic library, the
contents of the TOC save slot is saved in the jmp_buf.   Similarly, if
glibc is built as a dynamic library, longjmp *updates* the TOC save
slot with the r2 value from the jmp_buf before returning."

GLIBC setjmp/longjmp code explicitly differs for shared and static
versions of the library.  Musl libc needs equivalent functionality in
its implementation.

Thanks, David