Re: [9fans] OS X threads + dynamic linking

Re: [9fans] OS X threads + dynamic linking

[quanstro]

what is the backtrace?. is it possible to link against /lib/libc.a (or whatever apple calls it?)

- erik

[9fans] OS X threads + dynamic linking

[Russ Cox]

[The only claim this has to being on topic is that
I need an answer to make libthread work on OS X x86.
That and it has to do with dynamic linking.  Sorry.]

In the gcc program below, the function runonstack uses
some carefully crafted inline assembly to run the function fn
on the given stack (the pointer passed in is the top of the stack).
Because of the use of assembly, it is an x86-only program.

It works great on Linux and on FreeBSD.  On OS X, though,
the function running on the alternate stack dies when it
tries to call printf.  A gdb session running the program
is shown after the program.

It's dying in the dynamic linker trying to resolve printf.
If I call printf in main before calling runonstack, then printf
no longer fails.  However, if I then call exit(0) inside hello,
then that fails, because exit isn't resolved yet.

Thus, it appears that somehow the x86 OS X library routines
cannot handle being called from an alternate stack.
This just cannot be true.  I thought everyone had already
made that mistake and moved on.

If anyone can either confirm that this program has no hope
of working on OS X (that would be unfortunate) or can tell
me what I need to do to make it work (that would be better!),
I'd greatly appreciate it.

As of right now, it appears that the plan9port CVS tree
builds just fine on x86 OS X except that all the threaded
programs crash due to this problem.

Thanks for any help.
Russ



#include <stdio.h>
#include <stdlib.h>

char stack[1048576];
char *state;

void
hello(void)
{
	state = "calling printf";
	printf("hello, world\n");
	state = "done with printf";
}

void
runonstack(void (*fn)(void), char *stack)
{
	state = "in assembly";
	asm(
		"pushal\n"
		"movl 8(%ebp), %ebx\n"	/* ebx = fn */
		"movl 12(%ebp), %eax\n"	/* ecx = stack */
		"xchgl %esp, %eax\n"
		"pushl %eax\n"
		"call *%ebx\n"
		"popl %esp\n"
		"popal\n"
	);
	state = "out of assembly";
}

int
main(int argc, char **argv)
{
	runonstack(hello, stack+sizeof stack);
}

---

$ gdb a.out
GNU gdb 6.1-20040303 (Apple version gdb-437) (Fri Jan 13 18:45:48 GMT 2006)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-apple-darwin"...Reading symbols for shared libraries .. done

(gdb) run
Starting program: /Users/rsc/a.out
Reading symbols for shared libraries . done

Program received signal EXC_BAD_INSTRUCTION, Illegal instruction/operand.
0x8fe136e4 in __dyld_stub_binding_helper_interface ()
(gdb) print (char*)state
$1 = 0x1fa8 "calling printf"
(gdb) quit
The program is running.  Exit anyway? (y or n) y
$

Re: [9fans] OS X threads + dynamic linking

[Russ Cox]

> what is the backtrace?.

the backtrace is at the bottom of the message.

> is it possible to link against /lib/libc.a (or whatever apple calls it?)

gcc -static complains that it doesn't know where -lcrt0.o [sic] is.

russ

Re: [9fans] OS X threads + dynamic linking

[Don Bailey]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> [The only claim this has to being on topic is that
> I need an answer to make libthread work on OS X x86.
> That and it has to do with dynamic linking.  Sorry.]
>

Just to be clear because you didn't state that this
code *is* working on MacOSX on PowerPC, is it?
Because it works for me. Just thought I'd mention
it. If you want it I'll post but it's pretty straight forward
powerpc assembly.

Don "north" Bailey


-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.0.6 (Build 6060)

iQA/AwUBREh6gl/Ie1ANMtLuEQJlKQCg6pRgqeC/8UcBD94WPbm7xaRQvbgAoN0j
LUlimsbmEXeIIgve7B3RXS+0
=wGg9
-----END PGP SIGNATURE-----

Re: [9fans] OS X threads + dynamic linking

[Don Bailey]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> Thanks, but I already have working context-switch
> code for OS X on PowerPC.  The actual context-switch
> code I'm using for OS X on x86 is more careful than
> the assembly in the test program I posted.
>

Good deal.

> The test program was intended only to be a minimal
> illustration of the problem.
>

Right, that was assumed. However, if you hadn't
done it on PowerPC I figured I might as well toss
some code your way if you didn't feel like mucking
about.

> The fact that it works on PowerPC is one of the reasons
> I was so surprised it doesn't work on x86.  But maybe
> they felt the small number of x86 registers justified
> using the high-order bits of the stack pointer as
> some kind of per-thread identifier.  Everyone else
> seems to have made that mistake too, though everyone
> else has corrected it.
>

Huh, interesting. I don't have access to an x86 OSX
so I didn't notice this. It's odd they haven't changed
their code base to solve the tid in a more elegant
fashion, as you say, like most other OSs.

Don "north" Bailey



-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.0.6 (Build 6060)

iQA/AwUBREiDJ1/Ie1ANMtLuEQKHBwCggMAxtvNoVcWubFIeBjR4zIEHQ28Anjiz
W2loMIbT6eb4Tuy/J3JWcI6J
=ttLY
-----END PGP SIGNATURE-----

Re: [9fans] OS X threads + dynamic linking

[Russ Cox]

> Just to be clear because you didn't state that this
> code *is* working on MacOSX on PowerPC, is it?
> Because it works for me. Just thought I'd mention
> it. If you want it I'll post but it's pretty straight forward
> powerpc assembly.

Thanks, but I already have working context-switch
code for OS X on PowerPC.  The actual context-switch
code I'm using for OS X on x86 is more careful than
the assembly in the test program I posted.

The test program was intended only to be a minimal
illustration of the problem.

The fact that it works on PowerPC is one of the reasons
I was so surprised it doesn't work on x86.  But maybe
they felt the small number of x86 registers justified
using the high-order bits of the stack pointer as
some kind of per-thread identifier.  Everyone else
seems to have made that mistake too, though everyone
else has corrected it.

Russ

Re: [9fans] OS X threads + dynamic linking

[David Leimbach]

Interestingly enough if I compile with different optimization options
this goes from an illegal instruciton error to a bus error.

On 4/20/06, Russ Cox <rsc@swtch.com> wrote:
> [The only claim this has to being on topic is that
> I need an answer to make libthread work on OS X x86.
> That and it has to do with dynamic linking.  Sorry.]
>
> In the gcc program below, the function runonstack uses
> some carefully crafted inline assembly to run the function fn
> on the given stack (the pointer passed in is the top of the stack).
> Because of the use of assembly, it is an x86-only program.
>
> It works great on Linux and on FreeBSD.  On OS X, though,
> the function running on the alternate stack dies when it
> tries to call printf.  A gdb session running the program
> is shown after the program.
>
> It's dying in the dynamic linker trying to resolve printf.
> If I call printf in main before calling runonstack, then printf
> no longer fails.  However, if I then call exit(0) inside hello,
> then that fails, because exit isn't resolved yet.
>
> Thus, it appears that somehow the x86 OS X library routines
> cannot handle being called from an alternate stack.
> This just cannot be true.  I thought everyone had already
> made that mistake and moved on.
>
> If anyone can either confirm that this program has no hope
> of working on OS X (that would be unfortunate) or can tell
> me what I need to do to make it work (that would be better!),
> I'd greatly appreciate it.
>
> As of right now, it appears that the plan9port CVS tree
> builds just fine on x86 OS X except that all the threaded
> programs crash due to this problem.
>
> Thanks for any help.
> Russ
>
>
>
> #include <stdio.h>
> #include <stdlib.h>
>
> char stack[1048576];
> char *state;
>
> void
> hello(void)
> {
>         state = "calling printf";
>         printf("hello, world\n");
>         state = "done with printf";
> }
>
> void
> runonstack(void (*fn)(void), char *stack)
> {
>         state = "in assembly";
>         asm(
>                 "pushal\n"
>                 "movl 8(%ebp), %ebx\n"  /* ebx = fn */
>                 "movl 12(%ebp), %eax\n" /* ecx = stack */
>                 "xchgl %esp, %eax\n"
>                 "pushl %eax\n"
>                 "call *%ebx\n"
>                 "popl %esp\n"
>                 "popal\n"
>         );
>         state = "out of assembly";
> }
>
> int
> main(int argc, char **argv)
> {
>         runonstack(hello, stack+sizeof stack);
> }
>
> ---
>
> $ gdb a.out
> GNU gdb 6.1-20040303 (Apple version gdb-437) (Fri Jan 13 18:45:48 GMT 2006)
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "i386-apple-darwin"...Reading symbols for shared libraries .. done
>
> (gdb) run
> Starting program: /Users/rsc/a.out
> Reading symbols for shared libraries . done
>
> Program received signal EXC_BAD_INSTRUCTION, Illegal instruction/operand.
> 0x8fe136e4 in __dyld_stub_binding_helper_interface ()
> (gdb) print (char*)state
> $1 = 0x1fa8 "calling printf"
> (gdb) quit
> The program is running.  Exit anyway? (y or n) y
> $
>
>

Re: [9fans] OS X threads + dynamic linking

[David Leimbach]

Actually, this code violates the ABI.  The stack needs to be aligned
on a 16byte address.

http://developer.apple.com/documentation/DeveloperTools/Conceptual/LowLevelABI/index.html

So the assembly block should be:

       asm(
               "pushal\n"
               "movl 8(%ebp), %ebx\n"
               "movl 12(%ebp), %eax\n"
               "xchgl %esp, %eax\n"
	        "subl $0xc, %esp\n"  //need to align the stack... can't just pushl
               "pushl %eax\n"
               "call *%ebx\n"
               "popl %esp\n"
               "popal\n"
       );

Just don't compile with -Os because gcc freaking inlines runonstack. 
Whom to throttle for that one?

Dave

On 4/20/06, Russ Cox <rsc@swtch.com> wrote:
> [The only claim this has to being on topic is that
> I need an answer to make libthread work on OS X x86.
> That and it has to do with dynamic linking.  Sorry.]
>
> In the gcc program below, the function runonstack uses
> some carefully crafted inline assembly to run the function fn
> on the given stack (the pointer passed in is the top of the stack).
> Because of the use of assembly, it is an x86-only program.
>
> It works great on Linux and on FreeBSD.  On OS X, though,
> the function running on the alternate stack dies when it
> tries to call printf.  A gdb session running the program
> is shown after the program.
>
> It's dying in the dynamic linker trying to resolve printf.
> If I call printf in main before calling runonstack, then printf
> no longer fails.  However, if I then call exit(0) inside hello,
> then that fails, because exit isn't resolved yet.
>
> Thus, it appears that somehow the x86 OS X library routines
> cannot handle being called from an alternate stack.
> This just cannot be true.  I thought everyone had already
> made that mistake and moved on.
>
> If anyone can either confirm that this program has no hope
> of working on OS X (that would be unfortunate) or can tell
> me what I need to do to make it work (that would be better!),
> I'd greatly appreciate it.
>
> As of right now, it appears that the plan9port CVS tree
> builds just fine on x86 OS X except that all the threaded
> programs crash due to this problem.
>
> Thanks for any help.
> Russ
>
>
>
> #include <stdio.h>
> #include <stdlib.h>
>
> char stack[1048576];
> char *state;
>
> void
> hello(void)
> {
>         state = "calling printf";
>         printf("hello, world\n");
>         state = "done with printf";
> }
>
> void
> runonstack(void (*fn)(void), char *stack)
> {
>         state = "in assembly";
>         asm(
>                 "pushal\n"
>                 "movl 8(%ebp), %ebx\n"  /* ebx = fn */
>                 "movl 12(%ebp), %eax\n" /* ecx = stack */
>                 "xchgl %esp, %eax\n"
>                 "pushl %eax\n"
>                 "call *%ebx\n"
>                 "popl %esp\n"
>                 "popal\n"
>         );
>         state = "out of assembly";
> }
>
> int
> main(int argc, char **argv)
> {
>         runonstack(hello, stack+sizeof stack);
> }
>
> ---
>
> $ gdb a.out
> GNU gdb 6.1-20040303 (Apple version gdb-437) (Fri Jan 13 18:45:48 GMT 2006)
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "i386-apple-darwin"...Reading symbols for shared libraries .. done
>
> (gdb) run
> Starting program: /Users/rsc/a.out
> Reading symbols for shared libraries . done
>
> Program received signal EXC_BAD_INSTRUCTION, Illegal instruction/operand.
> 0x8fe136e4 in __dyld_stub_binding_helper_interface ()
> (gdb) print (char*)state
> $1 = 0x1fa8 "calling printf"
> (gdb) quit
> The program is running.  Exit anyway? (y or n) y
> $
>
>

Re: [9fans] OS X threads + dynamic linking

["Nils O. Selåsdal"]

David Leimbach wrote:
> Actually, this code violates the ABI.  The stack needs to be aligned
> on a 16byte address.
>
> http://developer.apple.com/documentation/DeveloperTools/Conceptual/LowLevelABI/index.html
>
> So the assembly block should be:
>
>        asm(
>                "pushal\n"
>                "movl 8(%ebp), %ebx\n"
>                "movl 12(%ebp), %eax\n"
>                "xchgl %esp, %eax\n"
> 	        "subl $0xc, %esp\n"  //need to align the stack... can't just pushl
>                "pushl %eax\n"
>                "call *%ebx\n"
>                "popl %esp\n"
>                "popal\n"
>        );
>
> Just don't compile with -Os because gcc freaking inlines runonstack.
*shrug*
> Whom to throttle for that one?
I guess you could tack on a __attribute__ ((noinline))
to the prototype.
(as always, gcc is beeing "nice" enough to provide hacks
that gets you out of the trouble it created for you in the first
place)

Re: [9fans] OS X threads + dynamic linking

[Russ Cox]

> Actually, this code violates the ABI.  The stack needs to be aligned
> on a 16byte address.
>
> http://developer.apple.com/documentation/DeveloperTools/Conceptual/LowLevelABI/index.html
>
> So the assembly block should be:
>
>        asm(
>                "pushal\n"
>                "movl 8(%ebp), %ebx\n"
>                "movl 12(%ebp), %eax\n"
>                "xchgl %esp, %eax\n"
> 	        "subl $0xc, %esp\n"  //need to align the stack... can't just pushl
>                "pushl %eax\n"
>                "call *%ebx\n"
>                "popl %esp\n"
>                "popal\n"
>        );
>
> Just don't compile with -Os because gcc freaking inlines runonstack.
> Whom to throttle for that one?

The real version doesn't use inline assembly, so it's not a problem.
I just wanted a single file demonstrating the problem.

After aligning the stack pointer properly, the x86 OS X code
now in CVS does appear to work.

Thanks very much for tracking this down.

Russ

Re: [9fans] OS X threads + dynamic linking

[David Leimbach]

On 4/21/06, Russ Cox <rsc@swtch.com> wrote:
> > Actually, this code violates the ABI.  The stack needs to be aligned
> > on a 16byte address.
> >
> > http://developer.apple.com/documentation/DeveloperTools/Conceptual/LowLevelABI/index.html
> >
> > So the assembly block should be:
> >
> >        asm(
> >                "pushal\n"
> >                "movl 8(%ebp), %ebx\n"
> >                "movl 12(%ebp), %eax\n"
> >                "xchgl %esp, %eax\n"
> >               "subl $0xc, %esp\n"  //need to align the stack... can't just pushl
> >                "pushl %eax\n"
> >                "call *%ebx\n"
> >                "popl %esp\n"
> >                "popal\n"
> >        );
> >
> > Just don't compile with -Os because gcc freaking inlines runonstack.
> > Whom to throttle for that one?
>
> The real version doesn't use inline assembly, so it's not a problem.
> I just wanted a single file demonstrating the problem.
>
> After aligning the stack pointer properly, the x86 OS X code
> now in CVS does appear to work.
>
> Thanks very much for tracking this down.
>

Hey no problem... I want to use it on my new system :)  Thanks for
"making it go".

> Russ
>
>