[9fans] _tos puzzle

[9fans] _tos puzzle

[Anant Narayanan]

Good day,

I got some plan9 binaries to work on Linux (8c, 8l, cat, sed, cal and
a few more were tested), but all others are failing at exactly the
same instruction:

plock+0x31 MOVL 0x30(CX), CX

(which is called whenever malloc is used). For some context:

acid: asm(plock+0x20)
plock+0x20	JEQ		plock+0x27(SB)
plock+0x22	CALL	abort(SB)
plock+0x27	MOVL	pv+0xc(SP), AX
plock+0x2b	MOVL	_tos(SB), CX
plock+0x31	MOVL	0x30(CX), CX
plock+0x32	DECL	CX
plock+0x33	XORB	CL, 0xc4830448(CX)
plock+0x39	ADCB	AL,BL

The line of interest is plock+0x2b: (_tos(SB), CX); which is supposed
to store a value at CX. However, when the binary is run in Linux, CX
becomes 0 after that instruction, so plock+0x31 becomes (MOVL 0x30,
CX) resulting in a segfault as 0x30 is an invalid address in the
process address space (it starts only at 0x1000).

acid: asm(_tos)
_tos		ADDB	AL, 0x0(AX)

I defined a TEXT section for _tos, in the 'Hello Assembly' program
discussed earlier, and used _tos after printing Hello on the screen.
acid tells me CX becomes 0 after that instruction, which is exactly
what happens on Linux too:

DATA 	string<>+0(SB)/8, $"Plan9\n\z\z"
GLOBL 	string<>+0(SB), $8

TEXT 	_main+0(SB), 1, $0
MOVL	$string<>+0(SB), 4(SP)
MOVL	$8, AX
MOVL	_tos(SB), CX
MOVL	0x30(CX), CX
INT		$64

TEXT	_tos+0(SB), 1, $0
ADDB	AL, 0x0(AX)

8.out: 1831: suicide: sys: trap: fault read addr=0x30 pc=0x00001033

What is _tos supposed to do, and why does it set CX to different
values for some plan9 binaries, but not in a standalone assembled
program and on linux (in both cases, CX is set to 0)?

Thanks in advance for your help!

Regards,
Anant

Re: [9fans] _tos puzzle

[Russ Cox]

Tos is a clumsy hack.  It refers to a structure stored
at the top of the stack that is used to pass data
between the kernel and user space without system calls.
See /sys/include/tos.h.

When a Plan 9 binary first starts running, the system
call return register (AX in this case) contains a pointer
to the Tos structure.  /sys/src/libc/386/main9.s
saves that value in the global variable named _tos.
It sounds like you are not setting AX correctly when
you start the programs, so _tos is not initialized
correctly.

How do you plan to implement rfork(RFMEM|RFPROC),
which must share the entire address space between
the parent and child except for the stack segment?

Russ

Re: [9fans] _tos puzzle

[Anant Narayanan]

Hi Russ,

Thanks for your response!

> When a Plan 9 binary first starts running, the system
> call return register (AX in this case) contains a pointer
> to the Tos structure.  /sys/src/libc/386/main9.s
> saves that value in the global variable named _tos.
> It sounds like you are not setting AX correctly when
> you start the programs, so _tos is not initialized
> correctly.

So I have to allocate 56 bytes (sizeof(Tos) on 386) of space above the
top of stack (before copying argc and argv) and set AX to that
address, correct?

> How do you plan to implement rfork(RFMEM|RFPROC),
> which must share the entire address space between
> the parent and child except for the stack segment?

The clone(2) system call in linux allows for creation of child
processes that share address space with its parent, with the exception
of the stack segment. I believe that should do?

Regards,
Anant

Re: [9fans] _tos puzzle

[Russ Cox]

> So I have to allocate 56 bytes (sizeof(Tos) on 386) of space above the
> top of stack (before copying argc and argv) and set AX to that
> address, correct?

Yes.  And you need to maintain it.
At the very least you need to initialize tos->pid
and update it on return from rfork.

> The clone(2) system call in linux allows for creation of child
> processes that share address space with its parent, with the exception
> of the stack segment.

It does?  I don't see that anywhere in the man page.

Russ

Re: [9fans] _tos puzzle

[Steven Vormwald]

On Thu, 2008-05-08 at 13:24 -0400, Russ Cox wrote:
> > So I have to allocate 56 bytes (sizeof(Tos) on 386) of space above the
> > top of stack (before copying argc and argv) and set AX to that
> > address, correct?
>
> Yes.  And you need to maintain it.
> At the very least you need to initialize tos->pid
> and update it on return from rfork.
>
> > The clone(2) system call in linux allows for creation of child
> > processes that share address space with its parent, with the exception
> > of the stack segment.
>
> It does?  I don't see that anywhere in the man page.
>
> Russ

Wouldn't the CLONE_VM flag do this?

CLONE_VM
       If CLONE_VM is set, the calling process and the child  processes
       run in the same memory space.  In particular, memory writes per‐
       formed by the calling process or by the child process  are  also
       visible  in  the other process.  Moreover, any memory mapping or
       unmapping performed with mmap(2) or munmap(2) by  the  child  or
       calling process also affects the other process.

Steven Vormwald

Re: [9fans] _tos puzzle

[Anant Narayanan]

>> So I have to allocate 56 bytes (sizeof(Tos) on 386) of space above
>> the
>> top of stack (before copying argc and argv) and set AX to that
>> address, correct?
>
> Yes.  And you need to maintain it.
> At the very least you need to initialize tos->pid
> and update it on return from rfork.

I set EAX before starting the executable, but still no luck :(
On Plan 9, for /bin/mk:

acid: symbols("_tos")
_tos		D	0x00016084
acid: mem(0x00016084, "X")
0xdfffefc8

I'm probing address 0x16084 on linux after every instruction (using
ptrace's singlestep), and it consistently returns 0. EIP at the
beginning of the program is 0x9fe4, and sure enough:

acid: asm(0x9fe4)
_main		SUBL	$0x48, SP
_main+0x3	MOVL	AX, _tos(SB)
...

So (MOVL, _tos(SB)) is definitely executed, but for some reason, the
value of AX is not stored in 0x16084. That brings me to the question
of how 8a decides what address to put values like that in? Is the
address the same everytime, i.e. hardcoded into the binary? (certainly
seems to be)

I strip the symbol table from plan 9 executables and store only data,
text and initialize bss in memory - maybe that has something to do
with it. Does the symbol table need to be in memory too for
instructions like (MOVL AX, _tos(SB)) to work?

Regards,
Anant

Re: [9fans] _tos puzzle

[Russ Cox]

> I set EAX before starting the executable, but still no luck :(
> On Plan 9, for /bin/mk:
>
> acid: symbols("_tos")
> _tos		D	0x00016084
> acid: mem(0x00016084, "X")
> 0xdfffefc8
>
> I'm probing address 0x16084 on linux after every instruction (using
> ptrace's singlestep), and it consistently returns 0. EIP at the
> beginning of the program is 0x9fe4, and sure enough:
>
> acid: asm(0x9fe4)
> _main		SUBL	$0x48, SP
> _main+0x3	MOVL	AX, _tos(SB)
> ...
>
> So (MOVL, _tos(SB)) is definitely executed, but for some reason, the
> value of AX is not stored in 0x16084.

You should print AX after every instruction too, to see if
you've actually set it up the way you think you did.

> That brings me to the question
> of how 8a decides what address to put values like that in? Is the
> address the same everytime, i.e. hardcoded into the binary? (certainly
> seems to be)

_tos is no different than any other global variable.
8a doesn't use any address at all - it leaves a slot for 8l to fill in.
The eventual address of _tos depends on what other data
is in the binary.  I don't know why you say the address is
the same every time:

cpu% nm /bin/cat |grep _tos
    600c D _tos
cpu% nm /bin/ls |grep _tos
    d060 D _tos
cpu% nm /bin/echo |grep _tos
    9008 D _tos
cpu%

> I strip the symbol table from plan 9 executables and store only data,
> text and initialize bss in memory - maybe that has something to do
> with it. Does the symbol table need to be in memory too for
> instructions like (MOVL AX, _tos(SB)) to work?

No, the symbol table is only for debuggers and the like.
Plan 9 doesn't load it into memory either.

The good news is that you've identified the program
behaving incorrectly after executing only *two* instructions.
That should narrow things down considerably.

Russ

Re: [9fans] _tos puzzle

[Anant Narayanan]

Hi Russ,

> You should print AX after every instruction too, to see if
> you've actually set it up the way you think you did.

I did, and to my surprise, AX is 0, even though I set it properly in
the loader. It appears as if Linux is setting AX to 0 sometime after
the loader finishes, but before the executable begins. All other
registers, however, are preserved (eip being the most important).

>> That brings me to the question
>> of how 8a decides what address to put values like that in? Is the
>> address the same everytime, i.e. hardcoded into the binary?
>> (certainly
>> seems to be)
>
> _tos is no different than any other global variable.
> 8a doesn't use any address at all - it leaves a slot for 8l to fill
> in.
> The eventual address of _tos depends on what other data
> is in the binary.  I don't know why you say the address is
> the same every time:

I meant the address being fixed for a particular binary.

> No, the symbol table is only for debuggers and the like.
> Plan 9 doesn't load it into memory either.
>
> The good news is that you've identified the program
> behaving incorrectly after executing only *two* instructions.
> That should narrow things down considerably.

Every program linked with libc starts with:

83 ec 48: SUBL 0x48, SP

And the following opcode is:

89 05 xx xx xx xx: MOVL AX, _tos(SB).

As a hack, as I'm padding the executables anyway, I check for those
bytes and change 89 05 to 89 1D during the padding - which makes the
instruction MOVL BX, _tos(SB). I set BX to the proper value in the
loader (which is preserved by linux), and voila!

A lot of executables are working now, with the exception of awk (which
tries to open #e, which isn't there) - Thanks!

--
Anant

Re: [9fans] _tos puzzle

[Russ Cox]

> I did, and to my surprise, AX is 0, even though I set it properly in
> the loader. It appears as if Linux is setting AX to 0 sometime after
> the loader finishes, but before the executable begins. All other
> registers, however, are preserved (eip being the most important).

The zero you see in AX is probably the return value from exec.

Russ

Re: [9fans] _tos puzzle

[Charles Forsyth]

> The clone(2) system call in linux allows for creation of child
> processes that share address space with its parent, with the exception
> of the stack segment. I believe that should do?

>Wouldn't the CLONE_VM flag do this?

no.

Linux shares the stack segment too: the underlying system call shares everything (given CLONE_VM),
and the clone library function simply switches the stack pointer to the address you give it,
but that's still in the shared address space.

rfork by contrast shares data and dss, but gives parent and child private [logical] copies of the stack.
that is, both initially have the same stack addresses and contents, but can subsequently change them
independently.  both stacks can grow.

clone therefore creates different memory contents for parent and child, with completely different
stack addresses and contents, and the code the child executes is distinguished by the clone call itself.

by contrast, rfork gives parent and child the same initial memory contents, sharing only the data,
and rfork's caller determines (based on the process ID return value) which does what after the call.