dynamic linker command line invocation

dynamic linker command line invocation

[N Jain]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

Hi All,

I am trying to add ldso functionality in my kernel. I am loading the
dynamic linker "ld-musl-arm.so.1" into memory and passing the other
application as command line which requires the dynamic libraries but the
linker is generating fault during stage 2 "__dls2" at some random location
0x464cc57f.

I am fairly new to dynamic linking code and trying to understand _dlstart_c
code functionality. Can any one explain what are the command line arguments
dynamic linker expects at this entry point ? I am giving numArgs = 1 and
argv = app.elf after loading "ld-musl-arm.so.1" into memory ? Is this
approach correct ? Do I have to also load app.elf into memory or the
dynamic linker will take care of loading it ?

Thanks,
NJ

[-- Attachment #2: Type: text/html, Size: 893 bytes --]

Re: dynamic linker command line invocation

[Markus Wichmann]

On Mon, Jan 04, 2016 at 11:59:16AM -0500, N Jain wrote:
> Hi All,
> 
> I am trying to add ldso functionality in my kernel. I am loading the
> dynamic linker "ld-musl-arm.so.1" into memory and passing the other
> application as command line which requires the dynamic libraries but the
> linker is generating fault during stage 2 "__dls2" at some random location
> 0x464cc57f.
> 

Why don't you just set ld-musl-arm.so.1 as dynamic interpreter and
launch the application through the kernel? (Link with -Wl,-I,/lib/ld...)

> I am fairly new to dynamic linking code and trying to understand _dlstart_c
> code functionality. Can any one explain what are the command line arguments
> dynamic linker expects at this entry point ? I am giving numArgs = 1 and
> argv = app.elf after loading "ld-musl-arm.so.1" into memory ? Is this
> approach correct ? Do I have to also load app.elf into memory or the
> dynamic linker will take care of loading it ?
> 

You say it's crashing stage 2, but only stage 3 handles arguments. In
any case, musl expects the application's ELF image in argv[1]. The rest
of argv is assumed to be the command line of the application. And
argv[0] is ignored, save for a comparison to "ldd" (if argv[0] ends in
"ldd", the dynlinker will just display all the loaded libraries).

> Thanks,
> NJ

Ciao,
Markus

Re: dynamic linker command line invocation

[Rich Felker]

On Mon, Jan 04, 2016 at 08:22:06PM +0100, Markus Wichmann wrote:
> On Mon, Jan 04, 2016 at 11:59:16AM -0500, N Jain wrote:
> > Hi All,
> > 
> > I am trying to add ldso functionality in my kernel. I am loading the
> > dynamic linker "ld-musl-arm.so.1" into memory and passing the other
> > application as command line which requires the dynamic libraries but the
> > linker is generating fault during stage 2 "__dls2" at some random location
> > 0x464cc57f.
> > 
> 
> Why don't you just set ld-musl-arm.so.1 as dynamic interpreter and
> launch the application through the kernel? (Link with -Wl,-I,/lib/ld...)

I believe you missed the point: N Jain is implementing a kernel.

Rich

Re: dynamic linker command line invocation

[Rich Felker]

On Mon, Jan 04, 2016 at 11:59:16AM -0500, N Jain wrote:
> Hi All,
> 
> I am trying to add ldso functionality in my kernel. I am loading the
> dynamic linker "ld-musl-arm.so.1" into memory and passing the other
> application as command line which requires the dynamic libraries but the
> linker is generating fault during stage 2 "__dls2" at some random location
> 0x464cc57f.
> 
> I am fairly new to dynamic linking code and trying to understand _dlstart_c
> code functionality. Can any one explain what are the command line arguments
> dynamic linker expects at this entry point ? I am giving numArgs = 1 and
> argv = app.elf after loading "ld-musl-arm.so.1" into memory ? Is this
> approach correct ? Do I have to also load app.elf into memory or the
> dynamic linker will take care of loading it ?

Are you providing a complete and correct aux vector after the argv[]
and environ[]? If it's missing or contains incorrect information this
would surely cause crashing.

The ideal way to load dynamic-linked programs is to have the kernel
load both the main executable and the dynamic linker (where the latter
is obtained from the PT_INTERP header in the main program). In this
case, AT_BASE needs to point to the offset at which the dynamic linker
was loaded, and AT_PHDR needs to point to the main program's program
headers (and AT_PHENT and AT_PHNUM should also be valid). AT_ENTRY
also needs to point to the main program's entry point (from the ELF
Ehdr).

On the other hand, if you want to just load the dynamic linker and
pass the name of the program to run as an argument, AT_BASE must be
either unset or 0, and AT_PHDR must point to the dynamic linker's
program headers. This approach is undesirable however because it's
subject to race conditions if the executable is moved/replaced.
There's also the issue that the address you loaded the dynamic linker
at may conflict with the address where the main program is to be
loaded, but this is a non-issue for PIE executables.

Rich

Re: dynamic linker command line invocation

[N Jain]

[-- Attachment #1: Type: text/plain, Size: 2898 bytes --]

Hi Rich,

Thanks. Right now I am following second approach due to simplicity.
I understand about race condition so does it mean that currently the MUSL
dynamic linker will not gracefully handle such situation where the
executable is moved/replaced ?

During further debug I found that passed parameters to dynamic linker are
correct and argv[0] does have dynamically "app.elf".
Also I am loading the dynamic linker at fixed load offset 0x8000 so I
needed to adjust my all dynamic linker LOAD sections accordingly which was
causing issues previously.

Now I am facing issue during __dls3 API (stage 3). I would like to know if
I can enable "dprintfs"  in MUSL code in order to easily debug ?
What I have to do in order to enable these "dprintfs" ?


Thanks,
NJ

On Mon, Jan 4, 2016 at 3:59 PM, Rich Felker <dalias@libc.org> wrote:

> On Mon, Jan 04, 2016 at 11:59:16AM -0500, N Jain wrote:
> > Hi All,
> >
> > I am trying to add ldso functionality in my kernel. I am loading the
> > dynamic linker "ld-musl-arm.so.1" into memory and passing the other
> > application as command line which requires the dynamic libraries but the
> > linker is generating fault during stage 2 "__dls2" at some random
> location
> > 0x464cc57f.
> >
> > I am fairly new to dynamic linking code and trying to understand
> _dlstart_c
> > code functionality. Can any one explain what are the command line
> arguments
> > dynamic linker expects at this entry point ? I am giving numArgs = 1 and
> > argv = app.elf after loading "ld-musl-arm.so.1" into memory ? Is this
> > approach correct ? Do I have to also load app.elf into memory or the
> > dynamic linker will take care of loading it ?
>
> Are you providing a complete and correct aux vector after the argv[]
> and environ[]? If it's missing or contains incorrect information this
> would surely cause crashing.
>
> The ideal way to load dynamic-linked programs is to have the kernel
> load both the main executable and the dynamic linker (where the latter
> is obtained from the PT_INTERP header in the main program). In this
> case, AT_BASE needs to point to the offset at which the dynamic linker
> was loaded, and AT_PHDR needs to point to the main program's program
> headers (and AT_PHENT and AT_PHNUM should also be valid). AT_ENTRY
> also needs to point to the main program's entry point (from the ELF
> Ehdr).
>
> On the other hand, if you want to just load the dynamic linker and
> pass the name of the program to run as an argument, AT_BASE must be
> either unset or 0, and AT_PHDR must point to the dynamic linker's
> program headers. This approach is undesirable however because it's
> subject to race conditions if the executable is moved/replaced.
> There's also the issue that the address you loaded the dynamic linker
> at may conflict with the address where the main program is to be
> loaded, but this is a non-issue for PIE executables.
>
> Rich
>

[-- Attachment #2: Type: text/html, Size: 3681 bytes --]

Re: dynamic linker command line invocation

[Rich Felker]

On Tue, Jan 05, 2016 at 11:45:01AM -0500, N Jain wrote:
> Hi Rich,
> 
> Thanks. Right now I am following second approach due to simplicity.
> I understand about race condition so does it mean that currently the MUSL
> dynamic linker will not gracefully handle such situation where the
> executable is moved/replaced ?

No, it's not something you can "handle" but an inherent race
condition. To avoid it the kernel would have to refrain from passing
the actual pathname to the file but instead some virtual pathname like
/proc/self/main_exe (hypothetical) that would be tied to the original
inode the user tried to execute.

> During further debug I found that passed parameters to dynamic linker are
> correct and argv[0] does have dynamically "app.elf".
> Also I am loading the dynamic linker at fixed load offset 0x8000 so I
> needed to adjust my all dynamic linker LOAD sections accordingly which was
> causing issues previously.

Hm? You should not be making any adjustments. The dynamic linker
applies its own relocations to itself when run. You just need to pass
it the right pointers in the aux vector. You never answered whether
you were setting up the aux vector right, but if not, that's
definitely going to cause problems.

> Now I am facing issue during __dls3 API (stage 3). I would like to know if
> I can enable "dprintfs"  in MUSL code in order to easily debug ?
> What I have to do in order to enable these "dprintfs" ?

dprintf is a standard function for printing to a file descriptor, not
a debug function. It's definitely usable by the time you reach stage 3
and should also be usable during stage 2. On x86 linked with
-Bsymbolic-functions (as we do now) it may even work at stage 1.

Rich

P.S. Please don't top-post to the list but instead reply inline/below
the text you're replying to, and edit to include only relevant
context.

Re: dynamic linker command line invocation

[N Jain]

[-- Attachment #1: Type: text/plain, Size: 2312 bytes --]

> You never answered whether
> you were setting up the aux vector right, but if not, that's
> definitely going to cause problems.

I am not setting any aux vectors. I only pass argv = "app.elf" and argc = 1
to dynamic linker.
What and where I need to set "aux" vectors ? Any pointers will help..

Thanks,
NJ

On Tue, Jan 5, 2016 at 12:32 PM, Rich Felker <dalias@libc.org> wrote:

> On Tue, Jan 05, 2016 at 11:45:01AM -0500, N Jain wrote:
> > Hi Rich,
> >
> > Thanks. Right now I am following second approach due to simplicity.
> > I understand about race condition so does it mean that currently the MUSL
> > dynamic linker will not gracefully handle such situation where the
> > executable is moved/replaced ?
>
> No, it's not something you can "handle" but an inherent race
> condition. To avoid it the kernel would have to refrain from passing
> the actual pathname to the file but instead some virtual pathname like
> /proc/self/main_exe (hypothetical) that would be tied to the original
> inode the user tried to execute.
>
> > During further debug I found that passed parameters to dynamic linker are
> > correct and argv[0] does have dynamically "app.elf".
> > Also I am loading the dynamic linker at fixed load offset 0x8000 so I
> > needed to adjust my all dynamic linker LOAD sections accordingly which
> was
> > causing issues previously.
>
> Hm? You should not be making any adjustments. The dynamic linker
> applies its own relocations to itself when run. You just need to pass
> it the right pointers in the aux vector. You never answered whether
> you were setting up the aux vector right, but if not, that's
> definitely going to cause problems.
>
> > Now I am facing issue during __dls3 API (stage 3). I would like to know
> if
> > I can enable "dprintfs"  in MUSL code in order to easily debug ?
> > What I have to do in order to enable these "dprintfs" ?
>
> dprintf is a standard function for printing to a file descriptor, not
> a debug function. It's definitely usable by the time you reach stage 3
> and should also be usable during stage 2. On x86 linked with
> -Bsymbolic-functions (as we do now) it may even work at stage 1.
>
> Rich
>
> P.S. Please don't top-post to the list but instead reply inline/below
> the text you're replying to, and edit to include only relevant
> context.
>

[-- Attachment #2: Type: text/html, Size: 3145 bytes --]

Re: dynamic linker command line invocation

[Rich Felker]

On Tue, Jan 05, 2016 at 01:00:37PM -0500, N Jain wrote:
> > You never answered whether
> > you were setting up the aux vector right, but if not, that's
> > definitely going to cause problems.
> 
> I am not setting any aux vectors. I only pass argv = "app.elf" and argc = 1
> to dynamic linker.
> What and where I need to set "aux" vectors ? Any pointers will help..

The ELF entry point ABI is that the initial stack pointer points to an
array of word-sized cells containing:

argc argv[0] argv[1] ... argv[argc-1] argv[argc](=0) anviron[0]
environ[1] ... 0 auxv[0].key auxv[0].val auxv[1].key auxv[1].val ... 0

The auxv items are key,val pairs where the key is one of the AT_*
constants from elf.h and the value is either an integer or pointer
(depending on which key it's for). At the very least you should be
passing:

AT_PHDR - points to the start of elf program headers
AT_PHENT - size of each program header
AT_PHNUM - number of program headers

If you load both the main program and "interpreter" (dynamic linker)
from the kernel, then these values should be for the main program's
headers, and in addition you need to pass:

AT_BASE - load address of the "interpreter" (dynamic linker)
AT_ENTRY - entry point address of the main program, from its header

On the other hand, if you only load the dynamic linker (treating it as
the main program), then the program header auxv entries should hold
the right values for the dynamic linker.

Rich

Re: dynamic linker command line invocation

[N Jain]

[-- Attachment #1: Type: text/plain, Size: 2828 bytes --]

Hi Rich,

Thanks I am able to get my dynamic linker loaded into memory but facing
issue during application load at __dls3 function.
I am getting following print from MUSL -

"musl libc
Version 1.1.10
Dynamic Program Loader
Usage: hello.elf [options] [--] pathname--2--"

Not sure why my application is not getting loaded in below code __dls3
function ?

char *ldname = argv[0];
size_t l = strlen(ldname);

*ldname is the application elf which I want to load "hello.elf"*
if (l >= 3 && !strcmp(ldname+l-3, "ldd")) ldd_mode = 1;
*I am not sure why "ldd" string is compared here ?*
argv++;
*I don't pass any options ? Do I need to pass some option value ?*
while (argv[0] && argv[0][0]=='-' && argv[0][1]=='-') {
char *opt = argv[0]+2;
*argv++ = (void *)-1;
if (!*opt) {
break;
} else if (!memcmp(opt, "list", 5)) {
ldd_mode = 1;
} else if (!memcmp(opt, "library-path", 12)) {
if (opt[12]=='=') env_path = opt+13;
else if (opt[12]) *argv = 0;
else if (*argv) env_path = *argv++;
} else if (!memcmp(opt, "preload", 7)) {
if (opt[7]=='=') env_preload = opt+8;
else if (opt[7]) *argv = 0;
else if (*argv) env_preload = *argv++;
} else {
argv[0] = 0;
}
}
*I see the argv[0] is set to 0 so I am not able to see my application
loaded ?*

Thanks,
NJ

On Tue, Jan 5, 2016 at 1:23 PM, Rich Felker <dalias@libc.org> wrote:

> On Tue, Jan 05, 2016 at 01:00:37PM -0500, N Jain wrote:
> > > You never answered whether
> > > you were setting up the aux vector right, but if not, that's
> > > definitely going to cause problems.
> >
> > I am not setting any aux vectors. I only pass argv = "app.elf" and argc
> = 1
> > to dynamic linker.
> > What and where I need to set "aux" vectors ? Any pointers will help..
>
> The ELF entry point ABI is that the initial stack pointer points to an
> array of word-sized cells containing:
>
> argc argv[0] argv[1] ... argv[argc-1] argv[argc](=0) anviron[0]
> environ[1] ... 0 auxv[0].key auxv[0].val auxv[1].key auxv[1].val ... 0
>
> The auxv items are key,val pairs where the key is one of the AT_*
> constants from elf.h and the value is either an integer or pointer
> (depending on which key it's for). At the very least you should be
> passing:
>
> AT_PHDR - points to the start of elf program headers
> AT_PHENT - size of each program header
> AT_PHNUM - number of program headers
>
> If you load both the main program and "interpreter" (dynamic linker)
> from the kernel, then these values should be for the main program's
> headers, and in addition you need to pass:
>
> AT_BASE - load address of the "interpreter" (dynamic linker)
> AT_ENTRY - entry point address of the main program, from its header
>
> On the other hand, if you only load the dynamic linker (treating it as
> the main program), then the program header auxv entries should hold
> the right values for the dynamic linker.
>
> Rich
>

[-- Attachment #2: Type: text/html, Size: 5700 bytes --]

Re: dynamic linker command line invocation

[Markus Wichmann]

On Thu, Jan 07, 2016 at 06:29:48PM -0500, N Jain wrote:
> Hi Rich,
> 
> Thanks I am able to get my dynamic linker loaded into memory but facing
> issue during application load at __dls3 function.
> I am getting following print from MUSL -
> 
> "musl libc
> Version 1.1.10
> Dynamic Program Loader
> Usage: hello.elf [options] [--] pathname--2--"
> 
> Not sure why my application is not getting loaded in below code __dls3
> function ?
> 

That's what I was trying to tell you in my first answer: You need to set
argv[0] to something (doesn't matter so long as it's not "ldd"), and
argv[1] to the program file you want to run. (And argc to 2, obviously).

The only alternative to that is to load the program file in the kernel
additionally to musl, and then set the following aux vectors:

AT_BASE - base address of dynamic linker
AT_PHDR - address of main executable's program headers
AT_PHNUM, AT_PHENT - number and size of main executable's program
headers, respectively.
AT_ENTRY - main executable's entry point

Ciao,
Markus

Re: dynamic linker command line invocation

[N Jain]

[-- Attachment #1: Type: text/plain, Size: 1951 bytes --]

Hi Marcus,

Thanks. I am able to get this part working.
I am facing one more issue during dynamic linking and need some help to
understand how to resolve it.

In below code part of __dls3 function I found that TPIDRURO register is
being used to read the threadId.
I would like to understand how kernel should set this value ?

As per my understanding kernel should allocate memory and set this register
for current task.
But when i do this in my kernel the values doesn't match up as MUSL is
using some "builtin_tls" ?

/* Temporarily set the tls size to the full size of
* builtin_tls so that __copy_tls will use the same layout
* as it did for before. Then check, just to be safe. */
libc.tls_size = sizeof builtin_tls;
if (__copy_tls((void*)builtin_tls) != self) a_crash();

Thanks,
NJ


On Fri, Jan 8, 2016 at 6:09 AM, Markus Wichmann <nullplan@gmx.net> wrote:

> On Thu, Jan 07, 2016 at 06:29:48PM -0500, N Jain wrote:
> > Hi Rich,
> >
> > Thanks I am able to get my dynamic linker loaded into memory but facing
> > issue during application load at __dls3 function.
> > I am getting following print from MUSL -
> >
> > "musl libc
> > Version 1.1.10
> > Dynamic Program Loader
> > Usage: hello.elf [options] [--] pathname--2--"
> >
> > Not sure why my application is not getting loaded in below code __dls3
> > function ?
> >
>
> That's what I was trying to tell you in my first answer: You need to set
> argv[0] to something (doesn't matter so long as it's not "ldd"), and
> argv[1] to the program file you want to run. (And argc to 2, obviously).
>
> The only alternative to that is to load the program file in the kernel
> additionally to musl, and then set the following aux vectors:
>
> AT_BASE - base address of dynamic linker
> AT_PHDR - address of main executable's program headers
> AT_PHNUM, AT_PHENT - number and size of main executable's program
> headers, respectively.
> AT_ENTRY - main executable's entry point
>
> Ciao,
> Markus
>

[-- Attachment #2: Type: text/html, Size: 2993 bytes --]

Re: dynamic linker command line invocation

[Markus Wichmann]

On Mon, Jan 11, 2016 at 12:03:37PM -0500, N Jain wrote:
> Hi Marcus,
> 
> Thanks. I am able to get this part working.
> I am facing one more issue during dynamic linking and need some help to
> understand how to resolve it.
> 
> In below code part of __dls3 function I found that TPIDRURO register is
> being used to read the threadId.
> I would like to understand how kernel should set this value ?
> 

It shouldn't. musl calls __set_thread_area() (in __init_tp()), and that
has to set this register.

The kernel is only involved, if musl has to do a syscall there.

> As per my understanding kernel should allocate memory and set this register
> for current task.

Generally, no. The allocation is up to the application. But on some
architectures, the kernel has to add a bit of stuff to the list of
things to be swapped on task switch (e.g. on i386 the TLS segment
descriptors in the GDT). For more information, ask your architecture's
system programming manual.

> But when i do this in my kernel the values doesn't match up as MUSL is
> using some "builtin_tls" ?
> 
> /* Temporarily set the tls size to the full size of
> * builtin_tls so that __copy_tls will use the same layout
> * as it did for before. Then check, just to be safe. */
> libc.tls_size = sizeof builtin_tls;
> if (__copy_tls((void*)builtin_tls) != self) a_crash();
> 

If you get there, __init_tp() was already called, so the thread pointer
should be set to builtin_tls. (Or rather, should be set such that
__copy_tls() will return the current thread pointer). If that isn't the
case then something has already gone wrong. Check __init_tp()!

Ciao,
Markus

Re: dynamic linker command line invocation

[Rich Felker]

On Tue, Jan 12, 2016 at 05:24:02PM +0100, Markus Wichmann wrote:
> On Mon, Jan 11, 2016 at 12:03:37PM -0500, N Jain wrote:
> > Hi Marcus,
> > 
> > Thanks. I am able to get this part working.
> > I am facing one more issue during dynamic linking and need some help to
> > understand how to resolve it.
> > 
> > In below code part of __dls3 function I found that TPIDRURO register is
> > being used to read the threadId.
> > I would like to understand how kernel should set this value ?
> > 
> 
> It shouldn't. musl calls __set_thread_area() (in __init_tp()), and that
> has to set this register.

Yes. The syscall made by __set_thread_area (it's an ARM-private
syscall on ARM, for no good reason, rather than actually being
SYS_set_thread_area) is supposed to set TPIDRURO for the calling
thread to the address provided by the caller.

Rich