Question regarding dynamic loader

Question regarding dynamic loader

[Gernot Reisinger]

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

Hi,
I recently stumbled upon an issue with preloading a shared object into a Go
application (see related Go ticket https://github.com/golang/go/issues/28909
).

In short - Go comes with an internal linker which will not link crt code to
the application. The entry point will directly execute Go standard library
code. As musl libc calls shared object constructors in crt code, the shared
objects constructors subsequently will never be invoked. Things will work
on glibc systems / processes. it It seems to be a subtle - but in this case
wide reaching - behavioral difference to glibc.

I wonder if calling constructor functions from crt code is an intended musl
libc behavior. My personal - non expert - gut feeling considers glibc
behavior "more correct". Is there a chance that musl will change this
behavior?
br
Gernot

[-- Attachment #2: Type: text/html, Size: 1013 bytes --]

Re: Question regarding dynamic loader

[Rich Felker]

On Wed, Nov 21, 2018 at 02:55:19PM +0100, Gernot Reisinger wrote:
> Hi,
> I recently stumbled upon an issue with preloading a shared object into a Go
> application (see related Go ticket https://github.com/golang/go/issues/28909
> ).
> 
> In short - Go comes with an internal linker which will not link crt code to
> the application. The entry point will directly execute Go standard library
> code. As musl libc calls shared object constructors in crt code, the shared

I don't think this assessment of what musl does is correct. It calls
the (initially loaded) shared object constructor via
__libc_start_main. If the program is not entered via
__libc_start_main, libc is not usable. Necessary initialization will
have been bypassed. This has little to do with whether the crt code
was linked, except that *crt1.o is normally responsible for calling
__libc_start_main. If the linking process bypasses crt1, it needs to
ensure that __libc_start_main ends up getting called in some other
way. As far as I know this is also true for glibc, so I'm not sure why
it differs.

> objects constructors subsequently will never be invoked. Things will work
> on glibc systems / processes. it It seems to be a subtle - but in this case
> wide reaching - behavioral difference to glibc.
> 
> I wonder if calling constructor functions from crt code is an intended musl
> libc behavior. My personal - non expert - gut feeling considers glibc
> behavior "more correct". Is there a chance that musl will change this
> behavior?

The musl behavior here is intentional. For FDPIC targets, it's
impossible to run *any* application code, in the main application or
shared libraries, before the main application's crt1 has executed,
because there are (essentially -- the equivalent of) self-relocations
performed at that stage that the dynamic linker can't see. If any
ctors were invoked directly by the dynamic linker before passing
control the the main application's entry point, they would run without
these relocations in the main application having been performed,
possibly resulting in runaway-wrong execution.

I believe Go is doing some bad hacks here with regard to its C FFI,
but it's likely fixable in some reasonable way. We should get more
eyes looking at it.

Rich

Re: Question regarding dynamic loader

[Gernot Reisinger]

[-- Attachment #1: Type: text/plain, Size: 2934 bytes --]

Thanks for your swift and extensive reply. Your explanations make a lot of
sense. (sorry for my sloppy description - __libc_start_main invoked by *crt
is the place where constructor calls happen as you did outline).
I did no extensive research how glibc executes these constructor calls. At
least the call stack indicates that they are partially executed in dynamic
linker context - _dl_start_user () in /lib64/ld-linux-x86-64.so
calling _dl_init.

I will add a reference to reply to the Go ticket.

Am Mi., 21. Nov. 2018 um 15:25 Uhr schrieb Rich Felker <dalias@libc.org>:

> On Wed, Nov 21, 2018 at 02:55:19PM +0100, Gernot Reisinger wrote:
> > Hi,
> > I recently stumbled upon an issue with preloading a shared object into a
> Go
> > application (see related Go ticket
> https://github.com/golang/go/issues/28909
> > ).
> >
> > In short - Go comes with an internal linker which will not link crt code
> to
> > the application. The entry point will directly execute Go standard
> library
> > code. As musl libc calls shared object constructors in crt code, the
> shared
>
> I don't think this assessment of what musl does is correct. It calls
> the (initially loaded) shared object constructor via
> __libc_start_main. If the program is not entered via
> __libc_start_main, libc is not usable. Necessary initialization will
> have been bypassed. This has little to do with whether the crt code
> was linked, except that *crt1.o is normally responsible for calling
> __libc_start_main. If the linking process bypasses crt1, it needs to
> ensure that __libc_start_main ends up getting called in some other
> way. As far as I know this is also true for glibc, so I'm not sure why
> it differs.
>
> > objects constructors subsequently will never be invoked. Things will work
> > on glibc systems / processes. it It seems to be a subtle - but in this
> case
> > wide reaching - behavioral difference to glibc.
> >
> > I wonder if calling constructor functions from crt code is an intended
> musl
> > libc behavior. My personal - non expert - gut feeling considers glibc
> > behavior "more correct". Is there a chance that musl will change this
> > behavior?
>
> The musl behavior here is intentional. For FDPIC targets, it's
> impossible to run *any* application code, in the main application or
> shared libraries, before the main application's crt1 has executed,
> because there are (essentially -- the equivalent of) self-relocations
> performed at that stage that the dynamic linker can't see. If any
> ctors were invoked directly by the dynamic linker before passing
> control the the main application's entry point, they would run without
> these relocations in the main application having been performed,
> possibly resulting in runaway-wrong execution.
>
> I believe Go is doing some bad hacks here with regard to its C FFI,
> but it's likely fixable in some reasonable way. We should get more
> eyes looking at it.
>
> Rich
>
>

[-- Attachment #2: Type: text/html, Size: 3587 bytes --]

Re: Question regarding dynamic loader

[Szabolcs Nagy]

* Gernot Reisinger <Gernot.Reisinger@omnino.at> [2018-11-21 16:52:53 +0100]:
> I did no extensive research how glibc executes these constructor calls. At
> least the call stack indicates that they are partially executed in dynamic
> linker context - _dl_start_user () in /lib64/ld-linux-x86-64.so
> calling _dl_init.

the dynamic linker runs the
- preinit_array functions of the main executable,
- the init_array and DT_INIT functions of shared libraries.

then via __libc_start_main the _init and init_array functions
of the main executable are run by libc_nonshared.a code that
is linked into the executable.

so part of the initialization (main exe) does require entry
via __libc_start_main (but this is not an issue for go).

however this design can change when glibc introduces a new
symbol version for __libc_start_main, so i don't see how
go can rely on any of this.

Re: Question regarding dynamic loader

[Gernot Reisinger]

[-- Attachment #1: Type: text/plain, Size: 1203 bytes --]

Thanks a lot for this exhaustive explanation - helps a lot to understand
the different initialization stages. I agree, one should not assume a
specific execution sequence of these initialization routines.


Am Mi., 21. Nov. 2018 um 17:41 Uhr schrieb Szabolcs Nagy <nsz@port70.net>:

> * Gernot Reisinger <Gernot.Reisinger@omnino.at> [2018-11-21 16:52:53
> +0100]:
> > I did no extensive research how glibc executes these constructor calls.
> At
> > least the call stack indicates that they are partially executed in
> dynamic
> > linker context - _dl_start_user () in /lib64/ld-linux-x86-64.so
> > calling _dl_init.
>
> the dynamic linker runs the
> - preinit_array functions of the main executable,
> - the init_array and DT_INIT functions of shared libraries.
>
> then via __libc_start_main the _init and init_array functions
> of the main executable are run by libc_nonshared.a code that
> is linked into the executable.
>
> so part of the initialization (main exe) does require entry
> via __libc_start_main (but this is not an issue for go).
>
> however this design can change when glibc introduces a new
> symbol version for __libc_start_main, so i don't see how
> go can rely on any of this.
>
>

[-- Attachment #2: Type: text/html, Size: 1618 bytes --]

Re: Question regarding dynamic loader

[Rich Felker]

On Wed, Nov 21, 2018 at 09:25:50AM -0500, Rich Felker wrote:
> On Wed, Nov 21, 2018 at 02:55:19PM +0100, Gernot Reisinger wrote:
> > I wonder if calling constructor functions from crt code is an intended musl
> > libc behavior. My personal - non expert - gut feeling considers glibc
> > behavior "more correct". Is there a chance that musl will change this
> > behavior?
> 
> The musl behavior here is intentional. For FDPIC targets, it's
> impossible to run *any* application code, in the main application or
> shared libraries, before the main application's crt1 has executed,
> because there are (essentially -- the equivalent of) self-relocations
> performed at that stage that the dynamic linker can't see. If any
> ctors were invoked directly by the dynamic linker before passing
> control the the main application's entry point, they would run without
> these relocations in the main application having been performed,
> possibly resulting in runaway-wrong execution.

For reference, this was initially done in commit
c87a52103399135d2f57a91a8bcc749d8cb2ca83. Of course these code paths
have changed significantly since then, but it gives some historical
context.

Rich

Re: Question regarding dynamic loader

[Szabolcs Nagy]

* Gernot Reisinger <Gernot.Reisinger@omnino.at> [2018-11-21 14:55:19 +0100]:
> Hi,
> I recently stumbled upon an issue with preloading a shared object into a Go
> application (see related Go ticket https://github.com/golang/go/issues/28909
> ).
> 
> In short - Go comes with an internal linker which will not link crt code to
> the application. The entry point will directly execute Go standard library

then calling into the c runtime later is undefined.

crt is required for the c runtime setup.

> code. As musl libc calls shared object constructors in crt code, the shared

this is not true, the crt code is a tiny stub that
calls the __libc_start_main setup code in libc.so,
where ctors are run (there are several mechanisms
to do ctors, but that's an elf thing: musl supports
both _init and initarry style initializers, former
is passed as argument to __libc_start_main the latter
requires begin/end symbols for the .initarray section,
in glibc is similar, but part of the initialization
happens in the dynamic linker and part of it in
libc_nonshared.a code which should be linked to the
application and not in libc.so. static linking is
another story but i assume you are using dynamic linking).

> objects constructors subsequently will never be invoked. Things will work
> on glibc systems / processes. it It seems to be a subtle - but in this case
> wide reaching - behavioral difference to glibc.

this is libc internal implementation detail that
callers should not try to guess or rely on.
(however it has to be abi stable within one libc
implementation because old crt1.o linked into
an executable must work with new libc.so, otoh
in glibc the abi can changed over time using
symbol versioning for backward compatibility,
and there were talks about doing exactly that
because the way it runs ctor code is a perfect
gadget for rop attacks and present in every
executable)

> I wonder if calling constructor functions from crt code is an intended musl
> libc behavior. My personal - non expert - gut feeling considers glibc
> behavior "more correct". Is there a chance that musl will change this

what made you think musl calls ctors from crt code?

> behavior?
> br
> Gernot

Re: Question regarding dynamic loader

[Florian Weimer]

* Gernot Reisinger:

> I recently stumbled upon an issue with preloading a shared object into
> a Go application (see related Go ticket
> https://github.com/golang/go/issues/28909).

The bug here is that Go does not call __libc_start_main at all and does
not link crt1.o.  That is a major toolchain bug.  It's surprising that
this works at all.

Thanks,
Florian