=?gb18030?B?UmU6IFJlOiBbbXVzbF0gob5MaW5rZXKhv0RvZXMgTVVTTCBzdXBwb3J0IHVzaW5nIFRMUyBmb3IgYm90aCBsaWJjIGFuZCBhcHAgZGVwZW5kZW5jaWVzPw==?=

=?gb18030?B?UmU6IFJlOiBbbXVzbF0gob5MaW5rZXKhv0RvZXMgTVVTTCBzdXBwb3J0IHVzaW5nIFRMUyBmb3IgYm90aCBsaWJjIGFuZCBhcHAgZGVwZW5kZW5jaWVzPw==?=

[=?gb18030?B?ODQ3NTY3MTYx?=]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="gb18030", Size: 3485 bytes --]

> Indeed, there currently is not support for that. It's kinda an
> omission that should be remedied, as someday we may want to support
> softfloat with fenv where TLS might come into play through linking the
> compiler runtime (libgcc.a) that contains thread-local floating point
> state. But for the time being that's not something anyone seems to be
> interested in doing.

We discovered the problem when integrating gwp_asan into musl libc.
gwp_asan is a static library that used tls of init-exec mode, It can't work with tsd for some reason.
So we think it is necessary for libc to support tls.

> That would also be the answer to your question: If the TLS is too large
> for builtin_tls, it gets allocated (which happens in line 2017).
> builtin_tls is merely an optimization to ensure small TLS sections don't
> need an allocation that can fail. This can also be understood as a
> quality improvement measure since this way, most applications cannot
> randomly on startup for TLS allocation reasons when the free memory runs
> low. Whatever good that does in an environment where the kernel gets to
> arbitrarily kill any process it wants to.

Why do we set tp in Stage 2b, is builtin_tls used before here? Can we set it later£¿
https://github.com/bminor/musl/blob/master/ldso/dynlink.c#L2034

> For static linking, main thread TLS is allocated with mmap in
> static_init_tls(). In that case there is at most 1 TLS module.
>
> >
> > 2¡¢About libc and other so use tls at the same time
> > I didn¡¯t see musl modify tls_offset when ldso uses tls, so when another so uses tls later, their tls offsets will conflict.

> >If we were ever to support TLS in libc/ldso, I think it would always
> > be accessed via the global-dynamic model, so that it doesn't have to
> >move; its DTV slot number would just get changed.

Why can't we support locl-dynamic and init-exec modes in libc?
Do you mean that 'move' refers to the dlopen scene? Could you give more details?

>As to the question: Libc itself cannot use TLS. It could in the static
>linking case (when the TLS just gets rolled into the same section as the
>application TLS), but not in the dynamic one. The dynlinker currently
>does not set up TLS in libc correctly. Not entirely familiar with the
>list of things that would be needed to allow libc to have TLS, but it is
>likely to be nontrivial. I already foresee an order-of-operations
>problem. See, the dynlinker on startup currently works like this:

>Stage 1: Process libc relative relocations
>Stage 2a: Process libc symbolic relocations
>Stage 2b: Load initial thread pointer
>Stage 3: Load dependencies, process all of the relocations.

>With TLS in libc, stage 2a would already encounter relocations
>referencing the TLS which gets allocated in stage 3 at the earliest,
>because that's when the allocator becomes available. A gordian knot.

Some of our ideas£º
1¡¢Anyway, we need to put the app's tls in the first block.
2¡¢We can put the libc tls in the first or second part, depending on whether the app uses tls or not.
3¡¢More details£º
  In Stage 2a, we can use aux to see whether the app uses tls and decode the tls size of the app.
  Then we can know the tls offset of libc, it can be used for libc symbolic relocations.
  Skip processing of libc in subsequent processing of TLS functions£¬like update_tls_size¡¢assign_tls and so on.

Re: Re: [musl] 【Linker】Does MUSL support using TLS for both libc and app dependencies?

[Rich Felker]

On Mon, Jan 22, 2024 at 10:28:49AM +0800, 847567161 wrote:
> > Indeed, there currently is not support for that. It's kinda an
> > omission that should be remedied, as someday we may want to support
> > softfloat with fenv where TLS might come into play through linking the
> > compiler runtime (libgcc.a) that contains thread-local floating point
> > state. But for the time being that's not something anyone seems to be
> > interested in doing.
> 
> We discovered the problem when integrating gwp_asan into musl libc.
> gwp_asan is a static library that used tls of init-exec mode, It can't work with tsd for some reason.
> So we think it is necessary for libc to support tls.

I wasn't sure what gwp_asan was so I looked it up, and it appears not
to actually be real asan, just a wrapper for detecting malloc usage
errors (overflows, uaf/df, etc.). I'm not sure what you expect to gain
out of putting it in libc. There is no nontrivial malloc usage in libc
itself, and on top of that, mallocng already detects the same types of
errors gwp_asan seems to be documented to detect.

If you really want to try this, you can probably just put a member in
the struct __pthread and have the gwp_asan code access its
thread-local data via __pthread_self()->new_member..

But I don't see a good motivation for doing any of this.

> > That would also be the answer to your question: If the TLS is too large
> > for builtin_tls, it gets allocated (which happens in line 2017).
> > builtin_tls is merely an optimization to ensure small TLS sections don't
> > need an allocation that can fail. This can also be understood as a
> > quality improvement measure since this way, most applications cannot
> > randomly on startup for TLS allocation reasons when the free memory runs
> > low. Whatever good that does in an environment where the kernel gets to
> > arbitrarily kill any process it wants to.
> 
> Why do we set tp in Stage 2b, is builtin_tls used before here?

Yes. Basically all libc functions use TLS because they set errno on
error.

> Can we set it later？
> https://github.com/bminor/musl/blob/master/ldso/dynlink.c#L2034

No, you'll crash very early if it's not setup at all.


> > For static linking, main thread TLS is allocated with mmap in
> > static_init_tls(). In that case there is at most 1 TLS module.
> >
> > >
> > > 2、About libc and other so use tls at the same time
> > > I didn’t see musl modify tls_offset when ldso uses tls, so when another so uses tls later, their tls offsets will conflict.
> 
> > >If we were ever to support TLS in libc/ldso, I think it would always
> > > be accessed via the global-dynamic model, so that it doesn't have to
> > >move; its DTV slot number would just get changed.
> 
> Why can't we support locl-dynamic and init-exec modes in libc?

Using initial-exec from shared libs is a hack, not the intended usage.
In this case it can work, but as you've noticed it would require
moving the libc TLS after setting up the main program's TLS, whose
location relative to the thread pointer is fixed by the local-exec
model used in the main program. This is even more of a hack, because
it causes objects that have already been seen to change address. And
it has very little value. With TLSDESC, GD model is not noticably
slow.

> Do you mean that 'move' refers to the dlopen scene? Could you give more details?

No, just that whatever address the libc TLS got placed at in stage 2b
would not necessarily be a valid place to keep it after loading the
main program in stage 3.

> >As to the question: Libc itself cannot use TLS. It could in the static
> >linking case (when the TLS just gets rolled into the same section as the
> >application TLS), but not in the dynamic one. The dynlinker currently
> >does not set up TLS in libc correctly. Not entirely familiar with the
> >list of things that would be needed to allow libc to have TLS, but it is
> >likely to be nontrivial. I already foresee an order-of-operations
> >problem. See, the dynlinker on startup currently works like this:
> 
> >Stage 1: Process libc relative relocations
> >Stage 2a: Process libc symbolic relocations
> >Stage 2b: Load initial thread pointer
> >Stage 3: Load dependencies, process all of the relocations.
> 
> >With TLS in libc, stage 2a would already encounter relocations
> >referencing the TLS which gets allocated in stage 3 at the earliest,
> >because that's when the allocator becomes available. A gordian knot.
> 
> Some of our ideas：
> 1、Anyway, we need to put the app's tls in the first block.
> 2、We can put the libc tls in the first or second part, depending on whether the app uses tls or not.
> 3、More details：
>   In Stage 2a, we can use aux to see whether the app uses tls and decode the tls size of the app.
>   Then we can know the tls offset of libc, it can be used for libc symbolic relocations.
>   Skip processing of libc in subsequent processing of TLS functions，like update_tls_size、assign_tls and so on.

You can't see the app at stage 2a or 2b. In the general case, the main
app has not even been loaded into memory until line 1906 (ldso
executed as a command rather than via the kernel).

Rich