Re: Thread-local memory for thread structures

[Markus Wichmann <nullplan@gmx.net>]

On Thu, Apr 11, 2019 at 12:12:46PM +0100, Raphael Cohn wrote:
> Dear List,
>
> I'm playing around with allocating 100s of bytes of TLS memory for
> various purposes. Something I noticed in the code for creating the
> mmap'd memory for TLS is that it does not (quite reasonably) assign it
> a NUMA memory policy.
>
> I'd like to assign a NUMA memory policy to the memory used for
> managing a thread. Is there anything 'underhanded' I can do to find
> out its location and size? I realize anything is likely to be brittle.

Worse, it's going to be arch-specific.  Location of the thread-local
memory is easy, however, you just read the thread pointer. Basically,
you follow your arch's ABI to find the TLS base pointer from the thread
pointer, which must be in some kind of register.

Finding the size is less easy. In the statically linked case, you can
just read the program headers with getauxval(AT_PHDR) to find the size
to the PT_TLS segment, but in the dynamically linked case, you would
then have to add up all the sizes of the TLS segments of all the
linked-in libraries. But you don't know their base pointers, only the
libc does. Well, you /could/ parse /proc/self/maps.

But at least once you add dlopen() into the mix, the task becomes next
to entirely infeasible. You'd have to parse /proc/self/maps again after
each dlopen(), find out the libs you missed earlier, and then add their
TLS sizes to your total. And then ring up every thread to tell them the
size has changed.

> Ideally what I'd like is a 'set NUMA memory policy of this thead's
> mmap'd management memory to the local NUMA node [once I've scheduled
> it to run a particular set of CPUs].
>
> Any suggestions?
>
> Raph

If the manpage is to be trusted, then:

| The thread memory policy is preserved across an execve(2), and is
| inherited by child threads created using fork(2) or clone(2).

So, you could use set_mempolicy() to set MPOL_LOCAL before spawning
threads. Of course, that would be horrible for shared memory, but that
is a cross you have to bear.

Ciao,
Markus