tcmalloc compatibility

tcmalloc compatibility

[Denis V. Razumovsky]

[-- Attachment #1: Type: text/plain, Size: 755 bytes --]

Dear Sirs,

Could you please be so kind to fix muls compatibility with tcmalloc
(libtcmalloc_minimal in particular). Seems that tcmalloc developers
can't fix all issues without  corresponding changes of musl.

Small cite from tcmalloc bug-report:


>> Doesn't look like musl will ever support replacing their malloc for
dynamically linked musl.


All appropriate details you can find at the page:

https://github.com/gperftools/gperftools/issues/693


Because of this many important software can't be ported properly to
Alpine linux because they depends from libtcmalloc_minimal.


My personal competencies are not enough for that fixes. Therefore, I
appeal in support of both libraries.


Thank you in advance,


Dennis.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3587 bytes --]

Re: tcmalloc compatibility

[Rich Felker]

On Tue, Apr 10, 2018 at 01:22:54PM +0300, Denis V. Razumovsky wrote:
> Dear Sirs,
> 
> Could you please be so kind to fix muls compatibility with tcmalloc
> (libtcmalloc_minimal in particular). Seems that tcmalloc developers
> can't fix all issues without  corresponding changes of musl.

malloc interposition is undefined behavior (as is any interposition of
standard functions), and is very difficult to actually support as an
extension in a way that doesn't have lots of serious problems. This
has been discussed before but I don't have links handy. I'll try to
dig them up later. The glibc folks are also aware that it's broken (on
glibc, it only works if you get lucky and follow unwritten rules) and
would probably like a fix but it's not easy on their side either.

> Small cite from tcmalloc bug-report:
> 
> 
> >> Doesn't look like musl will ever support replacing their malloc for
> dynamically linked musl.

This claim doesn't seem to be well-justified. Myself and members of
our community have written a lot on why existing malloc interposition
hacks are broken, but there's also an interest in what would take to
make it work, and I particularly am interested in this from a
standpoint that musl's malloc is not very good, and that being able to
dynamically interpose it would facilitate developing and testing a
replacement.

Note however that if malloc interposition is supported at some point,
there will be a specification for constraints on the malloc
implementation including what functions you can call from it (e.g.
something like AS-safety), and bug reports for implementations that do
things outside this spec (and thereby inherently can't work safely or
reliably) will not be considered bugs.

> All appropriate details you can find at the page:
> 
> https://github.com/gperftools/gperftools/issues/693
> 
> 
> Because of this many important software can't be ported properly to
> Alpine linux because they depends from libtcmalloc_minimal.

This seems false. You can just remove tcmalloc from them and use the
system malloc and they will work fine (possibly after patching out
unnecessary calls into the internals of the malloc replacement, e.g.
for debug/stats/etc.).

Rich

Re: tcmalloc compatibility

[Bobby Powers]

On Tue, Apr 10, 2018 at 2:34 PM Rich Felker <dalias@libc.org> wrote:
> This claim doesn't seem to be well-justified. Myself and members of
> our community have written a lot on why existing malloc interposition
> hacks are broken, but there's also an interest in what would take to
> make it work, and I particularly am interested in this from a
> standpoint that musl's malloc is not very good, and that being able to
> dynamically interpose it would facilitate developing and testing a
> replacement.

This sounds super interesting -- what needs to happen to make progress
on this?  I would love to help out.

> Note however that if malloc interposition is supported at some point,
> there will be a specification for constraints on the malloc
> implementation including what functions you can call from it (e.g.
> something like AS-safety), and bug reports for implementations that do
> things outside this spec (and thereby inherently can't work safely or
> reliably) will not be considered bugs.

That sounds reasonable.  Some existing software (like Hoard) goes out
of its way to interpose on all functions that might call into malloc
to ensure the system allocator isn't called indirectly:

https://github.com/emeryberger/Heap-Layers/blob/master/wrappers/wrapper.cpp

yours,
Bobby

Re: tcmalloc compatibility

[Rich Felker]

On Tue, Apr 10, 2018 at 10:45:03AM -0400, Bobby Powers wrote:
> On Tue, Apr 10, 2018 at 2:34 PM Rich Felker <dalias@libc.org> wrote:
> > This claim doesn't seem to be well-justified. Myself and members of
> > our community have written a lot on why existing malloc interposition
> > hacks are broken, but there's also an interest in what would take to
> > make it work, and I particularly am interested in this from a
> > standpoint that musl's malloc is not very good, and that being able to
> > dynamically interpose it would facilitate developing and testing a
> > replacement.
> 
> This sounds super interesting -- what needs to happen to make progress
> on this?  I would love to help out.

For allowing interposition, it's mainly working out policy so that
it's clear what can and can't be supported and we don't get stuck
seemingly promising something impossible. The actual code changes are
fairly small. We'd need to switch from -Wl,-Bsymbolic-functions when
linking to -Wl,--dynamic-list in order to exclude the malloc functions
from being bound at link time, and some changes might be necessary in
the dynamic linker in how it deals with donating gaps to malloc and
early allocations before the interposed malloc is available. There's
also a question of whether the dynamic linker should have code to
detect and refuse to run with incorrect malloc interposition (some but
not all functions interposed).

But back to the point, the main issue is specifying the constraints on
the interposing functions.

> > Note however that if malloc interposition is supported at some point,
> > there will be a specification for constraints on the malloc
> > implementation including what functions you can call from it (e.g.
> > something like AS-safety), and bug reports for implementations that do
> > things outside this spec (and thereby inherently can't work safely or
> > reliably) will not be considered bugs.
> 
> That sounds reasonable.  Some existing software (like Hoard) goes out
> of its way to interpose on all functions that might call into malloc
> to ensure the system allocator isn't called indirectly:
> 
> https://github.com/emeryberger/Heap-Layers/blob/master/wrappers/wrapper.cpp

This is really impossible to do correctly, for multiple reasons:

1. Some such functions are fundamentally not replacable, like the
   dynamic linker functions (dlopen).

2. There is no specification for which libc functions call into
   malloc; this is an implementation detail. The only related things
   that are parts of the public contract are whether they return
   memory "as if by malloc" and whether they're AS-safe or AC-safe (in
   which case it's not formally correct, but it's fairly reasonable to
   assume they don't call malloc). For example on glibc, qsort and
   printf call malloc (but qsort has, and necessarily has to have, a
   fallback for when it fails since qsort cant' fail).

3. Some functions which use malloc are sufficiently heavy that you'd
   be replacing (and possibly changing or reducing functionality in)
   whole major libc components if you wanted to replace them. For
   example, getaddrinfo (the whole resolver infrastructure), iconv,
   regex, ...

Note that, unless the malloc replacement and the system malloc somehow
step on each other's state, there's no harm in having both present and
getting called as long as the libc functions that return memory "as if
by malloc" (thus that's permissible to pass to realloc and free) use
the interposed malloc replacement. This would just be things like
strdup. So if you only replace those, it's a more managable task. But
I still think it's a wrong approach.

If musl does add support for malloc interposition, I'm strongly
leaning towards using the interposed malloc everywhere so that this
kind of issue does not matter. Otherwise there are too many
opportunities for subtle errors. The main argument for not calling the
interposed malloc from libc except when you have to is that you don't
have to deal with reentrancy & inconsistent state issues that could be
prone to incorrect usage, but getline() inherently has to return
memory as if by malloc, and thus you're already stuck with at least
one function that has to call the interposed malloc with stdio locks
held (or else work in a temp buffer managed by internal malloc, then
only move to a public-malloc-allocated buffer after finishing, but
that's an awful hack, and imposing libc implementation constraints
like that around the allowance for interposing malloc is exactly the
type of nasty situation I don't want to get into).

Rich

Re: tcmalloc compatibility

[Rich Felker]

On Tue, Apr 10, 2018 at 07:53:46PM +0200, Florian Weimer wrote:
> * Rich Felker:
> 
> > malloc interposition is undefined behavior (as is any interposition of
> > standard functions), and is very difficult to actually support as an
> > extension in a way that doesn't have lots of serious problems. This
> > has been discussed before but I don't have links handy. I'll try to
> > dig them up later. The glibc folks are also aware that it's broken (on
> > glibc, it only works if you get lucky and follow unwritten rules)
> 
> We have some documentation nowadays:
> 
> <https://www.gnu.org/software/libc/manual/html_node/Replacing-malloc.html>

Thanks. This is useful information. I'm a bit concerned about the
status of memalign, etc. though as described there. If malloc is
replaced but the memalign family is not, calls to whichever function
is not replaced will either use the system malloc (producing a result
that's unsafe to pass to free) or will call the interposed malloc then
assume they can doctor its book-keeping structures as if they matched
the system malloc. Neither of these is safe.

In musl, what we should probably do is have extra weak+hidden aliases,
so that memalign can do something like:

	if (malloc != __internal_malloc) return 0;

Maybe something similar would be appropriate in glibc?

> The remaining undocumented aspects concern cyclic dependencies, such
> as the suitability of certain TLS models for implementing a custom
> malloc, or using memory-allocating glibc functions such as fopen or
> backtrace from the allocator itself.
> 
> In practice,

The "in practice" here imposes strong constraints on implementation
internals -- things could stop working at any time in the future if
changes are made to ways/places in which malloc can be called.
This could especially bite glibc if further alloca/large stack usage
is replaced with malloc.

> malloc interposition works extremely well and causes few
> issues due to interposition itself.  Obviously, there are bugs, but
> most of them would still be bugs if the allocator was non-interposing.
> (Examples are lots of initial-exec TLS data, and incorrect alignment
> for allocations.)
> 
> I believe musl uses less malloc internally, so it should be even more
> compatible with an interposing malloc implementation than glibc.

Indeed. But while musl uses less malloc, it also tries to be more
opaque and provide fewer guarantees about implementation internals,
leaving more free to change.

There are some functions that musl makes AS-safe, beyond what POSIX
requires, where I intended for the safety to actually be a public
property, and I think those would be a good basis for what's allowed
to be called from a malloc implementation, if we actually take the
effort to document them. This would include a lot of useful things
like dprintf, snprintf, mutex lock and (non-robust-type) unlock,
semaphore wait (post is already AS-safe by POSIX), ...

Unfortunately it doesn't even include mmap/munmap right now, due to
tricks we're doing to work around the unsafety (inherent race
conditions) in Linux's robust mutex support (I think glibc bug #14485
is related). I don't think this actually affects malloc though, and
mmap/munmap could just be explicitly allowed for use in malloc even
without documenting them as AS-safe.

I think allowing any use of stdio (meaning functions that actually
operate on FILEs, not counting snprintf, etc.) from malloc is a bad
idea. Some functions (getline/getdelim) use malloc and some FILE types
will necessarily (memstream) or potentially (cookie) involve malloc in
potentially complex ways.

A more interesting question is whether pthread functions, especially
pthread_create (e.g. background thread managing migration between
heaps) are permissible. They seem useful but also hard to guarantee
safe to use in this context.

Rich

Re: tcmalloc compatibility

[Szabolcs Nagy]

* Florian Weimer <fw@deneb.enyo.de> [2018-04-10 19:53:46 +0200]:
> We have some documentation nowadays:
> 
> <https://www.gnu.org/software/libc/manual/html_node/Replacing-malloc.html>
> 
> The remaining undocumented aspects concern cyclic dependencies, such
> as the suitability of certain TLS models for implementing a custom
> malloc, or using memory-allocating glibc functions such as fopen or
> backtrace from the allocator itself.
> 

it's not documented what the interposed malloc may or may not do.

cyclic dependency (because malloc calls a memory-allocating libc
function) is one issue, but in general the libc can be in an
inconsistent state at the point of an internal malloc call (e.g.
some lock is held that should never be held when user code is
running) and thus certain other libc functions may not operate
as expected in an interposer.

documenting the set of libc apis that work as expected in an
interposer is difficult.

> In practice, malloc interposition works extremely well and causes few
> issues due to interposition itself.  Obviously, there are bugs, but
> most of them would still be bugs if the allocator was non-interposing.
> (Examples are lots of initial-exec TLS data, and incorrect alignment
> for allocations.)
> 

even with the very small set of libc apis that a malloc
interposer may want to use, there known (but undocumented)
caveats:

- glibc dlsym calls calloc (bites anybody who tries to wrap
calloc in the usual way)

- general dynamic tls may call malloc and uses global dl
lock so using tls can cause recursion or deadlock.

- using stdio for logging is problematic because of stdio
locks and recursive malloc calls.

- malloc may get called before the first ctor and after
the last dtor.

- malloc may be called on a thread with small stack
(e.g. gai helper thread) so it should not have excessive
stack usage.

- glibc fork tries to reset the malloc state before
calling the child atfork handler in an attempt to support
non-as-safe fork handlers, code that relies on this can
get broken with malloc interposition.

interposition only seems to work extremely well because
users already fixed their code and things dont change
much in this space.

> I believe musl uses less malloc internally, so it should be even more
> compatible with an interposing malloc implementation than glibc.

Re: tcmalloc compatibility

[Rich Felker]

On Tue, Apr 10, 2018 at 10:33:55PM +0200, Szabolcs Nagy wrote:
> * Florian Weimer <fw@deneb.enyo.de> [2018-04-10 19:53:46 +0200]:
> > We have some documentation nowadays:
> > 
> > <https://www.gnu.org/software/libc/manual/html_node/Replacing-malloc.html>
> > 
> > The remaining undocumented aspects concern cyclic dependencies, such
> > as the suitability of certain TLS models for implementing a custom
> > malloc, or using memory-allocating glibc functions such as fopen or
> > backtrace from the allocator itself.
> > 
> 
> it's not documented what the interposed malloc may or may not do.
> 
> cyclic dependency (because malloc calls a memory-allocating libc
> function) is one issue, but in general the libc can be in an
> inconsistent state at the point of an internal malloc call (e.g.
> some lock is held that should never be held when user code is
> running) and thus certain other libc functions may not operate
> as expected in an interposer.
> 
> documenting the set of libc apis that work as expected in an
> interposer is difficult.

I think limiting to AS-safe plus a small list of additional allowed
functions is reasonably easy to guarantee works.

> > In practice, malloc interposition works extremely well and causes few
> > issues due to interposition itself.  Obviously, there are bugs, but
> > most of them would still be bugs if the allocator was non-interposing.
> > (Examples are lots of initial-exec TLS data, and incorrect alignment
> > for allocations.)
> 
> even with the very small set of libc apis that a malloc
> interposer may want to use, there known (but undocumented)
> caveats:
> 
> - glibc dlsym calls calloc (bites anybody who tries to wrap
> calloc in the usual way)

Yes. While I think it's unreasonable that glibc calls calloc from
dlsym, I also think it's unreasonable to be calling dlsym from a
malloc replacement. I don't think I would include dlsym or other
heavy, AS-unsafe functions in an allowed-list for musl.

> - general dynamic tls may call malloc and uses global dl
> lock so using tls can cause recursion or deadlock.

Yes but this is a known glibc bug that's hopefully on the way to
being fixed. A robust/correct implementation (where TLS works in
signal handlers and without possibility of OOM-crashing) can't
allocate at access time.

> - using stdio for logging is problematic because of stdio
> locks and recursive malloc calls.

Yes.

> - malloc may get called before the first ctor and after
> the last dtor.

An implementation could avoid this, but yes, I agree, there should be
no contract that it won't.

> - malloc may be called on a thread with small stack
> (e.g. gai helper thread) so it should not have excessive
> stack usage.

Certainly increasing the default thread stack size could be a
requirement to use malloc replacements that have large stack usage,
but I tend to doubt that would happen. If malloc has large stack
usage, it's likely to be very slow.

> - glibc fork tries to reset the malloc state before
> calling the child atfork handler in an attempt to support
> non-as-safe fork handlers, code that relies on this can
> get broken with malloc interposition.

Interposed mallocs probably try to support this with pthread_atfork,
but state after MT-fork is permanently an AS context anyway.

> interposition only seems to work extremely well because
> users already fixed their code and things dont change
> much in this space.

Maybe. :)

Rich

Re: tcmalloc compatibility

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2018-04-10 16:44:11 -0400]:
> On Tue, Apr 10, 2018 at 10:33:55PM +0200, Szabolcs Nagy wrote:
> > - glibc dlsym calls calloc (bites anybody who tries to wrap
> > calloc in the usual way)
> 
> Yes. While I think it's unreasonable that glibc calls calloc from
> dlsym, I also think it's unreasonable to be calling dlsym from a
> malloc replacement. I don't think I would include dlsym or other
> heavy, AS-unsafe functions in an allowed-list for musl.

then the wrappers with dlsym(RTLD_NEXT,sym) would not work.
(malloc checkers, valgrind, sanitizers etc all do it)

in glibc there is __libc_calloc so you can get the address
without dlsym, but in general you cannot wrap libc functions
for instrumentation if dlsym is disallowed.

i guess one could run dlsym early to initialize global
pointers but then the interceptor still has to handle the
case when it gets called before the appropriate dlsym is done,
so users need to know what dlsym might call and how to do the
initialization early enough.

Re: tcmalloc compatibility

[ardi]

On Tue, Apr 10, 2018 at 11:17 PM, Szabolcs Nagy <nsz@port70.net> wrote:
>
> then the wrappers with dlsym(RTLD_NEXT,sym) would not work.
> (malloc checkers, valgrind, sanitizers etc all do it)

I've been using ElectricFence, as my only memory debugger since 1996
or so; mostly with the libc of commercial Unices, but also with glibc
in Linux, and with the OSX libc. I never considered I could run into
the issues commented in this thread, and in fact I never faced these
issues and it always worked as expected (however, I must admit I only
use multithreading for accelerating clearly isolated math-intensive
loops that don't call malloc-related functions from inside the loop).

Said this, when I'm linking with ElectricFence, my brain has the "hack
mode flag" ON (I mean, I always had the feeling that I'm working with
a temporary hack that can fail whenever my link line contains -lefence
, and I'm aware that things can go wrong --I didn't consider thread
safety, but anyway I know ElectricFence can fail if the OS syscalls
that allocate protected memory at buffer ends change their behaviour
in newer versions, or if there's some OS/CPU-dependent subtlety with
alignment, etc...)

I've not tried to use ElectricFence with musl yet... but reading this,
can I suppose it won't work? Is there any "hack mode ON" procedure
(yet easy) that would allow to use ElectricFence (assuming
non-threaded code, which is always my case).

I agree with your commitment to correctness, and I'm not asking for a
safe and guaranteed implementation of function interposition, just
that sometimes I need to break my binaries to make them crash hard as
soon as pointer accesses a byte it shouldn't access.

Cheers,

ardi

Re: tcmalloc compatibility

[Markus Wichmann]

On Sun, Apr 15, 2018 at 01:52:10PM +0200, ardi wrote:
> On Tue, Apr 10, 2018 at 11:17 PM, Szabolcs Nagy <nsz@port70.net> wrote:
> >
> > then the wrappers with dlsym(RTLD_NEXT,sym) would not work.
> > (malloc checkers, valgrind, sanitizers etc all do it)
> 
> I've been using ElectricFence, as my only memory debugger since 1996
> or so; mostly with the libc of commercial Unices, but also with glibc
> in Linux, and with the OSX libc. I never considered I could run into
> the issues commented in this thread, and in fact I never faced these
> issues and it always worked as expected (however, I must admit I only
> use multithreading for accelerating clearly isolated math-intensive
> loops that don't call malloc-related functions from inside the loop).
> 
> Said this, when I'm linking with ElectricFence, my brain has the "hack
> mode flag" ON (I mean, I always had the feeling that I'm working with
> a temporary hack that can fail whenever my link line contains -lefence
> , and I'm aware that things can go wrong --I didn't consider thread
> safety, but anyway I know ElectricFence can fail if the OS syscalls
> that allocate protected memory at buffer ends change their behaviour
> in newer versions, or if there's some OS/CPU-dependent subtlety with
> alignment, etc...)
> 
> I've not tried to use ElectricFence with musl yet... but reading this,
> can I suppose it won't work? Is there any "hack mode ON" procedure
> (yet easy) that would allow to use ElectricFence (assuming
> non-threaded code, which is always my case).
> 
> I agree with your commitment to correctness, and I'm not asking for a
> safe and guaranteed implementation of function interposition, just
> that sometimes I need to break my binaries to make them crash hard as
> soon as pointer accesses a byte it shouldn't access.
> 
> Cheers,
> 
> ardi

So long as you refrain from using dynamic linking (because of the memory
donation) and calloc() and memalign() (and posix_memalign()) are unused
or overloaded, you should be fine. Both of these functions use the
internal bookkeeping of musl's malloc. calloc() uses it to figure out if
a chunk was mmapped (in which case no initialization is necessary), and
memalign() uses it to construct a second chunk header to cause the
returned pointer to be aligned.

Most of the questioning here arose from that first part. Those are the
two big problems, actually, we need an interface to donate memory to the
malloc implementation, and the malloc implementation needs to provide
all of the hairier functions like memalign(). And we currently have no
way of enforcing either of these.

Ciao,
Markus

Re: tcmalloc compatibility

[Rich Felker]

On Mon, Apr 16, 2018 at 06:19:24AM +0200, Markus Wichmann wrote:
> On Sun, Apr 15, 2018 at 01:52:10PM +0200, ardi wrote:
> > On Tue, Apr 10, 2018 at 11:17 PM, Szabolcs Nagy <nsz@port70.net> wrote:
> > >
> > > then the wrappers with dlsym(RTLD_NEXT,sym) would not work.
> > > (malloc checkers, valgrind, sanitizers etc all do it)
> > 
> > I've been using ElectricFence, as my only memory debugger since 1996
> > or so; mostly with the libc of commercial Unices, but also with glibc
> > in Linux, and with the OSX libc. I never considered I could run into
> > the issues commented in this thread, and in fact I never faced these
> > issues and it always worked as expected (however, I must admit I only
> > use multithreading for accelerating clearly isolated math-intensive
> > loops that don't call malloc-related functions from inside the loop).
> > 
> > Said this, when I'm linking with ElectricFence, my brain has the "hack
> > mode flag" ON (I mean, I always had the feeling that I'm working with
> > a temporary hack that can fail whenever my link line contains -lefence
> > , and I'm aware that things can go wrong --I didn't consider thread
> > safety, but anyway I know ElectricFence can fail if the OS syscalls
> > that allocate protected memory at buffer ends change their behaviour
> > in newer versions, or if there's some OS/CPU-dependent subtlety with
> > alignment, etc...)
> > 
> > I've not tried to use ElectricFence with musl yet... but reading this,
> > can I suppose it won't work? Is there any "hack mode ON" procedure
> > (yet easy) that would allow to use ElectricFence (assuming
> > non-threaded code, which is always my case).
> > 
> > I agree with your commitment to correctness, and I'm not asking for a
> > safe and guaranteed implementation of function interposition, just
> > that sometimes I need to break my binaries to make them crash hard as
> > soon as pointer accesses a byte it shouldn't access.
> > 
> > Cheers,
> > 
> > ardi
> 
> So long as you refrain from using dynamic linking (because of the memory
> donation)

This is only a small part of the reason you can't use dynamic linking.
The other big part is that references in libc.so are bound at libc.so
link time, so functions like getline, open_memstream, strdup, etc.
will return pointers that won't be valid for you to free.

> and calloc() and memalign() (and posix_memalign()) are unused
> or overloaded, you should be fine. Both of these functions use the
> internal bookkeeping of musl's malloc. calloc() uses it to figure out if
> a chunk was mmapped (in which case no initialization is necessary), and
> memalign() uses it to construct a second chunk header to cause the
> returned pointer to be aligned.

Yes, but this rule always applies for interposing, with any
implementation. It's not musl-specific.

> Most of the questioning here arose from that first part. Those are the
> two big problems, actually, we need an interface to donate memory to the
> malloc implementation,

This isn't needed. It's fine for donation to donate to the internal
(unused) implementation if malloc is interposed, or for donation not
to happen at all. I don't think it's a good idea to create a public
interposable API for donation.

The big thing that does need to happen is getting rid of the call to
free() to do the donation, which is unsafe/incorrect if it's
interposed. Alexander Monakov's patch (which looks ok to commit with
minor changes described in the thread) should fix that.

> and the malloc implementation needs to provide
> all of the hairier functions like memalign(). And we currently have no
> way of enforcing either of these.

A way to enforce this was discussed earlier in the thread, so it looks
doable.

Rich

Re: tcmalloc compatibility

[Rich Felker]

On Mon, Apr 16, 2018 at 12:40:06AM -0400, Rich Felker wrote:
> On Mon, Apr 16, 2018 at 06:19:24AM +0200, Markus Wichmann wrote:
> > On Sun, Apr 15, 2018 at 01:52:10PM +0200, ardi wrote:
> > [...]
> > 
> > So long as you refrain from using dynamic linking (because of the memory
> > donation)
> 
> This is only a small part of the reason you can't use dynamic linking.
> The other big part is that references in libc.so are bound at libc.so
> link time, so functions like getline, open_memstream, strdup, etc.
> will return pointers that won't be valid for you to free.
> 
> > and calloc() and memalign() (and posix_memalign()) are unused
> > or overloaded, you should be fine. Both of these functions use the
> > internal bookkeeping of musl's malloc. calloc() uses it to figure out if
> > a chunk was mmapped (in which case no initialization is necessary), and
> > memalign() uses it to construct a second chunk header to cause the
> > returned pointer to be aligned.
> 
> Yes, but this rule always applies for interposing, with any
> implementation. It's not musl-specific.
> 
> > Most of the questioning here arose from that first part. Those are the
> > two big problems, actually, we need an interface to donate memory to the
> > malloc implementation,
> 
> This isn't needed. It's fine for donation to donate to the internal
> (unused) implementation if malloc is interposed, or for donation not
> to happen at all. I don't think it's a good idea to create a public
> interposable API for donation.
> 
> The big thing that does need to happen is getting rid of the call to
> free() to do the donation, which is unsafe/incorrect if it's
> interposed. Alexander Monakov's patch (which looks ok to commit with
> minor changes described in the thread) should fix that.
> 
> > and the malloc implementation needs to provide
> > all of the hairier functions like memalign(). And we currently have no
> > way of enforcing either of these.
> 
> A way to enforce this was discussed earlier in the thread, so it looks
> doable.

Today I pushed changes which should make malloc
replacement/interposition work reliably as long as you only use
AS-safe functions. If you try this, please let us know how it turns
out and if you run into any unexpected problems.

Rich

Re: tcmalloc compatibility

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2018-04-18 16:35:56 -0400]:
> Today I pushed changes which should make malloc
> replacement/interposition work reliably as long as you only use
> AS-safe functions. If you try this, please let us know how it turns
> out and if you run into any unexpected problems.

there was a report on irc that this change in memalign has falsepositives:

-       if (len > SIZE_MAX - align) {
+       if (len > SIZE_MAX - align || free != __internal_free) {

in particular &free can point to a plt entry in the main executable
while &__internal_free is in libc.so so they compare unequal.

Re: tcmalloc compatibility

[Rich Felker]

On Thu, Apr 19, 2018 at 08:27:44PM +0200, Szabolcs Nagy wrote:
> * Rich Felker <dalias@libc.org> [2018-04-18 16:35:56 -0400]:
> > Today I pushed changes which should make malloc
> > replacement/interposition work reliably as long as you only use
> > AS-safe functions. If you try this, please let us know how it turns
> > out and if you run into any unexpected problems.
> 
> there was a report on irc that this change in memalign has falsepositives:
> 
> -       if (len > SIZE_MAX - align) {
> +       if (len > SIZE_MAX - align || free != __internal_free) {
> 
> in particular &free can point to a plt entry in the main executable
> while &__internal_free is in libc.so so they compare unequal.

Indeed, this will need to be reworked. I'll probably remove these
checks for now so that git master is not broken.

Rich

Re: tcmalloc compatibility

[Florian Weimer]

* Szabolcs Nagy:

> even with the very small set of libc apis that a malloc
> interposer may want to use, there known (but undocumented)
> caveats:
>
> - glibc dlsym calls calloc (bites anybody who tries to wrap
> calloc in the usual way)

That's not what I meant with an interposed malloc.  Any such partial
interposition is going to be very tricky indeed.

Re: tcmalloc compatibility

[Rich Felker]

On Wed, Apr 18, 2018 at 10:50:24PM +0200, Florian Weimer wrote:
> * Szabolcs Nagy:
> 
> > even with the very small set of libc apis that a malloc
> > interposer may want to use, there known (but undocumented)
> > caveats:
> >
> > - glibc dlsym calls calloc (bites anybody who tries to wrap
> > calloc in the usual way)
> 
> That's not what I meant with an interposed malloc.  Any such partial
> interposition is going to be very tricky indeed.

I don't understand. By "partial interposition" do you mean partial in
the sense of "wrapping and calling the original via dlsym" or in the
sense of "replacing some but not all allocator functions"?

Rich