[musl] musl and jemalloc support

[musl] musl and jemalloc support

[Kaisrlík, Jan]

Hello guys,

I would like to ask about the support of external malloc and jemalloc.
I find an old mail in ML claiming that support is not there and using
external malloc function can potentially cause problems[1] and a
thread[2] on jemalloc saying the same. On the other hand, I find a lot
of pages comparing speed like pages [3] and also I noticed that Musl
claims "replacement of malloc is now allowed/supported" statement in
changelog [4]. So, my expectation is Musl from version 1.1.20 supports
the loading of external malloc function and it is working. Is my
assumption correct? Does it have any limitations, e.g., it has to be
done in compilation time, is LD_PRELOAD supported?

The reason why I'm asking is that some apps started crashing with a
weird coredump from the moment I used
LD_PRELOAD=/usr/lib/libjemalloc.so.2:/usr/lib/libpthread.so.

Thank you for your answer.

The best,
Jan

[1]: https://www.openwall.com/lists/musl/2014/08/08/11
[2]: https://github.com/jemalloc/jemalloc/issues/1443
[3]: https://users.rust-lang.org/t/optimizing-rust-binaries-observation-of-musl-versus-glibc-and-jemalloc-versus-system-alloc/8499/7
[4]: https://git.musl-libc.org/cgit/musl/tree/WHATSNEW#n1916

Re: [musl] musl and jemalloc support

[Szabolcs Nagy]

* Kaisrlík, Jan <jan.kaisrlik@avast.com> [2020-03-10 10:32:08 +0100]:
> I would like to ask about the support of external malloc and jemalloc.
> I find an old mail in ML claiming that support is not there and using
> external malloc function can potentially cause problems[1] and a
> thread[2] on jemalloc saying the same. On the other hand, I find a lot
> of pages comparing speed like pages [3] and also I noticed that Musl
> claims "replacement of malloc is now allowed/supported" statement in
> changelog [4]. So, my expectation is Musl from version 1.1.20 supports
> the loading of external malloc function and it is working. Is my
> assumption correct? Does it have any limitations, e.g., it has to be
> done in compilation time, is LD_PRELOAD supported?

yes, it just works.

> 
> The reason why I'm asking is that some apps started crashing with a
> weird coredump from the moment I used
> LD_PRELOAD=/usr/lib/libjemalloc.so.2:/usr/lib/libpthread.so.

the preloaded libraries obviously have to be built against musl.
(in some cases a glibc linked library would work, i would expect
a glibc linked malloc library to work with musl on x86_64, but
there is no guarantee)

the fact that you have libpthread.so means it's not a musl system
and preloading a libpthread.so from another libc is expected to
crash. (even on other systems you should not just preload libpthread,
but libjemalloc should have it in its dependencies if needed.)

> 
> Thank you for your answer.
> 
> The best,
> Jan
> 
> [1]: https://www.openwall.com/lists/musl/2014/08/08/11
> [2]: https://github.com/jemalloc/jemalloc/issues/1443
> [3]: https://users.rust-lang.org/t/optimizing-rust-binaries-observation-of-musl-versus-glibc-and-jemalloc-versus-system-alloc/8499/7
> [4]: https://git.musl-libc.org/cgit/musl/tree/WHATSNEW#n1916

Re: [musl] musl and jemalloc support

[Kaisrlík, Jan]

[-- Attachment #1: Type: text/plain, Size: 2239 bytes --]

On Tue, Mar 10, 2020 at 10:59 AM Szabolcs Nagy <nsz@port70.net> wrote:

> * Kaisrlík, Jan <jan.kaisrlik@avast.com> [2020-03-10 10:32:08 +0100]:
> > I would like to ask about the support of external malloc and jemalloc.
> > I find an old mail in ML claiming that support is not there and using
> > external malloc function can potentially cause problems[1] and a
> > thread[2] on jemalloc saying the same. On the other hand, I find a lot
> > of pages comparing speed like pages [3] and also I noticed that Musl
> > claims "replacement of malloc is now allowed/supported" statement in
> > changelog [4]. So, my expectation is Musl from version 1.1.20 supports
> > the loading of external malloc function and it is working. Is my
> > assumption correct? Does it have any limitations, e.g., it has to be
> > done in compilation time, is LD_PRELOAD supported?
>
> yes, it just works.
>

Thank you for your answer.  Glad to hear that.


>
> >
> > The reason why I'm asking is that some apps started crashing with a
> > weird coredump from the moment I used
> > LD_PRELOAD=/usr/lib/libjemalloc.so.2:/usr/lib/libpthread.so.
>
> the preloaded libraries obviously have to be built against musl.
> (in some cases a glibc linked library would work, i would expect
> a glibc linked malloc library to work with musl on x86_64, but
> there is no guarantee)
>

All my components are built against musl for armv7 architecture.


>
> the fact that you have libpthread.so means it's not a musl system
> and preloading a libpthread.so from another libc is expected to
> crash. (even on other systems you should not just preload libpthread,
> but libjemalloc should have it in its dependencies if needed.)
>

Sorry, I kept libpthread from one of my previous tests which is slightly
misleading in this case. My system has only one libpthread library coming
from musl.


>
> >
> > Thank you for your answer.
> >
> > The best,
> > Jan
> >
> > [1]: https://www.openwall.com/lists/musl/2014/08/08/11
> > [2]: https://github.com/jemalloc/jemalloc/issues/1443
> > [3]:
> https://users.rust-lang.org/t/optimizing-rust-binaries-observation-of-musl-versus-glibc-and-jemalloc-versus-system-alloc/8499/7
> > [4]: https://git.musl-libc.org/cgit/musl/tree/WHATSNEW#n1916
>

[-- Attachment #2: Type: text/html, Size: 4275 bytes --]

Re: [musl] musl and jemalloc support

[Rich Felker]

On Tue, Mar 10, 2020 at 12:08:57PM +0100, Kaisrlík, Jan wrote:
> > the fact that you have libpthread.so means it's not a musl system
> > and preloading a libpthread.so from another libc is expected to
> > crash. (even on other systems you should not just preload libpthread,
> > but libjemalloc should have it in its dependencies if needed.)
> >
> 
> Sorry, I kept libpthread from one of my previous tests which is slightly
> misleading in this case. My system has only one libpthread library coming
> from musl.

musl has no libpthread.so, only libpthread.a (which is empty). If you
have a libpthread.so, something probably went badly wrong in setting
up your system.

Rich

Re: [musl] musl and jemalloc support

[Kaisrlík, Jan]

On Tue, Mar 10, 2020 at 2:41 PM Rich Felker <dalias@libc.org> wrote:
>
> On Tue, Mar 10, 2020 at 12:08:57PM +0100, Kaisrlík, Jan wrote:
> > > the fact that you have libpthread.so means it's not a musl system
> > > and preloading a libpthread.so from another libc is expected to
> > > crash. (even on other systems you should not just preload libpthread,
> > > but libjemalloc should have it in its dependencies if needed.)
> > >
> >
> > Sorry, I kept libpthread from one of my previous tests which is slightly
> > misleading in this case. My system has only one libpthread library coming
> > from musl.
>
> musl has no libpthread.so, only libpthread.a (which is empty). If you
> have a libpthread.so, something probably went badly wrong in setting
> up your system.

Thank you for pointing this. Fortunately, it is symlink to libc.

ls -la usr/lib/libpthread.so
lrwxrwxrwx 1 X X 7 Mar  5 15:02 usr/lib/libpthread.so -> libc.so

>
> Rich

Re: [musl] musl and jemalloc support

[Rich Felker]

On Tue, Mar 10, 2020 at 03:04:48PM +0100, Kaisrlík, Jan wrote:
> On Tue, Mar 10, 2020 at 2:41 PM Rich Felker <dalias@libc.org> wrote:
> >
> > On Tue, Mar 10, 2020 at 12:08:57PM +0100, Kaisrlík, Jan wrote:
> > > > the fact that you have libpthread.so means it's not a musl system
> > > > and preloading a libpthread.so from another libc is expected to
> > > > crash. (even on other systems you should not just preload libpthread,
> > > > but libjemalloc should have it in its dependencies if needed.)
> > > >
> > >
> > > Sorry, I kept libpthread from one of my previous tests which is slightly
> > > misleading in this case. My system has only one libpthread library coming
> > > from musl.
> >
> > musl has no libpthread.so, only libpthread.a (which is empty). If you
> > have a libpthread.so, something probably went badly wrong in setting
> > up your system.
> 
> Thank you for pointing this. Fortunately, it is symlink to libc.
> 
> ls -la usr/lib/libpthread.so
> lrwxrwxrwx 1 X X 7 Mar  5 15:02 usr/lib/libpthread.so -> libc.so

That's still wrong. Recent musl will go out of its way to prevent you
from shooting yourself in the foot like that, but older musl will blow
up horribly. In either case the file should not exist.

Rich

Re: [musl] musl and jemalloc support

[Andre McCurdy]

On Tue, Mar 10, 2020 at 8:48 AM Rich Felker <dalias@libc.org> wrote:
> On Tue, Mar 10, 2020 at 03:04:48PM +0100, Kaisrlík, Jan wrote:
> > On Tue, Mar 10, 2020 at 2:41 PM Rich Felker <dalias@libc.org> wrote:
> > > On Tue, Mar 10, 2020 at 12:08:57PM +0100, Kaisrlík, Jan wrote:
> > > > > the fact that you have libpthread.so means it's not a musl system
> > > > > and preloading a libpthread.so from another libc is expected to
> > > > > crash. (even on other systems you should not just preload libpthread,
> > > > > but libjemalloc should have it in its dependencies if needed.)
> > > >
> > > > Sorry, I kept libpthread from one of my previous tests which is slightly
> > > > misleading in this case. My system has only one libpthread library coming
> > > > from musl.
> > >
> > > musl has no libpthread.so, only libpthread.a (which is empty). If you
> > > have a libpthread.so, something probably went badly wrong in setting
> > > up your system.
> >
> > Thank you for pointing this. Fortunately, it is symlink to libc.
> >
> > ls -la usr/lib/libpthread.so
> > lrwxrwxrwx 1 X X 7 Mar  5 15:02 usr/lib/libpthread.so -> libc.so
>
> That's still wrong. Recent musl will go out of its way to prevent you
> from shooting yourself in the foot like that, but older musl will blow
> up horribly. In either case the file should not exist.

OpenEmbedded provides those symlinks (although some are only as part
of a glibc compatibility package and so not generally exposed to
users):

  https://git.openembedded.org/openembedded-core/tree/meta/recipes-core/musl/musl_git.bb#n73

Is that wrong?

Re: [musl] musl and jemalloc support

[Rich Felker]

On Tue, Mar 10, 2020 at 12:52:27PM -0700, Andre McCurdy wrote:
> On Tue, Mar 10, 2020 at 8:48 AM Rich Felker <dalias@libc.org> wrote:
> > On Tue, Mar 10, 2020 at 03:04:48PM +0100, Kaisrlík, Jan wrote:
> > > On Tue, Mar 10, 2020 at 2:41 PM Rich Felker <dalias@libc.org> wrote:
> > > > On Tue, Mar 10, 2020 at 12:08:57PM +0100, Kaisrlík, Jan wrote:
> > > > > > the fact that you have libpthread.so means it's not a musl system
> > > > > > and preloading a libpthread.so from another libc is expected to
> > > > > > crash. (even on other systems you should not just preload libpthread,
> > > > > > but libjemalloc should have it in its dependencies if needed.)
> > > > >
> > > > > Sorry, I kept libpthread from one of my previous tests which is slightly
> > > > > misleading in this case. My system has only one libpthread library coming
> > > > > from musl.
> > > >
> > > > musl has no libpthread.so, only libpthread.a (which is empty). If you
> > > > have a libpthread.so, something probably went badly wrong in setting
> > > > up your system.
> > >
> > > Thank you for pointing this. Fortunately, it is symlink to libc.
> > >
> > > ls -la usr/lib/libpthread.so
> > > lrwxrwxrwx 1 X X 7 Mar  5 15:02 usr/lib/libpthread.so -> libc.so
> >
> > That's still wrong. Recent musl will go out of its way to prevent you
> > from shooting yourself in the foot like that, but older musl will blow
> > up horribly. In either case the file should not exist.
> 
> OpenEmbedded provides those symlinks (although some are only as part
> of a glibc compatibility package and so not generally exposed to
> users):
> 
>   https://git.openembedded.org/openembedded-core/tree/meta/recipes-core/musl/musl_git.bb#n73
> 
> Is that wrong?

Yes, it's wrong, and will lead to incorrect linking if these symlinks
are present when linking applications. Their presence does not help
with anything (glibc compat or otherwise) because musl's dynamic
linker intercepts these names and does not search for files.

If you really really really think you need them for some runtime
reason for some application, they should probably be empty .so files
rather than symlinks to libc, and they should be in a directory where
they won't be searched by ld at link time. But it's almost certain
that whatever the motivation for putting them there in the first place
was based on a misunderstanding.

Rich

Re: [musl] musl and jemalloc support

[Andre McCurdy]

On Tue, Mar 10, 2020 at 1:17 PM Rich Felker <dalias@libc.org> wrote:
> On Tue, Mar 10, 2020 at 12:52:27PM -0700, Andre McCurdy wrote:
> > On Tue, Mar 10, 2020 at 8:48 AM Rich Felker <dalias@libc.org> wrote:
> > > On Tue, Mar 10, 2020 at 03:04:48PM +0100, Kaisrlík, Jan wrote:
> > > > On Tue, Mar 10, 2020 at 2:41 PM Rich Felker <dalias@libc.org> wrote:
> > > > > On Tue, Mar 10, 2020 at 12:08:57PM +0100, Kaisrlík, Jan wrote:
> > > > > > > the fact that you have libpthread.so means it's not a musl system
> > > > > > > and preloading a libpthread.so from another libc is expected to
> > > > > > > crash. (even on other systems you should not just preload libpthread,
> > > > > > > but libjemalloc should have it in its dependencies if needed.)
> > > > > >
> > > > > > Sorry, I kept libpthread from one of my previous tests which is slightly
> > > > > > misleading in this case. My system has only one libpthread library coming
> > > > > > from musl.
> > > > >
> > > > > musl has no libpthread.so, only libpthread.a (which is empty). If you
> > > > > have a libpthread.so, something probably went badly wrong in setting
> > > > > up your system.
> > > >
> > > > Thank you for pointing this. Fortunately, it is symlink to libc.
> > > >
> > > > ls -la usr/lib/libpthread.so
> > > > lrwxrwxrwx 1 X X 7 Mar  5 15:02 usr/lib/libpthread.so -> libc.so
> > >
> > > That's still wrong. Recent musl will go out of its way to prevent you
> > > from shooting yourself in the foot like that, but older musl will blow
> > > up horribly. In either case the file should not exist.
> >
> > OpenEmbedded provides those symlinks (although some are only as part
> > of a glibc compatibility package and so not generally exposed to
> > users):
> >
> >   https://git.openembedded.org/openembedded-core/tree/meta/recipes-core/musl/musl_git.bb#n73
> >
> > Is that wrong?
>
> Yes, it's wrong, and will lead to incorrect linking if these symlinks
> are present when linking applications. Their presence does not help
> with anything (glibc compat or otherwise) because musl's dynamic
> linker intercepts these names and does not search for files.
>
> If you really really really think you need them for some runtime
> reason for some application, they should probably be empty .so files
> rather than symlinks to libc, and they should be in a directory where
> they won't be searched by ld at link time. But it's almost certain
> that whatever the motivation for putting them there in the first place
> was based on a misunderstanding.

Thanks. Khem created the links. Hopefully he's reading this and will
either remove them or continue the thread if anything else needs
clarification.