Re: [musl] Re: FYI: some observations when testing next-gen malloc

[Szabolcs Nagy <nsz@port70.net>]

* Pirmin Walthert <pirmin.walthert@wwcom.ch> [2020-03-09 19:14:59 +0100]:
> Am 09.03.20 um 18:12 schrieb Rich Felker:
> > On Mon, Mar 09, 2020 at 05:49:02PM +0100, Pirmin Walthert wrote:
> > > I'd like to mention that I am not yet entirely sure whether the
> > > following is a problem with the new malloc code or with asterisk
> > > itself but maybe you can already keep the following in the back of
> > > your head if someone else is reporting similar behavior with a
> > > different application:
> > > 
> > > We use asterisk (16.7) in a musl libc based distribution and for
> > > some operations asterisk forks (in a thread) the main process to
> > > execute a system command. When using libmallocng.so (newest version
> > > with "fix race condition in lock-free path of free" applied, but
> > > already without that change) some of these forked child processes
> > > will hang during a call to pthread_mutex_unlock.
> > > 
> > > Unfortunatelly the backtrace is not of much help I guess, but the
> > > child process always seems to hang on pthread_mutex_unlock. So
> > > something seems to happen with the mutex on fork:
> > > 
> > > #0  0x00007f2152a20092 in pthread_mutex_unlock () from
> > > /lib/ld-musl-x86_64.so.1
> > > No symbol table info available.
> > > #1  0x0000000000000008 in ?? ()
> > > No symbol table info available.
> > > #2  0x0000000000000000 in ?? ()
> > > No symbol table info available.
> > > 
> > > I will for sure try to dig into this further. For the moment the
> > > only thing I know is that I did not yet observe this on any of the
> > > several hundred systems with musl 1.1.23 (same asterisk version),
> > > not on any of the around 5 with 1.2.0 (same asterisk version, old
> > > malloc) but quite frequently on the two systems with 1.1.24 and
> > > libmallocng.so.
> > This is completely expected and should happen with old or new malloc.
> > I'm surprised you haven't hit it before. After a multithreaded process
> > calls fork, the child inherits a state where locks may be permanently
> > held. See https://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html
> > 
> >      - A process shall be created with a single thread. If a
> >        multi-threaded process calls fork(), the new process shall
> >        contain a replica of the calling thread and its entire address
> >        space, possibly including the states of mutexes and other
> >        resources. Consequently, to avoid errors, the child process may
> >        only execute async-signal-safe operations until such time as one
> >        of the exec functions is called.
> > 
> > It's not described very rigorously, but effectively it's in an async
> > signal context and can only call functions which are AS-safe.
> > 
> > A future version of the standard is expected to drop the requirement
> > that fork itself be async-signal-safe, and may thereby add
> > requirements to synchronize against some or all internal locks so that
> > the child can inherit a working context. But the right solution here is
> > always to stop using fork without exec.
> > 
> > Rich
> 
> Well, I have now changed the code a bit to make sure that no
> async-signal-unsafe command is being executed before execl. Things I've
> removed:
> 
> a call to cap_from_text, cap_set_proc and cap_free has been removed as well
> as sched_setscheduler. Now the only thing being executed before execl in the
> child process is closefrom()


closefrom is not as-safe.

i think it reads /proc/self/fd directory to close fds.
(haven't checked the specific asterisk version)
opendir calls malloc so it can deadlock.

> 
> However I got a hanging process again:
> 
> (gdb) bt full
> #0  0x00007f42f649c6da in __syscall_cp_c () from /lib/ld-musl-x86_64.so.1
> No symbol table info available.
> #1  0x0000000000000000 in ?? ()
> No symbol table info available.
> 
> Best regards,
> 
> Pirmin