[musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[Askar Safin]

I found a bug both in glibc and musl.

If a process does posix_spawn+waitpid, then attempting to pause it using Ctrl-Z
sometimes doesn't work and, worse, makes the process unkillable by usual Ctrl-Z or Ctrl-C.

The bug is described in full in this glibc issue: https://sourceware.org/bugzilla/show_bug.cgi?id=32565 .

It is reproducible with musl on the same system I used to reproduce it with glibc (see the link).

I compiled the code using "x86_64-linux-musl-gcc" wrapper provided by Debian.

Please, CC me when replying.

Output of some commands:

# dpkg -l musl
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  musl:amd64     1.2.5-1.1    amd64        standard C library
# /lib/x86_64-linux-musl/libc.so 
musl libc (x86_64)
Version 1.2.5
Dynamic Program Loader
Usage: /lib/x86_64-linux-musl/libc.so [options] [--] pathname [args]

--
Askar Safin
https://types.pl/@safinaskar

Re: [musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[Rich Felker]

On Fri, Jan 17, 2025 at 03:14:03AM +0400, Askar Safin wrote:
> I found a bug both in glibc and musl.
> 
> If a process does posix_spawn+waitpid, then attempting to pause it using Ctrl-Z
> sometimes doesn't work and, worse, makes the process unkillable by usual Ctrl-Z or Ctrl-C.
> 
> The bug is described in full in this glibc issue: https://sourceware.org/bugzilla/show_bug.cgi?id=32565 .
> 
> It is reproducible with musl on the same system I used to reproduce it with glibc (see the link).
> 
> I compiled the code using "x86_64-linux-musl-gcc" wrapper provided by Debian.
> 
> Please, CC me when replying.

OK, I think this should be fixable by, if SIGTSTP is to be SIG_DFL in
the spawned child, setting it to a no-op handler instead of SIG_DFL.
It might actually make sense to just do this for all signals.

Note that SIGSTOP, which is not blockable interceptible or ignorable,
can't be handled this way, but the pid has not yet leaked to anything
at this point, so the only way SIGSTOP can be generated is by a badly
behaved program signaling random pids, which is not a case that needs
to be handled gracefully.

In theory SIGTTIN and SIGTTOU might be hazards too, but I don't think
it's possible for a process to generate them without attempting to
perform io, which the pre-exec child doesn't do. Still handling them
might be a good safety measure in case I'm wrong.

I'll prepare one or more versions of a proposed patch.

Rich
 

Re: [musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[Rich Felker]

On Fri, Jan 17, 2025 at 01:37:09AM -0500, Rich Felker wrote:
> On Fri, Jan 17, 2025 at 03:14:03AM +0400, Askar Safin wrote:
> > I found a bug both in glibc and musl.
> > 
> > If a process does posix_spawn+waitpid, then attempting to pause it using Ctrl-Z
> > sometimes doesn't work and, worse, makes the process unkillable by usual Ctrl-Z or Ctrl-C.
> > 
> > The bug is described in full in this glibc issue: https://sourceware.org/bugzilla/show_bug.cgi?id=32565 .
> > 
> > It is reproducible with musl on the same system I used to reproduce it with glibc (see the link).
> > 
> > I compiled the code using "x86_64-linux-musl-gcc" wrapper provided by Debian.
> > 
> > Please, CC me when replying.
> 
> OK, I think this should be fixable by, if SIGTSTP is to be SIG_DFL in
> the spawned child, setting it to a no-op handler instead of SIG_DFL.
> It might actually make sense to just do this for all signals.
> 
> Note that SIGSTOP, which is not blockable interceptible or ignorable,
> can't be handled this way, but the pid has not yet leaked to anything
> at this point, so the only way SIGSTOP can be generated is by a badly
> behaved program signaling random pids, which is not a case that needs
> to be handled gracefully.
> 
> In theory SIGTTIN and SIGTTOU might be hazards too, but I don't think
> it's possible for a process to generate them without attempting to
> perform io, which the pre-exec child doesn't do. Still handling them
> might be a good safety measure in case I'm wrong.
> 
> I'll prepare one or more versions of a proposed patch.

One complication I'll need to address: the pre-exec child does not
have enough stack to execute (even a no-op) signal handler. So the
parent is going to need to handle checking the runtime-variable min
signal stack and ensuring it provides enough. And the no-op signal
handler will need to be installed to run with all signals blocked so
that recursive signals can't overflow a limit that only suffices for
one signal.

With those changes I think this approach works.

I think applying it to all signals is probably a bad idea in that it
would introduce a lot more syscall cost at spawn time. Just doing the
signals we need (and probably omitting SITTTIN/TTOU unless there's
good reason to believe they can happen) seems like the smart approach
not to make the fix annoyingly costly to users.

Rich

Re: [musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[Askar Safin]

Thanks a lot for answer!

 ---- On Fri, 17 Jan 2025 10:37:09 +0400  Rich Felker  wrote --- 
 > Note that SIGSTOP, which is not blockable interceptible or ignorable,
 > can't be handled this way, but the pid has not yet leaked to anything
 > at this point, so the only way SIGSTOP can be generated is by a badly
 > behaved program signaling random pids, which is not a case that needs
 > to be handled gracefully.

But what if somebody sends SIGSTOP to whole process group using kill(2)?

The bug is not reproducible with fork. Maybe this is kernel bug and should be
forwarded there instead?

--
Askar Safin
https://types.pl/@safinaskar

Re: [musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[Florian Weimer]

* Askar Safin:

> Thanks a lot for answer!
>
>  ---- On Fri, 17 Jan 2025 10:37:09 +0400  Rich Felker  wrote --- 
>  > Note that SIGSTOP, which is not blockable interceptible or ignorable,
>  > can't be handled this way, but the pid has not yet leaked to anything
>  > at this point, so the only way SIGSTOP can be generated is by a badly
>  > behaved program signaling random pids, which is not a case that needs
>  > to be handled gracefully.
>
> But what if somebody sends SIGSTOP to whole process group using kill(2)?

I would expect that they send SIGCONT afterwards to the same process
group, to resume execution of all processes.  Doesn't this avoid the
issue?

Re: [musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[Rich Felker]

On Sat, Jan 18, 2025 at 10:51:01AM +0100, Florian Weimer wrote:
> * Askar Safin:
> 
> > Thanks a lot for answer!
> >
> >  ---- On Fri, 17 Jan 2025 10:37:09 +0400  Rich Felker  wrote --- 
> >  > Note that SIGSTOP, which is not blockable interceptible or ignorable,
> >  > can't be handled this way, but the pid has not yet leaked to anything
> >  > at this point, so the only way SIGSTOP can be generated is by a badly
> >  > behaved program signaling random pids, which is not a case that needs
> >  > to be handled gracefully.
> >
> > But what if somebody sends SIGSTOP to whole process group using kill(2)?
> 
> I would expect that they send SIGCONT afterwards to the same process
> group, to resume execution of all processes.  Doesn't this avoid the
> issue?

I mean if you just want a heuristic fix.. I guess?

But certainly they could send SIGSTOP to the group then SIGCONT only
to the single known process.

Rich

Re: [musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[Florian Weimer]

* Rich Felker:

> On Sat, Jan 18, 2025 at 10:51:01AM +0100, Florian Weimer wrote:
>> * Askar Safin:
>> 
>> > Thanks a lot for answer!
>> >
>> >  ---- On Fri, 17 Jan 2025 10:37:09 +0400  Rich Felker  wrote --- 
>> >  > Note that SIGSTOP, which is not blockable interceptible or ignorable,
>> >  > can't be handled this way, but the pid has not yet leaked to anything
>> >  > at this point, so the only way SIGSTOP can be generated is by a badly
>> >  > behaved program signaling random pids, which is not a case that needs
>> >  > to be handled gracefully.
>> >
>> > But what if somebody sends SIGSTOP to whole process group using kill(2)?
>> 
>> I would expect that they send SIGCONT afterwards to the same process
>> group, to resume execution of all processes.  Doesn't this avoid the
>> issue?
>
> I mean if you just want a heuristic fix.. I guess?
>
> But certainly they could send SIGSTOP to the group then SIGCONT only
> to the single known process.

It seems this may be the hard-to-kill scenario with SIGTSTP, too.
After the problematic ^Z, the desired signal is only delivered after
typing “fg” or equivalent in the shell, which triggers that SIGCONT.

Re: [musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[Askar Safin]

 ---- On Sat, 18 Jan 2025 15:13:57 +0400  Florian Weimer  wrote --- 
 > It seems this may be the hard-to-kill scenario with SIGTSTP, too.
 > After the problematic ^Z, the desired signal is only delivered after
 > typing “fg” or equivalent in the shell, which triggers that SIGCONT.

I don't understand what you mean here. Original issue was so:
the user presses ^Z, then (from UI point of view) stopping doesn't
happen. I. e. bash prompt doesn't appear, and thus user cannot
enter "fg"

--
Askar Safin
https://types.pl/@safinaskar

Re: [musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[Rich Felker]

On Fri, Jan 17, 2025 at 09:55:50PM +0400, Askar Safin wrote:
> Thanks a lot for answer!
> 
>  ---- On Fri, 17 Jan 2025 10:37:09 +0400  Rich Felker  wrote --- 
>  > Note that SIGSTOP, which is not blockable interceptible or ignorable,
>  > can't be handled this way, but the pid has not yet leaked to anything
>  > at this point, so the only way SIGSTOP can be generated is by a badly
>  > behaved program signaling random pids, which is not a case that needs
>  > to be handled gracefully.
> 
> But what if somebody sends SIGSTOP to whole process group using kill(2)?
> 
> The bug is not reproducible with fork. Maybe this is kernel bug and should be
> forwarded there instead?

I don't understand what you think the kernel bug is. AFAICT it's
behaving exactly as it should. The spawn impl just isn't using the
kernel primitives exactly right.

Since SIGSTOP is an issue, I don't think my previous idea of using
no-op signal handlers makes sense. We'll have to handle an unblockable
stop situation anyway so might as well use the same for blockable
stops.

So, I think the parent, while waiting for the fd close event from the
child, also needs to watch for child status change to stopped, and if
this happens, send SIGCONT. This is unfortunately gratuitously
difficult, since we don't have a primitive to wait for
fd-activity-or-pid-activity short of depending on pidf (if this even
works; does pidfd poll readable for transition to stopped state?).

If there is a working pidfd approach, I guess we should use that, and
fallback to polling with short timeout and doing a waitpid/WNOHANG
between polls if pidfd isn't obtainable.

Rich

Re: [musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[Markus Wichmann]

Hi all,

here is my understanding of the bug first.

1. Foreground process calls posix_spawn without the POSIX_SPAWN_SETSID
or POSIX_SPAWN_SETPGROUP flags (either of those prevent the bug).
2. User presses terminal suspend character between the parent process
masking signals and the child process execing the target program.
3. Kernel sends SIGTSTP to foreground process group.
4. SIGTSTP is blocked in parent process, so parent process does not
stop. Parent process is blocked in trying to read the pipe to the child,
though.
5. Child process unblocks signals before calling exec(), thereby
unblocking SIGTSTP and stopping.
6. User has an issue mainly because parent process never acts on SIGTSTP
and stops (which is why the shell's wait() call never returns).

Looking at the ingredients of the problem, it seems that unblocking
signals before reading the pipe would be the simplest way out of this
pickle. We cannot avoid blocking signals before calling clone() to spawn
the child with blocked signals, and they cannot be unblocked in exec(),
because all exec() functions pass on the signal mask, but the parent
could read the pipe with unblocked signals.

The code for reading the pipe and waiting for the child process
obviously would need to account for the possibility of EINTR, and there
is a possibility the pipe FD would escape to fork-without-exec in a
signal handler. That could be helped with FD_CLOFORK emulation in libc,
though (keep track of CLOFORK FDs in an FD set and close them all in
_Fork()), since FD_CLOFORK is not in the kernel, sadly.

Or else you could tell applications that weird things happen if you fork
in a signal handler without execing (that's weird usage, anyway).

Ciao,
Markus

Re: [musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[Rich Felker]

On Sat, Jan 18, 2025 at 09:16:58PM +0100, Markus Wichmann wrote:
> Hi all,
> 
> here is my understanding of the bug first.
> 
> 1. Foreground process calls posix_spawn without the POSIX_SPAWN_SETSID
> or POSIX_SPAWN_SETPGROUP flags (either of those prevent the bug).
> 2. User presses terminal suspend character between the parent process
> masking signals and the child process execing the target program.
> 3. Kernel sends SIGTSTP to foreground process group.
> 4. SIGTSTP is blocked in parent process, so parent process does not
> stop. Parent process is blocked in trying to read the pipe to the child,
> though.
> 5. Child process unblocks signals before calling exec(), thereby
> unblocking SIGTSTP and stopping.
> 6. User has an issue mainly because parent process never acts on SIGTSTP
> and stops (which is why the shell's wait() call never returns).
> 
> Looking at the ingredients of the problem, it seems that unblocking
> signals before reading the pipe would be the simplest way out of this
> pickle. We cannot avoid blocking signals before calling clone() to spawn
> the child with blocked signals, and they cannot be unblocked in exec(),
> because all exec() functions pass on the signal mask, but the parent
> could read the pipe with unblocked signals.

I think this is a misunderstanding of the bug. My understanding is
that, due to signals sent from a controlling terminal or to a process
group, it's posssible for a process which logically does not exist yet
to enter a stopped state.

If the parent also stopped, most likely they would get resumed
together, but there is no requirement that this happen. In a worst
case, the child stop may be queued before the child changes to a new
process group; in that case, it's acted upon after the process group
has already changed (because that necessarily happens before signals
are unblocked), and sending SIGCONT to the parent process group (like
a shell would do) will not resume it.

This cannot happen in the case of a hard SIGSTOP though, only SIGTSTP.
So one could argue that my original fix for SIGTSTP suffices, if
you're willing to assume something sending hard SIGSTOP to a process
group will send the SIGCONT to the process group as well.

> The code for reading the pipe and waiting for the child process
> obviously would need to account for the possibility of EINTR, and there
> is a possibility the pipe FD would escape to fork-without-exec in a
> signal handler. That could be helped with FD_CLOFORK emulation in libc,
> though (keep track of CLOFORK FDs in an FD set and close them all in
> _Fork()), since FD_CLOFORK is not in the kernel, sadly.

This doesn't matter. It's always expected that libc-internal fds can
escape this way, and in this case it's completely harmless except for
the resource leak. If you _Fork from a signal handler you're in a
permanent AS context, and can't really do much except exec or _exit.
So the resource usage really doesn't matter. It does not block forward
progress of anything.

> Or else you could tell applications that weird things happen if you fork
> in a signal handler without execing (that's weird usage, anyway).

This is basically what the standard already does.

I'm not really convinced that unblocking signals in the parent is
relevant to fixing this bug, but it might be a better behavior, since
posix_spawn can block forward progress indefinitely if the child file
actions do stupid things like opening a file type that blocks in open.
While the implementation may of course block signals internally where
needed, generally this should follow the as-if rule whereby the
application can't see that they were blocked except by timing
differences. Blocking forward progress that can only occur by a signal
being handled seems like at least bad QoI if not nonconforming.

Rich

Re: [musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[Askar Safin]

 ---- On Sat, 18 Jan 2025 15:17:02 +0400  Rich Felker  wrote --- 
 > Since SIGSTOP is an issue, I don't think my previous idea of using
 > no-op signal handlers makes sense. We'll have to handle an unblockable
 > stop situation anyway so might as well use the same for blockable
 > stops.

When I mentioned SIGSTOP, this was just thinking out loud. I. e. purely theoretical
concern. I didn't even test it. The original bug report was about Ctrl-Z, i. e. SIGTSTP.

--
Askar Safin
https://types.pl/@safinaskar

Re: [musl] [bug] Ctrl-Z when process is doing posix_spawn makes the process hard to kill

[Askar Safin]

 ---- On Sat, 18 Jan 2025 15:17:02 +0400  Rich Felker  wrote --- 
 > I don't understand what you think the kernel bug is.

Recently I got a suggestion to use CLONE_VM on io-uring@vger.kernel.org
( https://lore.kernel.org/io-uring/9ee30fc7-0329-4a69-b686-3131ce323c97@gmail.com/ )

So I tried CLONE_VM and it worked! I. e. this Ctrl-Z bug was not reproduced.

Also I compared various methods for spawning. And my testing shows that all
methods based on vfork or CLONE_VFORK or posix_spawn (as well as I understand it
is based on vfork, too) are buggy, and all others are not.

For all methods I wrote in comments whether the bug is reproducible on glibc
and musl.

In the end of this letter you will find full source.

So it may be good idea to replace vfork with CLONE_VM in musl and glibc.

Also, my CLONE_VM-based implementation is essentially reimplementation of vfork, but
in userspace. And it works. I. e. actual kernel implementation of vfork doesn't work,
and its userspace emulation works. This is strong argument for point of view, that
vfork is buggy in kernel.

--
Askar Safin
https://types.pl/@safinaskar

Source:

#define _GNU_SOURCE

#include <spawn.h>
#include <err.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <sched.h>
#include <sys/wait.h>

char *args[] = {"/bin/true", NULL};
char *env[] = {"HOME=/", NULL};

// repro glibc
// repro musl
pid_t
spawn_via_posix_spawn (void)
{
    pid_t pid;
    if (posix_spawn (&pid, "/bin/true", NULL, NULL, args, env) != 0)
        {
            errx (1, "posix_spawn");
        }
    return pid;
}

// not repro glibc
// not repro musl
pid_t
spawn_via_fork (void)
{
    pid_t pid = fork ();
    if (pid == -1)
        {
            err (1, "fork");
        }
    if (pid == 0)
        {
            execve ("/bin/true", args, env);
            err (1, "execve");
        }
    return pid;
}

// repro glibc
// repro musl
pid_t
spawn_via_vfork (void)
{
    pid_t pid = vfork ();
    if (pid == -1)
        {
            err (1, "vfork");
        }
    if (pid == 0)
        {
            execve ("/bin/true", args, env);
            err (1, "execve");
        }
    return pid;
}

/* Okay, so below we will emulate vfork using CLONE_VM. We will do so using O_CLOEXEC pipe.
 * We will heavily rely on one important property: during execve Linux first destroys old memory,
 * and then closes all O_CLOEXEC fds. This is actually true, as we can see in Linux source:
 * https://elixir.bootlin.com/linux/v6.13-rc3/source/fs/exec.c#L1274
 * https://elixir.bootlin.com/linux/v6.13-rc3/source/fs/exec.c#L1312
 * As you can see, do_close_on_exec is called after exec_mmap
 */

int pipe_fd[2];

int
helper (void *a)
{
    if (syscall (SYS_close, pipe_fd[0]) != 0)
        {
            syscall (SYS_write, 2, "clo", 3);
            syscall (SYS_exit_group, 1);
        }

    syscall (SYS_execve, "/bin/true", args, env);
    syscall (SYS_write, 2, "exe", 3);
    syscall (SYS_exit_group, 1);
}

// not repro glibc
// not repro musl
pid_t
spawn_via_clone_vm (void)
{
    if (pipe2 (pipe_fd, O_CLOEXEC) == -1)
        {
            err (1, "pipe2");
        }

    // Begin of code, copied from "man 2 clone"

#define STACK_SIZE (1024 * 1024)    /* Stack size for cloned child */
    char            *stack;         /* Start of stack buffer */
    char            *stackTop;      /* End of stack buffer */

    /* Allocate memory to be used for the stack of the child. */
    stack = mmap(NULL, STACK_SIZE, PROT_READ | PROT_WRITE,
                        MAP_PRIVATE | MAP_ANONYMOUS | MAP_STACK, -1, 0);
    if (stack == MAP_FAILED)
        {
            err (1, "mmap");
        }

    stackTop = stack + STACK_SIZE;  /* Assume stack grows downward */

    pid_t pid = clone (helper, stackTop, CLONE_VM | SIGCHLD, NULL);

    if (pid == -1)
        {
            err (1, "clone");
        }

    // End of code, copied from "man 2 clone"

    // Okay, so now we should wait for "execve". We will do this using that pipe
    // We will use "syscall" to avoid messing with libc's state
    // We cannot even rely on errno, because it is probably shared now

    if (syscall (SYS_close, pipe_fd[1]) != 0)
        {
            syscall (SYS_write, 2, "clo", 3);
            syscall (SYS_exit_group, 1);
        }

    char buf[1];
    if (syscall (SYS_read, pipe_fd[0], buf, 1) != 0)
        {
            syscall (SYS_write, 2, "rea", 3);
            syscall (SYS_exit_group, 1);
        }

    // Okay, so the child did "execve", now we can continue running normally

    if (close (pipe_fd[0]) != 0)
        {
            err (1, "close");
        }

    return pid;
}

int
helper_clone_vfork (void *a)
{
    execve ("/bin/true", args, env);
    err (1, "execve");
}

// repro glibc
// repro musl
pid_t
spawn_via_clone_vfork (void)
{
    // Begin of code, copied from "man 2 clone"

#define STACK_SIZE (1024 * 1024)    /* Stack size for cloned child */
    char            *stack;         /* Start of stack buffer */
    char            *stackTop;      /* End of stack buffer */

    /* Allocate memory to be used for the stack of the child. */
    stack = mmap(NULL, STACK_SIZE, PROT_READ | PROT_WRITE,
                        MAP_PRIVATE | MAP_ANONYMOUS | MAP_STACK, -1, 0);
    if (stack == MAP_FAILED)
        {
            err (1, "mmap");
        }

    stackTop = stack + STACK_SIZE;  /* Assume stack grows downward */

    pid_t pid = clone (helper, stackTop, CLONE_VFORK | SIGCHLD, NULL);

    if (pid == -1)
        {
            err (1, "clone");
        }

    // End of code, copied from "man 2 clone"

    return pid;
}

int
main (void)
{
    for (;;)
        {
            pid_t pid = spawn_via_clone_vfork (); // You can replace this line with some other "spawn_via_..." function
            if (waitpid (pid, NULL, 0) != pid)
                {
                    err(1, "waitpid");
                }
        }
}