[musl] More thoughts on wrapping signal handling

[musl] More thoughts on wrapping signal handling

[Rich Felker]

In "Re: [musl] Re: [PATCH] Make abort() AS-safe (Bug 26275)."
(20201010002612.GC17637@brightrain.aerifal.cx,
https://www.openwall.com/lists/musl/2020/10/10/1) I raised the
longstanding thought of having libc wrap signal handling. This is a
little bit of a big hammer for what it was proposed for -- fixing an
extremely-rare race between abort and execve -- but today I had a
thought about another use of it that's really compelling.

What I noted before was that, by wrapping signal handlers, libc could
implement a sort of "rollback" to restart a critical section that was
interrupted. However this really only has any use when the critical
section has no side effects aside from its final completion, and
except for execve where replacement of the process gives the atomic
cutoff for rollback, it requires __cp_end-like asm label of the end of
the critical section. So it's of limited utility.

However, what's more interesting than restarting the critical section
when a signal is received is *allowing it to complete* before handling
the signal. This can be implemented by having the wrapper, upon seeing
that it interrupted a critical section, save the siginfo_t in TLS and
immediately return, leaving signals blocked, without executing the
application-installed signal handler. Then, when leaving the critical
section, the unlock function can see the saved siginfo_t and call the
application's signal handler. Effectively, it's as if the signal were
just blocked until the end of the critical section.

What is the value in this?

1. It eliminates the need for syscalls to mask and unmask signals
   around all existing AS-safe locks and critical sections that can't
   safely be interrupted by application code.

2. It makes it so we can make almost any function that was AS-unsafe
   due to locking AS-safe, without any added cost. Even malloc can be
   AS-safe.

3. It makes it so a signal handler that fails to return promptly in
   one thread can't arbitrarily delay other threads waiting for
   libc-internal locks, because application code never interrupts our
   internal critical sections.

This last property, #3, is the really exciting one -- it means that,
short of swapping etc. (e.g. with mlockall and other realtime measures
taken) most libc locks can be considered as held only for very small
bounded time, rather than potentially-unbounded due to interruption by
signal.

I'm not sure if this is something worth pursuing, and certainly not in
the immediate future, but it is sounding more appealing.

Rich

Re: [musl] More thoughts on wrapping signal handling

[Alexey Izbyshev]

On 2020-10-29 09:34, Rich Felker wrote:
> In "Re: [musl] Re: [PATCH] Make abort() AS-safe (Bug 26275)."
> (20201010002612.GC17637@brightrain.aerifal.cx,
> https://www.openwall.com/lists/musl/2020/10/10/1) I raised the
> longstanding thought of having libc wrap signal handling. This is a
> little bit of a big hammer for what it was proposed for -- fixing an
> extremely-rare race between abort and execve -- but today I had a
> thought about another use of it that's really compelling.
> 
> What I noted before was that, by wrapping signal handlers, libc could
> implement a sort of "rollback" to restart a critical section that was
> interrupted. However this really only has any use when the critical
> section has no side effects aside from its final completion, and
> except for execve where replacement of the process gives the atomic
> cutoff for rollback, it requires __cp_end-like asm label of the end of
> the critical section. So it's of limited utility.
> 
> However, what's more interesting than restarting the critical section
> when a signal is received is *allowing it to complete* before handling
> the signal. This can be implemented by having the wrapper, upon seeing
> that it interrupted a critical section, save the siginfo_t in TLS and
> immediately return, leaving signals blocked, without executing the
> application-installed signal handler. Then, when leaving the critical
> section, the unlock function can see the saved siginfo_t and call the
> application's signal handler. Effectively, it's as if the signal were
> just blocked until the end of the critical section.
> 
As described, that would call the application's signal handler on the 
wrong stack in case SA_ONSTACK was used.

And what happens if the application wants to modify ucontext via the 
third argument of the signal handler?

Re: [musl] More thoughts on wrapping signal handling

[Rich Felker]

On Thu, Oct 29, 2020 at 02:45:34PM +0300, Alexey Izbyshev wrote:
> On 2020-10-29 09:34, Rich Felker wrote:
> >In "Re: [musl] Re: [PATCH] Make abort() AS-safe (Bug 26275)."
> >(20201010002612.GC17637@brightrain.aerifal.cx,
> >https://www.openwall.com/lists/musl/2020/10/10/1) I raised the
> >longstanding thought of having libc wrap signal handling. This is a
> >little bit of a big hammer for what it was proposed for -- fixing an
> >extremely-rare race between abort and execve -- but today I had a
> >thought about another use of it that's really compelling.
> >
> >What I noted before was that, by wrapping signal handlers, libc could
> >implement a sort of "rollback" to restart a critical section that was
> >interrupted. However this really only has any use when the critical
> >section has no side effects aside from its final completion, and
> >except for execve where replacement of the process gives the atomic
> >cutoff for rollback, it requires __cp_end-like asm label of the end of
> >the critical section. So it's of limited utility.
> >
> >However, what's more interesting than restarting the critical section
> >when a signal is received is *allowing it to complete* before handling
> >the signal. This can be implemented by having the wrapper, upon seeing
> >that it interrupted a critical section, save the siginfo_t in TLS and
> >immediately return, leaving signals blocked, without executing the
> >application-installed signal handler. Then, when leaving the critical
> >section, the unlock function can see the saved siginfo_t and call the
> >application's signal handler. Effectively, it's as if the signal were
> >just blocked until the end of the critical section.
> >
> As described, that would call the application's signal handler on
> the wrong stack in case SA_ONSTACK was used.
> 
> And what happens if the application wants to modify ucontext via the
> third argument of the signal handler?

Yes, I kinda hand-waved over this with the word "call", which I
thought about annotating with (*). In the case of SA_ONSTACK you need
a primitive to "call on new stack", and while the ucontext is mostly
not meaningful/inspectable to the signal handler (because it's
interrupting libc code), the saved signal mask is. You can have the
caller restore it (in place of SYS_[rt_]sigreturn), but the natural
common solution to all of these needs is having a sort of makecontext.

Rich

Re: [musl] More thoughts on wrapping signal handling

[Alexander Monakov]

On Thu, 29 Oct 2020, Rich Felker wrote:

> Yes, I kinda hand-waved over this with the word "call", which I
> thought about annotating with (*). In the case of SA_ONSTACK you need
> a primitive to "call on new stack", and while the ucontext is mostly
> not meaningful/inspectable to the signal handler (because it's
> interrupting libc code), the saved signal mask is. You can have the
> caller restore it (in place of SYS_[rt_]sigreturn), but the natural
> common solution to all of these needs is having a sort of makecontext.

Alternatively you could re-raise the signal to have the kernel re-deliver
it with the correctly regenerated ucontext (and on the right stack)?
Would that be undesirable for some reason?

(apart from extra kernel roundtrip in the (rare) event that original signal
interrupted a critical section)

Alexander

Re: [musl] More thoughts on wrapping signal handling

[Alexander Monakov]

On Thu, 29 Oct 2020, Alexander Monakov wrote:

> On Thu, 29 Oct 2020, Rich Felker wrote:
> 
> > Yes, I kinda hand-waved over this with the word "call", which I
> > thought about annotating with (*). In the case of SA_ONSTACK you need
> > a primitive to "call on new stack", and while the ucontext is mostly
> > not meaningful/inspectable to the signal handler (because it's
> > interrupting libc code), the saved signal mask is. You can have the
> > caller restore it (in place of SYS_[rt_]sigreturn), but the natural
> > common solution to all of these needs is having a sort of makecontext.
> 
> Alternatively you could re-raise the signal to have the kernel re-deliver
> it with the correctly regenerated ucontext (and on the right stack)?
> Would that be undesirable for some reason?

Ah, because there's no way to propagate siginfo struct. Sorry :)

Alexander

Re: [musl] More thoughts on wrapping signal handling

[Florian Weimer]

* Alexander Monakov:

> On Thu, 29 Oct 2020, Alexander Monakov wrote:
>
>> On Thu, 29 Oct 2020, Rich Felker wrote:
>> 
>> > Yes, I kinda hand-waved over this with the word "call", which I
>> > thought about annotating with (*). In the case of SA_ONSTACK you need
>> > a primitive to "call on new stack", and while the ucontext is mostly
>> > not meaningful/inspectable to the signal handler (because it's
>> > interrupting libc code), the saved signal mask is. You can have the
>> > caller restore it (in place of SYS_[rt_]sigreturn), but the natural
>> > common solution to all of these needs is having a sort of makecontext.
>> 
>> Alternatively you could re-raise the signal to have the kernel re-deliver
>> it with the correctly regenerated ucontext (and on the right stack)?
>> Would that be undesirable for some reason?
>
> Ah, because there's no way to propagate siginfo struct. Sorry :)

Yes, and that's why I think copying it into TLS space will not work,
either.

Thanks,
Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

Re: [musl] More thoughts on wrapping signal handling

[Alexander Monakov]

On Thu, 29 Oct 2020, Florian Weimer wrote:

> * Alexander Monakov:
> 
> > On Thu, 29 Oct 2020, Alexander Monakov wrote:
> >
> >> On Thu, 29 Oct 2020, Rich Felker wrote:
> >> 
> >> > Yes, I kinda hand-waved over this with the word "call", which I
> >> > thought about annotating with (*). In the case of SA_ONSTACK you need
> >> > a primitive to "call on new stack", and while the ucontext is mostly
> >> > not meaningful/inspectable to the signal handler (because it's
> >> > interrupting libc code), the saved signal mask is. You can have the
> >> > caller restore it (in place of SYS_[rt_]sigreturn), but the natural
> >> > common solution to all of these needs is having a sort of makecontext.
> >> 
> >> Alternatively you could re-raise the signal to have the kernel re-deliver
> >> it with the correctly regenerated ucontext (and on the right stack)?
> >> Would that be undesirable for some reason?
> >
> > Ah, because there's no way to propagate siginfo struct. Sorry :)
> 
> Yes, and that's why I think copying it into TLS space will not work,
> either.

Actually I regret rushing that email: clearly as we are talking about wrapped
signal handlers, re-raising would call the wrapper, which would be perfectly
capable of substituting saved siginfo.

So I don't think propagating siginfo is more complicated with this re-raising
approach.

Alexander

Re: [musl] More thoughts on wrapping signal handling

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2020-10-29 02:34:50 -0400]:
> In "Re: [musl] Re: [PATCH] Make abort() AS-safe (Bug 26275)."
> (20201010002612.GC17637@brightrain.aerifal.cx,
> https://www.openwall.com/lists/musl/2020/10/10/1) I raised the
> longstanding thought of having libc wrap signal handling. This is a
> little bit of a big hammer for what it was proposed for -- fixing an
> extremely-rare race between abort and execve -- but today I had a
> thought about another use of it that's really compelling.
> 
> What I noted before was that, by wrapping signal handlers, libc could
> implement a sort of "rollback" to restart a critical section that was
> interrupted. However this really only has any use when the critical
> section has no side effects aside from its final completion, and
> except for execve where replacement of the process gives the atomic
> cutoff for rollback, it requires __cp_end-like asm label of the end of
> the critical section. So it's of limited utility.
> 
> However, what's more interesting than restarting the critical section
> when a signal is received is *allowing it to complete* before handling
> the signal. This can be implemented by having the wrapper, upon seeing
> that it interrupted a critical section, save the siginfo_t in TLS and
> immediately return, leaving signals blocked, without executing the
> application-installed signal handler. Then, when leaving the critical
> section, the unlock function can see the saved siginfo_t and call the
> application's signal handler. Effectively, it's as if the signal were
> just blocked until the end of the critical section.

this probably does not work with SIGSEGV and SIGBUS:
execution likely cannot continue to leave the critical
section, but the handlers must be invoked.

> 
> What is the value in this?
> 
> 1. It eliminates the need for syscalls to mask and unmask signals
>    around all existing AS-safe locks and critical sections that can't
>    safely be interrupted by application code.
> 
> 2. It makes it so we can make almost any function that was AS-unsafe
>    due to locking AS-safe, without any added cost. Even malloc can be
>    AS-safe.

i guess this can introduce delay into signal handling,
depending on how long libc internal locks are held.

> 
> 3. It makes it so a signal handler that fails to return promptly in
>    one thread can't arbitrarily delay other threads waiting for
>    libc-internal locks, because application code never interrupts our
>    internal critical sections.
> 
> This last property, #3, is the really exciting one -- it means that,
> short of swapping etc. (e.g. with mlockall and other realtime measures
> taken) most libc locks can be considered as held only for very small
> bounded time, rather than potentially-unbounded due to interruption by
> signal.

sounds interesting.

> 
> I'm not sure if this is something worth pursuing, and certainly not in
> the immediate future, but it is sounding more appealing.
> 
> Rich

Re: [musl] More thoughts on wrapping signal handling

[Rich Felker]

On Thu, Oct 29, 2020 at 05:18:20PM +0300, Alexander Monakov wrote:
> On Thu, 29 Oct 2020, Florian Weimer wrote:
> 
> > * Alexander Monakov:
> > 
> > > On Thu, 29 Oct 2020, Alexander Monakov wrote:
> > >
> > >> On Thu, 29 Oct 2020, Rich Felker wrote:
> > >> 
> > >> > Yes, I kinda hand-waved over this with the word "call", which I
> > >> > thought about annotating with (*). In the case of SA_ONSTACK you need
> > >> > a primitive to "call on new stack", and while the ucontext is mostly
> > >> > not meaningful/inspectable to the signal handler (because it's
> > >> > interrupting libc code), the saved signal mask is. You can have the
> > >> > caller restore it (in place of SYS_[rt_]sigreturn), but the natural
> > >> > common solution to all of these needs is having a sort of makecontext.
> > >> 
> > >> Alternatively you could re-raise the signal to have the kernel re-deliver
> > >> it with the correctly regenerated ucontext (and on the right stack)?
> > >> Would that be undesirable for some reason?
> > >
> > > Ah, because there's no way to propagate siginfo struct. Sorry :)
> > 
> > Yes, and that's why I think copying it into TLS space will not work,
> > either.
> 
> Actually I regret rushing that email: clearly as we are talking about wrapped
> signal handlers, re-raising would call the wrapper, which would be perfectly
> capable of substituting saved siginfo.
> 
> So I don't think propagating siginfo is more complicated with this re-raising
> approach.

The reraising is problematic because of how it works with signal
queueing and additional pending signals, I think. You might be able to
make that transparent but I think it's at least slightly nontrivial,
even just figuring out what to do to get them handled in the right
order if there's already another signal pending when you want to
re-raise. If you just unblock and try to handle both from the same
kernel-invoked signal handler, you'll miss the second if the first one
doesn't return normally. And if you try to re-raise to get the second
one, you just push back the issue again, possibly arbitrarily many
times. Maybe this works but it seems messy...

Rich

Re: [musl] More thoughts on wrapping signal handling

[Rich Felker]

On Thu, Oct 29, 2020 at 03:21:17PM +0100, Szabolcs Nagy wrote:
> * Rich Felker <dalias@libc.org> [2020-10-29 02:34:50 -0400]:
> > In "Re: [musl] Re: [PATCH] Make abort() AS-safe (Bug 26275)."
> > (20201010002612.GC17637@brightrain.aerifal.cx,
> > https://www.openwall.com/lists/musl/2020/10/10/1) I raised the
> > longstanding thought of having libc wrap signal handling. This is a
> > little bit of a big hammer for what it was proposed for -- fixing an
> > extremely-rare race between abort and execve -- but today I had a
> > thought about another use of it that's really compelling.
> > 
> > What I noted before was that, by wrapping signal handlers, libc could
> > implement a sort of "rollback" to restart a critical section that was
> > interrupted. However this really only has any use when the critical
> > section has no side effects aside from its final completion, and
> > except for execve where replacement of the process gives the atomic
> > cutoff for rollback, it requires __cp_end-like asm label of the end of
> > the critical section. So it's of limited utility.
> > 
> > However, what's more interesting than restarting the critical section
> > when a signal is received is *allowing it to complete* before handling
> > the signal. This can be implemented by having the wrapper, upon seeing
> > that it interrupted a critical section, save the siginfo_t in TLS and
> > immediately return, leaving signals blocked, without executing the
> > application-installed signal handler. Then, when leaving the critical
> > section, the unlock function can see the saved siginfo_t and call the
> > application's signal handler. Effectively, it's as if the signal were
> > just blocked until the end of the critical section.
> 
> this probably does not work with SIGSEGV and SIGBUS:
> execution likely cannot continue to leave the critical
> section, but the handlers must be invoked.

For async delivery of them (via kill, etc.) it's no problem. For
synchronous delivery, it means code inside libc faulted, and then you
can really do whatever you want since it means the caller invoked UB.
There's no contract for the appliction's SIGSEGV handler to run when
you pass an invalid pointer to libc functionr or otherwise invoke
"segfaulty" UB. But of course we could just execute them synchronously
in this case, rather than deferring, since the process is in an
undefined, unrecoverable state anyway.

> > What is the value in this?
> > 
> > 1. It eliminates the need for syscalls to mask and unmask signals
> >    around all existing AS-safe locks and critical sections that can't
> >    safely be interrupted by application code.
> > 
> > 2. It makes it so we can make almost any function that was AS-unsafe
> >    due to locking AS-safe, without any added cost. Even malloc can be
> >    AS-safe.
> 
> i guess this can introduce delay into signal handling,
> depending on how long libc internal locks are held.

Yes, I meant to mention that. It basically trades small controllable
delay in invocation of signal handlers (note: some such delay already
exists due to backing out of a blocking syscall in progress;
essentially the kernel is doing the same thing I talked about here,
but in kernelspace) to get the below:

> > 3. It makes it so a signal handler that fails to return promptly in
> >    one thread can't arbitrarily delay other threads waiting for
> >    libc-internal locks, because application code never interrupts our
> >    internal critical sections.
> > 
> > This last property, #3, is the really exciting one -- it means that,
> > short of swapping etc. (e.g. with mlockall and other realtime measures
> > taken) most libc locks can be considered as held only for very small
> > bounded time, rather than potentially-unbounded due to interruption by
> > signal.
> 
> sounds interesting.

Also note: while I think they're probably expensive enough this
wouldn't be a good default, for hard-realtime use one could patch musl
to use priority inheritance locks here.

One thing I glossed over is that I do think you need owner-tracking
locks for this to work. That's because the signal handler wrapper has
to be able to distinguish between the "waiting for lock" condition (in
which signal must not be deferred) and the "lock already acquired"
condition. Technically this could be done with asm introspection like
__cp_end, but I'd really prefer to avoid that; with lock ownership
tracked, the signal handler wrapper just needs to look at whether
*tls->pending_lock & 0x7fffffff == tls->tid.

Rich

Re: [musl] More thoughts on wrapping signal handling

[Alexey Izbyshev]

On 2020-10-29 16:38, Rich Felker wrote:
> On Thu, Oct 29, 2020 at 02:45:34PM +0300, Alexey Izbyshev wrote:
>> On 2020-10-29 09:34, Rich Felker wrote:
>> >In "Re: [musl] Re: [PATCH] Make abort() AS-safe (Bug 26275)."
>> >(20201010002612.GC17637@brightrain.aerifal.cx,
>> >https://www.openwall.com/lists/musl/2020/10/10/1) I raised the
>> >longstanding thought of having libc wrap signal handling. This is a
>> >little bit of a big hammer for what it was proposed for -- fixing an
>> >extremely-rare race between abort and execve -- but today I had a
>> >thought about another use of it that's really compelling.
>> >
>> >What I noted before was that, by wrapping signal handlers, libc could
>> >implement a sort of "rollback" to restart a critical section that was
>> >interrupted. However this really only has any use when the critical
>> >section has no side effects aside from its final completion, and
>> >except for execve where replacement of the process gives the atomic
>> >cutoff for rollback, it requires __cp_end-like asm label of the end of
>> >the critical section. So it's of limited utility.
>> >
>> >However, what's more interesting than restarting the critical section
>> >when a signal is received is *allowing it to complete* before handling
>> >the signal. This can be implemented by having the wrapper, upon seeing
>> >that it interrupted a critical section, save the siginfo_t in TLS and
>> >immediately return, leaving signals blocked, without executing the
>> >application-installed signal handler. Then, when leaving the critical
>> >section, the unlock function can see the saved siginfo_t and call the
>> >application's signal handler. Effectively, it's as if the signal were
>> >just blocked until the end of the critical section.
>> >
>> As described, that would call the application's signal handler on
>> the wrong stack in case SA_ONSTACK was used.
>> 
>> And what happens if the application wants to modify ucontext via the
>> third argument of the signal handler?
> 
> Yes, I kinda hand-waved over this with the word "call", which I
> thought about annotating with (*). In the case of SA_ONSTACK you need
> a primitive to "call on new stack", and while the ucontext is mostly
> not meaningful/inspectable to the signal handler (because it's
> interrupting libc code), the saved signal mask is. You can have the
> caller restore it (in place of SYS_[rt_]sigreturn), but the natural
> common solution to all of these needs is having a sort of makecontext.
> 
Such "sigcall/sigreturn" shims would have to emulate kernel behavior 
precisely. If a new feature is added into the kernel, and the 
application detects that it's supported based on what the *kernel* tells 
it, subtle breakage might occur due to imprecise emulation (as a random 
example, consider SS_AUTODISARM flag of sigaltstack()). So you'd have to 
intercept feature tests as well, and it starts to look messy IMO.

Re-raising the signal would avoid most of that emulation, but appears to 
be broken at least due to signal ordering issues as mentioned in 
https://www.openwall.com/lists/musl/2020/10/29/12.

Alexey