sigaltstack for implementation-internal signals?

sigaltstack for implementation-internal signals?

[Rich Felker]

It came up recently that some erlang component is doing awful hacks
(see https://twitter.com/RichFelker/status/1099816400036773888) to try
to get implementation-internal signals to run on the alternate stack
setup by sigaltstack instead of the main stack. This desire makes some
sense, as they have tons of really tiny stacks and for their
coroutine-like things that rapidly context-switch in userspace. And
since there's no portable or valid way to do this hack, it raised the
issue for me: should we just always deliver implementation-internal
signals on the alternate stack if it's setup?

Unfortunateley I don't think this is possible/safe, for reasons
related to how the kernel's signal frame setup works. When a signal is
delivered and is to be handled on the alt stack, Linux checks whether
the current stack pointer is already on the alt stack. If so, it
decrements it normally; if not, it sets the stack pointer to the
beginning of the alt stack. The first case is needed in case a signal
interrupts a signal handler already running on the alt stack; if it
weren't handled that way, the second one would clobber the first one's
state, and upon return Bad Things would happen.

Unfortunately, this can go wrong. Suppose the application's signal
handler running on the alt stack changes the stack pointer to
something off the alt stack -- for example, using swapcontext, or some
awful split-stack hack. This is known to be "unsafe" in general, and
was the motivation for the (problematic with respect to POSIX, but
fixed in http://austingroupbugs.net/view.php?id=1187) addition of
SS_AUTODISARM. However, in principle it was "already safe" (without
SS_AUTODISARM) to use swapcontext with sigaltstack if the signal
handler and swapped-to context kept all SA_ONSTACK-flagged signals
blocked for their duration. An application could clearly arrange for
this; for example it's fairly natural if you only use one signal
handler that's SA_ONSTACK.

If we add unblockable, implementation-internal signals that are
flagged SA_ONSTACK, however, this breaks; now even if an application
has taken the "proper precautions", they can be delivered in a state
where the alt stack is nonempty but the stack pointer doesn't point
into it, thereby causing it to get clobbered.

Perhaps there's a chance that this just isn't supported/valid usage,
that "leaving the alt stack" should always be seen as abandoning it,
and that anything that worked before to preserve it was "just by
mistake/luck".

Rich

Re: sigaltstack for implementation-internal signals?

[Florian Weimer]

* Rich Felker:

> If we add unblockable, implementation-internal signals that are
> flagged SA_ONSTACK, however, this breaks; now even if an application
> has taken the "proper precautions", they can be delivered in a state
> where the alt stack is nonempty but the stack pointer doesn't point
> into it, thereby causing it to get clobbered.

There's also the matter of applications which do not use signal handlers
at all (and thus never invoke sigaltstack) and have really small stacks,
or use the stack pointer register for something else.  Is either of that
supported?

I think it is not clear whether a libc implementation may generate
sporadic signals *at all* to support implementation needs.

Does musl use asynchronous implementation signals?  For glibc, we would
prefer synchronous delivery, that is, the handler runs before the
signal-generating system call returns.  This makes me wonder if we
should try to get the kernel to provide us a system call which allows us
to run code on a different thread, with signals disabled, but with the
caller's stack (from the original thread).  I think this would address
issues caused by strange stack pointer values in the target thread.

The Boehm-Demers-Weiser garbage collector would probably benefit from
that as well.

Thanks,
Florian

Re: sigaltstack for implementation-internal signals?

[Rich Felker]

On Wed, Mar 06, 2019 at 01:56:02PM +0100, Florian Weimer wrote:
> * Rich Felker:
> 
> > If we add unblockable, implementation-internal signals that are
> > flagged SA_ONSTACK, however, this breaks; now even if an application
> > has taken the "proper precautions", they can be delivered in a state
> > where the alt stack is nonempty but the stack pointer doesn't point
> > into it, thereby causing it to get clobbered.
> 
> There's also the matter of applications which do not use signal handlers
> at all (and thus never invoke sigaltstack) and have really small stacks,
> or use the stack pointer register for something else.  Is either of that
> supported?

In practice, mostly. Applications doing that need to be aware that
there are a few operations that may use signals internally, and must
avoid those operations. Also, on most nommu platforms, there's a
requirement that the stack always be valid, as interrupts are
delivered on the same stack they interrupted, so code that could
potentially be used on such platforms clearly can't do this.

Formally, I'd say no, it shouldn't be supported, but I'm open to
reasons why it should.

> I think it is not clear whether a libc implementation may generate
> sporadic signals *at all* to support implementation needs.

It certainly can if it defines as part of its ABI a contract that the
stack pointer must always be valid and have at least X available
space. Whether this is a good idea or compatible with existing
informal expectations is the question.

> Does musl use asynchronous implementation signals?  For glibc, we would
> prefer synchronous delivery, that is, the handler runs before the
> signal-generating system call returns.

The __synccall mechanism, used for multithreaded set*id, setrlimit,
etc. is fully synchronous, in multiple steps, with only one thread
doing anything at a time.

If the same mechanism needs to be used as a fallback implementation of
SYS_membarrier, it's less synchronized, so as not to have such awful
performance. All the threads can run concurrently, but the caller
waits for them all to post a semaphore before moving on.

For pthread_cancel, it's asynchronous. The signal can be blocked if
the cancellation signal handler has already run and determined that it
should not act on cancellation (disabled or not at a cancellation
point); in that case, it re-raises the signal and leaves it pending,
so that, if the determination was made while the thread was executing
a signal handler, it can re-evaluate when the signal handler returns
(possibly into restarting a blocking syscall that's a cancellation
point). I don't see any easy way to make this synchronous with respect
to the thread calling pthread_cancel, but maybe there is by breaking
it down into cases and using more than one signal.

> This makes me wonder if we
> should try to get the kernel to provide us a system call which allows us
> to run code on a different thread, with signals disabled, but with the
> caller's stack (from the original thread).  I think this would address
> issues caused by strange stack pointer values in the target thread.

Yes, this would be a very nice feature. Even nicer would be new set*id
and setrlimit syscalls that honor POSIX semantics and affect the whole
process, so that none of this hackery is necessary.

> The Boehm-Demers-Weiser garbage collector would probably benefit from
> that as well.

Yes, I think so.

Rich

Re: sigaltstack for implementation-internal signals?

[Markus Wichmann]

On Wed, Mar 06, 2019 at 01:56:02PM +0100, Florian Weimer wrote:
> * Rich Felker:
> 
> > If we add unblockable, implementation-internal signals that are
> > flagged SA_ONSTACK, however, this breaks; now even if an application
> > has taken the "proper precautions", they can be delivered in a state
> > where the alt stack is nonempty but the stack pointer doesn't point
> > into it, thereby causing it to get clobbered.
> 
> There's also the matter of applications which do not use signal handlers
> at all (and thus never invoke sigaltstack) and have really small stacks,
> or use the stack pointer register for something else.  Is either of that
> supported?
> 

If you or any library you use install signal handlers, your code is
constantly at an interface boundary, since a signal can appear at any
time.

libc uses signal handlers. And you use libc. Connect the dots...

> I think it is not clear whether a libc implementation may generate
> sporadic signals *at all* to support implementation needs.
> 

Well, then we're at an impasse, as POSIX requires certain semantics and
the kernel provides different ones, and we need the signals as a
go-between. Alright, what implementation-internal signals are there?

SIGTIMER:
That one is needed in timer_create() to be able to spawn a thread if
someone creates a timer with SIGEV_THREAD. Since the handling thread is
spawned immediately, the signal is taken only on the stack of the
internal thread. And the handler itself does nothing. BTW, is there a
reason that handler is installed with SA_SIGINFO if it does nothing with
the extra info?

Anyway, this signal has no impact on application stacks.

SIGCANCEL:
The cancelled thread will take this signal. Wanting to cancel a thread
without signalling it is like wanting to be washed without being made
wet; it simply makes no sense. Applications can trivially prevent the
generation of this signal by simply foregoing the use of
pthread_cancel().

SIGSYNCCALL:
This is going to be the sticking point, isn't it? This signal really
does appear in each thread, even application threads, without warning.
It is currently used to implement setrlimit(), setegid(), seteuid(),
setgid(), setuid(), setregid(), setreuid(), setresgid() and setresuid().
Except for setrlimit(), I would not expect applications with tiny stacks
and lots of threads to call any of these. Much less often enough to
cause an issue. So setrlimit() is probably the only call that will cause
problems in the kind of applications you speak of.

Thus the entire problem could be solved if the kernel allowed setting
rlimits for the entire process, or at least for other threads. Seems
like the smaller change to me.

> Does musl use asynchronous implementation signals?  For glibc, we would
> prefer synchronous delivery, that is, the handler runs before the
> signal-generating system call returns.  This makes me wonder if we
> should try to get the kernel to provide us a system call which allows us
> to run code on a different thread, with signals disabled, but with the
> caller's stack (from the original thread).  I think this would address
> issues caused by strange stack pointer values in the target thread.
> 

Hmmm... if the kernel supported remote sigaltstack() (i.e. setting a
signal stack for another thread), then this would all be done. But alas,
no such luck.

> Thanks,
> Florian

Ciao,
Markus

Re: sigaltstack for implementation-internal signals?

[Rich Felker]

On Wed, Mar 06, 2019 at 05:07:40PM +0100, Markus Wichmann wrote:
> On Wed, Mar 06, 2019 at 01:56:02PM +0100, Florian Weimer wrote:
> > * Rich Felker:
> > 
> > > If we add unblockable, implementation-internal signals that are
> > > flagged SA_ONSTACK, however, this breaks; now even if an application
> > > has taken the "proper precautions", they can be delivered in a state
> > > where the alt stack is nonempty but the stack pointer doesn't point
> > > into it, thereby causing it to get clobbered.
> > 
> > There's also the matter of applications which do not use signal handlers
> > at all (and thus never invoke sigaltstack) and have really small stacks,
> > or use the stack pointer register for something else.  Is either of that
> > supported?
> > 
> 
> If you or any library you use install signal handlers, your code is
> constantly at an interface boundary, since a signal can appear at any
> time.

No, only when signals are not blocked. You can block any signals that
are provided to the application for its use, but the libc-internal
signals are not blockable by the application.

> libc uses signal handlers. And you use libc. Connect the dots...

This is an implementation detail, not part of the specification.

> > I think it is not clear whether a libc implementation may generate
> > sporadic signals *at all* to support implementation needs.
> > 
> 
> Well, then we're at an impasse, as POSIX requires certain semantics and
> the kernel provides different ones, and we need the signals as a
> go-between. Alright, what implementation-internal signals are there?
> 
> SIGTIMER:
> That one is needed in timer_create() to be able to spawn a thread if
> someone creates a timer with SIGEV_THREAD. Since the handling thread is
> spawned immediately, the signal is taken only on the stack of the
> internal thread. And the handler itself does nothing.

This can be removed at some point anyway. SIGEV_THREAD is easier to do
in userspace without kernel timer objects and signals.

> BTW, is there a
> reason that handler is installed with SA_SIGINFO if it does nothing with
> the extra info?

		if (si.si_code == SI_TIMER && !setjmp(jb)) {

> Anyway, this signal has no impact on application stacks.

It does if the thread attribute object defines a custom stack, but I
think POSIX severely limits or forbids what you can do with this,
since the stack is never freed/freeable.

> SIGCANCEL:
> The cancelled thread will take this signal. Wanting to cancel a thread
> without signalling it is like wanting to be washed without being made
> wet; it simply makes no sense.

No, it's an implementation detail that the underlying kernel lacks any
way to make cancellation work without signals. It's not part of how
cancellation is specified, but there might be language in POSIX
suggesting that signals are a possible mechanism, in which case I
think that's good evidence that applications should account for the
possibility in their stack usage.

> Applications can trivially prevent the
> generation of this signal by simply foregoing the use of
> pthread_cancel().

Yes.

> SIGSYNCCALL:
> This is going to be the sticking point, isn't it? This signal really
> does appear in each thread, even application threads, without warning.
> It is currently used to implement setrlimit(), setegid(), seteuid(),
> setgid(), setuid(), setregid(), setreuid(), setresgid() and setresuid().
> Except for setrlimit(), I would not expect applications with tiny stacks
> and lots of threads to call any of these. Much less often enough to
> cause an issue. So setrlimit() is probably the only call that will cause
> problems in the kind of applications you speak of.
> 
> Thus the entire problem could be solved if the kernel allowed setting
> rlimits for the entire process, or at least for other threads. Seems
> like the smaller change to me.

Yes.

> > Does musl use asynchronous implementation signals?  For glibc, we would
> > prefer synchronous delivery, that is, the handler runs before the
> > signal-generating system call returns.  This makes me wonder if we
> > should try to get the kernel to provide us a system call which allows us
> > to run code on a different thread, with signals disabled, but with the
> > caller's stack (from the original thread).  I think this would address
> > issues caused by strange stack pointer values in the target thread.
> 
> Hmmm... if the kernel supported remote sigaltstack() (i.e. setting a
> signal stack for another thread), then this would all be done. But alas,
> no such luck.

It's not remote sigaltstack; that would break all sorts of things and
would not give conforming behavior. Rather it needs to be a mechanism
to deliver a specific signal (or better yet, not a signal, just a
caller-provided function) on a specific stack (so that it doesn't
affect delivery of any other signals at the same time).

Rich

Re: sigaltstack for implementation-internal signals?

[Florian Weimer]

* Rich Felker:

> For pthread_cancel, it's asynchronous. The signal can be blocked if
> the cancellation signal handler has already run and determined that it
> should not act on cancellation (disabled or not at a cancellation
> point); in that case, it re-raises the signal and leaves it pending,
> so that, if the determination was made while the thread was executing
> a signal handler, it can re-evaluate when the signal handler returns
> (possibly into restarting a blocking syscall that's a cancellation
> point). I don't see any easy way to make this synchronous with respect
> to the thread calling pthread_cancel, but maybe there is by breaking
> it down into cases and using more than one signal.

phtread_cancel is slightly different; I think think we can send a signal
for that.  It can't be made synchronous because if we block the
canceling thread while the canceled thread unwinds (running cancellation
handlers), we will probably end up introducing deadlocks involving
application locks.

Thanks,
Florian

Re: sigaltstack for implementation-internal signals?

[Rich Felker]

On Wed, Mar 06, 2019 at 06:21:40PM +0100, Florian Weimer wrote:
> * Rich Felker:
> 
> > For pthread_cancel, it's asynchronous. The signal can be blocked if
> > the cancellation signal handler has already run and determined that it
> > should not act on cancellation (disabled or not at a cancellation
> > point); in that case, it re-raises the signal and leaves it pending,
> > so that, if the determination was made while the thread was executing
> > a signal handler, it can re-evaluate when the signal handler returns
> > (possibly into restarting a blocking syscall that's a cancellation
> > point). I don't see any easy way to make this synchronous with respect
> > to the thread calling pthread_cancel, but maybe there is by breaking
> > it down into cases and using more than one signal.
> 
> phtread_cancel is slightly different; I think think we can send a signal
> for that.  It can't be made synchronous because if we block the
> canceling thread while the canceled thread unwinds (running cancellation
> handlers), we will probably end up introducing deadlocks involving
> application locks.

If not for the "re-raise a signal" aspect I described, you could do it
"synchronously". You don't have to wait for the cleanup functions to
run. Rather, you'd just wait for the target thread to exit the base
cancellation logic code, respond with an indication of what it's
doing, and get off the new stack. If acting on the cancellation, it
would modify the ucontext it's returning to with a new call frame on
it's main stack to run the unwind/cleanup in, then return.

Rich

Re: sigaltstack for implementation-internal signals?

[Rich Felker]

On Wed, Mar 06, 2019 at 06:11:19PM +0100, Markus Wichmann wrote:
> On Wed, Mar 06, 2019 at 11:25:08AM -0500, Rich Felker wrote:
> > On Wed, Mar 06, 2019 at 05:07:40PM +0100, Markus Wichmann wrote:
> > > libc uses signal handlers. And you use libc. Connect the dots...
> > 
> > This is an implementation detail, not part of the specification.
> 
> Well, then the spec must allow for this implementation (else it would
> not be conforming). Which means we move from "libc uses signal handlers"
> to "libc might use signal handlers". From a practical point of view,
> that is no difference at all.

The question is whether it's a conforming implementation at all. If
the use of signals is visible in ways it shouldn't be (i.e. without
doing hacks outside the scope of the language/API like setting a bogus
stack pointer) then that's non-conforming. That (aside from other
practical necessities to prevent breakage) is why we take such
measures to "hide" the internal use of signals, like preventing them
from being added to sigset_t.

> But the point about not being able to block these signals remains... not
> sure what the right thing is, really. On one hand, this breaks
> applications that assume no signal will arrive after blocking signals,
> on the other hand, blocking libc-internal signals might deadlock. (We
> had that thread about go a while back, remember?)

It doesn't break applications that assume no signal will arrive, since
its arrival isn't observable within the standard interfaces. It breaks
applications that are doing particular hacks outside of the scope of
what's specified.

> One thing I noticed: At least on Linux, the main thread must always have
> a valid stack pointer in order to facilitate the rather fragile stack
> expansion mechanism. As I recall, that mechanism is triggered every time
> a stack pointer reference causes a page fault, and if the ulimit for the
> stack size is exceeded, the process is signalled.

I don't see how this matters. If you set the stack pointer to an
invalid value (e.g. to use it as a general purpose register), you're
not going to be trying to dereference it.

> > > SIGTIMER:
> > > That one is needed in timer_create() to be able to spawn a thread if
> > > someone creates a timer with SIGEV_THREAD. Since the handling thread is
> > > spawned immediately, the signal is taken only on the stack of the
> > > internal thread. And the handler itself does nothing.
> > 
> > This can be removed at some point anyway. SIGEV_THREAD is easier to do
> > in userspace without kernel timer objects and signals.
> > 
> 
> How? clock_nanosleep() in a loop, then?

Yes. What we're doing now is essentially a huge pile of unnecessary
complexity and kernel resource consumption to get the same effect as
clock_nanosleep in a loop. However there is some convenience; it
avoids having to calculate overruns ourselves and allows
timer_settime, etc. to use the common kernel interface regardless of
which type of timer they're acting on. This is why I haven't converted
it over yet; converting it requires writing the equivalent logic for
"userspace timer threads".

> > > BTW, is there a
> > > reason that handler is installed with SA_SIGINFO if it does nothing with
> > > the extra info?
> > 
> > 		if (si.si_code == SI_TIMER && !setjmp(jb)) {
> > 
> 
> That is in the handling thread, and the signal info comes out of
> sigwaitinfo(). No, I meant timer_handler(), which is installed with
> SA_SIGINFO but does nothing.

Well before it was in the handler, but it's not clear that you can use
sigwaitinfo to get the info without SI_SIGINFO. See the application
usage notes in POSIX for sigwaitinfo:

    "Note that in order to ensure that generated signals are queued
    and signal values passed to sigqueue() are available in si_value,
    applications which use sigwaitinfo() or sigtimedwait() need to set
    the SA_SIGINFO flag for each signal in the set (see Signal
    Concepts). This means setting each signal to be handled by a
    three-argument signal-catching function, even if the handler will
    never be called. It is not possible (portably) to set a signal
    handler to SIG_DFL while setting the SA_SIGINFO flag, because
    assigning to the sa_handler member of struct sigaction instead of
    the sa_sigaction member would result in undefined behavior, and
    SIG_DFL need not be assignment-compatible with sa_sigaction. Even
    if an assignment of SIG_DFL to sa_sigaction is accepted by the
    compiler, the implementation need not treat this value as
    special-it could just be taken as the address of a signal-catching
    function."

Thus the SA_SIGINFO flag was kept with the understanding that it might
be required for the current logic to work.

Rich