[RFC] Support for segmentation-hardened SafeStack

[RFC] Support for segmentation-hardened SafeStack

[LeMay, Michael]

[-- Attachment #1: Type: text/plain, Size: 3693 bytes --]

Hi,

I submitted several patches to LLVM and Clang to harden SafeStack using segmentation on x86-32 [1].  See [2] for general background on SafeStack.  On Linux, I have been testing my compiler changes with a modified version of musl.  I currently plan to submit my musl patches if and when the prerequisite LLVM and Clang patches are accepted.  One of my LLVM patches depends on the details of my musl patches, which is the main reason that I am sending this RFC now.

Specifically, https://reviews.llvm.org/D19762 assumes that the unsafe stack pointer is stored at offset 0x24 in the musl thread control block.  This would be between the pid and tsd_used variables that are currently defined.  I also propose storing the base address of the unsafe stack at offset 0x28, but the compiler would not depend on that.

Here is an overview of some other changes that I plan to propose with my musl patches:

The segmentation-hardened SafeStack support would be enabled with a new configuration option, "--enable-safe-stack".

When this is enabled, many libraries routines require that both a safe stack and an unsafe stack be available.  I modified _start_c in crt1.c to temporarily setup a small, pre-allocated unsafe stack for the early initialization routines to use.  I also made similar changes in dlstart.c.  A larger unsafe stack is allocated and setup later from either __libc_start_main or __dls3, depending on whether static or dynamic linking is used.  I split __dls3 so that it only performs minimal initialization before allocating the larger unsafe stack and then performing the rest of its work in a new __dls4 function.

After the larger unsafe stack is allocated, I invoke the modify_ldt syscall to insert a segment descriptor with a limit that is below the beginning of the safe stacks.  I load that segment descriptor into the DS and ES segment registers to block memory accesses to DS and ES from accessing the safe stacks.  One purpose of my LLVM and Clang patches is to insert the necessary segment override prefixes to direct accesses to the appropriate segments.

Many instructions expect that argc, argv, the environment, and auxv are accessible in the DS and ES segments.
These are stored on the initial stack, which is above the limit of the restricted DS and ES segments.  I annotated auxv with an attribute to cause the compiler to emit SS segment-override prefixes when accessing auxv.  I copied the other data to the heap, which is accessible in DS and ES.

I modified the pthread routines to allocate and deallocate additional stacks as needed in the appropriate memory ranges.  The safe stacks are allocated at high addresses so that they are above the limit of the modified DS and ES segments.  The unsafe stack for each new thread is allocated below its TLS region and thread control block, which is where the stack is currently located by default.

The Linux vDSO code may be incompatible with programs that enable segmentation-hardened SafeStack.  For example, it may allocate data on the safe stack and then attempt to access it in DS or ES, which would result in an exception due to the segment limit violation.  My patches prevent the vDSO from being invoked when segmentation-hardened SafeStack is enabled.

Finally, the i386 __clone implementation is written in assembly language, so the compiler is unable to automatically add a stack segment override prefix to an instruction in that routine that accesses a safe stack.  I added that prefix manually in the source code.

Comments appreciated.

Thanks,
Michael

[1] http://lists.llvm.org/pipermail/llvm-dev/2016-May/100346.html
[2] http://clang.llvm.org/docs/SafeStack.html

[-- Attachment #2: Type: text/html, Size: 6787 bytes --]

Re: [RFC] Support for segmentation-hardened SafeStack

[Rich Felker]

On Thu, Sep 22, 2016 at 11:00:45PM +0000, LeMay, Michael wrote:
> Hi,
> 
> I submitted several patches to LLVM and Clang to harden SafeStack
> using segmentation on x86-32 [1]. See [2] for general background on
> SafeStack. On Linux, I have been testing my compiler changes with a
> modified version of musl. I currently plan to submit my musl patches
> if and when the prerequisite LLVM and Clang patches are accepted.
> One of my LLVM patches depends on the details of my musl patches,
> which is the main reason that I am sending this RFC now.

My understanding is that this is a different, incompatible ABI for
i386, i.e. code that uses safestack is not calling-compatible with
code that doesn't, and vice versa. Is that true? This is probably the
most significant determining factor in how we treat it.

> Specifically, https://reviews.llvm.org/D19762 assumes that the
> unsafe stack pointer is stored at offset 0x24 in the musl thread
> control block. This would be between the pid and tsd_used variables
> that are currently defined. I also propose storing the base address
> of the unsafe stack at offset 0x28, but the compiler would not
> depend on that.

Almost none of the existing fields are public; I think the only
exception is the stack-protector canary. IMO you should prefer
avoiding per-libc offset variation over preserving existing offsets.

> Here is an overview of some other changes that I plan to propose
> with my musl patches:
> 
> The segmentation-hardened SafeStack support would be enabled with a
> new configuration option, "--enable-safe-stack".
> 
> When this is enabled, many libraries routines require that both a
> safe stack and an unsafe stack be available. I modified _start_c in
> crt1.c to temporarily setup a small, pre-allocated unsafe stack for

I'd have to see exactly what you mean, but my leaning is that crt1 is
not a good place for anything new. For dynamic-linked programs,
crt1-to-__libc_start_main is the main permanent ABI boundary and not
something you want to have complexity that could need changing.

> the early initialization routines to use. I also made similar
> changes in dlstart.c. A larger unsafe stack is allocated and setup
> later from either __libc_start_main or __dls3, depending on whether
> static or dynamic linking is used. I split __dls3 so that it only
> performs minimal initialization before allocating the larger unsafe
> stack and then performing the rest of its work in a new __dls4
> function.
> 
> After the larger unsafe stack is allocated, I invoke the modify_ldt
> syscall to insert a segment descriptor with a limit that is below
> the beginning of the safe stacks. I load that segment descriptor
> into the DS and ES segment registers to block memory accesses to DS
> and ES from accessing the safe stacks. One purpose of my LLVM and
> Clang patches is to insert the necessary segment override prefixes
> to direct accesses to the appropriate segments.

The content on these stacks is purely return addresses, spills, and
other stuff that's only accessible to compiler-generated code, not
data whose addresses can be taken, right?

> Many instructions expect that argc, argv, the environment, and auxv
> are accessible in the DS and ES segments. These are stored on the
> initial stack, which is above the limit of the restricted DS and ES
> segments. I annotated auxv with an attribute to cause the compiler
> to emit SS segment-override prefixes when accessing auxv. I copied
> the other data to the heap, which is accessible in DS and ES.

Invasive arch-specific changes to unrelated code are highly frowned
upon in musl. I think to be acceptable upstream at all the auxv would
also have to be relocated by startup code to an address where it's
accessible.

> I modified the pthread routines to allocate and deallocate
> additional stacks as needed in the appropriate memory ranges. The
> safe stacks are allocated at high addresses so that they are above
> the limit of the modified DS and ES segments. The unsafe stack for
> each new thread is allocated below its TLS region and thread control
> block, which is where the stack is currently located by default.

This likely should be hidden inside __clone rather than in
non-arch-specific sources.

> The Linux vDSO code may be incompatible with programs that enable
> segmentation-hardened SafeStack. For example, it may allocate data
> on the safe stack and then attempt to access it in DS or ES, which
> would result in an exception due to the segment limit violation. My
> patches prevent the vDSO from being invoked when
> segmentation-hardened SafeStack is enabled.

That sounds reasonable but rather unfortunate.

> Finally, the i386 __clone implementation is written in assembly
> language, so the compiler is unable to automatically add a stack
> segment override prefix to an instruction in that routine that
> accesses a safe stack. I added that prefix manually in the source
> code.
> 
> Comments appreciated.

Hope the above are helpful.

Rich

Re: [RFC] Support for segmentation-hardened SafeStack

[Szabolcs Nagy]

* LeMay, Michael <michael.lemay@intel.com> [2016-09-22 23:00:45 +0000]:
> I submitted several patches to LLVM and Clang to harden SafeStack using segmentation on x86-32 [1].  See [2] for general background on SafeStack.
...
> [1] http://lists.llvm.org/pipermail/llvm-dev/2016-May/100346.html
> [2] http://clang.llvm.org/docs/SafeStack.html

is all runtime support in the libc with your patches?
(i.e. no static linked interposition code from compiler-rt)

can you call into non-instrumented code?
(as Rich noted this looks like a new abi on i386)
i assume the segmented variant breaks abi while the
non-segmented one does not.

what is the unsafe stack size of the main thread?
how much is the resource usage overhead?

what happens if unsafe stack allocation fails?
how does the stack get deallocated at thread exit?
i assume they are consistent with normal stack
handling if this is done in musl.. except for the
main thread.

can signal handlers work with sigaltstack?

Re: [RFC] Support for segmentation-hardened SafeStack

[LeMay, Michael]

On 9/22/2016 16:42, Rich Felker wrote:
> On Thu, Sep 22, 2016 at 11:00:45PM +0000, LeMay, Michael wrote:
>> Hi,
>>
>> I submitted several patches to LLVM and Clang to harden SafeStack
>> using segmentation on x86-32 [1]. See [2] for general background on
>> SafeStack. On Linux, I have been testing my compiler changes with a
>> modified version of musl. I currently plan to submit my musl patches
>> if and when the prerequisite LLVM and Clang patches are accepted.
>> One of my LLVM patches depends on the details of my musl patches,
>> which is the main reason that I am sending this RFC now.
> My understanding is that this is a different, incompatible ABI for
> i386, i.e. code that uses safestack is not calling-compatible with
> code that doesn't, and vice versa. Is that true? This is probably the
> most significant determining factor in how we treat it.
That's a good question, but I don't know how to answer it directly.

A salient requirement is that all code that runs while restricted 
segment limits are in effect for DS and ES must use appropriate segment 
override prefixes.  This is to direct safe stack accesses to SS and thus 
avoid violating the segment limits on DS and ES.  It is still possible 
to call code that does not satisfy that requirement (I will refer to 
such functions as "standard functions", in contrast to 
"segmentation-aware functions") in the same program, but the segment 
registers would need to be reverted to contain flat segment descriptors 
before calling such code.  Otherwise, a segment limit violation would 
occur if a standard function attempted to access data on the stack using 
DS or ES.  Of course, reverting to flat segment descriptors would leave 
the safe stacks unprotected.

Another consideration is that if a standard function invokes a 
segmentation-aware function, then there is the potential for the 
standard function to pass the address of a safe stack allocation to the 
segmentation-aware function.  This is problematic, because my proposed 
compiler pass for inserting segment override prefixes assumes that 
pointers to the safe stack are not passed between functions.  The pass 
only performs intraprocedural analysis.  If an allocation's address is 
taken and passed to a subroutine, then the segmentation-hardened 
SafeStack pass will move that allocation to the unsafe stack.

My proposed musl patches currently assume that all code that runs after 
restricted segment limits are enabled is segmentation-aware.
>
>> Specifically, https://reviews.llvm.org/D19762 assumes that the
>> unsafe stack pointer is stored at offset 0x24 in the musl thread
>> control block. This would be between the pid and tsd_used variables
>> that are currently defined. I also propose storing the base address
>> of the unsafe stack at offset 0x28, but the compiler would not
>> depend on that.
> Almost none of the existing fields are public; I think the only
> exception is the stack-protector canary. IMO you should prefer
> avoiding per-libc offset variation over preserving existing offsets.
The offset 0x24 is at least consistent with Bionic, according to the 
comments in the definition of 
X86TargetLowering::getSafeStackPointerLocation in 
<LLVM>/lib/Target/X86/X86ISelLowering.cpp.  However, I can't find any 
mention of SafeStack in the Bionic header linked from those LLVM comments.
>> Here is an overview of some other changes that I plan to propose
>> with my musl patches:
>>
>> The segmentation-hardened SafeStack support would be enabled with a
>> new configuration option, "--enable-safe-stack".
>>
>> When this is enabled, many libraries routines require that both a
>> safe stack and an unsafe stack be available. I modified _start_c in
>> crt1.c to temporarily setup a small, pre-allocated unsafe stack for
> I'd have to see exactly what you mean, but my leaning is that crt1 is
> not a good place for anything new. For dynamic-linked programs,
> crt1-to-__libc_start_main is the main permanent ABI boundary and not
> something you want to have complexity that could need changing.
I revised my patches to move the temporary unsafe stack initialization 
out of crt1 and into __libc_start_main.
>> the early initialization routines to use. I also made similar
>> changes in dlstart.c. A larger unsafe stack is allocated and setup
>> later from either __libc_start_main or __dls3, depending on whether
>> static or dynamic linking is used. I split __dls3 so that it only
>> performs minimal initialization before allocating the larger unsafe
>> stack and then performing the rest of its work in a new __dls4
>> function.
>>
>> After the larger unsafe stack is allocated, I invoke the modify_ldt
>> syscall to insert a segment descriptor with a limit that is below
>> the beginning of the safe stacks. I load that segment descriptor
>> into the DS and ES segment registers to block memory accesses to DS
>> and ES from accessing the safe stacks. One purpose of my LLVM and
>> Clang patches is to insert the necessary segment override prefixes
>> to direct accesses to the appropriate segments.
> The content on these stacks is purely return addresses, spills, and
> other stuff that's only accessible to compiler-generated code, not
> data whose addresses can be taken, right?
If an allocation's address is taken and passed as an argument to a 
subroutine, then the segmentation-hardened SafeStack pass will move that 
allocation to the unsafe stack.

Of course, function arguments are only one of many possible mechanisms 
for passing information between functions.  For example, one function 
may attempt to store a safe stack pointer into a structure that gets 
passed to another function, or into a global variable.  By default, my 
compiler pass detects and blocks such memory writes.  However, there are 
instances where such writes are necessary.  For example, the va_list 
object used to support variadic arguments stores a pointer to the safe 
stack.  I implemented other compiler patches to emit the SS segment 
override prefix when accessing variadic arguments, so storing the safe 
stack pointer into the va_list object should be allowed.  The 
intraprocedural analysis attempts to detect this type of write, but it 
currently has limitations on the complexity of pointer computations that 
it can handle.  Thus, I added compiler command line options to 
selectively override this analysis for certain files and allow safe 
stack pointers to be written to memory even when the compiler cannot 
verify that they are being written to va_list objects.

Note that manipulating safe stack addresses in registers within a single 
function is no problem.  For example, traverses_stack_p in expand_heap.c 
compares a safe stack address to other addresses. Actually, 
traverses_stack_p is interesting in other ways, since it assumes  that 
libc.auxv is on the main-thread stack.  My revised patches move auxv to 
the main-thread unsafe stack.  What checks should be performed in 
traverses_stack_p when multiple types of stacks are defined?
>
>> Many instructions expect that argc, argv, the environment, and auxv
>> are accessible in the DS and ES segments. These are stored on the
>> initial stack, which is above the limit of the restricted DS and ES
>> segments. I annotated auxv with an attribute to cause the compiler
>> to emit SS segment-override prefixes when accessing auxv. I copied
>> the other data to the heap, which is accessible in DS and ES.
> Invasive arch-specific changes to unrelated code are highly frowned
> upon in musl. I think to be acceptable upstream at all the auxv would
> also have to be relocated by startup code to an address where it's
> accessible.
I revised my patches to do this.
>> I modified the pthread routines to allocate and deallocate
>> additional stacks as needed in the appropriate memory ranges. The
>> safe stacks are allocated at high addresses so that they are above
>> the limit of the modified DS and ES segments. The unsafe stack for
>> each new thread is allocated below its TLS region and thread control
>> block, which is where the stack is currently located by default.
> This likely should be hidden inside __clone rather than in
> non-arch-specific sources.
I'll send out the current revision of my patches soon to show you 
exactly what I'm proposing and get your feedback on that.
>> The Linux vDSO code may be incompatible with programs that enable
>> segmentation-hardened SafeStack. For example, it may allocate data
>> on the safe stack and then attempt to access it in DS or ES, which
>> would result in an exception due to the segment limit violation. My
>> patches prevent the vDSO from being invoked when
>> segmentation-hardened SafeStack is enabled.
> That sounds reasonable but rather unfortunate.
>
>> Finally, the i386 __clone implementation is written in assembly
>> language, so the compiler is unable to automatically add a stack
>> segment override prefix to an instruction in that routine that
>> accesses a safe stack. I added that prefix manually in the source
>> code.
>>
>> Comments appreciated.
> Hope the above are helpful.
Yes, very.  Thank you!

  - Michael

Re: [RFC] Support for segmentation-hardened SafeStack

[LeMay, Michael]

On 9/23/2016 03:22, Szabolcs Nagy wrote:
> * LeMay, Michael <michael.lemay@intel.com> [2016-09-22 23:00:45 +0000]:
>> I submitted several patches to LLVM and Clang to harden SafeStack using segmentation on x86-32 [1].  See [2] for general background on SafeStack.
> ...
>> [1] http://lists.llvm.org/pipermail/llvm-dev/2016-May/100346.html
>> [2] http://clang.llvm.org/docs/SafeStack.html
> is all runtime support in the libc with your patches?
> (i.e. no static linked interposition code from compiler-rt)

For programs linked against my patched version of musl with 
segmentation-hardened SafeStack enabled, the SafeStack library in 
compiler-rt is not needed.

>
> can you call into non-instrumented code?
> (as Rich noted this looks like a new abi on i386)
> i assume the segmented variant breaks abi while the
> non-segmented one does not.

Restricting segment limits does introduce additional considerations that 
are not applicable to the original version of SafeStack, as I described 
in the reply to Rich that I just sent.

>
> what is the unsafe stack size of the main thread?
> how much is the resource usage overhead?

I arbitrarily chose to allocate a main-thread unsafe stack that is twice 
as large as the main-thread safe stack.  The unsafe stack sizes for new 
threads are computed similarly to the safe stack sizes.  I'll post the 
current revision of my patches soon for the sake of discussion.

>
> what happens if unsafe stack allocation fails?

A limitation of my current patches is that there is no support for 
dynamically expanding the size of the unsafe stack.  By the way, I think 
that this is also a limitation of the current compiler-rt support for 
the original version of SafeStack.

> how does the stack get deallocated at thread exit?
> i assume they are consistent with normal stack
> handling if this is done in musl.. except for the
> main thread.

Yes, the unsafe stack gets deallocated when non-main threads exit.

>
> can signal handlers work with sigaltstack?

That's an interesting question.  One thing to consider is that the 
kernel will only switch the safe stack when sigaltstack is used, not the 
unsafe stack.  Furthermore, for the segmentation-based hardening to 
apply to the stack passed to sigaltstack, that stack would need to be 
allocated above the restricted limits of DS and ES.

Thanks,
Michael

Re: [RFC] Support for segmentation-hardened SafeStack

[Rich Felker]

On Mon, Sep 26, 2016 at 10:28:40AM -0700, LeMay, Michael wrote:
> 
> 
> On 9/22/2016 16:42, Rich Felker wrote:
> >On Thu, Sep 22, 2016 at 11:00:45PM +0000, LeMay, Michael wrote:
> >>Hi,
> >>
> >>I submitted several patches to LLVM and Clang to harden SafeStack
> >>using segmentation on x86-32 [1]. See [2] for general background on
> >>SafeStack. On Linux, I have been testing my compiler changes with a
> >>modified version of musl. I currently plan to submit my musl patches
> >>if and when the prerequisite LLVM and Clang patches are accepted.
> >>One of my LLVM patches depends on the details of my musl patches,
> >>which is the main reason that I am sending this RFC now.
> >My understanding is that this is a different, incompatible ABI for
> >i386, i.e. code that uses safestack is not calling-compatible with
> >code that doesn't, and vice versa. Is that true? This is probably the
> >most significant determining factor in how we treat it.
> That's a good question, but I don't know how to answer it directly.
> 
> A salient requirement is that all code that runs while restricted
> segment limits are in effect for DS and ES must use appropriate
> segment override prefixes.  This is to direct safe stack accesses to
> SS and thus avoid violating the segment limits on DS and ES.  It is
> still possible to call code that does not satisfy that requirement
> (I will refer to such functions as "standard functions", in contrast
> to "segmentation-aware functions") in the same program, but the
> segment registers would need to be reverted to contain flat segment
> descriptors before calling such code.  Otherwise, a segment limit
> violation would occur if a standard function attempted to access
> data on the stack using DS or ES.  Of course, reverting to flat
> segment descriptors would leave the safe stacks unprotected.

In short, non-safestack code called from safestack code would try to
store data on the safe stack, which doesn't/can't work.

This could probably be avoided by a different approach that avoids
call/ret and instead uses manual pushes/pops to the safe stack and
jumps in place of call/ret. Then %ss could point to the non-safe stack
so that calling non-safestack code would work fine (but would be
unprotected as usual).

I suspect this is sufficiently costly to implement (either in
performance or implementation complexity or both) that you're not
interested in doing it that way, but I mention it for completeness.

> Another consideration is that if a standard function invokes a
> segmentation-aware function, then there is the potential for the
> standard function to pass the address of a safe stack allocation to
> the segmentation-aware function.  This is problematic, because my
> proposed compiler pass for inserting segment override prefixes
> assumes that pointers to the safe stack are not passed between
> functions.  The pass only performs intraprocedural analysis.  If an
> allocation's address is taken and passed to a subroutine, then the
> segmentation-hardened SafeStack pass will move that allocation to
> the unsafe stack.

This would also be solved by the above approach, I think.

> My proposed musl patches currently assume that all code that runs
> after restricted segment limits are enabled is segmentation-aware.

That seems reasonable, but then it has to be treated as a separate
subarch or even a separate arch.

> >>After the larger unsafe stack is allocated, I invoke the modify_ldt
> >>syscall to insert a segment descriptor with a limit that is below
> >>the beginning of the safe stacks. I load that segment descriptor
> >>into the DS and ES segment registers to block memory accesses to DS
> >>and ES from accessing the safe stacks. One purpose of my LLVM and
> >>Clang patches is to insert the necessary segment override prefixes
> >>to direct accesses to the appropriate segments.
> >The content on these stacks is purely return addresses, spills, and
> >other stuff that's only accessible to compiler-generated code, not
> >data whose addresses can be taken, right?
> If an allocation's address is taken and passed as an argument to a
> subroutine, then the segmentation-hardened SafeStack pass will move
> that allocation to the unsafe stack.
> 
> Of course, function arguments are only one of many possible
> mechanisms for passing information between functions.  For example,
> one function may attempt to store a safe stack pointer into a
> structure that gets passed to another function, or into a global
> variable.  By default, my compiler pass detects and blocks such
> memory writes.

This makes it sound like you're not implementing C but some weird
C-like language where pointers don't work. From what I can see, to
correctly implement this, "safe stack pointers" have to be an
oxymoron. Any object whose address is ever taken has to live in
addressible memory, i.e. not on the safe-stack.

> However, there are instances where such writes are
> necessary.  For example, the va_list object used to support variadic
> arguments stores a pointer to the safe stack.

Are your "safe stack pointers" just implementation-details like
va_list state? In that case I think they're valid since they can only
be accessed via va_arg. But I'm wondering why the argument list is on
the unsafe stack at all. This mandates that you copy incoming
(non-variadic) arguments rather than using them in-place, which is
very expensive for (moderately-)large aggregate-type arguments.

> I implemented other
> compiler patches to emit the SS segment override prefix when
> accessing variadic arguments, so storing the safe stack pointer into
> the va_list object should be allowed.  The intraprocedural analysis
> attempts to detect this type of write, but it currently has
> limitations on the complexity of pointer computations that it can
> handle.  Thus, I added compiler command line options to selectively
> override this analysis for certain files and allow safe stack
> pointers to be written to memory even when the compiler cannot
> verify that they are being written to va_list objects.

I don't think they should even be able to _arise_ except as
variadic-argument pointers. If they can't arise you don't need to
analyze where they're written.

> Note that manipulating safe stack addresses in registers within a
> single function is no problem.  For example, traverses_stack_p in
> expand_heap.c compares a safe stack address to other addresses.
> Actually, traverses_stack_p is interesting in other ways, since it
> assumes  that libc.auxv is on the main-thread stack.  My revised
> patches move auxv to the main-thread unsafe stack.  What checks
> should be performed in traverses_stack_p when multiple types of
> stacks are defined?

It can be omitted completely for safestack. It's just a fail-safe
workaround for buggy kernels that let you grow the brk across the
stack which presumably you don't have on x86 systems.

Rich

Re: [RFC] Support for segmentation-hardened SafeStack

[LeMay, Michael]

On 9/26/2016 11:08, Rich Felker wrote:
> On Mon, Sep 26, 2016 at 10:28:40AM -0700, LeMay, Michael wrote:
...
>> A salient requirement is that all code that runs while restricted
>> segment limits are in effect for DS and ES must use appropriate
>> segment override prefixes.  This is to direct safe stack accesses to
>> SS and thus avoid violating the segment limits on DS and ES.  It is
>> still possible to call code that does not satisfy that requirement
>> (I will refer to such functions as "standard functions", in contrast
>> to "segmentation-aware functions") in the same program, but the
>> segment registers would need to be reverted to contain flat segment
>> descriptors before calling such code.  Otherwise, a segment limit
>> violation would occur if a standard function attempted to access
>> data on the stack using DS or ES.  Of course, reverting to flat
>> segment descriptors would leave the safe stacks unprotected.
> In short, non-safestack code called from safestack code would try to
> store data on the safe stack, which doesn't/can't work.
>
> This could probably be avoided by a different approach that avoids
> call/ret and instead uses manual pushes/pops to the safe stack and
> jumps in place of call/ret. Then %ss could point to the non-safe stack
> so that calling non-safestack code would work fine (but would be
> unprotected as usual).
>
> I suspect this is sufficiently costly to implement (either in
> performance or implementation complexity or both) that you're not
> interested in doing it that way, but I mention it for completeness.

That's an interesting idea that could offer some benefits as you 
described, but it would indeed also have drawbacks.  One example that 
comes to mind is that to protect the safe stacks, only instructions that 
require access to the safe stacks should be able to use the segment that 
grants such access.  Thus, many instructions that access data sections 
and the heap would need to have segment override prefixes to direct 
their memory accesses to SS.  Using EBP as the base address for some of 
those accesses could reduce the need for segment override prefixes, but 
I don't know to what extent that would help.  A more fundamental 
challenge is that not adhering to the longstanding assumption that the 
stack resides in SS would probably necessitate major compiler 
revisions.  Another challenge is that standard functions would expect 
parameters to be passed on the stack pointed to by ESP, whereas some 
other register would be used as the safe stack pointer in this case.

...
>> However, there are instances where such writes are
>> necessary.  For example, the va_list object used to support variadic
>> arguments stores a pointer to the safe stack.
> Are your "safe stack pointers" just implementation-details like
> va_list state? In that case I think they're valid since they can only
> be accessed via va_arg. But I'm wondering why the argument list is on
> the unsafe stack at all. This mandates that you copy incoming
> (non-variadic) arguments rather than using them in-place, which is
> very expensive for (moderately-)large aggregate-type arguments.

Arguments, whether variadic or not, are still passed on the main (safe) 
stack like usual, and they can be used in-place.

>> I implemented other
>> compiler patches to emit the SS segment override prefix when
>> accessing variadic arguments, so storing the safe stack pointer into
>> the va_list object should be allowed.  The intraprocedural analysis
>> attempts to detect this type of write, but it currently has
>> limitations on the complexity of pointer computations that it can
>> handle.  Thus, I added compiler command line options to selectively
>> override this analysis for certain files and allow safe stack
>> pointers to be written to memory even when the compiler cannot
>> verify that they are being written to va_list objects.
> I don't think they should even be able to _arise_ except as
> variadic-argument pointers. If they can't arise you don't need to
> analyze where they're written.

I refreshed my memory on what allocations the SafeStack pass moves to 
the unsafe stack, and I think you're right.  If a pointer to the safe 
stack would be written to memory (e.g. into a structure passed to 
another function or into a global variable), then the SafeStack pass 
moves the allocation to the unsafe stack.  Sorry to have forgotten that 
when I wrote my previous message.  I found just a couple of exceptions 
to that rule.  Variadic argument handling is one and register spills are 
another.  Additional exceptions exist for certain function calls when 
segmentation-based hardening is disabled, but that's out of scope for 
this discussion.  So, I agree that the verification that I described 
above is unnecessary. However, I still need to track pointers to the 
safe stack in registers when they get spilled and filled so that I can 
emit the proper segment override prefixes.

>
>> Note that manipulating safe stack addresses in registers within a
>> single function is no problem.  For example, traverses_stack_p in
>> expand_heap.c compares a safe stack address to other addresses.
>> Actually, traverses_stack_p is interesting in other ways, since it
>> assumes  that libc.auxv is on the main-thread stack.  My revised
>> patches move auxv to the main-thread unsafe stack.  What checks
>> should be performed in traverses_stack_p when multiple types of
>> stacks are defined?
> It can be omitted completely for safestack. It's just a fail-safe
> workaround for buggy kernels that let you grow the brk across the
> stack which presumably you don't have on x86 systems.

I'll revise my patches to reflect this.

Thanks,
Michael

Re: [RFC] Support for segmentation-hardened SafeStack

[Rich Felker]

On Mon, Sep 26, 2016 at 11:05:06PM -0700, LeMay, Michael wrote:
> 
> 
> On 9/26/2016 11:08, Rich Felker wrote:
> >On Mon, Sep 26, 2016 at 10:28:40AM -0700, LeMay, Michael wrote:
> ....
> >>A salient requirement is that all code that runs while restricted
> >>segment limits are in effect for DS and ES must use appropriate
> >>segment override prefixes.  This is to direct safe stack accesses to
> >>SS and thus avoid violating the segment limits on DS and ES.  It is
> >>still possible to call code that does not satisfy that requirement
> >>(I will refer to such functions as "standard functions", in contrast
> >>to "segmentation-aware functions") in the same program, but the
> >>segment registers would need to be reverted to contain flat segment
> >>descriptors before calling such code.  Otherwise, a segment limit
> >>violation would occur if a standard function attempted to access
> >>data on the stack using DS or ES.  Of course, reverting to flat
> >>segment descriptors would leave the safe stacks unprotected.
> >In short, non-safestack code called from safestack code would try to
> >store data on the safe stack, which doesn't/can't work.
> >
> >This could probably be avoided by a different approach that avoids
> >call/ret and instead uses manual pushes/pops to the safe stack and
> >jumps in place of call/ret. Then %ss could point to the non-safe stack
> >so that calling non-safestack code would work fine (but would be
> >unprotected as usual).
> >
> >I suspect this is sufficiently costly to implement (either in
> >performance or implementation complexity or both) that you're not
> >interested in doing it that way, but I mention it for completeness.
> 
> That's an interesting idea that could offer some benefits as you
> described, but it would indeed also have drawbacks.  One example
> that comes to mind is that to protect the safe stacks, only
> instructions that require access to the safe stacks should be able
> to use the segment that grants such access.  Thus, many instructions
> that access data sections and the heap would need to have segment
> override prefixes to direct their memory accesses to SS.  Using EBP
> as the base address for some of those accesses could reduce the need
> for segment override prefixes, but I don't know to what extent that

If you actually load %ss with the safestack segment, you would not be
able to call non-safestack code since it might use addressing forms
that immplicitly use %ss. Instead, %fs would have to be loaded with
the safestack segment and all jumps for calls and returns would have
to use a %fs prefix to load the address. Of course that makes it all
the more work to implement (and more costly at runtime).

> ....
> >>However, there are instances where such writes are
> >>necessary.  For example, the va_list object used to support variadic
> >>arguments stores a pointer to the safe stack.
> >Are your "safe stack pointers" just implementation-details like
> >va_list state? In that case I think they're valid since they can only
> >be accessed via va_arg. But I'm wondering why the argument list is on
> >the unsafe stack at all. This mandates that you copy incoming
> >(non-variadic) arguments rather than using them in-place, which is
> >very expensive for (moderately-)large aggregate-type arguments.
> 
> Arguments, whether variadic or not, are still passed on the main
> (safe) stack like usual, and they can be used in-place.

Here I think we're just differing on what "used in-place" means. For
me that would include the ability to take their addresses. I assume
you're just talking about using the values.

> >>I implemented other
> >>compiler patches to emit the SS segment override prefix when
> >>accessing variadic arguments, so storing the safe stack pointer into
> >>the va_list object should be allowed.  The intraprocedural analysis
> >>attempts to detect this type of write, but it currently has
> >>limitations on the complexity of pointer computations that it can
> >>handle.  Thus, I added compiler command line options to selectively
> >>override this analysis for certain files and allow safe stack
> >>pointers to be written to memory even when the compiler cannot
> >>verify that they are being written to va_list objects.
> >I don't think they should even be able to _arise_ except as
> >variadic-argument pointers. If they can't arise you don't need to
> >analyze where they're written.
> 
> I refreshed my memory on what allocations the SafeStack pass moves
> to the unsafe stack, and I think you're right.  If a pointer to the
> safe stack would be written to memory (e.g. into a structure passed
> to another function or into a global variable), then the SafeStack
> pass moves the allocation to the unsafe stack.  Sorry to have
> forgotten that when I wrote my previous message.  I found just a
> couple of exceptions to that rule.  Variadic argument handling is
> one and register spills are another.

This is another place where I think we're just using terms
differently. From my perspective (the formal C language) variadic
argument handling does not involve taking or dereferencing addresses
on the stack; those are just va_list/va_arg implementation details. At
the level of the formal language I think there are no exceptions; in
all cases where the address on "the stack" leaks outside the scope of
what the compiler can see/control, "the stack" it's on has to be the
unsafe stack.

Rich

Re: [RFC] Support for segmentation-hardened SafeStack

[LeMay, Michael]

On 9/27/2016 07:43, Rich Felker wrote:
> On Mon, Sep 26, 2016 at 11:05:06PM -0700, LeMay, Michael wrote:
...
>> Arguments, whether variadic or not, are still passed on the main
>> (safe) stack like usual, and they can be used in-place.
> Here I think we're just differing on what "used in-place" means. For
> me that would include the ability to take their addresses. I assume
> you're just talking about using the values.

I see your point now.  Yes, when SafeStack determines that a local 
variable or argument may be accessed unsafely, it moves or copies 
(respectively) that allocation to the unsafe stack.  Incidentally, I 
thought that just taking the address of a local variable or argument 
(e.g. for pointer comparisons within a single function) would not 
necessarily result in it being moved to the unsafe stack, but re-reading 
the SafeStack pass and running some tests showed me that the pass 
currently does move such allocations to the unsafe stack.

...
>
> This is another place where I think we're just using terms
> differently. From my perspective (the formal C language) variadic
> argument handling does not involve taking or dereferencing addresses
> on the stack; those are just va_list/va_arg implementation details. At
> the level of the formal language I think there are no exceptions; in
> all cases where the address on "the stack" leaks outside the scope of
> what the compiler can see/control, "the stack" it's on has to be the
> unsafe stack.

Yes, we're in agreement.  For completeness, I'll note that there are 
other ways for safe stack pointers to leak: 
http://clang.llvm.org/docs/SafeStack.html#known-security-limitations

Thanks,
Michael