Changes for upstream?

Changes for upstream?

[Jewell Seay]

[-- Attachment #1: Type: text/plain, Size: 778 bytes --]

Hello,

The team I am on is in the beginning stages of making the following changes to musl, would upstream desire any of these?

- Heap hardening: adding cookies and validation to increase the likelihood of crashing if someone corrupts heap memory (as a security mitigation).
- Randomizing library locations in memory (while keeping the ordering of module _init and _fini calls stable).
- Shrink the memory footprint of the DATA and BSS sections.
- Return memory to the kernel within free().

The other question we have is that it does not appear that there is any standard way in musl to have certain functionality turned on or off. If any of these changes are desired to be optional then is there an accepted method for enabling or disabling the feature?

Jewell

[-- Attachment #2: Type: text/html, Size: 2852 bytes --]

Re: Changes for upstream?

[Rich Felker]

On Tue, Apr 02, 2019 at 07:11:14PM +0000, Jewell Seay wrote:
> Hello,
> 
> The team I am on is in the beginning stages of making the following
> changes to musl, would upstream desire any of these?
> 
> - Heap hardening: adding cookies and validation to increase the
> likelihood of crashing if someone corrupts heap memory (as a
> security mitigation).

Additional validation is desirable, but the existing malloc
implementation in musl is not intended to be kept long-term, so doing
and reviewing work on it is maybe not a good use of time...

> - Randomizing library locations in memory (while keeping the ordering
> of module _init and _fini calls stable).

I'm confused why this would be in musl and not just by the kernel's
normal ASLR. On 32-bit you really can't randomize positions because
the address space is just too small, but on 64-bit, doing strong mmap
ASLR in the kernel would get you this for free. Do you have a
different/better method in mind that could be achieved in the dynamic
linker but not in the kernel?

> - Shrink the memory footprint of the DATA and BSS sections.

That would be nice, but for the most part I don't think any further
reduction is possible without introducing places where no forward
progress is possible because memory needed was not reserved in
advance. Obviously the stdio buffer sizes could be decreased but that
would hurt performance a lot.

> - Return memory to the kernel within free().

This is already done via MADV_DONTNEED. Using MADV_FREE has been
suggested recently and would probably be a good idea; it looks like it
could be swapped in without any complex work. There are also a few
past threads where I discussed a desire to return not just the dirty
page pressure, but the commit charge, to the kernel when there are
huge free chunks. There are tradeoffs involved, and with the current
allocator slated to be replaced, I didn't really want to spend a lot
of effort considering how they'd fit in with it.

> The other question we have is that it does not appear that there is
> any standard way in musl to have certain functionality turned on or
> off. If any of these changes are desired to be optional then is
> there an accepted method for enabling or disabling the feature?

It's intentional that we don't have functionality switches, but of
course due to CFLAGS various degrees of hardeing are already possible
(stack protector, stack clash check, etc., even UBSan) so it's
conceivable that we could add things that require conditionals at the
source level. Any such things should not affect the exposed interfaces
or the observable behavior of programs with well-defined behavior. But
ideally good hardening measures are sufficiently inexpensive that
making them optional is not needed. The main cost, which should always
be minimized, is the source-level complexity, and that's same or worse
when they're optional.

Rich

RE: Changes for upstream?

[Jewell Seay]

> On Tue, Apr 02, 2019 at 07:11:14PM +0000, Jewell Seay wrote:
> > Hello,
> >
> > The team I am on is in the beginning stages of making the following
> > changes to musl, would upstream desire any of these?
> >
> > - Heap hardening: adding cookies and validation to increase the
> > likelihood of crashing if someone corrupts heap memory (as a security
> > mitigation).
> 
> Additional validation is desirable, but the existing malloc implementation in
> musl is not intended to be kept long-term, so doing and reviewing work on it
> is maybe not a good use of time...

Thank you for the heads up.

> 
> > - Randomizing library locations in memory (while keeping the ordering
> > of module _init and _fini calls stable).
> 
> I'm confused why this would be in musl and not just by the kernel's normal
> ASLR. On 32-bit you really can't randomize positions because the address
> space is just too small, but on 64-bit, doing strong mmap ASLR in the kernel
> would get you this for free. Do you have a different/better method in mind
> that could be achieved in the dynamic linker but not in the kernel?
> 

We have already modified the kernel under ARM to have a wider ASLR range, however this does not improve the Linux design where libraries are loaded in-order in memory by the dynamic loader. A single address leak of a library location allows easy address calculation to any other library for rop gadgets. Although the 32-bit address space is small, shuffling the order the libraries are located in memory results in more complex leaks being required to obtain needed library addresses for exploitation.

It is more specific to our system as any normal system can just load a library to a random location while we are attempting to minimize cache hits so there is a desire on our side to group them together just in different orders.
 
> > - Shrink the memory footprint of the DATA and BSS sections.
> 
> That would be nice, but for the most part I don't think any further reduction
> is possible without introducing places where no forward progress is possible
> because memory needed was not reserved in advance. Obviously the stdio
> buffer sizes could be decreased but that would hurt performance a lot.
> 

Thank you for the heads up. Our current build shows 2 pages of memory being used and we are attempting to get it down to 1 page due to memory impact.

> > - Return memory to the kernel within free().
> 
> This is already done via MADV_DONTNEED. Using MADV_FREE has been
> suggested recently and would probably be a good idea; it looks like it could
> be swapped in without any complex work. There are also a few past threads
> where I discussed a desire to return not just the dirty page pressure, but the
> commit charge, to the kernel when there are huge free chunks. There are
> tradeoffs involved, and with the current allocator slated to be replaced, I
> didn't really want to spend a lot of effort considering how they'd fit in with it.
> 

What allocator do you plan to go to or are there plans to have one created from scratch?

> > The other question we have is that it does not appear that there is
> > any standard way in musl to have certain functionality turned on or
> > off. If any of these changes are desired to be optional then is there
> > an accepted method for enabling or disabling the feature?
> 
> It's intentional that we don't have functionality switches, but of course due
> to CFLAGS various degrees of hardeing are already possible (stack protector,
> stack clash check, etc., even UBSan) so it's conceivable that we could add
> things that require conditionals at the source level. Any such things should
> not affect the exposed interfaces or the observable behavior of programs
> with well-defined behavior. But ideally good hardening measures are
> sufficiently inexpensive that making them optional is not needed. The main
> cost, which should always be minimized, is the source-level complexity, and
> that's same or worse when they're optional.
> 

If I am understanding correctly, any useful hardening measures should just expect to be on and not an optional compilation flag.

Re: Changes for upstream?

[Rich Felker]

On Tue, Apr 02, 2019 at 08:24:15PM +0000, Jewell Seay wrote:
> > > - Randomizing library locations in memory (while keeping the ordering
> > > of module _init and _fini calls stable).
> > 
> > I'm confused why this would be in musl and not just by the kernel's normal
> > ASLR. On 32-bit you really can't randomize positions because the address
> > space is just too small, but on 64-bit, doing strong mmap ASLR in the kernel
> > would get you this for free. Do you have a different/better method in mind
> > that could be achieved in the dynamic linker but not in the kernel?
> > 
> 
> We have already modified the kernel under ARM to have a wider ASLR
> range, however this does not improve the Linux design where
> libraries are loaded in-order in memory by the dynamic loader. A

As I understand it, some of the hardened kernel mods randomize the
relative location of all mmaps, not just the initial mmap base. This
seems like a better approach than doing it specifically in the dynamic
linker, since malloc and other users of mmap would also benefit from
the additional randomness. But as noted, you need a huge address space
for that to work well.

> single address leak of a library location allows easy address
> calculation to any other library for rop gadgets. Although the
> 32-bit address space is small, shuffling the order the libraries are
> located in memory results in more complex leaks being required to
> obtain needed library addresses for exploitation.
> 
> It is more specific to our system as any normal system can just load
> a library to a random location while we are attempting to minimize
> cache hits so there is a desire on our side to group them together
> just in different orders.

If you really want to shuffle just libraries, and not mmap in general,
I think it should be done by either wrapping mmap where ldso calls it
from map_library, or shuffling a whole batch of newly-loaded libraries
with mremap after success of the load has been committed. Compared to
randomizing order, both of the above two approaches avoid adding
complexity to existing program logic and avoid introduction of other
possibly-observable side effects (you mentioned ctor order, but there
could be symbol priority order, etc. too).

> > > - Return memory to the kernel within free().
> > 
> > This is already done via MADV_DONTNEED. Using MADV_FREE has been
> > suggested recently and would probably be a good idea; it looks like it could
> > be swapped in without any complex work. There are also a few past threads
> > where I discussed a desire to return not just the dirty page pressure, but the
> > commit charge, to the kernel when there are huge free chunks. There are
> > tradeoffs involved, and with the current allocator slated to be replaced, I
> > didn't really want to spend a lot of effort considering how they'd fit in with it.
> > 
> 
> What allocator do you plan to go to or are there plans to have one
> created from scratch?

The plan is something new from scratch. Almost everything out there
now trades massive amounts of fragmentation and overall memory usage,
and negative imppact on hardening, for dubious performance gains. I've
recently had some new ideas about solving some of the problems I
didn't know how to solve before, so I'm planning to start a new thread
on this topic soon, probably shortly after 1.1.22 release.

> > > The other question we have is that it does not appear that there is
> > > any standard way in musl to have certain functionality turned on or
> > > off. If any of these changes are desired to be optional then is there
> > > an accepted method for enabling or disabling the feature?
> > 
> > It's intentional that we don't have functionality switches, but of course due
> > to CFLAGS various degrees of hardeing are already possible (stack protector,
> > stack clash check, etc., even UBSan) so it's conceivable that we could add
> > things that require conditionals at the source level. Any such things should
> > not affect the exposed interfaces or the observable behavior of programs
> > with well-defined behavior. But ideally good hardening measures are
> > sufficiently inexpensive that making them optional is not needed. The main
> > cost, which should always be minimized, is the source-level complexity, and
> > that's same or worse when they're optional.
> 
> If I am understanding correctly, any useful hardening measures
> should just expect to be on and not an optional compilation flag.

I don't think that needs to be an absolute hard rule, but it's a good
goal to have in mind. Making something switchable should be a last
resort, not the default course of action.

Rich