malloc and linux memory layout

malloc and linux memory layout

[Rich Felker]

Since Luka has had a number of questions about when/why malloc could
fail, heap/brk/mmap allocation, etc. I thought I'd post a quick
description of the virtual address space is managed on Linux:

Each process has its own 32- or 64-bit virtual address space.
Initially, from bottom to top, it looks something like:

[low unmappable pages]
[main program text (code) segment]
[main program data segment]
[brk segment (heap)]
[....lots of open virtual address space...]
[mmap zone]
[main thread stack]
[reserved for kernelspace use]

(Note that there will be small randomized amounts of empty/unused
address space between these regions if ALSR is enabled.)x

The brk segment starts just above the program's static data and grows
upward into the big open space. The mmap zone (where mmaps are put by
default) starts just below the stack limit and continues downward as
more mappings are made. Previously-mapped ranges here are available
for reuse after they're unmapped; that's managed by the kernel. The
brk is not allowed to run into any other mapping when it grows upward,
so if there's a mapping right above the brk segment, attempts to grow
it will always fail.

Shared libraries are mapped in the mmap zone just like another other
mmap.

Typically malloc implementations will use a mix of brk and mmap to
satisfy malloc requests, using individual mmaps for large allocations
(letting the kernel take care of it and making it easy to free the
whole thing back to the system) and carving up the brk segment (and
possibly additional mmapped regions) to handle small allocations (this
avoids the overhead of syscalls and page alignment for small
allocations). musl only uses brk for small allocations, but glibc's
ptmalloc and others are happy to use mmap to make a new "heap region"
if the brk can't grow. The difference will generally only be seen if
virtual address space has already been exhausted.

Hope this helps..

Rich

Re: malloc and linux memory layout

[Vasiliy Kulikov]

On Wed, Aug 10, 2011 at 15:56 -0400, Rich Felker wrote:
> Each process has its own 32- or 64-bit virtual address space.
> Initially, from bottom to top, it looks something like:
> 
> [low unmappable pages]
> [main program text (code) segment]
> [main program data segment]
> [brk segment (heap)]
> [....lots of open virtual address space...]
> [mmap zone]
> [main thread stack]
> [reserved for kernelspace use]
> 
> (Note that there will be small randomized amounts of empty/unused
> address space between these regions if ALSR is enabled.)x
> 
> The brk segment starts just above the program's static data and grows
> upward into the big open space.

Btw, (randomize_va_space):

http://www.mjmwired.net/kernel/Documentation/sysctl/kernel.txt

> The mmap zone (where mmaps are put by
> default) starts just below the stack limit and continues downward as
> more mappings are made.

Not only this zone.  mmap() can return address before main program text if
there is enough space there.


Other minor (Linux-specific) things:

Low unmappable pages region might be absent if the task has CAP_SYS_RAWIO
capability.  But pages before mmap_min_addr will be mmap'ed only by
explicit mmap(addr, ..., MAP_FIXED, ...), no libs will be there.

"Reserved for kernelspace use" region might be absent too for 32-bit
tasks running on 286-64 system.

There could be some specific pages after the stack.  On x86-64 it is
VDSO and vsyscall pages.

-- 
Vasiliy

Re: malloc and linux memory layout

[Rich Felker]

On Thu, Aug 11, 2011 at 11:43:05AM +0400, Vasiliy Kulikov wrote:
> > The mmap zone (where mmaps are put by
> > default) starts just below the stack limit and continues downward as
> > more mappings are made.
> 
> Not only this zone.  mmap() can return address before main program text if
> there is enough space there.

Interesting.

> Other minor (Linux-specific) things:
> 
> Low unmappable pages region might be absent if the task has CAP_SYS_RAWIO
> capability.  But pages before mmap_min_addr will be mmap'ed only by
> explicit mmap(addr, ..., MAP_FIXED, ...), no libs will be there.

This is mostly irrelevant since it requires root or equivalent.
(CAP_SYS_RAWIO might as well be root since it lets you exploit
kernelspace null pointer dereference bugs to get root).

> "Reserved for kernelspace use" region might be absent too for 32-bit
> tasks running on 286-64 system.

Nice.

Rich