Native code stack overflow detection guarantees

[Michael Ekstrand <michael@elehack.net>]

Some time ago, I saw someone mention non-tail-recursive functions in
native code as a security problem.  Unfortunately, I cannot find where I
read that again, but the basic idea was that, if you have a recursive
function that uses stack linear in user-provided input, then the user
can trigger a stack overflow which, in native code, can allow your stack
pointer to go waltzing through memory and wreak general havoc since
stack overflows are not trapped.

I was attempting to do some further research on this subject today and
found section 11.5 in the OCaml manual (ocamlopt compatibility with the
bytecode compiler), where it described the stack overflow behavior:

 * By raising a Stack_overflow exception, like the bytecode compiler
does. (IA32/Linux, AMD64/Linux, PowerPC/MacOSX, MS Windows 32-bit ports).
 * By aborting the program on a “segmentation fault” signal. (All other
Unix systems.)
 * By terminating the program silently. (MS Windows 64 bits).

I have also turned up mailing list posts indicating that the stack
overflow exception is generated by a sigsegv signal handler on Linux.

My goal is to determine whether the ability for user input to cause a
stack overflow is merely a denial-of-service vulnerability (they shut
down the process, but can't do anything else), or whether it represents
an attack vector whereby they could modify memory, execute arbitrary
code, or perform other such shenanigans.

Therefore, I am wondering: are there documented guarantees on which the
native code stack overflow behavior rests?  Linux processes usually
receive a segmentation fault when they run out of stack space; is that
guaranteed, or is it simply the usual convenient behavior?  What about
for other systems?

If the program will always die, either via Stack_overflow or a
segmentation fault, as soon as the stack overflow occurs before there is
opportunity for more thorough memory corruption, then I think I can take
user-triggerable stack overflows to be inconvenient and undesirable but
not a viable attack vector other than for DoS attacks.

Thank you for any answers or insight you might be able to provide.

- Michael