Re: [Caml-list] Native code stack overflow detection guarantees

[Goswin von Brederlow <goswin-v-b@web.de>]

Michael Ekstrand <michael@elehack.net> writes:

> Some time ago, I saw someone mention non-tail-recursive functions in
> native code as a security problem.  Unfortunately, I cannot find where I
> read that again, but the basic idea was that, if you have a recursive
> function that uses stack linear in user-provided input, then the user
> can trigger a stack overflow which, in native code, can allow your stack
> pointer to go waltzing through memory and wreak general havoc since
> stack overflows are not trapped.

For that to happen you would have to get the stack pointer to overflow
so far that it actualy points into an allocated memory region
again. Stack frames usualy aren't that big and I'm pretty certain there
will be some unallocated space around the stack to catch
overflows. Isn't a stackframe for a recursive call in ocaml limited in
size (<< PAGE_SIZE)? Unless you have some varargs in there.

I don't see any security probem there other than DOS attacks. With an
exception you could catch it and continue running while a segfault kills
your program (usualy). So for native code you would have to inspect your
input and check if it will stack overflow before calling the recursive
function. Or just write the function tail recursive.

MfG
        Goswin