[Caml-list] Exception values may crash GC when interfacing C and Caml

[Caml-list] Exception values may crash GC when interfacing C and Caml

[Alexey Rodriguez]

Dear all,

We are experiencing crashes in Caml-calling C code. This happens if
garbage collection runs after Caml code has raised an exception. We
now understand why this happens but we are puzzled as to why the
"Interfacing C with Ocaml" chapter of the Ocaml manual doesn't warn
about this situation.

Suppose you have C code that calls Caml code as follows:

…
CAMLparam2(v1,v2);
CAMLlocal2(…,res);
res = callback2_exn(…,v1,v2);
foobar();
…

We have found that this code will crash with "Fatal error: out of
memory." if the following two things happen:
* the function called by [callback2_exn] raises an exception, and
* [foobar] triggers a garbage collection through the allocation of
values in the Caml heap. (just calling [caml_gc_full_major] is enough
to cause the crash).

The reason for this crash is that [res] will contain an invalid
pointer if an exception is thrown. The GC follows this bogus pointer
([res] is registered as a root by [CAMLlocal2]) which ultimately
causes a crash in the GC code. Why does [res] contain a bogus pointer?
It's not really a bogus pointer, but the lower bits are tagged in
order to denote a thrown exception. These bits are usually
tested/cleared by [Is_exception_result] and [Extract_exception].

We observed all this in an x86_64 platform but I can imagine this also
happens in other platforms. We are using Ocaml 3.12 and Ubuntu 11.10.
The example program is based on the one given at
http://caml.inria.fr/pub/docs/oreilly-book/html/book-ora117.html

We could do something like this to avoid the problem:

…
CAMLparam2(v1,v2);
CAMLlocal2(…,res);
bool was_exn=false;
res = callback2_exn(…,v1,v2);
if(Is_exception_result(res)) {
  was_exn=true;
  res = Extract_exception(res);
}
foobar(); // Now it's safe to Caml-allocate in foobar
…

It's a bit ugly but it could be encapsulated in a macro. The question
I have now is why isn't the GC+exn crash mentioned in the
documentation? Is there some best practice that we are missing from
the documentation?

Cheers,

Alexey

--
Alexey Rodriguez Yakushev

O +31 (0)40 8200960   |  D + 31 (0)40 8200974  |  F +31 (0)40 8200979
Vonderweg 22, 5616 RM  |  Eindhoven |  The Netherlands
www.vectorfabrics.com  |  alexey@vectorfabrics.com

RE: [Caml-list] Exception values may crash GC when interfacing C and Caml

[David Allsopp]

Alexey Rodriguez wrote:
> Dear all,
> 
> We are experiencing crashes in Caml-calling C code. This happens if
> garbage collection runs after Caml code has raised an exception. We now
> understand why this happens but we are puzzled as to why the "Interfacing
> C with Ocaml" chapter of the Ocaml manual doesn't warn about this
> situation.
> 
> Suppose you have C code that calls Caml code as follows:
> 
> ...
> CAMLparam2(v1,v2);
> CAMLlocal2(...,res);
> res = callback2_exn(...,v1,v2);
> foobar();
> ...
> 
> We have found that this code will crash with "Fatal error: out of memory."
> if the following two things happen:
> * the function called by [callback2_exn] raises an exception, and
> * [foobar] triggers a garbage collection through the allocation of values
> in the Caml heap. (just calling [caml_gc_full_major] is enough to cause
> the crash).
> 
> The reason for this crash is that [res] will contain an invalid pointer if
> an exception is thrown. The GC follows this bogus pointer ([res] is
> registered as a root by [CAMLlocal2]) which ultimately causes a crash in
> the GC code. Why does [res] contain a bogus pointer?
> It's not really a bogus pointer, but the lower bits are tagged in order to
> denote a thrown exception. These bits are usually tested/cleared by
> [Is_exception_result] and [Extract_exception].

This is already in the manual, but I agree that the requirement to do so could be stated more clearly. Section 18.7.1[1], last paragraph states "The return v of the caml_callback*_exn function **must** be tested with the macro Is_exception_result(v)". It also clearly indicates that v is only a valid [value] if Is_exception_result(v) returns false so storing the return of caml_callback*_exn in a local root and allowing the Gc to run before you update that root with the result of Extract_exception is "obviously" a Gc violation.


David

[1] http://caml.inria.fr/pub/docs/manual-ocaml/manual032.html#htoc245

Re: [Caml-list] Exception values may crash GC when interfacing C and Caml

[Alexey Rodriguez]

[-- Attachment #1: Type: text/plain, Size: 2629 bytes --]

I added a warning to the documentation. Can someone with commit rights
have a look at it and apply it? This would have definitely saved us a
lot of time. And I am sure this will help other people too.

Cheers,

Alexey

On Wed, May 2, 2012 at 1:28 PM, David Allsopp <dra-news@metastack.com> wrote:
> Alexey Rodriguez wrote:
>> Dear all,
>>
>> We are experiencing crashes in Caml-calling C code. This happens if
>> garbage collection runs after Caml code has raised an exception. We now
>> understand why this happens but we are puzzled as to why the "Interfacing
>> C with Ocaml" chapter of the Ocaml manual doesn't warn about this
>> situation.
>>
>> Suppose you have C code that calls Caml code as follows:
>>
>> ...
>> CAMLparam2(v1,v2);
>> CAMLlocal2(...,res);
>> res = callback2_exn(...,v1,v2);
>> foobar();
>> ...
>>
>> We have found that this code will crash with "Fatal error: out of memory."
>> if the following two things happen:
>> * the function called by [callback2_exn] raises an exception, and
>> * [foobar] triggers a garbage collection through the allocation of values
>> in the Caml heap. (just calling [caml_gc_full_major] is enough to cause
>> the crash).
>>
>> The reason for this crash is that [res] will contain an invalid pointer if
>> an exception is thrown. The GC follows this bogus pointer ([res] is
>> registered as a root by [CAMLlocal2]) which ultimately causes a crash in
>> the GC code. Why does [res] contain a bogus pointer?
>> It's not really a bogus pointer, but the lower bits are tagged in order to
>> denote a thrown exception. These bits are usually tested/cleared by
>> [Is_exception_result] and [Extract_exception].
>
> This is already in the manual, but I agree that the requirement to do so could be stated more clearly. Section 18.7.1[1], last paragraph states "The return v of the caml_callback*_exn function **must** be tested with the macro Is_exception_result(v)". It also clearly indicates that v is only a valid [value] if Is_exception_result(v) returns false so storing the return of caml_callback*_exn in a local root and allowing the Gc to run before you update that root with the result of Extract_exception is "obviously" a Gc violation.
>
>
> David
>
> [1] http://caml.inria.fr/pub/docs/manual-ocaml/manual032.html#htoc245



-- 
dr. Alexey Rodriguez Yakushev

Vector Fabrics included in EE Times 'Silicon 60' list of emerging startups

O +31 (0)40 8200960   |  D + 31 (0)40 8200974  |  F +31 (0)40 8200979
Vonderweg 22, 5616 RM  |  Eindhoven |  The Netherlands
www.vectorfabrics.com  |  alexey@vectorfabrics.com

[-- Attachment #2: warning_exception.patch --]
[-- Type: application/octet-stream, Size: 818 bytes --]

Index: manual/cmds/intf-c.etex
===================================================================
--- manual/cmds/intf-c.etex	(revision 12413)
+++ manual/cmds/intf-c.etex	(working copy)
@@ -1273,6 +1273,12 @@
 an exception escaped, and its value (the exception descriptor) can be
 recovered using "Extract_exception("\var{v}")".
 
+\paragraph{Warning:} If an exception did escape the OCaml function, \var{v}
+does not contain a valid value. It follows that prior to calling a memory-allocating
+function, \var{v} should not be reachable by the garbage collector. For instance,
+\var{v} should not be assigned to a variable declared with one of the "CAMLlocal*"
+macros.
+
 \subsection{Obtaining or registering OCaml closures for use in C functions}
 
 There are two ways to obtain OCaml function values (closures) to

Re: [Caml-list] Exception values may crash GC when interfacing C and Caml

[Wojciech Meyer]

Dear Alexey,

Alexey Rodriguez <alexey@vectorfabrics.com> writes:

> I added a warning to the documentation. Can someone with commit rights
> have a look at it and apply it? This would have definitely saved us a
> lot of time. And I am sure this will help other people too.

Yes, patches to the documentation are always welcome!

Please use [1] to file a ticket and describe the change and/or propose a
patch.

[1] http://caml.inria.fr/mantis/my_view_page.php

Regards,

- Wojciech

Re: [Caml-list] Exception values may crash GC when interfacing C and Caml

[Alexey Rodriguez]

Thank you Wojciech. I just added ticket 5606 [1] with the patch to the
documentation.

Cheers,

Alexey

[1] http://caml.inria.fr/mantis/view.php?id=5606

On Wed, May 2, 2012 at 9:37 PM, Wojciech Meyer
<wojciech.meyer@googlemail.com> wrote:
>
> Dear Alexey,
>
> Alexey Rodriguez <alexey@vectorfabrics.com> writes:
>
>> I added a warning to the documentation. Can someone with commit rights
>> have a look at it and apply it? This would have definitely saved us a
>> lot of time. And I am sure this will help other people too.
>
> Yes, patches to the documentation are always welcome!
>
> Please use [1] to file a ticket and describe the change and/or propose a
> patch.
>
> [1] http://caml.inria.fr/mantis/my_view_page.php
>
> Regards,
>
> - Wojciech



-- 
Alexey Rodriguez Yakushev

O +31 (0)40 8200960   |  D + 31 (0)40 8200974  |  F +31 (0)40 8200979
Vonderweg 22, 5616 RM  |  Eindhoven |  The Netherlands
www.vectorfabrics.com  |  alexey@vectorfabrics.com