Heaps size problems with "caml_alloc_small" in foreign function interfaces

Heaps size problems with "caml_alloc_small" in foreign function interfaces

[Sean Seefried]

Hi,

I'm having a problem where sometimes a call to "caml_alloc_small" from  
C results in a segmentation fault. If I increase the size of the stack  
using OCAMLRUNPARAM=s=1000k then I don't get the crash anymore. It  
seems strange that I have to increase the size of the heap manually  
like this. Is this because I'm calling this function from C?

If I want to increase the size of the heap in C how do I do this?  
Could I write a "safe" caml_alloc_small which first checks to see if  
there is enough memory and then increases the heap size if not?

Sean

Re: [Caml-list] Heaps size problems with "caml_alloc_small" in foreign function interfaces

[Richard Jones]

On Fri, Jul 11, 2008 at 06:21:51PM +1000, Sean Seefried wrote:
> I'm having a problem where sometimes a call to "caml_alloc_small" from  
> C results in a segmentation fault. If I increase the size of the stack  
> using OCAMLRUNPARAM=s=1000k then I don't get the crash anymore. It  
> seems strange that I have to increase the size of the heap manually  
> like this. Is this because I'm calling this function from C?

Seems like you have confusion over heap & stack.

First of all, caml_alloc_small is limited to small allocations, so
number of words allocated must be <= Max_young_wosize.  You don't post
any example code so we can't see whether that is true in your code.

Secondly (and much more likely to be the problem), the caml_alloc*
functions allocate uninitialised memory.  If the garbage collector
gets to run before you've properly initialized all the fields then the
GC will hit an uninitialised field and a segfault could be the result.

  eg: This is a fail:

  v = caml_alloc (2, 0);
  vv = caml_alloc (3, 0);  /* GC could run here */
  Store_field (v, 0, vv);

Changing the _stack_ size (or other tunables) probably just changes
something about when the garbage collector runs, and thus moves the
bug around.

> If I want to increase the size of the heap in C how do I do this?  
> Could I write a "safe" caml_alloc_small which first checks to see if  
> there is enough memory and then increases the heap size if not?

The "size of the heap in C" is (for most operating systems) extended
automatically by malloc.  What you're saying here isn't necessary -
you must have some other bug.

I suggest you post some code which exhibits the problem.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Heaps size problems with "caml_alloc_small" in foreign function interfaces

[Richard Jones]

On Fri, Jul 11, 2008 at 10:40:50AM +0100, Richard Jones wrote:
>   v = caml_alloc (2, 0);
>   vv = caml_alloc (3, 0);  /* GC could run here */

Ick, actually caml_alloc is OK, it's only caml_alloc_small which
doesn't initialize.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Heaps size problems with "caml_alloc_small" in foreign function interfaces

[Sean Seefried]

On 11/07/2008, at 7:40 PM, Richard Jones wrote:

> On Fri, Jul 11, 2008 at 06:21:51PM +1000, Sean Seefried wrote:
>> I'm having a problem where sometimes a call to "caml_alloc_small"  
>> from
>> C results in a segmentation fault. If I increase the size of the  
>> stack
>> using OCAMLRUNPARAM=s=1000k then I don't get the crash anymore. It
>> seems strange that I have to increase the size of the heap manually
>> like this. Is this because I'm calling this function from C?
>
> Seems like you have confusion over heap & stack.
>
> First of all, caml_alloc_small is limited to small allocations, so
> number of words allocated must be <= Max_young_wosize.  You don't post
> any example code so we can't see whether that is true in your code.
>

caml_alloc_small(32,0);

is what seemed to cause the problem. I'm shying away from posting more  
code since it is auto-generated from CamlIDL and is very verbose. I  
didn't write it myself.

> Secondly (and much more likely to be the problem), the caml_alloc*
> functions allocate uninitialised memory.  If the garbage collector

What do you mean by "uninitialised memory"?

> gets to run before you've properly initialized all the fields then the
> GC will hit an uninitialised field and a segfault could be the result.
>
>  eg: This is a fail:
>
>  v = caml_alloc (2, 0);
>  vv = caml_alloc (3, 0);  /* GC could run here */
>  Store_field (v, 0, vv);
>

What do you do about this? Is there a way to stop the GC from running  
for a period of time?

> Changing the _stack_ size (or other tunables) probably just changes
> something about when the garbage collector runs, and thus moves the
> bug around.
>
>> If I want to increase the size of the heap in C how do I do this?
>> Could I write a "safe" caml_alloc_small which first checks to see if
>> there is enough memory and then increases the heap size if not?
>
> The "size of the heap in C" is (for most operating systems) extended
> automatically by malloc.  What you're saying here isn't necessary -
> you must have some other bug.
>

I wasn't very clear but I meant changing the OCaml heap size using C  
functions.

Sean

Re: [Caml-list] Heaps size problems with "caml_alloc_small" in foreign function interfaces

[Richard Jones]

On Fri, Jul 11, 2008 at 08:21:45PM +1000, Sean Seefried wrote:
> caml_alloc_small(32,0);
> 
> is what seemed to cause the problem. I'm shying away from posting more  
> code since it is auto-generated from CamlIDL and is very verbose. I  
> didn't write it myself.

You need to post a reproducer.

> >Secondly (and much more likely to be the problem), the caml_alloc*
> >functions allocate uninitialised memory.  If the garbage collector
> 
> What do you mean by "uninitialised memory"?

Memory which hasn't been explicitly initialized, and so contains
random stuff.  The GC attempts to interpret the random stuff and
fails.

[...]
> What do you do about this? Is there a way to stop the GC from running  
> for a period of time?

The GC could possibly run any time you call an allocation function.
So don't call allocation functions.

Anyhow, you really need to post some code at this point.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Heaps size problems with "caml_alloc_small" in foreign function interfaces

[Xavier Leroy]

> I'm having a problem where sometimes a call to "caml_alloc_small" from C
> results in a segmentation fault. If I increase the size of the stack
> using OCAMLRUNPARAM=s=1000k then I don't get the crash anymore. It seems
> strange that I have to increase the size of the heap manually like this.

It's probably a root registration problem.  These are very sensitive
to the times when GC is triggered, which themselves are sensitive to
the heap sizes and memory behavior of your program.

> If I want to increase the size of the heap in C how do I do this? Could
> I write a "safe" caml_alloc_small which first checks to see if there is
> enough memory and then increases the heap size if not?

Don't try to hack around the real problem, but do make available a
repro case, no matter how large, on a Web site or as attachment to a
problem report on the bug tracking system, so that others can have a
look at it.

- Xavier Leroy

Re: [Caml-list] Heaps size problems with "caml_alloc_small" in foreign function interfaces

[Sean Seefried]

Could you give me a pointer to information on root registration?

What frustrates me is that this is CamlIDL generated code. Shouldn't  
it just work?

Sean

On 12/07/2008, at 0:11, Xavier Leroy <Xavier.Leroy@inria.fr> wrote:

>> I'm having a problem where sometimes a call to "caml_alloc_small"  
>> from C
>> results in a segmentation fault. If I increase the size of the stack
>> using OCAMLRUNPARAM=s=1000k then I don't get the crash anymore. It  
>> seems
>> strange that I have to increase the size of the heap manually like  
>> this.
>
> It's probably a root registration problem.  These are very sensitive
> to the times when GC is triggered, which themselves are sensitive to
> the heap sizes and memory behavior of your program.
>
>> If I want to increase the size of the heap in C how do I do this?  
>> Could
>> I write a "safe" caml_alloc_small which first checks to see if  
>> there is
>> enough memory and then increases the heap size if not?
>
> Don't try to hack around the real problem, but do make available a
> repro case, no matter how large, on a Web site or as attachment to a
> problem report on the bug tracking system, so that others can have a
> look at it.
>
> - Xavier Leroy

Re: [Caml-list] Heaps size problems with "caml_alloc_small" in foreign function interfaces

[Sean Seefried]

[-- Attachment #1: Type: text/plain, Size: 1219 bytes --]

>>

Hi Xavier,

Okay, here's some further clarification and this time I'm attaching a  
file. I've had to GZip it because it's so big.

The bug in question occurs on line 4078.

I should clarify. This file becomes part of a library that is linked  
into another OCaml program.  On one input file that this OCaml program  
takes as input it fails on line 4078.  On other files it fails on  
other lines. However, the code it fails at always seems to be of the  
form:

     _v2 = camlidl_alloc_small(18, 0);
     { mlsize_t _c8;
       for (_c8 = 0; _c8 < 18; _c8++) Field(_v2, _c8) = _v3[_c8];
     }

What's strange about this is that this is automatically generated code  
from CamlIDL (with a few extra debugging messages I threw in). Xavier,  
I'm using CamlDL to write a binding to a  C front-end which has an  
absolutely huge C data structure.  I've had to write several  
extensions to CamlIDL to make this a reality, but for the moment I'm  
keeping these in my own source control mainly because your coding  
style is very clean and many of my extensions could have been done a  
lot more elegantly.

However, the code that is being generated above was already generated  
by version 1.05 of CamlIDL.




Sean

[-- Attachment #2.1: Type: text/html, Size: 1972 bytes --]

[-- Attachment #2.2: test.c.gz --]
[-- Type: application/x-gzip, Size: 116331 bytes --]

[-- Attachment #2.3: Type: text/html, Size: 164 bytes --]