Disabling the OCaml garbage collector

Disabling the OCaml garbage collector

[Raj Bandyopadhyay]

Dear all

I am writing a foreign function interface between OCaml and another  
language (Python) which works via C. Since I have some memory issues  
currently, I would like to be able to disable the OCaml Garbage  
collector before jumping into C/Python, and re-enable it when I  
return to OCaml.

I am using the OCaml/C interface and the macros that it provides to  
manage values (CAMLlocal and CAMLparam), but I would like to be able  
to disable the GC entirely for short periods.

What facilities does OCaml provide to disable/enable the garbage  
collector? I looked at the GC module and didn't see anything obvious.  
It's fine if it is something at the C level rather than the OCaml level.

Thanks for your help!
Raj

Re: [Caml-list] Disabling the OCaml garbage collector

[Basile STARYNKEVITCH]

Raj Bandyopadhyay wrote:
> Dear all
> 
> I am writing a foreign function interface between OCaml and another 
> language (Python) which works via C. Since I have some memory issues 
> currently, I would like to be able to disable the OCaml Garbage 
> collector before jumping into C/Python, and re-enable it when I return 
> to OCaml.


I am not convinced it is possible or makes sense. More precisely, your 
foreign -coded in C, called by Ocaml code- function probably allocates 
some result, and this allocation can trigger the Ocaml garbage collector

In addition, AFAIK, the Ocaml minor garbage collector is only running 
when called,
either implicitly from some allocation routines, or explicitly.

So if you don't allocate things either directly or indirectly (e.g. by 
calling some callback coded in Ocaml) the Ocaml GC is not running.

> 
> I am using the OCaml/C interface and the macros that it provides to 
> manage values (CAMLlocal and CAMLparam), but I would like to be able to 
> disable the GC entirely for short periods.


You definitely may need to hack the C code inside byterun/minor_gc.c 
(for example), but I don't understand what you want exactly.

However, I do know that mixing GCs, like Ocaml & Python, is a nightmare.
Did you consider having two separate Unix processes, one for the Ocaml 
code and one for the Python code, and having them communicate thru 
standard stuff like pipes, memory mapped files, ...


-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: [Caml-list] Python and Caml (was: Disabling the OCaml garbage collector)

[Thomas Fischbacher]

Raj Bandyopadhyay wrote:

> I am writing a foreign function interface between OCaml and another  
> language (Python) which works via C. 

Acknowledging that there is a number of people interested in interfacing
python and caml, I just put up a website from which our bugfixed and
extended variant of Art Yerkes' "pycaml" module can be downloaded:

http://nmag.soton.ac.uk/tf/pycaml.html

(Note ad Debian developers: this fixes some major memory management bugs
that can cause crashes in the original pycaml module which is in Debian,
so, ideally, our variant should eventually supersede the code that is at
present in that Debian package. Interface-wise, our module provides more
than the original one, but I think I also had to remove two or three
very obscure functions.)

-- 
best regards,
Thomas Fischbacher
tf@functionality.de

Re: [Caml-list] Disabling the OCaml garbage collector

[Raj]

The issue for me is that I need to be able to modify mutable objects in 
OCaml (eg. array-modification in-place) from both OCaml and C/Python. 
However, the OCaml GC moves things around while the execution is in 
C/Python and this crashes my program.

The Python GC is very simple, based on reference counts, hence (I think) 
it suffices just to make sure that any object has a refcount > 0 to keep 
the Python GC off.

Well, I decided to go ahead and hack some C code to build a custom OCaml 
compiler. These are the steps I followed:

1) Add the following declarations to stdlib/gc.ml and stdlib/gc.mli

external disable: unit -> unit = "caml_gc_disable"
external enable: unit -> unit = "caml_gc_enable"

2) Modify byterun/gc_ctrl.c and implement the C functions. For example,

int gc_enabled = 1; //global variable

CAMLprim value caml_gc_enable(){
        CAMLparam0();
        gc_enabled = 1;
        CAMLreturn(Val_unit);
}

3) Modify the "main loop" in gc_ctrl.c as

CAMLprim value caml_gc_compaction(value v)
{
if(gc_enabled) {
  Assert (v == Val_unit);
  caml_empty_minor_heap ();
  caml_finish_major_cycle ();
  caml_finish_major_cycle ();
  caml_compact_heap ();
  caml_final_do_calls ();
}
  return Val_unit;
}


However, when I compile this code with 'make world' I get the following 
error while linking:

boot/ocamlrun boot/ocamlc -nostdlib -I boot  -linkall -o ocaml.tmp 
toplevel/toplevellib.cma toplevel/topstart.cmo
Error while linking boot/stdlib.cma(Gc):
The external function `caml_gc_enable' is not available
make: *** [ocaml] Error 2


Any ideas what I am missing?

Thanks!
Raj




>
> You definitely may need to hack the C code inside byterun/minor_gc.c 
> (for example), but I don't understand what you want exactly.
>
> However, I do know that mixing GCs, like Ocaml & Python, is a nightmare.
> Did you consider having two separate Unix processes, one for the Ocaml 
> code and one for the Python code, and having them communicate thru 
> standard stuff like pipes, memory mapped files, ...
>
>

Re: [Caml-list] Disabling the OCaml garbage collector

[Alain Frisch]

Raj wrote:
> The issue for me is that I need to be able to modify mutable objects in 
> OCaml (eg. array-modification in-place) from both OCaml and C/Python. 
> However, the OCaml GC moves things around while the execution is in 
> C/Python and this crashes my program.

Are you sure?

The OCaml GC is triggered only when Caml memory blocks are allocated 
(from Caml or C code) or when you call it explicitly.

-- Alain

Re: [Caml-list] Disabling the OCaml garbage collector

[Thomas Fischbacher]

Raj,

> The issue for me is that I need to be able to modify mutable objects in 
> OCaml (eg. array-modification in-place) from both OCaml and C/Python. 
> However, the OCaml GC moves things around while the execution is in 
> C/Python and this crashes my program.

You strongly should consider malloc()ing data and providing caml
functions to read and write entries from a C array.

In case the data chunk you want to modify from both caml and C holds
non-integer ml values, things become a bit more complicated. Then, you
have to register_global_root() these extra pointers into the ML heap.
Unregistering of global roots then has to be done in the finalizer of
the ML container value that retains a C pointer to malloc()d space.
There, you free() this as well.

Beware the dreaded global root reference loop though!

-- 
best regards,
Thomas Fischbacher
tf@functionality.de

Re: [Caml-list] Disabling the OCaml garbage collector

[Thomas Fischbacher]

Alain Frisch wrote:

>> The issue for me is that I need to be able to modify mutable objects 
>> in OCaml (eg. array-modification in-place) from both OCaml and 
>> C/Python. However, the OCaml GC moves things around while the 
>> execution is in C/Python and this crashes my program.
> 
> Are you sure?
> 
> The OCaml GC is triggered only when Caml memory blocks are allocated 
> (from Caml or C code) or when you call it explicitly.

Most likely, he calls back into another OCaml function from Python which
does some memory allocation... In real applications, this situation is
practically impossible to avoid.

-- 
best regards,
Thomas Fischbacher
tf@functionality.de

Re: [Caml-list] Disabling the OCaml garbage collector

[Xavier Leroy]

> Well, I decided to go ahead and hack some C code to build a custom OCaml
> compiler. These are the steps I followed: [...]
> However, when I compile this code with 'make world' I get the following
> error while linking:

That's a classic bootstrapping issue, and "make bootstrap; make all"
might solve it.  But you are on the wrong tracks anyway.

What your C code does is disable the compactor, which runs from time
to time to eliminate fragmentation in the major heap by moving objects
around.  You can disable it by using Gc.set with the "max_overhead"
field set to a suitably high value, as documented in the Gc module.
There is no need to hack the run-time system.

However, what you will never be able to disable is the minor
collector, which moves blocks from the minor heap to the major heap.
The reason it cannot be disabled is that the minor heap is of fixed
size, so if it gets full and the minor GC isn't executed to empty it,
your program cannot proceed.

You see, the Caml garbage collector is like a god from ancient
mythology: mighty, but very irritable.  If you mess with it, it'll
make you suffer in surprising ways.

The solution to your problem is to play nice with the Caml GC: on the
C/Python side, handle Caml "values" through an indirection via a
malloc-ed block, and register the Caml value contained within this
block with the Caml GC using register_global_roots(), as Thomas
Fischbacher outlined.  Members of this list can probably point you to
some existing C/Caml bindings that does just this and that you could
use as inspiration.

Hope this helps,

- Xavier Leroy

Re: [Caml-list] Disabling the OCaml garbage collector

[Raj]

You are absolutely right, Thomas. Internally, my program goes back and 
forth between Python and OCaml via C, and there can be chains of calls 
e.g. OCaml -> Python -> OCaml ->... and so on. If I'm passing a 
reference to a mutable object (e.g. an Array) around these call-chains, 
I need to make sure the GC doesn't move it around unexpectedly.

Thank you for the suggestions, folks. I'll follow them up and see how 
they work.

Regards
Raj



Thomas Fischbacher wrote:
>
> Most likely, he calls back into another OCaml function from Python which
> does some memory allocation... In real applications, this situation is
> practically impossible to avoid.
>

Re: [Caml-list] Disabling the OCaml garbage collector

[Raj]

Thank you for the advice, Xavier.

I would highly appreciate any pointers to existing code that does the 
'register_global_roots()' thing.  Examples would definitely help me code 
it correctly.

Regards
Raj


Xavier Leroy wrote:
>
>
> The solution to your problem is to play nice with the Caml GC: on the
> C/Python side, handle Caml "values" through an indirection via a
> malloc-ed block, and register the Caml value contained within this
> block with the Caml GC using register_global_roots(), as Thomas
> Fischbacher outlined.  Members of this list can probably point you to
> some existing C/Caml bindings that does just this and that you could
> use as inspiration.
>
> Hope this helps,
>
> - Xavier Leroy
>   

Re: [Caml-list] Disabling the OCaml garbage collector

[Thomas Fischbacher]

Raj,

> I would highly appreciate any pointers to existing code that does the 
> 'register_global_roots()' thing.  Examples would definitely help me code 
> it correctly.

I would suggest you download the nsim sources from:

http://nmag.soton.ac.uk/nmag/current/download/nmag-0.1-core.tar.gz

and study the nsim/mpi_petsc/petsc.ml and petsc_stubs.c files.

(Actually, the code quality of these functions varies a bit.
Most of the module is reasonably good style, but not everything.)

Basically, this is an OCaml interface to the PETSc "parallel sparse
linear algebra" library. (It also assimilated Xavier's old MPI
interface for OCaml. Had to do that, sorry.)

Of particular interest should be the function
with_petsc_vector_as_bigarray, which will take an opaque petsc vector
and a function that takes a double-float bigarray and calls that
function from C in such a way that it can address the Petsc vector as if
it were a bigarray.

Also, you might be interested in the "sundials" module, which interfaces
the CVODE time integrator from libsundials to OCaml.

-- 
best regards,
Thomas Fischbacher
tf@functionality.de

Re: [Caml-list] Disabling the OCaml garbage collector

[Frédéric van der Plancke]

Raj wrote:
> The Python GC is very simple, based on reference counts, hence (I 
> think) it suffices just to make sure that any object has a refcount > 
> 0 to keep the Python GC off.
No, besides doing basic refcounting, the Python GC is also able to 
collect cycles despite of refcount > 0.
I don't know the details you better ask the Python list.

Frédéric

Re: [Caml-list] Disabling the OCaml garbage collector

[Raj Bandyopadhyay]

Dear all,

As a followup to this discussion, I have been trying to understand  
the OCaml/C interface better. Here's a very small program that I've  
been trying to get to work. It's a mutually recursive function, one  
part in C and the other in OCaml.

(* (* Implement this in C *)
let factC g n = if n=0 then [] else ("C",n)::(g (n-1));;
*)

external factC: (int ->(string * int) list) -> int -> ((string * int)  
list)= "caml_factC"
let factO g n = if n=0 then [] else ("OCaml",n)::(g (n-1));;
let rec fact n = factO (factC fact) n;;

fact 12000;;


The C function corresponding to factC is quite short, however the  
program crashes for values of n > about 11,000. I have tried  
inserting the code to register global roots (currently commented  
out), however, that makes no difference to the point of crash. I was  
just wondering if there is some really obvious step that I am  
missing, or using the wrong allocation function in Caml or something  
like that. Any suggestions would be welcome. I apologize in advice  
for inflicting code on you all, but I am out of ideas right now :(

/*
let factC g n =
         if n=0 then
                 []
         else
                 ("C",n)::(g (n-1))
*/
value caml_factC(value g,value n){
         CAMLparam2(g,n);
         CAMLlocal3(e,l,new_l);

         //C value for n
         int locn = Int_val(n);

         //2 cases
         if (locn <= 0){
                 CAMLreturn(Val_int(0)); //empty list
         }
         else {
                 e = alloc_tuple(2);  //allocate a new list element  
("C",n)
                 Store_field(e,0,copy_string("C"));
                 Store_field(e,1,n);

                 //callback the closure g
                 if(Tag_val(g) == Closure_tag) {
			//PROGRAM CRASHES HERE for large n and never returns from callback
                         l = callback(g,Val_int(locn-1));
                 } else {
                         exit(1);
                 }
                 //cons this tuple to the list obtained by callback
                 new_l = alloc(2,0); //structured list value, tag is 0
                 Store_field(new_l,0,e);
                 Store_field(new_l,1,l);

                 //now we should register e,l and new_l with the GC
		//The program crashes at the same point (n>11000) regdless of  
whether the following
		//code is commented out or not.
                 /*
                 Vstore *roots = (Vstore *)caml_stat_alloc(sizeof 
(Vstore));
                 roots->v1 = e;
                 roots->v2 = l;
                 roots->v3 = new_l;
                 caml_register_global_root(&roots->v1);
                 caml_register_global_root(&roots->v2);
                 caml_register_global_root(&roots->v3);
                 CAMLreturn(roots->v3); //newly constructed list
                 */


                 CAMLreturn(new_l); //newly constructed list
         }
}


Thanks!
Raj

Re: [Caml-list] Disabling the OCaml garbage collector

[Jon Harrop]

On Tuesday 04 December 2007 20:14, Raj Bandyopadhyay wrote:
> Dear all,
>
> As a followup to this discussion, I have been trying to understand
> the OCaml/C interface better. Here's a very small program that I've
> been trying to get to work. It's a mutually recursive function, one
> part in C and the other in OCaml.
>
> (* (* Implement this in C *)
> let factC g n = if n=0 then [] else ("C",n)::(g (n-1));;
> *)
>
> external factC: (int ->(string * int) list) -> int -> ((string * int)
> list)= "caml_factC"
> let factO g n = if n=0 then [] else ("OCaml",n)::(g (n-1));;

Could it just be segfaulting because you've overflowed the stack? Try tail 
recursive functions, e.g. CPS.

I would love to see this working because I never managed to get this kind of 
stuff to work before.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Python and Caml

[Stéphane Glondu]

Le 27/11/2007 18:05, Thomas Fischbacher a écrit :
> http://nmag.soton.ac.uk/tf/pycaml.html

This link is broken.

> (Note ad Debian developers: this fixes some major memory management bugs
> that can cause crashes in the original pycaml module which is in Debian,
> so, ideally, our variant should eventually supersede the code that is at
> present in that Debian package. Interface-wise, our module provides more
> than the original one, but I think I also had to remove two or three
> very obscure functions.)

If you still think that, you should file a bugreport [1] against the 
pycaml package... or at least notify the Debian OCaml Maintainers list 
(in CC)...

Given the patches applied to the Debian packages that merely adapt it to 
newer Python versions, the absence of new upstream release for 6 years 
now, the absence of Art Yerkes in this thread (by the way, I put him in 
CC of this mail), I consider this project "dead" upstream. It would be 
nice if someone could take over upstream maintenance, and at least 
incorporate our patches [2] (or any other variant) for wider distribution.

As far as Debian is concerned, only coccinelle [3] depends on pycaml.

[1] http://www.debian.org/Bugs/Reporting
[2] http://patch-tracker.debian.org/package/pycaml
[3] http://coccinelle.lip6.fr/


Cheers,

-- 
Stéphane