[Caml-list] GC and caml_oldify_local_roots taking too much time

[Caml-list] GC and caml_oldify_local_roots taking too much time

[Christophe Raffalli]

I have a program (doing some shape optimisation) that runs slowler and 
slower. I did some profiling using grpof (and -pg option) and get the 
following result:

- the memory usage does not grow (stays bounded and reach very fast its 
maximum)
- The number of GC cycles and compactions per time unit stays constant 
(actually there is quite a lot of compaction (very approximatively 1 per 
minute).
- The function caml_oldify_local_roots is bellow 2% on a 5 minutes run 
but arround 50% on a 30 minutes run and prbably much more on a ten hour 
runs, but did not have the time to let the program run 10 hours with -pg 
(it will take 10 hours :-).
- All other GC functions are quite low in the profile.

What could cause the increase of time spent in caml_oldify_local_roots ?
And what is this function in the first place ?

Thanks in advance for any hint.

-- 
Christophe Raffalli
Université de Savoie
Batiment Le Chablais, bureau 21
73376 Le Bourget-du-Lac Cedex

tél: (33) 4 79 75 81 03
fax: (33) 4 79 75 87 42
mail: Christophe.Raffalli@univ-savoie.fr
www: http://www.lama.univ-savoie.fr/~RAFFALLI
---------------------------------------------
IMPORTANT: this mail is signed using PGP/MIME
At least Enigmail/Mozilla, mutt or evolution
can check this signature
---------------------------------------------

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] GC and caml_oldify_local_roots taking too much time

[Damien Doligez]

On Jul 19, 2004, at 19:59, Christophe Raffalli wrote:

> What could cause the increase of time spent in caml_oldify_local_roots 
> ?
> And what is this function in the first place ?

It is the function that finds the roots of the memory graph:

1. in the stack
2. in the global C roots
3. among finalised values
4. in the stacks of other threads, if any

I guess your program does one or more of the following:

1. allocate lots of stack space (deep recursion)
2. declare lots of C roots (with caml_register_global_root)
3. launch lots of threads that allocate big stacks

I don't think finalised values can be the problem, because this
particular set of roots is emptied after each minor collection.

It should be possible to do something better for case (2), but
if you declare arbitrary amounts of C roots, you will still pay
the price at each major collection (instead of each minor
collection as it is now).

-- Damien

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] GC and caml_oldify_local_roots taking too much time

[Christophe Raffalli]

Damien Doligez wrote:
> On Jul 19, 2004, at 19:59, Christophe Raffalli wrote:
> 
>> What could cause the increase of time spent in caml_oldify_local_roots ?
>> And what is this function in the first place ?
> 
> 
> It is the function that finds the roots of the memory graph:
> 
> 1. in the stack
> 2. in the global C roots
> 3. among finalised values
> 4. in the stacks of other threads, if any
> 
> I guess your program does one or more of the following:
> 
> 1. allocate lots of stack space (deep recursion)

That's not the case, I am in a while loop so recursion depth is bounded

> 2. declare lots of C roots (with caml_register_global_root)

That may be a bug in the lablGL code ... I had a short look but found 
nothing.

> 3. launch lots of threads that allocate big stacks

I launch only two threads (native posix threads)


> I don't think finalised values can be the problem, because this
> particular set of roots is emptied after each minor collection.

I removed call to finalise and saw no change. Anyway I have always less 
than 10 not collected object with finaliser.

> It should be possible to do something better for case (2), but
> if you declare arbitrary amounts of C roots, you will still pay
> the price at each major collection (instead of each minor
> collection as it is now).

I will look better at lablGL/Glut code now.

Thanks


> -- Damien
> 
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: 
> http://caml.inria.fr
> Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: 
> http://caml.inria.fr/FAQ/
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

-- 
Christophe Raffalli
Université de Savoie
Batiment Le Chablais, bureau 21
73376 Le Bourget-du-Lac Cedex

tél: (33) 4 79 75 81 03
fax: (33) 4 79 75 87 42
mail: Christophe.Raffalli@univ-savoie.fr
www: http://www.lama.univ-savoie.fr/~RAFFALLI
---------------------------------------------
IMPORTANT: this mail is signed using PGP/MIME
At least Enigmail/Mozilla, mutt or evolution
can check this signature
---------------------------------------------

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] GC and caml_oldify_local_roots taking too much time

[Christophe Raffalli]

I identified the problem:

Glut.timerFunc does register the callback to each call and therefore 
allocate one glocal C root.

However, the callback in never unregistred (I do not know how to do 
that) and moreover, I register always the same function and therefore, 
it is always the same Global C root which is registered !

Is there a function to unregister a callback ?

-- 
Christophe Raffalli
Université de Savoie
Batiment Le Chablais, bureau 21
73376 Le Bourget-du-Lac Cedex

tél: (33) 4 79 75 81 03
fax: (33) 4 79 75 87 42
mail: Christophe.Raffalli@univ-savoie.fr
www: http://www.lama.univ-savoie.fr/~RAFFALLI
---------------------------------------------
IMPORTANT: this mail is signed using PGP/MIME
At least Enigmail/Mozilla, mutt or evolution
can check this signature
---------------------------------------------

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] GC and caml_oldify_local_roots taking too much time

[Richard Jones]

On Mon, Jul 19, 2004 at 04:17:12PM -0400, Christophe Raffalli wrote:
> Is there a function to unregister a callback ?

Indeed there is: [caml_]remove_global_root.

Of course, when it is appropriate and correct to call this function is
another matter entirely ...

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
http://www.YouUnlimited.co.uk/ - management courses

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] GC and caml_oldify_local_roots taking too much time

[Richard Jones]

On Mon, Jul 19, 2004 at 03:28:34PM +0100, Richard Jones wrote:
> On Mon, Jul 19, 2004 at 04:17:12PM -0400, Christophe Raffalli wrote:
> > Is there a function to unregister a callback ?
> 
> Indeed there is: [caml_]remove_global_root.

Duh, sorry totally misread the description of your problem.

Callbacks, not global roots ...

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
http://www.YouUnlimited.co.uk/ - management courses

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] GC and caml_oldify_local_roots taking too much time

[Olivier Andrieu]

 Christophe Raffalli [Mon, 19 Jul 2004]:
 > 
 > I identified the problem:
 > 
 > Glut.timerFunc does register the callback to each call and therefore 
 > allocate one glocal C root.
 > 
 > However, the callback in never unregistred (I do not know how to do 
 > that) and moreover, I register always the same function and therefore, 
 > it is always the same Global C root which is registered !
 > 
 > Is there a function to unregister a callback ?

No, but you needn't register the callback itself, you can register a
reference to it.

,----
| let callback = ref None
| external setup_c_callback : string -> unit = "setup_c_callback"
| let _ = 
|   let id = "my_caml_callback" in
|   Callback.register id callback ;
|   setup_c_callback id
| 
| let set_callback f = callback := Some f
| let unset_callback () = callback := None
`----

,----
| static value *my_caml_callback;
| 
| static void my_c_callback()
| {
|   if (my_caml_callback != NULL) {
|     value contents = Field(*my_caml_callback, 0);
|     if (Is_block(contents)) {
|       value closure = Field(contents, 0);
|       callback_exn(closure, Val_unit);
|     }
|   }
| }
| 
| CAMLprim value setup_c_callback(value id)
| {
|   my_caml_callback = caml_named_value(String_val(id));
|   setup_callback(&my_c_callback);
| }
`----

That way, Callback.register is only called once.

-- 
   Olivier

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] GC and caml_oldify_local_roots taking too much time

[Christophe Raffalli]

It seems it is a problem (a bug ?) with caml_register_named_value in 
asmrun/callback.c:

when a value v is registered under a name x, v is registered as a global 
root.

when another value w is registered under the same name, the global root 
v is not released.

Moreover, it seems stranges not to have a function to unregister a named 
value: Glut.timerFunc register a callback that will be used only once, 
so it could unregister the callback in the callback itself.


-- 
Christophe Raffalli
Université de Savoie
Batiment Le Chablais, bureau 21
73376 Le Bourget-du-Lac Cedex

tél: (33) 4 79 75 81 03
fax: (33) 4 79 75 87 42
mail: Christophe.Raffalli@univ-savoie.fr
www: http://www.lama.univ-savoie.fr/~RAFFALLI
---------------------------------------------
IMPORTANT: this mail is signed using PGP/MIME
At least Enigmail/Mozilla, mutt or evolution
can check this signature
---------------------------------------------

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

[Caml-list] Calback (was caml_oldify_local_roots taking too much time)

[Christophe Raffalli]

Thanks for the help on the list. I decided to change Glut.timerFunc for 
the following code, passing the Caml function and its argument using a C 
int (transmitted to the times) which is in fact a pointer to a Caml 
pair. This pointer is registered as a global root and removed by the 
callback itself. I give the code bellow if someone wants to write 
something similar.

A small question: why do people use callback trough a name and a 
hashtable instead of passing the callback inside C variables (global or 
local) ? It could save a lot of time ?

For instance Glut.idleFunc will do a search in the hashtable of caml 
names at each callback and if the callback was transmitted  through a C 
global variable there vould only be one indirection ?

I am about to change all the callback mechnism used in Glut. So if 
anyone knows a good reason to use names and hashtables ?

----- 8< ----- here is the code

----- First the caml code:


(* timerFunc is non-window-dependent *)
external _glutTimerFunc : int->((value:'a->'b) * 'a)->unit = 
"ml_glutTimerFunc"
let timerFunc ~ms ~cb ~value:v : (unit) =
     _glutTimerFunc ms (cb,v);; (* register the callback with GLUT *)

----- Then the C code:


--
// for timer we can not use Register.callback because it grows 
infinitely the
// number of global root

static void glutTimerFunc_cb(int fun_arg)
{
   // fun_arg is a pointer on a caml pair hidden in an integer.
   // Moreover we remove the global root because each callback is used once
   // by the timer.
   value *v = (value) fun_arg;
   leave_blocking_section ();
   value fun = Field(*v, 0);
   value arg = Field(*v, 1);
   caml_remove_global_root(v);
   free(v);
   callback (fun, arg);
   enter_blocking_section ();
}

CAMLprim value ml_glutTimerFunc(value millis_val, value fun_arg) // set 
Timer callback
{
   // fun_arg is a caml pair with a function and its argument we 
register a root on this pair and set the callback.
     unsigned int millis;
     value *v = (value*) malloc(sizeof(value));
     *v = fun_arg;
     caml_register_global_root(v);
     millis = Int_val(millis_val);
     glutTimerFunc(millis, &glutTimerFunc_cb, (int) v); // register with 
GLUT
     return Val_unit;
}

--
Christophe Raffalli
Université de Savoie
Batiment Le Chablais, bureau 21
73376 Le Bourget-du-Lac Cedex

tél: (33) 4 79 75 81 03
fax: (33) 4 79 75 87 42
mail: Christophe.Raffalli@univ-savoie.fr
www: http://www.lama.univ-savoie.fr/~RAFFALLI
---------------------------------------------
IMPORTANT: this mail is signed using PGP/MIME
At least Enigmail/Mozilla, mutt or evolution
can check this signature
---------------------------------------------

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Calback

[Olivier Andrieu]

 Christophe Raffalli [Tue, 20 Jul 2004]:
 > Thanks for the help on the list. I decided to change Glut.timerFunc
 > for the following code, passing the Caml function and its argument
 > using a C int (transmitted to the times) which is in fact a pointer
 > to a Caml pair. This pointer is registered as a global root and
 > removed by the callback itself. I give the code bellow if someone
 > wants to write something similar.
 > 
 > A small question: why do people use callback trough a name and a
 > hashtable instead of passing the callback inside C variables
 > (global or local) ? It could save a lot of time ?

(caml_named_value uses a linked list, not a hashtable).

 > ----- First the caml code:
 > 
 > 
 > (* timerFunc is non-window-dependent *)
 > external _glutTimerFunc : int->((value:'a->'b) * 'a)->unit = 
 > "ml_glutTimerFunc"
 > let timerFunc ~ms ~cb ~value:v : (unit) =
 >      _glutTimerFunc ms (cb,v);; (* register the callback with GLUT *)

It's just easier not to bother with the argument and have the callback
be of type `unit -> unit' (or `unit -> 'a'), since you can include any
argument you like in the closure.

 > // for timer we can not use Register.callback because it grows 
 > infinitely the
 > // number of global root
 > 
 > static void glutTimerFunc_cb(int fun_arg)
 > {
 >    // fun_arg is a pointer on a caml pair hidden in an integer.
 >    // Moreover we remove the global root because each callback is used once
 >    // by the timer.
 >    value *v = (value) fun_arg;

I'm afraid this will break on plaforms where `sizeof (int) != sizeof (long)'.

-- 
   Olivier

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Calback

[Christophe Raffalli]

Olivier Andrieu wrote:

 >  Christophe Raffalli [Tue, 20 Jul 2004]:
 >  > Thanks for the help on the list. I decided to change Glut.timerFunc
 >  > for the following code, passing the Caml function and its argument
 >  > using a C int (transmitted to the times) which is in fact a pointer
 >  > to a Caml pair. This pointer is registered as a global root and
 >  > removed by the callback itself. I give the code bellow if someone
 >  > wants to write something similar.
 >  >  > A small question: why do people use callback trough a name and a
 >  > hashtable instead of passing the callback inside C variables
 >  > (global or local) ? It could save a lot of time ?
 >
 > (caml_named_value uses a linked list, not a hashtable).
 >

This is even worth !

 >  > ----- First the caml code:
 >  >  >  > (* timerFunc is non-window-dependent *)
 >  > external _glutTimerFunc : int->((value:'a->'b) * 'a)->unit =  > 
"ml_glutTimerFunc"
 >  > let timerFunc ~ms ~cb ~value:v : (unit) =
 >  >      _glutTimerFunc ms (cb,v);; (* register the callback with GLUT *)
 >
 > It's just easier not to bother with the argument and have the callback
 > be of type `unit -> unit' (or `unit -> 'a'), since you can include any
 > argument you like in the closure.


This is to follow the same api than GLUT (generalized a bit because in 
glut the argument is always int.

 >  > // for timer we can not use Register.callback because it grows  > 
infinitely the
 >  > // number of global root
 >  >  > static void glutTimerFunc_cb(int fun_arg)
 >  > {
 >  >    // fun_arg is a pointer on a caml pair hidden in an integer.
 >  >    // Moreover we remove the global root because each callback is 
used once
 >  >    // by the timer.
 >  >    value *v = (value) fun_arg;
 >
 > I'm afraid this will break on plaforms where `sizeof (int) != sizeof 
(long)'.
 >

It was just a first try. You are right. Do you have such a platform ? 
Could check in that case the prototype of glutTimerFunc in GL/glut.h ?

If it is still int, I could do a test in the C code and go through a C 
array if (sizeof(int) != sizeof(void*))

-- 
Christophe Raffalli
Université de Savoie
Batiment Le Chablais, bureau 21
73376 Le Bourget-du-Lac Cedex

tél: (33) 4 79 75 81 03
fax: (33) 4 79 75 87 42
mail: Christophe.Raffalli@univ-savoie.fr
www: http://www.lama.univ-savoie.fr/~RAFFALLI
---------------------------------------------
IMPORTANT: this mail is signed using PGP/MIME
At least Enigmail/Mozilla, mutt or evolution
can check this signature
---------------------------------------------

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Calback

[Jere Sanisalo]

> > I'm afraid this will break on plaforms where `sizeof (int) != sizeof 
> > (long)'.
>If it is still int, I could do a test in the C code and go through a C 
>array if (sizeof(int) != sizeof(void*))

Why not do something along the lines of:
#define Meta_Assert(expr) typedef int __META_ASSERT__[(expr)? 1 : -1];

and then simply asserting it runtime? Like:
  Meta_Assert( sizeof(int) = sizeof(void*) );
Of course if you DO want it to work with different sizes, a different
approach is wanted. But mostly thses kind of things are nice, as they check
at compile time that certain truths about the runtime are true.

-- 
Jere Sanisalo [xm@xmunkki.org] - http://www.xmunkki.org/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Calback

[Christophe Raffalli]

>  > static void glutTimerFunc_cb(int fun_arg)
>  > {
>  >    // fun_arg is a pointer on a caml pair hidden in an integer.
>  >    // Moreover we remove the global root because each callback is used once
>  >    // by the timer.
>  >    value *v = (value) fun_arg;
> 
> I'm afraid this will break on plaforms where `sizeof (int) != sizeof (long)'.
> 

Anyway, for GlSurf, I am just using this (which is faster) and for most 
case I think this a the right code (except that set_timer should raise 
an exception it init_timer was not called previously).

This code allow to change the callback associated with timers but does 
not allow to have two timers with different callbacks. But the later is 
only usefull (in GLUT) for timers, all other callback could use code 
similar to this.

-- ML -------------------------------

(* init_timer f set the function to be called by all timer *)
external init_timer : (value:int -> unit) -> unit = "myinit_glutTimerFunc"

(* set_timer ms val : set a timer in ms milliseconds calling the 
function registered by init_timer with val as argument *)
external set_timer : int -> int -> unit = "myml_glutTimerFunc"

-- C --------------------------------


static value* caml_glutTimerFunc_cb = NULL;

static void myglutTimerFunc_cb(int val)
{
   leave_blocking_section ();
   callback (*caml_glutTimerFunc_cb, Val_int(val));
   enter_blocking_section ();
}

CAMLprim value myml_glutTimerFunc(value millis_val, value v) // set 
Timer callback
{
     glutTimerFunc(Int_val(millis_val), &myglutTimerFunc_cb, Int_val(v));
     return Val_unit;
}

CAMLprim value myinit_glutTimerFunc(value v) {
   if (caml_glutTimerFunc_cb)   {
     caml_remove_global_root(caml_glutTimerFunc_cb);
   } else {
     caml_glutTimerFunc_cb = (value*) malloc(sizeof(value));
   }
   *caml_glutTimerFunc_cb = v;
   caml_register_global_root(caml_glutTimerFunc_cb);
   return Val_unit;
}

-- 
Christophe Raffalli
Université de Savoie
Batiment Le Chablais, bureau 21
73376 Le Bourget-du-Lac Cedex

tél: (33) 4 79 75 81 03
fax: (33) 4 79 75 87 42
mail: Christophe.Raffalli@univ-savoie.fr
www: http://www.lama.univ-savoie.fr/~RAFFALLI
---------------------------------------------
IMPORTANT: this mail is signed using PGP/MIME
At least Enigmail/Mozilla, mutt or evolution
can check this signature
---------------------------------------------

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners