[Caml-list] Accuracy of Gc.stat ()

[Caml-list] Accuracy of Gc.stat ()

[Daniel Bünzli]

Since my first attempt [1] didn't really get through, I try to 
reformulate my post.

To compare different implementations of a function I would like to 
profile its execution in time and heap memory usage (at least orders of 
magnitude). To do so, I use the code given at the end of this email. 
Basically I do a full major collection, get the gc statistics via the 
Gc.stat () function, run my function, call again Gc.stat (), and 
substract the former statistics to the latters.

My questions are :

1) What is the accuracy of these results ?

E.g. I read in the documentation of the Gc module that the field 
minor_words is only an approximation in programs compiled to native 
code. Is it also true for the other fields ? Would the figure 
minor+major-promoted be accurate ? How much can I trust the figures I 
get ?

2) When I start profiling should I prefer a Gc.compact to a 
Gc.full_major ?

3) Is it possible to know at runtime whether we are running native code 
or interpreted bytecode ?

Regarding time profiling, a binding in the Unix module to the 
getrusage() function would definitvely be nice.

Thanks for your answers,

Daniel

[1] <http://caml.inria.fr/archives/200311/msg00217.html>

-- profile.ml --

type t = { minor_bytes : float;
	   promoted_bytes : float;
	   major_bytes : float;
	   allocated_bytes : float;
	
	   minor_collections : float;
	   major_collections : float;
	
	   user_time : float;
	   system_time : float }

(* Bytes per words *)
let bpw = float_of_int (Sys.word_size / 8)

(* Heap allocation overhead due to profiling *)
let heap_overhead =
   let s = Gc.stat() in
   ignore(Unix.times());
   ignore(Unix.times());
   let s' = Gc.stat() in
   ((s'.Gc.minor_words +. s'.Gc.major_words -. s'.Gc.promoted_words) -.
   (s.Gc.minor_words +. s.Gc.major_words -. s.Gc.promoted_words)) *. bpw

let execution_stats_n n f a =
   Gc.full_major ();
   let s = Gc.stat () in
   let t = Unix.times () in
   for i = 1 to n do
     ignore(f a)
   done;
   let t' = Unix.times () in
   let s' = Gc.stat () in
   let mi, pro, ma =
     ((s'.Gc.minor_words -. s.Gc.minor_words) *. bpw) -. heap_overhead,
     (s'.Gc.promoted_words -. s.Gc.promoted_words) *. bpw,
     (s'.Gc.major_words -. s.Gc.major_words) *. bpw in
   let n' = float_of_int n in
   { minor_bytes = mi /. n';
     promoted_bytes = pro /. n';
     major_bytes = ma /. n';
     allocated_bytes = (mi +. ma -. pro) /. n';

     minor_collections =
       (float_of_int (s'.Gc.minor_collections - s.Gc.minor_collections)) 
/. n';
     major_collections =
       (float_of_int (s'.Gc.major_collections - s.Gc.major_collections)) 
/. n';

     user_time = (t'.Unix.tms_utime -. t.Unix.tms_utime) /. n';
     system_time = (t'.Unix.tms_stime -. t.Unix.tms_stime) /. n'
   }

let execution_stats f a = execution_stats_n 1000 f a
let execution f a = execution_stats_n 1 f a

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Accuracy of Gc.stat ()

[Damien Doligez]

On Wednesday, November 19, 2003, at 06:52 PM, Daniel Bünzli wrote:

> Since my first attempt [1] didn't really get through, I try to 
> reformulate my post.

Sorry I didn't answer earlier, the traffic on caml-list is getting
quite heavy these days.

> 1) What is the accuracy of these results ?
>
> E.g. I read in the documentation of the Gc module that the field 
> minor_words is only an approximation in programs compiled to native 
> code.

Because of the way we trigger a minor collection in native code, the
counters will overestimate the number of words allocated by a few words
per minor GC.  Note that it is always an overestimation, and the error
is at most 256 words per collection.

>  Is it also true for the other fields ?

No.  The other fields are accurate.

>  Would the figure minor+major-promoted be accurate ?

No.  The error in minor will propagate.

>  How much can I trust the figures I get ?

You can repeat your experiment after changing the size of the
minor heap, that will give you a good idea of the error.  The
bigger the heap, the smaller the error, and it converges to
error=0 for an infinite heap.

> 2) When I start profiling should I prefer a Gc.compact to a 
> Gc.full_major ?

If you're only interested in minor+major-promoted, you don't even
need a Gc.full_major.  A Gc.compact would be if you are timing a
very allocation-intensive program, and even then I doubt it will
make much difference.

> 3) Is it possible to know at runtime whether we are running native 
> code or interpreted bytecode ?

Good question.  I don't know the answer.  Maybe you should file this
as a feature wish in the bug tracking system:
< http://caml.inria.fr/bin/caml-bugs >.

> Regarding time profiling, a binding in the Unix module to the 
> getrusage() function would definitvely be nice.

Another candidate for the bug tracking system.

-- Damien

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Accuracy of Gc.stat ()

[Kim Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1645 bytes --]

Hi,

On Wed, 19 Nov 2003 18:52:10 +0100
Daniel Bünzli <daniel.buenzli@epfl.ch> wrote:

> 
> 3) Is it possible to know at runtime whether we are running native code 
> or interpreted bytecode ?
> 

There is actualy an (ugly) hack which seems to work, but it requires few lines of C codes :

--- prog.ml --

(* An external declaration, first C function is used when compiled to bytecode, 
    second when compiled to native code.
*)

external is_bytecode : unit -> bool = "is_bytecode_bytecode" "is_bytecode_native"

(* And now you can use it... *)
let _ = 
	if is_bytecode () 
	then 
	print_endline "Bytecode !!!"
	else
	print_endline "Native code !!!"

-----------

--- bytecode.c -----

#include <caml/mlvalues.h>

CAMLprim value is_bytecode_bytecode(value unit){
  return Val_true;
}
CAMLprim value is_bytecode_native(value unit){
  return Val_false;
}

-------------

Et voilà !

Linking native caml code with this C code isn't a big deal but, linking with
bytecode make you somehow loose the portability of the later. 
See the corresponding chapter in the Ocaml manual (Chapter 18).

$ gcc -c bytecode.c
$ ocamlc -make-runtime -o myruntime bytecode.o
$ ocamlc -o prog -use-runtime ./myruntime prog.ml
$ ocamlopt -o prog.opt bytecode.o prog.ml
$ ./prog
Bytecode !!!
$ ./prog.opt
Native code !!!

I wonder if there is a cleaner way to do this. Maybe there could be a flag like
 the Sys.interactive one. I'd like to know how "safe" is this code, the behavior 
of the compiler is only described for function with more than 5 arguments in 
the manual.


Cheers.

Kim Nguyen.



[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [Caml-list] Accuracy of Gc.stat ()

[Richard Jones]

[-- Attachment #1: Type: text/plain, Size: 828 bytes --]

On Sat, Nov 22, 2003 at 01:20:54AM +0100, Kim Nguyen wrote:
> external is_bytecode : unit -> bool = "is_bytecode_bytecode" "is_bytecode_native"

Neat trick. This _ought_ to work, but it doesn't for some reason:

	let arg = (true, false) 
	external is_bytecode : bool * bool -> bool = "%field0" "%field1" 
 
	let () = 
	  if is_bytecode arg then 
	    print_endline "Bytecode!" 
	  else 
	    print_endline "Native!" 

Perhaps someone can explain why ...

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://freshmeat.net/users/rwmj
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
"One serious obstacle to the adoption of good programming languages is
the notion that everything has to be sacrificed for speed. In computer
languages as in life, speed kills." -- Mike Vanier

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [Caml-list] Accuracy of Gc.stat ()

[Richard Jones]

[-- Attachment #1: Type: text/plain, Size: 1180 bytes --]

On Sat, Nov 22, 2003 at 11:43:50AM +0000, rich wrote:
> On Sat, Nov 22, 2003 at 01:20:54AM +0100, Kim Nguyen wrote:
> > external is_bytecode : unit -> bool = "is_bytecode_bytecode" "is_bytecode_native"
> 
> Neat trick. This _ought_ to work, but it doesn't for some reason:
> 
> 	let arg = (true, false) 
> 	external is_bytecode : bool * bool -> bool = "%field0" "%field1" 
>  
> 	let () = 
> 	  if is_bytecode arg then 
> 	    print_endline "Bytecode!" 
> 	  else 
> 	    print_endline "Native!" 
> 
> Perhaps someone can explain why ...

OK, closer inspection of the manual[*] reveals why this doesn't work.
Surely OCaml should flag the above code as a bug, because there are
fewer than the magical 6 arguments?

Rich.

[*] http://caml.inria.fr/ocaml/htmlman/manual032.html

-- 
Richard Jones. http://www.annexia.org/ http://freshmeat.net/users/rwmj
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
'There is a joke about American engineers and French engineers. The
American team brings a prototype to the French team. The French team's
response is: "Well, it works fine in practice; but how will it hold up
in theory?"'

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Self-detection of native code execution (Was Re: [Caml-list] Accuracy of Gc.stat ())

[Daniel Bünzli]

Le 22 nov. 03, à 12:49, Richard Jones a écrit :

>> Neat trick. This _ought_ to work, but it doesn't for some reason:
>>
>> 	let arg = (true, false)
>> 	external is_bytecode : bool * bool -> bool = "%field0" "%field1"
>>
>> 	let () =
>> 	  if is_bytecode arg then
>> 	    print_endline "Bytecode!"
>> 	  else
>> 	    print_endline "Native!"
>>
>> Perhaps someone can explain why ...

Nice try, too bad it doesn't work. I would even prefer that one :

let arg = true
external is_bytecode : bool ->  bool = "%identity" "%boolnot"

let () =
   if is_bytecode arg then
     print_endline "Bytecode!"
   else
     print_endline "Native!"


> OK, closer inspection of the manual[*] reveals why this doesn't work.
> Surely OCaml should flag the above code as a bug, because there are
> fewer than the magical 6 arguments?

The strange thing is that the function given by Kim also have fewer 
than 6 arguments.

Daniel

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Accuracy of Gc.stat ()

[Kim Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1723 bytes --]

Hello,

On Sat, 22 Nov 2003 11:49:08 +0000
Richard Jones <rich@annexia.org> wrote:


> > Neat trick. This _ought_ to work, but it doesn't for some reason:
> > 
> > 	let arg = (true, false) 
> > 	external is_bytecode : bool * bool -> bool = "%field0" "%field1" 
> >  
> > 	let () = 
> > 	  if is_bytecode arg then 
> > 	    print_endline "Bytecode!" 
> > 	  else 
> > 	    print_endline "Native!" 
> > 
> > Perhaps someone can explain why ...
> 
> OK, closer inspection of the manual[*] reveals why this doesn't work.
> Surely OCaml should flag the above code as a bug, because there are
> fewer than the magical 6 arguments?
> 
> Rich.

Well, my guess is that '%' primitives pre-defined in Ocaml are not compiled as
function. Some are just a way to bypass the typechecker (like %identity) and others
are directly translated to a sequence of bytecode/native instructions and inlined 
in the code. Thus when you use a "%primitive" in an external only the first declaration
 would by used, and so, the %field1 above would get ignored by the compiler. 
You can see it if you try to mix both kinds of primitive :

external is_bytecode : bool * bool -> bool = "field0_bytecode" "%field1" 

the compilation process fine but at link time, the linker complain about an undefiened
 symbol '$25field1'.
if you use :
external is_bytecode : bool * bool -> bool = "%field0" "field1_native"
then the "field1_native" is ignored %field0 is used in both cases.

Anyhow, I just meant to toy with the compiler and don't think using this 'feature' of
the compiler should be considered safe, as it is not specified in the documentation
(unless we have clearer explanation by an Ocaml guru :-). It could be a nice feature
though.

Regards,

Kim.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Self-detection of native code execution (Was Re: [Caml-list] Accuracy of Gc.stat ())

[Richard Jones]

On Sat, Nov 22, 2003 at 03:20:40PM +0100, Daniel B?nzli wrote:
> 
> Le 22 nov. 03, ? 12:49, Richard Jones a ?crit :
> 
> >>Neat trick. This _ought_ to work, but it doesn't for some reason:
> >>
> >>	let arg = (true, false)
> >>	external is_bytecode : bool * bool -> bool = "%field0" "%field1"
> >>
> >>	let () =
> >>	  if is_bytecode arg then
> >>	    print_endline "Bytecode!"
> >>	  else
> >>	    print_endline "Native!"
> >>
> >>Perhaps someone can explain why ...
> 
> Nice try, too bad it doesn't work. I would even prefer that one :
> 
> let arg = true
> external is_bytecode : bool ->  bool = "%identity" "%boolnot"
> 
> let () =
>   if is_bytecode arg then
>     print_endline "Bytecode!"
>   else
>     print_endline "Native!"
> 
> 
> >OK, closer inspection of the manual[*] reveals why this doesn't work.
> >Surely OCaml should flag the above code as a bug, because there are
> >fewer than the magical 6 arguments?
> 
> The strange thing is that the function given by Kim also have fewer 
> than 6 arguments.

True, but his code doesn't actually use the values of the arguments
passed to the C functions.

What is needed is a primitive function which ignores its arguments and
returns a known value. But I don't think there is such a primitive in
the current compiler. (Not as far as I can see by looking at
bytecomp/translcore.ml anyway).

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://freshmeat.net/users/rwmj
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
"One serious obstacle to the adoption of good programming languages is
the notion that everything has to be sacrificed for speed. In computer
languages as in life, speed kills." -- Mike Vanier

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Accuracy of Gc.stat ()

[Richard Jones]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

On Sat, Nov 22, 2003 at 03:28:16PM +0100, Kim Nguyen wrote:
> Anyhow, I just meant to toy with the compiler and don't think using this 'feature' of
> the compiler should be considered safe, as it is not specified in the documentation
> (unless we have clearer explanation by an Ocaml guru :-). It could be a nice feature
> though.

OK, but it was fun anyway :-)

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://freshmeat.net/users/rwmj
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
NET::FTPSERVER is a full-featured, secure, configurable, database-backed
FTP server written in Perl: http://www.annexia.org/freeware/netftpserver/

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]