Re: [Caml-list] Benchmarking different dispatch types

[Christophe TROESTLER <Christophe.Troestler+ocaml@umh.ac.be>]

[-- Attachment #1: Type: Text/Plain, Size: 2594 bytes --]

>             (warning: too few iterations for a reliable count)
>
> Interesting, but are they meaningful?  The warnings from Benchmark are
> troubling, but I didn't have any immediate ideas on how to get rid of
> them.

The warnings mean what they say.  The running time of the calls (as
opposed to the whole process: calls + repetition loop), is estimated
to be less than 0.1 second and is therefore too imprecise to produce
meaningful results.

On Wed, 17 Jan 2007, "Nathaniel Gray" <n8gray@gmail.com> wrote:
> 
>    let results = latencyN 40000
>       [("function", call_f, ());
>        ("method", call_o, ());
>        ("closure", call_fc, ());
>        ("obj. closure", call_foc, ())]

Knowing the right number of calls to have a meaningful measurment is
not always easy, espcially for fast running code like yours.
Actually, as Edgar Friendly pointed out, in your case, even max_int is
not enough...  In order to get rid of this you should:

- Use the throughputN function which allows you to set the running
  time of the function instead of the number of calls;

- Use the new version Benchmark module which now uses Int64.t to store
  the number of calls.  Moreover, with this new version,...

On Wed, 17 Jan 2007, Edgar Friendly <thelema314@gmail.com> wrote:
> 
>                        Rate
>     function -5.36871e+10/s
               ^
  ...this cannot happen anymore.

You are also advised to set the ~repeat flag to a value of 5-10 in
order to trigger some statistical tests that will put the relative
rates of the tested functions in between brackets if they are not
significantly different.

For your problem, your code (attached) gives the following results (on
a Intel 2Ghz dual core):

                    Rate               method obj. closure    closure   function
      method 156624689+-  738397/s         --         -60%       -68%       -84%
obj. closure 387882394+- 4665352/s       148%           --       -20%       -61%
     closure 486459254+- 1330609/s       211%          25%         --       -51%
    function 984016184+-24771072/s       528%         154%       102%         --

Since the (i,j) entry is rij = (ri / rj - 1) where ri is the rate of
line i and rj the rate of col j, if we want to compare the times ti =
1/ri, one gets tj = (1 + rij) ti.  Thus one has to look at the last
line to have the times relative to the function one:

t(closure)     = (1 + 102%) t(function) ~ 2x
t(obj.closure) = (1 + 154%) t(function) ~ 2.5x
t(method)      = (1 + 528%) t(function) ~ 6.3x

This is quite coherent with Jacques Garrigue results.

Hope this helps,
ChriS

[-- Attachment #2: gray.ml --]
[-- Type: Text/Plain, Size: 779 bytes --]

(* Test method dispatch vs. function dispatch vs. closure dispatch *)

let f x = x + 100
let call_f () = f 1

let o = object
   method f_o x = x + 100
end
let call_o () = o#f_o 1

let f_c () x = x + 100
let f_c' = f_c ()
let call_fc () = f_c' 1

let o_c = object
   method f_oc () x = x + 100
end
let f_oc' = o_c#f_oc ()
let call_foc () = f_oc' 1


open Benchmark

let () =
  let results = latencyN ~repeat:5 1100000000L
(*   let results = throughputN ~repeat:5 1 *)
    [("function", call_f, ());
     ("method", call_o, ());
     ("closure", call_fc, ());
     ("obj. closure", call_foc, ())]
  in
  tabulate results

(* ocamlopt -o gray.com -inline 0 -I benchmark unix.cmxa benchmark.cmxa gray.ml
   ocamlc -o gray.exe -inline 0 -I benchmark unix.cma benchmark.cma gray.ml
*)