From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=AWL autolearn=disabled version=3.1.3 X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from discorde.inria.fr (discorde.inria.fr [192.93.2.38]) by yquem.inria.fr (Postfix) with ESMTP id AD9B5BC69 for ; Fri, 9 Feb 2007 15:22:41 +0100 (CET) Received: from ipmail01.adl2.internode.on.net (ipmail01.adl2.internode.on.net [203.16.214.140]) by discorde.inria.fr (8.13.6/8.13.6) with ESMTP id l19EMd11005403 for ; Fri, 9 Feb 2007 15:22:40 +0100 Received: from ppp27-234.lns1.syd6.internode.on.net (HELO rosella) ([59.167.27.234]) by ipmail01.adl2.internode.on.net with ESMTP; 10 Feb 2007 00:52:37 +1030 X-IronPort-AV: i="4.13,306,1167571800"; d="scan'208"; a="86115160:sNHT23999045" Subject: Re: [Caml-list] Multiplication of matrix in C and OCaml From: skaller To: ls-ocaml-developer-2006@m-e-leypold.de Cc: caml-list@yquem.inria.fr In-Reply-To: References: <45CAF3E2.7020807@univ-paris12.fr> <45CAFF5A.2020607@inria.fr> <45CB3ED0.9040200@univ-paris12.fr> <20070209.115842.106265091.garrigue@math.nagoya-u.ac.jp> Content-Type: text/plain Date: Sat, 10 Feb 2007 01:22:33 +1100 Message-Id: <1171030953.5302.51.camel@rosella.wigram> Mime-Version: 1.0 X-Mailer: Evolution 2.6.1 Content-Transfer-Encoding: 7bit X-j-chkmail-Score: MSGID : 45CC83AF.000 on discorde : j-chkmail score : X : 0/20 1 0.000 -> 1 X-Miltered: at discorde with ID 45CC83AF.000 by Joe's j-chkmail (http://j-chkmail . ensmp . fr)! X-Spam: no; 0.00; ocaml:01 0100,:01 matrices:01 associative:01 ocamlopt:01 paging:98 sourceforge:01 sourceforge:01 wrote:01 imho:01 integer:01 integer:01 caml-list:01 writes:01 arithmetic:01 On Fri, 2007-02-09 at 11:32 +0100, ls-ocaml-developer-2006@m-e-leypold.de wrote: > ls-ocaml-developer-2006@m-e-leypold.de writes: > > > optimizations. I'd feel better if the code is benchmarked in a way > > that the result of the multiplication is output to a file and to > > subtract the constant contribution of that to the run time that the > > time is measured for various problem sizes (number of matrices). > > Correction: ... the constant contribution of writing the result to a > file should be subtracted from the run time by measuring with various > problem sizes. > > Hope that is understandable now. There is no need for that IMHO. The right way to benchmark simple things like matrix operations AND simultaneously validate them is to evaluate known laws of arithmetic, such as the associative law, on various samples. This doesn't require any I/O, nor a 'known correct' program to generate the comparison data. Generating the samples is harder .. I once used things like Fibonacci to generate input for integer calculator tests. You really want to run these benchmarks for at least 5 minutes and randomly to eliminate cache effects. If you can generate increasing sized problems based on a single linear integer parameter, my Python test harness can handle randomising the tests, run multiple processes, and also draw a nice plot of the results, such as these: http://felix.sourceforge.net/speed/en_flx_perf_0005.html http://felix.sourceforge.net/speed/en_flx_perf_0012.html full sources etc are here to read: http://felix.sourceforge.net/speed/en_flx_perf_top.html or you can grab from Felix svn archive (the test harness is independent of Felix). Or I can email to you (saves building Felix just to get the test harness ..) Ocamlopt scores quite well thank you! This data is from two AMD64 machines running Ubuntu Linux and is the result of HOURS of run time. I don't know if it is cache, random context switches, or what, but on my boxes variations up to 20% on tests under 5 seconds are normal. Note the measurements are real time. Other measurements are suspect .. you really SHOULD count VM paging for example. -- John Skaller Felix, successor to C++: http://felix.sf.net