Re: [Caml-list] Benchmark suggestion (Was: Programming Language Shootout)

["Wolfgang Müller" <Wolfgang.Mueller2@uni-bayreuth.de>]

Well, I knew OCaml before the shootout :-) , but I loved the shootout
results a lot.

OCaml helped me once in a project I had to code on a small slow portable
computer, where c++ was too slow to compile, JAVA needed too much RAM, and
Python/Perl Scripting was out of the question for execution speed reasons.

We then recently did some benchmarking at our place comparing JAVA against
c++ for an indexing application (inverted files). I also did some initial
tests with OCaml, however, as it was out of the question to convince my
peers to use OCaml I abandoned :-(

What was emphasized by the results of that benchmark is that you can
always do fancy optimisations, but what matters in projects that have to
be finished in time is the relation between programmer time and (execution
time || functionality).

JAVA took in our precise scenario about twice the time of c++, which we
deemed in our context fast enough, as we can save developer time when
using JAVA (e.g. faster debug-build-execute). In other contexts this might
be different.

In general you might approximate programmer time via number of lines or
the suggested compressed number of bytes (in absence of a better measure),
and the execution time you can measure, as well as the functionality.

What I would dream of, would be two classes of shootout-like benchmarks:

1. Gurus in every language solve given standard problems, like e.g.
parsing of web pages, indexing text, etc.etc. (suitable problems would
need to be found).

Here we would learn how fast and/or short someone really good can make it,
some upper bound of the performance reachable.

2. Graduate students with the usual experience in functional and
imperative/OO programming solve the problem given ample documentation of
the language and three days of time (like the ICFP contest?).

This would give an idea if one reaches reasonably quickly useful results
when switching to a given language. In my experience OCaml is also very
good at that.

What do you think about these ideas?

Cheers,
Wolfgang


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners