Benchmarks against imperative languages

Benchmarks against imperative languages

[Sarah Mount]

Informal benchmarks (Doug Bagley, Jon Harrop, ...) of OCaml code
against other languages seem to suggest that Ocaml code performs about
as well as C++ code in many cases. Does anyone know of any published
(as in dead-tree) work that might confirm/deny this?

Thanks!

Sarah
--
http://www.mis.coventry.ac.uk/research/imd/
http://varspool.blogspot.com
http://varspool.blogspot.com/2005/11/how-to-pass-your-final-year-thesis.html

Re: [Caml-list] Benchmarks against imperative languages

[Basile STARYNKEVITCH]

Le Sat, Mar 04, 2006 at 02:04:44PM +0000, Sarah Mount écrivait/wrote:
> Informal benchmarks (Doug Bagley, Jon Harrop, ...) of OCaml code
> against other languages seem to suggest that Ocaml code performs about
> as well as C++ code in many cases. Does anyone know of any published
> (as in dead-tree) work that might confirm/deny this?

I don't believe this question has a precise, practical answer.

We all know (by experience) that Ocaml performs quite well. We also
know that for most (but not all) of the software we are coding, the
cost and time of development does significantly matter, and a 10%
decrease in performance is not that important, hence Ocaml brings a
real win.

A real answer would be to have a team of programmers fluent in Ocaml
write a code (an real-sized application of hundreds of KLOC of source
code, representing several man-years of effort) which has exactly the
same precise specification than an existing code written in C. But
this will never happen (it is too costly but quite useless an
experiment). For example, nobody will recode in Ocaml an exact clone
of gcc-4.1, firefox-1.5, or mysql-5.0!


I don't even know about big-sized Ocaml applications which have C++
written competitors....

Why do you need more than the informal benchmarks, and your personal
experience?


Maybe a related question is "why corporations (or professionals) are
switching from C++ to Ocaml" but this is a question that won't be (for
social and political reasons easy to guess) easily and really
answered.

Is your real question: "help me to convince my boss to let me code in
OCaml"?

Regards.


-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/ 
email: basile<at>starynkevitch<dot>net 
aliases: basile<at>tunes<dot>org = bstarynk<at>nerim<dot>net
8, rue de la Faïencerie, 92340 Bourg La Reine, France

Re: [Caml-list] Benchmarks against imperative languages

[David Teller]

Le samedi 04 mars 2006 à 15:36 +0100, Basile STARYNKEVITCH a écrit :
> Le Sat, Mar 04, 2006 at 02:04:44PM +0000, Sarah Mount écrivait/wrote:
> A real answer would be to have a team of programmers fluent in Ocaml
> write a code (an real-sized application of hundreds of KLOC of source
> code, representing several man-years of effort) which has exactly the
> same precise specification than an existing code written in C. But
> this will never happen (it is too costly but quite useless an
> experiment). For example, nobody will recode in Ocaml an exact clone
> of gcc-4.1, firefox-1.5, or mysql-5.0!

Unfortunately... I don't know about MySQL, but both gcc and Firefox are
applications which shouldn't have been written in C/C++ in the first
place and which spend more LOC emulating other paradigms (respectively
ML/lisp-style and Objective-C/Python-style) than actually doing anything
else -- in the case of Firefox, my experience is that at least 2/3 of
the code is just the application of the dynamic objects layer. 

I actually think that exactly replicating an application would yield any
benefits, as applications are typically designed to be implemented in
one specific language, and around the limitations of this language. On
the other hand, I would consider it interesting to plan and implement an
independent large-scale end-user-oriented open-source project in OCaml.
The question being, of course "what kind of application" ? I'm not sure
that OCaml is the best language for end-user-oriented applications, but
I'd like to be proved wrong.

On the third hand (yeah, Chernobyl), I think that the Next Big Thing in
language is distribution, not a new paradigm or a new take on an old
paradigm. OCaml itself is not that Next Big Thing. JoCaml could be, or
perhaps Acute.

Cheers,
 David
-- 
Read, Write, and Publish Standard eBooks
  Free, Open Software, Open Standards and multi-platform
    The OpenBerg project http://www.openberg.org

Re: [Caml-list] Benchmarks against imperative languages

[Richard Jones]

On Sat, Mar 04, 2006 at 06:01:25PM +0000, David Teller wrote:
> Unfortunately... I don't know about MySQL, but both gcc and Firefox are
> applications which shouldn't have been written in C/C++ in the first
> place[...]

I picked up on Firefox too.  PrinceXML[1] (which is commercial
software) is kind-of a Firefox equivalent, and IIRC it's written in
some sort of interesting logic programming language.  It would be
fascinating to find out how many LOC it contains and how it compares
to Firefox in other ways (bugs, speed, maintainability, etc.)

Rich.

[1] http://www.princexml.com/overview/

-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Team Notepad - intranets and extranets for business - http://team-notepad.com

Re: [Caml-list] Benchmarks against imperative languages

[Christophe TROESTLER]

On Sun, 5 Mar 2006, Richard Jones <rich@annexia.org> wrote:
> 
> PrinceXML[1] (which is commercial software) is kind-of a Firefox
> equivalent, and IIRC it's written in some sort of interesting logic
> programming language.

Seems it is Mercury (http://en.wikipedia.org/wiki/Prince_XML).

Looking for suggestions on self-referential object definitions

[David Powers]

I am in the middle of hacking together a rogue-like game in OCaml for 
fun and to get a better feel for the language, and I have come across a 
stumbling block.  Specifically I began to model items in the game as 
objects deriving from a base item class.  All well and good until I 
tried to come up with a way to model a container (like a backpack, or 
sack).  The container itself was an item, that could hold other items - 
including, possibly, other containers.

Some brief dabbling led me to the idea that I could store the items in 
the container in a list using a variant type to differentiate the 
specific types of items - but I can't for the life of me think how to 
add containers to that type list without having defined containers 
first.  I've included the simple code below so that hopefully some smart 
person can point out how dumb I'm being.  ;)

-David


class virtual item =
   object (self)
     val mutable name = ""

     method name = name

     method set_name newname = name <- newname
   end
;;


class weapon =
   object (self)
     inherit item
   end
;;


class container =
   object (self)
     inherit item

     val mutable items = []

     method add newitem = items <- (newitem :: items)

     method contents = items

     method remove i = items <- List.filter (fun x -> x != i) items

     method contents_to_string =
       let print_item i =
         match i with
           | `Weapon w -> Printf.sprintf "%s (weapon)" w#name
           | `Container c -> Printf.sprintf "%s (container) - 
Containing:\n%s" c#name c#contents_to_string
       in
         List.map print_item items

   end
;;

Re: [Caml-list] Looking for suggestions on self-referential object definitions

[Jonathan Roewen]

> class virtual item =
>   object (self)
>     val mutable name = ""
>
>     method name = name
>
>     method set_name newname = name <- newname
>   end
> ;;
>
>
> class weapon =
>   object (self)
>     inherit item
>   end
> ;;
>
>
> class container =
>   object (self)
>     inherit item
>
>     val mutable items = []
>
>     method add newitem = items <- (newitem :: items)
>
>     method contents = items
>
>     method remove i = items <- List.filter (fun x -> x != i) items
>
>     method contents_to_string =
>       let print_item i =
>         match i with
>           | `Weapon w -> Printf.sprintf "%s (weapon)" w#name
>           | `Container c -> Printf.sprintf "%s (container) -
> Containing:\n%s" c#name c#contents_to_string
>       in
>         List.map print_item items
>
>   end
> ;;

First off, method contents_to_string has conflicting types to be
recursive. Second, adding a type constraint to items should fix your
problems.

My changes:

class container =
  .....
    val mutable items : [ `Weapon of weapon | `Container of container
] list = []
  ...
    method contents_to_string =
       .....
        List.fold_left (fun a b -> if a = "" then print_item b else a
^ "; " ^ print_item b) "" items

  end
;;

Jonathan

Re: [Caml-list] Looking for suggestions on self-referential object definitions

[David Powers]

I had no idea you could reference types in variant definitions before 
the types were fully defined - mostly due to some failure at trying to 
define the type before I even opened the class definition for container. 
  That said, obviously you can, which is enormously helpful.  ;)

I assume (based on some simple experiments) that this will only work out 
using polymorphic variants.

-David

Jonathan Roewen wrote:
>> class virtual item =
>>   object (self)
>>     val mutable name = ""
>>
>>     method name = name
>>
>>     method set_name newname = name <- newname
>>   end
>> ;;
>>
>>
>> class weapon =
>>   object (self)
>>     inherit item
>>   end
>> ;;
>>
>>
>> class container =
>>   object (self)
>>     inherit item
>>
>>     val mutable items = []
>>
>>     method add newitem = items <- (newitem :: items)
>>
>>     method contents = items
>>
>>     method remove i = items <- List.filter (fun x -> x != i) items
>>
>>     method contents_to_string =
>>       let print_item i =
>>         match i with
>>           | `Weapon w -> Printf.sprintf "%s (weapon)" w#name
>>           | `Container c -> Printf.sprintf "%s (container) -
>> Containing:\n%s" c#name c#contents_to_string
>>       in
>>         List.map print_item items
>>
>>   end
>> ;;
> 
> First off, method contents_to_string has conflicting types to be
> recursive. Second, adding a type constraint to items should fix your
> problems.
> 
> My changes:
> 
> class container =
>   .....
>     val mutable items : [ `Weapon of weapon | `Container of container
> ] list = []
>   ...
>     method contents_to_string =
>        .....
>         List.fold_left (fun a b -> if a = "" then print_item b else a
> ^ "; " ^ print_item b) "" items
> 
>   end
> ;;
> 
> Jonathan

Re: [Caml-list] Looking for suggestions on self-referential object definitions

[Martin Jambon]

On Sat, 4 Mar 2006, David Powers wrote:

> I am in the middle of hacking together a rogue-like game in OCaml for fun and 
> to get a better feel for the language, and I have come across a stumbling 
> block.  Specifically I began to model items in the game as objects deriving 
> from a base item class.  All well and good until I tried to come up with a 
> way to model a container (like a backpack, or sack).  The container itself 
> was an item, that could hold other items - including, possibly, other 
> containers.
>
> Some brief dabbling led me to the idea that I could store the items in the 
> container in a list using a variant type to differentiate the specific types 
> of items - but I can't for the life of me think how to add containers to that 
> type list without having defined containers first.  I've included the simple 
> code below so that hopefully some smart person can point out how dumb I'm 
> being.  ;)
>
> -David

What you are doing is correct, you just need to tell the compiler about 
the type of the items.
There are a several ways of doing this, here is one:


class virtual item =
    object (self)
      val mutable name = ""

      method name = name

      method set_name newname = name <- newname
    end
;;


class weapon =
    object (self)
      inherit item
    end
;;


type 'a item = [ `Container of 'a
 	       | `Weapon of weapon ]

class container =
    object (self)
      inherit item

      val mutable items : container item list = []

      method add newitem = items <- (newitem :: items)

      method contents = items

      method remove i = items <- List.filter (fun x -> x != i) items

      method contents_to_string =
        let print_item i =
          match i with
            | `Weapon w -> Printf.sprintf "%s (weapon)" w#name
            | `Container c -> Printf.sprintf "%s (container) -
Containing:\n%s" c#name c#contents_to_string
        in
          String.concat "\n" (List.map print_item items)

    end
;;


Another possibility is to define the type of the objects without defining 
a class or class type, so that you can write directly a recursive type 
definition separately from the class definition:

type obj = < content : item list >
and item = [ `A | `B of obj ]

(* BTW I don't know how to tell the compiler that the class creates
    objects of type obj, but this can be done: *)
class c = object method content : item list = [] end


Martin

--
Martin Jambon, PhD
http://martin.jambon.free.fr

Visit http://wikiomics.org, bioinformatics wiki

Re: [Caml-list] Looking for suggestions on self-referential object definitions

[Oliver Bandel]

I doubt "self-referential" is the item you look for.
"Recursive" datastructure fits better.



Ciao,
   Oliver

Re: [Caml-list] Benchmarks against imperative languages

[Jon Harrop]

On Saturday 04 March 2006 14:36, Basile STARYNKEVITCH wrote:
> We all know (by experience) that Ocaml performs quite well. We also
> know that for most (but not all) of the software we are coding, the
> cost and time of development does significantly matter, and a 10%
> decrease in performance is not that important, hence Ocaml brings a
> real win.

Yes and no. I think the relative performance of OCaml is quite variable. Here 
is a range of examples:

1. Straightforwardly-implemented algorithms can be vastly faster when written 
in a functional style and optimising imperative equivalents can be tricky, 
e.g. set theoretic operations and the "n"th-nearest neighbour example from my 
ocaml book:

  http://www.ffconsultancy.com/products/ocaml_for_scientists/complete/

2. Any problems that are near the limit of practical feasibility are likely to 
benefit enormously from improved development time. This includes the limits 
of computational complexity, memory usage and algorithm-implementation 
complexity.

3. Floating-point intensive algorithms can be miraculously (IMHO) fast for an 
FPL, e.g. my ray tracer benchmark on AMD (particularly AMD64):

  http://www.ffconsultancy.com/free/ray_tracer/languages.html

4. Certain operations can be surprisingly slow, e.g. my Sudoku solver is 
several times slower in OCaml because ocamlopt does not optimise integer div 
or mod by a constant:

  http://www.ffconsultancy.com/free/sudoku/

The relative development speed in OCaml and C++ is also quite variable. Some 
applications, like compilers and interpreters, clearly benefit enormously 
from languages like OCaml. Other applications, like small and simple 
numerical programs (e.g. the ray tracer) benefit from the availability of 
libraries like Blitz++ and Boost for C++. In the former case, development can 
be an order of magnitude faster in OCaml. In the latter case, the time taken 
to develop similar-performance programs can be roughly the same.

> A real answer would be to have a team of programmers fluent in Ocaml
> write a code (an real-sized application of hundreds of KLOC of source
> code, representing several man-years of effort) which has exactly the
> same precise specification than an existing code written in C. But
> this will never happen (it is too costly but quite useless an
> experiment). For example, nobody will recode in Ocaml an exact clone
> of gcc-4.1, firefox-1.5, or mysql-5.0!

I have cited my most relevant example before - a vector graphics library which 
was 4x longer in C++, the worst-case performance was 5x faster in OCaml and 
the development time was an order of magnitude faster in OCaml. It was a 
rewrite from C++ to OCaml, so it is biased. However, I ditched C++ because it 
was infeasible to do the rewrite in C++ and I have not looked back since. 
Also, I can't say how OCaml compares to other FPLs. The code size was 
~10kLOC.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists

Re: [Caml-list] Benchmarks against imperative languages

[skaller]

On Sun, 2006-03-05 at 11:54 +0000, Jon Harrop wrote:
> On Saturday 04 March 2006 14:36, Basile STARYNKEVITCH wrote:
> > We all know (by experience) that Ocaml performs quite well. We also
> > know that for most (but not all) of the software we are coding, the
> > cost and time of development does significantly matter, and a 10%
> > decrease in performance is not that important, hence Ocaml brings a
> > real win.
> 
> Yes and no. I think the relative performance of OCaml is quite variable.

The relative performance of many algorithms can be highly sensitive 
to tiny changes in coding. And the significance of small changes in
efficiency are also sensitive to context.

So you think 10% performance isn't significant? Well, I make
that 36.5 DAYS out of the year I am going to take out of your holidays!
What? What do you mean, you only get 28 days a year holidays? :)

-- 
John Skaller <skaller at users dot sourceforge dot net>
Async PL, Realtime software consultants
Checkout Felix: http://felix.sourceforge.net