[Caml-list] How can I check for the use of polymorphic equality?

[Caml-list] How can I check for the use of polymorphic equality?

[Neel Krishnaswami]

I spent the last few hours tracking down a bug that was the result of
accidentally using the polymorphic equality rather than a custom
comparison function. Is there any way I can tell the compiler not to
allow the use of polymorphic equality at certain types, or even just
to raise an exception when it tries to compare values of those types?

-- 
Neel Krishnaswami
neelk@alum.mit.edu

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] How can I check for the use of polymorphic equality?

[brogoff]

Quite a conundrum, since equality wouldn't be polymorphic if it didn't work for 
all (non-functional) values!

Following that silly parenthetical comment, you could explicitly tag values with 
a functional fields. As long as they are the first field you're OK (try 
swapping the field order if you don't see what I mean). 

# type 'a no_eq = { no_eq_tag : unit -> int; data : 'a };;
type 'a no_eq = { no_eq_tag : unit -> int; data : 'a; } 
# let x0 = {data = 0; no_eq_tag = fun () -> 0};;          
val x0 : int no_eq = {no_eq_tag = <fun>; data = 0}
# let x1 = {data = 1; no_eq_tag = fun () -> 1};;
val x1 : int no_eq = {no_eq_tag = <fun>; data = 1}
# x0 = x1;;
Exception: Invalid_argument "equal: functional value".

You can use tuples isntead of records if you wish too. BTW, this tagging  
strikes me as very ugly. 

A cleaner solution IMO is be to use your own order data type patterned after 
every other FPL, type order = Less | Equal | Greater, or your own equality data 
type, to avoid this if it starts becoming a problem.

Equality is a rough edge in just about every programming language. 

-- Brian

On Mon, 31 Mar 2003, Neel Krishnaswami wrote:

> 
> I spent the last few hours tracking down a bug that was the result of
> accidentally using the polymorphic equality rather than a custom
> comparison function. Is there any way I can tell the compiler not to
> allow the use of polymorphic equality at certain types, or even just
> to raise an exception when it tries to compare values of those types?
> 
> -- 
> Neel Krishnaswami
> neelk@alum.mit.edu
> 
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
> Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> 

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] How can I check for the use of polymorphic equality?

[Jason Hickey]

Neel Krishnaswami wrote:
> I spent the last few hours tracking down a bug that was the result of
> accidentally using the polymorphic equality rather than a custom
> comparison function. Is there any way I can tell the compiler not to
> allow the use of polymorphic equality at certain types, or even just
> to raise an exception when it tries to compare values of those types?


This has been bothering me for quite some time.  There are times when I 
would like to modify an implementation to use a coarser comparison 
function than Pervasives.compare.  This requires manually finding all 
uses of =, <, etc. and modifying them to use the new comparison 
function.  It would be great if the typechecker would help, since it is 
easy to forget and reintroduce the use of = by accident.

I suppose the real answer would be to extend the type system with 
equality types, similar to those in SML.

In the meantime, I use a hack to help catch errors at runtime.  The idea 
is this.
    1. The type you care about is probably abstract.
    2. Add an abstract value to the data in your type, so that equality
       will fail (at runtime).

Suppose that your type is declared like this:

    type t = t1

    let create ... : t = e

Modify this as follows:

    type t = t1 * Obj.t

    let create ... : t =
        let bogus = Obj.new_block Obj.abstract_tag 1 in
           e, bogus

You'll see the following behavior:

    let o1 = create ... in
    let o2 = create ... in
        o1 = o2

Exception: Invalid_argument "equal: abstract value"

To find these errors, you'll probably want to use exception tracing. 
Use byte-code, and set the environment variable OCAMLRUNPARAM=b.

The "bogus" values are wasted space, so you may want to remove this 
wrapper code if you are confident that you found all the errors.

Jason


-- 
Jason Hickey                  http://www.cs.caltech.edu/~jyh
Caltech Computer Science      Tel: 626-395-6568 FAX: 626-792-4257

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] How can I check for the use of polymorphic equality?

[Pierre Weis]

[...]
> In the meantime, I use a hack to help catch errors at runtime.  The idea 
> is this.
>     1. The type you care about is probably abstract.
>     2. Add an abstract value to the data in your type, so that equality
>        will fail (at runtime).
[...]
> -- 
> Jason Hickey                  http://www.cs.caltech.edu/~jyh
> Caltech Computer Science      Tel: 626-395-6568 FAX: 626-792-4257

Hi,

Your trick is fine and clever but it seems overcomplex to me: to check
usage of equality, I would just redefine equality!

If you add, in the scope of each module (for instance by systematically
opening a basic code module), a new binding for ( = ) as in

let ( = ) (x : 'a) (y : 'a) = assert false;;

then the generic equality is no more used and any occurrence of ( = )
will raise an assertion failure at runtime.

You could also define assign a special type for your ``fake'' generic
equality that would trigger a typecheckeking error statically. For
instance, define

let ( = ) (x : int) (y : bool) = assert false;;

You probably will catch every reasonable usage of ( = ) that way...

Simple, no ?

Pierre Weis

INRIA, Projet Cristal, Pierre.Weis@inria.fr, http://pauillac.inria.fr/~weis/


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] How can I check for the use of polymorphic equality?

[Chris Hecker]

>You probably will catch every reasonable usage of ( = ) that way...
>Simple, no ?

But that a) requires you to put it in scope at the top of each file, and 
more importantly b) means you can't use = anywhere, doesn't it?  That seems 
kinda strong for the very specific check, and the abstract type thing 
(which was going to be my suggestion as well, but I forgot you could do it 
with Obj all in caml!) is both localized and only catches the ones you care 
about.

Chris

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] How can I check for the use of polymorphic equality?

[Pierre Weis]

> >You probably will catch every reasonable usage of ( = ) that way...
> >Simple, no ?
> 
> But that a) requires you to put it in scope at the top of each file,

Absolutely, but remember it is just a debugging phase, not a
production state.

> and more importantly b) means you can't use = anywhere, doesn't it?

No: it means you have to do something more than just writing = when
you need to call the predefined generic equality. You first have to
think if there is not another predicate that is not more appropriate
to the situation, and if there is none, you just have to insert a mere

  let ( = ) = Pervasives.( = ) in

before each relevant occurrence of ( = ). Those single lines can be
easily removed after proper debugging (use an emacs keyboard macro + a
M-x grep -n -e ... *.ml).

Still sounds rather simple and elegant to me...

> That seems kinda strong for the very specific check, and the
> abstract type thing (which was going to be my suggestion as well,
> but I forgot you could do it with Obj all in caml!) is both
> localized and only catches the ones you care about.
>
> Chris

No doubt that you could use an abstract type trick (possibly using the
Obj module :(), if you fill more comfortable with this trick. The hack
I proposed is just simpler, only based on simple core language
features, and has the additional advantage to force you to have a look
at all occurrences of ( = ) in your source code, which in my mind is a
desirable side effect.

Best regards,

Pierre Weis

INRIA, Projet Cristal, Pierre.Weis@inria.fr, http://pauillac.inria.fr/~weis/


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] How can I check for the use of polymorphic equality?

[Jason Hickey]

Pierre Weis wrote:
>>and more importantly b) means you can't use = anywhere, doesn't it?
> 
> No: it means you have to do something more than just writing = when
> you need to call the predefined generic equality. You first have to
> think if there is not another predicate that is not more appropriate
> to the situation, and if there is none, you just have to insert a mere
> 
>   let ( = ) = Pervasives.( = ) in
> 
> before each relevant occurrence of ( = ). Those single lines can be
> easily removed after proper debugging (use an emacs keyboard macro + a
> M-x grep -n -e ... *.ml).
 >
 > ...
 > Pierre Weis


One really nice thing about your solution is that we can get type 
inference to catch uses of =.  But it can be awkward because we still 
need relations on base types.  Imagine that we have a Set module that 
should use Set.equal, not =.  Consider the fixpoint code below.

type empty

let (=) (x : empty) (y : empty) = assert false
let (>) (x : empty) (y : empty) = assert false
...

let rec fixpoint s1 s2 =
    if Set.equal s1 s2 then
       s1
    (* Can't use >, so use Pervasives.(>) directly *)
    else if Pervasives.(>) (Set.cardinal s1) (Set.cardinal s2) then
       fixpoint (f s1) s2
    else
       fixpoint s1 (f s2)

It can get ugly if we have to use the Pervasives.(=) a lot.  Of course, 
we could be a little smarter, and define something like the following 
for each of the base types.

let (=@) : int -> int -> bool = Pervasives.(=)

Perhaps even

let (=$) : 'a set -> 'a set -> bool = Set.equal

This might be a compromise, though remembering all the equality suffixes 
would be a hassle.

Jason

-- 
Jason Hickey                  http://www.cs.caltech.edu/~jyh
Caltech Computer Science      Tel: 626-395-6568 FAX: 626-792-4257

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] How can I check for the use of polymorphic equality?

[Andreas Rossberg]

Chris Hecker wrote:
> 
>> You probably will catch every reasonable usage of ( = ) that way...
>> Simple, no ?
> 
> But that a) requires you to put it in scope at the top of each file, and 
> more importantly b) means you can't use = anywhere, doesn't it?

And c) puts the responsibility to avoid misuses of structural comparison 
into the hands of the clients of an abstract type, instead of enabling 
its implementer to prevent them. That counters the very idea of type 
abstraction and is hence not The Right Thing. ;-)

   - Andreas

-- 
Andreas Rossberg, rossberg@ps.uni-sb.de

"Computer games don't affect kids; I mean if Pac Man affected us
  as kids, we would all be running around in darkened rooms, munching
  magic pills, and listening to repetitive electronic music."
  - Kristian Wilson, Nintendo Inc.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

[Caml-list] ocaml and large development projects

[Traudt, Mark]

I am new to ObjectiveCaml and am wondering whether it has been used for any
large projects (i.e. hundreds of thousands or more lines of code, possible
>100 modules)?  If so, how well has it scaled, in terms of performance,
memory usage, compile time, and any other relevant characteristics?  If not,
is this because there are known issues that would prevent it from being
effectively used beyond a certain size project?

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Jason Hickey]

Traudt, Mark wrote:
> I am new to ObjectiveCaml and am wondering whether it has been used for any
> large projects (i.e. hundreds of thousands or more lines of code, possible
> 
>>100 modules)?  If so, how well has it scaled, in terms of performance,
> 
> memory usage, compile time, and any other relevant characteristics?  If not,
> is this because there are known issues that would prevent it from being
> effectively used beyond a certain size project?

We use OCaml in several projects.  Two of them are in the medium-sized 
category: a compiler (mojave.caltech.edu, ~300K lines), and a theorem 
prover (metaprl.org, ~250K lines).  Together, about 3000 files, 50-100 
functors.  The OCaml toolset scales well, with no real issues.  Compile 
time appears to scale linearly in the size of the project, with a slight 
exception for linking.  Typical numbers are 3-10min (it depends on the 
project) for a full native-code build, about 10sec link time, on a 1GHz 
PIII-class machine, Linux, 256MB.  I haven't measured memory usage; it 
hasn't been an issue.

One issue to be aware of: if you build native code, when you change a 
module, then all dependent modules must be recompiled.  This can be 
surprising when a small change to a utility module causes a near full 
recompile of the project.  The byte-code compiler does not have this 
issue, and is sufficient for testing in most cases.

Jason

-- 
Jason Hickey                  http://www.cs.caltech.edu/~jyh
Caltech Computer Science      Tel: 626-395-6568 FAX: 626-792-4257

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Chris Hecker]

>One issue to be aware of: if you build native code, when you change a 
>module, then all dependent modules must be recompiled.  This can be 
>surprising when a small change to a utility module causes a near full 
>recompile of the project.  The byte-code compiler does not have this 
>issue, and is sufficient for testing in most cases.

Is this true?  The cmi file for the interface doesn't insulate you from 
this in native code?  My project is native code, but I haven't been using 
mlis yet, because of the annoying interface-code-duplication-issue when 
changing things often is a huge pain, but I assumed this would just work 
correctly (meaning if the mli doesn't change then dependents won't 
recompiler) when I started using mlis to speed up my builds.  I've seen 
problems when doing simple tests before ("Files b.cmx and a.cmx make 
inconsistent assumptions over implementation A") that I just attributed to 
my screwups in the quick tests, but now I think they're this 
problem.  Another quick test indicates you're right.

Uh, isn't this a major flaw in the compiler?  Why does this happen?  This 
seems like a vital part of scaling native code projects that's broken...I 
was certainly counting on it working correctly quite soon.

Chris


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[David Brown]

On Sat, May 17, 2003 at 10:32:54PM -0700, Chris Hecker wrote:
> 
> >One issue to be aware of: if you build native code, when you change a 
> >module, then all dependent modules must be recompiled.  This can be 
> >surprising when a small change to a utility module causes a near full 
> >recompile of the project.  The byte-code compiler does not have this 
> >issue, and is sufficient for testing in most cases.
> 
> Uh, isn't this a major flaw in the compiler?  Why does this happen?  This 
> seems like a vital part of scaling native code projects that's broken...I 
> was certainly counting on it working correctly quite soon.

Inlining.  Inlining is an important optimization, especially for
functional languages, where many functions are typically short.

However, inlinign causes a dependency on the bodies that are being
inlined.

These dependencies are correctly displayed by ocamldep, though.

Dave

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Chris Hecker]

> > Uh, isn't this a major flaw in the compiler?  Why does this happen?
>Inlining.  Inlining is an important optimization, especially for
>functional languages, where many functions are typically short.

Sure, but there has to be a better way to do this than forcing a recompile 
of the entire program when you change a debug print in your lowest level 
completely encapsulated library.  That's insanely bad software engineering, 
and this is a huge issue.  The only reason this hasn't been #1 on my list 
of "must fix"'s is because there's no way I would have even guessed this 
was the actual behavior...I didn't even bother to check this it's so 
absurd, I just assumed it worked and was some stupid makefile thing I was 
doing in my quick tests (since it works for bytecode).  Any production C++ 
programmer evaluating caml as a possible alternative for large scale 
software would simply laugh and write off the language as an option for 
this behavior alone, in my opinion.  Certainly all of the game developers I 
know would.

Do people think I'm overreacting there?  I can't believe anyone thinking 
about using ocaml for big native code projects wouldn't consider this a 
100% showstopper.  I certainly would have thought differently about using 
ocaml if I'd known this when I started my project.

>These dependencies are correctly displayed by ocamldep, though.

Well, that answers another confusion of mine when I did my quick tests a 
while back, but that certainly isn't an excuse for the behavior.  That's 
like saying "our programming language can't add two integers, but the 
compiler catches this and issues an error, so it's okay."

Don't people consider separate compilation and the ability to change 
implementation without complete project recompiles a fundamental 
requirement of non-toy languages?

If this can't be fixed, there needs to be a way to disable cross-file 
inlining so that it can be worked around (and the checksums should reflect 
this, and ocamldep should do the right thing, etc.), and then when you want 
to "globally optimize" you can recompile the world.

Totally flabbergasted,
Chris


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[John Carr]

> Don't people consider separate compilation and the ability to change 
> implementation without complete project recompiles a fundamental 
> requirement of non-toy languages?

Ten or fifteen years ago (when building a large project took a day of
compile time, and building the complete OS took several days) I would
have answered "yes".

Now it is an important feature but not an essential feature.


> If this can't be fixed, there needs to be a way to disable cross-file 
> inlining so that it can be worked around (and the checksums should reflect 
> this, and ocamldep should do the right thing, etc.), and then when you want 
> to "globally optimize" you can recompile the world.

At least two possibilities come to mind:

1. An inining threshold that affects generation of .cmx files instead
of their use.  -gen-inline 0 means don't generate any inlining information,
-gen-inline 1 means put small functions in the .cmx file, etc.

2. Only put functions defined in the .mli file in the .cmx file

Either way you have to avoid letting regeneration of an identical
.cmx file trigger a rebuild of the whole project.  Two solutions:

1. Make ocamlopt leave an existing .cmx file if the new contents
would be identical.

2. Use scons, a python-based make replacement that uses file contents
instead of modification time.  It is designed for large projects.
I'll have to try it out with ocaml some day.  The startup cost is
high because it works not at all like make.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Ed L Cashin]

John Carr <jfc@MIT.EDU> writes:

>> Don't people consider separate compilation and the ability to change 
>> implementation without complete project recompiles a fundamental 
>> requirement of non-toy languages?
>
> Ten or fifteen years ago (when building a large project took a day of
> compile time, and building the complete OS took several days) I would
> have answered "yes".
>
> Now it is an important feature but not an essential feature.

Working with an O.S. kernel or mozilla on a one-year-old PC reminds me
that it still essential for some projects.

-- 
--Ed L Cashin     PGP public key: http://noserose.net/e/pgp/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Lex Stein]

You are missing the point. It is not true that building a program requires
full recompilation. The point that Prof. Hickey made was that a program
requires full recompilation if you make a change then build *native* code.
The bytecode compiler (ocamlc) does allow separate compilation. I believe
that nightly builds are done for checking code compilability, ensuring
that interfaces still compile against one another, and for generating an
executable to run correctness regression tests against. I don't believe
nightly builds exist to either generate an executable target for
performance tests or generate binary product code; the reasons one would
use ocamlopt over ocamlc.

Lex

On Sun, 18 May 2003, Ed L Cashin wrote:

> John Carr <jfc@MIT.EDU> writes:
>
> >> Don't people consider separate compilation and the ability to change
> >> implementation without complete project recompiles a fundamental
> >> requirement of non-toy languages?
> >
> > Ten or fifteen years ago (when building a large project took a day of
> > compile time, and building the complete OS took several days) I would
> > have answered "yes".
> >
> > Now it is an important feature but not an essential feature.
>
> Working with an O.S. kernel or mozilla on a one-year-old PC reminds me
> that it still essential for some projects.
>
> --
> --Ed L Cashin     PGP public key: http://noserose.net/e/pgp/
>
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
> Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Ed L Cashin]

Lex Stein <stein@eecs.harvard.edu> writes:

> You are missing the point. It is not true that building a program
> requires full recompilation. The point that Prof. Hickey made was
> that a program requires full recompilation if you make a change then
> build *native* code.

I've never used ocamlc -- I have no reason to, so I tend to think of
ocamlopt as the ocaml compiler.  Ideally, ocamlc and ocamlopt would be
equally flexible.  

It never occurred to me that I'd have to create a more complex
makefile just to get real separate compilation during development by
using ocamlc, doing ocamlopt only at the end.  

-- 
--Ed L Cashin     PGP public key: http://noserose.net/e/pgp/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Lex Stein]

Fortunately you can amortize the cost of this change to your Makefile
across all your builds. In the limit it approaches 0.

Lex

On Sun, 18 May 2003, Ed L Cashin wrote:

> Lex Stein <stein@eecs.harvard.edu> writes:
>
> > You are missing the point. It is not true that building a program
> > requires full recompilation. The point that Prof. Hickey made was
> > that a program requires full recompilation if you make a change then
> > build *native* code.
>
> I've never used ocamlc -- I have no reason to, so I tend to think of
> ocamlopt as the ocaml compiler.  Ideally, ocamlc and ocamlopt would be
> equally flexible.
>
> It never occurred to me that I'd have to create a more complex
> makefile just to get real separate compilation during development by
> using ocamlc, doing ocamlopt only at the end.
>
> --
> --Ed L Cashin     PGP public key: http://noserose.net/e/pgp/
>
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
> Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Markus Mottl]

On Sun, 18 May 2003, Ed L Cashin wrote:
> It never occurred to me that I'd have to create a more complex makefile
> just to get real separate compilation during development by using
> ocamlc, doing ocamlopt only at the end.

OCamlMakefile does that for you:

  http://www.ai.univie.ac.at/~markus/home/ocaml_sources.html#OCamlMakefile

A "make bc" (or simply: "make") will build byte code, a "make nc" will
build native code for the sources you specify. In each case "make"
will only build what is still missing.

OCamlMakefile should scale up fairly well for medium-sized
OCaml-projects. The limitations / problems that appear with larger
projects are mostly related to general issues concerning "make", which
isn't a particularly well-suited tool for this purpose, but the only
one which is reliably installed on most development platforms.

Regards,
Markus Mottl

-- 
Markus Mottl          http://www.oefai.at/~markus          markus@oefai.at

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Nicolas Cannasse]

> > It never occurred to me that I'd have to create a more complex makefile
> > just to get real separate compilation during development by using
> > ocamlc, doing ocamlopt only at the end.
>
> OCamlMakefile does that for you:

You can also use Ocamake : http://tech.motion-twin.com/ocamake which is more
like a "make for ocaml" , entirely written in OCaml, and does not rely on
external tools such as bash , make, and so on... ( in particular, one
doesn't need to install cygwin gnu make on Windows platform to compile ) .
OCamake can then be called as "fully portable" since it works exactly the
same on all platforms OCaml is working on ( and who would like to compile
ocaml code on a platform where ocaml doesn't work ?? )

Nicolas Cannasse

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[John Max Skaller]

Markus Mottl wrote:

>The limitations / problems that appear with larger
> projects are mostly related to general issues concerning "make", which
> isn't a particularly well-suited tool for this purpose, but the only
> one which is reliably installed on most development platforms.

That really is NOT true. Make is NOT installed
on a wide class of platforms (Windows).

On the other hand Python is available for Windows
and most other platforms.

So why bother with crud like make, when you can
write a Python script?

I'm lazy, I just made a list of module,
and check the file times for where to start,
but there is no reason you can't write a more
sophisticated checker -- after all, you can


	os.system('ocamldep ' + filename)

to get dependencies, then build a dependency
tree from them in Python without that much fuss.

Yeah, of course .. you could use Ocaml instead
of Python .. and that is GUARRANTEED to be installed,
since you're using it, right?

So unless I'm totally wrong .. for a single directory
build you could make a harness:

	ocamlbuild prog1 prog2 ..


that would just work, no makefile needed.

-- 
John Max Skaller, mailto:skaller@ozemail.com.au
snail:10/1 Toxteth Rd, Glebe, NSW 2037, Australia.
voice:61-2-9660-0850


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Chris Hecker]

>You are missing the point. It is not true that building a program requires
>full recompilation. The point that Prof. Hickey made was that a program
>requires full recompilation if you make a change then build *native* code.

Actually, this kind of misses the point I was making (perhaps 
poorly).  There are some programs that have a minimum performance floor 
below which they fail to function correctly because they have a "real time" 
user feedback loop.  Games are one example.  You can't actually do useful 
work on a game without being able to run it at reasonably interactive 
framerates because of this feedback loop.  It gets worse, because you have 
another layer of algorithms that attempt to compensate for framerate 
variances, and so you're not even running the same code at low framerates, 
often times.  So, I can't use the bytecode compiler for development and 
haven't been able to for most of the lifetime of my project.  (See posts 
from over a year ago wishing I could link bytecode and native code modules 
together for this feedback reason.)

It may be that the intersection of these two characteristics ("large scale 
projects" and "realtime feedback requiring native compilation") is not an 
interesting set for the caml team to target, although I would argue it's an 
important target to support if a language is to be considered "general 
purpose" and compete with C++.  Unfortunately, this rules out a lot of 
games, which is my area, and one of the places where caml could have a 
chance at success because the project-based nature of the industry means 
there are clear frequent opportunities to make a big language switch 
(unlike industries that evolve code for decades).  This is why I am 
evaluating caml for games by writing my current game in it.  I just wish I 
would have known this "bug" existed before starting because it might have 
changed my mind.  Is this documented somewhere that I overlooked?

>In my experience, it isn't as bad as you make it sound like.  Touching
>early modules doesn't cause a recompile of everything, only modules that
>directly depend on it.  The inlining seems to only happen to one level.

I don't think this is true, since the recompile of the parent will mean the 
recompile of its parents, won't it (it does in my tests)?  And, even if it 
was true, it's still not great, because you really want to be able to edit 
algorithms in math3d.ml without recompiling the whole game.

Chris


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[David Brown]

On Sat, May 17, 2003 at 11:10:03PM -0700, Chris Hecker wrote:

> Do people think I'm overreacting there?  I can't believe anyone thinking 
> about using ocaml for big native code projects wouldn't consider this a 
> 100% showstopper.  I certainly would have thought differently about using 
> ocaml if I'd known this when I started my project.

> If this can't be fixed, there needs to be a way to disable cross-file 
> inlining so that it can be worked around (and the checksums should reflect 
> this, and ocamldep should do the right thing, etc.), and then when you want 
> to "globally optimize" you can recompile the world.

I do agree that there should be an option to disable cross-file
inlining.  Somehow that would have to be coordinated with the
dependencies.  I don't know if this would be easy to implement, though,
since I don't think the code doing the inlining knows where the
functions came from.

In my experience, it isn't as bad as you make it sound like.  Touching
early modules doesn't cause a recompile of everything, only modules that
directly depend on it.  The inlining seems to only happen to one level.

However, ocaml is not alone in supporting cross-file inlining.  The
GNU-Ada compiler (GNAT) supports cross-module inlining.  It is an
option, that is not enabled by default.  The compiler automatically
detects the extra dependencies when it is enabled, and recompiles as
necessary.

There are a few C compilers that support cross-file inlining, but the
ones that I know of do it at "link" time.  So instead of extra
recompiles of source files, you end up with extremely long link times,
since it is doing the global analysis of everything each time you link.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Ville-Pertti Keinonen]

> I do agree that there should be an option to disable cross-file
> inlining.  Somehow that would have to be coordinated with the
> dependencies.  I don't know if this would be easy to implement, though,
> since I don't think the code doing the inlining knows where the
> functions came from.

Preventing dependencies between compilation units would also require 
preventing constant values from being picked up.

> In my experience, it isn't as bad as you make it sound like.  Touching
> early modules doesn't cause a recompile of everything, only modules 
> that
> directly depend on it.  The inlining seems to only happen to one level.

Are you sure?  You could probably get make to do that by depending on 
.ml files instead of .cmx files, but that isn't what ocamldep generates.

> There are a few C compilers that support cross-file inlining, but the
> ones that I know of do it at "link" time.  So instead of extra
> recompiles of source files, you end up with extremely long link times,
> since it is doing the global analysis of everything each time you link.

Some C compilers support inlining across compilation units when you 
compile multiple files at a time, i.e. cc *.c can produce better 
optimized code than the equivalent compiled using a reasonable Makefile.

I doubt that many people make use of such features in C compilers other 
than for benchmarks.  Not because it isn't a useful optimization in 
general, but because C/C++ programming practices work around it by 
putting more things in header files.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Brian Hurt]

On Sat, 17 May 2003, Chris Hecker wrote:

> Any production C++ 
> programmer evaluating caml as a possible alternative for large scale 
> software would simply laugh and write off the language as an option for 
> this behavior alone, in my opinion.  

Except C++ has *exactly* the same problem.  Change a private member of a 
base class, and watch *everything* recompile.  I've seen this more often 
then I want to remember.  This is, of course, assuming you don't have an 
"everything.h" include file, which is quite common if you precompile 
headers.  At which point, change anything in a header and watch everything 
recompile.

And the Java programmers?  The build tool they all (at least all the ones 
I know about) rave about is Ant, which doesn't even support partial 
recompilation unless you jump through serious hoops.  "It's almost as fast 
just to recompile the module" they say, "as it is just to check if it's 
changed."  Of course, checking time stamps on files is way too obvious- 
instead they MD5 checksum the file.  And Java doesn't even try to optimize 
the code on compile.  So they may be right, for their language.

> Don't people consider separate compilation and the ability to change 
> implementation without complete project recompiles a fundamental 
> requirement of non-toy languages?

Maybe.  C++ and Java are toy languages, then.

Brian


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Chris Hecker]

>Except C++ has *exactly* the same problem.  Change a private member of a
>base class, and watch *everything* recompile.  I've seen this more often
>then I want to remember.  This is, of course, assuming you don't have an
>"everything.h" include file, which is quite common if you precompile
>headers.  At which point, change anything in a header and watch everything
>recompile.

It would be silly to turn this thread into a language war, but this 
statement is just false, in practice.  C++ sucks, for sure, and a naive 
programmer can easily make a mess.  But, there are well known and heavily 
used techniques to avoid these problems, and they work.  It's a waste of 
time to debate that.  You can (and most people who have a clue do) make it 
so that you have interfaces and implementation, and touching the latter 
doesn't require anything more than a recompile of that file and a relink.

It does not appear that this is even possible now with ocamlopt.  That is 
the problem I'm talking about.  In C++ you can give up some features and 
get highly encapsulated code with minimal build dependencies.  I don't 
think this is possible now with ocamlopt regardless of whether you're 
willing to give up features or not, given my understanding of the 
situation.  I would love to hear differently, or have someone from the caml 
team comment on this.

>Maybe.  C++ and Java are toy languages, then.

This statement plays well on a mailing list for an alternative language, 
but the reality is fairly different and most people writing large software 
in C++ would write you off as a loon if you said this to them.  There's an 
important difference between a sucky language and a toy language.

Still flabbergasted,
Chris

PS.  A related issue is going to come up with respect to disabling 
cross-file inlining when we get native DLLs.  You want to be able to 
control what gets made available for inlining when building a DLL, since 
one of the uses of DLLs is to be able to supply a new version of code and 
so you can't have it be already inlined in the client application.


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Seth Kurtzberg]

On Monday, May 19, 2003, at 12:31 PM, Chris Hecker wrote:

>
>> Except C++ has *exactly* the same problem.  Change a private member 
>> of a
>> base class, and watch *everything* recompile.  I've seen this more 
>> often
>> then I want to remember.  This is, of course, assuming you don't have 
>> an
>> "everything.h" include file, which is quite common if you precompile
>> headers.  At which point, change anything in a header and watch 
>> everything
>> recompile.

What does an "everything.h" header file have to do with anything?  The 
compilers that support precompiled headers are smart enough to do with 
per header file granularity.  In addition, if one of my programmers 
organized header files based on the compiler's behavior W.R.T. 
precompiled headers, that programmer would soon halt any such behavior 
or start pounding the pavement.

>
> It would be silly to turn this thread into a language war, but this 
> statement is just false, in practice.  C++ sucks, for sure, and a 
> naive programmer can easily make a mess.  But, there are well known 
> and heavily used techniques to avoid these problems, and they work.  
> It's a waste of time to debate that.  You can (and most people who 
> have a clue do) make it so that you have interfaces and 
> implementation, and touching the latter doesn't require anything more 
> than a recompile of that file and a relink.

This is true although I would argue that use of such techniques is 
unwise due because they impose a huge cost in errors that the compiler 
is unable to detect.  I agree with the general thrust of what Chris has 
been saying, and I don't think he is suggesting that such strategies 
are things that ocaml might emulate; I'm just stating for emphasis that 
this would be a bad thing.

>
> It does not appear that this is even possible now with ocamlopt.  That 
> is the problem I'm talking about.  In C++ you can give up some 
> features and get highly encapsulated code with minimal build 
> dependencies.  I don't think this is possible now with ocamlopt 
> regardless of whether you're willing to give up features or not, given 
> my understanding of the situation.  I would love to hear differently, 
> or have someone from the caml team comment on this.

I agree; I don't know of any such techniques but please correct that 
impression if it is incorrect.

>
>> Maybe.  C++ and Java are toy languages, then.
>
> This statement plays well on a mailing list for an alternative 
> language, but the reality is fairly different and most people writing 
> large software in C++ would write you off as a loon if you said this 
> to them.  There's an important difference between a sucky language and 
> a toy language.
>
> Still flabbergasted,
> Chris
>
> PS.  A related issue is going to come up with respect to disabling 
> cross-file inlining when we get native DLLs.  You want to be able to 
> control what gets made available for inlining when building a DLL, 
> since one of the uses of DLLs is to be able to supply a new version of 
> code and so you can't have it be already inlined in the client 
> application.
>
>
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: 
> http://caml.inria.fr
> Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: 
> http://caml.inria.fr/FAQ/
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>
>
-----------------------------------------------------------------
Seth Kurtzberg
CTO
ISEC Research and Network Operations Center
480-314-1540
888-879-5206
seth@isec.us
-----------------------------------------------------------------

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

[Caml-list] ocaml as *.so (was: ...and large development projects)

[Wolfgang Müller]

Hi,

> PS.  A related issue is going to come up with respect to disabling
> cross-file inlining when we get native DLLs.  You want to be able to
> control what gets made available for inlining when building a DLL, since
> one of the uses of DLLs is to be able to supply a new version of code and
> so you can't have it be already inlined in the client application.

I would like to rephrase the question: is it possible to use OCaml code (plus 
possibly some C code) for generating dynamically linked libraries that are 
used as easily as c(++) from c(++)? I've got the impression that this is the 
case. At least page 218 of the docs (chapter 17.7. at the end) suggests so. 

As I am currently deciding if I want to start a project in OCaml that possibly 
could become a library that would have to be used from C or JAVA, I would be 
very interested in the OCaml-Gurus' comment on that. 

Cheers,
Wolfgang

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

[Caml-list] Reading a file

[Siegfried Gonzi]

Hi:

Is there a better way in Ocaml to read a file line by line than via the 
read_line function?

I use read_line on a file, perform some tasks on this line and store the 
results in a list and after having red the file I use List.rev. The 
problem actually is on big files the function is awfully slow. As 
similar Clean function takes 15 seconds, my Bigloo program takes 25 
second and my C++ programs (via templates) takes 25 secondes but my 
Ocaml program takes 8 minutes.

I am not sure how quick List.rev actually is? In Bigloo reversing a list 
has more or less no overhead. My Bigloo function is similar to my OCaml 
function. Could it be that OCaml is that slow because I use "try and 
with" constructs in order to check for the end of a file?

Why my Clean function is that fast is incomprehensible for me. Does one 
know whether there exists a function in OCaml which converts a String to 
a character-list? I use this construct in Clean then in order to extract 
floating point numbers from that character list: ['1','.','2',...] and 
store this floating point numbers via pattern matching in my result-list.

S. Gonzi


>
>



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] Reading a file

[Mattias Waldau]

If the files are not of megabyte size, I just read 
them into a string. Very fast.

(** return the contents of filename as a string *)
let read_as_string filename =
  (* read at most len chars into string s, return the number of chars
read *)
  let rec my_input ic s ofs len =
    if len <= 0 then ofs else begin
      let r = input ic s ofs len in
      if r = 0
      then ofs
      else my_input ic s (ofs+r) (len-r)
    end in
  let ic = open_in_bin filename in
  let max_size = in_channel_length ic in
  let buf = String.create max_size in
  let read_chars = my_input ic buf 0 max_size in
  close_in ic;
  String.sub buf 0 read_chars ;;

> -----Original Message-----
> From: owner-caml-list@pauillac.inria.fr 
> [mailto:owner-caml-list@pauillac.inria.fr] On Behalf Of 
> Siegfried Gonzi
> Sent: den 20 maj 2003 10:43
> To: 'caml-list@inria.fr'
> Subject: [Caml-list] Reading a file
> 
> 
> Hi:
> 
> Is there a better way in Ocaml to read a file line by line 
> than via the 
> read_line function?
> 
> I use read_line on a file, perform some tasks on this line 
> and store the 
> results in a list and after having red the file I use List.rev. The 
> problem actually is on big files the function is awfully slow. As 
> similar Clean function takes 15 seconds, my Bigloo program takes 25 
> second and my C++ programs (via templates) takes 25 secondes but my 
> Ocaml program takes 8 minutes.
> 
> I am not sure how quick List.rev actually is? In Bigloo 
> reversing a list 
> has more or less no overhead. My Bigloo function is similar 
> to my OCaml 
> function. Could it be that OCaml is that slow because I use "try and 
> with" constructs in order to check for the end of a file?
> 
> Why my Clean function is that fast is incomprehensible for 
> me. Does one 
> know whether there exists a function in OCaml which converts 
> a String to 
> a character-list? I use this construct in Clean then in order 
> to extract 
> floating point numbers from that character list: 
> ['1','.','2',...] and 
> store this floating point numbers via pattern matching in my 
> result-list.
> 
> S. Gonzi
> 
> 
> >
> >
> 
> 
> 
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: 
http://caml.inria.fr Bug reports: http://caml.inria.fr/bin/caml-bugs
FAQ: http://caml.inria.fr/FAQ/ Beginner's list:
http://groups.yahoo.com/group/ocaml_beginners

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Nicolas Cannasse]

> Is there a better way in Ocaml to read a file line by line than via the
> read_line function?

You might want to use the ExtLib for this.
You can look at the CVS here :

http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/ocaml-lib/

The code you need :
-------------------------------
open ExtLib

let e = input_enum my_file in
let e = Enum.map my_conversion_function e in
List.of_enum e (* will return you the mapped list without any rev done *)
-------------------------------------
(You'll need the ExtList and the Enum modules for it)

> Does one
> know whether there exists a function in OCaml which converts a String to
> a character-list? I use this construct in Clean then in order to extract
> floating point numbers from that character list: ['1','.','2',...] and
> store this floating point numbers via pattern matching in my result-list.

you can use ExtString.enum to get a char Enumeration on your string.
It is more efficient then to use it as an Enum, since you don't create any
data structure to store the chars , but you still can get a list from it by
doing List.of_enum.

Nicolas Cannasse

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Markus Mottl]

Siegfried Gonzi schrieb am Dienstag, den 20. Mai 2003:
> Is there a better way in Ocaml to read a file line by line than via the 
> read_line function?

Yes: use "input_line"! "read_line" flushes stdout, which you'll probably
not need.

> I am not sure how quick List.rev actually is?

As quick as it can be...

This problem is most likely not related to List.rev.

> In Bigloo reversing a list 
> has more or less no overhead. My Bigloo function is similar to my OCaml 
> function. Could it be that OCaml is that slow because I use "try and 
> with" constructs in order to check for the end of a file?

If you create the exception handler within a loop, this will also be a bit
costly. The loop should be within the exception handler, not vice versa.

Regards,
Markus Mottl

-- 
Markus Mottl          http://www.oefai.at/~markus          markus@oefai.at

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Michal Moskal]

On Tue, May 20, 2003 at 10:42:44AM +0200, Siegfried Gonzi wrote:
> Hi:
> 
> Is there a better way in Ocaml to read a file line by line than via the 
> read_line function?
> 
> I use read_line on a file, perform some tasks on this line and store the 
> results in a list and after having red the file I use List.rev. The 
> problem actually is on big files the function is awfully slow. As 
> similar Clean function takes 15 seconds, my Bigloo program takes 25 
> second and my C++ programs (via templates) takes 25 secondes but my 
> Ocaml program takes 8 minutes.
                      ^^^^^^^^^^

[...]
> Why my Clean function is that fast is incomprehensible for me. Does one 
> know whether there exists a function in OCaml which converts a String to 
> a character-list? I use this construct in Clean then in order to extract 
> floating point numbers from that character list: ['1','.','2',...] and 
> store this floating point numbers via pattern matching in my result-list.

If you expand each line of megabyte file to list of characters -- it
cannot be fast.

On my box reading 32M file (800k lines) takes 3.4s with List.rev at the
end and 2.8s without. Reading the same file into array takes 2.3s. All
using ocamlopt, x86 linux, glibc 2.3.2. Doing the same thing in C
(using strdup() to remember lines) takes 0.7s (the C program is flawed
in some aspects, so don't take it too seriously).

-- 
: Michal Moskal :: http://www.kernel.pl/~malekith : GCS {C,UL}++++$ a? !tv
: PLD Linux ::::::::: Wroclaw University, CS Dept : {E-,w}-- {b++,e}>+++ h

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Siegfried Gonzi]

Michal Moskal wrote:

>
>>Why my Clean function is that fast is incomprehensible for me. Does one 
>>know whether there exists a function in OCaml which converts a String to 
>>a character-list? I use this construct in Clean then in order to extract 
>>floating point numbers from that character list: ['1','.','2',...] and 
>>store this floating point numbers via pattern matching in my result-list.
>>
>
>If you expand each line of megabyte file to list of characters -- it
>cannot be fast.
>

I will post the Clean snippet tomorrow. In the evening I will try other 
OCaml solutions. When I was first running my Clean program I couldn't 
believe it how fast it was; I was thinking of something like: "Clean is 
may be cheating me due to lazy evaluation, because as a return result I 
used the first list entry and I thought lazy evaluation just reads the 
first line of the 50MB file and nothing more". I did the test again but 
this time with estimating the length of the return list which ensures 
that the complete file becomes worked through.

Converting strings to list of chars is fast in Clean, as it seems. As 
far as I can remember I did the conversion ala:

[x\\x <-: y]

where y is the string (btw: '<-:' denotes "from array to list")

S. Gonzi


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Siegfried Gonzi]

Michal Moskal wrote:

>
>If you expand each line of megabyte file to list of characters -- it
>cannot be fast.
>
Enclosed the OCaml version in question:

'split' has been pinched up from comp.lang.functional. A year ago I had
a conversation there and someone posted this split function tailored to
my request: split "nil,2.23,3.34,nil" (-1.0) = [-1.0,2.23,3.34,-1.0]

'extractFloats' opens a file and applies split to every line and stores
the result into a list:

==
let split s c =
    let rec loop start acc =
 	try
  	  let next = String.index_from s start c in
  	  let substring = String.sub s start (next-start) in
  	  loop (next+1) (substring :: acc)
  	with
  	  Not_found ->
		let len = String.length s in
  		let substring = String.sub s start (len-start) in
  		List.rev (substring :: acc)
     in loop 0 []
;;


let frob userval s =
    match s with
    | "n/a" -> userval
    | "nil" -> userval
    | _ -> float_of_string s
;;

let extractFloats file del nanProxy =
  let rec readLoop i acc =
    try
      let line = input_line file in
      let floatL = List.map (frob nanProxy) (split line del) in
      readLoop  (i+1) (floatL :: acc)
    with
      End_of_file ->
	List.rev acc
  in
    readLoop 0 []
;;



let f = open_in "/home/gonzi/test.txt";;
let erg2 = extractFloats f ',' (-1.0);;
let rows = List.length erg2;;
rows;;
====

Enclosed also the Clean function. This version would be way more
readable than the Ocaml version. But I do not know how to translate it
to OCaml. My Clean function reads line after line and passes this
string-line on to RealsFromString. The latter function converts the
string-line to a char-list: [x\\x <-: string-line] and uses takeWhile,
toString, dropWhile and toReal in order to get the double numbers. As I
said the function is incredibly fast and takes for a 50MB file about 15
seconds.

Ocaml takes 8minutes. If I try to read the file line by line only
(without the conversion to double numbers) then Ocaml would take
 about 1 minutes. Where is the bottleneck here? List.map or what?

I think everybody has one specific task which he tries to implement in
every programming language he encounters. My specific task is this
floating-point extraction from string-files.

I didn't play around with different OCaml solutions, because I
had to play a bit with OCaml's Psilab implementation (if you need
something like Python+Numeric+Dislin you could give Psilab a try).

If you need the whole Clean program drop me a note. By the way: my
Scheme version is clumsy and is more or less similar to the OCaml
version. I wrote this verbose Scheme (Bigloo) version a year ago when I
was a beginner of Scheme. The performance of the Scheme (Bigloo) version
is about 30 seconds for this 50MB file and is therefore similar to the
C++-template version which takes about 30 seconds.

Oh yes: do not draw to close out a comment when I write "clumsy" which
implies OCaml is clumsy too; I have the strong believing that OCaml's
exception handling mechanism is more or less better than Clean's one
because Clean does not posses such a thing as exception handling, so to
speak. 

S. Gonzi

====
////////////////////////////////////////////////
// The dead as Latin functional language
// whith the most readable syntax out there
// and one of the /fastest functional languages/:
// Clean (In the meantime open(source?)
// for Linux/Unix). But as life plays:
// nobody jumps onto the Clean-bandwagon. Is this
// a pity or a bless? Why doesn't the "most"
// readable syntax plays a role in real life?
// Do not get me wrong, but why does always the
// "punctuation syntax" win in real life?
////////////////////////////////////////////////
FExtractReals:: HeaderKeys File -> [[Real]]
FExtractReals h file
		  | sfend file = []
		  # (line,nextline) = sfreadline file
		  = [(RealsFromString line h.del h.nan h.nanProxy) :
		     (FExtractReals h nextline)]



RealsFromString:: String Char String Real -> [Real]
RealsFromString line del nan nanProxy= searchDel [x\\x<-:line]
where
	searchDel:: [Char] -> [Real]
	searchDel [] = []
	searchDel linerest
		  # val = toString( takeWhile notDelNl linerest )
		  # rest = dropWhile ((<>)del) linerest
		  = [toRealNaN val nan : searchDel (drop 1 rest)]
	notDelNl::Char -> Bool
	notDelNl x
	       | x==del = False
	       | x==' ' = False
	       | x=='\t' = False
	       | x=='\n' = False
	       = True
	toRealNaN:: String String -> Real
	toRealNaN s nan
		  | s==nan = nanProxy
		  = toReal(s)
====





-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Siegfried Gonzi]

Siegfried Gonzi wrote:

> Michal Moskal wrote:
>
>>
>> If you expand each line of megabyte file to list of characters -- it
>> cannot be fast.
>>
> Enclosed the OCaml version in question:
>
> 'split' has been pinched up from comp.lang.functional. A year ago I had
> a conversation there and someone posted this split function tailored to
> my request: split "nil,2.23,3.34,nil" (-1.0) = [-1.0,2.23,3.34,-1.0] 


sorry:  split "nil,2.23,.3.34,nil" ','

delivers: ["nil","2.23","3.34","nil"]

I use then List.map (frob (-1.0)) ["nil","2.23","3.34","nil"]

in order to convert it to floating-point numbers.


S. Gonzi

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Siegfried Gonzi]

Siegfried Gonzi wrote:

>>
>> my request: split "nil,2.23,3.34,nil" (-1.0) = [-1.0,2.23,3.34,-1.0] 
>
>
>
> sorry:  split "nil,2.23,.3.34,nil" ',' 


This is one of the reasons why I do not get it that nobody annotates 
types in OCaml. If I had  to deal with a Clean function I would not have 
made the mistake of using split in a wrong manner, because:

split:: String Char -> [String]
split s c ....

but in OCaml:

split s c ...

I agree the compiler would rebel if I pass wrong types, however, this 
does not increase readability as opposed to Clean.

S. Gonzi

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Markus Mottl]

Siegfried Gonzi schrieb am Mittwoch, den 21. Mai 2003:
> This is one of the reasons why I do not get it that nobody annotates 
> types in OCaml. If I had  to deal with a Clean function I would not have 
> made the mistake of using split in a wrong manner, because:

But you can use type annotations in OCaml - it's optional! I suppose
that Clean, too, doesn't enforce explicit types for all functions,
but I am not sure.

> split:: String Char -> [String]
> split s c ....
> 
> but in OCaml:
> 
> split s c ...

You mean:

  let split s c = ...

But you can also write:

  let split (s : string) (c : char) : string list = ...

> I agree the compiler would rebel if I pass wrong types, however, this 
> does not increase readability as opposed to Clean.

It's really just a matter of practice: I never use type annotations,
because my functions usually have parameter names that speak for
themselves. Only library functions get an explicit interface, both for
abstraction and documentation - in a separate file.

Regards,
Markus Mottl

-- 
Markus Mottl          http://www.oefai.at/~markus          markus@oefai.at

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Eray Ozkural]

On Wednesday 21 May 2003 12:16, Markus Mottl wrote:
> It's really just a matter of practice: I never use type annotations,
> because my functions usually have parameter names that speak for
> themselves. Only library functions get an explicit interface, both for
> abstraction and documentation - in a separate file.

Err. Yes, but anybody who's used Haskell will know that those annotations can 
be handy when you're writing the code. They are harmless too, why don't we 
have such a syntax?

Thanks,

-- 
Eray Ozkural (exa) <erayo@cs.bilkent.edu.tr>
Comp. Sci. Dept., Bilkent University, Ankara  KDE Project: http://www.kde.org
www: http://www.cs.bilkent.edu.tr/~erayo  Malfunction: http://mp3.com/ariza
GPG public key fingerprint: 360C 852F 88B0 A745 F31B  EA0F 7C07 AE16 874D 539C

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[brogoff]

On Wed, 21 May 2003, Eray Ozkural wrote:
> Err. Yes, but anybody who's used Haskell will know that those annotations can 
> be handy when you're writing the code. They are harmless too, why don't we 
> have such a syntax?

It's more than that. In Haskell 98, using polymorphic recursion requires 
that you explicitly annotate the recursive function. Some other languages 
like Clean and Mercury use type inference. OCaml requires that you explicitly 
annotate the functions to get polymorphic recursion, whether you use explicit 
polymorphism of record fields, polymorphic methods, or the experimental 
recursive module feature. My current opinion is that explicit annotation is 
the right way. 

WRT your main point, I agree that having a separate signature close to the 
function is often more readable. In OCaml, the syntax is 

  let function_name : function_type = 
    fun arg0 arg1 ... argN -> 

and while I admit that I prefer the Haskell/Clean separate signatures, this 
is often good enough. I'd file this one under "petty complaint" for now. 
There are bigger fish to fry. 

-- Brian


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Michal Moskal]

On Wed, May 21, 2003 at 08:11:40AM +0200, Siegfried Gonzi wrote:
> let split s c =
>    let rec loop start acc =
> 	try
>  	  let next = String.index_from s start c in
>  	  let substring = String.sub s start (next-start) in
>  	  loop (next+1) (substring :: acc)
>  	with
>  	  Not_found ->
> 		let len = String.length s in
>  		let substring = String.sub s start (len-start) in
>  		List.rev (substring :: acc)
>     in loop 0 []
> ;;
> 
> 
> let frob userval s =
>    match s with
>    | "n/a" -> userval
>    | "nil" -> userval
>    | _ -> float_of_string s
> ;;
> 
> let extractFloats file del nanProxy =
>  let rec readLoop i acc =
>    try
>      let line = input_line file in
>      let floatL = List.map (frob nanProxy) (split line del) in
>      readLoop  (i+1) (floatL :: acc)
>    with
>      End_of_file ->
> 	List.rev acc
>  in
>    readLoop 0 []
> ;;

This function is *not* tail recursive. You should try something like
this:

let try_input file =
  try
    Some (input_line file)
  with None -> None

let extractFloats file del nanProxy =
 let rec readLoop i acc =
   match try_input file with
   | Some line ->
     let floatL = List.map (frob nanProxy) (split line del) in
     readLoop  (i+1) (floatL :: acc)
   | None ->
     List.rev acc
 in
   readLoop 0 []

This alone makes your program 2 times faster (maybe more for you data).

split (and List.map for that matter) function also ain't tail recursive,
but this shouldn't be the case if your lines aren't excessively long
(say over 1000 floats).

Next you should note that float lists are boxed, maybe use arrays?

Hope that helps.

-- 
: Michal Moskal :: http://www.kernel.pl/~malekith : GCS {C,UL}++++$ a? !tv
: PLD Linux ::::::::: Wroclaw University, CS Dept : {E-,w}-- {b++,e}>+++ h

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

[Caml-list] PsiLAB works fine under Linux SuSE 8

[Siegfried Gonzi]

Have you ever wondered whether PsiLAB works on your SuSE distribution? A 
year ago I failed to compile PsiLAB for SuSE 8. Had a conversation with 
the author but didn't request from him to ponder over my problem because 
I was just a lurker and it wouldn't pay off for him.

But yesterday I found out that the second pre-compiled PsiLAB version 
just /works/ fine under SuSE 8.

PsiLAB sounds interesting and is similar to Python+Numeric+DISLIN.

This was always on my wishlist to have something like PsiLAB for Clean, 
but as I wrote Clean is dead and the community is dead too (you normally 
do not get help or feedback from there).

Until now I used Bigloo and my self-written Bigloo binding to the high 
quality plotting library DISLIN.

Does one know whether it is possible to use the PsiLAB command-line 
interpreter in Emacs mode?

S. Gonzi

>



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Markus Mottl]

Siegfried Gonzi schrieb am Mittwoch, den 21. Mai 2003:
> Enclosed the OCaml version in question:
[snip]

There are some horrendously inefficient things in the code snippet
you sent, mostly related to lack of tail-recursion due to bad use of
exception handlers within loops.

Try this as a starting point:

---------------------------------------------------------------------------
let rec my_index_from_loop s len i c =
  if i >= len then -1
  else if String.unsafe_get s i = c then i
  else my_index_from_loop s len (i + 1) c
;;

let my_index_from s start c =
  my_index_from_loop s (String.length s) start c
;;

let split s c =
  let rec loop start acc =
    let next = my_index_from s start c in
    if next = -1 then
      let substring = String.sub s start (String.length s - start) in
      List.rev_append acc [substring]
    else
      let substring = String.sub s start (next - start) in
      loop (next + 1) (substring :: acc) in
  loop 0 []
;;

let frob userval = function
  | "n/a" | "nil" -> userval
  | s -> float_of_string s
;;

let extract_floats file del nan_proxy =
  let my_frob = frob nan_proxy in
  let rec loop acc =
    match
      try
        let line = input_line file in
        Some line
      with End_of_file -> None
    with
    | Some line -> loop (List.map my_frob (split line del) :: acc)
    | None -> List.rev acc in
  loop []
;;

let f = open_in "/home/gonzi/test.txt";;
let erg2 = extract_floats f ',' (-1.0);;
let rows = List.length erg2;;
rows;;
---------------------------------------------------------------------------

Regards,
Markus Mottl

-- 
Markus Mottl          http://www.oefai.at/~markus          markus@oefai.at

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Siegfried Gonzi]

Markus Mottl wrote:

>
>
>
Thanks for this version. Without any insult, it works well and takes for
this legendary file now 25 seconds, though, I find the version ugly. The
Clean, Bigloo, Fortran 95 and C++ version more or less takes the same time.

 One thing which surprised me most is the fact that reading this file
takes /only/ 40 seconds under PsiLAB. For all the innocent: PsiLAB is
based on OCaml 3.02 and is like an interpreter for scientific computing
ala Matlab or Python+Numeric+Dislin. Why you should try PsiLAB:

a) You are really using OCaml 3.02 with the goody of some important
linear algebra routines and a good plotting library (PLPLOT)

 b) As it seems very good performance

c) To go on: the syntax is not as crippled as Python's Numeric stuff.
I regard Franky D. and his team for fostering Numeric. I think it is
important to have one from the acknowledged LLNL who is using Python,
because the LLNL is high ranked and nobody will laugh at you when you
say: why are you using Matlab or IDL guy there are plenty of
alternatives out there. But the Numeric library syntax is crippled and
I never got used to. However, I think Python is overly complicated to master.


But there remains my requirement for a better OCaml solution. I mean we
are talking about functional programming. My C++ version based on
templates has some elegance too, though, it is more or less based on
this awkward "string indexing". However, t should be possible to walk up
Clean's way:

1) Convert the string-line to a character-list: [x\\x<-: string-line]
2) Use takeWhile until you encounter the seperator ','
3) Convert the string-number of 2) to a real number and store it in the list
4) Use dropWhile until you encounter ',' and go on with rest of the
string until you encounter the next occurence of ',' in point 2) 

This, oh dear Watson is what I mean with elegance. No need of string
indexing counting or other error prone stuff.

My Bigloo (Scheme) version does exactly the same
as my Clean version. Okay, it uses takeWhile and dropWhile which is not
standard Scheme, but the two functions can be easily implemenetd by a few
lines and will never spoil the "platform" independence of the code.
Biglo does exactly what I described in the aformentioned points. First
it converts the string to a character-list (standard Scheme function) and then
it goes on.

So, this must also be possible in OCaml - or must not?

S. Gonzi


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Markus Mottl]

Siegfried Gonzi schrieb am Donnerstag, den 22. Mai 2003:
> Thanks for this version. Without any insult, it works well and takes for
> this legendary file now 25 seconds, though, I find the version ugly. The
> Clean, Bigloo, Fortran 95 and C++ version more or less takes the same time.

There is always room for making things faster or shorter: it depends on
what you prefer.

If you want to have an even faster version, you could use float arrays
instead of float lists. E.g. instead of this line in the previous
solution:

  | Some line -> loop (List.map my_frob (split line del) :: acc)

Use this:

  | Some line ->
      let str_ar = Array.of_list (split line del) in
      loop (Array.map my_frob str_ar :: acc)

This is probably what you really need anyway.

If you want to have a short version, which is still pretty fast, use this:

---------------------------------------------------------------------------
let frob userval = function
  | "n/a" | "nil" -> userval
  | s -> float_of_string s
;;

let extract_floats file del nan_proxy =
  let rex = Str.regexp_string (String.make 1 del) in
  let my_frob = frob nan_proxy in
  let acc_ref = ref [] in
  (try
    while true do
      let line = input_line file in
      let str_ar = Array.of_list (Str.split rex line) in
      acc_ref := Array.map my_frob str_ar :: !acc_ref
    done
  with End_of_file -> ());
  List.rev !acc_ref
;;

let f = open_in "/home/gonzi/test.txt";;
let erg2 = extract_floats f ',' (-1.0);;
let rows = List.length erg2;;
rows;;
---------------------------------------------------------------------------

> But there remains my requirement for a better OCaml solution.

Better = faster or smaller...?

> 1) Convert the string-line to a character-list: [x\\x<-: string-line]

This would be _very_ inefficient.

> This, oh dear Watson is what I mean with elegance. No need of string
> indexing counting or other error prone stuff.

Use regular expressions: fast enough and very short.

> So, this must also be possible in OCaml - or must not?

You cannot just take one solution of one language and assume that it
is equally efficient or elegant in another language. This also heavily
depends on availability and functionality of libraries. E.g. there is no
"take_while" or "drop_while" in the OCaml standard library - you'd have
to write them yourself.

Regards,
Markus Mottl

-- 
Markus Mottl          http://www.oefai.at/~markus          markus@oefai.at

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Siegfried Gonzi]

Markus Mottl wrote:

>You cannot just take one solution of one language and assume that it
>is equally efficient or elegant in another language. This also heavily
>depends on availability and functionality of libraries. E.g. there is no
>"take_while" or "drop_while" in the OCaml standard library - you'd have
>to write them yourself.
>

I got this version and I think I like it best (except for this single 
semicolon; may be C could get some punctuation hell-ideas from Ocaml):

 let map_on_lines ic f =
|   let lines = ref [] in
|   begin try while true do
|     let l = input_line ic in
|     lines := l :: !lines
|   done 
|   with End_of_file -> close_in ic
|   end ;
|   List.rev_map f !lines
| 
| let extract_floats reg userval l =
|   List.map 
|     (function
|       | "n/a" | "nil" -> userval
|       | s -> float_of_string s)
|     (Str.split reg l)
| 
| let main = 
|   let ic = open_in "foo" in
|   map_on_lines ic 
|     (extract_floats (Str.regexp ",") (-1.0))


The problem what I am now faced with is as follows: how to use it. Do I 
have to link against a regex.o library? There is no Readme in the 
additional libraay folder of Ocaml how to use regex in Ocaml programs.

Regards,
S. Gonzi

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Reading a file

[Siegfried Gonzi]

Siegfried Gonzi wrote:

>
> I got this version and I think I like it best

==
| let map_on_lines ic f =
|   let lines = ref [] in
|   begin try while true do
|     let l = input_line ic in
|     lines := l :: !lines
|   done 
|   with End_of_file -> close_in ic
|   end ;
|   List.rev_map f !lines
| 
| let extract_floats reg userval l =
|   List.map 
|     (function
|       | "n/a" | "nil" -> userval
|       | s -> float_of_string s)
|     (Str.split reg l)
| 
| let main = 
|   let ic = open_in "foo" in
|   map_on_lines ic 
|     (extract_floats (Str.regexp ",") (-1.0))
==



It is due to Olivier Andrieu

S. Gonzi



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml as *.so (was: ...and large development projects)

[Nicolas Cannasse]

> > PS.  A related issue is going to come up with respect to disabling
> > cross-file inlining when we get native DLLs.  You want to be able to
> > control what gets made available for inlining when building a DLL, since
> > one of the uses of DLLs is to be able to supply a new version of code
and
> > so you can't have it be already inlined in the client application.
>
> I would like to rephrase the question: is it possible to use OCaml code
(plus
> possibly some C code) for generating dynamically linked libraries that are
> used as easily as c(++) from c(++)? I've got the impression that this is
the
> case. At least page 218 of the docs (chapter 17.7. at the end) suggests
so.

One of the best way to do what you want right now is to use ODLL (
http://tech.motion-twin.com/odll ). This will give you a good idea of how
you can handle it.

Nicolas Cannasse

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml as *.so (was: ...and large development projects)

[Wolfgang Müller]

> One of the best way to do what you want right now is to use ODLL (
> http://tech.motion-twin.com/odll ). This will give you a good idea of how
> you can handle it.

Looks great! Thanks! But this is for Windows, right? Do you think there are 
any obstacles in porting it to Linux (i.e. I would not be able to spend much 
time on it soon)?

Cheers,
Wolfgang

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml as *.so (was: ...and large development projects)

[Nicolas Cannasse]

> > One of the best way to do what you want right now is to use ODLL (
> > http://tech.motion-twin.com/odll ). This will give you a good idea of
how
> > you can handle it.
>
> Looks great! Thanks! But this is for Windows, right? Do you think there
are
> any obstacles in porting it to Linux (i.e. I would not be able to spend
much
> time on it soon)?

Yes, right now only windows DLL can be produced, but I think with some minor
changes you can extend ODLL to make it works with GCC, since this is only a
matter of "calling the good command with the good arguments". If you're
interested in modifying it , I'll be happy to answer your questions and to
include the diffs in the distribution.

Nicolas Cannasse

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml as *.so (was: ...and large development projects)

[Wolfgang Müller]

> Yes, right now only windows DLL can be produced, but I think with some
> minor changes you can extend ODLL to make it works with GCC, since this is
> only a matter of "calling the good command with the good arguments". If
> you're interested in modifying it , I'll be happy to answer your questions
> and to include the diffs in the distribution.
>
> Nicolas Cannasse

OK. Great. So I can start doing my code then. When I use/extend ODLL, I will 
let you know. 

Cheers,
Wolfgang

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml as *.so (was: ...and large development projects)

[John Max Skaller]

Wolfgang Müller wrote:

 
> I would like to rephrase the question: is it possible to use OCaml code (plus 
> possibly some C code) for generating dynamically linked libraries 


No. Not yet :-)

-- 
John Max Skaller, mailto:skaller@ozemail.com.au
snail:10/1 Toxteth Rd, Glebe, NSW 2037, Australia.
voice:61-2-9660-0850


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml as *.so

[Remi Vanicat]

John Max Skaller <skaller@ozemail.com.au> writes:

> Wolfgang Müller wrote:
>
>  
>> I would like to rephrase the question: is it possible to use OCaml
>> code (plus possibly some C code) for generating dynamically linked
>> libraries
>
>
> No. Not yet :-)

Their is odll (http://tech.motion-twin.com/) that could help (never
try it, but it seem it just do that).

-- 
Rémi Vanicat
vanicat@labri.u-bordeaux.fr
http://dept-info.labri.u-bordeaux.fr/~vanicat

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[John Max Skaller]

Chris Hecker wrote:


>> Maybe.  C++ and Java are toy languages, then.
> 
> 
> This statement plays well on a mailing list for an alternative language, 
> but the reality is fairly different and most people writing large 

> software in C++ would write you off as a loon if you said this to them.  


	With due respect: most software houses emply
smart people that know no computer science. One can blame
universities for not teaching it I suppose.

	These software houses insist on using bad tools
all the time, they insist on using engineering methodology
entirely inappropriate for software development, they DARE
to rant and rave about how good object orientation is, and then
organise their people in heirarchical teams.

	Basically, they're total morons on an organisational
level, even if the individiuals are quite smart.

	Unfortunately one must cry, that these people
are well paid to rip off investors, consumers, taxpayers,
and just about everyone else. Most haven't even heard
of functional programming.

	In other words, their opinion about
separate compilation just doesn't count for anything
in my books: why would I listen to some idiot that
thinks Java is a good language?


	In my opinion, the major weakness of
Ocaml industrially is that one cannot currently generate
shared libraries. (Interfacing with C/C++ is also problematic
but possible).

	The reason is that the modern separate
compilation model *enforced* my operating system
technology, and used extensively for large scale projects,
is not separate compilation and static linkage into a program,
but building of shared libraries: Ocaml(opt) is targetting the wrong
kind of object, no one is interested in programs these days
(they're built with scripting languages). They're interested
in reusable binary components.

	I can't argue if this is right or wrong, but it
is a fact that that is what people need to build.
The Ocaml team surely knows this and is working towards
implementing it (witness preparatory work on linking
C parts of Ocaml as shared libraries).

-- 
John Max Skaller, mailto:skaller@ozemail.com.au
snail:10/1 Toxteth Rd, Glebe, NSW 2037, Australia.
voice:61-2-9660-0850


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[John Max Skaller]

Chris Hecker wrote:

> 
>> > Uh, isn't this a major flaw in the compiler?  Why does this happen?
>> Inlining.  Inlining is an important optimization, especially for
>> functional languages, where many functions are typically short.
> 
> 
> Sure, but there has to be a better way to do this than forcing a 
> recompile of the entire program when you change a debug print in your 
> lowest level completely encapsulated library.  That's insanely bad 
> software engineering, and this is a huge issue.  


No it isn't. With due respect, Chris, you come
from the C++ world where compiles take an hour,
the NFS service is screwed, and everyone is
alway rebuilding everying in a totally
unmodular fashion.

So you are expecting poor performance where it
simply doesn't exist.

This is not C++ where you #include the whole
of the standard library and half the rest
of the world in every small translation
unit and have to recompile the interface
from TEXT every time.

This is Ocaml where you can selectively include
module interfaces which have been already compiled.
So compiles of tranlsation units (ml files)
are very very fast because 90% of the work
a C++ compiler does is recompile the same thing
over and over again ......

Added to that Ocaml has a simpler language
structure, and doesn't need to instantiate
10000 templates ..

Trust me. Don't worry about it!
The existing compiler performance optimisations
and separate compilation model have superb performance
from an engineering viewpoint (if there is a weakness
it is in the unexpected slowness of inference of
some nasty types, which occasionally bites you)


-- 
John Max Skaller, mailto:skaller@ozemail.com.au
snail:10/1 Toxteth Rd, Glebe, NSW 2037, Australia.
voice:61-2-9660-0850


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[John Max Skaller]

Chris Hecker wrote:

> 
>> One issue to be aware of: if you build native code, when you change a 
>> module, then all dependent modules must be recompiled.  This can be 
>> surprising when a small change to a utility module causes a near full 
>> recompile of the project.  The byte-code compiler does not have this 
>> issue, and is sufficient for testing in most cases.
> 
> 
> Is this true?  The cmi file for the interface doesn't insulate you from 
> this in native code?  


Yes its true, but it should rarely be an issue.
I personally have a sequence of compiles:

m1.mli
m2.mli
..
mn.mli
m1.ml
m2.ml
..
mn.ml

and I simply start at the first needed place
and recompile everything afterwards. I have something
like 20 modules, and a 550 MHz PIII, and the ONLY
program that compiles so slowly I see any delay
is ocamllex (which is REALLY slow).


In other words, don't worry about it. Ocaml is FAST.
Don't try to finess building only required files,
it will take longer for (** expletive deleted **)
make and friends to find out which modules to
compile than to actually compile the lot.

I actually use a Python script, and I suspect the
timestamp comparisons needed take about the same
time as an actual compile .. I left checking
in to eliminate the need to rebuild the lexer
and parser (run ocamllex and ocamlyacc).

Which is all long winded praise to the
Ocaml team for producing a product that
not only generates fast code .. it generates it fast :-)

-- 
John Max Skaller, mailto:skaller@ozemail.com.au
snail:10/1 Toxteth Rd, Glebe, NSW 2037, Australia.
voice:61-2-9660-0850


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Chris Hecker]

>In other words, don't worry about it. Ocaml is FAST. [...]
>So you are expecting poor performance where it
>simply doesn't exist.

Uh, I must have missed it when you stopped by my office and came to this 
conclusion from looking at actual data from my project instead of just 
asserting it on a mailing list.  :)  Another example of 
You-Don't-Need-This-itis.

Actually, ocamlopt compile speeds are already a problem for me, and my 
project is only about 20% done, LOC-wise.  This is a small problem for me 
now that's going to be a huge problem soon enough unless I can get it 
fixed.  Perhaps I have a lower process-overhead tolerance than you.  I've 
already had to change my project around so only the files that need it use 
camlp4 (for streams) because it was too slow.  I was counting on ocamlopt 
working "correctly" and allowing this decoupling as well, and again, I'm 
amazed it doesn't.  I'm hoping that the ocaml team takes this seriously and 
fixes it soon (although no one from Inria has responded to this thread, 
which worries me).

>[lots of ranting about software houses snipped]

In spite of these generalizations about developers and organizations, it is 
not hard in C++ to decouple a large project.  All C++ compilers support 
this "feature".  It's done every day in large projects, and it mostly 
works.  It appears impossible with ocamlopt given the current state of the 
compiler.  There is a fundamental and important difference between "often 
people don't do it in C++ and John thinks they're stupid" and "it is 
physically impossible to do so in ocaml".

Anyway, back to the problem:  it seems like the hacked way to enable this 
is to checksum the cmi and compare the new one to the old one and if they 
match then don't update the new one during cmx compilation.  That, coupled 
with a way to disable code inlining would allow you to control decoupling 
fairly well.  Unless I'm missing something; I haven't looked into this yet 
(we had a baby :).

Chris


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Michal Moskal]

On Mon, Jun 02, 2003 at 05:31:46PM -0700, Chris Hecker wrote:
> Anyway, back to the problem:  it seems like the hacked way to enable this 
> is to checksum the cmi and compare the new one to the old one and if they 
> match then don't update the new one during cmx compilation.  That, coupled 
> with a way to disable code inlining would allow you to control decoupling 
> fairly well.  Unless I'm missing something; I haven't looked into this yet 
> (we had a baby :).

Maybe try simple shell script to do that? Something like
move-if-changed. This should work, and you could report how much it
helps.

-- 
: Michal Moskal :: http://www.kernel.pl/~malekith : GCS {C,UL}++++$ a? !tv
: PLD Linux ::::::::: Wroclaw University, CS Dept : {E-,w}-- {b++,e}>+++ h

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Chris Hecker]

>Maybe try simple shell script to do that? Something like
>move-if-changed. This should work, and you could report how much it
>helps.

I think the cmi/cmx files have a checksum/timestamp already in them that's 
unique to a compile, so the naive external checksum won't work (this is 
supposition on my part, though, based on the error messages since I haven't 
looked at it yet).  I think it'd have to be smart about things, and at that 
point it might be easier just to put it in the compiler to test it.

Chris

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[art yerkes]

On Mon, 02 Jun 2003 17:31:46 -0700
Chris Hecker <checker@d6.com> wrote:

> 
> >In other words, don't worry about it. Ocaml is FAST. [...]
> >So you are expecting poor performance where it
> >simply doesn't exist.
> 
> Uh, I must have missed it when you stopped by my office and came to this 
> conclusion from looking at actual data from my project instead of just 
> asserting it on a mailing list.  :)  Another example of 
> You-Don't-Need-This-itis.
> 
> Actually, ocamlopt compile speeds are already a problem for me, and my 
> project is only about 20% done, LOC-wise.  This is a small problem for me 
> now that's going to be a huge problem soon enough unless I can get it 
> fixed.  Perhaps I have a lower process-overhead tolerance than you.  I've 
> already had to change my project around so only the files that need it use 
> camlp4 (for streams) because it was too slow.  I was counting on ocamlopt 
> working "correctly" and allowing this decoupling as well, and again, I'm 
> amazed it doesn't.  I'm hoping that the ocaml team takes this seriously and 
> fixes it soon (although no one from Inria has responded to this thread, 
> which worries me).
> 
> >[lots of ranting about software houses snipped]
> 
> In spite of these generalizations about developers and organizations, it is 
> not hard in C++ to decouple a large project.  All C++ compilers support 
> this "feature".  It's done every day in large projects, and it mostly 
> works.  It appears impossible with ocamlopt given the current state of the 
> compiler.  There is a fundamental and important difference between "often 
> people don't do it in C++ and John thinks they're stupid" and "it is 
> physically impossible to do so in ocaml".
> 
> Anyway, back to the problem:  it seems like the hacked way to enable this 
> is to checksum the cmi and compare the new one to the old one and if they 
> match then don't update the new one during cmx compilation.  That, coupled 
> with a way to disable code inlining would allow you to control decoupling 
> fairly well.  Unless I'm missing something; I haven't looked into this yet 
> (we had a baby :).
> 
> Chris

Have you thought of trying to use Gerd Stolpmann's Dl package?  Although that's
not exactly its purpose you'll be able to use it to implement each module in
a separate shared library.  It's fairly trivial to adapt it to work on win32.

In order to generate the loader for each part, you use:

ocamldlgen original_implementation.ml implementing_library.dll > stub_impl.ml

And build your bits as dlls with something like:

ocamlopt -pack <*.cmx> -output-obj -o packed.obj
link /dll /out:implementing_library.dll packed.obj $(LIBS)

This would allow you to separate your project into as many separately compiled
subparts as you want without sacrificing inlining within the parts.

-- 
`No, you don't understand,' the Knight said, looking a little vexed. 
`That's what the name is called. The name really is "The Aged Aged 
Man."'
-- Lewis Carroll

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Jason Hickey]

Chris Hecker wrote:
> Actually, ocamlopt compile speeds are already a problem for me, and my 
> project is only about 20% done, LOC-wise.

I've been following this thread only vaguely.  To summarize, minor 
changes to a utility file can cause a project recompile with ocamlopt. 
For large projects this can take a long time.  For instance, it takes 
3-10 min for the projects I work on.  Too long to twiddle thumbs, but 
not long enough to go get coffee.

This is not an issue for the byte-code compiler, ocamlc.  Can't you use 
that?  It should be fairly easy to add byte-code support to your build 
system.  It is quite straightforward if you use make, for example.  For 
us, byte-code performance is usually within a factor of 3 or so of 
native code, which is perfectly fine for testing.

This is not really an issue with the design of ocaml, I would say.  It 
is just how inlining works.  One easy design solution might be to have 
an option to mark .cmx files with a -noinline flag.  Then you would 
partition your project into parts, disabling inlining across the parts 
with something like "ocamlopt -pack -noinline".  Dependencies would 
propagate across the parts only if the interface changes.  The OCaml 
designers would have to weigh in on some scheme like this.

You could consider our omake build system (http://mojave.caltech.edu) if 
you want to use MD5 digests for dependency analysis, but in general that 
will not solve the dependency problem.

Congrats on the new baby!  We just had a new baby too.

Jason

-- 
Jason Hickey                  http://www.cs.caltech.edu/~jyh
Caltech Computer Science      Tel: 626-395-6568 FAX: 626-792-4257

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[Chris Hecker]

>This is not an issue for the byte-code compiler, ocamlc.  Can't you use that?

This was covered earlier.  My project is a game which has "real time" 
feedback loops and it just isn't runnable on the bytecode 
compiler.  There's more detail in the archive.

The noinline workaround would be fine with me, as I've said before.  Or, 
better yet, some kind of checksum thing that also handles inlined functions 
and knows when they've changed as well.  Both would be the best, because 
with the former you could completely control when you get a rebuild, and 
with the latter you can minimize them with no performance penalty.  Both 
useful tools to have at your disposal.

>Congrats on the new baby!  We just had a new baby too.

Thanks, and likewise!  Even the power of functional programming has yet to 
make me productive in the face of this.  :)

Chris


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] ocaml and large development projects

[John Max Skaller]

Jason Hickey wrote:

> Chris Hecker wrote:
> 
>> Actually, ocamlopt compile speeds are already a problem for me, and my 
>> project is only about 20% done, LOC-wise.
> 
> 
> I've been following this thread only vaguely.  To summarize, minor 
> changes to a utility file can cause a project recompile with ocamlopt. 
> For large projects this can take a long time.  For instance, it takes 
> 3-10 min for the projects I work on.  Too long to twiddle thumbs, but 
> not long enough to go get coffee.


The solution is obvious. Install a faster coffee maker :-)



-- 
John Max Skaller, mailto:skaller@ozemail.com.au
snail:10/1 Toxteth Rd, Glebe, NSW 2037, Australia.
voice:61-2-9660-0850


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners