[Caml-list] A few questions regarding the compiler

[Caml-list] A few questions regarding the compiler

[Marcin 'Qrczak' Kowalczyk]

I'm using OCaml as the target language of my compiler of a custom
language, because the OCaml compiler has good quality and the OCaml
language has appropriate runtime characteristics for my language
(regarding side effects, modules, recursion, memory management etc.).

1. Is it possible to setup a custom handler / printer of exceptions
   raised at the toplevel? The language defines how to convert objects
   to strings but of cource OCaml doesn't know that when printing
   exceptions.

   I can ignore the problem. I can also let my compiler wrap each
   toplevel binding which can fail. But perhaps it can be done easier.

2. Why ocamlc & ocamlopt are compiled to bytecode and ocamlc.opt &
   ocamlopt.opt have these names instead of being the default?

   I understand that the native code compiler is not as portable as the
   bytecode compiler. What should be a good policy for using one of the
   four versions of the compiler? I think the -O switch of my compiler
   will change from ocamlc/ocamlc.opt to ocamlopt/ocamlopt.opt,
   with some unknown yet way to choose between * and *.opt - probably
   a configuration file which sets program names (commandline switch
   is not good, I don't need makefiles for my language - the compiler
   manages dependencies itself).

3. I will want to have an interactive interpreter for my language,
   in addition to a compiler of standalone executables.

   I think it should be possible to start ocaml (or a custom toplevel)
   in a subprocess and talk with it through a pipe, giving it
   declarations compiled on the fly. How to suppress printing the
   prompt and the resulting value and type?

   Is there a better way? This one probably won't work on Windows.

4. Am I right that modules are executed exactly in the order provided
   during linking, and that the only requirement is that each module
   is listed after those it depends on?

-- 
 __("<  Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/
 \__/
  ^^
QRCZAK

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] A few questions regarding the compiler

[Xavier Leroy]

> 1. Is it possible to setup a custom handler / printer of exceptions
>    raised at the toplevel? The language defines how to convert objects
>    to strings but of cource OCaml doesn't know that when printing
>    exceptions.

In 3.03 alpha and in the forthcoming 3.03, additional "hooks" were
added to the toplevel so that it can call user-provided functions for
displaying evaluation results, including uncaught exceptions.  The
Camlp4 preprocessor makes good use of this new feature.

> 2. Why ocamlc & ocamlopt are compiled to bytecode and ocamlc.opt &
>    ocamlopt.opt have these names instead of being the default?

Largely because the native-code compiler is not always available.
Also, the main purpose of the .opt compilers is to test the ocamlopt
compiler by bootstrapping it.  Unless your source code is very large,
or your machine very slow, the regular ocamlc and ocamlopt are
reasonably fast already.

> 3. I will want to have an interactive interpreter for my language,
>    in addition to a compiler of standalone executables.
> 
>    I think it should be possible to start ocaml (or a custom toplevel)
>    in a subprocess and talk with it through a pipe, giving it
>    declarations compiled on the fly.

It can be done this way indeed, although again Camlp4 provides another
alternative where your preprocessor and the Caml toplevel live in the
same process.

> How to suppress printing the
> prompt and the resulting value and type?

Not easily.  You could link with the toplevel and call
"Toploop.use_silently" to execute code from a file, suppressing the
output.

>    Is there a better way? This one probably won't work on Windows.

The pipe thing can be made to work under Windows, however there is an
issue with interrupting a looping inferior OCaml process.  We're
working on this for 3.03.

> 4. Am I right that modules are executed exactly in the order provided
>    during linking, and that the only requirement is that each module
>    is listed after those it depends on?

Yes, exactly.

- Xavier Leroy
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] A few questions regarding the compiler

[Marcin 'Qrczak' Kowalczyk]

Sun, 11 Nov 2001 18:38:30 +0100, Xavier Leroy <xavier.leroy@inria.fr> pisze:

> In 3.03 alpha and in the forthcoming 3.03, additional "hooks" were
> added to the toplevel so that it can call user-provided functions for
> displaying evaluation results, including uncaught exceptions.  The
> Camlp4 preprocessor makes good use of this new feature.

Only toplevel? I would like to have it in standalone executables.
It doesn't have high priority though - often exceptions are defined
in OCaml, wrapped in objects of my language only when catching,
reraised in their original form, so then they are printed reasonably.

I was thinking about backtraces. It would be very helpful because the
compiled language is dynamically typed so errors are often catched
at runtime. Unfortunately translating an OCaml backtrace to something
related to source positions would be very tricky. It would even have
to be pasted manually to an external tool because the program proper
doesn't have access to its backtrace when it fails - it's just printed
to stderr when it's too late.

>>    I think it should be possible to start ocaml (or a custom
>>    toplevel) in a subprocess and talk with it through a pipe,
>>    giving it declarations compiled on the fly.
> 
> It can be done this way indeed, although again Camlp4 provides
> another alternative where your preprocessor and the Caml toplevel
> live in the same process.

I don't see how Camlp4 could help. The compiler is not written in
OCaml, it only generates OCaml output.

> Not easily.  You could link with the toplevel and call
> "Toploop.use_silently" to execute code from a file, suppressing
> the output.

I looked closer and realized that it's quite impossible to plug into
toplevellib something other than the default Topmain, although it's
very close to being possible.

For example the Clflags module is not exported (i.e. clflags.cmi
is not installed) so I can't set any options. Initialization is
hardwired into Toploop.loop, but calling Toploop.loop is too much:
it reads from stdin (which must be available to the program so it
can't be /dev/null) and writes to stdout. Initializing by hand is
not possible because the relevant modules are not exported.

I don't want to have to compile a custom OCaml version - it should
work with whatever is installed with OCaml. Unfortunately now it's
impossible.

Another question. Low-level libraries for my language contain many
embedded OCaml snippets written by hand, so they often contain errors.
Is it possible to tell the OCaml compiler about original source
positions for parts of the program, so it reports them in error
messages? I mean like '# line_no "filename"' in cpp. Maybe Camlp4
could help?

Yet another question. OCaml doesn't accept an identifier as the
rhs of let rec (an identifier defined earlier). Why? I had to make
a little workaround to not generate such bindings. I guess it's the
only difference between "what is accepted in let rec" and "what allows
generalization of free type variables"?

-- 
 __("<  Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/
 \__/
  ^^
QRCZAK

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] A few questions regarding the compiler

[Patrick M Doane]

On Sun, 11 Nov 2001, Marcin 'Qrczak' Kowalczyk wrote:

> Sun, 11 Nov 2001 18:38:30 +0100, Xavier Leroy <xavier.leroy@inria.fr> pisze:
>
> > Not easily.  You could link with the toplevel and call
> > "Toploop.use_silently" to execute code from a file, suppressing
> > the output.
>
> I looked closer and realized that it's quite impossible to plug into
> toplevellib something other than the default Topmain, although it's
> very close to being possible.
>
> For example the Clflags module is not exported (i.e. clflags.cmi
> is not installed) so I can't set any options. Initialization is
> hardwired into Toploop.loop, but calling Toploop.loop is too much:
> it reads from stdin (which must be available to the program so it
> can't be /dev/null) and writes to stdout. Initializing by hand is
> not possible because the relevant modules are not exported.

See my implementation of Fort which works around some of these issues. For
example, initialization is also hardwired into "run_script" so we can it:

  let save_arg = Sys.argv.(0) in
  Sys.argv.(0) <- "__dummy";
  let ignore_formatter =
    Format.make_formatter (fun _ _ _ -> ()) (fun _ -> ()) in
  ignore (Toploop.run_script ignore_formatter "__dummy" Sys.argv);
  Sys.argv.(0) <- save_arg

> I don't want to have to compile a custom OCaml version - it should
> work with whatever is installed with OCaml. Unfortunately now it's
> impossible.

I agree that the top-level interface should be improved. I've been able to
find workarounds so far, but it sounds like you have some issues that
require some definite changes.

Another gripe of mine is that Topdirs.dir_load does not print good
diagnostic information when it fails, so in practice I create a dummy
script containing a single "#load" directive to get the error messages to
display properly.

Patrick

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] A few questions regarding the compiler

[Xavier Leroy]

> > In 3.03 alpha and in the forthcoming 3.03, additional "hooks" were
> > added to the toplevel so that it can call user-provided functions for
> > displaying evaluation results, including uncaught exceptions.  The
> > Camlp4 preprocessor makes good use of this new feature.
> 
> Only toplevel? I would like to have it in standalone executables.

For standalone executables, it is very easy to wrap your generated
program in a catch-all "try...with" construct, and print the escaping
exception as you see fit.

> I don't see how Camlp4 could help. The compiler is not written in
> OCaml, it only generates OCaml output.

Camlp4 (among other features) provides an easy way to construct OCaml
abstract syntax trees and have then compiled or executed by the OCaml
compilers or toplevel.  It exploits a number of hooks in the compilers
and toplevel to achieve fairly tight integration with them.  So, I'm
pretty sure it could help, provided your compiler is written in OCaml
indeed.

> [toplevel interface not doing what you want]
> I don't want to have to compile a custom OCaml version - it should
> work with whatever is installed with OCaml.

Methinks you're asking for too much here :-)

> Another question. Low-level libraries for my language contain many
> embedded OCaml snippets written by hand, so they often contain errors.
> Is it possible to tell the OCaml compiler about original source
> positions for parts of the program, so it reports them in error
> messages? I mean like '# line_no "filename"' in cpp. Maybe Camlp4
> could help?

The ocamlc and ocamlopt compilers honor '# lineno "filename"'
directives.

> Yet another question. OCaml doesn't accept an identifier as the
> rhs of let rec (an identifier defined earlier). Why? I had to make
> a little workaround to not generate such bindings. I guess it's the
> only difference between "what is accepted in let rec" and "what allows
> generalization of free type variables"?

IIRC, the reason was that this offered no increase in expressive
power, yet would require a nasty special case in the compilation
scheme for "let rec" definitions.

- Xavier Leroy
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] A few questions regarding the compiler

[Marcin 'Qrczak' Kowalczyk]

Tue, 13 Nov 2001 21:36:57 +0100, Xavier Leroy <xavier.leroy@inria.fr> pisze:

> For standalone executables, it is very easy to wrap your generated
> program in a catch-all "try...with" construct, and print the escaping
> exception as you see fit.

Then it would be impossible to dump the backtrace. A backtrace
which shows OCaml source positions might be better than nothing.

Well, I could reraise the exception after printing, but then it would
be printed twice, formatted by me and by OCaml, which is a bit silly
especially when there is no backtrace (the usual case).

It would be ideal to be able to obtain the backtrace programmatically
after I catch an exception. I could even think about translating
positions to real source in this case. Is that possible?

> Camlp4 (among other features) provides an easy way to construct OCaml
> abstract syntax trees and have then compiled or executed by the OCaml
> compilers or toplevel.  It exploits a number of hooks in the compilers
> and toplevel to achieve fairly tight integration with them.  So, I'm
> pretty sure it could help, provided your compiler is written in OCaml
> indeed.

It will be "written" in OCaml after I rewrite it in itself. It's
not going to happen soon, but it would not be much of work. The
gap between being so small to be easily implemented and so big to
implement something is shrinking as I'm completing the library.

>> [toplevel interface not doing what you want]
>> I don't want to have to compile a custom OCaml version - it should
>> work with whatever is installed with OCaml.
> 
> Methinks you're asking for too much here :-)

It's so close... Why else would topmain.cmo be separate? Don't say
that it's because the linker would otherwise throw the library out,
treating it as unused. I know that it's separate for being able to
replace topmain with a customized version :-)

The only problem is that some key features are not exported.

> The ocamlc and ocamlopt compilers honor '# lineno "filename"'
> directives.

Ah, I could have checked before asking.

BTW, I think it's not quite true that special-casing the representation
of float arrays while pretending it's the same type has little cost.
Almost all functions in the Array module do a check in their inner
loop. Since code generated by my compiler uses arrays very often,
I don't call functions like Array.of_list but use inline loops or
specialized functions.

Besides float arrays, I see that writing to an array whose element
type is statically known to be always an immediate word is faster than
the other case which must call 'modify'. Are such details described
somewhere?

I guess that it's fine to initialize an array of non-floats with
'Obj.magic 0' even if the type doesn't have nullary constructors,
provided that I overwrite elements with valid objects before use?
I don't always have a proper element at hand. Is it true that the
Array module cares to always find a valid element for initialization
only because of float arrays? And is it true that 'Obj.magic 0' can be
sometimes more efficient than a pointer because some case in make_vect
uses 'initialize(&Field(res, i), init)' instead of 'Field(res, i)
= init' in its inner loop? Or slightly less efficient because later
initialization must do one round more... Why are small arrays and
large arrays treated differently?

-- 
 __("<  Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/
 \__/
  ^^
QRCZAK

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] A few questions regarding the compiler

[Markus Mottl]

On Wed, 14 Nov 2001, Marcin 'Qrczak' Kowalczyk wrote:
> I guess that it's fine to initialize an array of non-floats with
> 'Obj.magic 0' even if the type doesn't have nullary constructors,
> provided that I overwrite elements with valid objects before use?

Yes, this is safe for non-float arrays. The GC will only interpret some
array element as a pointer to follow if it is not tagged as integer
(or nullary constructor, which is the same representation). I abuse this
"feature" in my "Res"-module to implement automatically resizing arrays.

> I don't always have a proper element at hand. Is it true that the
> Array module cares to always find a valid element for initialization
> only because of float arrays?

There is no function that allows you to create an array without passing
an initial element in the Array-module. It would be nice if there
were a function that allows one to create uninitialized, unboxed float
arrays. This is easy to implement and could be useful for efficiency
purposes in some occasions. It's, of course, not possible for other
kinds of arrays, because the memory region might contain things that
look like a pointer to the GC.

Btw., wouldn't it be possible (or is this actually implemented - haven't
looked) that the GC allocates arrays of integers always within the same
region of memory? Then it could reuse this region as a pool for creating
uninitialized integer arrays. Only if there is not enough memory, more
would have to be allocated and initialized.

> Why are small arrays and large arrays treated differently?

I think this is part of the GC-strategy to gain efficiency. The heuristics
is that large arrays (or other large chunks of memory) are not likely
to be short-lived.

I hope my explanations were not too far from the truth... ;)

Regards,
Markus Mottl

-- 
Markus Mottl                                             markus@oefai.at
Austrian Research Institute
for Artificial Intelligence                  http://www.oefai.at/~markus
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] A few questions regarding the compiler

[David Mentre]

"Marcin 'Qrczak' Kowalczyk" <qrczak@knm.org.pl> writes:

> It would be ideal to be able to obtain the backtrace programmatically
> after I catch an exception.

Maybe I have missed something but the point of raising an exception and
getting its location has already been discussed on the list:
http://caml.inria.fr/archives/200007/msg00028.html (look also at the thread)

To generate a backtrace, just do an "assert(false)" with the environment
variable OCAMLRUNPARAM set to "b=1" (without quotes, see man ocamlrun).

> I could even think about translating positions to real source in this
> case. Is that possible?

Yes. Somebody has written a tool to convert a trace in source position:
http://caml.inria.fr/archives/200109/msg00111.html

Best regards,
d.
-- 
 David.Mentre@inria.fr
 Opinions expressed here are only mine.
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] A few questions regarding the compiler

[Marcin 'Qrczak' Kowalczyk]

14 Nov 2001 14:20:37 +0100, David Mentre <David.Mentre@inria.fr> pisze:

> Maybe I have missed something but the point of raising an exception and
> getting its location has already been discussed on the list:
> http://caml.inria.fr/archives/200007/msg00028.html (look also at the thread)
> 
> To generate a backtrace, just do an "assert(false)" with the environment
> variable OCAMLRUNPARAM set to "b=1" (without quotes, see man ocamlrun).

I know, but the backtrace is just dumped to stderr just before the
program dies. I would like the program itself to have access to its
own backtrace, so it can do something with it instead of displaying
it in the default format and dying.

I can wrap each top-level value binding which can fail in a 'try
... with exn -> my_handler exn' (i.e. I let my compiler generate such
code), where my_handler does some magic to access the backtrace and
translates it to something readable. I'll now try to do it myself
basing on backtrace.c, but I don't understand yet how it works.

A more convenient way would be to call some magic hook which will
install my_handler for unhandled exceptions instead of the builtin
fatal_uncaught_exception, so I would save those 'try ...' for each
binding. Looking at the code I guess that it's now impossible, calling
fatal_uncaught_exception is just hardwired in appropriate places.

> Yes. Somebody has written a tool to convert a trace in source position:
> http://caml.inria.fr/archives/200109/msg00111.html

This is an external tool which reads original sources and the
backtrace. I want to customize the backtrace itself. Especially as
OCaml sources is only an intermediate step between my language and
executable - they should not be required to be available for producing
a proper backtrace. I will set up my own translation of OCaml positions
to original source positions and embed it in the program.

-- 
 __("<  Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/
 \__/
  ^^
QRCZAK

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] A few questions regarding the compiler

[Marcin 'Qrczak' Kowalczyk]

Wed, 14 Nov 2001 14:47:13 +0000 (UTC), Marcin 'Qrczak' Kowalczyk <qrczak@knm.org.pl> pisze:

> I'll now try to do it myself basing on backtrace.c, but I don't
> understand yet how it works.

Why all interesting things are private again? :-(

Yes, it works, but it required copying parts of various .h files
(they are not installed by default). Fortunately it's C and not OCaml
so names exported from any module can be accessed.

E.g. I had to copy the 'instructions' type to obtain the values of
RAISE and RERAISE. I don't think it's very stable across releases.
It makes the whole solution more version-dependent than necessary
and so ugly that I'm not sure it's worth doing.

-- 
 __("<  Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/
 \__/
  ^^
QRCZAK

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr