problem with ocamlmktop -output-obj

problem with ocamlmktop -output-obj

[Thierry Bravier]

Dear ocamlers,
 
I have encountered a problem while trying to build a toplevel
(ocamlmktop) linked with
- external C code and
- ML code compiled as a C object (-output-obj and implicitely -custom).
 
I have no problem when I build an ordinary batch (non-toplevel) program.
 
Is this a limitation of the system or I am doing it wrong ?
 
Thanks.
 
Thierry Bravier              Dassault Aviation - DGT / DPR / DESA
78, Quai Marcel Dassault       F-92214 Saint-Cloud Cedex - France
Telephone : (33) 01 47 11 53 07   Telecopie : (33) 01 47 11 52 83
E-Mail :              mailto:thierry.bravier@dassault-aviation.fr
 
PS. Here are 3 small files demonstrating the issue.
 
I understand it does not make sense to use -output-obj just to call
printf from C.
In fact, I use it to link ML and C++ code; in this case, C++ needs to
compile main() and link itself (as explained in the manual section
`Embedding the Caml code in the C code', ordinary -custom wants to link
but -output-obj builds a linkable .o file which seems to fit my needs)
 
PS2:
I work on solaris
% uname -a
SunOS scdprs20 5.5.1 Generic sun4u sparc SUNW,Ultra-2
 
/* ==================================================================
 *        File: mail_main.c
 * =============================================================== */
#include <stdio.h>
#include <caml/mlvalues.h>
#include <caml/callback.h>
 
/* =============================================================== */
int main (int argc, char **argv) {
  printf ("entering C\n");
  caml_startup (argv);
  printf ("5! = %d\n",
          Int_val(callback(*caml_named_value("f"),Val_int(5))));
  printf ("exiting C\n");
}
/* =============================================================== */
  
(* ==================================================================
 *        File: mail_prgm.ml
 * =============================================================== *)
let _ = Printf.printf "entering ML\n"
let rec f = function 0 -> 1 | n -> n * f (pred n)
let _ = Callback.register "f" f
let _ = Printf.printf "exiting ML\n"
 
(* =============================================================== *)
 
#!/bin/sh -x
#====================================================================
#         File: mail_build
#====================================================================
STDLIB=/logiciel/GNU/LOCAL/ocaml/ocaml/stdlib   # or whatever
LDLIBS="-lcamlrun -ltermlib -lsocket -lnsl -lm" # or whatever
CC=gcc                                          # or whatever
 
#====================================================================
${CC} -c -I${STDLIB} mail_main.c
ocamlc -c mail_prgm.ml
 
ocamlc     -output-obj -o mail_batch.o    mail_prgm.cmo
ocamlmktop -output-obj -o mail_toplevel.o mail_prgm.cmo
 
${CC} -o mail_batch    -L${STDLIB} mail_main.o mail_batch.o    ${LDLIBS}
${CC} -o mail_toplevel -L${STDLIB} mail_main.o mail_toplevel.o ${LDLIBS}
 
#====================================================================
./mail_batch                                    # works as expected
# entering C
# entering ML
# exiting ML
# 5! = 120
# exiting C
 
./mail_toplevel                                 # fails at runtime
# entering C
# Fatal error: uncaught exception Failure("input_value: bad object")
 
#====================================================================

-- 
Thierry Bravier                     Dassault Aviation - DGT / DPR / DESA
78, Quai Marcel Dassault              F-92214 Saint-Cloud Cedex - France
Telephone : (33) 01 47 11 53 07          Telecopie : (33) 01 47 11 52 83
E-Mail :                     mailto:thierry.bravier@dassault-aviation.fr

Re: problem with ocamlmktop -output-obj

[Xavier Leroy]

> I have encountered a problem while trying to build a toplevel
> (ocamlmktop) linked with
> - external C code and
> - ML code compiled as a C object (-output-obj and implicitely -custom).
> I have no problem when I build an ordinary batch (non-toplevel) program.
> Is this a limitation of the system or I am doing it wrong ?

Toplevels need access to their own symbol table, and they look inside
their executable file to find it.  The symbol table information is not
available if -output-obj is selected.

A regular bytecode executable file is composed of several sections:
code, data, symbols, and debug info.  Currently, only the first two
are available when a C object is generated instead of a regular
executable file.

For the same reasons, you can't debug a Caml program built with
-output-obj.

Hope this helps,

- Xavier Leroy

Re: problem with ocamlmktop -output-obj

[Thierry Bravier]

Xavier Leroy wrote:
> 
> > I have encountered a problem while trying to build a toplevel
> > (ocamlmktop) linked with
> > - external C code and
> > - ML code compiled as a C object (-output-obj and implicitely -custom).
> > I have no problem when I build an ordinary batch (non-toplevel) program.
> > Is this a limitation of the system or I am doing it wrong ?
> 
> Toplevels need access to their own symbol table, and they look inside
> their executable file to find it.  The symbol table information is not
> available if -output-obj is selected.
> 
> A regular bytecode executable file is composed of several sections:
> code, data, symbols, and debug info.  Currently, only the first two
> are available when a C object is generated instead of a regular
> executable file.
> 
> For the same reasons, you can't debug a Caml program built with
> -output-obj.
> 
> Hope this helps,
> 
> - Xavier Leroy

Thank you for answering

I must now face a problem:

Thierry Bravier wrote:
> I understand it does not make sense to use -output-obj just to call
> printf from C.
> In fact, I use it to link ML and C++ code; in this case, C++ needs to
> compile main() and link itself (as explained in the manual section
> `Embedding the Caml code in the C code', ordinary -custom wants to
> link but -output-obj builds a linkable .o file which seems to fit my
> needs

It would be really nice if I could use -custom but ...
 
How can I ask ocamlc -custom to :
1) use a user-provided compiled main() (potentially compiled with C++)
2) link with a user-provided linker (aka g++)
3) I know that internally, some C code is output and compiled
   (for the external primitives I think), I think this code should
   still be compiled with the C compiler used to build ocaml.

Are there any clues to help me build this toplevel with C++ code
without -output-obj ?

Thanks

-- 
Thierry Bravier                     Dassault Aviation - DGT / DPR / DESA
78, Quai Marcel Dassault              F-92214 Saint-Cloud Cedex - France
Telephone : (33) 01 47 11 53 07          Telecopie : (33) 01 47 11 52 83
E-Mail :                     mailto:thierry.bravier@dassault-aviation.fr

Re: problem with ocamlmktop -output-obj

[Thierry Bravier]

Thierry Bravier wrote:
> I must now face a problem:
> 
> It would be really nice if I could use -custom but ...
> 
> How can I ask ocamlc -custom to :
> 1) use a user-provided compiled main() (potentially compiled with C++)
> 2) link with a user-provided linker (aka g++)
> 3) I know that internally, some C code is output and compiled
>    (for the external primitives I think), I think this code should
>    still be compiled with the C compiler used to build ocaml.
> 
> Are there any clues to help me build this toplevel with C++ code
> without -output-obj ?
> 
> Thanks

Here is a suggestion so that I can link ML with C++
and avoid using -output-obj when I don't need it (particularly
to build a toplevel).

with two new options for ocamlc, one could choose ...

* ocamlc -cc my-ccompiler
  ... an alternative C compiler internally used
  to compile the internal primitive table and other C codes.
* ocamlc -clinker my-clinker
  ... an alternative linker used to link native objects before
  concatenation of ocaml cmo files
  this one is most important to C++ users because it makes it
  possible to let a C++ tool link object files.

I have emulated this thanks to an ugly hack that I would like
to remove. I have made my local gcc startup script
read an environment variable and link with another
linker when the variable is set (in my case, a C++ linker).

It gives me the ability to build a C++/ML program (or toplevel) with
ocamlc (or ocamlmktop) -custom which is just what I need.

I also have a request about the toplevel module:

The #quit directive happens to call directly exit
this is a problem if the toplevel is embedded in a C (or C++)
main () function because after #quit, the program
exits instead of finishing main ()'s code.

In the case of C++ it's even worse because destructors are
not called.

I would suggest an internal toplevel_exit exception
raised by #quit
and handled by the toplevel loop to return properly from caml_main ()

Thank you.

-- 
Thierry Bravier                     Dassault Aviation - DGT / DPR / DESA
78, Quai Marcel Dassault              F-92214 Saint-Cloud Cedex - France
Telephone : (33) 01 47 11 53 07          Telecopie : (33) 01 47 11 52 83
E-Mail :                     mailto:thierry.bravier@dassault-aviation.fr

Re: problem with ocamlmktop -output-obj

[Pascal Brisset]

Thierry Bravier writes:
 > > Are there any clues to help me build this toplevel with C++ code
 > > without -output-obj ?

I've been calling C++ libraries from caml code since ocaml-1.03
without resorting to any particular hack (see example below). I tend
to avoid things like global objects and c++ exceptions (which were
seriously flawed at that time), so this might not cover all of your
problems. Also, it may fail with compilers other than gcc (because gcc
called as a linker resolves c++ symbols).

- Pascal Brisset <pascal.brisset@cnet.francetelecom.fr> +33296051928 -
- France Telecom CNET DTL/MSV | 2 av Pierre Marzin | F-22307 Lannion -

-------------------------- mlcell.C ---------------------------------

#include <stdio.h>

extern "C" {
# include <caml/mlvalues.h>
# include <caml/alloc.h>
}

class Cell {
public:
  Cell(int init) : val(init) { }
  void set(int x) { val = x; }
  int get() { return val; }
private:
  int val;
};

typedef struct {
  final_fun f;
  Cell *c;
} mlcell;

static void free_cell(value mlc) {
  printf("freeing %p\n", mlc);
  delete ((mlcell*)mlc)->c;
}

extern "C" value caml_cell_create(value mlv) {
  mlcell *res = (mlcell*)alloc_final(sizeof(mlcell)/sizeof(value),
				     free_cell, 1, 1000); /* ? */
  res->c = new Cell(Int_val(mlv));
  return (value)res;
}
			  
extern "C" value caml_cell_set(value mlc, value mlv) {
  ((mlcell*)mlc)->c->set(Int_val(mlv));
  return Val_unit;
}

extern "C" value caml_cell_get(value mlc) {
  int v = ((mlcell*)mlc)->c->get();
  return Val_int(v);
}

-------------------------- cell.ml --------------------------------

module Cell = struct
  type t
  external create : int -> t = "caml_cell_create"
  external set : t -> int -> unit = "caml_cell_set"
  external get : t -> int = "caml_cell_get"
end

let _ =
  let c = Cell.create 99 in
  Printf.printf "%d\n" (Cell.get c);
  Cell.set c 42;
  Printf.printf "%d\n" (Cell.get c);
  flush stdout

let _ = Gc.full_major ()

-------------------------- Testing --------------------------------

$ g++ --version
2.8.1
$ g++ -I/usr/local/lib/ocaml -c mlcell.C
$ ocamlc -v -custom mlcell.o cell.ml -o cell.out
The Objective Caml compiler, version 1.07
Standard library directory: /usr/local/lib/ocaml
$ ./cell.out
99
42
freeing a2944

------------------------------------------------------------------

Re: problem with ocamlmktop (contd)

[Pascal Brisset]

[-- Attachment #1: Type: text/plain, Size: 1160 bytes --]

Here is a more complete example demonstrating:

(1) destructors of global objects being called correctly on exit;
(2) translation of C++ exceptions to Caml exceptions;
(3) catching a C++ exception generated by a C++ primitive called
    through a Caml callback.

The only trick is that if you really need (3), you have to modify
libcamlrun.a (found in ocaml-1.07/byterun) as follows:

  - Insert `extern "C" {' at the beginning of interp.c and callback.c
  - Insert `}' at the end of interp.c and callback.c
  - Compile interp.c and callback.c with g++ (This will add
    ".eh_frame" sections which are required for exceptions handling):
          g++ -O -fno-defer-pop -Wall   -c interp.c -o interp.o
          g++ -O -fno-defer-pop -Wall   -c callback.c -o callback.o
  - Compile everything else normally (make libcamlrun.a)

The Makefile assumes that the modified libcamlrun.a is in
/tmp/ocaml-1.07/byterun/. This was tested with ocaml-1.07 and
g++-2.8.1. Again, things seem to have improved a lot since gcc-2.7.

- Pascal Brisset <pascal.brisset@cnet.francetelecom.fr> +33296051928 -
- France Telecom CNET DTL/MSV | 2 av Pierre Marzin | F-22307 Lannion -


[-- Attachment #2: Makefile --]
[-- Type: text/plain, Size: 271 bytes --]

LIBCAMLRUN=-cclib /tmp/ocaml-1.07/byterun/libcamlrun.a

run:
	g++ -I/usr/local/lib/ocaml -c libcell.C
	g++ -I/usr/local/lib/ocaml -c mlcell.C
	ocamlc -custom libcell.o mlcell.o cell.ml -o cell.out $(LIBCAMLRUN)
	./cell.out

clean:
	/bin/rm -f *.out *.o *.cm[io] *~ \#*\#

[-- Attachment #3: cell.ml --]
[-- Type: text/plain, Size: 963 bytes --]

module Cell = struct
  type t
  external global : unit -> t = "caml_global_cell"
  external create : int -> t = "caml_cell_create"
  external set : t -> int -> unit = "caml_cell_set"
  external get : t -> int = "caml_cell_get"

  external throw : unit -> string = "caml_cell_throw"
  external call : string -> string = "caml_cell_call"
end

let test_cell c =
  Printf.printf "c=%d\n" (Cell.get c); flush stdout;
  Printf.printf "set 42... "; flush stdout;
  Cell.set c 42;
  Printf.printf "c=%d\n" (Cell.get c); flush stdout;
  begin try
    Printf.printf "set -1... "; flush stdout;
    Cell.set c (-1);
  with e -> print_endline (Printexc.to_string e); flush stdout
  end

let _ =
  print_endline "start"; flush stdout;
  test_cell (Cell.create 271828);
  Gc.full_major (); print_newline ();
  test_cell (Cell.global ()); print_newline ()

let _ =
  Callback.register "caml-throw" Cell.throw;
  print_endline ("callback: "^Cell.call "caml-throw"); flush stdout

[-- Attachment #4: libcell.C --]
[-- Type: text/plain, Size: 345 bytes --]

#include <stdio.h>
#include "libcell.h"

Exc::Exc(const char *m) : msg(m) { } 

Cell::Cell(int init) : val(init) { printf("init %p=%d\n", this, init); }
Cell::~Cell() { printf("free %p (was %d)\n", this, val); }

void Cell::set(int x) {
  if ( x < 0 ) throw Exc("< 0");
  val = x;
}
 
int Cell::get() { return val; }

Cell global_cell(3141592);

[-- Attachment #5: libcell.h --]
[-- Type: text/plain, Size: 189 bytes --]

class Exc {
public:
  Exc(const char *m);
  const char *msg;
};

class Cell {
public:
  Cell(int);
  ~Cell();
  void set(int);
  int get();
private:
  int val;
};

extern Cell global_cell;

[-- Attachment #6: mlcell.C --]
[-- Type: text/plain, Size: 1182 bytes --]

#include <stdio.h>

extern "C" {
# include <caml/mlvalues.h>
# include <caml/alloc.h>
# include <caml/callback.h>
extern void failwith(char *s);
}

#include "libcell.h"

typedef struct {
  final_fun f;
  Cell *c;
} mlcell;

static void free_cell(value mlc) {
  delete ((mlcell*)mlc)->c;
}

static mlcell mlglobal_cell = { free_cell, &global_cell };

extern "C" value caml_global_cell(value) {
  return (value)&mlglobal_cell;
}

extern "C" value caml_cell_create(value mlv) {
  value res = alloc_final(sizeof(mlcell)/sizeof(value),
				     free_cell, 1, 1000); /* ? */
  ((mlcell*)res)->c = new Cell(Int_val(mlv));
  return res;
}
			  
extern "C" value caml_cell_set(value mlc, value mlv) {
  try { ((mlcell*)mlc)->c->set(Int_val(mlv)); }
  catch (Exc e) { failwith((char*)e.msg); }
  return Val_unit;
}

extern "C" value caml_cell_get(value mlc) {
  int v = ((mlcell*)mlc)->c->get();
  return Val_int(v);
}

extern "C" value caml_cell_throw(value) {
  throw Exc("caml_cell_throw");
}

extern "C" value caml_cell_call(value mlname) {
  value f = *caml_named_value(String_val(mlname));
  try { return callback(f, Val_unit); }
  catch (Exc e) { return copy_string((char*)e.msg); }
}

Re: problem with ocamlmktop (contd)

[Pascal Brisset]

Pascal Brisset writes:
 > (3) catching a C++ exception generated by a C++ primitive called
 >     through a Caml callback.

Oops, I shouldn't have included the last part... Here is a tentative
list of issues regarding exceptions and Caml/C++ interfacing.
I hope others will contribute; this is definitely tricky.

* g++-2.8.1 seems to link C/C++ reliably, therefore nesting Caml/C++
  and C++/Caml calls without exceptions on either side should be safe.

* C++ primitive raising a Caml exception:

  The following code is incorrect because mlraise uses siglongjmp,
  which does not unwind the C++ stack (obj will not be destroyed):
   |   extern "C" value caml_stub(value) {
   |     Object obj(...);
   |     mlraise(...);
   |   }

  This might be safe for practical purposes:
   |   extern "C" value caml_stub(value) {
   |     try { /* pure C++ stuff */; }
   |     catch (...) { failwith("Uncaught C++ exception"); }
   |   }
  but only a C++ expert could tell whether the exception's destructor
  (if any) will be called...

* Callback raising a Caml exception:

  This is unsafe for the same reason:
   |   extern "C" value caml_stub(value) {
   |     Object obj(...);
   |     callback(...);
   |   }

* C++ primitive throwing a C++ exception:

  We might want to have the interpreter translate all C++ exceptions
  into Caml exceptions:
   |   Instruct(C_CALL1):
   |     Setup_for_c_call;
   |     try { accu = cprim[*pc](accu); }
   |     catch (...) {
   |       local_roots = initial_local_roots;
   |       failwith("Uncaught C++ exception");
   |     }
   |     Restore_after_c_call;
   |     ...
  The local_roots assignment is meant to handle exceptions thrown from
  within a Begin_roots/End_roots block (i.e. don't let failwith()
  allocate memory while local_roots points to an obsolete stack
  frame). Is this enough ?

* Caml callback throwing a C++ exception (i.e. trying to embed Caml
  code in C++ code transparently like in my example):

  I believe similar changes are required in order to return from
  interprete() gracefully (catch the C++ exception, handle things like
  external_raise, callback_depth and local_roots, then re-throw the
  exception). Has anyone done this ?

- Pascal Brisset <pascal.brisset@cnet.francetelecom.fr> +33296051928 -
- France Telecom CNET DTL/MSV | 2 av Pierre Marzin | F-22307 Lannion -

Re: problem with ocamlmktop (contd)

[Thierry Bravier]

Pascal Brisset wrote:
> 
> Here is a more complete example demonstrating:
> 
> (1) destructors of global objects being called correctly on exit;
> (2) translation of C++ exceptions to Caml exceptions;
> (3) catching a C++ exception generated by a C++ primitive called
>     through a Caml callback.
> 
> The only trick is that if you really need (3), you have to modify
> libcamlrun.a (found in ocaml-1.07/byterun) as follows:
> 
>

Thanks for helping,

I am not currently trying to map ocaml and C++ exceptions
because it seems that basically, C++ exceptions don't mix
well with setjmp/longjmp. so (2) and (3) are not my priorities
although they are challenging.

(1) is important to me ! Your example uses final ML objects
successfully but I am also interested in making C++ manage
its own C++ global values when entering main and
most of all when leaving main ().

I agree C++ global variables should be avoided and I do my very
best to achieve this goal, unfortunately static variables
in functions (and of course in classes too) also need to be
destroyed at exit () time. This means, for most C++ compilers,
compiling main () in C++ and linking with the C++ linker.

The features of gcc (it can replace g++ as a linker) are
not fully satisfying in my case because I mainly use egcs-1.1
(is it linkable with gcc-2.8.1 ?) and I often try to compile
and link code with other C++ compilers (mainly SparcWorks and xlC).

This explains why I need the -cc and -clinker options
I have previously suggested.

I keep the exceptions management for further explorations.

Cheers

-- 
Thierry Bravier                     Dassault Aviation - DGT / DPR / DESA
78, Quai Marcel Dassault              F-92214 Saint-Cloud Cedex - France
Telephone : (33) 01 47 11 53 07          Telecopie : (33) 01 47 11 52 83
E-Mail :                     mailto:thierry.bravier@dassault-aviation.fr

Re: problem with ocamlmktop -output-obj

[luther]

On Thu, Oct 15, 1998 at 07:22:43PM +0200, Xavier Leroy wrote:
> > I have encountered a problem while trying to build a toplevel
> > (ocamlmktop) linked with
> > - external C code and
> > - ML code compiled as a C object (-output-obj and implicitely -custom).
> > I have no problem when I build an ordinary batch (non-toplevel) program.
> > Is this a limitation of the system or I am doing it wrong ?
> 
> Toplevels need access to their own symbol table, and they look inside
> their executable file to find it.  The symbol table information is not
> available if -output-obj is selected.
> 
> A regular bytecode executable file is composed of several sections:
> code, data, symbols, and debug info.  Currently, only the first two
> are available when a C object is generated instead of a regular
> executable file.
> 
> For the same reasons, you can't debug a Caml program built with
> -output-obj.

Je pense que j'ai un probleme similaire.
I think i have a similar problem.

Je veut utiliser plus d'un source ocaml embarque dans du C.
i want to use more than one ocaml source file embedded in C.

donc je prend l'exemple de la doc, j'ajoute un other.ml contenant :
so i take the example, and add an other.ml file containing :

        let my_format_result n = Printf.sprintf "Result is: %d\n" n
		
et modifie mod.ml comme suit :
and modify mod.ml as follows :

	(* File mod.ml -- some ``useful'' Caml functions *)
	
	    let rec fib n = if n < 2 then 1 else fib(n-1) + fib(n-2)
	
	    let format_result n = Other.my_format_result n
		
	(* Export those two functions to C *)
	
        let _ = Callback.register "fib" fib
        let _ = Callback.register "format_result" format_result
					
et a la compilation j'ai le probleme suivant :
and when compiling i have the following error :

	ocamlc -custom -output-obj -o other.o other.ml
	ocamlc -custom -output-obj -o modcaml.o modcaml.ml
	Error while linking modcaml.cmo: Reference to undefined global `Other'

Est-il possible de compiler plusieurs source ocaml, en une librairie .cma par
exemple, puis de faire un -output-obj de celle-ci ? Est ce que cela resoudrait
le probleme ? Ou alors dois-je creer un fichier source geant, et transformer
mon other.ml en module Other = struct (contenu de other.ml) end, eventuellement
automatiquement ? Pas tres joli, mais cela marche.

IS it possible to compile more than 1 caml source, as a .cma library for
example, and then embedd it in C code ? will this solve my problem, or should i
do alone giant caml source file that include all other source files as modules
?

Amicalement,

Sven LUTHER