[Caml-list] is it possible to embed an OCaml interpreter into an OCaml Module?

[Caml-list] is it possible to embed an OCaml interpreter into an OCaml Module?

[Christoph Höger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dear all,

I already asked this on stackoverflow and was pointed to
compiler-libs.toplevel - indeed this API seems to be sufficient to run
an OCaml interpreter from inside an OCaml program (as that seems to be
what utop does).

But ist it also possible in some way to embed that interpreter safely
in an OCaml Module (so I can reuse it e.g. from within utop)?
Currently, it seems that there is exactly one dedicated toplevel for
every running bytecode interpreter and when running utop, it is
already in use.

So what I would need would be the ability to execute a phrase from
within a call of execute_phrase. I already clonded the toploop module
and for tehe time being I am fine with that. What I need is a way to

a) safe the already set ('outer') toplevel value bindings
b) restore the nested value bindings
c) execute the compiled bytecode
d) restore the 'outer' value bindings

is there someone who can point me to a solution?
- -- 
Christoph Höger

Technische Universität Berlin
Fakultät IV - Elektrotechnik und Informatik
Übersetzerbau und Programmiersprachen

Sekr. TEL12-2, Ernst-Reuter-Platz 7, 10587 Berlin

Tel.: +49 (30) 314-24890
E-Mail: christoph.hoeger@tu-berlin.de
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iEYEARECAAYFAlRTn9kACgkQhMBO4cVSGS97QwCfX59enOE89WWwvBu555V9LNM/
mCwAoLJ4PLkDlrLwA6A8OKzj8Elnpu0B
=uWdx
-----END PGP SIGNATURE-----

Re: [Caml-list] is it possible to embed an OCaml interpreter into an OCaml Module?

[Peter Zotov]

On 2014-10-31 17:42, Christoph Höger wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Dear all,
> 
> I already asked this on stackoverflow and was pointed to
> compiler-libs.toplevel - indeed this API seems to be sufficient to run
> an OCaml interpreter from inside an OCaml program (as that seems to be
> what utop does).
> 
> But ist it also possible in some way to embed that interpreter safely
> in an OCaml Module (so I can reuse it e.g. from within utop)?
> Currently, it seems that there is exactly one dedicated toplevel for
> every running bytecode interpreter and when running utop, it is
> already in use.
> 
> So what I would need would be the ability to execute a phrase from
> within a call of execute_phrase. I already clonded the toploop module
> and for tehe time being I am fine with that. What I need is a way to
> 
> a) safe the already set ('outer') toplevel value bindings
> b) restore the nested value bindings
> c) execute the compiled bytecode
> d) restore the 'outer' value bindings
> 
> is there someone who can point me to a solution?

There is nothing special about the toplevel or the OCaml interpreter.
Indeed, you can run a chunk of OCaml code at any time using
the caml_callback C call--this is exactly what the runtime would
invoke at startup, or what the toplevel would do to run a freshly
compiled phrase.

However, the OCaml runtime has quite some global data:
a cursory look brings up GC, backtrace machinery, polymorphic
comparison (!), debugger, dynlink (and its symbol tables),
exceptions, finalizers, and a few more things. Depending on your
desired level of isolation, it might be very hard to provide
a clean environment.

Incidentally, can someone explain how the runtime manages to be
multithreaded while it has so much global data? Please don't
tell me the answer is "every thread copies it back and forth
on context switch".

-- 
Peter Zotov

Re: [Caml-list] is it possible to embed an OCaml interpreter into an OCaml Module?

[Stephen Dolan]

On Fri, Oct 31, 2014 at 3:03 PM, Peter Zotov <whitequark@whitequark.org> wrote:
> Incidentally, can someone explain how the runtime manages to be
> multithreaded while it has so much global data? Please don't
> tell me the answer is "every thread copies it back and forth
> on context switch".

In current OCaml, there is a global lock. The global data can only be
accessed holding this lock, so only one thread can run OCaml code at a
time.

In the multicore branch, the data is thread-local.

Stephen

Re: [Caml-list] is it possible to embed an OCaml interpreter into an OCaml Module?

[Christoph Höger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thanks for the hint. I wasn't able to figure out, how MetaOCaml does
the trick, though. So, here is my solution.

The trick is to store away the old "toploop" global variable from the
Symtable module. This seems to work, although it looks a bit hacky. I
am unsure whether this works _outside_ of a toplevel. The sane way
would be to also fork Translmod and add a completely new global
variable for my interpreter, but that seems to be impossible (adding
that global variable programatically). I do not understand, why there
is no API dealing with it. Is there a deeper reason?

let toploop_id = Ident.create_persistent "Toploop"

let execute_phrase print_outcome ppf phr =
  match phr with
  | Ptop_def sstr ->
      let outer_toploop = Symtable.get_global_value toploop_id in
      let _ = Symtable.assign_global_value toploop_id (Obj.repr
toplevel_value_api) in
      let oldenv = !toplevel_env in
      Typecore.reset_delayed_checks ();
      let (str, sg, newenv) = Typemod.type_toplevel_phrase oldenv sstr in
      if !Clflags.dump_typedtree then Printtyped.implementation ppf str;
      let sg' = Typemod.simplify_signature sg in
      ignore (Includemod.signatures oldenv sg sg');
      Typecore.force_delayed_checks ();
      let lam = Translmod.transl_toplevel_definition str in
      Warnings.check_fatal ();
      begin try
        toplevel_env := newenv;
        let res = load_lambda ppf lam in
        let out_phr =
          match res with
          | Result v ->
	     let res =
               if print_outcome then
                 Printtyp.wrap_printing_env oldenv (fun () ->
						    match str.str_items with
						    | [ { str_desc = Tstr_eval (exp, _attrs) }] ->
						       let outv = outval_of_value newenv v exp.exp_type in
						       let ty = Printtyp.tree_of_type_scheme exp.exp_type in
						       Ophr_eval (outv, ty)
						    | [] -> Ophr_signature []
						    | _ -> Ophr_signature (item_list newenv sg'))
               else Ophr_signature []
	     in Symtable.assign_global_value toploop_id outer_toploop ; res
          | Exception exn ->
	     Symtable.assign_global_value toploop_id outer_toploop ;
             toplevel_env := oldenv;
             if exn = Out_of_memory then Gc.full_major();
             let outv =
               outval_of_value !toplevel_env (Obj.repr exn)
Predef.type_exn
             in
             Ophr_exception (exn, outv)
        in
        !print_out_phrase ppf out_phr;
        begin match out_phr with
        | Ophr_eval (_, _) | Ophr_signature _ -> true
        | Ophr_exception _ -> false
        end
      with x ->
        toplevel_env := oldenv; raise x				
      end
  | Ptop_dir(dir_name, dir_arg) ->
      let d =
        try Some (Hashtbl.find directive_table dir_name)
        with Not_found -> None
      in
      begin match d with
      | None ->
          fprintf ppf "Unknown directive `%s'.@." dir_name;
          false
      | Some d ->
          match d, dir_arg with
          | Directive_none f, Pdir_none -> f (); true
          | Directive_string f, Pdir_string s -> f s; true
          | Directive_int f, Pdir_int n -> f n; true
          | Directive_ident f, Pdir_ident lid -> f lid; true
          | Directive_bool f, Pdir_bool b -> f b; true
          | _ ->
              fprintf ppf "Wrong type of argument for directive `%s'.@."
                dir_name;
              false
      end


- -- 
Christoph Höger

Technische Universität Berlin
Fakultät IV - Elektrotechnik und Informatik
Übersetzerbau und Programmiersprachen

Sekr. TEL12-2, Ernst-Reuter-Platz 7, 10587 Berlin

Tel.: +49 (30) 314-24890
E-Mail: christoph.hoeger@tu-berlin.de
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iEYEARECAAYFAlRXjZMACgkQhMBO4cVSGS946ACdG788ViovGE5X6j0XbuswsZ3z
uJ4An2BXxlFRMSiagkIYi9EQb0SqrHwz
=xCBd
-----END PGP SIGNATURE-----

Re: [Caml-list] is it possible to embed an OCaml interpreter into an OCaml Module?

[oleg]

Christoph Hoeger wrote:

> But ist it also possible in some way to embed that interpreter safely
> in an OCaml Module (so I can reuse it e.g. from within utop)?
> Currently, it seems that there is exactly one dedicated toplevel for
> every running bytecode interpreter and when running utop, it is
> already in use.
>
> So what I would need would be the ability to execute a phrase from
> within a call of execute_phrase. I already clonded the toploop module
> and for tehe time being I am fine with that. What I need is a way to
>
> a) safe the already set ('outer') toplevel value bindings
> b) restore the nested value bindings
> c) execute the compiled bytecode
> d) restore the 'outer' value bindings

The MetaOCaml top-level does exactly what you have described. It is
the standard OCaml toplevel with the ability to execute the generated
code. Please look at the file metalib/runcode.ml in the MetaOCaml
distribution. If you get it from the metaocaml.bundle, you get the
current version. Otherwise, please to make sure the beginning of
run_bytecode starts as follows:

let run_bytecode' exp =
  if !initial_env = Env.empty then begin
    let old_time = Ident.current_time() in
    (* does Ident.reinit() and may corrupt the timestamp if we
       run in top-level. See Ident.reinit code
     *)
    initial_env := Compmisc.initial_env(); 
    Ident.set_current_time old_time
   end;
  (* Ctype.init_def(Ident.current_time());  *)
  ....