[Caml-list] Calling the toplevel from the toplevel

[Caml-list] Calling the toplevel from the toplevel

[rixed]

I'm trying to build an application that's run from the toplevel, using
the toplevel to customize the application from itself (for instance,
setting some global parameters using the toplevel to configure the
application behavior while it's running).

But I cannot allow the toplevel to read directly from stdin nor to
write anything into stdout (since my application uses ncurses).

I've looked for various ways to do this, and the simpler I found to
prevent the toplevel to use stdout is to call myself
Toploop.execute_phrase with a custom formatter (so that I can display
the output where and how I want).

Up to now all seams to work except for minor annoyances :

- I cannot start the application directly by linking the custom toplevel
  with something like "let _ = start_application ()" but I have to call
  "start_application();;" from the toplevel manually (or from the
  .ocamlinit file), otherwise the application bindings are not
  available.

- I cannot let the user uses the toplevel directives "use" and "load"
  because both writes into stdout whatever the formatter passed to
  Toploop.execute_phrase (for "use" this is easily solvable by shadowing
  the toplevel implementation by another one that call Toploop.use_file
  with my own formatter, but for "load" I would have to copy a lot of
  code from topdirs.ml

I'd like to know if it is safe to call the Toploop evaluation functions
from code evaluated from the toplevel itself ? Or if someone can suggest
a better way to prevent the toplevel from using stdout ?

Re: [Caml-list] Calling the toplevel from the toplevel

[Guillaume Yziquel]

Le Saturday 26 Feb 2011 à 22:44:04 (+0100), rixed@happyleptic.org a écrit :
> I'm trying to build an application that's run from the toplevel, using
> the toplevel to customize the application from itself (for instance,
> setting some global parameters using the toplevel to configure the
> application behavior while it's running).
> 
> But I cannot allow the toplevel to read directly from stdin nor to
> write anything into stdout (since my application uses ncurses).
> 
> I've looked for various ways to do this, and the simpler I found to
> prevent the toplevel to use stdout is to call myself
> Toploop.execute_phrase with a custom formatter (so that I can display
> the output where and how I want).
> 
> Up to now all seams to work except for minor annoyances :
> 
> - I cannot start the application directly by linking the custom toplevel
>   with something like "let _ = start_application ()" but I have to call
>   "start_application();;" from the toplevel manually (or from the
>   .ocamlinit file), otherwise the application bindings are not
>   available.

For this, you can change the Toploop.read_interactive_input reference to
what you want to control what you input to the toplevel. You therefore
do not need to consider workarounds such as calling evaluation functions
from code evaluated from the toplevel itself.

But it's a bit surprising that application bindings are not available
when calling start_application from some code that is being loaded. If
you insist on loading a .cmo instead of a .cma, the .cmo code is
executed when loaded, while code in the .cma often seems to be executed
only when required. That may solve your issue. Dunno.

You've also got a Toplevel.toplevel_startup_hook ref that may be useful.

> - I cannot let the user uses the toplevel directives "use" and "load"
>   because both writes into stdout whatever the formatter passed to
>   Toploop.execute_phrase (for "use" this is easily solvable by shadowing
>   the toplevel implementation by another one that call Toploop.use_file
>   with my own formatter, but for "load" I would have to copy a lot of
>   code from topdirs.ml
> 
> I'd like to know if it is safe to call the Toploop evaluation functions
> from code evaluated from the toplevel itself ? Or if someone can suggest
> a better way to prevent the toplevel from using stdout ?

Well, the Toploop module provides some interesting values to control
output:

val parse_toplevel_phrase : (Lexing.lexbuf -> Parsetree.toplevel_phrase)
ref
val parse_use_file : (Lexing.lexbuf -> Parsetree.toplevel_phrase list)
ref
val print_location : formatter -> Location.t -> unit
val print_error : formatter -> Location.t -> unit
val print_warning : Location.t -> formatter -> Warnings.t -> unit
val input_name : string ref

val print_out_value :
  (formatter -> Outcometree.out_value -> unit) ref
val print_out_type :
  (formatter -> Outcometree.out_type -> unit) ref
val print_out_class_type :
  (formatter -> Outcometree.out_class_type -> unit) ref
val print_out_module_type :
  (formatter -> Outcometree.out_module_type -> unit) ref
val print_out_sig_item :
  (formatter -> Outcometree.out_sig_item -> unit) ref
val print_out_signature :
  (formatter -> Outcometree.out_sig_item list -> unit) ref
val print_out_phrase :
  (formatter -> Outcometree.out_phrase -> unit) ref

I'd also look in the toplevel code of lwt, specifically in the source
code of lwt.top package. Contains quite a lot of interesting toplevel
tricks that would likely be of use to you. I'm not so sure, but it seems
to have some amount of control over toplevel output.

-- 
     Guillaume Yziquel

Re: [Caml-list] Calling the toplevel from the toplevel

[rixed]

> For this, you can change the Toploop.read_interactive_input reference to
> what you want to control what you input to the toplevel.

Yes, I tried this initially, having found this into the lwt top level.
Actually, there is many way to replace stdin by something else. My
main concern is for stdout, which appears explicitly in the code for the
toplevel directives.

> You therefore
> do not need to consider workarounds such as calling evaluation functions
> from code evaluated from the toplevel itself.

Calling the evaluator is the only way I've found to prevent the toplevel from
writing into stdout (apart from manipulations of the file descriptors themself,
which are error prone since ncurses also insist on writing on file descriptor
1).

> But it's a bit surprising that application bindings are not available
> when calling start_application from some code that is being loaded. If
> you insist on loading a .cmo instead of a .cma, the .cmo code is
> executed when loaded, while code in the .cma often seems to be executed
> only when required. That may solve your issue. Dunno.

The application is not loaded as a cmo nor as a cma, but linked into
a custom toplevel with mktoplevel, so I'm unsure if this remark applies.

> You've also got a Toplevel.toplevel_startup_hook ref that may be useful.

I will have a look to this one, thank you!

Re: [Caml-list] Calling the toplevel from the toplevel

[Jérémie Dimino]

On Sat, Feb 26, 2011 at 10:44:04PM +0100, rixed@happyleptic.org wrote:
> Up to now all seams to work except for minor annoyances :
> 
> - I cannot start the application directly by linking the custom toplevel
>   with something like "let _ = start_application ()" but I have to call
>   "start_application();;" from the toplevel manually (or from the
>   .ocamlinit file), otherwise the application bindings are not
>   available.

Actually you can, you just have to make sure that .cmi files are
available in the search paths of the toplevel when you run your
application.

> - I cannot let the user uses the toplevel directives "use" and "load"
>   because both writes into stdout whatever the formatter passed to
>   Toploop.execute_phrase (for "use" this is easily solvable by shadowing
>   the toplevel implementation by another one that call Toploop.use_file
>   with my own formatter, but for "load" I would have to copy a lot of
>   code from topdirs.ml

Why not use Topdirs.dir_load with a custom formatter ?

If you are interested, i started some time ago an emacs-like editor
which can run in both curses and gtk mode and which integrates a
toplevel. I add to face the same problems. I solved them by using
Toploop.execute_phrase directly and redefining all directives.

The code is available here:

  http://solaria.dimino.org/cgi-bin/darcsweb.cgi?r=peps;a=summary

The file that may interest you is src/core/peps_top.ml

Also another problem i had is that Toploop.execute_phrase does not
prints errors on the given formatter but raises an exception instead,
and the printer used in Toploop.loop (Errors.report_error) is in the
module Errors for which the cmi is not available. I used a hack which
consists on parsing the output of Toploop.use_silently (file
peps_top.ml, function eval_command).

-- 
Jérémie

Re: [Caml-list] Calling the toplevel from the toplevel

[rixed]

> Actually you can, you just have to make sure that .cmi files are
> available in the search paths of the toplevel when you run your
> application.

Yes, it works now. AFAICT the cmi have always been available,
but anyway...

> > - I cannot let the user uses the toplevel directives "use" and "load"
> >   because both writes into stdout whatever the formatter passed to
> >   Toploop.execute_phrase (for "use" this is easily solvable by shadowing
> >   the toplevel implementation by another one that call Toploop.use_file
> >   with my own formatter, but for "load" I would have to copy a lot of
> >   code from topdirs.ml
> 
> Why not use Topdirs.dir_load with a custom formatter ?

Because I had not noticed that these functions were available.
This is handy. So I end up with something similar to you, except
I don't shadow that much directives (only those I know and use
regularly, and I'm not very experienced with the toplevel to say
the least).

> If you are interested, i started some time ago an emacs-like editor
> which can run in both curses and gtk mode and which integrates a
> toplevel.

Haha!, this is the very same reason why I was asking these questions.
Except that my project is just a small pet project consisting only
of less than 1k LOC, and I hope it will stay that way. I'm just used
to mix C and scheme to use the scheme repl to manipulate the running
C app and wanted to know how easy/hard it would be to have the same
"hackability" with Ocaml.

> I add to face the same problems. I solved them by using
> Toploop.execute_phrase directly and redefining all directives.

I've seen. We use very similar approach.
The funny thing is that when I started I was convinced that it was
not possible to call the Toploop evaluator from within itself, and
tried many other solutions, until I noticed that topfind.ml actually
does this to enrich the toplevel directives.

> Also another problem i had is that Toploop.execute_phrase does not
> prints errors on the given formatter but raises an exception instead,
> and the printer used in Toploop.loop (Errors.report_error) is in the
> module Errors for which the cmi is not available.

I give the exception to Toploop.print_exception_outcome :

        let top_eval cmd =
            let cmd = Rope.to_string cmd in
            Log.p "Executing '%s'" cmd ;
            try
                let l = Lexing.from_string cmd in
                let ph = !Toploop.parse_toplevel_phrase l in
                Toploop.execute_phrase true self#formatter ph
            with exn ->
                Toploop.print_exception_outcome self#formatter exn ;
                false in

but the exceptions are only pretty printed to a depth of 1, further
values being replaced by "_", so it's far from perfect.

> I used a hack which
> consists on parsing the output of Toploop.use_silently (file
> peps_top.ml, function eval_command).

Ooooh, I see! I'd never come up with this idea.

BTW, do you have any future plan for peps?

Re: [Caml-list] Calling the toplevel from the toplevel

[Jérémie Dimino]

On Mon, Feb 28, 2011 at 07:28:13PM +0100, rixed@happyleptic.org wrote:
> BTW, do you have any future plan for peps?

Yes. There is still work to do on the 'core' part and on the GTK ui,
then i will terminate the ocaml mode and plugins for syntax extensions
and then i will do a first release (and use it for writing my programs).

-- 
Jérémie

Re: [Caml-list] Calling the toplevel from the toplevel

[Yoann Padioleau]

On Feb 28, 2011, at 12:30 PM, Jérémie Dimino wrote:

> On Mon, Feb 28, 2011 at 07:28:13PM +0100, rixed@happyleptic.org wrote:
>> BTW, do you have any future plan for peps?
> 
> Yes. There is still work to do on the 'core' part and on the GTK ui,
> then i will terminate the ocaml mode and plugins for syntax extensions
> and then i will do a first release (and use it for writing my programs).

Could be interesting to integrate peps with codemap, a code browser for large
codebase: https://github.com/facebook/pfff/wiki/CodeMap
> 
> -- 
> Jérémie
> 
> -- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 

Re: [Caml-list] Calling the toplevel from the toplevel

[rixed]

> Actually you can, you just have to make sure that .cmi files are
> available in the search paths of the toplevel when you run your
> application.

Now I think I'm seeing what you was refering to.
This problem seams to be general to ocaml. For instance, if I have
a directory foo with object files bar.cmo and cmi in it, then from
ocaml:

# #load "foo/bar.cmo";;
# module X = Bar;;
Error: Unbound module Bar

While if I copy bar.cmi into the current directory the module Bar is
bound.

I find it weird that the cmi is not looked for in the same directory
where the cmo was found. Is there a way to change this ?