[Caml-list] Go Oracle like facility for Ocaml?

[Caml-list] Go Oracle like facility for Ocaml?

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]

If anybody knows what Go's oracle is you'll know that its a great
accelerator for your time; it allows expressive and meaningful searches to
be done over a source repository. It's fast and dead useful. Opengrok is
much the same, but to a lesser extent (having links is nice, but not quite
as powerful as oracle, I could be wrong).

Is there anything like this for OCaml?

[-- Attachment #2: Type: text/html, Size: 429 bytes --]

Re: [Caml-list] Go Oracle like facility for Ocaml?

[Gabriel Scherer]

(One can find a description of Go oracle's design in
  https://docs.google.com/document/d/1WmMHBUjQiuy15JfEnT8YBROQmEv-7K6bV-Y_K53oi5Y/view
and its user manual in
  https://docs.google.com/document/d/1SLk36YRjjMgKqe490mSRzOPYEDe0Y_WQNRv-EiFYUyw/view#heading=h.kthq8ap0mdwi
)

The ecosystem of OCaml tooling is not as refined as Go's (but
contributions are welcome). There is no centralized tool provider with
a common interface, but several contributors have developped separate
tool to anayze different aspects of OCaml programs:

- ocamlspotter: https://bitbucket.org/camlspotter/ocamlspot
- ocp-index: http://typerex.ocamlpro.com/ocp-index.html
- pfff: https://github.com/facebook/pfff
- merlin: http://the-lambda-church.github.io/merlin/

These tool provide a relatively complete coverage of the information
that can easily be retrieved from the typedtree of a program (>=4.01
versions of the OCaml compiler have the option to generate a reified
typedtree for external tools): the occurences of a declared/defined
name, the definition place of a name, the type of an expression, etc.
As far as I'm aware, there is not much in the direction of the more
advanced static analysis feature Go's oracle supports: points-to
information, "who may update this mutable field", etc. I'm not
familiar with Pfff's capabilities, it may be the more advanced in this
regard.

(There is also more experimental work going on, for example Thomas
Blanc's work on static analysis of exception flow at OCamlPro:
https://github.com/OCamlPro/socaml-analyzer )

I think merlin is the best-positioned tool to deal with
partially-incorrect files (typical of an edition session) and
incrementality. It also incorporates some query/analysis feature, but
it's unclear whether those should grow inside a monolithic tool (eg.
it could encompass the current feature set of ocp-index and
ocamlspotter, if it does not already), or rather try to communicate
with external analysis/query plugins. It also interacts with existing
editors through a reasonable query-answer interface, but does not
provide a direct command-line interface (anyone interested in this
could work on it, it may be relatively easy to implement).

There are fairly orthogonal aspects to a "answering questions about
programs" toolbox, among which:
1) user-interface, interactive use, and interface with existing editors
2) support for incrementality and robustness under partially-incorrect files
3) knowledge of what the "project", or whole program, is; which
dependencies are required to understand the work? (build system
knowledge)
4) implementation of various program analyses and transformations

Is it possible to provide them in separate programs and have them
interact to form a useful whole? Or would it be easier, faster and
more robust to implement them all in a monolithic program? What are
the necessary interdependence between these aspects and what interface
should them provide to each other?

On Tue, Nov 18, 2014 at 6:00 AM, Kenneth Adam Miller
<kennethadammiller@gmail.com> wrote:
> If anybody knows what Go's oracle is you'll know that its a great
> accelerator for your time; it allows expressive and meaningful searches to
> be done over a source repository. It's fast and dead useful. Opengrok is
> much the same, but to a lesser extent (having links is nice, but not quite
> as powerful as oracle, I could be wrong).
>
> Is there anything like this for OCaml?

Re: [Caml-list] Go Oracle like facility for Ocaml?

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 3799 bytes --]

Wow, you did a fantastic job fielding this question. I only even found one
of those tools, OCamlspotter and some etags, otags and cscope
functionality! :)

On Tue, Nov 18, 2014 at 4:06 AM, Gabriel Scherer <gabriel.scherer@gmail.com>
wrote:

> (One can find a description of Go oracle's design in
>
> https://docs.google.com/document/d/1WmMHBUjQiuy15JfEnT8YBROQmEv-7K6bV-Y_K53oi5Y/view
> and its user manual in
>
> https://docs.google.com/document/d/1SLk36YRjjMgKqe490mSRzOPYEDe0Y_WQNRv-EiFYUyw/view#heading=h.kthq8ap0mdwi
> )
>
> The ecosystem of OCaml tooling is not as refined as Go's (but
> contributions are welcome). There is no centralized tool provider with
> a common interface, but several contributors have developped separate
> tool to anayze different aspects of OCaml programs:
>
> - ocamlspotter: https://bitbucket.org/camlspotter/ocamlspot
> - ocp-index: http://typerex.ocamlpro.com/ocp-index.html
> - pfff: https://github.com/facebook/pfff
> - merlin: http://the-lambda-church.github.io/merlin/
>
> These tool provide a relatively complete coverage of the information
> that can easily be retrieved from the typedtree of a program (>=4.01
> versions of the OCaml compiler have the option to generate a reified
> typedtree for external tools): the occurences of a declared/defined
> name, the definition place of a name, the type of an expression, etc.
> As far as I'm aware, there is not much in the direction of the more
> advanced static analysis feature Go's oracle supports: points-to
> information, "who may update this mutable field", etc. I'm not
> familiar with Pfff's capabilities, it may be the more advanced in this
> regard.
>
> (There is also more experimental work going on, for example Thomas
> Blanc's work on static analysis of exception flow at OCamlPro:
> https://github.com/OCamlPro/socaml-analyzer )
>
> I think merlin is the best-positioned tool to deal with
> partially-incorrect files (typical of an edition session) and
> incrementality. It also incorporates some query/analysis feature, but
> it's unclear whether those should grow inside a monolithic tool (eg.
> it could encompass the current feature set of ocp-index and
> ocamlspotter, if it does not already), or rather try to communicate
> with external analysis/query plugins. It also interacts with existing
> editors through a reasonable query-answer interface, but does not
> provide a direct command-line interface (anyone interested in this
> could work on it, it may be relatively easy to implement).
>
> There are fairly orthogonal aspects to a "answering questions about
> programs" toolbox, among which:
> 1) user-interface, interactive use, and interface with existing editors
> 2) support for incrementality and robustness under partially-incorrect
> files
> 3) knowledge of what the "project", or whole program, is; which
> dependencies are required to understand the work? (build system
> knowledge)
> 4) implementation of various program analyses and transformations
>
> Is it possible to provide them in separate programs and have them
> interact to form a useful whole? Or would it be easier, faster and
> more robust to implement them all in a monolithic program? What are
> the necessary interdependence between these aspects and what interface
> should them provide to each other?
>
> On Tue, Nov 18, 2014 at 6:00 AM, Kenneth Adam Miller
> <kennethadammiller@gmail.com> wrote:
> > If anybody knows what Go's oracle is you'll know that its a great
> > accelerator for your time; it allows expressive and meaningful searches
> to
> > be done over a source repository. It's fast and dead useful. Opengrok is
> > much the same, but to a lesser extent (having links is nice, but not
> quite
> > as powerful as oracle, I could be wrong).
> >
> > Is there anything like this for OCaml?
>

[-- Attachment #2: Type: text/html, Size: 5110 bytes --]

Re: [Caml-list] Go Oracle like facility for Ocaml?

[yoann padioleau]

[-- Attachment #1: Type: text/plain, Size: 5118 bytes --]

Hi,

Regarding pfff that gabriel mentioned: some of my recent work have been focused actually on reproducing some of the features
of go oracles (a great tool), but mainly for C. The idea was to use datalog to declaratively specify complex inter procedural data flow analysis: https://github.com/facebook/pfff/blob/master/h_program-lang/datalog_code.dtl 
and
https://github.com/facebook/pfff/blob/master/lang_c/analyze/datalog_c.ml

I was able to scale this analysis using bddbddb to C programs of 100 000 lines (the plan9 kernel mostly).

I would like to extend this approach to data/control flow analysis of ocaml, so one can know for instance where a function can be called from (a tricky problem when
your code becomes more complex and you start
returning functions, encapsulate functions in wrappers,
store function in fields, etc).

pfff has some good support for ocaml via the typed tree
and can generate a “code graph” 
https://github.com/facebook/pfff/wiki/CodeGraph
https://github.com/facebook/pfff/blob/master/lang_cmt/analyze/graph_code_cmt.ml
but I have not yet written the datalog fact generator for ocaml. Contributions are welcome though :)

On Nov 18, 2014, at 1:06 AM, Gabriel Scherer <gabriel.scherer@gmail.com> wrote:

> (One can find a description of Go oracle's design in
>  https://docs.google.com/document/d/1WmMHBUjQiuy15JfEnT8YBROQmEv-7K6bV-Y_K53oi5Y/view
> and its user manual in
>  https://docs.google.com/document/d/1SLk36YRjjMgKqe490mSRzOPYEDe0Y_WQNRv-EiFYUyw/view#heading=h.kthq8ap0mdwi
> )
> 
> The ecosystem of OCaml tooling is not as refined as Go's (but
> contributions are welcome). There is no centralized tool provider with
> a common interface, but several contributors have developped separate
> tool to anayze different aspects of OCaml programs:
> 
> - ocamlspotter: https://bitbucket.org/camlspotter/ocamlspot
> - ocp-index: http://typerex.ocamlpro.com/ocp-index.html
> - pfff: https://github.com/facebook/pfff
> - merlin: http://the-lambda-church.github.io/merlin/
> 
> These tool provide a relatively complete coverage of the information
> that can easily be retrieved from the typedtree of a program (>=4.01
> versions of the OCaml compiler have the option to generate a reified
> typedtree for external tools): the occurences of a declared/defined
> name, the definition place of a name, the type of an expression, etc.
> As far as I'm aware, there is not much in the direction of the more
> advanced static analysis feature Go's oracle supports: points-to
> information, "who may update this mutable field", etc. I'm not
> familiar with Pfff's capabilities, it may be the more advanced in this
> regard.
> 
> (There is also more experimental work going on, for example Thomas
> Blanc's work on static analysis of exception flow at OCamlPro:
> https://github.com/OCamlPro/socaml-analyzer )
> 
> I think merlin is the best-positioned tool to deal with
> partially-incorrect files (typical of an edition session) and
> incrementality. It also incorporates some query/analysis feature, but
> it's unclear whether those should grow inside a monolithic tool (eg.
> it could encompass the current feature set of ocp-index and
> ocamlspotter, if it does not already), or rather try to communicate
> with external analysis/query plugins. It also interacts with existing
> editors through a reasonable query-answer interface, but does not
> provide a direct command-line interface (anyone interested in this
> could work on it, it may be relatively easy to implement).
> 
> There are fairly orthogonal aspects to a "answering questions about
> programs" toolbox, among which:
> 1) user-interface, interactive use, and interface with existing editors
> 2) support for incrementality and robustness under partially-incorrect files
> 3) knowledge of what the "project", or whole program, is; which
> dependencies are required to understand the work? (build system
> knowledge)
> 4) implementation of various program analyses and transformations
> 
> Is it possible to provide them in separate programs and have them
> interact to form a useful whole? Or would it be easier, faster and
> more robust to implement them all in a monolithic program? What are
> the necessary interdependence between these aspects and what interface
> should them provide to each other?
> 
> On Tue, Nov 18, 2014 at 6:00 AM, Kenneth Adam Miller
> <kennethadammiller@gmail.com> wrote:
>> If anybody knows what Go's oracle is you'll know that its a great
>> accelerator for your time; it allows expressive and meaningful searches to
>> be done over a source repository. It's fast and dead useful. Opengrok is
>> much the same, but to a lesser extent (having links is nice, but not quite
>> as powerful as oracle, I could be wrong).
>> 
>> Is there anything like this for OCaml?
> 
> -- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs


[-- Attachment #2: Type: text/html, Size: 6676 bytes --]