[ANN] OCamlSpotter: OCaml compiler extension for source browsing, version 1.0

[ANN] OCamlSpotter: OCaml compiler extension for source browsing, version 1.0

[Jun Furuse]

Hi, OCaml programmers,

Here is your free holiday gift, OCamlSpotter version 1.0, a small
patch for OCaml 3.11.0.

OCamlSpotter : OCaml source browsing
====================================

  Camel spotter: Oh, it was extremely interesting, very, very -
    quite... it was dull; dull, dull, dull, oh God it was dull.
    Sitting in the Waterloo waiting room. ...

                                     from You're No Fun Anymore,
                                    Monty Python's Flying Circus

OCaml module language is powerful. So extremely powerful that you can
be lost yourself easily inside a huge OCaml project with tons of modules.
Spotting the definition of a variable is sometimes "extremely interesting".
Let's see such an "exciting" example:

  include M
  open N
  open O
  let _ = f 1

Suppose you are not familiar with the function f and want to spot its
definition. Where to find it? Just from the code you are looking at,
it is not sure: if you are enough lucky it may be f in m.ml, n.ml or
o.ml. Otherwise, it may be O.f in n.ml. Or probably N.O.f in
m.ml. If you are unlucky and the project is so sophisticated, there could
be complex module operations in m.ml (i.e includes, functor applications)
and therefore the definition might be found somewhere completely unexpected.
Module packing, module load paths and library installation without .ml files
complicate the situation even worse.

You first days after joining a large OCaml project should be this kind
of manual value definition spotting with find + grep commands. Yes, actually
it is very educational: you can learn a lot about the project struggling
in the source tree (only if your collegues have organized things very
well :-P), but it is still a dull job...

To say short, OCamlSpotter is a tool which does this dull job automatically
for you and permits your energy for something more "interesting" like Yeti
spotting:

 - The -annot option of ocamlc and ocamlopt is extended and creates
   <module>.spot files (<module>.spit for .mli), which record the location
   information of the names defined and used in the module.

 - A small application ocamlspot provides automatic where-about spotting
   of the definition of the name you are interested in.

 - ocamlspot.el provides interactive ocaml-spotting of definition
   locations in emacs.

 - Interfaces for other editors such as vi can be built easily, if you want.

Donwload
=======
http://www.furuse.info/jun/hacks/ocamlspotter/ocamlspotter-1.0.tgz?attredirects=0

Have fun, and have a good new year.

Jun FURUSE

RE: [Caml-list] [ANN] OCamlSpotter: OCaml compiler extension for source browsing, version 1.0

[David Allsopp]

This sounds really cool - but is it really necessary to patch OCaml 3.11.0
to achieve it? The .annot files produced by OCaml 3.11 give external
reference information already...

e.g.

[Test.ml]
open List;;

rev [];;

# ocamlc -c -annot Test.ml

[Test.annot]
"Test.ml" 3 15 15 "Test.ml" 3 15 18
type(
  'a list -> 'a list
)
ident(
  ext_ref List.rev
)
<snip>

The second entry in the .annot file shows the full module path required for
the call to rev - taking into account all open, include statements and
giving the full path in terms of packed modules and so on.

Surely all you need once armed with the .annot file is the command line
which was used to compile the module so that you can translate the module
paths into actual files .cm* files (and then look for corresponding .ml
files) - the code for tools such as cmigrep (not to mention the compiler
itself) should contain everything necessary for analysing .cm[ox], .cmi and
.cm{x,}a files to turn a given module path into the actual file reference.

Capturing the command line used to make the .annot file would seem to me to
be a better job for ocamlfind - I would have thought that you're more likely
to get Gerd to accept a patch for a simple hook interface in findlib than to
get a patch which emits additional output files accepted by the OCaml team
(however, it might be worth raising a feature request for the OCaml team to
add an entry to .annot files which records the command line used in the
.annot file itself as I can't imagine that's particularly hard or a
maintenance problem)...

Happy Boxing Day, everyone!


David


> -----Original Message-----
> From: caml-list-bounces@yquem.inria.fr [mailto:caml-list-
> bounces@yquem.inria.fr] On Behalf Of Jun Furuse
> Sent: 25 December 2008 13:47
> To: caml-list@yquem.inria.fr
> Subject: [Caml-list] [ANN] OCamlSpotter: OCaml compiler extension for
> source browsing, version 1.0
> 
> Hi, OCaml programmers,
> 
> Here is your free holiday gift, OCamlSpotter version 1.0, a small
> patch for OCaml 3.11.0.
> 
> OCamlSpotter : OCaml source browsing
> ====================================
> 
>   Camel spotter: Oh, it was extremely interesting, very, very -
>     quite... it was dull; dull, dull, dull, oh God it was dull.
>     Sitting in the Waterloo waiting room. ...
> 
>                                      from You're No Fun Anymore,
>                                     Monty Python's Flying Circus
> 
> OCaml module language is powerful. So extremely powerful that you can
> be lost yourself easily inside a huge OCaml project with tons of
> modules.
> Spotting the definition of a variable is sometimes "extremely
> interesting".
> Let's see such an "exciting" example:
> 
>   include M
>   open N
>   open O
>   let _ = f 1
> 
> Suppose you are not familiar with the function f and want to spot its
> definition. Where to find it? Just from the code you are looking at,
> it is not sure: if you are enough lucky it may be f in m.ml, n.ml or
> o.ml. Otherwise, it may be O.f in n.ml. Or probably N.O.f in
> m.ml. If you are unlucky and the project is so sophisticated, there
> could
> be complex module operations in m.ml (i.e includes, functor
> applications)
> and therefore the definition might be found somewhere completely
> unexpected.
> Module packing, module load paths and library installation without .ml
> files
> complicate the situation even worse.
> 
> You first days after joining a large OCaml project should be this kind
> of manual value definition spotting with find + grep commands. Yes,
> actually
> it is very educational: you can learn a lot about the project
> struggling
> in the source tree (only if your collegues have organized things very
> well :-P), but it is still a dull job...
> 
> To say short, OCamlSpotter is a tool which does this dull job
> automatically
> for you and permits your energy for something more "interesting" like
> Yeti
> spotting:
> 
>  - The -annot option of ocamlc and ocamlopt is extended and creates
>    <module>.spot files (<module>.spit for .mli), which record the
> location
>    information of the names defined and used in the module.
> 
>  - A small application ocamlspot provides automatic where-about
> spotting
>    of the definition of the name you are interested in.
> 
>  - ocamlspot.el provides interactive ocaml-spotting of definition
>    locations in emacs.
> 
>  - Interfaces for other editors such as vi can be built easily, if you
> want.
> 
> Donwload
> =======
> http://www.furuse.info/jun/hacks/ocamlspotter/ocamlspotter-
> 1.0.tgz?attredirects=0
> 
> Have fun, and have a good new year.
> 
> Jun FURUSE
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] [ANN] OCamlSpotter: OCaml compiler extension for source browsing, version 1.0

[Jun Furuse]

Hi,

On Fri, Dec 26, 2008 at 8:15 PM, David Allsopp <dra-news@metastack.com> wrote:
> This sounds really cool - but is it really necessary to patch OCaml 3.11.0
> to achieve it? The .annot files produced by OCaml 3.11 give external
> reference information already...

> e.g.
>
> [Test.ml]
> open List;;
>
> rev [];;
>
> # ocamlc -c -annot Test.ml
>
> [Test.annot]
> "Test.ml" 3 15 15 "Test.ml" 3 15 18
> type(
>  'a list -> 'a list
> )
> ident(
>  ext_ref List.rev
> )
> <snip>

> The second entry in the .annot file shows the full module path required for
> the call to rev - taking into account all open, include statements and
> giving the full path in terms of packed modules and so on.

Sorry, but I do not think so:

  include List
  let g = length

ocamlc 3.11.0 with -annot never informs about length, included from
List. This is why a compiler patch is required at least for version
3.11.0.

Besides this version specific reason, annot is a text and therefore it
must be pretty-printed at creation and parsed back at the use. It is
costly and sometimes files become very huge (few MBs sometimes). In
addition, some very important properties for browsing such as the
identifier stamp integers are simply lost at pretty printing and never
be recovered at parsing. From these difficulties, rather than pretty
printed, I think those information should be simply marshaled.
Marshaled spot files are always smaller than annots, and loading is
straightforward and pretty fast.

> Surely all you need once armed with the .annot file is the command line
> which was used to compile the module so that you can translate the module
> paths into actual files .cm* files (and then look for corresponding .ml
> files) - the code for tools such as cmigrep (not to mention the compiler
> itself) should contain everything necessary for analysing .cm[ox], .cmi and
> .cm{x,}a files to turn a given module path into the actual file reference.

It is a very interesting approach but I did not take it, firstly
because this only works with compiled/compilable modules: it is often
very handy to browse sources with errors, even they are just partially
browsable. Secondly because I was (and am still) not very confident
whether cm* files contain all the information I need for spotter.

> Capturing the command line used to make the .annot file would seem to me to
> be a better job for ocamlfind - I would have thought that you're more likely
> to get Gerd to accept a patch for a simple hook interface in findlib than to
> get a patch which emits additional output files accepted by the OCaml team
> (however, it might be worth raising a feature request for the OCaml team to
> add an entry to .annot files which records the command line used in the
> .annot file itself as I can't imagine that's particularly hard or a
> maintenance problem)...

OCamlFind is a great tool, but there are many people who are happy
without it. I would rather like to take an effort to move the caml-dev
team to accept some part of my patch. :-) It is almost trivial: dump
things available at typing, removing references to functions, which
prevents output_value.

=
j

RE: [Caml-list] [ANN] OCamlSpotter: OCaml compiler extension for source browsing, version 1.0

[David Allsopp]

Jun Furuse wrote:
> <snip>
> 
> Sorry, but I do not think so:
> 
>   include List
>   let g = length
> 
> ocamlc 3.11.0 with -annot never informs about length, included from
> List. This is why a compiler patch is required at least for version
> 3.11.0.

This is a bug and should be reported in Mantis - [length] comes from an
external file so there should be an ext_ref annotation in the .annot file.

> > Besides this version specific reason, annot is a text and therefore it
> > must be pretty-printed at creation and parsed back at the use. It is
> > costly and sometimes files become very huge (few MBs sometimes). In
> > addition, some very important properties for browsing such as the
> > identifier stamp integers are simply lost at pretty printing and never
> > be recovered at parsing. From these difficulties, rather than pretty
> > printed, I think those information should be simply marshaled.
> > Marshaled spot files are always smaller than annots, and loading is
> > straightforward and pretty fast.

Define costly! I've never noticed Vim's interpreted, probably non-optimised
internal script language taking a significant amount of time to process the
.annot files (many of which are several megabytes) for all of my sources
every time I open the corresponding .ml file.

> > It is a very interesting approach but I did not take it, firstly
> > because this only works with compiled/compilable modules: it is often
> > very handy to browse sources with errors, even they are just partially
> > browsable. Secondly because I was (and am still) not very confident
> > whether cm* files contain all the information I need for spotter.

How much have you changed in the compiler? While compiling foo.ml, all
external references are already compiled (or you'll get an error that the
.cmi file for an interface is missing). The compiler also gives as much info
in the .annot file for sources which contain errors already. Surely the .cmi
file must contain enough information virtually by definition - it's all the
compiler had to go on in the first place!

> > OCamlFind is a great tool, but there are many people who are happy
> > without it.

My impression reading this list is that the majority of the community would
like it included in the main OCaml distribution but let's not re-open that
thread of discussion again ;o) If someone's willing to install a tool like
ocamlspot involving, at present, a patched compiler then I'd imagine that
they'd not have too much issue installing findlib too...


David