Extending .annot files

Extending .annot files

[Nathaniel Gray]

On Thu, Sep 18, 2008 at 10:02 AM, Xavier Leroy <Xavier.Leroy@inria.fr> wrote:
>
> From what I've heard, there's also an OCaml summer of code project
> that enriched the info found in .annot files.  So, it's certainly time
> to discuss extensions to .annot files, but let's do that globally, not
> one at a time.  It is probably too late for inclusion in 3.11, but as
> long as these extensions are backward compatible, inclusion in bugfix
> releases can be considered.

I'm really happy to hear that you're open to including some of this
stuff.  I think there are actually only a few data that one wants to
have in .annot files (and that the compiler can reasonably provide).

For any identifier it would be good to know:
1. Its inferred type
2. Its fully-qualified module "path"
3. Where it was defined, if it was defined in the current file.

In addition, for each module referenced in the file it would be good
to know what file the module was read from.  (This will allow some
hope of tracking down definitions in other files)

It's hard for me to think of anything else that belongs in .annot
files.  If I stretch a bit I suppose annotating variable definitions
with their range of scope might be cute.  Maybe other people can think
of more original ideas.

Finally, it may be worth putting a little work into reducing the size
of .annot files.  One could certainly do much better with very little
effort.

Cheers,
-n8

-- 
>>>-- Nathaniel Gray -- Caltech Computer Science ------>
>>>-- Mojave Project -- http://mojave.cs.caltech.edu -->

Re: Extending .annot files

[Jun Furuse]

Hi,

> I'm really happy to hear that you're open to including some of this
> stuff.  I think there are actually only a few data that one wants to
> have in .annot files (and that the compiler can reasonably provide).
>
> For any identifier it would be good to know:
> 1. Its inferred type
> 2. Its fully-qualified module "path"
> 3. Where it was defined, if it was defined in the current file.
>
> In addition, for each module referenced in the file it would be good
> to know what file the module was read from.  (This will allow some
> hope of tracking down definitions in other files)

In addition, ocamlspot records the following:
- compilation load paths to find referred modules
- abstractions of internally defined modules to find definitions of
variables obtained from module aliases and functor applications
- some hack for packed modules

> It's hard for me to think of anything else that belongs in .annot
> files.  If I stretch a bit I suppose annotating variable definitions
> with their range of scope might be cute.  Maybe other people can think
> of more original ideas.

Many things: constructor/type/exception/module definition locations,
corresponding .mli entry positions and so on. Some of them, especially
type related things, require compiler side support definitely, but the
others could be done by CamlP4 or something similar, I guess. I
personally like to modify the compiler since all the interesting
information is available at compilation for free and all we need is
just to export it to some file, in an extensible format. Probably in a
text file like the current annot, or as a marshaled value like
ocamlspot does. Marshalled values are nice since we do not need to
write parsers for them, and we can use lots of tool functions in the
compiler to handle them. But of course, it is more difficult to keep
backward compatibility, which Xavier is anxious about.

> Finally, it may be worth putting a little work into reducing the size
> of .annot files.  One could certainly do much better with very little
> effort.

The size of the current annot files are more than 600% larger than the
source ml files (I measured this by compiling ocaml compiler tree with
-annot). Gzipping them makes them 1/10 (about 60% of the source size).
However, the compiler should not rely on any external compression tool
is available by default. You can easily add few rules to your OCaml
project Makefiles to gzip annots automatically, and I guess it would
be also easy to extend caml-types.el to support gzipped annot files.
Still, it is possible to reduce the size of annotation files, by
outputting as marshaled value, which makes things 35%, still more than
twice larger than the source :-(.

=
j

Re: [Caml-list] Extending .annot files

[Maxence Guesdon]

On Fri, 19 Sep 2008 11:20:36 -0700
"Nathaniel Gray" <n8gray@gmail.com> wrote:

> On Thu, Sep 18, 2008 at 10:02 AM, Xavier Leroy <Xavier.Leroy@inria.fr> wrote:
> >
> > From what I've heard, there's also an OCaml summer of code project
> > that enriched the info found in .annot files.  So, it's certainly time
> > to discuss extensions to .annot files, but let's do that globally, not
> > one at a time.  It is probably too late for inclusion in 3.11, but as
> > long as these extensions are backward compatible, inclusion in bugfix
> > releases can be considered.
> 
> I'm really happy to hear that you're open to including some of this
> stuff.  I think there are actually only a few data that one wants to
> have in .annot files (and that the compiler can reasonably provide).
> 
> For any identifier it would be good to know:
> 1. Its inferred type
> 2. Its fully-qualified module "path"

All identifiers have not a fully qualified path (think of y in
  let x = let y = ... in ...) and two identifiers may have the same
fully-qualified path. (e.g.: let x = 1;; let x = 2;;)

> 3. Where it was defined, if it was defined in the current file.

Would it be possible to have the location of the definition of all
identifiers, including the ones defined in another module (if there is
a .annot file for this module, of course) ?

> In addition, for each module referenced in the file it would be good
> to know what file the module was read from.  (This will allow some
> hope of tracking down definitions in other files)
> 
> It's hard for me to think of anything else that belongs in .annot
> files.  If I stretch a bit I suppose annotating variable definitions
> with their range of scope might be cute.  Maybe other people can think
> of more original ideas.

A module containing the functions to read .annot files, so that
all developers using these files do not develop their own module.

Regards,


-- 
Maxence Guesdon                           http://yquem.inria.fr/~guesdon/
Service Expérimentation et Développements https://devel.inria.fr/rocq/
INRIA Paris-Rocquencourt                  http://www.inria.fr/rocquencourt/

Re: [Caml-list] Re: Extending .annot files

[Nathaniel Gray]

On Sun, Sep 21, 2008 at 7:15 PM, Jun Furuse <jun.furuse@gmail.com> wrote:
> Hi,
>
>> I'm really happy to hear that you're open to including some of this
>> stuff.  I think there are actually only a few data that one wants to
>> have in .annot files (and that the compiler can reasonably provide).
>>
>> For any identifier it would be good to know:
>> 1. Its inferred type
>> 2. Its fully-qualified module "path"
>> 3. Where it was defined, if it was defined in the current file.
>>
>> In addition, for each module referenced in the file it would be good
>> to know what file the module was read from.  (This will allow some
>> hope of tracking down definitions in other files)
>
> In addition, ocamlspot records the following:
> - compilation load paths to find referred modules

Is this different from my last sentence?  I'm not sure if you mean the
path to the module file or the module search path.  I would personally
rather have the path to the specific file for each module so that the
client doesn't have to write code for searching out modules.

> - abstractions of internally defined modules to find definitions of
> variables obtained from module aliases and functor applications

Are there cases where you can't figure this out by following back the
chain of definitions?

> - some hack for packed modules
>
>> It's hard for me to think of anything else that belongs in .annot
>> files.  If I stretch a bit I suppose annotating variable definitions
>> with their range of scope might be cute.  Maybe other people can think
>> of more original ideas.
>
> Many things: constructor/type/exception/module definition locations,

Sure, by using the word "identifiers" above I meant to include these entities.

> corresponding .mli entry positions and so on.

That's a good idea, assuming the compiler has this information at the
right time.

> Some of them, especially
> type related things, require compiler side support definitely, but the
> others could be done by CamlP4 or something similar, I guess. I
> personally like to modify the compiler since all the interesting
> information is available at compilation for free and all we need is
> just to export it to some file, in an extensible format. Probably in a
> text file like the current annot, or as a marshaled value like
> ocamlspot does. Marshalled values are nice since we do not need to
> write parsers for them, and we can use lots of tool functions in the
> compiler to handle them. But of course, it is more difficult to keep
> backward compatibility, which Xavier is anxious about.

Marshalled values are awful for .annot files because only OCaml code
can read them without lots of effort.  I'd venture a guess that most
people want to exploit .annot files in their text editors.  My text
editor isn't (yet) written in OCaml, and I know I'm not the only one.

>> Finally, it may be worth putting a little work into reducing the size
>> of .annot files.  One could certainly do much better with very little
>> effort.
>
> The size of the current annot files are more than 600% larger than the
> source ml files (I measured this by compiling ocaml compiler tree with
> -annot). Gzipping them makes them 1/10 (about 60% of the source size).
> However, the compiler should not rely on any external compression tool
> is available by default. You can easily add few rules to your OCaml
> project Makefiles to gzip annots automatically, and I guess it would
> be also easy to extend caml-types.el to support gzipped annot files.

I recently pointed out this very fact about gzipping .annot files, but
if we're putting together a wish list for .annot file changes I would
still like to see them be less bloated by default.

Cheers,
-n8

-- 
>>>-- Nathaniel Gray -- Caltech Computer Science ------>
>>>-- Mojave Project -- http://mojave.cs.caltech.edu -->

Re: [Caml-list] Extending .annot files

[Nathaniel Gray]

On Sun, Sep 21, 2008 at 11:08 PM, Maxence Guesdon
<maxence.guesdon@inria.fr> wrote:
> On Fri, 19 Sep 2008 11:20:36 -0700
> "Nathaniel Gray" <n8gray@gmail.com> wrote:
>>
>> For any identifier it would be good to know:
>> 1. Its inferred type
>> 2. Its fully-qualified module "path"
>
> All identifiers have not a fully qualified path (think of y in
>  let x = let y = ... in ...)

That's a good point.  I tend to think of those variables as being
private parts of the current module, but perhaps that's inaccurate.
So we can amend this request to be for "its fully qualified module
path, if such a path exists".

> and two identifiers may have the same
> fully-qualified path. (e.g.: let x = 1;; let x = 2;;)

That's fine.  Nobody said it had to be unique.  :)

>> 3. Where it was defined, if it was defined in the current file.
>
> Would it be possible to have the location of the definition of all
> identifiers, including the ones defined in another module (if there is
> a .annot file for this module, of course) ?

This strikes me as being beyond the scope of what the compiler can
reasonably provide.  If we're given the info that:

1. x resolves to Foo.Bar.x
2. Foo.Bar came from file /home/n8gray/src/foo/bar.cmo

We can write the code that decides whether/how to find bar.annot,
since that will be very build-system dependent.

>> In addition, for each module referenced in the file it would be good
>> to know what file the module was read from.  (This will allow some
>> hope of tracking down definitions in other files)
>>
>> It's hard for me to think of anything else that belongs in .annot
>> files.  If I stretch a bit I suppose annotating variable definitions
>> with their range of scope might be cute.  Maybe other people can think
>> of more original ideas.
>
> A module containing the functions to read .annot files, so that
> all developers using these files do not develop their own module.

I agree, but this is something that can be developed outside the INRIA
team if necessary.

Cheers,
-n8

-- 
>>>-- Nathaniel Gray -- Caltech Computer Science ------>
>>>-- Mojave Project -- http://mojave.cs.caltech.edu -->

Re: [Caml-list] Re: Extending .annot files

[Jun Furuse]

Hi,

>> In addition, ocamlspot records the following:
>> - compilation load paths to find referred modules
>
> Is this different from my last sentence?  I'm not sure if you mean the
> path to the module file or the module search path.  I would personally
> rather have the path to the specific file for each module so that the
> client doesn't have to write code for searching out modules.

Yeah, if we recode module file path names, search paths are no longer
required. But it is very easy to find annotation files in search
paths, only if our external tool is written in OCaml and uses compiler
modules. (And I explain why the first external tool should be written
in OCaml later.)

BTW, having module search paths and compilation options in the
annotation files opens a door to more evil hacks: possibility of easy
flymake style recompilation of source files without accessing
Makefiles. :-)

>> - abstractions of internally defined modules to find definitions of
>> variables obtained from module aliases and functor applications
>
> Are there cases where you can't figure this out by following back the
> chain of definitions?

Yes, they are required for definition point analysis at queries.

> Marshalled values are awful for .annot files because only OCaml code
> can read them without lots of effort.  I'd venture a guess that most
> people want to exploit .annot files in their text editors.  My text
> editor isn't (yet) written in OCaml, and I know I'm not the only one.

I am not enforcing you to write any parser of marshalled values in
your editor extension language :-) I cannot do it!
Instead, you can easily write a small OCaml program to dump it to
whatever friendly to you and your editor, say, s-exp for emacs, or XML
for some crazy editor. Then it should be easier to write a parser for
the dump in your editor language than writing a parser for some fixed
textual format which may not match with your editor language.

I push marshalled values also since I have a plan to use recorded type
expressions in annotations like ocamlbrowser. Of course marshalled
values have downsides: annotation files become compiler version
dependent. But I think we can avoid this problem somehow carefully
choosing data types we output_value, as far as the compiler keeps the
format of marshalled values and changes at type expression structures
are small.

We are discussing an official OCaml compiler extension in future, and
I think it is very important to keep such an extensions small as
possible; it is natural that INRIA people (and any other compiler
developer) reject any extension not maintainable by themselves or
something which adds extra library or tool requirements to the
compiler itself. Therefore our extension should write down things
available for free at compile time to the annotation files very
simply.
Any other work required can be done by external tools such as
ocamlspot: definition location analysis over modules, compression,
checking annotation file time stamps, and dumping them to editor
friendly formats, and so on.

I wrote ocamlspot in the above way, but some are still very ad-hoc.
Upcoming beta2 version will be cleaner and have an example of textual
dump.

Regards,
=
j