[Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[Francois Berenger]

Hello,

Is there some recommended tool/script to generate a .mli
from the corresponding .ml?

I want a little more than ocamlc -i:

- I think there should be tags in the .ml file as comments
   that say "export this" to the .mli.
   By default, things are not exported.
- maybe it should have an option to say to replicate
   the ocamldoc comments in the .mli.
- it could be nice if the order in which things are exported
   to the .mli can be specified, maybe as an argument of the tag.
   So that the .mli can be more readable (only backward references
   to concepts, etc.)

If there is a need to create a tool, let's call it "nomli". :)

Regards,
F.

PS: I'm not going to maintain both a .mli and a .ml.
     I feel it is a dumb and error-prone job and that
     itches me.

Re: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[Malcolm Matalka]

I know of no such tool, but in counter to your premise: I used to think
maintaining a .ml and .mli was foolish, however I no longer do.  .mli is
effectively documentation for me.  It contains a lot of comments and is
generally written to reflect how the API should be used rather than the
order in which I must express functions to get ta .ml to compile.  On
top of that, ocamlc will fail to compile if your .ml and .mli don't
match, so it's a valuable check that what I think my module does is also
what the compiler does.  I also tend to write the .mli first, then write
the .ml.  I find it to be a great way to develop.

In short, I think it's a good thing to maintain these things by hand.
But as for your original question I'm completely useless, sorry.

/M

Francois Berenger <berenger@riken.jp> writes:

> Hello,
>
> Is there some recommended tool/script to generate a .mli
> from the corresponding .ml?
>
> I want a little more than ocamlc -i:
>
> - I think there should be tags in the .ml file as comments
>   that say "export this" to the .mli.
>   By default, things are not exported.
> - maybe it should have an option to say to replicate
>   the ocamldoc comments in the .mli.
> - it could be nice if the order in which things are exported
>   to the .mli can be specified, maybe as an argument of the tag.
>   So that the .mli can be more readable (only backward references
>   to concepts, etc.)
>
> If there is a need to create a tool, let's call it "nomli". :)
>
> Regards,
> F.
>
> PS: I'm not going to maintain both a .mli and a .ml.
>     I feel it is a dumb and error-prone job and that
>     itches me.

Re: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[Francois Berenger]

On 05/31/2013 02:31 PM, Malcolm Matalka wrote:
> I know of no such tool, but in counter to your premise: I used to think
> maintaining a .ml and .mli was foolish, however I no longer do.

It itches me and it won't stop.

>  .mli is
> effectively documentation for me.  It contains a lot of comments and is
> generally written to reflect how the API should be used rather than the
> order in which I must express functions to get ta .ml to compile.

I'm proposing to add some number to tags so that things can
be reordered when dumped to the .mli.

>  On
> top of that, ocamlc will fail to compile if your .ml and .mli don't
> match,

The tool I am thinking about is a preprocessor.
It would be used to generate all .mli files before the compilation
of the .ml ones (at least in _MY_ OCaml projects).

 > so it's a valuable check that what I think my module does is also
> what the compiler does.  I also tend to write the .mli first, then write
> the .ml.  I find it to be a great way to develop.

I don't.

> In short, I think it's a good thing to maintain these things by hand.

I don't:
dumb_job + error_prone + can_be_automated + duplication_of_information 
--> the computer will do it for me.

I'm thinking about something small.
It may not work on all cases people can come up with.
I don't do very elaborated things usually so it will work for me,
at least.

> But as for your original question I'm completely useless, sorry.
>
> /M
>
> Francois Berenger <berenger@riken.jp> writes:
>
>> Hello,
>>
>> Is there some recommended tool/script to generate a .mli
>> from the corresponding .ml?
>>
>> I want a little more than ocamlc -i:
>>
>> - I think there should be tags in the .ml file as comments
>>    that say "export this" to the .mli.
>>    By default, things are not exported.
>> - maybe it should have an option to say to replicate
>>    the ocamldoc comments in the .mli.
>> - it could be nice if the order in which things are exported
>>    to the .mli can be specified, maybe as an argument of the tag.
>>    So that the .mli can be more readable (only backward references
>>    to concepts, etc.)
>>
>> If there is a need to create a tool, let's call it "nomli". :)
>>
>> Regards,
>> F.
>>
>> PS: I'm not going to maintain both a .mli and a .ml.
>>      I feel it is a dumb and error-prone job and that
>>      itches me.
>

Re: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[Romain Bardou]

I also used to believe the .mli file should be somehow included in the
.ml file. Now I don't.

But if you design such a tool, here are some things to consider.

- Using numbers to order stuff creates some issues of scalability. In
particular, inserting a declaration between #3 and #4 may require you to
move #4 to #5, #5 to #6 and so on, which (in particular if everything is
not in the right order in the implementation) can prove more annoying
than reordering stuff in an .mli.

- Even without numbers it will be harder to get the big picture from a
module. An approach I like about this is Pascal units, with an interface
section and an implementation section, in the same file. But it does not
solve the duplication issue.

- You need a way to specify abstraction. For instance:
  * specify that a type should be exported as "type t" instead of "type
t = A | B";
  * specify that a type should be exported as "type t = private A | B"
instead of "type t = A | B";
  * specify that a value should be exported as "val f: t -> t" instead
of "val f: int -> int"
  * specify that a value should be exported as "val f: int -> t" instead
of "val f: 'a -> 'a"
Often, people who want automatic .mli generation do not use abstraction,
which is the main point of interfaces.

- Don't forget that you need to take into account the whole OCaml
language, including objects, modules, polymorphic variants… It may be
more work than expected. And this project will need to be maintained and
follow the changes to the language.

Cheers,

-- 
Romain

Le 31/05/2013 08:26, Francois Berenger a écrit :
> On 05/31/2013 02:31 PM, Malcolm Matalka wrote:
>> I know of no such tool, but in counter to your premise: I used to think
>> maintaining a .ml and .mli was foolish, however I no longer do.
> 
> It itches me and it won't stop.
> 
>>  .mli is
>> effectively documentation for me.  It contains a lot of comments and is
>> generally written to reflect how the API should be used rather than the
>> order in which I must express functions to get ta .ml to compile.
> 
> I'm proposing to add some number to tags so that things can
> be reordered when dumped to the .mli.
> 
>>  On
>> top of that, ocamlc will fail to compile if your .ml and .mli don't
>> match,
> 
> The tool I am thinking about is a preprocessor.
> It would be used to generate all .mli files before the compilation
> of the .ml ones (at least in _MY_ OCaml projects).
> 
>> so it's a valuable check that what I think my module does is also
>> what the compiler does.  I also tend to write the .mli first, then write
>> the .ml.  I find it to be a great way to develop.
> 
> I don't.
> 
>> In short, I think it's a good thing to maintain these things by hand.
> 
> I don't:
> dumb_job + error_prone + can_be_automated + duplication_of_information
> --> the computer will do it for me.
> 
> I'm thinking about something small.
> It may not work on all cases people can come up with.
> I don't do very elaborated things usually so it will work for me,
> at least.
> 
>> But as for your original question I'm completely useless, sorry.
>>
>> /M
>>
>> Francois Berenger <berenger@riken.jp> writes:
>>
>>> Hello,
>>>
>>> Is there some recommended tool/script to generate a .mli
>>> from the corresponding .ml?
>>>
>>> I want a little more than ocamlc -i:
>>>
>>> - I think there should be tags in the .ml file as comments
>>>    that say "export this" to the .mli.
>>>    By default, things are not exported.
>>> - maybe it should have an option to say to replicate
>>>    the ocamldoc comments in the .mli.
>>> - it could be nice if the order in which things are exported
>>>    to the .mli can be specified, maybe as an argument of the tag.
>>>    So that the .mli can be more readable (only backward references
>>>    to concepts, etc.)
>>>
>>> If there is a need to create a tool, let's call it "nomli". :)
>>>
>>> Regards,
>>> F.
>>>
>>> PS: I'm not going to maintain both a .mli and a .ml.
>>>      I feel it is a dumb and error-prone job and that
>>>      itches me.
>>
> 
> 

[Caml-list] Re: automatic extaction of the .mli (and a little more) from the .ml

[Hongbo Zhang]

On 5/31/13 1:31 AM, Malcolm Matalka wrote:
> I know of no such tool, but in counter to your premise: I used to think
> maintaining a .ml and .mli was foolish, however I no longer do.  .mli is
> effectively documentation for me.  It contains a lot of comments and is
> generally written to reflect how the API should be used rather than the
> order in which I must express functions to get ta .ml to compile.  On
> top of that, ocamlc will fail to compile if your .ml and .mli don't
> match, so it's a valuable check that what I think my module does is also
> what the compiler does.  I also tend to write the .mli first, then write
> the .ml.  I find it to be a great way to develop.
>
> In short, I think it's a good thing to maintain these things by hand.
> But as for your original question I'm completely useless, sorry.
>
I had the same experience with you, after writing some large software in 
ML, I found it better to maintain mli by hand.
> /M
>
> Francois Berenger <berenger@riken.jp> writes:
>
>> Hello,
>>
>> Is there some recommended tool/script to generate a .mli
>> from the corresponding .ml?
>>
>> I want a little more than ocamlc -i:
>>
>> - I think there should be tags in the .ml file as comments
>>    that say "export this" to the .mli.
>>    By default, things are not exported.
>> - maybe it should have an option to say to replicate
>>    the ocamldoc comments in the .mli.
>> - it could be nice if the order in which things are exported
>>    to the .mli can be specified, maybe as an argument of the tag.
>>    So that the .mli can be more readable (only backward references
>>    to concepts, etc.)
>>
>> If there is a need to create a tool, let's call it "nomli". :)
>>
>> Regards,
>> F.
>>
>> PS: I'm not going to maintain both a .mli and a .ml.
>>      I feel it is a dumb and error-prone job and that
>>      itches me.
>

Re: [Caml-list] Re: automatic extaction of the .mli (and a little more) from the .ml

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 2450 bytes --]

I concur.  I think this is largely the accepted wisdom among experienced
OCaml developers.

y
On May 31, 2013 11:21 AM, "Hongbo Zhang" <bobzhang1988@gmail.com> wrote:

> On 5/31/13 1:31 AM, Malcolm Matalka wrote:
>
>> I know of no such tool, but in counter to your premise: I used to think
>> maintaining a .ml and .mli was foolish, however I no longer do.  .mli is
>> effectively documentation for me.  It contains a lot of comments and is
>> generally written to reflect how the API should be used rather than the
>> order in which I must express functions to get ta .ml to compile.  On
>> top of that, ocamlc will fail to compile if your .ml and .mli don't
>> match, so it's a valuable check that what I think my module does is also
>> what the compiler does.  I also tend to write the .mli first, then write
>> the .ml.  I find it to be a great way to develop.
>>
>> In short, I think it's a good thing to maintain these things by hand.
>> But as for your original question I'm completely useless, sorry.
>>
>>  I had the same experience with you, after writing some large software in
> ML, I found it better to maintain mli by hand.
>
>> /M
>>
>> Francois Berenger <berenger@riken.jp> writes:
>>
>>  Hello,
>>>
>>> Is there some recommended tool/script to generate a .mli
>>> from the corresponding .ml?
>>>
>>> I want a little more than ocamlc -i:
>>>
>>> - I think there should be tags in the .ml file as comments
>>>    that say "export this" to the .mli.
>>>    By default, things are not exported.
>>> - maybe it should have an option to say to replicate
>>>    the ocamldoc comments in the .mli.
>>> - it could be nice if the order in which things are exported
>>>    to the .mli can be specified, maybe as an argument of the tag.
>>>    So that the .mli can be more readable (only backward references
>>>    to concepts, etc.)
>>>
>>> If there is a need to create a tool, let's call it "nomli". :)
>>>
>>> Regards,
>>> F.
>>>
>>> PS: I'm not going to maintain both a .mli and a .ml.
>>>      I feel it is a dumb and error-prone job and that
>>>      itches me.
>>>
>>
>>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/**arc/caml-list<https://sympa.inria.fr/sympa/arc/caml-list>
> Beginner's list: http://groups.yahoo.com/group/**ocaml_beginners<http://groups.yahoo.com/group/ocaml_beginners>
> Bug reports: http://caml.inria.fr/bin/caml-**bugs<http://caml.inria.fr/bin/caml-bugs>
>

[-- Attachment #2: Type: text/html, Size: 3371 bytes --]

Re: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[oliver]

On Fri, May 31, 2013 at 03:26:11PM +0900, Francois Berenger wrote:
> On 05/31/2013 02:31 PM, Malcolm Matalka wrote:
> >I know of no such tool, but in counter to your premise: I used to think
> >maintaining a .ml and .mli was foolish, however I no longer do.
> 
[...]


[...]
> > so it's a valuable check that what I think my module does is also
> >what the compiler does.  I also tend to write the .mli first, then write
> >the .ml.  I find it to be a great way to develop.
> 
> I don't.
[...]


I also not always start with the mli-file, but it is a good way
to do it, it#s a good habit.

Why?

Because the interface is designed first.
And the implementation will follow.

Especially in teams, this is a really helpful issue:

  1.: Let the team agree on a interface (mli-file)

  2.: Let the project leader or other person who controls the interface for the modules,
      create the mli-files.
      (the interface for modules && responsibility of certain persons for asertain modules
        => interface between responsibilities)

  3.: Put the mli-files into a certain directory, where they have to live.
      Give read-permissions to all the programmers of the team and write permissions
      to the mli-files only for the person, who is responsible for them.


Then necessarily all programmers of the team know, what the interface is,
and can use these files, but not change them. Creating his/her own mli-files
will not make sense, sooner or later, the official mli-files must be used.

And each programmer can write ml-files in personal style, can write one
or more implementations, which need to fir the interface files.

      
But even if these team-worflow would not be used: deigning the
interface first (thinking about the program in types) can often be helpful
to make the design clear from the beginning.

Of course, especially if things are explored rather experimentally,
for example if there is a new field of topics, where the interface
is NOT clear in advance, starting with the mli-files might be rather
a show stopper.
if things are rather experimentally, starting with the ml-file might
be easier to handle.

So, which way to go depends on the field, on the knowledge of the field,
on how much planning (of interface) is possible or necessary and
if it is done in a team or as a single person.

So, both ways have their advantages and disadvantages.

camlc -i   most often worked for me.

In private projects I not necessarily use mli-files always.
At job I have to use other languages, but the above mentioned
way of read-only mli-files for the programmers of a team
would make sense to me.
This would make interfaces much clearer in the real world,
and IMHO is a good argument for OCaml at work.


If some small programs are one at home,
and the mli files follow the ml-files,
I wonder if they are necessary at all.

I sometimes use modules inside a compilation unit,
and if they clutter the interface too much, use signatures
for these inside-modules, not necessarily for the compilation
unit itself.

But it depends....


If ocamlc could print both: the default signature of the ml-file,
as well as the resulting signature, as delimited by the mli-file,
depending on cli-switches, as well as a diff between the ml-file's
default signature and the signature provided by the mli-file,
this would make handling much easier.
Same call of ocamlc, but just change the switch to get the interface
of the ml-file, the mli-file's signature or the diff between them,
this would be nice.

Something like

  ocamlc -i   foobar.ml   # prints default   sig of foobar.ml
  ocamlc -ii  foobar.ml   # prints resulting sig of foobar.ml (either def.-sig or mli-file-sig)

  ocamlc -id   foobar.ml   # prints diff between default sig and resulting sig (mli-sig if there is a mli-file)
  ocamlc -iid  foobar.ml   # prints diff between resulting sig (mli-sig if there is a mli-file) and default sig


Why ocamlc -ii foobar.ml, if 'cat foobar.mli' or 'ocamlc -i foobar.mli' would
give the same result?
Just because the handling would be unified: same command, same filename, just a
different switch.


Hope, this does make some sense not only for me ;-)


Ciao,
   Oliver

Re: [Caml-list] Re: automatic extaction of the .mli (and a little more) from the .ml

[Jacques Le Normand]

[-- Attachment #1: Type: text/plain, Size: 2712 bytes --]

I'm surprised no one has mentioned that it can speed up compilation by not
having to recompile cmx files unnecessarily.
On May 31, 2013 8:42 AM, "Yaron Minsky" <yminsky@janestreet.com> wrote:

> I concur.  I think this is largely the accepted wisdom among experienced
> OCaml developers.
>
> y
> On May 31, 2013 11:21 AM, "Hongbo Zhang" <bobzhang1988@gmail.com> wrote:
>
>> On 5/31/13 1:31 AM, Malcolm Matalka wrote:
>>
>>> I know of no such tool, but in counter to your premise: I used to think
>>> maintaining a .ml and .mli was foolish, however I no longer do.  .mli is
>>> effectively documentation for me.  It contains a lot of comments and is
>>> generally written to reflect how the API should be used rather than the
>>> order in which I must express functions to get ta .ml to compile.  On
>>> top of that, ocamlc will fail to compile if your .ml and .mli don't
>>> match, so it's a valuable check that what I think my module does is also
>>> what the compiler does.  I also tend to write the .mli first, then write
>>> the .ml.  I find it to be a great way to develop.
>>>
>>> In short, I think it's a good thing to maintain these things by hand.
>>> But as for your original question I'm completely useless, sorry.
>>>
>>>  I had the same experience with you, after writing some large software
>> in ML, I found it better to maintain mli by hand.
>>
>>> /M
>>>
>>> Francois Berenger <berenger@riken.jp> writes:
>>>
>>>  Hello,
>>>>
>>>> Is there some recommended tool/script to generate a .mli
>>>> from the corresponding .ml?
>>>>
>>>> I want a little more than ocamlc -i:
>>>>
>>>> - I think there should be tags in the .ml file as comments
>>>>    that say "export this" to the .mli.
>>>>    By default, things are not exported.
>>>> - maybe it should have an option to say to replicate
>>>>    the ocamldoc comments in the .mli.
>>>> - it could be nice if the order in which things are exported
>>>>    to the .mli can be specified, maybe as an argument of the tag.
>>>>    So that the .mli can be more readable (only backward references
>>>>    to concepts, etc.)
>>>>
>>>> If there is a need to create a tool, let's call it "nomli". :)
>>>>
>>>> Regards,
>>>> F.
>>>>
>>>> PS: I'm not going to maintain both a .mli and a .ml.
>>>>      I feel it is a dumb and error-prone job and that
>>>>      itches me.
>>>>
>>>
>>>
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/**arc/caml-list<https://sympa.inria.fr/sympa/arc/caml-list>
>> Beginner's list: http://groups.yahoo.com/group/**ocaml_beginners<http://groups.yahoo.com/group/ocaml_beginners>
>> Bug reports: http://caml.inria.fr/bin/caml-**bugs<http://caml.inria.fr/bin/caml-bugs>
>>
>

[-- Attachment #2: Type: text/html, Size: 3835 bytes --]

Re: [Caml-list] Re: automatic extaction of the .mli (and a little more) from the .ml

[Florent Monnier]

2013/05/31, Hongbo Zhang wrote:
[...]
> I had the same experience with you, after writing some large software in
> ML, I found it better to maintain mli by hand.

It depends which situation, which kind of things you're doing.

Most often I will prefer writing mli by hand, and sometimes I would
prefer to use a tool like nomli.

--

Re: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[Francois Berenger]

On 06/01/2013 08:13 AM, oliver wrote:
 > [...]
> Of course, especially if things are explored rather experimentally,
> for example if there is a new field of topics, where the interface
> is NOT clear in advance, starting with the mli-files might be rather
> a show stopper.
> if things are rather experimentally, starting with the ml-file might
> be easier to handle.

This is exactly my case.

Also, I grow programs organically because it is fun and motivating.
It is like playing with LEGO.

Re: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[Francois Berenger]

On 05/31/2013 06:10 PM, Romain Bardou wrote:
> I also used to believe the .mli file should be somehow included in the
> .ml file. Now I don't.
>
> But if you design such a tool, here are some things to consider.
>
> - Using numbers to order stuff creates some issues of scalability. In
> particular, inserting a declaration between #3 and #4 may require you to
> move #4 to #5, #5 to #6 and so on, which (in particular if everything is
> not in the right order in the implementation) can prove more annoying
> than reordering stuff in an .mli.

This is trivial to solve.

> - Even without numbers it will be harder to get the big picture from a
> module.

I just propose to generate the .mli automatically (for my projects).
Not to get rid of them.

 > An approach I like about this is Pascal units, with an interface
> section and an implementation section, in the same file. But it does not
> solve the duplication issue.
>
> - You need a way to specify abstraction. For instance:
>    * specify that a type should be exported as "type t" instead of "type
> t = A | B";
>    * specify that a type should be exported as "type t = private A | B"
> instead of "type t = A | B";
>    * specify that a value should be exported as "val f: t -> t" instead
> of "val f: int -> int"
>    * specify that a value should be exported as "val f: int -> t" instead
> of "val f: 'a -> 'a"

I did not think about this yet.

> Often, people who want automatic .mli generation do not use abstraction,
> which is the main point of interfaces.
>
> - Don't forget that you need to take into account the whole OCaml
> language, including objects, modules, polymorphic variants… It may be
> more work than expected. And this project will need to be maintained and
> follow the changes to the language.

I cannot provide any guarantee, on any of these.

Regards,
F.

Re: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[Malcolm Matalka]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]

I don't think LEGOs are a good analogy here since they have a very well
defined and consistent interface :)

But I have never found handling mlis by hand to get in the way of program
evolution.  If you take time in your interfaces program evolution should be
more about plugging in functionality in unique ways not changing interfaces
willy nilly.
 On Jun 3, 2013 1:28 PM, "Francois Berenger" <berenger@riken.jp> wrote:

> On 06/01/2013 08:13 AM, oliver wrote:
> > [...]
>
>> Of course, especially if things are explored rather experimentally,
>> for example if there is a new field of topics, where the interface
>> is NOT clear in advance, starting with the mli-files might be rather
>> a show stopper.
>> if things are rather experimentally, starting with the ml-file might
>> be easier to handle.
>>
>
> This is exactly my case.
>
> Also, I grow programs organically because it is fun and motivating.
> It is like playing with LEGO.
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/**arc/caml-list<https://sympa.inria.fr/sympa/arc/caml-list>
> Beginner's list: http://groups.yahoo.com/group/**ocaml_beginners<http://groups.yahoo.com/group/ocaml_beginners>
> Bug reports: http://caml.inria.fr/bin/caml-**bugs<http://caml.inria.fr/bin/caml-bugs>
>

[-- Attachment #2: Type: text/html, Size: 1858 bytes --]

Re: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[Alain Frisch]

I've created a proof-of-concept implementation of such a tool as part of 
the "extension_points" branch, which introduces a syntax for attributes 
on various syntactic items (including type declarations and value bindings).

Note that I *don't* endorse the idea of generating .mli files 
automatically :-)


Here is the code:

http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/branches/extension_points/experimental/frisch/nomli.ml?revision=HEAD&view=markup

and an example:

http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/branches/extension_points/experimental/frisch/test_nomli.ml?revision=HEAD&view=markup


On 05/31/2013 05:43 AM, Francois Berenger wrote:
> - I think there should be tags in the .ml file as comments
>    that say "export this" to the .mli.

Encoding annotations in comments is very fragile, because they are not 
kept by the parser in the generated Parsetree, and it is not 
well-defined to which component each component refers to.  ocamldoc uses 
comments, and as a consequence, it does not combine well with 
preprocessors (Camlp4/Camlp5/Fan/ppx extensions cannot generate 
documentation fragments, and it is likely that they break existing 
ones).  A version of ocamldoc based on attributes of the 
extension_points branch would be much more robust (and simple).  I've 
written another toy POC implementation of a mini-ocamldoc based on 
attributes:

http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/branches/extension_points/experimental/frisch/minidoc.ml?revision=HEAD&view=markup

http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/branches/extension_points/experimental/frisch/testdoc.mli?revision=HEAD&view=markup


(experimental/frisch/Makefile in the extenstion_point branch contains 
everything to compile and run these little demos.)

-- Alain

Re: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[Francois Berenger]

On 06/04/2013 02:12 AM, Alain Frisch wrote:
> I've created a proof-of-concept implementation of such a tool as part of
> the "extension_points" branch, which introduces a syntax for attributes
> on various syntactic items (including type declarations and value
> bindings).
>
> Note that I *don't* endorse the idea of generating .mli files
> automatically :-)
>
> Here is the code:
>
> http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/branches/extension_points/experimental/frisch/nomli.ml?revision=HEAD&view=markup
>
> and an example:
>
> http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/branches/extension_points/experimental/frisch/test_nomli.ml?revision=HEAD&view=markup

Alain, I am amazed!

Would it be possible to handle the two following things?

- being able to assign priorities to things exported in the
   .mli file (to re-order the .mli in some way that makes it easier to
   grasp). I think floats should be used and the priority assigned
   should be shown in the .mli so that users can quickly see what
   priority they need to change in the .ml to obtain the order
   they want in the .mli

- can you also handle the transfer of ocamldoc comments in the .ml
   to the .mli (I read later that you also have a final solution
   for ocamldoc too)?

The tool I was thinking of would not do anything more...

> On 05/31/2013 05:43 AM, Francois Berenger wrote:
>> - I think there should be tags in the .ml file as comments
>>    that say "export this" to the .mli.
>
> Encoding annotations in comments is very fragile, because they are not
> kept by the parser in the generated Parsetree, and it is not
> well-defined to which component each component refers to.  ocamldoc uses
> comments, and as a consequence, it does not combine well with
> preprocessors (Camlp4/Camlp5/Fan/ppx extensions cannot generate
> documentation fragments, and it is likely that they break existing
> ones).  A version of ocamldoc based on attributes of the
> extension_points branch would be much more robust (and simple).

The same is true for qtest I guess.

>  I've
> written another toy POC implementation of a mini-ocamldoc based on
> attributes:
>
> http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/branches/extension_points/experimental/frisch/minidoc.ml?revision=HEAD&view=markup
>
> http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/branches/extension_points/experimental/frisch/testdoc.mli?revision=HEAD&view=markup

Thanks a lot for the code and explanations.

Best regards,
Francois.

RE: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[David Allsopp]

Romain Bardou wrote:
> I also used to believe the .mli file should be somehow included in the .ml
> file. Now I don't.

+1, FWIW! It's easy to start without the .mli when prototyping and then add it as things mature.
 
> But if you design such a tool, here are some things to consider.
> 
> - Using numbers to order stuff creates some issues of scalability. In
> particular, inserting a declaration between #3 and #4 may require you to
> move #4 to #5, #5 to #6 and so on, which (in particular if everything is
> not in the right order in the implementation) can prove more annoying than
> reordering stuff in an .mli.

This is horrible, IMO - any design decision which looks like it'll *require* refactoring support from an editor to be able to work with it strikes me as a bad design decision. If you have hundreds of these in a file, at some point you'll need a "renumber everything" macro.

I have wondered if this problem is perhaps looked at the wrong way around - in other words, the complaint takes the form "how can we export to the .mli file automatically" rather than "how can we *import* from the .ml file automatically". The thing I do find irritating maintaining .mli/.ml files is having to type anything out twice - and for the most part that means fully exported type declarations. Say you have a simple module:

Foo.mli (separate file, so the items can appear in a different order trivially)
=======

(** Foo. This module is an example *)

(** Annotation *)
type t = A (** Annotation *) | B (** Annotation *)

val bar : t -> bool
(** [bar x] returns true if [x] is [A] *)

Foo.ml
======

type t = A | B (* This duplication is annoying *)

let bar x = (x = A) (* No type declaration needed here *)

For me, "val bar" in the .mli and "let bar" in the .ml is not duplication as one is the implementation and the other is the exported type. However, I do find having to maintain type t in two files irritating (say if I want to add the constructor C) - especially if t has hundreds of constructors where you have to make sure the order is the same too, of course. What would be nice would be to be able to put this in the /.ml/ file: 

(** Annotation *)
type t = A (** Annotation *) | B (** Annotation *)

and then this in the /.mli/ file:

type t from implementation

or 

export type t

which would export the entire declaration and also its ocamldoc comments. For an opaque type, you have different declarations anyway, so just write them. The same would be true of module types and so on.

Just my 2p!


David

Re: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[Alain Frisch]

On 06/04/2013 09:53 AM, David Allsopp wrote:
> I have wondered if this problem is perhaps looked at the wrong way
> around - in other words, the complaint takes the form "how can we
> export to the .mli file automatically" rather than "how can we
> *import* from the .ml file automatically". The thing I do find
> irritating maintaining .mli/.ml files is having to type anything out
> twice - and for the most part that means fully exported type
> declarations. Say you have a simple module:

I'd rather do it the other way around: maintain the type declaration in 
the .mli and import it in the .ml file.

And, guess what, there is also a POC of such a tool in the 
extension_points branch.  It allows you to write in the .ml file:

   type t = [%copy_typedef]

to copy the concrete definition from the .mli file.  This also works for 
module types, and you can also import a definition from another .ml or 
.mli file:

   module type S = [%copy_typedef]
   type loc = [%copy_typedef "../../parsing/location.mli" t]


The [%id ...] syntax is the other new syntactic feature introduced by 
the extension_points branch (in addition to attributes).  Contrary to 
attributes which annotate already well-formed code and which can thus be 
ignored by the type-checker, [%...] extensions can used in place of an 
expression, type, pattern, etc, and they need to be expanded by a -ppx 
rewriter.



The tool:

http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/branches/extension_points/experimental/frisch/copy_typedef.ml?revision=HEAD&view=markup

And an example:

http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/branches/extension_points/experimental/frisch/test_copy_typedef.mli?revision=HEAD&view=markup

http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/branches/extension_points/experimental/frisch/test_copy_typedef.ml?revision=HEAD&view=markup



-- Alain

Re: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[Romain Bardou]

Le 04/06/2013 09:53, David Allsopp a écrit :
> Romain Bardou wrote:
>> I also used to believe the .mli file should be somehow included in the .ml
>> file. Now I don't.
> 
> +1, FWIW! It's easy to start without the .mli when prototyping and then add it as things mature.
>  
>> But if you design such a tool, here are some things to consider.
>>
>> - Using numbers to order stuff creates some issues of scalability. In
>> particular, inserting a declaration between #3 and #4 may require you to
>> move #4 to #5, #5 to #6 and so on, which (in particular if everything is
>> not in the right order in the implementation) can prove more annoying than
>> reordering stuff in an .mli.
> 
> This is horrible, IMO - any design decision which looks like it'll *require* refactoring support from an editor to be able to work with it strikes me as a bad design decision. If you have hundreds of these in a file, at some point you'll need a "renumber everything" macro.

Agreed, obviously. It seems François proposes floats as a solution, but
while it does allow you to insert without renumbering everything, it
seems that with time, you'd end up with a messy file.

> I have wondered if this problem is perhaps looked at the wrong way around - in other words, the complaint takes the form "how can we export to the .mli file automatically" rather than "how can we *import* from the .ml file automatically". The thing I do find irritating maintaining .mli/.ml files is having to type anything out twice - and for the most part that means fully exported type declarations. Say you have a simple module:

I agree with the idea of looking the other way around, but I'm not sure
I agree with your proposal, which seems to *not* be the other way
around: in your example, you write "type t from implementation", in the
.mli. I would prefer "type t from interface" from the .ml.

Also, I agree that val and let is not duplication as long as the let is
not annotated. Duplication comes from types and module type
declarations, exceptions, and maybe also classes.

Regarding val and let, if the types don't match, the error is better
located than before in recent versions of OCaml. But it could be even
better if the let could be retyped with inlined type annotations.

Here is an example:

val f: int -> int

let f x = x +. 1.

The error is located on f (it has type float -> float instead of int ->
int). But maybe it would be better if it was located on x or +. as if
the function was actually written like this:

let f (x: int): int = x +. 1.

Why do I bring that up? Well this would go in the sense of avoiding
duplication by avoiding the need of adding the type annotations yourself
to get better error location.

But this brings up other questions, such as: what if a function f is
polymorphic in the implementation and used as such, but is exported as a
monomorphic instance: if you import type annotations from the .mli, you
cannot use the polymorphism of f in the implementation anymore. Unless
the type annotations are only used to obtain better error messages, only
when there is a type mismatch between the mli and the ml.

Cheers,

-- 
Romain Bardou

Re: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[Alain Frisch]

On 06/04/2013 02:30 AM, Francois Berenger wrote:
> - being able to assign priorities to things exported in the
>    .mli file (to re-order the .mli in some way that makes it easier to
>    grasp). I think floats should be used and the priority assigned
>    should be shown in the .mli so that users can quickly see what
>    priority they need to change in the .ml to obtain the order
>    they want in the .mli

Yes, it's quite easy to add some priority information as an extra 
argument to the [@mli] attribute.  (That said, since extension_points is 
not yet merged in the official version, I don't think it's worth 
spending time on the prototype right now.)

Showing the priority in the .mli (in the form of attributes) is 
straightforward (currently, the source syntax pretty-printer 
pprintast.ml does not support printing attributes, but this will be fixed).

> - can you also handle the transfer of ocamldoc comments in the .ml
>    to the .mli

In this form, this is very tricky, because the tool is based on the 
result of the OCaml parser which does not store the ocamldoc comments. 
The tool would need to steal from ocamldoc the (complex) logic required 
to re-parse the source code, extract the attributes, attach them to 
components of the parsetree/typedtree.

What is doable is to have the tool accept more attributes (or more 
arguments to the [@mli] attribute) representing text to be used as 
ocamldoc comments in the generated .mli.  Instead of producing an AST 
for the generated signatures (which is then pretty-printed), the tool 
would directly emit the content of the .mli (with ocamldoc attributes).


> (I read later that you also have a final solution
>    for ocamldoc too)?

Certainly not!  I'm only suggesting that a tool similar to ocamldoc 
could be based on attributes, and that it would be more robust and much 
easier to implement.  But designing and implementing this tool is still 
a serious engineering task, and unless someone puts the required 
manpower in this project, this is not going to happen.


Alain

RE: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[David Allsopp]

Alain Frisch wrote:
> On 06/04/2013 09:53 AM, David Allsopp wrote:
> > I have wondered if this problem is perhaps looked at the wrong way
> > around - in other words, the complaint takes the form "how can we
> > export to the .mli file automatically" rather than "how can we
> > *import* from the .ml file automatically". The thing I do find
> > irritating maintaining .mli/.ml files is having to type anything out
> > twice - and for the most part that means fully exported type
> > declarations. Say you have a simple module:
> 
> I'd rather do it the other way around: maintain the type declaration in
> the .mli and import it in the .ml file.

My thinking was that the type declaration is part of the implementation at that point so should be in the .ml file - but actually I agree that it's better to do it the other way around (given that installing .mli files is quite common, for example).
 
> And, guess what, there is also a POC of such a tool in the
> extension_points branch.  It allows you to write in the .ml file:
> 
>    type t = [%copy_typedef]

This is very cool - will have to check out this branch and have a play when I have a moment!


David

RE: [Caml-list] automatic extaction of the .mli (and a little more) from the .ml

[David Allsopp]

Romain Bardou wrote:
> Le 04/06/2013 09:53, David Allsopp a écrit :
> > Romain Bardou wrote:
> >> I also used to believe the .mli file should be somehow included in
> >> the .ml file. Now I don't.
> >
> > +1, FWIW! It's easy to start without the .mli when prototyping and then
> add it as things mature.
> >
> >> But if you design such a tool, here are some things to consider.
> >>
> >> - Using numbers to order stuff creates some issues of scalability. In
> >> particular, inserting a declaration between #3 and #4 may require you
> >> to move #4 to #5, #5 to #6 and so on, which (in particular if
> >> everything is not in the right order in the implementation) can prove
> >> more annoying than reordering stuff in an .mli.
> >
> > This is horrible, IMO - any design decision which looks like it'll
> *require* refactoring support from an editor to be able to work with it
> strikes me as a bad design decision. If you have hundreds of these in a
> file, at some point you'll need a "renumber everything" macro.
> 
> Agreed, obviously. It seems François proposes floats as a solution, but
> while it does allow you to insert without renumbering everything, it seems
> that with time, you'd end up with a messy file.

Yes - I'd said in a private exchange that using floats is no different from the way Basic programmers used to leave gaps between line numbers! Eventually you need to renumber...

> > I have wondered if this problem is perhaps looked at the wrong way
> around - in other words, the complaint takes the form "how can we export
> to the .mli file automatically" rather than "how can we *import* from the
> .ml file automatically". The thing I do find irritating maintaining
> .mli/.ml files is having to type anything out twice - and for the most
> part that means fully exported type declarations. Say you have a simple
> module:
> 
> I agree with the idea of looking the other way around, but I'm not sure I
> agree with your proposal, which seems to *not* be the other way
> around: in your example, you write "type t from implementation", in the
> .mli. I would prefer "type t from interface" from the .ml.

My perspective was that it looks at what we write in the .ml file being *exported* to the .mli file and I was proposing that what we write in the .mli file be *imported* from the .ml file (which I would say is indeed the other way around). But as I said in response to Alain, perhaps that's not actually the good idea I thought it was, given that .mli files are used in contexts where the .ml file isn't available. I agree that putting the copy statement in the .ml file is better. 

> Also, I agree that val and let is not duplication as long as the let is
> not annotated. Duplication comes from types and module type declarations,
> exceptions, and maybe also classes.
> 
> Regarding val and let, if the types don't match, the error is better
> located than before in recent versions of OCaml. But it could be even
> better if the let could be retyped with inlined type annotations.
> 
> Here is an example:
> 
> val f: int -> int
> 
> let f x = x +. 1.
> 
> The error is located on f (it has type float -> float instead of int ->
> int). But maybe it would be better if it was located on x or +. as if the
> function was actually written like this:
> 
> let f (x: int): int = x +. 1.
> 
> Why do I bring that up? Well this would go in the sense of avoiding
> duplication by avoiding the need of adding the type annotations yourself
> to get better error location.

I guess that depends on what you regard as the error, though - the definition of [f] with floats is not incorrect, it just doesn't comply with the signature you've imposed. Personally, I'm happy with the idea that the compiler reports that the signature of [f] differs between the implementation and interface and leaves me to figure out why that's a mistake, but... 

> But this brings up other questions, such as: what if a function f is
> polymorphic in the implementation and used as such, but is exported as a
> monomorphic instance: if you import type annotations from the .mli, you
> cannot use the polymorphism of f in the implementation anymore. Unless the
> type annotations are only used to obtain better error messages, only when
> there is a type mismatch between the mli and the ml.

That's kind of why I was thinking that you only have an "extension" for types which are *actually* duplicated between the two. However, perhaps it's something that is relatively easy with the kind of extension Alain proposes so that you could write in the .ml file:

let [%unify_typedef] f = x +. 1.

so that you're saying that the type of [f] must be unifiable (rather than copied) from the .mli... which would catch the case of f : float -> float and state that either [+.] or [x] is the problem but also allow you to have a polymorphic function in the implementation with a monomorphic exported type.


David