[Caml-list] Ocaml compiler documentation

[Caml-list] Ocaml compiler documentation

[Yotam Barnoy]

[-- Attachment #1: Type: text/plain, Size: 2121 bytes --]

Hi everybody

It's been mentioned before that the OCaml compiler's documentation is
somewhat lacking. I've been going over the compiler code gradually (both
the frontend and the backend) and while some parts are understandable
enough, others are missing some basic explanations. Some explanations are
also spread out throughout the codebase, making it hard to know what
something means unless you've read another part of the codebase that
relates to it.

Since the call to submit documentation commits has gone mostly unanswered,
I'd like to suggest a method of making both my own progress through the
code easier and hopefully making it easier for others who will follow.

What I'm going to do is, focusing on more or less one file at a time, I'll
post newbie questions to the list about the code. Once I'm satisfied that I
have a good enough understanding, I'll add comments to the aforementioned
files and submit pull requests for them. I also encourage others to do the
same.

What I need from the list, and especially from the more knowledgeable
members (who already know the compiler code) is the willingness to explain
the concepts and answer my questions, annoying as they may be. I have a
pretty decent background in compilers, ASTs, code generation, etc, but not
so much in type inference.

I'm not suggesting a particular timeframe for this process -- I'm doing
this on the side while working on a research project and TAing, but I
really would like to get to the point where I can make significant
contributions to the toolchain, and if I can help others who follow in my
footsteps, then that's a nice bonus.

While I could have skipped this introduction and just proceeded with
inundating the list with questions, I felt that this (hopefully) gives a
purpose and perhaps motivation for those who have the answers to answer my
questions even if they get annoying. In particular, I may often miss some
parts that may seem obvious because I don't necessarily have the time to
read all the connected code in depth. Hopefully you'll bear with me.

Does this sound reasonable to the fine folks on the list?

Yotam

[-- Attachment #2: Type: text/html, Size: 2354 bytes --]

Re: [Caml-list] Ocaml compiler documentation

[Milan Stanojević]

Thank you for doing this, I'm interested in learning more about how
compiler works.

Are you creating a separate file(s) to document the compiler or you
are adding comments to ml files?

On Mon, Mar 31, 2014 at 11:39 AM, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
> Hi everybody
>
> It's been mentioned before that the OCaml compiler's documentation is
> somewhat lacking. I've been going over the compiler code gradually (both the
> frontend and the backend) and while some parts are understandable enough,
> others are missing some basic explanations. Some explanations are also
> spread out throughout the codebase, making it hard to know what something
> means unless you've read another part of the codebase that relates to it.
>
> Since the call to submit documentation commits has gone mostly unanswered,
> I'd like to suggest a method of making both my own progress through the code
> easier and hopefully making it easier for others who will follow.
>
> What I'm going to do is, focusing on more or less one file at a time, I'll
> post newbie questions to the list about the code. Once I'm satisfied that I
> have a good enough understanding, I'll add comments to the aforementioned
> files and submit pull requests for them. I also encourage others to do the
> same.
>
> What I need from the list, and especially from the more knowledgeable
> members (who already know the compiler code) is the willingness to explain
> the concepts and answer my questions, annoying as they may be. I have a
> pretty decent background in compilers, ASTs, code generation, etc, but not
> so much in type inference.
>
> I'm not suggesting a particular timeframe for this process -- I'm doing this
> on the side while working on a research project and TAing, but I really
> would like to get to the point where I can make significant contributions to
> the toolchain, and if I can help others who follow in my footsteps, then
> that's a nice bonus.
>
> While I could have skipped this introduction and just proceeded with
> inundating the list with questions, I felt that this (hopefully) gives a
> purpose and perhaps motivation for those who have the answers to answer my
> questions even if they get annoying. In particular, I may often miss some
> parts that may seem obvious because I don't necessarily have the time to
> read all the connected code in depth. Hopefully you'll bear with me.
>
> Does this sound reasonable to the fine folks on the list?
>
> Yotam

Re: [Caml-list] Ocaml compiler documentation

[Yotam Barnoy]

[-- Attachment #1: Type: text/plain, Size: 3334 bytes --]

I think it depends on how much feedback I get on any particular question.
By default, I would like comments to go in the code. Additionally, there's
the ocaml-internals wiki at
https://github.com/ocamllabs/ocaml-internalswhich will be useful for
any concepts that span multiple files, or that are
too beginner-oriented. I'm guessing that for many things, it will just have
to be decided on a case-by-case basis.

Of course, the most important ingredient for the success of this 'project'
is the willing, patient participation of the core team, as well as the
other experts on this list.

-Yotam


On Mon, Mar 31, 2014 at 1:06 PM, Milan Stanojević <milanst@gmail.com> wrote:

> Thank you for doing this, I'm interested in learning more about how
> compiler works.
>
> Are you creating a separate file(s) to document the compiler or you
> are adding comments to ml files?
>
> On Mon, Mar 31, 2014 at 11:39 AM, Yotam Barnoy <yotambarnoy@gmail.com>
> wrote:
> > Hi everybody
> >
> > It's been mentioned before that the OCaml compiler's documentation is
> > somewhat lacking. I've been going over the compiler code gradually (both
> the
> > frontend and the backend) and while some parts are understandable enough,
> > others are missing some basic explanations. Some explanations are also
> > spread out throughout the codebase, making it hard to know what something
> > means unless you've read another part of the codebase that relates to it.
> >
> > Since the call to submit documentation commits has gone mostly
> unanswered,
> > I'd like to suggest a method of making both my own progress through the
> code
> > easier and hopefully making it easier for others who will follow.
> >
> > What I'm going to do is, focusing on more or less one file at a time,
> I'll
> > post newbie questions to the list about the code. Once I'm satisfied
> that I
> > have a good enough understanding, I'll add comments to the aforementioned
> > files and submit pull requests for them. I also encourage others to do
> the
> > same.
> >
> > What I need from the list, and especially from the more knowledgeable
> > members (who already know the compiler code) is the willingness to
> explain
> > the concepts and answer my questions, annoying as they may be. I have a
> > pretty decent background in compilers, ASTs, code generation, etc, but
> not
> > so much in type inference.
> >
> > I'm not suggesting a particular timeframe for this process -- I'm doing
> this
> > on the side while working on a research project and TAing, but I really
> > would like to get to the point where I can make significant
> contributions to
> > the toolchain, and if I can help others who follow in my footsteps, then
> > that's a nice bonus.
> >
> > While I could have skipped this introduction and just proceeded with
> > inundating the list with questions, I felt that this (hopefully) gives a
> > purpose and perhaps motivation for those who have the answers to answer
> my
> > questions even if they get annoying. In particular, I may often miss some
> > parts that may seem obvious because I don't necessarily have the time to
> > read all the connected code in depth. Hopefully you'll bear with me.
> >
> > Does this sound reasonable to the fine folks on the list?
> >
> > Yotam
>

[-- Attachment #2: Type: text/html, Size: 4088 bytes --]

Re: [Caml-list] Ocaml compiler documentation

[Mark Shinwell]

I would suggest that it's probably better to keep the documentation as
comments where possible.  However, I think it is important to avoid
excessive commentary, especially if it is likely to get out of sync as
a result of future modifications to the code.  It may be that in some
cases making alterations to the code (for example, improving the name
of a variable) is a more satisfactory approach than adding a comment.

Thanks for working on this.

Mark

On 31 March 2014 18:51, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
> I think it depends on how much feedback I get on any particular question. By
> default, I would like comments to go in the code. Additionally, there's the
> ocaml-internals wiki at https://github.com/ocamllabs/ocaml-internals which
> will be useful for any concepts that span multiple files, or that are too
> beginner-oriented. I'm guessing that for many things, it will just have to
> be decided on a case-by-case basis.
>
> Of course, the most important ingredient for the success of this 'project'
> is the willing, patient participation of the core team, as well as the other
> experts on this list.
>
> -Yotam
>
>
> On Mon, Mar 31, 2014 at 1:06 PM, Milan Stanojević <milanst@gmail.com> wrote:
>>
>> Thank you for doing this, I'm interested in learning more about how
>> compiler works.
>>
>> Are you creating a separate file(s) to document the compiler or you
>> are adding comments to ml files?
>>
>> On Mon, Mar 31, 2014 at 11:39 AM, Yotam Barnoy <yotambarnoy@gmail.com>
>> wrote:
>> > Hi everybody
>> >
>> > It's been mentioned before that the OCaml compiler's documentation is
>> > somewhat lacking. I've been going over the compiler code gradually (both
>> > the
>> > frontend and the backend) and while some parts are understandable
>> > enough,
>> > others are missing some basic explanations. Some explanations are also
>> > spread out throughout the codebase, making it hard to know what
>> > something
>> > means unless you've read another part of the codebase that relates to
>> > it.
>> >
>> > Since the call to submit documentation commits has gone mostly
>> > unanswered,
>> > I'd like to suggest a method of making both my own progress through the
>> > code
>> > easier and hopefully making it easier for others who will follow.
>> >
>> > What I'm going to do is, focusing on more or less one file at a time,
>> > I'll
>> > post newbie questions to the list about the code. Once I'm satisfied
>> > that I
>> > have a good enough understanding, I'll add comments to the
>> > aforementioned
>> > files and submit pull requests for them. I also encourage others to do
>> > the
>> > same.
>> >
>> > What I need from the list, and especially from the more knowledgeable
>> > members (who already know the compiler code) is the willingness to
>> > explain
>> > the concepts and answer my questions, annoying as they may be. I have a
>> > pretty decent background in compilers, ASTs, code generation, etc, but
>> > not
>> > so much in type inference.
>> >
>> > I'm not suggesting a particular timeframe for this process -- I'm doing
>> > this
>> > on the side while working on a research project and TAing, but I really
>> > would like to get to the point where I can make significant
>> > contributions to
>> > the toolchain, and if I can help others who follow in my footsteps, then
>> > that's a nice bonus.
>> >
>> > While I could have skipped this introduction and just proceeded with
>> > inundating the list with questions, I felt that this (hopefully) gives a
>> > purpose and perhaps motivation for those who have the answers to answer
>> > my
>> > questions even if they get annoying. In particular, I may often miss
>> > some
>> > parts that may seem obvious because I don't necessarily have the time to
>> > read all the connected code in depth. Hopefully you'll bear with me.
>> >
>> > Does this sound reasonable to the fine folks on the list?
>> >
>> > Yotam
>
>

Re: [Caml-list] Ocaml compiler documentation

[Yotam Barnoy]

[-- Attachment #1: Type: text/plain, Size: 6049 bytes --]

Ok I think a good place to start a tour of the compiler is in
parsing/parsetree.mli. This file is actually very well documented, with
terse but effective examples of almost every constructor and type.

I had to refer to the OCaml manual for a few of the corner cases. For
example, I didn't know about the #class type shortcut. I think a few
comments explaining the more obscure facets of the language could be
helpful.

Since the file is so well documented, I only have a few questions. I'll
accept an answer or a hunch from anyone -- don't feel shy because you think
you're not sure about the answer:

1. What is the difference between an extension and an attribute? From what
I understand, they are both means of integrating additional metadata into
the AST that can then be parsed by implementations of the ast-mapper, but
why are there 2 mechanisms?

2. What is demonstrated in lines 114-117 regarding polymorphic variant row
fields:

  | Rtag of label * bool * core_type list
        (* [`A]                   ( true,  [] )
           [`A of T]              ( false, [T] )
           [`A of T1 & .. & Tn]   ( false, [T1;...Tn] )
           [`A of & T1 & .. & Tn] ( true,  [T1;...Tn] )
         *)

What does the bool value represent?
Why are the type separators in the comments using the & symbol?
What is the difference between the 3rd and 4th example?

3. line 684: what is the purpose of the override flag on Pstr_open? It's
not explained by the comment.

4. The toplevel phrases are not clear. What is the purpose of Ptop_dir on
line 721?

Like I said, feel free to jump in and answer any one of these questions.

Thanks in advance for everyone's help

-Yotam




On Tue, Apr 1, 2014 at 6:03 AM, Mark Shinwell <mshinwell@janestreet.com>wrote:

> I would suggest that it's probably better to keep the documentation as
> comments where possible.  However, I think it is important to avoid
> excessive commentary, especially if it is likely to get out of sync as
> a result of future modifications to the code.  It may be that in some
> cases making alterations to the code (for example, improving the name
> of a variable) is a more satisfactory approach than adding a comment.
>
> Thanks for working on this.
>
> Mark
>
> On 31 March 2014 18:51, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
> > I think it depends on how much feedback I get on any particular
> question. By
> > default, I would like comments to go in the code. Additionally, there's
> the
> > ocaml-internals wiki at https://github.com/ocamllabs/ocaml-internalswhich
> > will be useful for any concepts that span multiple files, or that are too
> > beginner-oriented. I'm guessing that for many things, it will just have
> to
> > be decided on a case-by-case basis.
> >
> > Of course, the most important ingredient for the success of this
> 'project'
> > is the willing, patient participation of the core team, as well as the
> other
> > experts on this list.
> >
> > -Yotam
> >
> >
> > On Mon, Mar 31, 2014 at 1:06 PM, Milan Stanojević <milanst@gmail.com>
> wrote:
> >>
> >> Thank you for doing this, I'm interested in learning more about how
> >> compiler works.
> >>
> >> Are you creating a separate file(s) to document the compiler or you
> >> are adding comments to ml files?
> >>
> >> On Mon, Mar 31, 2014 at 11:39 AM, Yotam Barnoy <yotambarnoy@gmail.com>
> >> wrote:
> >> > Hi everybody
> >> >
> >> > It's been mentioned before that the OCaml compiler's documentation is
> >> > somewhat lacking. I've been going over the compiler code gradually
> (both
> >> > the
> >> > frontend and the backend) and while some parts are understandable
> >> > enough,
> >> > others are missing some basic explanations. Some explanations are also
> >> > spread out throughout the codebase, making it hard to know what
> >> > something
> >> > means unless you've read another part of the codebase that relates to
> >> > it.
> >> >
> >> > Since the call to submit documentation commits has gone mostly
> >> > unanswered,
> >> > I'd like to suggest a method of making both my own progress through
> the
> >> > code
> >> > easier and hopefully making it easier for others who will follow.
> >> >
> >> > What I'm going to do is, focusing on more or less one file at a time,
> >> > I'll
> >> > post newbie questions to the list about the code. Once I'm satisfied
> >> > that I
> >> > have a good enough understanding, I'll add comments to the
> >> > aforementioned
> >> > files and submit pull requests for them. I also encourage others to do
> >> > the
> >> > same.
> >> >
> >> > What I need from the list, and especially from the more knowledgeable
> >> > members (who already know the compiler code) is the willingness to
> >> > explain
> >> > the concepts and answer my questions, annoying as they may be. I have
> a
> >> > pretty decent background in compilers, ASTs, code generation, etc, but
> >> > not
> >> > so much in type inference.
> >> >
> >> > I'm not suggesting a particular timeframe for this process -- I'm
> doing
> >> > this
> >> > on the side while working on a research project and TAing, but I
> really
> >> > would like to get to the point where I can make significant
> >> > contributions to
> >> > the toolchain, and if I can help others who follow in my footsteps,
> then
> >> > that's a nice bonus.
> >> >
> >> > While I could have skipped this introduction and just proceeded with
> >> > inundating the list with questions, I felt that this (hopefully)
> gives a
> >> > purpose and perhaps motivation for those who have the answers to
> answer
> >> > my
> >> > questions even if they get annoying. In particular, I may often miss
> >> > some
> >> > parts that may seem obvious because I don't necessarily have the time
> to
> >> > read all the connected code in depth. Hopefully you'll bear with me.
> >> >
> >> > Does this sound reasonable to the fine folks on the list?
> >> >
> >> > Yotam
> >
> >
>

[-- Attachment #2: Type: text/html, Size: 7775 bytes --]

Re: [Caml-list] Ocaml compiler documentation

[Mark Shinwell]

On 3 April 2014 03:48, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
> 3. line 684: what is the purpose of the override flag on Pstr_open? It's not
> explained by the comment.

I believe when this is in "override" mode it corresponds to the new
"open!" syntax, which suppresses the "unused open" warning, and the
warnings about an "open" shadowing existing identifiers or labels.

Mark

Re: [Caml-list] Ocaml compiler documentation

[Jeremy Yallop]

On 3 April 2014 03:48, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
> 4. The toplevel phrases are not clear. What is the purpose of Ptop_dir on
> line 721?

Ptop_dir represents top level directives such as '#quit' and
'#directory', which are documented in the manual:

   http://caml.inria.fr/pub/docs/manual-ocaml-400/manual023.html#s:toplevel-directives

RE: [Caml-list] Ocaml compiler documentation

[David Allsopp]

Yotam Barnoy wrote:
> 2. What is demonstrated in lines 114-117 regarding polymorphic variant row fields:
> 
>  | Rtag of label * bool * core_type list
>        (* [`A]                   ( true,  [] )
>           [`A of T]              ( false, [T] )
>           [`A of T1 & .. & Tn]   ( false, [T1;...Tn] )
>           [`A of & T1 & .. & Tn] ( true,  [T1;...Tn] )
>         *)
>
> What does the bool value represent?

It indicates that the constructor is constant, or that one of the types in the conjunction is "empty" (i.e. constant - I don't know what Jacques' terminology for that is!)

> Why are the type separators in the comments using the & symbol?

For example:

type t = [ `A of string & int ]
Error: The present constructor A has a conjunctive type

(see typing/typetexp.ml)

but, adapting "Advanced use" example in section 4.2 of the manual:

# let f x = (function `A x -> x = 1 | `B -> true) x && (function `A x -> x = "a" | `B -> true) x;;
val f : [< `A of string & int | `B ] -> bool = <fun>

in order to write an annotation:

# let f : [< `A of string & int | `B] ] -> bool = fun x -> ...

> What is the difference between the 3rd and 4th example?

The conjunction may include the constant constructor case:

# let f x = (function `A -> false | `B -> true) x && (function `A x -> x = "a" | `B -> true) x;;
val f : [< `A of & string | `B ] -> bool = <fun>

The constructor Reither will give you other pointers into where this information is needed in typing/typecore.ml.

HTH,


David

Re: [Caml-list] Ocaml compiler documentation

[Simon Cruanes]

[-- Attachment #1: Type: text/plain, Size: 1490 bytes --]

Le Thu, 03 Apr 2014, David Allsopp a écrit :
> Yotam Barnoy wrote:
> > 2. What is demonstrated in lines 114-117 regarding polymorphic variant row fields:
> > 
> >  | Rtag of label * bool * core_type list
> >        (* [`A]                   ( true,  [] )
> >           [`A of T]              ( false, [T] )
> >           [`A of T1 & .. & Tn]   ( false, [T1;...Tn] )
> >           [`A of & T1 & .. & Tn] ( true,  [T1;...Tn] )
> >         *)
> >
> > What does the bool value represent?
> 
> It indicates that the constructor is constant, or that one of the types in the conjunction is "empty" (i.e. constant - I don't know what Jacques' terminology for that is!)

I might miss something, but wouldn't this be more readable if a variant
was used instead of booleans? Say,

type includes_constant =
    | IncludesConstant
    | DoesNotIncludeConstant

...
    | RTag of label * includes_constant * core_type list
        (*  [`A]            (IncludesConstant, [])
            [`A of T]       (DoesNotIncludeConstant, [T])
....

It would also be easier to extend. That's just nitpicking but I think it
would make the code more explicit by itself.

In addition, I never heard about conjunctions in variants, so I'm happy
Yotam started this initiative...

Cheers,

-- 
Simon

http://weusepgp.info/
key 49AA62B6 fingerprint 949F EB87 8F06 59C6 D7D3  7D8D 4AC0 1D08 49AA 62B6

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

RE: [Caml-list] Ocaml compiler documentation

[David Allsopp]

Simon Cruanes wrote:
> Le Thu, 03 Apr 2014, David Allsopp a écrit :
> > Yotam Barnoy wrote:
> > > 2. What is demonstrated in lines 114-117 regarding polymorphic variant
> row fields:
> > >
> > >  | Rtag of label * bool * core_type list
> > >        (* [`A]                   ( true,  [] )
> > >           [`A of T]              ( false, [T] )
> > >           [`A of T1 & .. & Tn]   ( false, [T1;...Tn] )
> > >           [`A of & T1 & .. & Tn] ( true,  [T1;...Tn] )
> > >         *)
> > >
> > > What does the bool value represent?
> >
> > It indicates that the constructor is constant, or that one of the
> > types in the conjunction is "empty" (i.e. constant - I don't know what
> > Jacques' terminology for that is!)
> 
> I might miss something, but wouldn't this be more readable if a variant
> was used instead of booleans? Say,
> 
> type includes_constant =
>     | IncludesConstant
>     | DoesNotIncludeConstant

Possibly (it certainly can't make any difference for efficiency, as type bool is simply a two-constructor variant itself!). I guess historically the worry would be conflicting names between different types, but that's not a problem now.

> In addition, I never heard about conjunctions in variants, so I'm happy
> Yotam started this initiative...

+1!

I already knew about conjunctions (it is worth perusing that section of the manual as most, if not all, of the weird corner cases are covered in there somewhere) but not how it was implemented in the parser/type checker!


David

Re: [Caml-list] Ocaml compiler documentation

[Yotam Barnoy]

[-- Attachment #1: Type: text/plain, Size: 2384 bytes --]

Wow that was awesome! Thanks so much for your help and encouragement, guys!

I think for this file I'll submit a simple PR with only a few minor
clarifications for the less obvious features. I'll also make an 'OCaml
Nuances' page in the wiki for listing some of these less well-known
features.

After that, I think I'll proceed to typing/typedtree.mli, which is the same
AST, only after the typechecker has added types to it.



On Thu, Apr 3, 2014 at 6:46 AM, David Allsopp <dra-news@metastack.com>wrote:

> Simon Cruanes wrote:
> > Le Thu, 03 Apr 2014, David Allsopp a écrit :
> > > Yotam Barnoy wrote:
> > > > 2. What is demonstrated in lines 114-117 regarding polymorphic
> variant
> > row fields:
> > > >
> > > >  | Rtag of label * bool * core_type list
> > > >        (* [`A]                   ( true,  [] )
> > > >           [`A of T]              ( false, [T] )
> > > >           [`A of T1 & .. & Tn]   ( false, [T1;...Tn] )
> > > >           [`A of & T1 & .. & Tn] ( true,  [T1;...Tn] )
> > > >         *)
> > > >
> > > > What does the bool value represent?
> > >
> > > It indicates that the constructor is constant, or that one of the
> > > types in the conjunction is "empty" (i.e. constant - I don't know what
> > > Jacques' terminology for that is!)
> >
> > I might miss something, but wouldn't this be more readable if a variant
> > was used instead of booleans? Say,
> >
> > type includes_constant =
> >     | IncludesConstant
> >     | DoesNotIncludeConstant
>
> Possibly (it certainly can't make any difference for efficiency, as type
> bool is simply a two-constructor variant itself!). I guess historically the
> worry would be conflicting names between different types, but that's not a
> problem now.
>
> > In addition, I never heard about conjunctions in variants, so I'm happy
> > Yotam started this initiative...
>
> +1!
>
> I already knew about conjunctions (it is worth perusing that section of
> the manual as most, if not all, of the weird corner cases are covered in
> there somewhere) but not how it was implemented in the parser/type checker!
>
>
> David
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 3316 bytes --]

Re: [Caml-list] Ocaml compiler documentation

[Gabriel Scherer]

> 1. What is the difference between an extension and an attribute? From what I
> understand, they are both means of integrating additional metadata into the
> AST that can then be parsed by implementations of the ast-mapper, but why
> are there 2 mechanisms?

Attributes are supposed to be "optional" information attached to
existing AST nodes:
  let@shadow x = e1 in e2

If you erase attributes, you get back a valid AST. The implicit
contract that syntax-extension designers that use attributes ought to
respect is that the presence of attributes should not change the
semantics of the underlying code (but it can direct additional code
generation, etc.): at first approximation it should be possible to
read the code without attributes, understand what it means, and then
refine one's understanding with knowledge of the particular attributes
used.

The compiler can ignore unknown attributes.

On the contrary, extensions are AST nodes by themselves: [% foo] is an
expression (when used in an expression position). If you erase the
extension, you get an invalid AST with a missing hole. They are
designed for situations where it makes no sense to forget about the
extended-syntax part.

For more information, see
  https://github.com/ocaml/ocaml/blob/trunk/experimental/frisch/extension_points.txt


On Thu, Apr 3, 2014 at 4:48 AM, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
> Ok I think a good place to start a tour of the compiler is in
> parsing/parsetree.mli. This file is actually very well documented, with
> terse but effective examples of almost every constructor and type.
>
> I had to refer to the OCaml manual for a few of the corner cases. For
> example, I didn't know about the #class type shortcut. I think a few
> comments explaining the more obscure facets of the language could be
> helpful.
>
> Since the file is so well documented, I only have a few questions. I'll
> accept an answer or a hunch from anyone -- don't feel shy because you think
> you're not sure about the answer:
>
> 1. What is the difference between an extension and an attribute? From what I
> understand, they are both means of integrating additional metadata into the
> AST that can then be parsed by implementations of the ast-mapper, but why
> are there 2 mechanisms?
>
> 2. What is demonstrated in lines 114-117 regarding polymorphic variant row
> fields:
>
>   | Rtag of label * bool * core_type list
>         (* [`A]                   ( true,  [] )
>            [`A of T]              ( false, [T] )
>            [`A of T1 & .. & Tn]   ( false, [T1;...Tn] )
>            [`A of & T1 & .. & Tn] ( true,  [T1;...Tn] )
>          *)
>
> What does the bool value represent?
> Why are the type separators in the comments using the & symbol?
> What is the difference between the 3rd and 4th example?
>
> 3. line 684: what is the purpose of the override flag on Pstr_open? It's not
> explained by the comment.
>
> 4. The toplevel phrases are not clear. What is the purpose of Ptop_dir on
> line 721?
>
> Like I said, feel free to jump in and answer any one of these questions.
>
> Thanks in advance for everyone's help
>
> -Yotam
>
>
>
>
> On Tue, Apr 1, 2014 at 6:03 AM, Mark Shinwell <mshinwell@janestreet.com>
> wrote:
>>
>> I would suggest that it's probably better to keep the documentation as
>> comments where possible.  However, I think it is important to avoid
>> excessive commentary, especially if it is likely to get out of sync as
>> a result of future modifications to the code.  It may be that in some
>> cases making alterations to the code (for example, improving the name
>> of a variable) is a more satisfactory approach than adding a comment.
>>
>> Thanks for working on this.
>>
>> Mark
>>
>> On 31 March 2014 18:51, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
>> > I think it depends on how much feedback I get on any particular
>> > question. By
>> > default, I would like comments to go in the code. Additionally, there's
>> > the
>> > ocaml-internals wiki at https://github.com/ocamllabs/ocaml-internals
>> > which
>> > will be useful for any concepts that span multiple files, or that are
>> > too
>> > beginner-oriented. I'm guessing that for many things, it will just have
>> > to
>> > be decided on a case-by-case basis.
>> >
>> > Of course, the most important ingredient for the success of this
>> > 'project'
>> > is the willing, patient participation of the core team, as well as the
>> > other
>> > experts on this list.
>> >
>> > -Yotam
>> >
>> >
>> > On Mon, Mar 31, 2014 at 1:06 PM, Milan Stanojević <milanst@gmail.com>
>> > wrote:
>> >>
>> >> Thank you for doing this, I'm interested in learning more about how
>> >> compiler works.
>> >>
>> >> Are you creating a separate file(s) to document the compiler or you
>> >> are adding comments to ml files?
>> >>
>> >> On Mon, Mar 31, 2014 at 11:39 AM, Yotam Barnoy <yotambarnoy@gmail.com>
>> >> wrote:
>> >> > Hi everybody
>> >> >
>> >> > It's been mentioned before that the OCaml compiler's documentation is
>> >> > somewhat lacking. I've been going over the compiler code gradually
>> >> > (both
>> >> > the
>> >> > frontend and the backend) and while some parts are understandable
>> >> > enough,
>> >> > others are missing some basic explanations. Some explanations are
>> >> > also
>> >> > spread out throughout the codebase, making it hard to know what
>> >> > something
>> >> > means unless you've read another part of the codebase that relates to
>> >> > it.
>> >> >
>> >> > Since the call to submit documentation commits has gone mostly
>> >> > unanswered,
>> >> > I'd like to suggest a method of making both my own progress through
>> >> > the
>> >> > code
>> >> > easier and hopefully making it easier for others who will follow.
>> >> >
>> >> > What I'm going to do is, focusing on more or less one file at a time,
>> >> > I'll
>> >> > post newbie questions to the list about the code. Once I'm satisfied
>> >> > that I
>> >> > have a good enough understanding, I'll add comments to the
>> >> > aforementioned
>> >> > files and submit pull requests for them. I also encourage others to
>> >> > do
>> >> > the
>> >> > same.
>> >> >
>> >> > What I need from the list, and especially from the more knowledgeable
>> >> > members (who already know the compiler code) is the willingness to
>> >> > explain
>> >> > the concepts and answer my questions, annoying as they may be. I have
>> >> > a
>> >> > pretty decent background in compilers, ASTs, code generation, etc,
>> >> > but
>> >> > not
>> >> > so much in type inference.
>> >> >
>> >> > I'm not suggesting a particular timeframe for this process -- I'm
>> >> > doing
>> >> > this
>> >> > on the side while working on a research project and TAing, but I
>> >> > really
>> >> > would like to get to the point where I can make significant
>> >> > contributions to
>> >> > the toolchain, and if I can help others who follow in my footsteps,
>> >> > then
>> >> > that's a nice bonus.
>> >> >
>> >> > While I could have skipped this introduction and just proceeded with
>> >> > inundating the list with questions, I felt that this (hopefully)
>> >> > gives a
>> >> > purpose and perhaps motivation for those who have the answers to
>> >> > answer
>> >> > my
>> >> > questions even if they get annoying. In particular, I may often miss
>> >> > some
>> >> > parts that may seem obvious because I don't necessarily have the time
>> >> > to
>> >> > read all the connected code in depth. Hopefully you'll bear with me.
>> >> >
>> >> > Does this sound reasonable to the fine folks on the list?
>> >> >
>> >> > Yotam
>> >
>> >
>
>

Re: [Caml-list] Ocaml compiler documentation

[Alain Frisch]

On 4/3/2014 4:48 AM, Yotam Barnoy wrote:
> Ok I think a good place to start a tour of the compiler is in
> parsing/parsetree.mli. This file is actually very well documented, with
> terse but effective examples of almost every constructor and type.

Good idea indeed, especially that the Parsetree will gain in the next 
release a more important status, with -ppx rewriters, annotations and 
attributes.

> I had to refer to the OCaml manual for a few of the corner cases. For
> example, I didn't know about the #class type shortcut. I think a few
> comments explaining the more obscure facets of the language could be
> helpful.

Generally speaking, a good place to document the language is the user 
manual.  Don't hesitate to suggest patches to the manual as well!  (The 
source code is in the same repository: 
http://caml.inria.fr/cgi-bin/viewvc.cgi/ocamldoc/trunk/ ).

That said, for the specific case of Parsetree, it might indeed be useful 
to give some hints about rare language features in the source code as well.

> 1. What is the difference between an extension and an attribute? From
> what I understand, they are both means of integrating additional
> metadata into the AST that can then be parsed by implementations of the
> ast-mapper, but why are there 2 mechanisms?

An extension is something which will be rejected by the type-checker. 
It is placeholder for "sub-languages", to be processed by -ppx filters.

An attribute is indeed a way to integrate meta-data into a (hopefully) 
valid AST.  They could be used by -ppx filters to drive their behavior, 
by external tools to get some extra information (e.g. Bisect 
annotations), and they are propagated to the typedtreed, and hence to 
.cmt/.cmti files, again for external tools which read those files.  They 
are also kepts on some kinds of declarations (e.g. values and types) so 
that they are part of the Types structures, and hence found in .cmi 
files as well.  The compiler also give a built-in meaning to some 
attributes (to be documented).

Some more information: 
http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/trunk/experimental/frisch/extension_points.txt?revision=HEAD&view=markup

> 3. line 684: what is the purpose of the override flag on Pstr_open? It's
> not explained by the comment.

The override flag (i.e. "open!" as opposed to "open") is used to silence 
the new warning which signals when an open statement shadows an existing 
identifier which is later used.   (It does not affect the 'unused open' 
warning.)

> 4. The toplevel phrases are not clear. What is the purpose of Ptop_dir
> on line 721?

Those are #-directives understood by the toplevel (#use, #load, etc).


-- Alain

Re: [Caml-list] Ocaml compiler documentation

[François Bobot]

On 03/04/2014 22:16, Alain Frisch wrote:
> Generally speaking, a good place to document the language is the user 
> manual.  Don't hesitate to suggest patches to the manual as well!  (The 
> source code is in the same repository: 
> http://caml.inria.fr/cgi-bin/viewvc.cgi/ocamldoc/trunk/ ).

Why the usermanual of ocaml is in not in the ocaml repository? Is it for an historical, practical or
another reason?

When you add a new feature in a branch of the ocaml repository do you create also a branch in the
ocamldoc one for documenting the feature?


PS: viewvc.cgi/ocamldoc and viewvc.cgi/ocaml are on the same svn but not the same repository since
they don't share the same trunk/branches/tags/version directories.


Thank you,

-- 
François

Re: [Caml-list] Ocaml compiler documentation

[Mark Shinwell]

Alain, I think "open!" does affect the "unused open" warning.  (This
is a useful feature when you are trying to establish a consistent set
of in-scope names, a good example being Core's "open
Int.Replace_polymorphic_compare", which you may well want to have even
if your file doesn't happen to use a comparison operator right now.)

I checked this using the following test program and "-w +a".

module M = struct
  let foo x = x + 42
end
open! M         (or "open")

Mark

On 3 April 2014 21:16, Alain Frisch <alain@frisch.fr> wrote:
> On 4/3/2014 4:48 AM, Yotam Barnoy wrote:
>>
>> Ok I think a good place to start a tour of the compiler is in
>> parsing/parsetree.mli. This file is actually very well documented, with
>> terse but effective examples of almost every constructor and type.
>
>
> Good idea indeed, especially that the Parsetree will gain in the next
> release a more important status, with -ppx rewriters, annotations and
> attributes.
>
>
>> I had to refer to the OCaml manual for a few of the corner cases. For
>> example, I didn't know about the #class type shortcut. I think a few
>> comments explaining the more obscure facets of the language could be
>> helpful.
>
>
> Generally speaking, a good place to document the language is the user
> manual.  Don't hesitate to suggest patches to the manual as well!  (The
> source code is in the same repository:
> http://caml.inria.fr/cgi-bin/viewvc.cgi/ocamldoc/trunk/ ).
>
> That said, for the specific case of Parsetree, it might indeed be useful to
> give some hints about rare language features in the source code as well.
>
>
>> 1. What is the difference between an extension and an attribute? From
>> what I understand, they are both means of integrating additional
>> metadata into the AST that can then be parsed by implementations of the
>> ast-mapper, but why are there 2 mechanisms?
>
>
> An extension is something which will be rejected by the type-checker. It is
> placeholder for "sub-languages", to be processed by -ppx filters.
>
> An attribute is indeed a way to integrate meta-data into a (hopefully) valid
> AST.  They could be used by -ppx filters to drive their behavior, by
> external tools to get some extra information (e.g. Bisect annotations), and
> they are propagated to the typedtreed, and hence to .cmt/.cmti files, again
> for external tools which read those files.  They are also kepts on some
> kinds of declarations (e.g. values and types) so that they are part of the
> Types structures, and hence found in .cmi files as well.  The compiler also
> give a built-in meaning to some attributes (to be documented).
>
> Some more information:
> http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/trunk/experimental/frisch/extension_points.txt?revision=HEAD&view=markup
>
>
>> 3. line 684: what is the purpose of the override flag on Pstr_open? It's
>> not explained by the comment.
>
>
> The override flag (i.e. "open!" as opposed to "open") is used to silence the
> new warning which signals when an open statement shadows an existing
> identifier which is later used.   (It does not affect the 'unused open'
> warning.)
>
>
>> 4. The toplevel phrases are not clear. What is the purpose of Ptop_dir
>> on line 721?
>
>
> Those are #-directives understood by the toplevel (#use, #load, etc).
>
>
> -- Alain
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Ocaml compiler documentation

[Alain Frisch]

On 04/14/2014 08:12 AM, Mark Shinwell wrote:
> Alain, I think "open!" does affect the "unused open" warning.  (This
> is a useful feature when you are trying to establish a consistent set
> of in-scope names, a good example being Core's "open
> Int.Replace_polymorphic_compare", which you may well want to have even
> if your file doesn't happen to use a comparison operator right now.)

Indeed, you're right!  (And this is amusing, considering that I 
implemented both warnings...)

I've feel slightly unhappy about using "open!" for two unrelated things, 
but since you consider this behavior to be useful, I won't touch it 
unless someone else has a strong opinion about it.

-- Alain

>
> I checked this using the following test program and "-w +a".
>
> module M = struct
>    let foo x = x + 42
> end
> open! M         (or "open")
>
> Mark
>
> On 3 April 2014 21:16, Alain Frisch <alain@frisch.fr> wrote:
>> On 4/3/2014 4:48 AM, Yotam Barnoy wrote:
>>>
>>> Ok I think a good place to start a tour of the compiler is in
>>> parsing/parsetree.mli. This file is actually very well documented, with
>>> terse but effective examples of almost every constructor and type.
>>
>>
>> Good idea indeed, especially that the Parsetree will gain in the next
>> release a more important status, with -ppx rewriters, annotations and
>> attributes.
>>
>>
>>> I had to refer to the OCaml manual for a few of the corner cases. For
>>> example, I didn't know about the #class type shortcut. I think a few
>>> comments explaining the more obscure facets of the language could be
>>> helpful.
>>
>>
>> Generally speaking, a good place to document the language is the user
>> manual.  Don't hesitate to suggest patches to the manual as well!  (The
>> source code is in the same repository:
>> http://caml.inria.fr/cgi-bin/viewvc.cgi/ocamldoc/trunk/ ).
>>
>> That said, for the specific case of Parsetree, it might indeed be useful to
>> give some hints about rare language features in the source code as well.
>>
>>
>>> 1. What is the difference between an extension and an attribute? From
>>> what I understand, they are both means of integrating additional
>>> metadata into the AST that can then be parsed by implementations of the
>>> ast-mapper, but why are there 2 mechanisms?
>>
>>
>> An extension is something which will be rejected by the type-checker. It is
>> placeholder for "sub-languages", to be processed by -ppx filters.
>>
>> An attribute is indeed a way to integrate meta-data into a (hopefully) valid
>> AST.  They could be used by -ppx filters to drive their behavior, by
>> external tools to get some extra information (e.g. Bisect annotations), and
>> they are propagated to the typedtreed, and hence to .cmt/.cmti files, again
>> for external tools which read those files.  They are also kepts on some
>> kinds of declarations (e.g. values and types) so that they are part of the
>> Types structures, and hence found in .cmi files as well.  The compiler also
>> give a built-in meaning to some attributes (to be documented).
>>
>> Some more information:
>> http://caml.inria.fr/cgi-bin/viewvc.cgi/ocaml/trunk/experimental/frisch/extension_points.txt?revision=HEAD&view=markup
>>
>>
>>> 3. line 684: what is the purpose of the override flag on Pstr_open? It's
>>> not explained by the comment.
>>
>>
>> The override flag (i.e. "open!" as opposed to "open") is used to silence the
>> new warning which signals when an open statement shadows an existing
>> identifier which is later used.   (It does not affect the 'unused open'
>> warning.)
>>
>>
>>> 4. The toplevel phrases are not clear. What is the purpose of Ptop_dir
>>> on line 721?
>>
>>
>> Those are #-directives understood by the toplevel (#use, #load, etc).
>>
>>
>> -- Alain
>>
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>