[Caml-list] Compiling with camlp4 extensions

[Caml-list] Compiling with camlp4 extensions

[Aaron Bohannon]

[-- Attachment #1: Type: text/plain, Size: 451 bytes --]

Hi,

I have been trying to use the new camlp4 to write an OCaml syntax
extension.  All the examples I have seen so far suggest that I use the
extension by passing ocamlc the "-pp" option.  But it seems that all the
location info for error messages gets lost when I do this unless I catch
and report the parse error myself within the extension.  Is there some way
to get ocamlc to report the parse error at the correct location
automatically?

- Aaron

[-- Attachment #2: Type: text/html, Size: 486 bytes --]

Re: [Caml-list] Compiling with camlp4 extensions

[Gabriel Scherer]

All nodes in a Camlp4 AST are annotated with location information; the
locations you get from the parser are correct, and it is your
responsibility, as an extension writer, to ensure that any new nodes
you generate also have (approximately) correct location information.

If you build AST nodes "by hand", you have to provide this location
explicitly. If you use the concrete syntax quotations, the location
used is the value _loc present in the environment, whatever it may be.
So to have correct locations, you have to make sure that, at every AST
you produce through a quotation, there is a "_loc" variable in scope
with the correct value. If you match AST pieces with quotation
patterns (match e with <:expr< $a$ + $b$ >> -> ...), you may bind the
location variable through the syntax "<:expr@foo<", for example:
(match e with <:expr@_loc< $a$ + $b$ >> -> ...). Finally, if you're
inside an EXTEND block defining a parsing rule, the idenfitier _loc is
implicitely bound to a location corresponding to what was parsed by
this rule.

See for example the toy extension pa_refutable, that has example of
those various things:
  http://bluestorm.info/camlp4/pa_refutable.ml.html

In some very rare cases (or if you are perfectionist), you may want to
give to a new node a location that is not quite the location of any of
the parsed node you're working on. You may use various functions of
the Loc submodule of your syntax definition to forge new locations; in
particular, Loc.merge merges two (supposed contiguous) locations.
  http://bluestorm.info/camlp4/camlp4-doc/Sig.Loc.html





On Fri, Jun 22, 2012 at 5:53 PM, Aaron Bohannon <bohannon@seas.upenn.edu> wrote:
> Hi,
>
> I have been trying to use the new camlp4 to write an OCaml syntax
> extension.  All the examples I have seen so far suggest that I use the
> extension by passing ocamlc the "-pp" option.  But it seems that all the
> location info for error messages gets lost when I do this unless I catch and
> report the parse error myself within the extension.  Is there some way to
> get ocamlc to report the parse error at the correct location automatically?
>
> - Aaron

[Caml-list] Re: Compiling with camlp4 extensions

[Hongbo Zhang]

On 6/22/12 11:53 AM, Aaron Bohannon wrote:
> Hi,
Hi,
If you are writing code generator, write a rule making use of 
myocamlbuild, for every file, compiling file_pp.cmo(the preprocessed 
output). and if it's correct, go ahead compiling file.cmo.
For syntax extension, it's your repsonsibility to make it correct.
>
> I have been trying to use the new camlp4 to write an OCaml syntax
> extension.  All the examples I have seen so far suggest that I use the
> extension by passing ocamlc the "-pp" option.  But it seems that all the
> location info for error messages gets lost when I do this unless I catch
> and report the parse error myself within the extension.  Is there some
> way to get ocamlc to report the parse error at the correct location
> automatically?
>
> - Aaron
>

Re: [Caml-list] Compiling with camlp4 extensions

[Aaron Bohannon]

Thanks for the reply.  The example is helpful.  However, I should have
been more clear: I don't exactly want to write a syntax extension, per
se.  Rather, I am trying to use camlp4 to parse a non-OCaml grammar
and to generate an OCaml AST.  So the "Register.OCamlSyntaxExtension"
functor doesn't seem like it will work for me.  Instead, I tried using
"Printers.Ocaml.print_implem" in my "extension" code and everything
works fine, except for error locations.  Of course, I realize this is
because the AST is being printed and then re-parsed, but I don't know
how to prevent it from being reparsed.  I looked through all the
Camlp4 interfaces and thought that perhaps I need to use the function
"Register.register_str_item_parser".  But I couldn't make that work.
Either that's not the function I need or else I don't know how to use
it -- I can't tell which.

 - Aaron

On Fri, Jun 22, 2012 at 10:36 AM, Gabriel Scherer
<gabriel.scherer@gmail.com> wrote:
> All nodes in a Camlp4 AST are annotated with location information; the
> locations you get from the parser are correct, and it is your
> responsibility, as an extension writer, to ensure that any new nodes
> you generate also have (approximately) correct location information.
>
> If you build AST nodes "by hand", you have to provide this location
> explicitly. If you use the concrete syntax quotations, the location
> used is the value _loc present in the environment, whatever it may be.
> So to have correct locations, you have to make sure that, at every AST
> you produce through a quotation, there is a "_loc" variable in scope
> with the correct value. If you match AST pieces with quotation
> patterns (match e with <:expr< $a$ + $b$ >> -> ...), you may bind the
> location variable through the syntax "<:expr@foo<", for example:
> (match e with <:expr@_loc< $a$ + $b$ >> -> ...). Finally, if you're
> inside an EXTEND block defining a parsing rule, the idenfitier _loc is
> implicitely bound to a location corresponding to what was parsed by
> this rule.
>
> See for example the toy extension pa_refutable, that has example of
> those various things:
>  http://bluestorm.info/camlp4/pa_refutable.ml.html
>
> In some very rare cases (or if you are perfectionist), you may want to
> give to a new node a location that is not quite the location of any of
> the parsed node you're working on. You may use various functions of
> the Loc submodule of your syntax definition to forge new locations; in
> particular, Loc.merge merges two (supposed contiguous) locations.
>  http://bluestorm.info/camlp4/camlp4-doc/Sig.Loc.html
>
>
>
>
>
> On Fri, Jun 22, 2012 at 5:53 PM, Aaron Bohannon <bohannon@seas.upenn.edu> wrote:
>> Hi,
>>
>> I have been trying to use the new camlp4 to write an OCaml syntax
>> extension.  All the examples I have seen so far suggest that I use the
>> extension by passing ocamlc the "-pp" option.  But it seems that all the
>> location info for error messages gets lost when I do this unless I catch and
>> report the parse error myself within the extension.  Is there some way to
>> get ocamlc to report the parse error at the correct location automatically?
>>
>> - Aaron

Re: [Caml-list] Compiling with camlp4 extensions

[Gabriel Scherer]

See the "full parser tutorial" in the Camlp4 wiki, it has information
for what, if I have correctly understood, is your use case, including
location handling.
  http://brion.inria.fr/gallium/index.php/Full_parser_tutorial

On Sat, Jun 23, 2012 at 2:43 AM, Aaron Bohannon <bohannon@seas.upenn.edu> wrote:
> Thanks for the reply.  The example is helpful.  However, I should have
> been more clear: I don't exactly want to write a syntax extension, per
> se.  Rather, I am trying to use camlp4 to parse a non-OCaml grammar
> and to generate an OCaml AST.  So the "Register.OCamlSyntaxExtension"
> functor doesn't seem like it will work for me.  Instead, I tried using
> "Printers.Ocaml.print_implem" in my "extension" code and everything
> works fine, except for error locations.  Of course, I realize this is
> because the AST is being printed and then re-parsed, but I don't know
> how to prevent it from being reparsed.  I looked through all the
> Camlp4 interfaces and thought that perhaps I need to use the function
> "Register.register_str_item_parser".  But I couldn't make that work.
> Either that's not the function I need or else I don't know how to use
> it -- I can't tell which.
>
>  - Aaron
>
> On Fri, Jun 22, 2012 at 10:36 AM, Gabriel Scherer
> <gabriel.scherer@gmail.com> wrote:
>> All nodes in a Camlp4 AST are annotated with location information; the
>> locations you get from the parser are correct, and it is your
>> responsibility, as an extension writer, to ensure that any new nodes
>> you generate also have (approximately) correct location information.
>>
>> If you build AST nodes "by hand", you have to provide this location
>> explicitly. If you use the concrete syntax quotations, the location
>> used is the value _loc present in the environment, whatever it may be.
>> So to have correct locations, you have to make sure that, at every AST
>> you produce through a quotation, there is a "_loc" variable in scope
>> with the correct value. If you match AST pieces with quotation
>> patterns (match e with <:expr< $a$ + $b$ >> -> ...), you may bind the
>> location variable through the syntax "<:expr@foo<", for example:
>> (match e with <:expr@_loc< $a$ + $b$ >> -> ...). Finally, if you're
>> inside an EXTEND block defining a parsing rule, the idenfitier _loc is
>> implicitely bound to a location corresponding to what was parsed by
>> this rule.
>>
>> See for example the toy extension pa_refutable, that has example of
>> those various things:
>>  http://bluestorm.info/camlp4/pa_refutable.ml.html
>>
>> In some very rare cases (or if you are perfectionist), you may want to
>> give to a new node a location that is not quite the location of any of
>> the parsed node you're working on. You may use various functions of
>> the Loc submodule of your syntax definition to forge new locations; in
>> particular, Loc.merge merges two (supposed contiguous) locations.
>>  http://bluestorm.info/camlp4/camlp4-doc/Sig.Loc.html
>>
>>
>>
>>
>>
>> On Fri, Jun 22, 2012 at 5:53 PM, Aaron Bohannon <bohannon@seas.upenn.edu> wrote:
>>> Hi,
>>>
>>> I have been trying to use the new camlp4 to write an OCaml syntax
>>> extension.  All the examples I have seen so far suggest that I use the
>>> extension by passing ocamlc the "-pp" option.  But it seems that all the
>>> location info for error messages gets lost when I do this unless I catch and
>>> report the parse error myself within the extension.  Is there some way to
>>> get ocamlc to report the parse error at the correct location automatically?
>>>
>>> - Aaron

Re: [Caml-list] Compiling with camlp4 extensions

[Aaron Bohannon]

Ah, yes.  That is helpful.  I had thought of trying to "extend" OCaml
by replacing the grammar with a different one, although I didn't know
exactly how to do it.

Of course, it seemed obvious to me that I wouldn't be able to use my
own lexer if I did that.  I'm not sure if I will want to do that or
not yet, but I was thinking I would just learn to do it that way so
I'd have that flexibility if I need it.  Unfortunately, the page stops
short of explaining how to pursue that approach. :(

 - Aaron

On Sat, Jun 23, 2012 at 3:42 AM, Gabriel Scherer
<gabriel.scherer@gmail.com> wrote:
> See the "full parser tutorial" in the Camlp4 wiki, it has information
> for what, if I have correctly understood, is your use case, including
> location handling.
>  http://brion.inria.fr/gallium/index.php/Full_parser_tutorial
>
> On Sat, Jun 23, 2012 at 2:43 AM, Aaron Bohannon <bohannon@seas.upenn.edu> wrote:
>> Thanks for the reply.  The example is helpful.  However, I should have
>> been more clear: I don't exactly want to write a syntax extension, per
>> se.  Rather, I am trying to use camlp4 to parse a non-OCaml grammar
>> and to generate an OCaml AST.  So the "Register.OCamlSyntaxExtension"
>> functor doesn't seem like it will work for me.  Instead, I tried using
>> "Printers.Ocaml.print_implem" in my "extension" code and everything
>> works fine, except for error locations.  Of course, I realize this is
>> because the AST is being printed and then re-parsed, but I don't know
>> how to prevent it from being reparsed.  I looked through all the
>> Camlp4 interfaces and thought that perhaps I need to use the function
>> "Register.register_str_item_parser".  But I couldn't make that work.
>> Either that's not the function I need or else I don't know how to use
>> it -- I can't tell which.
>>
>>  - Aaron
>>
>> On Fri, Jun 22, 2012 at 10:36 AM, Gabriel Scherer
>> <gabriel.scherer@gmail.com> wrote:
>>> All nodes in a Camlp4 AST are annotated with location information; the
>>> locations you get from the parser are correct, and it is your
>>> responsibility, as an extension writer, to ensure that any new nodes
>>> you generate also have (approximately) correct location information.
>>>
>>> If you build AST nodes "by hand", you have to provide this location
>>> explicitly. If you use the concrete syntax quotations, the location
>>> used is the value _loc present in the environment, whatever it may be.
>>> So to have correct locations, you have to make sure that, at every AST
>>> you produce through a quotation, there is a "_loc" variable in scope
>>> with the correct value. If you match AST pieces with quotation
>>> patterns (match e with <:expr< $a$ + $b$ >> -> ...), you may bind the
>>> location variable through the syntax "<:expr@foo<", for example:
>>> (match e with <:expr@_loc< $a$ + $b$ >> -> ...). Finally, if you're
>>> inside an EXTEND block defining a parsing rule, the idenfitier _loc is
>>> implicitely bound to a location corresponding to what was parsed by
>>> this rule.
>>>
>>> See for example the toy extension pa_refutable, that has example of
>>> those various things:
>>>  http://bluestorm.info/camlp4/pa_refutable.ml.html
>>>
>>> In some very rare cases (or if you are perfectionist), you may want to
>>> give to a new node a location that is not quite the location of any of
>>> the parsed node you're working on. You may use various functions of
>>> the Loc submodule of your syntax definition to forge new locations; in
>>> particular, Loc.merge merges two (supposed contiguous) locations.
>>>  http://bluestorm.info/camlp4/camlp4-doc/Sig.Loc.html
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jun 22, 2012 at 5:53 PM, Aaron Bohannon <bohannon@seas.upenn.edu> wrote:
>>>> Hi,
>>>>
>>>> I have been trying to use the new camlp4 to write an OCaml syntax
>>>> extension.  All the examples I have seen so far suggest that I use the
>>>> extension by passing ocamlc the "-pp" option.  But it seems that all the
>>>> location info for error messages gets lost when I do this unless I catch and
>>>> report the parse error myself within the extension.  Is there some way to
>>>> get ocamlc to report the parse error at the correct location automatically?
>>>>
>>>> - Aaron

Re: [Caml-list] Compiling with camlp4 extensions

[Gabriel Scherer]

If you want to implement your own lexer, you have to provide the
MakeGram functor your own module satisfying the Lexer signature.
  http://bluestorm.info/camlp4/camlp4-doc/Sig.Lexer.html

If you can reuse Camlp4's predefined lexer, however, you should not
hesitate to do that. There is little use in being original on the
lexing part, and users that already know OCaml will appreciate the
consistency in the lexical conventions. Camlp4's token type for OCaml
is rich enough to integrate comments and whitespace information, so
you can even define an indentation-dependent language on top of the
pre-existing lexer, using a filtering function on the token stream :
  http://bluestorm.info/camlp4/camlp4-doc/Sig.Token.Filter.html

On Sat, Jun 23, 2012 at 3:41 PM, Aaron Bohannon <bohannon@seas.upenn.edu> wrote:
> Ah, yes.  That is helpful.  I had thought of trying to "extend" OCaml
> by replacing the grammar with a different one, although I didn't know
> exactly how to do it.
>
> Of course, it seemed obvious to me that I wouldn't be able to use my
> own lexer if I did that.  I'm not sure if I will want to do that or
> not yet, but I was thinking I would just learn to do it that way so
> I'd have that flexibility if I need it.  Unfortunately, the page stops
> short of explaining how to pursue that approach. :(
>
>  - Aaron
>
> On Sat, Jun 23, 2012 at 3:42 AM, Gabriel Scherer
> <gabriel.scherer@gmail.com> wrote:
>> See the "full parser tutorial" in the Camlp4 wiki, it has information
>> for what, if I have correctly understood, is your use case, including
>> location handling.
>>  http://brion.inria.fr/gallium/index.php/Full_parser_tutorial
>>
>> On Sat, Jun 23, 2012 at 2:43 AM, Aaron Bohannon <bohannon@seas.upenn.edu> wrote:
>>> Thanks for the reply.  The example is helpful.  However, I should have
>>> been more clear: I don't exactly want to write a syntax extension, per
>>> se.  Rather, I am trying to use camlp4 to parse a non-OCaml grammar
>>> and to generate an OCaml AST.  So the "Register.OCamlSyntaxExtension"
>>> functor doesn't seem like it will work for me.  Instead, I tried using
>>> "Printers.Ocaml.print_implem" in my "extension" code and everything
>>> works fine, except for error locations.  Of course, I realize this is
>>> because the AST is being printed and then re-parsed, but I don't know
>>> how to prevent it from being reparsed.  I looked through all the
>>> Camlp4 interfaces and thought that perhaps I need to use the function
>>> "Register.register_str_item_parser".  But I couldn't make that work.
>>> Either that's not the function I need or else I don't know how to use
>>> it -- I can't tell which.
>>>
>>>  - Aaron
>>>
>>> On Fri, Jun 22, 2012 at 10:36 AM, Gabriel Scherer
>>> <gabriel.scherer@gmail.com> wrote:
>>>> All nodes in a Camlp4 AST are annotated with location information; the
>>>> locations you get from the parser are correct, and it is your
>>>> responsibility, as an extension writer, to ensure that any new nodes
>>>> you generate also have (approximately) correct location information.
>>>>
>>>> If you build AST nodes "by hand", you have to provide this location
>>>> explicitly. If you use the concrete syntax quotations, the location
>>>> used is the value _loc present in the environment, whatever it may be.
>>>> So to have correct locations, you have to make sure that, at every AST
>>>> you produce through a quotation, there is a "_loc" variable in scope
>>>> with the correct value. If you match AST pieces with quotation
>>>> patterns (match e with <:expr< $a$ + $b$ >> -> ...), you may bind the
>>>> location variable through the syntax "<:expr@foo<", for example:
>>>> (match e with <:expr@_loc< $a$ + $b$ >> -> ...). Finally, if you're
>>>> inside an EXTEND block defining a parsing rule, the idenfitier _loc is
>>>> implicitely bound to a location corresponding to what was parsed by
>>>> this rule.
>>>>
>>>> See for example the toy extension pa_refutable, that has example of
>>>> those various things:
>>>>  http://bluestorm.info/camlp4/pa_refutable.ml.html
>>>>
>>>> In some very rare cases (or if you are perfectionist), you may want to
>>>> give to a new node a location that is not quite the location of any of
>>>> the parsed node you're working on. You may use various functions of
>>>> the Loc submodule of your syntax definition to forge new locations; in
>>>> particular, Loc.merge merges two (supposed contiguous) locations.
>>>>  http://bluestorm.info/camlp4/camlp4-doc/Sig.Loc.html
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Jun 22, 2012 at 5:53 PM, Aaron Bohannon <bohannon@seas.upenn.edu> wrote:
>>>>> Hi,
>>>>>
>>>>> I have been trying to use the new camlp4 to write an OCaml syntax
>>>>> extension.  All the examples I have seen so far suggest that I use the
>>>>> extension by passing ocamlc the "-pp" option.  But it seems that all the
>>>>> location info for error messages gets lost when I do this unless I catch and
>>>>> report the parse error myself within the extension.  Is there some way to
>>>>> get ocamlc to report the parse error at the correct location automatically?
>>>>>
>>>>> - Aaron