ocamlbuild, menhir and keeping tokens in a separate file

ocamlbuild, menhir and keeping tokens in a separate file

[Joel Reymont]

How can I specify with ocamlbuild that to build my easy_parser.mly I  
need to include easy_tokens.mly on the menhir command line, as well  
as give menhir --base easy_parser?

The command I'm looking for is below:

menhir --explain --infer --base easy_parser easy_tokens.mly  
easy_parser.mly

	Thanks, Joel

--
http://wagerlabs.com/

Re: ocamlbuild, menhir and keeping tokens in a separate file

[Joel Reymont]

I forgot to mention that easy_tokens.mly should always be used in the  
same command line as easy_parser.mly, even during the dependency run.  
I see this right now, which obviously doesn't work.

menhir --raw-depend --ocamldep 'ocamldep.opt -modules'  
easy_parser.mly > easy_parser.mly.depends

File "easy_parser.mly", line 453, characters 4-14:
Error: DARK_GREEN is undefined.
        ^^^
is in easy_tokens.mly

	Thanks, Joel

On Apr 28, 2007, at 12:38 AM, Joel Reymont wrote:

> How can I specify with ocamlbuild that to build my easy_parser.mly  
> I need to include easy_tokens.mly on the menhir command line, as  
> well as give menhir --base easy_parser?
>
> The command I'm looking for is below:
>
> menhir --explain --infer --base easy_parser easy_tokens.mly  
> easy_parser.mly

--
http://wagerlabs.com/

Re: ocamlbuild, menhir and keeping tokens in a separate file

[Joel Reymont]

This is what I came up with:

flag ["ocaml"; "parser"; "mehir" ] (A"--explain");
flag ["ocaml"; "menhir_ocamldep" ] (A"easy_tokens.mly");
flag ["ocaml"; "menhir_ocamldep" ] (A"--base");
flag ["ocaml"; "menhir_ocamldep" ] (A"easy_parser");

Can the 3 different (A"..")s be combined?

This works but ocamlbuild doesn't know anything about easy_tokens and  
doesn't copy it to _build which results in a "file not found" error.

Is there a way to say that this token file should always go on the  
command line with this mly file? Hardcoding like above only works if  
there's one parser in the project.

I tried the following off of the top of my head but it doesn't work.

dep ["ocaml"; "parser"; "menhir"; "tokens"] ["easy_tokens.mly"];

in _tags

"easy_parser.mly": tokens

	Thanks, Joel

--
http://wagerlabs.com/

Re: ocamlbuild, menhir and keeping tokens in a separate file

[Joel Reymont]

And one last request... Since I'm manually including easy_tokens.mly  
in the menhir command line I tell ocamlbuild to ignore it with - 
ignore Easy_tokens. I do open Easy_tokens in the lexer.

How can I make the lexer module depend on the parser so that the  
parser is always built first?

This is what happens now. Have I painted myself into a corner?

ocamlfind ocamlc -package ounit -c -o easy_lexer.cmo easy_lexer.ml
+ ocamlfind ocamlc -package ounit -c -o easy_lexer.cmo easy_lexer.ml
File "easy_lexer.mll", line 6, characters 2-18:
Unbound module Easy_tokens
Command exited with code 2.

	Thanks, Joel

--
http://wagerlabs.com/

Re: [Caml-list] Re: ocamlbuild, menhir and keeping tokens in a separate file

[Nicolas Pouillard]

On 4/28/07, Joel Reymont <joelr1@gmail.com> wrote:
> This is what I came up with:
>
> flag ["ocaml"; "parser"; "mehir" ] (A"--explain");
> flag ["ocaml"; "menhir_ocamldep" ] (A"easy_tokens.mly");
> flag ["ocaml"; "menhir_ocamldep" ] (A"--base");
> flag ["ocaml"; "menhir_ocamldep" ] (A"easy_parser");
>
> Can the 3 different (A"..")s be combined?

flag ["ocaml"; "menhir_ocamldep" ] (S[A"easy_tokens.mly"; A"--base";
A"easy_parser"]);

> This works but ocamlbuild doesn't know anything about easy_tokens and
> doesn't copy it to _build which results in a "file not found" error.
>
> Is there a way to say that this token file should always go on the
> command line with this mly file? Hardcoding like above only works if
> there's one parser in the project.
>
> I tried the following off of the top of my head but it doesn't work.
>
> dep ["ocaml"; "parser"; "menhir"; "tokens"] ["easy_tokens.mly"];
>
> in _tags
>
> "easy_parser.mly": tokens

Hum this is supposed to works.

Can you try these one

dep ["ocaml"; "parser"; "menhir"] ["easy_tokens.mly"];

dep ["ocaml"; "parser"; "menhir"; "file:easy_parser.mly"] ["easy_tokens.mly"];

-- 
Nicolas Pouillard

Re: [Caml-list] Re: ocamlbuild, menhir and keeping tokens in a separate file

[Nicolas Pouillard]

On 4/28/07, Joel Reymont <joelr1@gmail.com> wrote:
> And one last request... Since I'm manually including easy_tokens.mly
> in the menhir command line I tell ocamlbuild to ignore it with -
> ignore Easy_tokens. I do open Easy_tokens in the lexer.

Ignoring is not a good idea, I hope that you will find how to include
dependencies to make it works.

-- 
Nicolas Pouillard

Re: [Caml-list] Re: ocamlbuild, menhir and keeping tokens in a separate file

[Joel Reymont]

I need the same options to Menhir when calculating dependencies so I  
thought I would replace  "ocaml"; "parser"; "menhir" with  
"menhir_ocamldep". This works, for example:

flag ["ocaml"; "menhir_ocamldep" ] (S[A"--external-tokens";  
A"EasyToken";]);

Still,

_tags:

"EasyParser.mly": use_tokens

myocamlbuild.ml:

dep ["ocaml"; "menhir_ocamldep"; "use_tokens"] ["EasyToken.mly"];

+ /usr/local/bin/menhir --external-tokens EasyToken --raw-depend -- 
ocamldep '/usr/local/bin/ocamldep.opt -modules' EasyParser.mly >  
EasyParser.mly.depends

Doesn't seem to work, I don't see EasyToken.mly included on the  
command line. I was expecting to see "EasyParser.mly EasyToken.mly"

On Apr 28, 2007, at 4:12 PM, Nicolas Pouillard wrote:

> Can you try these one
>
> dep ["ocaml"; "parser"; "menhir"] ["easy_tokens.mly"];

dep ["ocaml"; "menhir_ocamldep"] ["EasyToken.mly"];

+ /usr/local/bin/menhir --external-tokens EasyToken --raw-depend -- 
ocamldep '/usr/local/bin/ocamldep.opt -modules' EasyParser.mly >  
EasyParser.mly.depends

Not working.

> dep ["ocaml"; "parser"; "menhir"; "file:easy_parser.mly"]  
> ["easy_tokens.mly"];

dep ["ocaml"; "menhir_ocamldep"; "file:EasyParser.mly"]  
["EasyToken.mly"];

+ /usr/local/bin/menhir --external-tokens EasyToken --raw-depend -- 
ocamldep '/usr/local/bin/ocamldep.opt -modules' EasyParser.mly >  
EasyParser.mly.depends

Doesn't work either.


--
http://wagerlabs.com/

Re: [Caml-list] Re: ocamlbuild, menhir and keeping tokens in a separate file

[Nicolas Pouillard]

On 5/2/07, Joel Reymont <joelr1@gmail.com> wrote:
> I need the same options to Menhir when calculating dependencies so I
> thought I would replace  "ocaml"; "parser"; "menhir" with
> "menhir_ocamldep". This works, for example:
>
> flag ["ocaml"; "menhir_ocamldep" ] (S[A"--external-tokens";
> A"EasyToken";]);
>
> Still,
>
> _tags:
>
> "EasyParser.mly": use_tokens
>
> myocamlbuild.ml:
>
> dep ["ocaml"; "menhir_ocamldep"; "use_tokens"] ["EasyToken.mly"];
>
> + /usr/local/bin/menhir --external-tokens EasyToken --raw-depend --
> ocamldep '/usr/local/bin/ocamldep.opt -modules' EasyParser.mly >
> EasyParser.mly.depends
>
> Doesn't seem to work, I don't see EasyToken.mly included on the
> command line. I was expecting to see "EasyParser.mly EasyToken.mly"

The main behavior of dep is just to build EasyToken.mly before running
the command, not to include it into the command line. You can use flag
to insert it into the command line.

-- 
Nicolas Pouillard

Re: [Caml-list] Re: ocamlbuild, menhir and keeping tokens in a separate file

[Joel Reymont]

On May 2, 2007, at 12:54 PM, Nicolas Pouillard wrote:

> The main behavior of dep is just to build EasyToken.mly before running
> the command, not to include it into the command line. You can use flag
> to insert it into the command line.

This is what I use now:

let menhir_opts = S[A"--external-tokens";
                     A"EasyToken";
                     A"--base";
                     A"EasyParser";
                     A"EasyToken.mly"
                    ];;

flag ["ocaml"; "parser"; "menhir" ] (menhir_opts);
flag ["ocaml"; "menhir_ocamldep" ] (menhir_opts);

Two things happen, though, both undesirable:

1) Menhir cannot find EasyToken.mly unless I manually go into _build  
and symlink the file from ../

2) ocamlbuild tries to compile EasyToken.mly. I want to avoid this as  
this file is not set up for compilation.

How do I prevent ocamlbuild from trying to compile EasyToken.mly  
while still copying it into _build/ for Menhir to find it?

Alternatively, how do I specify custom flags for the Menhir run on  
EasyToken.mly? I need to add --token-only, for example.

And one last thing, how do I apply menhir_opts to certain mly files  
but not others?

	Thanks, Joel

--
http://wagerlabs.com/

Re: [Caml-list] Re: ocamlbuild, menhir and keeping tokens in a separate file

[Joel Reymont]

Nicolas,

Thank you very much for your help and answers. I'm slowly learning!

I have the following in my plugin now...

 > flag ["ocaml"; "parser"; "menhir" ] (A"--explain");

I always want to run Menhir with --explain in the parsing phase.

 > flag ["ocaml"; "parser"; "menhir"; "only_tokens" ] (A"--only- 
tokens");
 > flag ["ocaml"; "menhir_ocamldep"; "only_tokens" ] (A"--only-tokens");

I wish the tail portion could be combined, i.e. "only_tokens" ] (A"-- 
only-tokens"); created once and then appended somehow. Just a minor  
wish, though, although comments are always appreciated.

 > flag ["ocaml"; "parser"; "menhir"; "ext_tokens" ] (S[A"--external- 
tokens";
                                                        A"EasyToken";
                                                         
A"EasyToken.mly";
                                                       ]);
 > flag ["ocaml"; "menhir_ocamldep"; "ext_tokens" ] (S[A"--external- 
tokens";
                                                     A"EasyToken";
                                                     A"EasyToken.mly";
                                                    ]);

Is there a way to create a "hole" for A"EasyToken" that can be filled  
from the _tags file?

I may have different parsers in the project and not all of then would  
use EasyToken as the external token module.

 > flag ["ocaml"; "parser"; "menhir"; "token_base" ] (S[A"--base";
                                                        A"EasyParser";
                                                       ]);
 > flag ["ocaml"; "menhir_ocamldep"; "token_base" ] (S[A"--base";
                                                       A"EasyParser";
                                                      ]);

Another tag for Menhir.

 > dep ["ocaml"; "menhir_ocamldep"; "use_tokens"] ["EasyToken.mly"];
 > dep ["ocaml"; "parser"; "menhir"; "use_tokens"] ["EasyToken.mly"];

The token file needs to be generated before the parser. Can a hole be  
created here to be filled from _tags? My token file won't always be  
EasyToken.mly.

Now in the _tags file I have...

 > true: use_menhir

I want to always use Menhir.

 > "EasyParser.mly": token_base, use_tokens, ext_tokens

I thought what I'm telling ocamlbuild is that to generate EasyParser  
it needs to pass Menhir the --base ... and ext_tokens flags, and that  
EasyParser.mly depends on EasyToken.mly.

I may have gotten the dependency wrong since EasyToken.mly is not a  
generated file, it just needs to be processed before. In fact, I may  
not even need the dependency since I'm force-feeding Menhir the file  
on the command line.

 > "EasyToken.mly": only_tokens

I want to compile the above with _just_ the --only-tokens option.

What happens when I run ocamlbuild?

ocamlbuild Test.byte
+ /usr/local/bin/menhir --external-tokens EasyToken EasyToken.mly -- 
base EasyParser --raw-depend --ocamldep '/usr/local/bin/ocamldep.opt - 
modules' EasyParser.mly > EasyParser.mly.depends
Error: EasyToken.mly: No such file or directory

Why are the external tokens and base options used above???

And why is EasyToken.mly not found?

What did I get wrong in my plugin and tags file?

	Thanks, Joel

--
http://wagerlabs.com/

Re: [Caml-list] Re: ocamlbuild, menhir and keeping tokens in a separate file

[Nicolas Pouillard]

On 5/2/07, Joel Reymont <joelr1@gmail.com> wrote:
> Nicolas,
>
> Thank you very much for your help and answers. I'm slowly learning!
>
> I have the following in my plugin now...
>
>  > flag ["ocaml"; "parser"; "menhir" ] (A"--explain");
>
> I always want to run Menhir with --explain in the parsing phase.
>

Ok

>  > flag ["ocaml"; "parser"; "menhir"; "only_tokens" ] (A"--only-
> tokens");
>  > flag ["ocaml"; "menhir_ocamldep"; "only_tokens" ] (A"--only-tokens");
>
> I wish the tail portion could be combined, i.e. "only_tokens" ] (A"--
> only-tokens"); created once and then appended somehow. Just a minor
> wish, though, although comments are always appreciated.

That kind of factorisation is not yet available.

>  > flag ["ocaml"; "parser"; "menhir"; "ext_tokens" ] (S[A"--external-
> tokens";
>                                                         A"EasyToken";
>
> A"EasyToken.mly";
>                                                        ]);
>  > flag ["ocaml"; "menhir_ocamldep"; "ext_tokens" ] (S[A"--external-
> tokens";
>                                                      A"EasyToken";
>                                                      A"EasyToken.mly";
>                                                     ]);
>
> Is there a way to create a "hole" for A"EasyToken" that can be filled
> from the _tags file?
>
> I may have different parsers in the project and not all of then would
> use EasyToken as the external token module.

No but you can have some rules for each parser.

You can generate these rules:

let mk_ext_token parser token =
  flag ["ocaml"; "menhir_ocamldep"; "file:"^parser^".mly"]
    (S[A"--external-tokens"; A token; A(token^".mly")]);
  flag ["ocaml"; "parser"; "menhir"; "file:"^parser^".mly"]
    (S[A"--base"; A parser]);
  ....

[...]

> What happens when I run ocamlbuild?
>
> ocamlbuild Test.byte
> + /usr/local/bin/menhir --external-tokens EasyToken EasyToken.mly --
> base EasyParser --raw-depend --ocamldep '/usr/local/bin/ocamldep.opt -
> modules' EasyParser.mly > EasyParser.mly.depends
> Error: EasyToken.mly: No such file or directory
>
> Why are the external tokens and base options used above???
>
> And why is EasyToken.mly not found?
>
> What did I get wrong in my plugin and tags file?

>From here I don't know why it fails...

-- 
Nicolas Pouillard

Re: [Caml-list] Re: ocamlbuild, menhir and keeping tokens in a separate file

[Joel Reymont]

On May 2, 2007, at 2:08 PM, Nicolas Pouillard wrote:

> You can generate these rules:
>
> let mk_ext_token parser token =
>  flag ["ocaml"; "menhir_ocamldep"; "file:"^parser^".mly"]
>    (S[A"--external-tokens"; A token; A(token^".mly")]);
>  flag ["ocaml"; "parser"; "menhir"; "file:"^parser^".mly"]
>    (S[A"--base"; A parser]);

What would I need to put in my _tags file to use the above? I just  
have this for now:

"EasyToken.mly": only_tokens
"EasyParser.mly": token_base, ext_tokens

>> Error: EasyToken.mly: No such file or directory
>>
> From here I don't know why it fails...

It fails because EasyToken.mly has not been copied to _build. How do  
I force a copy?

For now I have to use "../EasyToken.mly" below

flag ["ocaml"; "parser"; "menhir"; "ext_tokens" ] (S[A"--external- 
tokens";
                                                      A"EasyToken";
                                                      A"../ 
EasyToken.mly";
                                                     ]);

Still, there's something wrong with dependencies

ocamlbuild Test.byte
+ /usr/local/bin/menhir --ocamlc 'ocamlfind ocamlc -package ounit' -- 
explain --external-tokens EasyToken ../EasyToken.mly --base  
EasyParser --infer EasyParser.mly
File "EasyParser.mly", line 54, characters 27-44:
Unbound type constructor EasySymtab.symbol

The problem is that EasySymtab has not been compiled yet. _build/ 
EasyParser.mly.depends is size 0 which is wrong since EasyParser.mly  
opens a bunch of modules. I tried this a couple of times from scratch  
and it's always size 0.

The tail of the log file below shows that ocamlbuild runs  
dependencies on EasyLexer but proceeds to compile EasyParser without  
figuring out its dependencies.

# Target: EasyLexer.mll, tags: { extension:mll, file:EasyLexer.mll,  
lexer, ocaml, ocamllex, quiet, traverse, use_menhir }
/usr/local/bin/ocamllex.opt -q EasyLexer.mll # cached

# Target: EasyLexer.ml.depends, tags: { extension:ml,  
file:EasyLexer.ml, ocaml, ocamldep, quiet, traverse, use_menhir }
/usr/local/bin/ocamldep.opt -modules EasyLexer.ml >  
EasyLexer.ml.depends # cached

# Target: EasyParser.mly, tags: { ext_tokens, extension:mly,  
file:EasyParser.mly, menhir, ocaml, parser, quiet, token_base,  
traverse, use_menhir }
/usr/local/bin/menhir --ocamlc 'ocamlfind ocamlc -package ounit' -- 
explain --external-tokens EasyToken ../EasyToken.mly --base  
EasyParser --infer EasyParser.mly


--
http://wagerlabs.com/

Re: [Caml-list] Re: ocamlbuild, menhir and keeping tokens in a separate file

[Joel Reymont]

On May 2, 2007, at 2:08 PM, Nicolas Pouillard wrote:

> From here I don't know why it fails...

Apparently, dependencies are not being generated correctly for mly  
files.

both _build/EasyToken.mly.depends and _build/EasyParser.mly.depends  
are empty.

It's strange because there's no _build/EasyLexer.mll.depends, for  
example, there's the *.ml.depends instead.

I added a manual dependency on EasySymtab and tagged EasyParser.mly  
accordingly, now everything works around the empty dependency files.

myocamlbuild.ml:

dep ["ocaml"; "menhir_ocamldep"; "symtab"] ["EasySymtab"];
dep ["ocaml"; "parser"; "menhir"; "symtab"] ["EasySymtab"];

_tags:

"EasyParser.mly": token_base, ext_tokens, symtab

Is this a bug in ocamlbuild? Menhir?

	Thanks, Joel

--
http://wagerlabs.com/

Re: [Caml-list] Re: ocamlbuild, menhir and keeping tokens in a separate file

[Joel Reymont]

["EasySymtab"] did not make sense to ocamlbuild, apparently, so it  
never tried to build my parser. I had to change to the following. It  
does trigger complication of EasyParser.mly but is it correct?

dep ["ocaml"; "menhir_ocamldep"; "symtab"] ["EasySymtab.ml"];
dep ["ocaml"; "parser"; "menhir"; "symtab"] ["EasySymtab.cmo"];


--
http://wagerlabs.com/

Re: [Caml-list] Re: ocamlbuild, menhir and keeping tokens in a separate file

[Nicolas Pouillard]

On 5/2/07, Joel Reymont <joelr1@gmail.com> wrote:
> ["EasySymtab"] did not make sense to ocamlbuild, apparently, so it
> never tried to build my parser. I had to change to the following. It
> does trigger complication of EasyParser.mly but is it correct?
>
> dep ["ocaml"; "menhir_ocamldep"; "symtab"] ["EasySymtab.ml"];
> dep ["ocaml"; "parser"; "menhir"; "symtab"] ["EasySymtab.cmo"];

cmi should be sufficent instead of cmo.

"file:EasyParser.mly" should replace "symtab" it's shorter since it
doesn't appear in the tag file.

-- 
Nicolas Pouillard