How can I generate an AST?

How can I generate an AST?

[Loup Vaillant]

Hello,

I would like to write a preprocessor of my own (to make a lisp
syntax). In the process, I am slowly catching up with the Ocaml AST.

However, I am clueless about how I should feed the compiler with my
generated AST. Should I use Marshall? should I output the same as the
option -dparsetree? Should I use some "purple magic"?

Can anybody give me a pointer about how to do this?

Thanks,
Loup Vaillant

Re: [Caml-list] How can I generate an AST?

[Daniel de Rauglaudre]

Hi,

On Sun, Aug 19, 2007 at 07:39:50PM +0200, Loup Vaillant wrote:

> I would like to write a preprocessor of my own (to make a lisp
> syntax). In the process, I am slowly catching up with the Ocaml AST.

If you just need a Lisp syntax, you can use Camlp5 which proposes a
Lisp syntax among its parsing kits:
   http://pauillac.inria.fr/~ddr/camlp5/doc/html/scheme.html

Perhaps (probably) incomplete, but you can complete or fix it if you
want.

If you are interested on the fact of *programming* yourself a new
syntax, use Camlp5: make your parser generating Camlp5 AST and the
rest (dumping syntax tree) is already done by Camlp5. See the doc
or ask who knows for hints (me or other).

-- 
Daniel de Rauglaudre
http://pauillac.inria.fr/~ddr/

Re: [Caml-list] How can I generate an AST?

[Loup Vaillant]

2007/8/19, Daniel de Rauglaudre <daniel.de_rauglaudre@inria.fr>:
> Hi,
>
> On Sun, Aug 19, 2007 at 07:39:50PM +0200, Loup Vaillant wrote:
>
> > I would like to write a preprocessor of my own (to make a lisp
> > syntax). In the process, I am slowly catching up with the Ocaml AST.
>
> If you just need a Lisp syntax, you can use Camlp5 which proposes a
> Lisp syntax among its parsing kits:
>   http://pauillac.inria.fr/~ddr/camlp5/doc/html/scheme.html
>
> Perhaps (probably) incomplete, but you can complete or fix it if you
> want.

Looks great, but unfortunately, I think it is unsuitable for what I
want to do : a macro system based on a lisp-like syntax. I want reader
macros as well as regular ones. (By the way, have I overlooked a
"defmacro" in your syntax?)

> If you are interested on the fact of *programming* yourself a new
> syntax, use Camlp5: make your parser generating Camlp5 AST and the
> rest (dumping syntax tree) is already done by Camlp5. See the doc
> or ask who knows for hints (me or other).

This may be a better approach. However, I am not eager to use camlp*,
for it will introduce a additional dependency. I may choose this
however, if the camlp* AST is simpler than the Ocaml AST. Where can I
find it?

Thank you,
Loup

Re: [Caml-list] How can I generate an AST?

[Daniel de Rauglaudre]

Hi,

On Mon, Aug 20, 2007 at 10:22:17AM +0200, Loup Vaillant wrote:

> This may be a better approach. However, I am not eager to use camlp*,
> for it will introduce a additional dependency. I may choose this
> however, if the camlp* AST is simpler than the Ocaml AST. Where can I
> find it?

Solution using camlp5:

  For the syntax tree, look at:
    http://pauillac.inria.fr/~ddr/camlp5/doc/html/ml_ast.html

  You can generate abstract syntax tree by using concrete syntax. For
  example, the syntax tree for the "if" statement is (in revised syntax):
    MLast.ExIfe loc e1 e2 e3

  but you can write it with quotations (in revised syntax):
    <:expr< if $e1$ then $e2$ else $e3$ >>

  and build syntax trees as complicated as you want using quotations.

  But, well, you need to know camlp5, understand the quotation system,
  perhaps the revised syntax too, in one word, read the doc, and I can
  understand that if you did not use camlp5 before, it is some work...

Solution using camlp4:

  The principle is the same but the syntax tree is different.

Other solution not using camlp*:

  Read the ocaml sources, file "parsing/parser.mly" to understand how
  the ocaml syntax tree works. Program your parser to generate this
  syntax tree. Once done, to give your syntax tree directly to the
  ocaml compiler, output:

  1/ the magic number you can find in ocaml sources at utils/config.ml
       - ast_impl_magic_number for an implementation
         ast_intf_magic_number for an interface
  2/ your input file name
  3/ your syntax tree in binary with "output_value"

-- 
Daniel de Rauglaudre
http://pauillac.inria.fr/~ddr/

Re: [Caml-list] How can I generate an AST?

[Loup Vaillant]

2007/8/20, Daniel de Rauglaudre <daniel.de_rauglaudre@inria.fr>:
> On Mon, Aug 20, 2007 at 10:22:17AM +0200, Loup Vaillant wrote:
>
> > This may be a better approach. However, I am not eager to use camlp*,
> > for it will introduce a additional dependency. I may choose this
> > however, if the camlp* AST is simpler than the Ocaml AST. Where can I
> > find it?
>
> Solution using camlp5:
> [...]
>  But, well, you need to know camlp5, understand the quotation system,
>  perhaps the revised syntax too, in one word, read the doc, and I can
>  understand that if you did not use camlp5 before, it is some work...

Indeed. I didn't work with camlp* before, so the learning curve may be
a problem, I don't know.

> Other solution not using camlp*:
>
>  Read the ocaml sources, file "parsing/parser.mly" to understand how
>  the ocaml syntax tree works. Program your parser to generate this
>  syntax tree.

I have already read "parsing/parstree.mli", so I assume half the work
is done! (well, I hope so, for I don't know .mly files...)

> Once done, to give your syntax tree directly to the
>  ocaml compiler, output:
>
>  1/ the magic number you can find in ocaml sources at utils/config.ml
>       - ast_impl_magic_number for an implementation
>         ast_intf_magic_number for an interface
>  2/ your input file name
>  3/ your syntax tree in binary with "output_value"

That, is what I need. Thank you. I'll notify the list once I managed
to produce some output.

I have still one problem, though : how should I access this code? It
is not part of the standard library, so I can't just "open Config;;".
I can't hard-copy this code, either, that is too ugly. Maybe there is
a way to install a "development package" of Ocaml so I can link it?

Thanks,
Loup

Re: [Caml-list] How can I generate an AST?

[Massimiliano Brocchini]

[-- Attachment #1: Type: text/plain, Size: 2336 bytes --]

On Debian or Archlinux you have the "ocaml-compiler-libs" package which
contains what you need.

Massimiliano Brocchini

On 8/20/07, Loup Vaillant <loup.vaillant@gmail.com> wrote:
>
> 2007/8/20, Daniel de Rauglaudre <daniel.de_rauglaudre@inria.fr>:
> > On Mon, Aug 20, 2007 at 10:22:17AM +0200, Loup Vaillant wrote:
> >
> > > This may be a better approach. However, I am not eager to use camlp*,
> > > for it will introduce a additional dependency. I may choose this
> > > however, if the camlp* AST is simpler than the Ocaml AST. Where can I
> > > find it?
> >
> > Solution using camlp5:
> > [...]
> >  But, well, you need to know camlp5, understand the quotation system,
> >  perhaps the revised syntax too, in one word, read the doc, and I can
> >  understand that if you did not use camlp5 before, it is some work...
>
> Indeed. I didn't work with camlp* before, so the learning curve may be
> a problem, I don't know.
>
> > Other solution not using camlp*:
> >
> >  Read the ocaml sources, file "parsing/parser.mly" to understand how
> >  the ocaml syntax tree works. Program your parser to generate this
> >  syntax tree.
>
> I have already read "parsing/parstree.mli", so I assume half the work
> is done! (well, I hope so, for I don't know .mly files...)
>
> > Once done, to give your syntax tree directly to the
> >  ocaml compiler, output:
> >
> >  1/ the magic number you can find in ocaml sources at utils/config.ml
> >       - ast_impl_magic_number for an implementation
> >         ast_intf_magic_number for an interface
> >  2/ your input file name
> >  3/ your syntax tree in binary with "output_value"
>
> That, is what I need. Thank you. I'll notify the list once I managed
> to produce some output.
>
> I have still one problem, though : how should I access this code? It
> is not part of the standard library, so I can't just "open Config;;".
> I can't hard-copy this code, either, that is too ugly. Maybe there is
> a way to install a "development package" of Ocaml so I can link it?
>
> Thanks,
> Loup
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 3347 bytes --]

Re: [Caml-list] How can I generate an AST?

[Daniel de Rauglaudre]

Hi,

On Mon, Aug 20, 2007 at 03:24:50PM +0200, Loup Vaillant wrote:

> I have still one problem, though : how should I access this code? It
> is not part of the standard library, so I can't just "open Config;;".
> I can't hard-copy this code, either, that is too ugly. Maybe there is
> a way to install a "development package" of Ocaml so I can link it?

For Camlp5, and Camlp4 (before it was integrated into OCaml), I had to
"hard copy" this code, like you say, since it is not exported indeed.

Notice that you have the same problem with the definition of the
abstract syntax tree, since is not exported either.

Perhaps convice the ocaml team to do it, in a specific sub-directory,
since I see that there are already: "ocamldoc", "camlp4", "labltk",
"threads", "vmthreads", "stublibs".

A new installed directory, named, e.g. "syntaxtree" would be great,
containing "parseetree.mli", "parsetree.cmi" and all its dependencies,
as well as this "pconfig.ml" and its object file or a subpart of it,
containing at least these "ast_impl_magic_number" and
"ast_intf_magic_number".

I can do it in the ocaml compiler if I have the permission.

Xavier ?

-- 
Daniel de Rauglaudre
http://pauillac.inria.fr/~ddr/

Re: [Caml-list] How can I generate an AST?

[Loup Vaillant]

(Woops, did not use the list to reply)
Hi,

2007/8/20, Daniel de Rauglaudre <daniel.de_rauglaudre@inria.fr>:
> Hi,
>
> On Mon, Aug 20, 2007 at 03:24:50PM +0200, Loup Vaillant wrote:
>
> > I have still one problem, though : how should I access this code? It
> > is not part of the standard library, so I can't just "open Config;;".
> > I can't hard-copy this code, either, that is too ugly. Maybe there is
> > a way to install a "development package" of Ocaml so I can link it?
>
> [...]
>
> A new installed directory, named, e.g. "syntaxtree" would be great,
> containing "parsetree.mli", "parsetree.cmi" and all its dependencies,
> as well as this "pconfig.ml" and its object file or a subpart of it,
> containing at least these "ast_impl_magic_number" and
> "ast_intf_magic_number".

That would be great. It would really help third parties to add any
fancy preprocessor they like, without worrying too much about the AST
implementation details. Could we expect this to appear in a future
release, or in CVS, even?

I wonder : if you expose the AST in an accessible library, will it be
considered part of the standard (and as such an additional burden)? Or
could it be subject to changes like camlp4 (risking a schism)?

> I can do it in the ocaml compiler if I have the permission.
>
> Xavier ?

please ;-(

Loup

Re: [Caml-list] How can I generate an AST?

[Alain Frisch]

Loup Vaillant wrote:
> Can anybody give me a pointer about how to do this?

The function Pparse.file is the one that makes the compiler accepts
either source code or marshaled ASTs. By looking at its implementation
and its call sites, you'll discover that the binary format is the
concatenation of a magic string Config.ast_impl_magic_number (resp.
Config.ast_intf_magic_number) and the marshaled version of a
Parsetree.structure (resp. a Parsetree.signature).

-- Alain

Re: [Caml-list] How can I generate an AST?

[Loup Vaillant]

2007/8/19, Alain Frisch <alain@frisch.fr>:
> Loup Vaillant wrote:
> > Can anybody give me a pointer about how to do this?
>
> The function Pparse.file is the one that makes the compiler accepts
> either source code or marshaled ASTs. By looking at its implementation
> and its call sites, you'll discover that the binary format is the
> concatenation of a magic string Config.ast_impl_magic_number (resp.
> Config.ast_intf_magic_number) and the marshaled version of a
> Parsetree.structure (resp. a Parsetree.signature).

Great. Thank you. I'll go and try some toy example.

Loup