AST transformation and scrapping boilerplate code

AST transformation and scrapping boilerplate code

[Joel Reymont]

Folks,

I have a large AST [2] that I would like to strip of token locations.  
Using the "Scrap your boilerplate" approach [1] the Haskell code  
looks like this:

strip :: (Data a) => a -> a
strip = everywhere (mkT f)
     where f (TokenPos a _) = a
           f x = x

Is there a way to accomplish a similar feat in OCaml without writing  
out heaps of code that recursively invokes strip for various  
constructors to get to expr and strip it of TokenPos?

	Thanks, Joel

[1] http://tinyurl.com/36jj3q

[2] Incomplete AST

let statement =
     [
     ...
     | `InputDecls of input_decl list
     | `VarDecls of var_decl list
     ...
     ]

and subscript = expr list

and input_decl =
     [
     | `InputDecl of id * ty * expr
     | `FunArgDecl of id * ty * subscript
     ]

and expr =
     [
     ...
     | `FunCall of ty * string * expr list * bars_ago * instrument
     | `Mul of expr * expr
     ...
     | `PrintExpr of expr * expr * expr
     ...
     | `TokenPos of expr * pos list
     ]

--
http://wagerlabs.com/

Re: [Caml-list] AST transformation and scrapping boilerplate code

[Nicolas Pouillard]

On 4/29/07, Joel Reymont <joelr1@gmail.com> wrote:
> Folks,
>
> I have a large AST [2] that I would like to strip of token locations.
> Using the "Scrap your boilerplate" approach [1] the Haskell code
> looks like this:
>
> strip :: (Data a) => a -> a
> strip = everywhere (mkT f)
>      where f (TokenPos a _) = a
>            f x = x
>
> Is there a way to accomplish a similar feat in OCaml without writing
> out heaps of code that recursively invokes strip for various
> constructors to get to expr and strip it of TokenPos?
>

The new Camlp4 can generate extensible map and fold traversals for a
given data structure. alphaCaml [1] also generetates some similar code
(and a lot more since its goal is to treat bindings).

However these to generators don't handle polymorphic variants.

Here is an example using camlp4, but without polymorphic variants:

-------------------8<----------------------------------------------------------------------------------
type statement =
  | InputDecls of input_decl list
  | VarDecls of var_decl list

and subscript = expr list

and input_decl =
    | InputDecl of id * ty * expr
    | FunArgDecl of id * ty * subscript

and expr =
    | Num of int
    | Mul of expr * expr
    | PrintExpr of expr * expr * expr
    | TokenPos of expr * pos list

and id = string
and ty = Int
and var_decl = Var_decl
and pos = int
;;

class map = Camlp4Filters.GenerateMap.generated;;
class fold = Camlp4Filters.GenerateFold.generated;;

let strip_postions = object inherit map as super
  method expr e =
    match super#expr e with
    | TokenPos(a, _) -> a
    | e -> e
end

let x =
  InputDecls
    [InputDecl("foo", Int,
      TokenPos(
        Mul(TokenPos(Num 3,[12]), Num 4),
        [15]))]
;;

let y = strip_postions#statement x
--------------------------------------8<---------------------------------------------------------------------

$ ocamlc -pp "camlp4o -filter map -filter fold" /tmp/test.ml

Hope this helps,

[1]: http://cristal.inria.fr/~fpottier/alphaCaml/

-- 
Nicolas Pouillard

Re: [Caml-list] AST transformation and scrapping boilerplate code

[Joel Reymont]

Thank you Nicolas! This helps a lot!

What needs to be done to make the generators handle polymorphic  
variants?

On Apr 30, 2007, at 9:21 AM, Nicolas Pouillard wrote:

> The new Camlp4 can generate extensible map and fold traversals for a
> given data structure. alphaCaml [1] also generetates some similar code
> (and a lot more since its goal is to treat bindings).
>
> However these to generators don't handle polymorphic variants.

--
http://wagerlabs.com/

Re: [Caml-list] AST transformation and scrapping boilerplate code

[Joel Reymont]

A combination of polymoprhic variants with generated map and fold  
traversals would be incredibly useful in AST transformations.

Imagine the case where some constructors are "shared" between  
different ASTs, e.g. type ty = [`A a|`B b] in one module and type ty  
= [`B b|`C c|` d] in another. You need to transform the first type  
into the second.

You could use the example below and add matches for special cases  
where either the target constructor is different or different actions  
need to be taken.

Is adding polymorphic variants to camlp4 something that can be  
accomplished by mere mortals? Any pointers on where to start and how  
to proceed?

	Thanks, Joel

On Apr 30, 2007, at 9:21 AM, Nicolas Pouillard wrote:

> However these to generators don't handle polymorphic variants.
> [...]
> class map = Camlp4Filters.GenerateMap.generated;;
> class fold = Camlp4Filters.GenerateFold.generated;;
>
> let strip_postions = object inherit map as super
>  method expr e =
>    match super#expr e with
>    | TokenPos(a, _) -> a
>    | e -> e
> end

--
http://wagerlabs.com/

Re: [Caml-list] AST transformation and scrapping boilerplate code

[Nicolas Pouillard]

On 4/30/07, Joel Reymont <joelr1@gmail.com> wrote:
> Thank you Nicolas! This helps a lot!
>
> What needs to be done to make the generators handle polymorphic
> variants?

Adding just 28 lines in fact, so I've done it.

I technically can't commit this patch right now, but here it is:
http://pastie.caboo.se/57636

The example using polymorphic variants: http://pastie.caboo.se/57635

Regards,

-- 
Nicolas Pouillard

Re: [Caml-list] AST transformation and scrapping boilerplate code

[Joel Reymont]

I get this error when try to compile my version of the token remover.  
The only difference is that my version has "open AST" at the top  
whereas in the example everything is in one file.

What am I missing?

	Thanks, Joel

---

./build.sh test.byte
+ ocamlfind ocamlc -package ounit -c -I +camlp4 -pp 'camlp4o -filter  
map -filter fold' -o easy_strip.cmo easy_strip.ml
File "easy_strip.ml", line 8, characters 10-15:
This expression has no method expr
Command exited with code 2.


--
http://wagerlabs.com/

Re: [Caml-list] AST transformation and scrapping boilerplate code

[Nicolas Pouillard]

On 4/30/07, Joel Reymont <joelr1@gmail.com> wrote:
> I get this error when try to compile my version of the token remover.
> The only difference is that my version has "open AST" at the top
> whereas in the example everything is in one file.
>
> What am I missing?

The generator needs to see youre types definitions.

-- 
Nicolas Pouillard

Re: [Caml-list] AST transformation and scrapping boilerplate code

[Joel Reymont]

On Apr 30, 2007, at 1:42 PM, Nicolas Pouillard wrote:

> The generator needs to see youre types definitions.

What does it mean? Open <AST module> doesn't do it?

How do I make the type defs in a separate module available to the  
generator?

	Thanks, Joel

--
http://wagerlabs.com/

Re: [Caml-list] AST transformation and scrapping boilerplate code

[Nicolas Pouillard]

On 4/30/07, Joel Reymont <joelr1@gmail.com> wrote:
>
> On Apr 30, 2007, at 1:42 PM, Nicolas Pouillard wrote:
>
> > The generator needs to see youre types definitions.
>
> What does it mean? Open <AST module> doesn't do it?
>
> How do I make the type defs in a separate module available to the
> generator?
>

Here is the hack I use:

module Camlp4Trash = struct
  INCLUDE "AST.ml";;
end;;

Use camlp4of (since INCLUDE is provided by camlp4 macros).
Also add -filter trash to the camlp4 options (after map and fold), in
order to remove the Camlp4Trash module.

-- 
Nicolas Pouillard

Re: [Caml-list] AST transformation and scrapping boilerplate code

[Joel Reymont]

On Apr 30, 2007, at 1:58 PM, Nicolas Pouillard wrote:

> Use camlp4of (since INCLUDE is provided by camlp4 macros).
> Also add -filter trash to the camlp4 options (after map and fold), in
> order to remove the Camlp4Trash module.

Including the AST module worked and I'm almost there.

+ ocamlfind ocamlc -package ounit -c -I +camlp4 -pp 'camlp4of -filter  
map -filter fold -filter trash ' -o easy_strip.cmo easy_strip.ml
File "ghost-location", line 1, characters 0-0:
Unbound type constructor var_decl

This is surprising since the AST looks like this, i.e. var_decl is  
part of it.

What is the explanation for this and how should I fix it?

...
and statement =
     [
     ...
     | `VarDecls of var_decl list
     ...
     ]

...

and var_decl =
     [
     | `VarDecl of id * ty * expr
     ]

	Thanks, Joel

--
http://wagerlabs.com/

Re: [Caml-list] AST transformation and scrapping boilerplate code

[Nicolas Pouillard]

On 4/30/07, Joel Reymont <joelr1@gmail.com> wrote:
>
> On Apr 30, 2007, at 1:58 PM, Nicolas Pouillard wrote:
>
> > Use camlp4of (since INCLUDE is provided by camlp4 macros).
> > Also add -filter trash to the camlp4 options (after map and fold), in
> > order to remove the Camlp4Trash module.
>
> Including the AST module worked and I'm almost there.
>
> + ocamlfind ocamlc -package ounit -c -I +camlp4 -pp 'camlp4of -filter
> map -filter fold -filter trash ' -o easy_strip.cmo easy_strip.ml
> File "ghost-location", line 1, characters 0-0:
> Unbound type constructor var_decl

Some open AST is missing I think.

-- 
Nicolas Pouillard