Parsing simple type expressions

Parsing simple type expressions

[Paolo Donadeo]

For a serializer I'm writing I need to parse simple OCaml type
expressions composed by OCaml basic types, tuples, options and lists.
Given a string like "(int * string option) list" and this type:

type types =
  | Int
  | String
  | Float
  | Char
  | Bool
  | Option of types
  | List of types
  | Tuple of types list

the function I need should return something like List (Tuple ([Int;
Option(String)]))

Before starting with low level sscanf functions I looked at the Genlex
module, but it wasn't so inspiring. Then I tried with Camlp4 but the
documentation doesn't really shine :-)

So is there a simple way to write this function using some standard module?

TIA,


-- 
Paolo
~
~
:wq

RE: [Caml-list] Parsing simple type expressions

[David Allsopp]

ocamlyacc - you can get most of it for free out of parsing/parser.mly in the OCaml sources... the section on type expressions starts at line 1144 for OCaml 3.11.0.


David

> -----Original Message-----
> From: caml-list-bounces@yquem.inria.fr [mailto:caml-list-
> bounces@yquem.inria.fr] On Behalf Of Paolo Donadeo
> Sent: 06 January 2009 14:04
> To: OCaml mailing list
> Subject: [Caml-list] Parsing simple type expressions
> 
> For a serializer I'm writing I need to parse simple OCaml type
> expressions composed by OCaml basic types, tuples, options and lists.
> Given a string like "(int * string option) list" and this type:
> 
> type types =
>   | Int
>   | String
>   | Float
>   | Char
>   | Bool
>   | Option of types
>   | List of types
>   | Tuple of types list
> 
> the function I need should return something like List (Tuple ([Int;
> Option(String)]))
> 
> Before starting with low level sscanf functions I looked at the Genlex
> module, but it wasn't so inspiring. Then I tried with Camlp4 but the
> documentation doesn't really shine :-)
> 
> So is there a simple way to write this function using some standard
> module?
> 
> TIA,
> 
> 
> --
> Paolo
> ~
> ~
> :wq
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Parsing simple type expressions

[Martin Jambon]

David Allsopp wrote:
> ocamlyacc - you can get most of it for free out of parsing/parser.mly in the OCaml sources... the section on type expressions starts at line 1144 for OCaml 3.11.0.

Our json-wheel library is a complete example:

http://martin.jambon.free.fr/json-wheel.html


Martin

>> -----Original Message-----
>> From: caml-list-bounces@yquem.inria.fr [mailto:caml-list-
>> bounces@yquem.inria.fr] On Behalf Of Paolo Donadeo
>> Sent: 06 January 2009 14:04
>> To: OCaml mailing list
>> Subject: [Caml-list] Parsing simple type expressions
>>
>> For a serializer I'm writing I need to parse simple OCaml type
>> expressions composed by OCaml basic types, tuples, options and lists.
>> Given a string like "(int * string option) list" and this type:
>>
>> type types =
>>   | Int
>>   | String
>>   | Float
>>   | Char
>>   | Bool
>>   | Option of types
>>   | List of types
>>   | Tuple of types list
>>
>> the function I need should return something like List (Tuple ([Int;
>> Option(String)]))
>>
>> Before starting with low level sscanf functions I looked at the Genlex
>> module, but it wasn't so inspiring. Then I tried with Camlp4 but the
>> documentation doesn't really shine :-)
>>
>> So is there a simple way to write this function using some standard
>> module?
>>
>> TIA,
>>
>>
>> --
>> Paolo
>> ~
>> ~
>> :wq

-- 
http://mjambon.com/

Re : [Caml-list] Parsing simple type expressions

[Matthieu Wipliez]

[-- Attachment #1: Type: text/plain, Size: 3650 bytes --]

Dear caml-list,

here you can find another way of doing what Paolo wants to do.

Basically the point was to reuse as much (well-tested) code as I could, so I tried to use Camlp4 to parse a type definition, and work from there.

You can find in Common the definition of the AST that Paolo provided us with, along with a string_of function and a parse_type function overidden at runtime. In Pa_exp_ext a syntax extension extends the OCaml standard syntax with a rule that parses a type and End Of Input, and in Pa_exp is the driver that loads:
  - Camlp4OCamlRevisedParser the revised syntax parser,
  - Camlp4OCamlParser the "normal" syntax parser,
  - and "pa_exp_ext" our parser.

Actually, Pa_exp acts as the standard "camlp4" executable, loading parsers and stuff.

Please note that this example is incomplete because Pa_exp_ext.Make.convert_type does not take into account the syntax of applied types, which are tranformed using the function "ident_of_ctyp" from Camlp4Ast.mlast which I did not have time to code.

However it is my belief that these files provide a (very interesting) complete example of how camlp4 can be used to parse parts of OCaml syntax without having to write the syntax rules over and over, while using the mysterious quotations :-)

Having spent a couple of hours on this, I hope this helps anyone that is having a look at camlp4 to do what they would like to do,

Cheers and la multi ani "happy new year",
Matei



----- Message d'origine ----
> De : Martin Jambon <martin.jambon@ens-lyon.org>
> À : David Allsopp <dra-news@metastack.com>
> Cc : OCaml mailing list <caml-list@yquem.inria.fr>; Paolo Donadeo <p.donadeo@gmail.com>
> Envoyé le : Mardi, 6 Janvier 2009, 23h10mn 05s
> Objet : Re: [Caml-list] Parsing simple type expressions
> 
> David Allsopp wrote:
> > ocamlyacc - you can get most of it for free out of parsing/parser.mly in the 
> OCaml sources... the section on type expressions starts at line 1144 for OCaml 
> 3.11.0.
> 
> Our json-wheel library is a complete example:
> 
> http://martin.jambon.free.fr/json-wheel.html
> 
> 
> Martin
> 
> >> -----Original Message-----
> >> From: caml-list-bounces@yquem.inria.fr [mailto:caml-list-
> >> bounces@yquem.inria.fr] On Behalf Of Paolo Donadeo
> >> Sent: 06 January 2009 14:04
> >> To: OCaml mailing list
> >> Subject: [Caml-list] Parsing simple type expressions
> >>
> >> For a serializer I'm writing I need to parse simple OCaml type
> >> expressions composed by OCaml basic types, tuples, options and lists.
> >> Given a string like "(int * string option) list" and this type:
> >>
> >> type types =
> >>   | Int
> >>   | String
> >>   | Float
> >>   | Char
> >>   | Bool
> >>   | Option of types
> >>   | List of types
> >>   | Tuple of types list
> >>
> >> the function I need should return something like List (Tuple ([Int;
> >> Option(String)]))
> >>
> >> Before starting with low level sscanf functions I looked at the Genlex
> >> module, but it wasn't so inspiring. Then I tried with Camlp4 but the
> >> documentation doesn't really shine :-)
> >>
> >> So is there a simple way to write this function using some standard
> >> module?
> >>
> >> TIA,
> >>
> >>
> >> --
> >> Paolo
> >> ~
> >> ~
> >> :wq
> 
> -- 
> http://mjambon.com/
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs



      

[-- Attachment #2: common.ml --]
[-- Type: application/octet-stream, Size: 525 bytes --]

type types =
  | Int
  | String
  | Float
  | Char
  | Bool
  | Option of types
  | List of types
  | Tuple of types list

let rec string_of_type = function
  | Int -> "int"
  | String -> "string"
  | Float -> "float"
  | Char -> "char"
  | Bool -> "bool"
  | Option types -> string_of_type types ^ " option"
  | List types -> string_of_type types ^ " list"
  | Tuple types ->
    let tup = String.concat " * " (List.map string_of_type types) in
    "(" ^ tup ^ ")"

let parse_type : (string -> unit) ref = ref (fun _ -> ())

[-- Attachment #3: pa_exp.ml --]
[-- Type: application/octet-stream, Size: 665 bytes --]

open Camlp4
open PreCast
open Printf
  
let _ =
  let dl = DynLoader.mk () in
  DynLoader.load dl "Camlp4OCamlRevisedParser.cmo";
  DynLoader.load dl "Camlp4OCamlParser.cmo";
  DynLoader.load dl "pa_exp_ext.cmo";
  
  List.iter print_endline !Register.loaded_modules;
  Register.iter_and_take_callbacks
    (fun (name, module_callback) ->
      module_callback ());
  (*parse_type "int * string option list";
  parse_type "(int * string option) list";*)
  try
    !Common.parse_type "int option"
  with Loc.Exc_located (loc, e) ->
      print_endline (Loc.to_string loc);
	  print_endline (Printexc.to_string e)
  | e -> 
	  print_endline (Printexc.to_string e)
  

[-- Attachment #4: pa_exp_ext.ml --]
[-- Type: application/octet-stream, Size: 1227 bytes --]

open Camlp4
open PreCast
open Printf
 
module Id = struct
  let name = "pa_exp"
  let version = "1.0"
end

module Make (Syntax : Sig.Camlp4Syntax) = struct
  open Sig
  include Syntax

  let rtyuio = Gram.Entry.mk "rtyuio"

  let rec convert_type = function
  | <:ctyp< $lid:i$ >> when i = "int" -> Common.Int
  | <:ctyp< $lid:i$ >> when i = "string" -> Common.String
  | <:ctyp< $lid:i$ >> when i = "float" -> Common.Float
  | <:ctyp< $lid:i$ >> when i = "char" -> Common.Char
  | <:ctyp< $lid:i$ >> when i = "bool" -> Common.Bool
  | <:ctyp< $t1$ $t2$ >> ->
    let t = convert_type t2 in
    (match t1 with
	| <:ctyp< $lid:i$ >> when i = "option" -> Common.Option t
	| <:ctyp< $lid:i$ >> when i = "list" -> Common.List t
	| _ -> failwith "only option and list allowed for type application")
  | _ -> failwith "this case is not representable using this AST"

  (* Grammar definition *)
  EXTEND Gram
  
    rtyuio: [
  		[ t = ctyp; EOI -> convert_type t ]
  	];

  END
  
  let my_parse_type str =
    print_endline (Common.string_of_type (Gram.parse rtyuio (Loc.mk str) (Stream.of_string str)))

  let _ =
    Common.parse_type := my_parse_type
end
  
let _=
  let module M = Register.OCamlSyntaxExtension (Id) (Make) in ()

[-- Attachment #5: build.sh --]
[-- Type: application/octet-stream, Size: 205 bytes --]

#!/bin/sh
ocamlc -g -c common.ml
ocamlc -g -c -I +camlp4 pa_exp.ml
ocamlc -g -c -I +camlp4 -pp camlp4of pa_exp_ext.ml
ocamlc -g -o pa_exp.exe -I +camlp4 dynlink.cma camlp4fulllib.cma common.cmo pa_exp.cmo 

Re: [Caml-list] Parsing simple type expressions

[Paolo Donadeo]

Thanks, Matthieu and Jake, this is exactly what I had in mind.

At the end, Camlp4 was the right solution to my simple problem, as I
suspected from the beginning. Camlp4 is an extremely powerful tool and
it's a pity it couldn't be used by everyone for lack of documentation.
What is really needed, in my opinion, is a description of *what* one
can do with Camlp4 without being an expert. For example a list of all
syntax extension available out of the box might spread the use of
Camlp4 among common programmers.

Searching in the source code I can see things named:
Camlp4ExceptionTracer, Camlp4FoldGenerator, Camlp4MetaGenerator, and
so on. The names sounds great but... what is it? The source code is
cryptic to every "regular" OCaml programmer like me. The same
considerations apply to all the (many!) libraries available and
related to Camlp4.

What is needed to spread the use of this tool are some... recipes,
like those available for ocamlbuild in its wiki pages.


-- 
Paolo
~
~
:wq

Re: [Caml-list] Parsing simple type expressions

[Jake Donham]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

On Tue, Jan 6, 2009 at 6:03 AM, Paolo Donadeo <p.donadeo@gmail.com> wrote:

> For a serializer I'm writing I need to parse simple OCaml type
> expressions composed by OCaml basic types, tuples, options and lists.


This is pretty easy with Camlp4, although as you say there isn't much in the
docs to point the way. You might take a look at how orpc does it--see the
parse_type function in

  http://code.google.com/p/orpc2/source/browse/trunk/src/generator/parse.ml

(I'm going to cover using Camlp4 in this way on the blog I posted yesterday,
but not for another week or two.)

You might also look at some other Camlp4-based serialization tools, like
bin-prot/type-conv at

  http://www.ocaml.info/home/ocaml_sources.html

Jake

[-- Attachment #2: Type: text/html, Size: 1198 bytes --]

Re: [Caml-list] Parsing simple type expressions

[Paolo Donadeo]

> This is pretty easy with Camlp4, although as you say there isn't much in the
> docs to point the way. You might take a look at how orpc does it--see the
> parse_type function

Thanks for this pointer!


-- 
Paolo
~
~
:wq