Generation of Java code from OCaml

Generation of Java code from OCaml

[Mykola Stryebkov]

Hi,

I need to generate arbitrary text (more specifically, Java code) from  
Ocaml structures, pretty similar to how json-static lib generates  
ocaml code from ocaml structures.

Which approach would you recommend?
Is it possible to use ocamlp4 for it?

-- 
Mykola Stryebkov
Blog: http://mykola.org/blog/
Public key: http://mykola.org/pubkey.txt
fpr: 0226 54EE C1FF 8636 36EF  2AC9 BCE9 CFC7 9CF4 6747

Re: [Caml-list] Generation of Java code from OCaml

[Richard Jones]

On Wed, Sep 23, 2009 at 09:15:08PM +0300, Mykola Stryebkov wrote:
> I need to generate arbitrary text (more specifically, Java code) from  
> Ocaml structures, pretty similar to how json-static lib generates  
> ocaml code from ocaml structures.
> 
> Which approach would you recommend?
> Is it possible to use ocamlp4 for it?

Do you want to generate valid Java code (or an AST) or just string
bits of text together?

FWIW we generate 100's of thousands of lines of C, Java and other
languages using OCaml in libguestfs (by stringing bits of text
together):

http://git.et.redhat.com/?p=libguestfs.git;a=blob;f=src/generator.ml;hb=HEAD

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Generation of Java code from OCaml

[Mykola Stryebkov]

Hi, Richard.

On 23 Вер 2009, at 22:57, Richard Jones wrote:

> On Wed, Sep 23, 2009 at 09:15:08PM +0300, Mykola Stryebkov wrote:
>> I need to generate arbitrary text (more specifically, Java code) from
>> Ocaml structures, pretty similar to how json-static lib generates
>> ocaml code from ocaml structures.
>>
>> Which approach would you recommend?
>> Is it possible to use ocamlp4 for it?
>
> Do you want to generate valid Java code (or an AST) or just string
> bits of text together?

I'm going to use stringing bits of text together. But text generating  
is not
an issue here. The issue is how to make this stringing driven by
description of ocaml records.

I have few hundreds of types. For each ocaml type I want to generate
corresponding Java class with function (constructor) that initialize an
object of this class from json string. E.g.:

type json create_order_response = {
   order_id : int;
   order_code : string;
   order_price : int;
   ...
}

... must be translated in something like:

public class CreateOrderResponse extends ... implements ... {
	private int id = 0;
	private String code = "";
	private int price = 0;
	...
	public CreateOrderResponse(String json) { ... }
};

P.S.: I'm using json-static library and its preprocessor module
(http://martin.jambon.free.fr/json-static.html) to automatically  
generate
functions, used to convert records (marked by "json" keyword) to strings
in json format and vice versa.

-- 
Mykola Stryebkov
Blog: http://mykola.org/blog/
Public key: http://mykola.org/pubkey.txt
fpr: 0226 54EE C1FF 8636 36EF  2AC9 BCE9 CFC7 9CF4 6747

RE: [Caml-list] Generation of Java code from OCaml

[David Allsopp]

Mykola Stryebkov wrote: 
> Hi, Richard.
> 
> On 23 Вер 2009, at 22:57, Richard Jones wrote:

<snip>

> I'm going to use stringing bits of text together. But text generating
> is not an issue here. The issue is how to make this stringing driven by
> description of ocaml records.

camlp4 would be the neatest, most stable way of doing this - as it sounds so similar in operation (if not function) to json-static, why not use its code as a reference/starting point? The alternative if you're happier with text processing and want to avoid the learning-curve for camlp4 would be to parse the output of `ocamlc -i` which is, in syntactic terms, canonical (where a source file is not)


David

Re: [Caml-list] Generation of Java code from OCaml

[blue storm]

In case it helps, below is a basic patch against json-static (SVN
trunk, 3.10 version). On your example it produces the following code :

let javadef_of_create_order_response =
  "public class Create_order_response {\n\tprivate int
order_id;\n\tprivate String order_code;\n\tprivate int
order_price;\n\tpublic Create_order_response { ... }\n}"

Wich, printed to sdout, gives the following output :

public class Create_order_response {
	private int order_id;
	private String order_code;
	private int order_price;
	public Create_order_response { ... }
}

I choosed to generate the Java code inside an ocaml string declared
inside the program to keep coherency with the other json code
generators, but if you want you could easily output the java code to a
separate file, and that would remove some of the string escaping in
play.

--- json-static/pa_json_static.ml.310	2009-09-24 11:40:28.000000000 +0200
+++ json-static-new/pa_json_static.ml.310	2009-09-24 11:39:52.000000000 +0200
@@ -1,3 +1,4 @@
+# 1 "pa_json_static.ml.310"
 (*
   Conversion between OCaml types and JSON types as provided by the json-wheel
   library.
@@ -532,6 +533,39 @@
   <:str_item< value rec $defs$ >>


+let make_javadef _loc l =
+  let rec print name (_loc, l) = match l with
+    | Record r ->
+        let class_name = String.capitalize name in (* TODO : real
java naming conventions *)
+        Printf.sprintf
+          "public class %s {\\n\
+             \\t%s;\\n\
+             \\tpublic %s { ... }\\n\
+          }" (* TODO : real constructor body *)
+          class_name
+          (String.concat ";\\n\\t" (List.map print_field r))
+          class_name
+    | _ -> Loc.raise _loc
+        (Failure "javadef patch draft : only records are supported")
+  and print_field f =
+    Printf.sprintf "private %s %s"
+      (print_type f.field_type)
+      f.field_caml_name
+  and print_type (_loc, t) = match t with
+    | String -> "String"
+    | Bool -> "bool"
+    | Int -> "int"
+    | Float -> "double"
+    | _ -> Loc.raise _loc
+        (Failure "javadef patch draft : only base types are supported") in
+  let defs =
+    List.fold_right
+      (fun ((_loc, name), x) acc ->
+	 let fname = "javadef_of_" ^ name in
+	 <:binding< $lid:fname$ = $str:print name x.def$ and $acc$ >>)
+      l <:binding<>> in
+  <:str_item< value $defs$ >>
+
 let expand_typedefs _loc l =
   check_unique (fun (name, _) -> name) l;
   let names =
@@ -541,7 +575,8 @@
   let typedef = make_typedef _loc names l in
   let ofjson = make_ofjson _loc l in
   let tojson = make_tojson _loc l in
-  <:str_item< $typedef$; $ofjson$; $tojson$ >>
+  let javadef = make_javadef _loc l in
+  <:str_item< $typedef$; $ofjson$; $tojson$; $javadef$>>

 let o2b = function None -> false | _ -> true

Re: [Caml-list] Generation of Java code from OCaml

[Martin Jambon]

blue storm wrote:
> In case it helps, below is a basic patch against json-static (SVN
> trunk, 3.10 version). On your example it produces the following code :
[...]

Yes, patching the original code of json-static would work but it's not ideal.

Maybe Deriving is a better choice (http://code.google.com/p/deriving/).  I
just say "maybe" because I've never used it.

Or you can write your own camlp4 extension from scratch and find out how to
use one of the predefined filters that would let you output only
record-derived data.


Note that if you're already using json-static but need to duplicate type
definitions, you can do as follows:

type foo = { bar : int }
type json foo = predefined { bar : int }



Martin

-- 
http://mjambon.com/

Re: [Caml-list] Generation of Java code from OCaml

[blue storm]

On Thu, Sep 24, 2009 at 1:18 PM, Martin Jambon
<martin.jambon@ens-lyon.org> wrote:
> Yes, patching the original code of json-static would work but it's not ideal.

>From the example given, it seems that the original poster already uses
json-static. In that case, I think that reusing the code logic is a
good idea (coherent behaviour, etc.). I suppose that you dislike the
"patching" solution as it is not easily reusable (one cannot use the
upstream json-static anymore).

What about factorising json-static to allow adding arbitrary code
generators (represented as functions from (string * type_def) list to
a camlp4 Ast) at camlp4-time ? You would have a design similar to
type-conv, wich allows adding new generators without modifying
type-conv itself.

Re: [Caml-list] Generation of Java code from OCaml

[Martin Jambon]

blue storm wrote:
> On Thu, Sep 24, 2009 at 1:18 PM, Martin Jambon
> <martin.jambon@ens-lyon.org> wrote:
>> Yes, patching the original code of json-static would work but it's not ideal.
> 
>>From the example given, it seems that the original poster already uses
> json-static. In that case, I think that reusing the code logic is a
> good idea (coherent behaviour, etc.). I suppose that you dislike the
> "patching" solution as it is not easily reusable (one cannot use the
> upstream json-static anymore).

Of course, the internals of json-static are much more likely to change than
its user interface.  This may however not be such a big problem here in practice.

The problem is more that it's not compatible with other possible extensions of
json-static provided by 3rd parties.

> What about factorising json-static to allow adding arbitrary code
> generators (represented as functions from (string * type_def) list to
> a camlp4 Ast) at camlp4-time ? You would have a design similar to
> type-conv, wich allows adding new generators without modifying
> type-conv itself.

Oh yes, there's type-conv too.  I don't know the pros and cons of using either
type-conv or deriving.  If anyone knows, a brief comparison would be helpful.


Martin

-- 
http://mjambon.com/

Re: [Caml-list] Generation of Java code from OCaml

[ygrek]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

On Thu, 24 Sep 2009 14:02:14 +0200
blue storm <bluestorm.dylc@gmail.com> wrote:

> What about factorising json-static to allow adding arbitrary code
> generators (represented as functions from (string * type_def) list to
> a camlp4 Ast) at camlp4-time ? You would have a design similar to
> type-conv, wich allows adding new generators without modifying
> type-conv itself.

That would be really useful for other kinds of serialization. I would be glad to 
add json-static-like interface for bencode and php_serialize.

-- 
 ygrek
 http://ygrek.org.ua

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [Caml-list] Generation of Java code from OCaml

[Anil Madhavapeddy]

[-- Attachment #1: Type: text/plain, Size: 1459 bytes --]

On 24 Sep 2009, at 13:19, Martin Jambon wrote:
>
> Oh yes, there's type-conv too.  I don't know the pros and cons of  
> using either
> type-conv or deriving.  If anyone knows, a brief comparison would be  
> helpful.
>

The nice thing about using type-conv is that syntax extensions can be  
composed quite easily, as long as the generators themselves don't  
depend too much on custom type syntax (e.g. when using sexplib, don't  
use the special sexp_option/sexp_opaque types).

I've ported your json-static extension over to type-conv so that we  
can use a combination of json/sexp and the (still in development) ORM  
extension side-by-side with a single type declaration, e.g.

type t = {
   foo: string;
   bar: int
} with orm (debug: all), json

... which generates both persistence functions and the json functions  
from the single type.  Right now it's fairly awkward to do this with  
json-static and the 'predefined' keyword as it requires duplicating  
types.

The ported version of json-static to type-conv is available at:
http://github.com/avsm/caml-type-conv-json/

The only thing I haven't quite worked out yet is the quotation to  
pattern-match type applications to detect things like "(string, unit)  
Hashtbl.t" the way the current json-static does via the grammar  
extension.  Most other types like char, int32/64, unit, poly variants,  
etc should all work.  There's a simple testcase in check_tc.ml in the  
same repository.

-anil

[-- Attachment #2: Type: text/html, Size: 1989 bytes --]

Re: [Caml-list] Generation of Java code from OCaml

[blue storm]

On Sat, Oct 3, 2009 at 2:16 PM, Anil Madhavapeddy <anil@recoil.org> wrote:
> The only thing I haven't quite worked out yet is the quotation to
> pattern-match type applications to detect things like "(string, unit)
> Hashtbl.t" the way the current json-static does via the grammar extension.
> -anil

Below are two patches (from `git log -u`) adding the relevant features.

##########################
diff --git a/json-static/pa_json_tc.ml b/json-static/pa_json_tc.ml
index f1d21e7..09b7937 100644
--- a/json-static/pa_json_tc.ml
+++ b/json-static/pa_json_tc.ml
@@ -494,11 +494,15 @@ and process_td _loc = function
  | <:ctyp< int64 >> -> Int64
  | <:ctyp< unit >> -> Unit
  | <:ctyp< char >> -> Char
+ | <:ctyp< number >> -> Number

  | <:ctyp< option $t$ >> -> Option (_loc, process_td _loc t)
  | <:ctyp< list $t$ >> -> List (_loc, process_td _loc t)
  | <:ctyp< array $t$ >> -> Array (_loc, process_td _loc t)
-
+ | <:ctyp< assoc $t$ >> as assoc ->
+   (match t with
+      | <:ctyp< (string * $t$) >> -> Assoc (_loc, process_td _loc t)
+      | _ -> failwith "must be of the form (string * ...) assoc")
  | <:ctyp< < $cs$ > >> -> Object (process_fields _loc cs)
  | <:ctyp< { $cs$ } >> -> Record (process_fields _loc cs)

@@ -512,8 +516,13 @@ and process_td _loc = function
        (Ast.list_of_ctyp tp []) in
    Tuple tps

- | <:ctyp< $uid:id$.t >> -> Custom id (* XXX broken, how to check for TyApp? *)
+ | <:ctyp< Hashtbl.t string $x$ >> -> Hashtbl (_loc, process_td _loc x)
+ | <:ctyp< json_type >>
+ | <:ctyp< Json_type.json_type >>
+ | <:ctyp< Json_type.t >>
+   -> Raw
  | <:ctyp< $lid:id$ >> -> Name id
+ | <:ctyp< $uid:id$.t >> -> Custom id
  | _ -> failwith "unknown type"

 open Pa_type_conv


##########################
diff --git a/json-static/check.ml b/json-static/check.ml
index 19bac81..ff0186b 100644
--- a/json-static/check.ml
+++ b/json-static/check.ml
@@ -33,3 +33,4 @@ and b = int

 type json c = (string * d * d) list
 and d = [ `A ]
+
diff --git a/json-static/check_tc.ml b/json-static/check_tc.ml
index b362ad2..3105800 100644
--- a/json-static/check_tc.ml
+++ b/json-static/check_tc.ml
@@ -31,3 +31,6 @@ let _ =
   assert (json_o#foo = o#foo);
   assert (json_o#bar = o#bar);
   prerr_endline json_string
+
+type c = (string, unit) Hashtbl.t with json
+type d = (string * float) assoc with json
diff --git a/json-static/pa_json_tc.ml b/json-static/pa_json_tc.ml
index 09b7937..5c76819 100644
--- a/json-static/pa_json_tc.ml
+++ b/json-static/pa_json_tc.ml
@@ -448,6 +448,9 @@ let expand_typedefs _loc l =
   let tojson = make_tojson _loc l in
   <:str_item< $ofjson$; $tojson$ >>

+let type_fail ctyp msg =
+  Loc.raise (Ast.loc_of_ctyp ctyp) (Failure msg)
+
 let rec process_tds tds =
   let rec fn ty =
     match ty with
@@ -455,7 +458,7 @@ let rec process_tds tds =
        fn tyl @ (fn tyr)
     |Ast.TyDcl (_loc, id, _, ty, []) ->
        [ (_loc, id ) , (_loc, process_td _loc ty) ]
-    |_ -> failwith "process_tds: unexpected type"
+    | other -> type_fail other "process_tds: unexpected AST"
    in fn tds

 and process_fields _loc cs =
@@ -463,7 +466,7 @@ and process_fields _loc cs =
     | <:ctyp< $t1$; $t2$ >> -> fn t1 @ (fn t2)
     | <:ctyp< $lid:id$ : mutable $t$ >> -> fnt ~mut:true ~id ~t
     | <:ctyp< $lid:id$ : $t$ >> ->  fnt ~mut:false ~id ~t
-    | _ -> failwith "unexpected ast"
+    | other -> type_fail other "process_fields: unexpected AST"
   and fnt ~mut ~id ~t =
     [ { field_caml_name = id; field_json_name = id;
         field_type = (_loc, process_td _loc t);
@@ -482,7 +485,7 @@ and process_constructor _loc rf =
     | <:ctyp< $uid:id$ >> ->
        { cons_caml_name=id; cons_json_name=id; cons_caml_loc=_loc;
          cons_json_loc=_loc; cons_args=[] }
-    | _ -> failwith "process_constructor: unexpected AST"
+    | other -> type_fail other "process_constructor: unexpected AST"
   ) (Ast.list_of_ctyp rf [])

 and process_td _loc = function
@@ -502,7 +505,7 @@ and process_td _loc = function
  | <:ctyp< assoc $t$ >> as assoc ->
    (match t with
       | <:ctyp< (string * $t$) >> -> Assoc (_loc, process_td _loc t)
-      | _ -> failwith "must be of the form (string * ...) assoc")
+      | other -> type_fail assoc "must be of the form (string * ...) assoc")
  | <:ctyp< < $cs$ > >> -> Object (process_fields _loc cs)
  | <:ctyp< { $cs$ } >> -> Record (process_fields _loc cs)

@@ -523,7 +526,7 @@ and process_td _loc = function
    -> Raw
  | <:ctyp< $lid:id$ >> -> Name id
  | <:ctyp< $uid:id$.t >> -> Custom id
- | _ -> failwith "unknown type"
+ | other -> type_fail other "unknown type"

 open Pa_type_conv
 let _ =

Re: [Caml-list] Generation of Java code from OCaml

[Anil Madhavapeddy]

[-- Attachment #1: Type: text/plain, Size: 1174 bytes --]

On 3 Oct 2009, at 18:27, blue storm wrote:

> On Sat, Oct 3, 2009 at 2:16 PM, Anil Madhavapeddy <anil@recoil.org>  
> wrote:
>> The only thing I haven't quite worked out yet is the quotation to
>> pattern-match type applications to detect things like "(string, unit)
>> Hashtbl.t" the way the current json-static does via the grammar  
>> extension.
>> -anil
>
> Below are two patches (from `git log -u`) adding the relevant  
> features.
>
(snip)
> -
> + | <:ctyp< assoc $t$ >> as assoc ->
> +   (match t with
> +      | <:ctyp< (string * $t$) >> -> Assoc (_loc, process_td _loc t)
> +      | _ -> failwith "must be of the form (string * ...) assoc")
>  | <:ctyp< < $cs$ > >> -> Object (process_fields _loc cs)
>  | <:ctyp< { $cs$ } >> -> Record (process_fields _loc cs)

Thanks; my goal is to avoid putting in special types into type-conv  
generators though ('assoc' here), so they can be used alongside other  
generators too.

It seems reasonable to just special-case "Hashtbl.t string $t$" and  
"list ( string * $t$ )", since JSON objects must have a string as the  
key. I've pushed a patch which does that to my tree, along with some  
of your other changes.

-anil

[-- Attachment #2: Type: text/html, Size: 1959 bytes --]