Re: [Caml-list] Problems with printing MetaOCaml generated code

["Ömer Sinan Ağacan" <omeragacan@gmail.com>]

Thanks for all the answers,

> MetaOCaml always prints the generated code. However, if the generated code
> contains non-serializable CSP, they are printed in a way that makes saving
> the code to a file and then compiling separately impossible.

Can we talk about what is non-serializable here? I pushed some code to a
public repo for demonstration purposes. I have a simple interpreter for
a simple language. I'm experimenting with different levels of
interpreter specializations(aka. first Futamura projections). As a
simplest specialization, I want to eliminate parsing costs. The code
that's supposed to do this is here:

https://github.com/osa1/int-proj/blob/master/Unlambda.ml#L204

`p'` in that code is lifted, and it's parsed syntax tree of given source
file. Syntax tree is defined here:

https://github.com/osa1/int-proj/blob/master/Syntax.ml#L63

I think it doesn't have anything that's not serializable: It's just a
simple monomorphic type, doesn't have any closures etc.

But MetaOCaml is failing to print this code. To try it yourself, make
sure you have metaocamlc in $PATH, and run:

  ➜  unlambda git:(master) ✗ make unlambda_metaocaml
  metaocamlc -c Syntax.ml
  metaocamlc Syntax.cmo Unlambda.ml -o unlambda_metaocaml

  ➜  unlambda git:(master) ✗ ./unlambda_metaocaml programs/Hello.unl stage0
  File "Unlambda.ml", line 204, characters 67-69:
  Warning 22: The CSP value is a closure or too deep to serialize
  File "Unlambda.ml", line 204, characters 58-66:
  Warning 22: The CSP value is a closure or too deep to serialize
  .<(* CSP eval_ref *) (* CSP p' *) []>.

To make my point clear: I don't actually need to print the code, even
though offline compilation would be awesome, I'm fine with just (.!)ing
my code for now. But I should be able to see the transformations I'm
doing. I'm perfectly fine with just (.!)ing my code, as long as I can
see what I'm running.

But in this case MetaOCaml is failing to print a code that IMO is
completely printable.

Thanks again for very helpful responses.

2015-05-01 7:21 GMT-04:00  <oleg@okmij.org>:
>
> After Jeremy's thorough reply, there is little more to say. I'd like
> to reinforce a few points. First of all, I was confused by
>
>   > The problem is MetaOCaml never prints generated code,
>   > it's always failing with `Warning 22: The CSP value is a closure or
>   > too deep to serialize`.
>
> because Warning 22 is a warning, it is not an error or
> failure. Second, MetaOCaml always prints the generated code, with the
> proviso that non-seriealizable CSP are printed in a way that make the
> code unsuitable for separate compilation. But the code is still
> suitable for Runcode.run. (see below). Your concrete example helped.
>
>> So now that I've expressed my frustration about this, I guess my main
>> question is: In what cases does MetaOCaml prints generated code? In
>> what cases it doesn't? Note that it fails to generate code even after
>> calling `print_closed_code (close_code ...)`, so being closed or not
>> doesn't seem very relevent here. Is this a bug in MetaOCaml? Are there
>> any workarounds? Any ideas what am I doing wrong?
>
> MetaOCaml always prints the generated code. However, if the generated
> code contains non-serializable CSP, they are printed in a way that
> makes saving the code to a file and then compiling separately
> impossible.
> Warning 22 was specifically introduced to tell that the generated code
> contains a non-serializable CSP. That means that separately compiling
> such code is impossible in principle. (I should admit that when CSP is
> serializable, it may still be in many cases printed badly -- but at
> least that problem is fixable in principle. Or even in practice: CSPs
> used internally by MetaOCaml are serializable and are printed in a
> good way.)
>
> BTW, one should distinguish ``code with badly printed CSP'' from
> ``fails to generate code''. The code with badly printed CSP is still a
> good code and can be run. So, seeing (* CSP x *) in the output is NOT
> a failure of code generation.
>
> Perhaps a concrete example will help.
>
> # let lift x = .<x>.;;
> val lift : 'a -> 'a code = <fun>
>
> # lift "aa";;
> - : string code = .<"aa">.
>
> The generated code can be saved to a file and compiled separately.
>
> # lift 1;;
> - : int code = .<(* CSP x *) Obj.magic 1>.
>
> Here, the generated code looks odd but it is still a well-formed OCaml
> code, which may be saved into a file and compiled separately.
>
>
> # let p = lift (+);;
> val p : (int -> int -> int) code = .<(* CSP x *)>.
>
> This code cannot be compiled separately (and it is accompanied by
> Warning 22). Here, CSP is a function (closure). How do you print
> closures (other than a sequence of bytes in memory, which is not very
> informative)? Saving it to a file is also problematic.
>
> Yet this is NO failure of code generation. The code p can be used as
> any other code value, for example being spliced:
>
> # let p1 = .<.~p 1 2>.;;
> val p1 : int code = .<(* CSP x *) 1 2>.
>
> And we can run the result:
>
> # Runcode.run p1;;
> - : int = 3
>
> There is no problems here.
>
> I should stress that there are two ways of using MetaOCaml: run-time
> code specialization and generating code for off-line use. In the
> former case, it is appropriate, and even beneficial, to share values
> between the generator and the generated code, via the common
> heap. This is especially good for large values or for closures, open
> channels, etc. Such sharing through the common heap is bad for
> off-line use, because there is no common heap any more. Warning 22 is
> emitted when there is shared through the common heap. This warning is
> issued at run-time of the generator: it is difficult to detect
> heap-based sharing at compile time: look at the function lift above.
> It is polymorphic, so we can't tell if sharing through the heap will
> be required from the type of the function (or its definition). In some
> uses of this function, like (lift "aa") and (lift 1), no heap sharing
> occurs (or, at least, it is not essential and irrelevant). In some
> cases, like (lift (+)), the heap-sharing is essential.
>
> That said, there is something we can do. For example, we can prohibit
> polymorphic lifting, like lift above. The user will have to supply
> their own function of the type t -> t code for some the concrete type
> t. Perhaps that is the way to go, and I am evaluating it. Warning 22
> is specifically introduced to help determine how much work the user
> will have to do if this solution is implemented. Basically, Warning 22
> will become a compile-time error, and the user will have to supply
> their own lifting function. That's where modular implicits will be of
> great help.
>
>
>> and `print_code_as_ast` is printing something that definitely doesn't
>> look like an AST for any imaginable programming language.(okay... I
>> might have exaggerated a bit, but it's definitely not OCaml AST and it
>> contains some references to generator program source, like
>> "Generator.ml[248]" for example)
>
> It may be hard to accept this, but what you see as the output of
> print_code_as_ast is indeed the OCaml AST, printed by OCaml's own
> Printast.implementation function.
>