Deriving + type-conv + OCaml-Templates + camlp4* = ?

Deriving + type-conv + OCaml-Templates + camlp4* = ?

[David Teller]

    Dear list,

 Browsing around the net, I've found three
"scrap-your-boilerplate"-style projects for OCaml: the simple Type-conv,
the ambitious Deriving and the unmaintained OCaml-Templates, and of
course the ability to use camlp4/camlp5 for the same purpose. I imagine
that there are a number of nice boilerplate-based modules just waiting
to be implemented or adopted.

 As far as I understand, there's no interaction between the project
authors, which is a shame. So this is an open call to whoever is in
charge of each work: do you think it would be possible for you all to
join forces and produce something robust, simple and possible to adopt
as a standard ?

Cheers,
 David

-- 
David Teller
 Security of Distributed Systems
  http://www.univ-orleans.fr/lifo/Members/David.Teller
 Angry researcher: French Universities need reforms, but the LRU act brings liquidations. 

Re: [Caml-list] Deriving + type-conv + OCaml-Templates + camlp4* = ?

[Markus Mottl]

On Tue, Feb 26, 2008 at 3:27 PM, David Teller <David.Teller@ens-lyon.org> wrote:
>   Browsing around the net, I've found three
>  "scrap-your-boilerplate"-style projects for OCaml: the simple Type-conv,
>  the ambitious Deriving and the unmaintained OCaml-Templates, and of
>  course the ability to use camlp4/camlp5 for the same purpose. I imagine
>  that there are a number of nice boilerplate-based modules just waiting
>  to be implemented or adopted.
>
>   As far as I understand, there's no interaction between the project
>  authors, which is a shame. So this is an open call to whoever is in
>  charge of each work: do you think it would be possible for you all to
>  join forces and produce something robust, simple and possible to adopt
>  as a standard ?

I think that different projects have different trade-offs that are
hard, if not impossible, to combine.  Deriving is a very elegant,
general approach, but last time I checked it would be hard to generate
highly optimized code with it, and it was also not complete (e.g.
handling hard cases like polymorphic variants with inherited types,
etc.).  Type-conv was mostly factored out of sexplib to make it
possible to have different type converters in different preprocessors,
because we are planning to release our binary protocol library very
soon.  Type-conv only combines a few often needed constructs, but in
my experience the very vast part of new type converters has to be
written by hand anyway (at least if you care about performance / good
code generation) so there is little to gain here.  I don't know much
about OCaml-Templates.

Regards,
Markus

-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com

Re: [Caml-list] Deriving + type-conv + OCaml-Templates + camlp4* = ?

[Jeremy Yallop]

Markus Mottl wrote:
> On Tue, Feb 26, 2008 at 3:27 PM, David Teller 
> <David.Teller@ens-lyon.org> wrote:
>> Browsing around the net, I've found three 
>> "scrap-your-boilerplate"-style projects for OCaml: the simple 
>> Type-conv, the ambitious Deriving and the unmaintained 
>> OCaml-Templates, and of course the ability to use camlp4/camlp5 for
>>  the same purpose. I imagine that there are a number of nice 
>> boilerplate-based modules just waiting to be implemented or 
>> adopted.
>> 
>> As far as I understand, there's no interaction between the project 
>> authors, which is a shame. So this is an open call to whoever is in
>> charge of each work: do you think it would be possible for you all
>> to join forces and produce something robust, simple and possible to
>> adopt as a standard ?
> 
> I think that different projects have different trade-offs that are 
> hard, if not impossible, to combine.

I fear that you may be right, although (as the author of deriving) I'd
be more than happy to consider collaboration, given a concrete proposal.

> Deriving is a very elegant, general approach, but last time I checked
>  it would be hard to generate highly optimized code with it,

Thanks for the kind words!  I expect you're right about generating
optimized code; as you suggest this hasn't been a focus.  Performance of
generated code should be at least reasonable, though, and it is possible
to substitute handwritten code at critical points, as with any such
approach.

> and it was also not complete (e.g. handling hard cases like
> polymorphic variants with inherited types, etc.).

This is a little more surprising, since completeness *is* a focus of 
deriving.

If I understand you rightly, then the case you mention is actually 
handled: are you referring to using deriving with a polymorphic variant 
type that extends another, such as

    type a = [`A]
    type ab = [a|`B]

?

This sort of thing is certainly handled; indeed, deriving handles 
considerably more complicated cases as well, e.g. involving various 
sorts of recursion:

    type ('a,'b) x = [`One of 'a | `Two of 'b]           deriving (Show)
    type 'a y = [`Four of ([('z,'a) x | `Three] as 'z)]  deriving (Show)
    type z = z y                                         deriving (Show)

    let _ = print_endline (Show.show<z>
                            (`Four (`One (`Two (`Four `Three)))))

In fact, the way that deriving treats structural types with inline 
recursion (i.e. the "as" syntax) is perhaps an example of a difficulty 
in combining projects: deriving converts types into a normalized 
representation in which this recursion is shifted to the type 
declaration level: a sort of ANF for types.  This works well (and makes 
it much easier to write new generic functions, since it removes the need 
to deal with "as"), but might be a little heavyweight for projects with 
other focuses.

To avoid giving the impression that deriving is absolutely complete, I 
should mention that nested (i.e. irregular) types are not handled.  This 
is in some ways a limitation of the approach of using modules for 
recursion.  I don't think this limitation is significant for most 
people, but I may look into adding support at some point.

Jeremy.

Re: [Caml-list] Deriving + type-conv + OCaml-Templates + camlp4* = ?

[Markus Mottl]

On Wed, Feb 27, 2008 at 5:56 AM, Jeremy Yallop <jeremy.yallop@ed.ac.uk> wrote:
>  > I think that different projects have different trade-offs that are
>  > hard, if not impossible, to combine.
>
>  I fear that you may be right, although (as the author of deriving) I'd
>  be more than happy to consider collaboration, given a concrete proposal.

We'd certainly be very happy to have a general framework for deriving
code from type definitions, since we (Jane Street Capital) have been
using this idea very successfully in our projects.  Having written two
complete and fairly complex libraries of that kind, I found that the
overlap between them was so small that it almost doesn't count in
comparison to the total effort required.  Thus I'm rather skeptical
that a general purpose library for type-derived code will be possible
without giving up on performance, but I'd surely like to be
surprised....

>  This is a little more surprising, since completeness *is* a focus of
>  deriving.
>
>  If I understand you rightly, then the case you mention is actually
>  handled: are you referring to using deriving with a polymorphic variant
>  type that extends another, such as
>
>     type a = [`A]
>     type ab = [a|`B]

Right.

>  This sort of thing is certainly handled; indeed, deriving handles
>  considerably more complicated cases as well, e.g. involving various
>  sorts of recursion:

Ok, I possibly misremembered something.  Sorry about that.

Sexplib treats this issue (inherited polymorphic variant types) by
using backtracking.  This has the advantage of a clear semantics
concerning which type converter will be used if types overlap (always
the first one left-to-right).  This doesn't seem to be the case with
some Deriving generators, but this may not be a big deal there.

>  To avoid giving the impression that deriving is absolutely complete, I
>  should mention that nested (i.e. irregular) types are not handled.  This
>  is in some ways a limitation of the approach of using modules for
>  recursion.  I don't think this limitation is significant for most
>  people, but I may look into adding support at some point.

This is surely a very minor limitation.  I can't remember the last
time I encountered irregular types in real life.  Sexplib seems to
support them, though one has to pass the -rectypes flag to the
compiler, of course.

Regards,
Markus

-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com

Re: [Caml-list] Deriving + type-conv + OCaml-Templates + camlp4* = ?

[Till Varoquaux]

type-conv in itself just puts the base down to be a common plate for
type converter. The main reason for its inception was to avoid
conflicts between different camlp4 extensions that  scaffold types
(e.g. sexp), it does not dispense from writing all that gruesome
camlp4 code (this is in no way a critizisme of Nicolas work, hats off
to both him and Daniel De Raugladre for the sheer amount of work they
have done). Markus Mottl would be able to tell you more.

I have not tried Deriving so I am not able to say much about it. I can
however point out two approaches I have been involved in:
_We (jane street) have recently been experimenting with a Camlp4
extension that automatically projects to and from universal types. It
is still in its infancy but seems quite promising (disclaimer: I am
the main author of the extention). Since this is such an amazing
exciting, revolutionary, ground-breaking etc... project (what kind of
monster would say his newborn is ugly) I will preceed the question
that is bound to arise: why is not open sourced?
   1) It is not my decision to make (this is not as big of an issue as
it might seem)
   2) Releasing bad code would hurt our credibility and give a bad
image. Documenting and polishing this extension is a lot more work
that what meets the eyes (Pareto principle....)
   3) The code in our system is deeply intertwined doing a clean
separation whilst not introducing artificial problems or code
duplication is yet again an additional burden.
I am fairly confident that if 2 and 3 are solved 1 will not be much of
an issue. Of course this is a pie in the sky.
_I have been toying with recursive modules, polymorphic variants and
Camlp4 to build a toy layered compiler. The code is a nice proof of
concept, but, as previously, does not address any of the exotic cases
of the long trail (nor does it have proper documentation or error
reporting). If you are not scared of having a bumping ride it can be
browsed online at:
http://till.varoquaux.free.fr/projects/janus/

Now I feel I should give credit (and disclaimers) where due:
_I am currently the only person using the first extension. I am fixing
it when it breaks down it is not used in critical production code.Do
not be under the impression that the code at Jane Street is not
carefully peer reviewed. This work was based on work by David  Powers
and others. As always my statements are purely personnel and do not
reflect the views of my employer (although I do not think I am parting
much with them).
_Jerome Vouillon, Vincent Balat counseled me and guided (err..
suffered) me through the creation of Janus. I should be held
responsible for the sorry state it is in, but they are to be praised
for the little that has been made. Jacques Garrigue was also of great
help, not to mention I stole most of my ideas from his articles.

Till

On Tue, Feb 26, 2008 at 8:27 PM, David Teller <David.Teller@ens-lyon.org> wrote:
>     Dear list,
>
>   Browsing around the net, I've found three
>  "scrap-your-boilerplate"-style projects for OCaml: the simple Type-conv,
>  the ambitious Deriving and the unmaintained OCaml-Templates, and of
>  course the ability to use camlp4/camlp5 for the same purpose. I imagine
>  that there are a number of nice boilerplate-based modules just waiting
>  to be implemented or adopted.
>
>   As far as I understand, there's no interaction between the project
>  authors, which is a shame. So this is an open call to whoever is in
>  charge of each work: do you think it would be possible for you all to
>  join forces and produce something robust, simple and possible to adopt
>  as a standard ?
>
>  Cheers,
>   David
>
>  --
>  David Teller
>   Security of Distributed Systems
>   http://www.univ-orleans.fr/lifo/Members/David.Teller
>   Angry researcher: French Universities need reforms, but the LRU act brings liquidations.
>
>  _______________________________________________
>  Caml-list mailing list. Subscription management:
>  http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>  Archives: http://caml.inria.fr
>  Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>  Bug reports: http://caml.inria.fr/bin/caml-bugs
>



-- 
http://till-varoquaux.blogspot.com/