Re: [Caml-list] [ANN] ppx_protobuf

[Peter Zotov <whitequark@whitequark.org>]

On 2014-05-04 19:18, Malcolm Matalka wrote:
> In my fantasy scenario you could annotate the accessor functions in a
> module.

I have just found this article:
http://cedeela.fr/universal-serialization-and-deserialization.html

Hopefully it can give some inspiration on how to implement such
a serialization library.

> 
> Peter Zotov <whitequark@whitequark.org> writes:
> 
>> On 2014-05-04 08:49, Malcolm Matalka wrote:
>>> Not exactly. I don't mean I want a functor, I just used that style to
>>> express that I think it would be best if these sort of things worked 
>>> on
>>> a module-to-module level rather than type.  That way I can separate 
>>> out
>>> the data type and it's business logic from its encoding/decoding 
>>> logic.
>>> I want to decouple a type definition from all of the transformations
>>> that can be done on the type.  Everything an still happen at a
>>> preprocessor point, but I just want it to happen on a module level.
>> 
>> Still not a good idea. Consider the annotations like @key and 
>> @encoding:
>> where would you specify them? If right on the type signature, then 
>> what
>> is the point of separation?
>> 
>>> 
>>> 
>>> Peter Zotov <whitequark@whitequark.org> writes:
>>> 
>>>> On 2014-05-03 22:46, Malcolm Matalka wrote:
>>>>> The idea I mean is more to do this at the module level than the 
>>>>> type
>>>>> level, like a functor.  So rather than defining protobuf for a type
>>>>> definition, define it for a module, and have some convention for 
>>>>> how to
>>>>> pick out setter/getter functions.  Then create a new module from 
>>>>> that.
>>>> 
>>>> Oh! You want a functor which would be able to examine the structure
>>>> of the module that was passed to it.
>>>> 
>>>> It's probably technically feasible (you need a syntactic extension
>>>> which would essentially serialize the module that will be passed), 
>>>> but
>>>> it is a really horrible solution:
>>>> 
>>>>   * You won't be able to report some interesting errors (such as
>>>>     incorrect annotations... [@key -1] until runtime.
>>>>   * It will be really slow, because the implementation of the 
>>>> functor
>>>>     will have to traverse the lists of fields dynamically and invoke
>>>>     accessors one by one. My current implementation directly pattern
>>>>     matches the input.
>>>>   * It is just really complicated and does too much at runtime.
>>>> 
>>>>> 
>>>>> For example of the top of my head:
>>>>> 
>>>>> module Foo = sig
>>>>>    type t
>>>>>    val set_x : t -> int -> t
>>>>>    val get_x : t -> int
>>>>> end
>>>>> 
>>>>> Then I can do:
>>>>> 
>>>>> module Foo_protobuf = Protobuf.Make(Foo)
>>>>> 
>>>>> In this case I stole how most people to functors to make it clear 
>>>>> the
>>>>> translation is actually module to module.
>>>>> 
>>>>> The reason I prefer this is because I can also do:
>>>>> 
>>>>> module Foo_xml = Xml.Make(Foo)
>>>>> module Foo_json = Json.Make(Foo)
>>>>> 
>>>>> By separating the mechanism for creating the decoders from the type
>>>>> definition, I can add decoders for any type I want without 
>>>>> disturbing
>>>>> the original definition.  This feels more right to me.  But I have 
>>>>> no
>>>>> idea how to do it.
>>>>> 
>>>>> 
>>>>> Peter Zotov <whitequark@whitequark.org> writes:
>>>>> 
>>>>>> On 2014-05-03 20:08, Malcolm Matalka wrote:
>>>>>>> Nice, great work!
>>>>>>> 
>>>>>>> I'm not actually a huge fan of mixing type definitions and the 
>>>>>>> protocols
>>>>>>> they can be encoded/decoded from.  How hard would it be to take a 
>>>>>>> module
>>>>>>> definition accessors on a type and produce a new module with
>>>>>>> encode/decode functions?  That way I could create JSON, XML, 
>>>>>>> Protobufs,
>>>>>>> etc modules from one module.
>>>>>> 
>>>>>> Do you suggest generating the following signature instead of the 
>>>>>> current
>>>>>> one?
>>>>>> 
>>>>>> type t = ... [@@protobuf]
>>>>>> module Protobuf_t : sig
>>>>>>   val decode : Protobuf.Decoder.t -> t
>>>>>>   val encode : Protobuf.Encoder.t -> t -> unit
>>>>>> end
>>>>>> 
>>>>>> This would be similar to what deriving currently does.
>>>>>> 
>>>>>> In principle, this is not a complex change. It would add just a 
>>>>>> few lines
>>>>>> to ppx_protobuf.
>>>>>> 
>>>>>> However, I don't like it conceptually. I think the flat signature 
>>>>>> is
>>>>>> more natural, it mimics what one would usually write by hand 
>>>>>> without
>>>>>> introducing too much deep nesting of modules. You may notice how
>>>>>> ppx_protobuf doesn't generate the signature items for you; this is
>>>>>> because ppx_protobuf is a mere implementation detail, a convenient
>>>>>> way to generate the serializer/deserializer.
>>>>>> 
>>>>>> I'm not going to oppose addition of such a mode for two reasons:
>>>>>>   * I don't like fighting over minute details.
>>>>>>   * More importantly, deriving, when rewritten with ppx in mind,
>>>>>>     will surely contain this mode for compatibility. ppx_protobuf
>>>>>>     will be (ideally) rewritten over deriving some day.
>>>>>> 
>>>>>> I will happily merge a PR adding such a mode to ppx_protobuf.
>>>>>> 
>>>>>>> 
>>>>>>> Just an idea!
>>>>>>> 
>>>>>>> Peter Zotov <whitequark@whitequark.org> writes:
>>>>>>> 
>>>>>>>> Greetings.
>>>>>>>> 
>>>>>>>> I have just released the first version of ppx_protobuf, a 
>>>>>>>> complete
>>>>>>>> Protocol Buffers implementation. Unlike Google's implementation,
>>>>>>>> ppx_protobuf derives the message structure directly from OCaml 
>>>>>>>> type
>>>>>>>> definitions, which allows a much more seamless integration with
>>>>>>>> OCaml's types. In particular, ppx_protobuf natively supports
>>>>>>>> sum types, while maintaining full backwards compatibility with
>>>>>>>> protoc.
>>>>>>>> 
>>>>>>>> ppx_protobuf uses the extension points API, and thus requires
>>>>>>>> a recent (>= 2014-04-29) 4.02 (trunk) compiler. It also requires
>>>>>>>> an unreleased version of ppx_tools. It is probably easiest
>>>>>>>> to install both from the source repositories[1][2].
>>>>>>>> 
>>>>>>>> The API is extensively documented at [3].
>>>>>>>> 
>>>>>>>> [1]: https://github.com/whitequark/ocaml-ppx_protobuf.git
>>>>>>>> [2]: https://github.com/alainfrisch/ppx_tools.git
>>>>>>>> [3]: 
>>>>>>>> https://github.com/whitequark/ocaml-ppx_protobuf/blob/master/README.md
>>>>>>>> 
>>>>>>>> --
>>>>>>>>   WBR, Peter Zotov.

-- 
Peter Zotov
sip:whitequark@sipnet.ru