From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id 2C48E7F726 for ; Mon, 5 May 2014 00:21:13 +0200 (CEST) Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of whitequark@whitequark.org) identity=pra; client-ip=176.58.103.125; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="whitequark@whitequark.org"; x-sender="whitequark@whitequark.org"; x-conformance=sidf_compatible Received-SPF: Pass (mail3-smtp-sop.national.inria.fr: domain of whitequark@whitequark.org designates 176.58.103.125 as permitted sender) identity=mailfrom; client-ip=176.58.103.125; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="whitequark@whitequark.org"; x-sender="whitequark@whitequark.org"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@mail.whitequark.org) identity=helo; client-ip=176.58.103.125; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="whitequark@whitequark.org"; x-sender="postmaster@mail.whitequark.org"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjtgADe8ZlOwOmd9/2dsb2JhbABYg1WrKwEBAQEBAQUBgSiYYoEmdIIlAQEEAScRAj8QBAcOCh0RLCsGEwgTiB4MCcoFF4VWgz4dhBSBDQeEPwSEWgKFJZBviUKHdoM5OA X-IPAS-Result: AjtgADe8ZlOwOmd9/2dsb2JhbABYg1WrKwEBAQEBAQUBgSiYYoEmdIIlAQEEAScRAj8QBAcOCh0RLCsGEwgTiB4MCcoFF4VWgz4dhBSBDQeEPwSEWgKFJZBviUKHdoM5OA X-Spam-Status: Yes X-IronPort-AV: E=Sophos;i="4.97,984,1389740400"; d="scan'208";a="60519735" Received: from fehu.whitequark.org (HELO mail.whitequark.org) ([176.58.103.125]) by mail3-smtp-sop.national.inria.fr with ESMTP; 05 May 2014 00:21:12 +0200 Received: by mail.whitequark.org (Postfix, from userid 33) id 0207A4D743; Sun, 4 May 2014 22:21:09 +0000 (UTC) To: Malcolm Matalka X-PHP-Originating-Script: 1000:rcube.php MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Mon, 05 May 2014 02:21:09 +0400 From: Peter Zotov Cc: caml-list , info@gerd-stolpmann.de In-Reply-To: <87bnvd374f.fsf@gmail.com> References: <6232dace806569a16f7fbfaa1689ef42@whitequark.org> <87wqe23kvw.fsf@gmail.com> <0f8bb41e5638586926c6975e1def53e9@whitequark.org> <87ppju3dk6.fsf@gmail.com> <3beba9cb0078930fd8a48c4408fa1773@whitequark.org> <87iopm2lnk.fsf@gmail.com> <3d5dc7368c12c92ab9cb1a45f1210360@whitequark.org> <87bnvd374f.fsf@gmail.com> Message-ID: X-Sender: whitequark@whitequark.org User-Agent: Roundcube Webmail/1.0.0 Subject: Re: [Caml-list] [ANN] ppx_protobuf On 2014-05-04 19:18, Malcolm Matalka wrote: > In my fantasy scenario you could annotate the accessor functions in a > module. I have just found this article: http://cedeela.fr/universal-serialization-and-deserialization.html Hopefully it can give some inspiration on how to implement such a serialization library. > > Peter Zotov writes: > >> On 2014-05-04 08:49, Malcolm Matalka wrote: >>> Not exactly. I don't mean I want a functor, I just used that style to >>> express that I think it would be best if these sort of things worked >>> on >>> a module-to-module level rather than type. That way I can separate >>> out >>> the data type and it's business logic from its encoding/decoding >>> logic. >>> I want to decouple a type definition from all of the transformations >>> that can be done on the type. Everything an still happen at a >>> preprocessor point, but I just want it to happen on a module level. >> >> Still not a good idea. Consider the annotations like @key and >> @encoding: >> where would you specify them? If right on the type signature, then >> what >> is the point of separation? >> >>> >>> >>> Peter Zotov writes: >>> >>>> On 2014-05-03 22:46, Malcolm Matalka wrote: >>>>> The idea I mean is more to do this at the module level than the >>>>> type >>>>> level, like a functor. So rather than defining protobuf for a type >>>>> definition, define it for a module, and have some convention for >>>>> how to >>>>> pick out setter/getter functions. Then create a new module from >>>>> that. >>>> >>>> Oh! You want a functor which would be able to examine the structure >>>> of the module that was passed to it. >>>> >>>> It's probably technically feasible (you need a syntactic extension >>>> which would essentially serialize the module that will be passed), >>>> but >>>> it is a really horrible solution: >>>> >>>> * You won't be able to report some interesting errors (such as >>>> incorrect annotations... [@key -1] until runtime. >>>> * It will be really slow, because the implementation of the >>>> functor >>>> will have to traverse the lists of fields dynamically and invoke >>>> accessors one by one. My current implementation directly pattern >>>> matches the input. >>>> * It is just really complicated and does too much at runtime. >>>> >>>>> >>>>> For example of the top of my head: >>>>> >>>>> module Foo = sig >>>>> type t >>>>> val set_x : t -> int -> t >>>>> val get_x : t -> int >>>>> end >>>>> >>>>> Then I can do: >>>>> >>>>> module Foo_protobuf = Protobuf.Make(Foo) >>>>> >>>>> In this case I stole how most people to functors to make it clear >>>>> the >>>>> translation is actually module to module. >>>>> >>>>> The reason I prefer this is because I can also do: >>>>> >>>>> module Foo_xml = Xml.Make(Foo) >>>>> module Foo_json = Json.Make(Foo) >>>>> >>>>> By separating the mechanism for creating the decoders from the type >>>>> definition, I can add decoders for any type I want without >>>>> disturbing >>>>> the original definition. This feels more right to me. But I have >>>>> no >>>>> idea how to do it. >>>>> >>>>> >>>>> Peter Zotov writes: >>>>> >>>>>> On 2014-05-03 20:08, Malcolm Matalka wrote: >>>>>>> Nice, great work! >>>>>>> >>>>>>> I'm not actually a huge fan of mixing type definitions and the >>>>>>> protocols >>>>>>> they can be encoded/decoded from. How hard would it be to take a >>>>>>> module >>>>>>> definition accessors on a type and produce a new module with >>>>>>> encode/decode functions? That way I could create JSON, XML, >>>>>>> Protobufs, >>>>>>> etc modules from one module. >>>>>> >>>>>> Do you suggest generating the following signature instead of the >>>>>> current >>>>>> one? >>>>>> >>>>>> type t = ... [@@protobuf] >>>>>> module Protobuf_t : sig >>>>>> val decode : Protobuf.Decoder.t -> t >>>>>> val encode : Protobuf.Encoder.t -> t -> unit >>>>>> end >>>>>> >>>>>> This would be similar to what deriving currently does. >>>>>> >>>>>> In principle, this is not a complex change. It would add just a >>>>>> few lines >>>>>> to ppx_protobuf. >>>>>> >>>>>> However, I don't like it conceptually. I think the flat signature >>>>>> is >>>>>> more natural, it mimics what one would usually write by hand >>>>>> without >>>>>> introducing too much deep nesting of modules. You may notice how >>>>>> ppx_protobuf doesn't generate the signature items for you; this is >>>>>> because ppx_protobuf is a mere implementation detail, a convenient >>>>>> way to generate the serializer/deserializer. >>>>>> >>>>>> I'm not going to oppose addition of such a mode for two reasons: >>>>>> * I don't like fighting over minute details. >>>>>> * More importantly, deriving, when rewritten with ppx in mind, >>>>>> will surely contain this mode for compatibility. ppx_protobuf >>>>>> will be (ideally) rewritten over deriving some day. >>>>>> >>>>>> I will happily merge a PR adding such a mode to ppx_protobuf. >>>>>> >>>>>>> >>>>>>> Just an idea! >>>>>>> >>>>>>> Peter Zotov writes: >>>>>>> >>>>>>>> Greetings. >>>>>>>> >>>>>>>> I have just released the first version of ppx_protobuf, a >>>>>>>> complete >>>>>>>> Protocol Buffers implementation. Unlike Google's implementation, >>>>>>>> ppx_protobuf derives the message structure directly from OCaml >>>>>>>> type >>>>>>>> definitions, which allows a much more seamless integration with >>>>>>>> OCaml's types. In particular, ppx_protobuf natively supports >>>>>>>> sum types, while maintaining full backwards compatibility with >>>>>>>> protoc. >>>>>>>> >>>>>>>> ppx_protobuf uses the extension points API, and thus requires >>>>>>>> a recent (>= 2014-04-29) 4.02 (trunk) compiler. It also requires >>>>>>>> an unreleased version of ppx_tools. It is probably easiest >>>>>>>> to install both from the source repositories[1][2]. >>>>>>>> >>>>>>>> The API is extensively documented at [3]. >>>>>>>> >>>>>>>> [1]: https://github.com/whitequark/ocaml-ppx_protobuf.git >>>>>>>> [2]: https://github.com/alainfrisch/ppx_tools.git >>>>>>>> [3]: >>>>>>>> https://github.com/whitequark/ocaml-ppx_protobuf/blob/master/README.md >>>>>>>> >>>>>>>> -- >>>>>>>> WBR, Peter Zotov. -- Peter Zotov sip:whitequark@sipnet.ru