Re: [Caml-list] Re: Serialisation of PXP DTDs

["Markus Mottl" <markus.mottl@gmail.com>]

On Sat, Oct 25, 2008 at 2:58 PM, Mauricio Fernandez <mfp@acm.org> wrote:
> Unfortunately, growing sum types is far from being the only protocol extension
> of interest. There's a trivial extension which, I suspect, will be at
> least as common in practice, namely adding new fields to a record (or new
> elements to a tuple). bin-prot is unable to handle it adequately --- a
> self-describing format like the one I'm working on is required.

If you add a tag to a sum type, previous protocol implementations
cannot read these values, whereas new implementations will be able to
read both protocols.  With records / tuples it is exactly the other
way round: you could, in principle, read both in the old
implementation, which just needs to drop new, unknown fields, whereas
the new implementation requires these fields and hence cannot parse
old protocols.

I don't see how any approach could "hande" the respective unsolvable
case.  If a receiver doesn't know how to handle a tag, or if it
requires data that is not there, you'll be stuck.

Note, too, that even if you created an implementation which allows
handling extended records in old protocols, this would undoubtly come
at a pretty hefty cost.  The only efficient way to do that would be to
exchange protocols and generate code at runtime to translate quickly
between protocols.  I don't think it's worth it.

> You might argue that this extension is subsumed by the ability to grow sum types,
> since you can go from
>
>    type record = { a : int } with bin_io
>    type msg = A of record
>
> to
>
>    type record1 = { a : int } with bin_io
>    type record2 = { a' : int; b : int } with bin_io
>    type msg = A of record1 | B of record2
>
> (Note how special care has to be taken to tag the record --- "explicit
> tagging" in ASN.1 parlance.)

This is surely a clean way to extend protocols without losing backward
compatibility.

> My design lifts that restriction and allows an old consumer to read the data
> from a new producer when new fields have been added to a record or a tuple.

I'd probably bet that simply putting a protocol translator in front of
some old application you don't want to / cannot recompile would be
about as efficient.  Unless, of course, you go for the "generate
efficient translation code from a new protocol specifications at
runtime" approach, which seems very hard to implement.  And it
wouldn't even be as general, since an intermediate translator could
translate between previously completely unrelated, arbitrary protocols
(as long as you can define a meaningful translation).  It's hard to
imagine that anybody wouldn't want to use a type safe language with
pattern matching (like OCaml) to specify that part...

> AFAICS the ability to process data not understood in full requires the use of
> a self-describing format like the one I'm working on.

I'd go for the protocol translator.  Especially if two protocols share
a lot of structure, it should be trivial to define translations.
Another very reasonable approach, which does not diminish performance,
would be to exchange protocol versions.  Assuming that one side is
always more recent than the other, they should be able to support old
protocols directly.

Regards,
Markus

-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com