[Caml-list] extending a type with Marshal

[Caml-list] extending a type with Marshal

[Ker Lutyn]

Marshal provides a convenient way to pass information between
components. It requires that types be the same at either end. But for
multi-machine production systems that must serve traffic continuously,
you cannot count on upgrading all your systems simultaneously.

It is reasonable to want to add capabilities to, say, a web server, in
a backward compatible way so that, say, a legacy content server can
still be used. Later a new content server can be deployed that
responds to the new capabilities. Conversely, we might want to augment
the content server with additional information that will later be used
by an upgraded web server.

    Foo -- A --> Bar

In other words, Foo sends message A to Bar. We'd like to be able to
handle two possibilities:

(1) Foo sends message A' to Bar, who interprets it as A.
(2) Foo sends message A to Bar, who interprets it as A'.

...where A' is an extension of A.

To do this it would be necessary to 'extend' a type at one end of a
Marshal connection without extending it at the other end. Is this
possible? Do people have any suggestions?

__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] extending a type with Marshal

[Richard Jones]

Why not just prefix each message with a version number. Recipients
then need to be able to support just current_version and
current_version-1[*] of each message type.

Rich.

[*] Implying the knowledge to upgrade from this to current_version.

-- 
Richard Jones. http://www.annexia.org/ http://freshmeat.net/users/rwmj
Merjis Ltd. http://www.merjis.com/ - all your business data are belong to you.
"My karma ran over your dogma"

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] extending a type with Marshal

[Peter Jolly]

>It is reasonable to want to add capabilities to, say, a web server, in
>a backward compatible way so that, say, a legacy content server can
>still be used. Later a new content server can be deployed that
>responds to the new capabilities. Conversely, we might want to augment
>the content server with additional information that will later be used
>by an upgraded web server.

Isn't this precisely the sort of problem that extensible data interchange
formats such as XML are intended to solve? 

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] extending a type with Marshal

[skaller]

On Thu, 2003-10-09 at 07:22, Ker Lutyn wrote:
> Marshal provides a convenient way to pass information between
> components. It requires that types be the same at either end. But for
> multi-machine production systems that must serve traffic continuously,
> you cannot count on upgrading all your systems simultaneously.
> 

If you have p2p connection: Negotiate.
That's the traditional solution: used by modems,
for example.

Fixing a negotiation protocol isn't hard.

IMHO the hard part is identifying 'versions'
and 'capbilities'. Probably the best way is
to use a string encoding (and use a parser).

Nastiest example I can think of is requesting a font.

Note this suggestion requires both servers and clients
to maintain capability to handle several 'old' versions.

This isn't extension of the Marshalled data, rather
quite distinct data for each 'version' is possible,
so the solution seems more general: the cost is extra
handshakes -- how expensive that is depends on how long
the agreement lasts (until explicitly refreshed? until
end of session? only for a single data fetch?)


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] extending a type with Marshal

[Christian Rinderknecht]

Hello Ker,

On Wed, 2003-10-08 at 23:22, Ker Lutyn wrote:
> [...]
> In other words, Foo sends message A to Bar. We'd like to be able to
> handle two possibilities:
> 
> (1) Foo sends message A' to Bar, who interprets it as A.
> (2) Foo sends message A to Bar, who interprets it as A'.
> 
> ...where A' is an extension of A.
> 
> To do this it would be necessary to 'extend' a type at one end [...]

This kind of concept is already present (subtyping, versioning,
relaying) in a specification language called Abstract Syntax Notation
One (ASN.1), and, at least, easily supported by one of its encoding rule
called Basic Encoding Rule (BER).

The point of ASN.1 is sharing the same ASN.1 module between the two
possibly hetereogeneous peers, also agreeing on an encoding/decoding
scheme (e.g. BER) and then each peer compile the ASN.1 module into type
definitions and codecs in his application language (obviously, you seems
interested in O'Caml). Then compile and link.

In my PhD thesis I wrote a parser and a checker for ASN.1:1988/90, but
no code generation for O'Caml. I remember that Erlang had such a
facility integrated transparently. It is worth looking this way.

For all information about ASN.1, see the excellent site of France
Telecom R&D at http://asn1.elibel.tm.fr/. By the way, these folks
develop some ASN.1 tools in O'Caml, but do not disclose their source
code in general.


Hope this helps,


Christian

-- 
---------------------------------------------------------------------------
Christian Rinderknecht                        Phone: +33 1 41 16 70 39
Assistant Professor                           Fax:   +33 1 41 16 71 71
D.E.R. Genie Informatique (ESILV)
Pole Universitaire Leonard de Vinci
F-92916 Paris La Defense Cedex (France)

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] extending a type with Marshal

[skaller]

On Fri, 2003-10-10 at 19:19, Christian Rinderknecht wrote:
> Hello Ker,
> 

> This kind of concept is already present (subtyping, versioning,
> relaying) in a specification language called Abstract Syntax Notation
> One (ASN.1), and, at least, easily supported by one of its encoding rule
> called Basic Encoding Rule (BER).

There is also a FFAU (free for any use) open source ASN 1.1 library
around somewhere for C++, and tools for compiling ASN specifications
(into C++ sources). I can't give a reference, sorry, I ran into
these beasts while working at a (now defunct) telco. I imagine
google can find it though :-)



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] extending a type with Marshal

[Martin Berger]

 > In my PhD thesis I wrote a parser and a checker for ASN.1:1988/90, but
 > no code generation for O'Caml. I remember that Erlang had such a
 > facility integrated transparently. It is worth looking this way.
 >
 > For all information about ASN.1, see the excellent site of France
 > Telecom R&D at http://asn1.elibel.tm.fr/. By the way, these folks
 > develop some ASN.1 tools in O'Caml, but do not disclose their source
 > code in general.

for a long time i have been toying with the idea of writing a proper
network management systems (with layer 2 topology detection) in
ocaml. it seems an ideal language for this purpose. it  would also be
a "killer application" for ocaml, if successful. one of the things
that have prevented me from doing so was my dread at writing parser,
especially for things like ASN.1 ... if however usable and reasonably
licensed ocaml libraries for that sort thing already existed, i *may*
reconsider ... so my question for you is: do these libraries exist
in the public domain, would  your phd's parser be suitable for such
a project, or, if not, can the france telecom R&D suits be convinced
to donate a bit of code?

martin


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] extending a type with Marshal

[William Chesters]

Christian Rinderknecht writes:
 > Hello Ker,
 > 
 > On Wed, 2003-10-08 at 23:22, Ker Lutyn wrote:
 > > [...]
 > > In other words, Foo sends message A to Bar. We'd like to be able to
 > > handle two possibilities:
 > > 
 > > (1) Foo sends message A' to Bar, who interprets it as A.
 > > (2) Foo sends message A to Bar, who interprets it as A'.
 > > 
 > > ...where A' is an extension of A.
 > > 
 > > To do this it would be necessary to 'extend' a type at one end [...]
 > 
 > This kind of concept is already present (subtyping, versioning,
 > relaying) in a specification language called Abstract Syntax Notation
 > One (ASN.1), and, at least, easily supported by one of its encoding rule
 > called Basic Encoding Rule (BER).
 > 
 > The point of ASN.1 is sharing the same ASN.1 module between the two
 > possibly hetereogeneous peers, also agreeing on an encoding/decoding
 > scheme (e.g. BER) and then each peer compile the ASN.1 module into type
 > definitions and codecs in his application language (obviously, you seems
 > interested in O'Caml). Then compile and link.

   I think it's only fair to point out that many people dislike ASN.1,
and that it may not be a perfect match for what you're trying to
achieve.

   It's true that protocol forward compatibility and negotiation is
supported, but to get it you have to use ASN.1, plus the encoding
rules (nearly always BER, will be handled by the ASN.1 library if you
find one), plus the OSI-inspired TCAP meta-protocol.  All of these are
surprisingly complicated, obscure, and heavyweight given what they
actually do; and of course it's ultimately left up to the client app
to decide what to do when talking to an older server, so they don't
magically solve all your problems.

   And the much trumpted forward compatibility in ASN.1 basically
boils down to the following facts:

   -- Sum types (in ocaml terms) can be encoded in ASN.1 using
so-called "explicit tags": a number is prefixed saying which variant
is present, just like in ocaml's marshal representation.  Because you
are required to give your own numeric values for these tags, you can
achieve forward compatibility by adding new tag-values rather than
re-using old ones when you extend the datatype.

   -- Product types (records and tuples) can be encoded using
"sequences"; again, the elements of the sequence are tagged, and
forward compatibility comes from inventing new tags for new members.

   That's all.  As I said TCAP adds a simple kind of "protocol
negotiation protocol".

   Why go to the trouble of constructing a mapping between the ocaml
values you want to transmit and ASN.1, when you could just encode them
directly in a homegrown encoding with less fuss?  In fact I think with
care you could get upwards compatibility using marshal directly.
Why not just write

      (* version 2 *)

      type foo = { a: string; b: int; extensions: unit option }

      type bar = First of int
               | Second
               | Extended of unit

      (* version 3 *)

      type foo = { a: string; b: int; extensions: foo_v3 option }
      and foo_v3 = { c: float }

      type bar = First of int
               | Second
               | Extended of bar_v3
      and bar_v3 = Third of string
                 | Fourth

This will work because of the way ocaml in practice lays things out,
and automatically induces the right protocol-fallback structure in
your code:

     let doit (foo: foo) =
       printf "a %s b %d " foo.a foo.b;
       match foo.extensions with
         | Some foo_v3 -> printf "c %f" foo_v3.c
         | _ -> printf "(no v3 extensions")



   What would be much more interesting than ASN.1 would be something
that ensured upward compatibility but was (a) more tightly integrated
with ocaml datatype declarations, perhaps via camlp4, so that it
was closer to Marshal's trasparency, and (b) capable of preserving
value identity/pointer aliasing relationships, which is a
completely unknown concept to ASN.1, but a very useful feature of
Marshal.

   In summary my feeling is that unless ASN.1 is mandated for a
certiain application it's best avoided because it creates problems
of complexity and over-engineering without actually offering
very much.  (It would be nice to think that at least the
interoperability nightmares that used to arise because of everyone's
differently broken implementations of the "standard" are a thing of
the past nowadays, but I suspect they're not.)

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] extending a type with Marshal

[William Chesters]

William Chesters writes:
 >    I think it's only fair to point out that many people dislike ASN.1,

Oh, by the way, try http://www.google.com/search?q=asn.1%20rant if
you don't believe me.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

About ASN.1 and network encoding (was: Re: [Caml-list] extending a type with Marshal)

[David MENTRE]

Hello,

William Chesters <williamc@paneris.org> writes:

>    In summary my feeling is that unless ASN.1 is mandated for a
> certiain application it's best avoided because it creates problems
> of complexity and over-engineering without actually offering
> very much.

As a side note, for our own project which needs a client/server encoding
convention, I have decided to use XDR. It has some nice properties of
ASN.1: standardized (see corresponding IETF RFC); declarative message
description; automatic generation of encoder/decoder from/to OCaml data
structures. All of that without ASN.1's weight.

For OCaml, Gerd Stolpman has written a very nice XDR generator in his
RPC package.

Yours,
david
-- 
 David Mentré <dmentre@linux-france.org>
   http://www.linux-france.org/~dmentre/david-mentre-public-key.asc
 GnuPG key fingerprint: A7CD 7357 3EC4 1163 745B  7FD3 FB3E AD7C 2A18 BE9E

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners