[Caml-list] Marshalling objects (was: French interactive fiction, anyone ?)

[Caml-list] Marshalling objects (was: French interactive fiction, anyone ?)

[Berke Durak]

Well, I've written a very small prototype game, using
objects. Container objects are somewhat tricky (lots of weird typing
errors) but not impossible to define. However, if I can't save the
game state, I'll have to trash everything and ``simulate'' a class
model using conventional datatypes (not necessarily a bad thing, but I
would have been nice to use classical interactive fiction programming
style).

So, I have to ask : Will we be able to marshall objects soon ? There
seem to be some (unused) code fragments in extern.c...
--
Berke Durak
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Marshalling objects (was: French interactive fiction, anyone ?)

[Alan Schmitt]

* Berke Durak (berke@altern.org) wrote:
> Well, I've written a very small prototype game, using
> objects. Container objects are somewhat tricky (lots of weird typing
> errors) but not impossible to define. However, if I can't save the
> game state, I'll have to trash everything and ``simulate'' a class
> model using conventional datatypes (not necessarily a bad thing, but I
> would have been nice to use classical interactive fiction programming
> style).
> 
> So, I have to ask : Will we be able to marshall objects soon ? There
> seem to be some (unused) code fragments in extern.c...

Since in some sense objects point to their method code, you're asking
about the marshalling of code. Some work on this has been done and is
being done for JoCaml (see http://pauillac.inria.fr/jocaml/ ) and might
be some day integrated with ocaml.

Best,

Alan Schmitt

-- 
The hacker: someone who figured things out and made something cool happen.
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Marshalling objects (was: French interactive fiction, anyone ?)

[Berke Durak]

On Tue, Jun 18, 2002 at 08:57:23AM +0200, Alan Schmitt wrote:
> * Berke Durak (berke@altern.org) wrote:
> > Well, I've written a very small prototype game, using
> > objects. Container objects are somewhat tricky (lots of weird typing
> > errors) but not impossible to define. However, if I can't save the
> > game state, I'll have to trash everything and ``simulate'' a class
> > model using conventional datatypes (not necessarily a bad thing, but I
> > would have been nice to use classical interactive fiction programming
> > style).
> > 
> > So, I have to ask : Will we be able to marshall objects soon ? There
> > seem to be some (unused) code fragments in extern.c...
> 
> Since in some sense objects point to their method code, you're asking
> about the marshalling of code. Some work on this has been done and is
> being done for JoCaml (see http://pauillac.inria.fr/jocaml/ ) and might
> be some day integrated with ocaml.
> 
> Best,
> 
> Alan Schmitt

Well, this is not exactly about marshalling code : I'm not asking
Ocaml to be able to transfer pieces of code between different
programs. Just being able to save an object and reload it in the same
executable later would suffice : since Caml can save closures under
the same constraints, the reason why objects can't be saved must be
something else.
-- 
Berke Durak
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Marshalling objects (was: French interactive fiction, anyone ?)

[Jacques Garrigue]

From: Berke Durak <berke@altern.org>

> Well, this is not exactly about marshalling code : I'm not asking
> Ocaml to be able to transfer pieces of code between different
> programs. Just being able to save an object and reload it in the same
> executable later would suffice : since Caml can save closures under
> the same constraints, the reason why objects can't be saved must be
> something else.

Actually I cannot remember any such reason.
I actually tried, just commenting out the Object_tag case in
byterun/extern.c, and it works!
There's a single glitch: as it just handles objects as normal data,
oid's are not updated. This means that equality on objects (which is
oid based) will be incorrect.
Can somebody comment about other possible problems?
I join some test code at the end of this mail, to run after commenting
out Object_tag.

On the other hand, objects are not just closures, and it would be nice
to be able to serialize their data in a code-independent way. Not so
unreasonable: class names are unique. This would require a bit of
runtime support, to be able to retrieve method tables.

Jacques Garrigue

(* objext.ml *)
class c (n : int) =
  object (self)
    method private n = n
    method get_n = self#n
    method show = string_of_int self#n
  end

let () =
  let o =
    if Sys.file_exists "objext.dat" then
      let ic = open_in "objext.dat" in
      let o : c = input_value ic in
      close_in ic;
      Printf.printf "old value: %s\n" o#show;
      let o' = new c (o#get_n + 1) in
      Printf.printf "new value: %s\n" o'#show;
      Printf.printf "old %s new\n" (if o = o' then "=" else "<>");
      o'
    else begin
      print_endline "created objext.dat";
      new c 0
    end
  in
  let oc = open_out "objext.dat" in
  Marshal.to_channel oc o [Marshal.Closures];
  close_out oc
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Marshalling objects (was: French interactive fiction, anyone ?)

[Christopher Quinn]

Jacques Garrigue wrote:
> Actually I cannot remember any such reason.
> I actually tried, just commenting out the Object_tag case in
> byterun/extern.c, and it works!
> There's a single glitch: as it just handles objects as normal data,
> oid's are not updated. This means that equality on objects (which is
> oid based) will be incorrect.
> Can somebody comment about other possible problems?
> I join some test code at the end of this mail, to run after commenting
> out Object_tag.
> 
> On the other hand, objects are not just closures, and it would be nice
> to be able to serialize their data in a code-independent way. Not so
> unreasonable: class names are unique. This would require a bit of
> runtime support, to be able to retrieve method tables.
> 

Is not Field(obj,0) still a *pointer* to a double hash table, and so 
unique to a particular compilation?
I have a crummy type reflection module which exports this pointer so 
that by comparison of types at runtime an imported object created under 
another program can be amended with the current version of this pointer.
I have a vague memory that reflection is a theoretical no-go for ML. Is 
this true?

- chris

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Marshalling objects (was: French interactive fiction, anyone ?)

[Xavier Leroy]

> > since Caml can save closures under
> > the same constraints, the reason why objects can't be saved must be
> > something else.
> 
> Actually I cannot remember any such reason.
> I actually tried, just commenting out the Object_tag case in
> byterun/extern.c, and it works!

I'm always suspicious about "it works!" claims :-)

IIRC, one of the potential issues here is that methods are numbered at
program start-up time, via the Oo.new_method support function.  The
numbering of methods, then, determines the layout of vtables.  With
your solution, you're saving and reloading the vtable of the object as
a normal data structure.  This is correct only if the program that
saves the object and the program that reads it assign the same numbers
to methods.  But it isn't entirely obvious that two runs of the same
executable code on different data will assign the same numbers to
methods.  (Think e.g. of a conditionalized "let module" containing a
class declaration.)  It seems to work in the current compiler, but I
feel this scheme of dynamic allocation of method identifiers is quite
fragile.

Another potential issue is the un-sharing of vtables: in the present
system, all instances of a class share a common vtable; this is no
longer true with your marshaling/unmarshaling scheme.  Again, this
doesn't seem to break anything, but I'd like to be certain that
pointer equality on vtables is nowhere used.

> There's a single glitch: as it just handles objects as normal data,
> oid's are not updated. This means that equality on objects (which is
> oid based) will be incorrect.

This issue could be addressed by special-casing Object_tag in input_value,
and making the OID counter available from C.

> On the other hand, objects are not just closures, and it would be nice
> to be able to serialize their data in a code-independent way. Not so
> unreasonable: class names are unique.

Methinks you're confusing Caml with Java :-)  What kind of unique
class names do you have in mind?

- Xavier Leroy
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Marshalling objects (was: French interactive fiction, anyone ?)

[Jacques Garrigue]

Great, an implementer's discussion on the Caml list!
Where is Jerome ?!

From: Xavier Leroy <xavier.leroy@inria.fr>
> > > since Caml can save closures under
> > > the same constraints, the reason why objects can't be saved must be
> > > something else.
> > 
> > Actually I cannot remember any such reason.
> > I actually tried, just commenting out the Object_tag case in
> > byterun/extern.c, and it works!
> 
> I'm always suspicious about "it works!" claims :-)

The bang expressed an experimental truth, which can be reverted by
other experiments :-)

> IIRC, one of the potential issues here is that methods are numbered at
> program start-up time, via the Oo.new_method support function.  The
> numbering of methods, then, determines the layout of vtables.  With
> your solution, you're saving and reloading the vtable of the object as
> a normal data structure.  This is correct only if the program that
> saves the object and the program that reads it assign the same numbers
> to methods.

After a deeper look at -dlambda and oo.ml, this holds, but only as
long as you're not using let modules.  That is, public labels are
correctly lifted to the toplevel, but private labels may be generated
at class initialization. Since classes are toplevel structures inside
modules, class initializations occur in a fixed order as long as there
are no let modules.
But even that is easy to fix: private labels are local to a class, and
they need not be kept between runs. However public labels must not
change. The current scheme has one single counter for both kinds of
labels, but it should be easy to use two counters, putting public and
private methods in different buckets.

> Another potential issue is the un-sharing of vtables: in the present
> system, all instances of a class share a common vtable; this is no
> longer true with your marshaling/unmarshaling scheme.  Again, this
> doesn't seem to break anything, but I'd like to be certain that
> pointer equality on vtables is nowhere used.

It's not a scheme, it's inaction...
I should look more carefully, but I don't see any point in using pointer
equality on vtables, except for some kind of unspecified reflection.
Note that it does not guarantee type equality, as classes may be
parameterized.

> > There's a single glitch: as it just handles objects as normal data,
> > oid's are not updated. This means that equality on objects (which is
> > oid based) will be incorrect.
> 
> This issue could be addressed by special-casing Object_tag in input_value,
> and making the OID counter available from C.

Yes, the only subtlety is that the counter is on the caml side, so you
need a callback for that.

> > On the other hand, objects are not just closures, and it would be nice
> > to be able to serialize their data in a code-independent way. Not so
> > unreasonable: class names are unique.
> 
> Methinks you're confusing Caml with Java :-)  What kind of unique
> class names do you have in mind?

Since a class definition also defines a type, module semantics say that
every class name is unique inside its module. From that I thought we
could get the same thing as exception names: use the path. But maybe I
overlooked something?
Note that let modules would be a problem though.

        Jacques
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Marshalling objects (was: French interactive fiction, anyone ?)

[Jacques Garrigue]

Hi,

I've made a patch for the CVS version which seems to work.
(Note how this is stronger than "it works!")

It solves all the issues evocated in previous mails:
* let modules can change public method ids
  (I checked, and could easily obtain a segfault; it's a bit hard to
   trigger as you must have several compilation units)
* oids not updated

You can now marshal objects as data with closures, and both oids and
method ids are properly handled.
About sharing, well, inputed objects do not not share vtables with
internally created objects, but they share between themselves if they
are in a single data structure.

To ouput an object: Marshal.to_channel oc o [Marshal.Closures]
To input an object: input_value ic

Why a patch? It doesn't look too dangerous (no typing implied), but
I'm not sure whether it will go into CVS any time soon.

Independent problem: CamlinternalOO doesn't seem thread-safe: if two
threads create new objects simultaneously, how can we be sure that
they will have different ids?

Jacques Garrigue

Index: byterun/extern.c
===================================================================
RCS file: /net/pauillac/caml/repository/csl/byterun/extern.c,v
retrieving revision 1.41
diff -u -r1.41 extern.c
--- extern.c	2002/06/07 09:49:37	1.41
+++ extern.c	2002/06/21 08:50:38
@@ -328,9 +328,11 @@
     case Forward_tag:
       v = Forward_val (v);
       goto tailcall;
+/*
     case Object_tag:
       extern_invalid_argument("output_value: object value");
       break;
+*/
     case Custom_tag: {
       unsigned long sz_32, sz_64;
       char * ident = Custom_ops_val(v)->identifier;
Index: byterun/intern.c
===================================================================
RCS file: /net/pauillac/caml/repository/csl/byterun/intern.c,v
retrieving revision 1.48
diff -u -r1.48 intern.c
--- intern.c	2002/06/03 14:21:50	1.48
+++ intern.c	2002/06/21 08:50:38
@@ -26,6 +26,7 @@
 #include "mlvalues.h"
 #include "misc.h"
 #include "reverse.h"
+#include "callback.h"
 
 static unsigned char * intern_src;
 /* Reading pointer in block holding input data. */
@@ -137,6 +138,22 @@
         dest = (value *) (intern_dest + 1);
         *intern_dest = Make_header(size, tag, intern_color);
         intern_dest += 1 + size;
+        if (tag == Object_tag && size >= 2) {
+          static value *last_id = NULL;
+          value id0;
+          if (!last_id) {
+            last_id = caml_named_value ("CamlinternalOO.last_id");
+            if (!last_id)
+              fatal_error ("Cannot access CamlinternalOO.last_id from C");
+          }
+          id0 = Field(*last_id, 0);
+          Field(*last_id, 0) = id0 + 2; /* caml int */
+          intern_rec(dest++);
+          intern_rec(dest);
+          *dest++ = id0;
+          size = size - 2;
+          if (size == 0) return;
+        }
         for(/*nothing*/; size > 1; size--, dest++)
           intern_rec(dest);
         goto tailcall;
Index: stdlib/Makefile
===================================================================
RCS file: /net/pauillac/caml/repository/csl/stdlib/Makefile,v
retrieving revision 1.74
diff -u -r1.74 Makefile
--- Makefile	2002/06/05 12:08:38	1.74
+++ Makefile	2002/06/21 08:50:39
@@ -30,8 +30,8 @@
   set.cmo map.cmo stack.cmo queue.cmo stream.cmo buffer.cmo \
   printf.cmo format.cmo scanf.cmo \
   arg.cmo printexc.cmo gc.cmo \
-  digest.cmo random.cmo camlinternalOO.cmo oo.cmo \
-  genlex.cmo callback.cmo weak.cmo \
+  digest.cmo random.cmo callback.cmo camlinternalOO.cmo oo.cmo \
+  genlex.cmo weak.cmo \
   lazy.cmo filename.cmo int32.cmo int64.cmo nativeint.cmo complex.cmo
 LABELLED=arrayLabels.ml listLabels.ml stringLabels.ml moreLabels.ml
 
Index: stdlib/camlinternalOO.ml
===================================================================
RCS file: /net/pauillac/caml/repository/csl/stdlib/camlinternalOO.ml,v
retrieving revision 1.1
diff -u -r1.1 camlinternalOO.ml
--- camlinternalOO.ml	2002/04/24 09:49:06	1.1
+++ camlinternalOO.ml	2002/06/21 08:50:39
@@ -21,6 +21,9 @@
 let new_id () =
   let id = !last_id in incr last_id; id
 
+(* Used in intern.c *)
+let () = Callback.register "CamlinternalOO.last_id" last_id
+
 let set_id o id =
   let id0 = !id in
   Array.unsafe_set (Obj.magic o : int array) 1 id0;
@@ -57,6 +60,7 @@
 let first_bucket = 0
 let bucket_size = 32                    (* Must be 256 or less *)
 let initial_object_size = 2
+let label_kinds = 2        (* Alternate public and private labels *)
 
 (**** Index ****)
 
@@ -67,8 +71,9 @@
 let next label =
   incr label_count;
   let label = label + step in
+  (* alternate private and public buckets: skip one bucket here *)
   if label mod (step * bucket_size) = 0 then
-    label + step * (65536 - bucket_size)
+    label + step * (label_kinds * 65536 - bucket_size)
   else
     label
 
@@ -219,8 +224,14 @@
     Hashtbl.add methods met label;
     label
 
-let new_anonymous_method =
-  new_label
+(* Private labels are numbered independently *)
+let first_private_label = first_label + 65536 * step
+let last_private_label = ref first_private_label
+
+let new_anonymous_method () =
+  let label = !last_private_label in
+  last_private_label := next !last_private_label;
+  label
 
 (**** Types ****)
 
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners