Safe Obj.magic container ?

Safe Obj.magic container ?

[David Teller]

   Along the lines of this discussion on Obj.magic, I have a question
about the semantics of the beast. 

 Assuming that I have two types t and u and a value v of type t, under
which conditions on t and u is it safe to Obj.magic v to type u, pass u
around a few times and Obj.magic it back to type t ? 


In particular
1. can I assume that this is always going to work if u is 'a.'a   ?

2. if t is a polymorphic variant written inline without a type name, can
I assume that if I write the same inline definition in another module,
it will have the same internal representation ?

3. assuming the answer to 1. is No, is there a type u or a simple
manipulation which should work for any type t ? Or perhaps just for
polymorphic variants ?

My guesses are 
1. No
2. Yes
3. Possibly if I restrict myself to polymorphic variants. Otherwise,
perhaps by playing with Deriving's Typeable.

Thanks,
 David
-- 
David Teller
 Security of Distributed Systems
  http://www.univ-orleans.fr/lifo/Members/David.Teller
 Angry researcher: French Universities need reforms, but the LRU act brings liquidations. 

Re: [Caml-list] Safe Obj.magic container ?

[Luc Maranget]

>    Along the lines of this discussion on Obj.magic, I have a question
> about the semantics of the beast. 

I have not followed the discussion, but here are a few clues

The semantics of Obj.magic is as follows:

1. Obj.magic does not exist. Don't use it.

2. Obj.magic is black magic, you can see it as the identity
   (fun x -> x)  with type 'a -> 'b.

As I see it, 2. steems from the uniform representation of values by OCaml.

A value is either a scalar, or a pointer, and this can be tested
dynamically (by looking at the low order bit) by
Obj.is_int/Obj.is_block.



-- 
Luc

Re: [Caml-list] Safe Obj.magic container ?

[Tom Primožič]

[-- Attachment #1: Type: text/plain, Size: 780 bytes --]

The following are my experience and guesses, nothing
scientific/official/thoroughly tested:

> 1. can I assume that this is always going to work if u is 'a.'a   ?


Well, there is no type 'a.'a. You can have:
type u = { everything: 'a . 'a }
but this presupposes that values of type u are blocks (heap-allocated).
Therefore, if you cast a boolean (true) to type u, and then try to access
the everything field of the "new" value, you will get a segfault.

3. assuming the answer to 1. is No, is there a type u or a simple
> manipulation which should work for any type t ? Or perhaps just for
> polymorphic variants ?
>

type u = int
this will work for any type. You can even do pointer manipulation with it
(adding and subtracting even numbers of bytes).

Sorry everybody.

 - Tom

[-- Attachment #2: Type: text/html, Size: 1219 bytes --]

Re: [Caml-list] Safe Obj.magic container ?

[David Teller]

On Wed, 2008-02-27 at 12:23 +0100, Tom Primožič wrote:
> The following are my experience and guesses, nothing
> scientific/official/thoroughly tested:
>         1. can I assume that this is always going to work if u is
>         'a.'a   ?
> 
> Well, there is no type 'a.'a. You can have:
> type u = { everything: 'a . 'a }

That's actually what I meant.

> but this presupposes that values of type u are blocks
> (heap-allocated). Therefore, if you cast a boolean (true) to type u,
> and then try to access the everything field of the "new" value, you
> will get a segfault.

Ok.

> 
>         3. assuming the answer to 1. is No, is there a type u or a
>         simple
>         manipulation which should work for any type t ? Or perhaps
>         just for
>         polymorphic variants ?
> 
> type u = int
> this will work for any type. You can even do pointer manipulation with
> it (adding and subtracting even numbers of bytes).
>
> Sorry everybody.

Er... I can't quite figure if it's a joke.

>  - Tom

Cheers,
 David

-- 
David Teller
 Security of Distributed Systems
  http://www.univ-orleans.fr/lifo/Members/David.Teller
 Angry researcher: French Universities need reforms, but the LRU act brings liquidations. 

Re: [Caml-list] Safe Obj.magic container ?

[Damien Doligez]

On 2008-02-27, at 10:43, Luc Maranget wrote:

> The semantics of Obj.magic is as follows:
>
> 1. Obj.magic does not exist. Don't use it.
>
> 2. Obj.magic is black magic, you can see it as the identity
>  (fun x -> x)  with type 'a -> 'b.
>
> As I see it, 2. steems from the uniform representation of values by  
> OCaml.
>
> A value is either a scalar, or a pointer, and this can be tested
> dynamically (by looking at the low order bit) by
> Obj.is_int/Obj.is_block.


If that was the whole story, Obj.magic would be pretty safe.  But
there are exceptions to the uniform representation of values, and
they make Obj.magic really hard to use.

The most important exception is floating-point numbers.  They are
not always represented as one word each, and that means even
Obj.repr is unsafe.

Example (on a 32-bit machine):

        Objective Caml version 3.10.2+rc1

# let x = Obj.repr 1;;
val x : Obj.t = <abstr>
# let y = Obj.repr 1.0;;
val y : Obj.t = <abstr>
# [| x; y |];;
- : Obj.t array = [|<abstr>; <abstr>|]
# [| y; x |];;
Bus error

My advice: don't use the Obj module if you were surprised by this
example.

-- Damien

Re: [Caml-list] Safe Obj.magic container ?

[David Teller]

Interesting. Can I assume that, if my type is boxed (in this case, a
polymorphic variant), I can successfully convert it to Obj.t and back ?

Thanks,
 David

On Thu, 2008-02-28 at 15:23 +0100, Damien Doligez wrote:
> If that was the whole story, Obj.magic would be pretty safe.  But
> there are exceptions to the uniform representation of values, and
> they make Obj.magic really hard to use.
> 
> The most important exception is floating-point numbers.  They are
> not always represented as one word each, and that means even
> Obj.repr is unsafe.

-- 
David Teller
 Security of Distributed Systems
  http://www.univ-orleans.fr/lifo/Members/David.Teller
 Angry researcher: French Universities need reforms, but the LRU act brings liquidations. 

Re: [Caml-list] Safe Obj.magic container ?

[Damien Doligez]

On 2008-02-28, at 16:29, David Teller wrote:

> Interesting. Can I assume that, if my type is boxed (in this case, a
> polymorphic variant), I can successfully convert it to Obj.t and  
> back ?


I don't know enough about the representation of polymorphic variants to
give an authoritative answer.

-- Damien

Re: [Caml-list] Safe Obj.magic container ?

[Jake Donham]

[-- Attachment #1: Type: text/plain, Size: 401 bytes --]

On Thu, Feb 28, 2008 at 6:23 AM, Damien Doligez <damien.doligez@inria.fr>
wrote:

> Example (on a 32-bit machine):
>
>        Objective Caml version 3.10.2+rc1
>
> # let x = Obj.repr 1;;
> val x : Obj.t = <abstr>
> # let y = Obj.repr 1.0;;
> val y : Obj.t = <abstr>
> # [| x; y |];;
> - : Obj.t array = [|<abstr>; <abstr>|]
> # [| y; x |];;
> Bus error


Could you please explain this behavior?

Jake

[-- Attachment #2: Type: text/html, Size: 721 bytes --]

Re: [Caml-list] Safe Obj.magic container ?

[Richard Jones]

On Thu, Feb 28, 2008 at 11:20:53AM -0800, Jake Donham wrote:
> On Thu, Feb 28, 2008 at 6:23 AM, Damien Doligez <damien.doligez@inria.fr>
> wrote:
> 
> > Example (on a 32-bit machine):
> >
> >        Objective Caml version 3.10.2+rc1
> >
> > # let x = Obj.repr 1;;
> > val x : Obj.t = <abstr>
> > # let y = Obj.repr 1.0;;
> > val y : Obj.t = <abstr>
> > # [| x; y |];;
> > - : Obj.t array = [|<abstr>; <abstr>|]
> > # [| y; x |];;
> > Bus error
> 
> 
> Could you please explain this behavior?

I _think_ this is what's going on:

Obj.repr is just the %identity primitive.  So 'x' and 'y' still look
like an int and a float respectively to the garbage collector.
ie. 'x' is still shifted left 1 place with the bottom bit set.  'y' is
still a pointer to a boxed 8 bit float (with a header on the heap
identifying it as such).

Constructing the array calls the caml_make_array primitive, which
looks at the tag of the first element to see if it's a float.  This
would cause the unboxing optimization.

Unfortunately if caml_make_array spots the possible unboxing of an
array of floats, then it blindly copies the remaining elements, which
dereferences the int (thinking it's a pointer to a float) and that
causes the segfault.

This also explains why it works the other way around, because having
the int first doesn't trigger the optimization.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Safe Obj.magic container ?

[Jacques Garrigue]

From: David Teller <David.Teller@univ-orleans.fr>

> Interesting. Can I assume that, if my type is boxed (in this case, a
> polymorphic variant), I can successfully convert it to Obj.t and back ?

Yes : polymorphic variants are either ints or normal blocks, so this
should work. This is also true for normal variants. Actually I don't
know of any non-uniform representation problem outside of floats.

Note however that this problem of non-uniform representation is no the
only danger when using Obj.magic imprudently. I recall another problem
with functional values. I'm afraid only Xavier could explain that one.

If you have only (polymorphic) variants, and if you keep the types
monomorphic (i.e. always add complete type annotations to Obj.magic or
Obj.obj), things should work properly in the current implementation.

Of course you should limit that kind of uses to things like
persistant storage or C interfaces, where you have to go through an
untyped world anyway, and avoid it at all costs in plain ml programs.

Jacques Garrigue

Re: [Caml-list] Safe Obj.magic container ?

[David Teller]

Thanks for the clarification. Again, I only have polymorphic variants,
no functions, no floats, no lazy values, etc. 

The reason why I need to go through some Obj conversion is essentially
the same reason safe marshalling needs to go through that conversion:
storing typed information on an untyped medium (or, in this case,
polymorphically typed information on a non-polymorphically typed medium)
-- along with a type certificate. It just happens that this
less-richly-typed medium is an exception.

Cheers,
 David

P.S.:
 I have tried alternative solutions but the best I can come up with
involves something like 30% slowdown on my testsuite. Which is a bit too
much for something which is supposed to be speed-critical.

On Fri, 2008-02-29 at 10:52 +0900, Jacques Garrigue wrote:
> From: David Teller <David.Teller@univ-orleans.fr>
> 
> > Interesting. Can I assume that, if my type is boxed (in this case, a
> > polymorphic variant), I can successfully convert it to Obj.t and back ?
> 
> Yes : polymorphic variants are either ints or normal blocks, so this
> should work. This is also true for normal variants. Actually I don't
> know of any non-uniform representation problem outside of floats.
> 
> Note however that this problem of non-uniform representation is no the
> only danger when using Obj.magic imprudently. I recall another problem
> with functional values. I'm afraid only Xavier could explain that one.
> 
> If you have only (polymorphic) variants, and if you keep the types
> monomorphic (i.e. always add complete type annotations to Obj.magic or
> Obj.obj), things should work properly in the current implementation.
> 
> Of course you should limit that kind of uses to things like
> persistant storage or C interfaces, where you have to go through an
> untyped world anyway, and avoid it at all costs in plain ml programs.
> 
> Jacques Garrigue
> 
-- 
David Teller
 Security of Distributed Systems
  http://www.univ-orleans.fr/lifo/Members/David.Teller
 Angry researcher: French Universities need reforms, but the LRU act brings liquidations.