[Caml-list] Marshall.from_channel and segmentation fault

[Caml-list] Marshall.from_channel and segmentation fault

[Kenichi Asai]

The OCaml manual for the Marshall module says:

> (Marshal.from_channel chan : type).  Anything can happen at run-time
> if the object in the file does not belong to the given type.

and this "Anything" contains segmentation fault.  Is it difficult to
avoid this segmentation fault and, e.g., raise an exception instead?

Sincerely,

-- 
Kenichi Asai

Re: [Caml-list] Marshall.from_channel and segmentation fault

[Romain Bardou]

On 12/06/2015 07:27, Kenichi Asai wrote:
> The OCaml manual for the Marshall module says:
>
>> (Marshal.from_channel chan : type).  Anything can happen at run-time
>> if the object in the file does not belong to the given type.
>
> and this "Anything" contains segmentation fault.  Is it difficult to
> avoid this segmentation fault and, e.g., raise an exception instead?
>
> Sincerely,
>

You have to check that you will not dereference invalid pointers. 
Basically you need to type-check the value at runtime before using it. 
If you have a runtime representation of the type of your value, you may 
be able to do so using Obj. But if you have a runtime representation of 
the type, Marshal suddenly becomes less interesting as you can use this 
representation to guide serialization anyway.

Because of this, Marshal is mostly used when one knows through other 
means that only values of the right type are deserialized. This means 
that Marshal should not be used for network applications where the 
remote peer cannot be trusted to always send values of the right type. 
An attacker could send an ill-formed value to crash the server, for 
instance. Or, the remote peer may simply not be up-to-date and use other 
types for its values.

Marshal is better suited to saving data locally. It will still fail if 
one tries to use values from another application or another version of 
the same application with incompatible types. In other words it will not 
be backward compatible when types change, so Marshal is better suited 
for temporary files. For instance, .cmi files are marshaled values, if 
I'm not mistaken.

To sum up: yes, it is difficult.

Cheers,

-- 
Romain Bardou

Re: [Caml-list] Marshall.from_channel and segmentation fault

[Kenichi Asai]

Dear Romain,

Thanks for the explanation.  It seems you are suggesting that I should
use something else that performs runtime type checking rather than the
Marshall module.  Is there some standard module I can use?

I want to send various data along the Unix socket to create
interactive games using the Universe library for OCaml:

http://pllab.is.ocha.ac.jp/~asai/Universe/

Because the library is used by beginning students, I want to avoid
segmentation fault without any useful error messages when they
mistakenly use the sent data as having different types.

Sincerely,

-- 
Kenichi Asai


On Fri, Jun 12, 2015 at 09:39:39AM +0200,
 Romain Bardou wrote:

> On 12/06/2015 07:27, Kenichi Asai wrote:
> >The OCaml manual for the Marshall module says:
> >
> >>(Marshal.from_channel chan : type).  Anything can happen at run-time
> >>if the object in the file does not belong to the given type.
> >
> >and this "Anything" contains segmentation fault.  Is it difficult to
> >avoid this segmentation fault and, e.g., raise an exception instead?
> >
> >Sincerely,
> >
> 
> You have to check that you will not dereference invalid pointers. 
> Basically you need to type-check the value at runtime before using it. 
> If you have a runtime representation of the type of your value, you may 
> be able to do so using Obj. But if you have a runtime representation of 
> the type, Marshal suddenly becomes less interesting as you can use this 
> representation to guide serialization anyway.
> 
> Because of this, Marshal is mostly used when one knows through other 
> means that only values of the right type are deserialized. This means 
> that Marshal should not be used for network applications where the 
> remote peer cannot be trusted to always send values of the right type. 
> An attacker could send an ill-formed value to crash the server, for 
> instance. Or, the remote peer may simply not be up-to-date and use other 
> types for its values.
> 
> Marshal is better suited to saving data locally. It will still fail if 
> one tries to use values from another application or another version of 
> the same application with incompatible types. In other words it will not 
> be backward compatible when types change, so Marshal is better suited 
> for temporary files. For instance, .cmi files are marshaled values, if 
> I'm not mistaken.
> 
> To sum up: yes, it is difficult.
> 
> Cheers,

Re: [Caml-list] Marshall.from_channel and segmentation fault

[Francois Berenger]

On 06/12/2015 07:27 AM, Kenichi Asai wrote:
> The OCaml manual for the Marshall module says:
>
>> (Marshal.from_channel chan : type).  Anything can happen at run-time
>> if the object in the file does not belong to the given type.
>
> and this "Anything" contains segmentation fault.  Is it difficult to
> avoid this segmentation fault and, e.g., raise an exception instead?

Maybe bin_prot from janestreet is better for your use case.
I think it will throw an exception if you try to unmarshal something
of the wrong type.

> Sincerely,
>

-- 
Regards,
Francois.
"When in doubt, use more types"

Re: [Caml-list] Marshall.from_channel and segmentation fault

[Francois Berenger]

On 06/12/2015 09:39 AM, Romain Bardou wrote:
> On 12/06/2015 07:27, Kenichi Asai wrote:
>> The OCaml manual for the Marshall module says:
>>
>>> (Marshal.from_channel chan : type).  Anything can happen at run-time
>>> if the object in the file does not belong to the given type.
>>
>> and this "Anything" contains segmentation fault.  Is it difficult to
>> avoid this segmentation fault and, e.g., raise an exception instead?
>>
>> Sincerely,
>>
>
> You have to check that you will not dereference invalid pointers.
> Basically you need to type-check the value at runtime before using it.
> If you have a runtime representation of the type of your value, you may
> be able to do so using Obj. But if you have a runtime representation of
> the type, Marshal suddenly becomes less interesting as you can use this
> representation to guide serialization anyway.
>
> Because of this, Marshal is mostly used when one knows through other
> means that only values of the right type are deserialized. This means
> that Marshal should not be used for network applications where the
> remote peer cannot be trusted to always send values of the right type.
> An attacker could send an ill-formed value to crash the server,

You could first check a message authentication code before
trying to unmarshall.

Given that the secret key to cryptographically signed messages
is only known to authorized members of the communication group,
an attacker would not be able to inject messages into your system
without you detecting it and not trying to unmarshall from them.

 > for
> instance. Or, the remote peer may simply not be up-to-date and use other
> types for its values.
>
> Marshal is better suited to saving data locally. It will still fail if
> one tries to use values from another application or another version of
> the same application with incompatible types. In other words it will not
> be backward compatible when types change, so Marshal is better suited
> for temporary files. For instance, .cmi files are marshaled values, if
> I'm not mistaken.
>
> To sum up: yes, it is difficult.
>
> Cheers,
>

-- 
Regards,
Francois.
"When in doubt, use more types"

Re: [Caml-list] Marshall.from_channel and segmentation fault

[Romain Bardou]

If the game is simple and values can easily be seen as sequences of char 
/ bytes, 32-bit integers and fixed-length strings, your students could 
code their own serialization functions using the following functions 
from the Pervasives module:
- output_char / input_char
- output_byte / input_byte
- output_binary_int / input_binary_int
- output_string / really_input_string
Serialization is an important part of network applications, so it may be 
an interesting skill for your students to learn.

If on the other hand you feel that it would make it too complicated, 
there are several libraries which allow some form of serialization. I'm 
actually not that familiar with them as I tend to write serialization 
functions myself, but you may take a look at bin_prot, sexplib, biniou, 
yojson, piqi...

Some of them use human-readable protocols such as JSON or s-expressions. 
They will be easier for your students to debug, but they are less 
efficient, which may be an issue for a game.

Some of them have syntax extensions to generate the serialization 
functions from the source code of the type you want to serialize. Those 
will be the easiest to use. But I believe they tend to confuse students 
a lot, as the language is no longer pure OCaml and the build system 
needs to become aware of the syntax extension.

Cheers,

-- Romain Bardou

On 12/06/2015 10:26, Kenichi Asai wrote:
> Dear Romain,
>
> Thanks for the explanation.  It seems you are suggesting that I should
> use something else that performs runtime type checking rather than the
> Marshall module.  Is there some standard module I can use?
>
> I want to send various data along the Unix socket to create
> interactive games using the Universe library for OCaml:
>
> http://pllab.is.ocha.ac.jp/~asai/Universe/
>
> Because the library is used by beginning students, I want to avoid
> segmentation fault without any useful error messages when they
> mistakenly use the sent data as having different types.
>
> Sincerely,
>