DSL for handling parsing binary memory structures?

DSL for handling parsing binary memory structures?

[Richard Jones]

I wonder if anyone has thought about or developed a domain-specific
language in OCaml for handling the processing of binary structures?

I was thinking of writing something to make system calls into the Xen
hypervisor.  The main problem is that the hypervisor takes & returns
binary structures which have very precise size / alignment
requirements, bitfields, etc. (and the requirements change with
versions of the hypervisor).  It sounds like the ideal use for a DSL
to describe the binary structure of each version of each call and
automatically build accessors.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] DSL for handling parsing binary memory structures?

[Joel Reymont]

Rich,

Take a look at the Erlang bit syntax:

http://tinyurl.com/35rup2

Could something like this be handled by camlp4?

	Joel

On May 17, 2007, at 10:21 PM, Richard Jones wrote:

>
> I wonder if anyone has thought about or developed a domain-specific
> language in OCaml for handling the processing of binary structures?

--
http://wagerlabs.com/

Re: [Caml-list] DSL for handling parsing binary memory structures?

[Gerd Stolpmann]

Am Donnerstag, den 17.05.2007, 22:21 +0100 schrieb Richard Jones:
> I wonder if anyone has thought about or developed a domain-specific
> language in OCaml for handling the processing of binary structures?
> 
> I was thinking of writing something to make system calls into the Xen
> hypervisor.  The main problem is that the hypervisor takes & returns
> binary structures which have very precise size / alignment
> requirements, bitfields, etc. (and the requirements change with
> versions of the hypervisor).  It sounds like the ideal use for a DSL
> to describe the binary structure of each version of each call and
> automatically build accessors.

Ocamlnet contains support for XDR, a simplified version of what you are
looking for (smallest unit are 4 bytes). But this might be a starting
point for this project.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] DSL for handling parsing binary memory structures?

[Jon Harrop]

On Thursday 17 May 2007 22:21:57 Richard Jones wrote:
> I was thinking of writing something to make system calls into the Xen
> hypervisor.  The main problem is that the hypervisor takes & returns
> binary structures which have very precise size / alignment
> requirements, bitfields, etc. (and the requirements change with
> versions of the hypervisor).  It sounds like the ideal use for a DSL
> to describe the binary structure of each version of each call and
> automatically build accessors.

I think XenSource sell a toolstack for Xen that is written in OCaml. So you 
might start there.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
The F#.NET Journal
http://www.ffconsultancy.com/products/fsharp_journal/?e

Re: [Caml-list] DSL for handling parsing binary memory structures?

[Christian Lindig]

If you are interested in the research aspect of this, Kathleen Fisher  
did interesting work on parsing ad-hoc data (mostly ascii):

http://www.research.att.com/~kfisher/hancock/release.php
http://www.research.att.com/~kfisher/hancock/release.php

I remember to have seen something like this for OCaml in the recent  
proceedings of ICFP or POPL but can't remember right now.

My own work in the context of calling conventions proposes an algebra  
to describe structures in memory. This could be a starting point for  
a DSL; it is also implemented in the Quick C-- compiler (in OCaml) in  
module block.nw.

http://www.st.cs.uni-sb.de/~lindig/papers/blocks/next.pdf
http://www.cminusminus.org/rsync/qc--/src/block.nw

-- Christian

Re: [Caml-list] DSL for handling parsing binary memory structures?

[Mike Furr]

Christian Lindig wrote:
> If you are interested in the research aspect of this, Kathleen Fisher
> did interesting work on parsing ad-hoc data (mostly ascii):
[...]
> I remember to have seen something like this for OCaml in the recent
> proceedings of ICFP or POPL but can't remember right now.

I believe you're thinking of PADS/ML (http://www.padsproj.org/) which
was presented at POPL'07 by Fisher et al.

Cheers,
-mike

Re: [Caml-list] DSL for handling parsing binary memory structures?

[Markus Mottl]

On 5/17/07, Richard Jones <rich@annexia.org> wrote:
> I wonder if anyone has thought about or developed a domain-specific
> language in OCaml for handling the processing of binary structures?
>
> I was thinking of writing something to make system calls into the Xen
> hypervisor.  The main problem is that the hypervisor takes & returns
> binary structures which have very precise size / alignment
> requirements, bitfields, etc. (and the requirements change with
> versions of the hypervisor).  It sounds like the ideal use for a DSL
> to describe the binary structure of each version of each call and
> automatically build accessors.

I'm currently developing a preprocessor that generates readers/writers
for binary protocols from OCaml type specifications.  It should be
very easy to add custom binary decoders by specifying an abstract type
for such values + custom readers/writer.  Or one can (ab)use the
existing ones to get the intended result (e.g. just use the int32 type
to read a 32bit integer from some memory loccation, etc.).

The protocol is very compact (even slightly more so than the
OCaml-marshal format), and performance is extremely high, usually even
higher than the standard marshal function, though it doesn't handle
shared/cyclic datastructures.  I also haven't looked into endianness
functionality yet (should be easy to add), but both 32bit and 64bit
platforms are supported.

In later stages there will also be a feature for a "protocol
handshake", which guarantees that e.g. the type specifications used by
two communicating machines are compatible.

We (Jane Street Capital) will eventually release this library as open
source so stay tuned...

Regards,
Markus

-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com

Re: [Caml-list] DSL for handling parsing binary memory structures?

[Richard Jones]

On Thu, May 17, 2007 at 10:21:57PM +0100, Richard Jones wrote:
> 
> I wonder if anyone has thought about or developed a domain-specific
> language in OCaml for handling the processing of binary structures?
> 
> I was thinking of writing something to make system calls into the Xen
> hypervisor.  The main problem is that the hypervisor takes & returns
> binary structures which have very precise size / alignment
> requirements, bitfields, etc. (and the requirements change with
> versions of the hypervisor).  It sounds like the ideal use for a DSL
> to describe the binary structure of each version of each call and
> automatically build accessors.

I had a play with some code to do this.  A write-up of my experiment
is here:

http://et.redhat.com/~rjones/hvcalls/

Rich.

-- 
Richard Jones
Red Hat