[Caml-list] IPv6 packet parsing

[Caml-list] IPv6 packet parsing

[Johan Mazel]

[-- Attachment #1: Type: text/plain, Size: 308 bytes --]

Hi
I am aware of several implementation for IPv4 packet parsing (ocaml-packet,
melange, promiwag and maybe others that I missed).
However, up to my knowledge, none of these implementations offer IPv6
parsing functionnalities.
I would like to know if there is anything available ?
Thanks for your time.
Johan

[-- Attachment #2: Type: text/html, Size: 384 bytes --]

Re: [Caml-list] IPv6 packet parsing

[Stéphane Glondu]

Le 18/10/2013 06:15, Johan Mazel a écrit :
> I am aware of several implementation for IPv4 packet parsing
> (ocaml-packet, melange, promiwag and maybe others that I missed).
> However, up to my knowledge, none of these implementations offer IPv6
> parsing functionnalities.
> I would like to know if there is anything available ?

Using bitstring makes parsing binary structures such as IPv6 (and IPv4)
packets easy:

  https://github.com/glondu/netacct-crans/blob/master/netacct-crans.ml


Cheers,

-- 
Stéphane

Re: [Caml-list] IPv6 packet parsing

[rixed]

IPv6 is on the TODO list for robinet[1], a small lib that parse
network traffic (with the help of bitstring which was already
suggested). It's hard to tell if this small library fits your use
case but if it does I'd gladly add support for IPv6.

[1]: https://github.com/rixed/RobiNet

Re: [Caml-list] IPv6 packet parsing

[Anil Madhavapeddy]

On Fri, Oct 18, 2013 at 01:55:53PM +0200, rixed@happyleptic.org wrote:
> IPv6 is on the TODO list for robinet[1], a small lib that parse
> network traffic (with the help of bitstring which was already
> suggested). It's hard to tell if this small library fits your use
> case but if it does I'd gladly add support for IPv6.
> 
> [1]: https://github.com/rixed/RobiNet

One feature I'd really like to see in Bitstring is support for Bigarray,
since that avoids a copy into the OCaml heap and lets us do quite high
performance parsing.  If I remember right, there was a patch on the
Bitstring issue tracker, but it wasn't parameterised (so it's either
Bistring+string or Bitstring+bigarray, which isn't ideal).

-- 
Anil Madhavapeddy                                 http://anil.recoil.org

Re: [Caml-list] IPv6 packet parsing

[rixed]

-[ Fri, Oct 18, 2013 at 12:59:55PM +0100, Anil Madhavapeddy ]----
> One feature I'd really like to see in Bitstring is support for Bigarray,
> since that avoids a copy into the OCaml heap and lets us do quite high
> performance parsing.  If I remember right, there was a patch on the
> Bitstring issue tracker, but it wasn't parameterised (so it's either
> Bistring+string or Bitstring+bigarray, which isn't ideal).

Pardon my lack of familiarity with bigarrays, but I can't see what's the
difference between copying packets from pcap ring buffer into a bigarray
or into a string. Or do you mean using Bigarray.map_file on the whole
raw ring buffer and handle it without pcap help?

In my own experience, what slows down bitstring the most is the small
size of its internal buffers (128 bytes, ie. too small for many packet).
This simple change
(https://github.com/rixed/bitstring/commit/59076eb4c5ee1c8ed8648fc559c62e108b9c5037)
got me approx 20% improvement.

Re: [Caml-list] IPv6 packet parsing

[Anil Madhavapeddy]

On Fri, Oct 18, 2013 at 02:16:12PM +0200, rixed@happyleptic.org wrote:
> -[ Fri, Oct 18, 2013 at 12:59:55PM +0100, Anil Madhavapeddy ]----
> > One feature I'd really like to see in Bitstring is support for Bigarray,
> > since that avoids a copy into the OCaml heap and lets us do quite high
> > performance parsing.  If I remember right, there was a patch on the
> > Bitstring issue tracker, but it wasn't parameterised (so it's either
> > Bistring+string or Bitstring+bigarray, which isn't ideal).
> 
> Pardon my lack of familiarity with bigarrays, but I can't see what's the
> difference between copying packets from pcap ring buffer into a bigarray
> or into a string. Or do you mean using Bigarray.map_file on the whole
> raw ring buffer and handle it without pcap help?

We have a number of use-cases that run OCaml in kernel mode, directly
operating on packets read from a network driver that's also written in
OCaml.  Bigarrays are used as the mechanism for passing around externally
allocated memory (i.e. network card buffers) directly, whereas inspecting
them with a string-based Bigarray requires an expensive data copy.

See: http://anil.recoil.org/papers/2013-asplos-mirage.pdf
or http://www.openmirage.org

-- 
Anil Madhavapeddy                                 http://anil.recoil.org

Re: [Caml-list] IPv6 packet parsing

[rixed]

> We have a number of use-cases that run OCaml in kernel mode, directly
> operating on packets read from a network driver that's also written in
> OCaml.  Bigarrays are used as the mechanism for passing around externally
> allocated memory (i.e. network card buffers) directly, whereas inspecting
> them with a string-based Bigarray requires an expensive data copy.

Ok. We don't play in the same league :-)

Re: [Caml-list] IPv6 packet parsing

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 3342 bytes --]

Am Freitag, den 18.10.2013, 13:20 +0100 schrieb Anil Madhavapeddy:
> On Fri, Oct 18, 2013 at 02:16:12PM +0200, rixed@happyleptic.org wrote:
> > -[ Fri, Oct 18, 2013 at 12:59:55PM +0100, Anil Madhavapeddy ]----
> > > One feature I'd really like to see in Bitstring is support for Bigarray,
> > > since that avoids a copy into the OCaml heap and lets us do quite high
> > > performance parsing.  If I remember right, there was a patch on the
> > > Bitstring issue tracker, but it wasn't parameterised (so it's either
> > > Bistring+string or Bitstring+bigarray, which isn't ideal).
> > 
> > Pardon my lack of familiarity with bigarrays, but I can't see what's the
> > difference between copying packets from pcap ring buffer into a bigarray
> > or into a string. Or do you mean using Bigarray.map_file on the whole
> > raw ring buffer and handle it without pcap help?

Without knowing details: maybe no copy is required at all? The pcap ring
buffer could be directly wrapped as Bigarray.

> We have a number of use-cases that run OCaml in kernel mode, directly
> operating on packets read from a network driver that's also written in
> OCaml.  Bigarrays are used as the mechanism for passing around externally
> allocated memory (i.e. network card buffers) directly, whereas inspecting
> them with a string-based Bigarray requires an expensive data copy.
> 
> See: http://anil.recoil.org/papers/2013-asplos-mirage.pdf
> or http://www.openmirage.org

For similar reasons, I also added some Bigarray functions to Ocamlnet:

http://projects.camlcity.org/projects/dl/ocamlnet-3.7.3/doc/html-main/Netsys_mem.html

If you look at the stub behind e.g. Unix.read, you'll see that the data
is first read into an internal unaligned buffer, and then copied to the
string buffer. This means usually two copies of the data: one from the
kernel buffer to the internal buffer, and one from there to the string.

If you use a Bigarray instead the internal buffer becomes superfluous:
Bigarrays are malloc'ed memory, and cannot be moved by the GC. Hence,
you can invoke the read() syscall directly with the Bigarray as buffer.
If you additionally ensure that the Bigarray is page-aligned, the kernel
can sometimes even avoid copying at all (though only some OS seem to
implement such a strategy, as changing the page mapping or doing some
direct I/O can be more costly than copying).

Another advantage here is that you can freely choose the size of the
buffer (Unix.read et al use fixed-size 64K for the internal buffer).
Also you can allocate the buffer in a shared area.

Ocamlnet now prefers Bigarrays as primary buffers where reasonable, and
where a speedup (or lower CPU consumption) can be expected. E.g. The
HTTP client first reads data into a bigarray, splits the header there
into lines (which are then normal strings again), and gathers the data
chunks from the HTTP body (which can be strings or Bigarrays, at the
user's choice).

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Caml-list] IPv6 packet parsing

[rixed]

-[ Fri, Oct 18, 2013 at 03:52:07PM +0200, Gerd Stolpmann ]----
> > > Or do you mean using Bigarray.map_file on the whole
> > > raw ring buffer and handle it without pcap help?
> 
> Without knowing details: maybe no copy is required at all? The pcap ring
> buffer could be directly wrapped as Bigarray.

Yes that's what I was trying to say.
Unfortunately, such implementation would be hard to maintain
(Linux capture ring buffer have already several implementations, to say
nothing about supporting other OSes).
Ideally, I'd like to pass const pointers to packet bytes provided by
libpcap to user callbacks up to bitstring through OCaml (say via a
sufficiently large int type?).

Re: [Caml-list] IPv6 packet parsing

[Markus Mottl]

On Fri, Oct 18, 2013 at 8:20 AM, Anil Madhavapeddy <anil@recoil.org> wrote:
> We have a number of use-cases that run OCaml in kernel mode, directly
> operating on packets read from a network driver that's also written in
> OCaml.  Bigarrays are used as the mechanism for passing around externally
> allocated memory (i.e. network card buffers) directly, whereas inspecting
> them with a string-based Bigarray requires an expensive data copy.
>
> See: http://anil.recoil.org/papers/2013-asplos-mirage.pdf
> or http://www.openmirage.org

It may also be of interest to investigate the use of "splice" and
"tee" on Linux to avoid frequent switches between user- and kernel
mode and to duplicate I/O streams without having to copy the data
(zero copy).  There is an implementation in
Core_extended.Extended_Linux in the core_extended OPAM package.

Regards,
Markus

Re: [Caml-list] IPv6 packet parsing

[Stéphane Glondu]

Le 18/10/2013 16:13, rixed@happyleptic.org a écrit :
> Ideally, I'd like to pass const pointers to packet bytes provided by
> libpcap to user callbacks up to bitstring through OCaml [...]

AFAICT, this is what is done in my earlier link:

  https://github.com/glondu/netacct-crans/blob/master/netacct-crans.ml

rawstring in line 240 is a direct pointer to libpcap's buffer (in
particular, it is outside the OCaml heap). Of course, it being typed as
string is unsafe but it allows me to wrap it directly into a bitstring
(the "dark magic" that comes later) and then bitmatch-ed.


Cheers,

-- 
Stéphane

Re: [Caml-list] IPv6 packet parsing

[Paul Pelzl]

On Fri, Oct 18, 2013 at 9:20 AM, Markus Mottl <markus.mottl@gmail.com> wrote:
> It may also be of interest to investigate the use of "splice" and
> "tee" on Linux to avoid frequent switches between user- and kernel
> mode and to duplicate I/O streams without having to copy the data
> (zero copy).  There is an implementation in
> Core_extended.Extended_Linux in the core_extended OPAM package.

Before investigating too far and getting disappointed by mediocre
performance, it's worth pointing out that SPLICE_F_MOVE support got
pulled from the kernel quite some time ago [1] and still hasn't
returned [2]. It does look like someone is actively working on it,
however [3].

Paul

[1] https://lkml.org/lkml/2008/9/23/115
[2] http://lxr.free-electrons.com/ident?v=3.11;i=SPLICE_F_MOVE
[3] https://lkml.org/lkml/2013/10/7/532

Re: [Caml-list] IPv6 packet parsing

[rixed]

-[ Fri, Oct 18, 2013 at 10:20:19AM -0400, Markus Mottl ]----
> It may also be of interest to investigate the use of "splice" and
> "tee" on Linux to avoid frequent switches between user- and kernel
> mode and to duplicate I/O streams without having to copy the data
> (zero copy).

I feel sorry to be clueless twice in a row, but how are these new syscalls
supposed to help sniffing & parsing packets?

Re: [Caml-list] IPv6 packet parsing

[Markus Mottl]

On Fri, Oct 18, 2013 at 1:52 PM,  <rixed@happyleptic.org> wrote:
> I feel sorry to be clueless twice in a row, but how are these new syscalls
> supposed to help sniffing & parsing packets?

There are several ways in which they could be beneficial.  In fact,
you'll usually have to combine them.  Packet sniffers often need to
analyze some things in realtime while also logging all sniffed data to
a file for later reference.

"splice" allows you to move data from one descriptor to another
without ever having to copy it to user land in between.  "tee" allows
you to cheaply duplicate the content of one pipe to another.  E.g. you
can thus parse packets from one pipe for realtime analysis while
splicing duplicated (via "tee") data to a file.

The packet sniffer might also have another, non-verbatim output stream
that contains only filtered parts of the communication.  You could use
"vmsplice" to send data contained in your user buffers to a pipe
(without copying!) and then splice the data to a file while e.g. also
duplicating it to a socket, etc.

Anyway, I agree with Paul that such performance tweaks may often
disappoint.  Other things (e.g. batching of I/O transactions to
minimize system calls) are usually much more important.

Regards,
Markus

-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com

Re: [Caml-list] IPv6 packet parsing

[Johan Mazel]

[-- Attachment #1: Type: text/plain, Size: 175 bytes --]

I took a look at the code and I think it fits my use-case: I am parsing
packets to build statistics on them for measurements so I need to access
some (not all) packet fields.

[-- Attachment #2: Type: text/html, Size: 196 bytes --]

Re: [Caml-list] IPv6 packet parsing

[rixed]

> I took a look at the code and I think it fits my use-case: I am parsing
> packets to build statistics on them for measurements so I need to access
> some (not all) packet fields.

Ok, I'm going to add IPv6 then. Thank you for providing the motivation :)
I let you know in private when it's done (allow me a few days since I've other
things on the TODO list right now).

Re: [Caml-list] IPv6 packet parsing

[Johan Mazel]

[-- Attachment #1: Type: text/plain, Size: 530 bytes --]

Ok, thank you very much. :)
I am also interested by ICMPv6 if you have enough time. :)
Regards.
Johan


2013/10/21 <rixed@happyleptic.org>

> > I took a look at the code and I think it fits my use-case: I am parsing
> > packets to build statistics on them for measurements so I need to access
> > some (not all) packet fields.
>
> Ok, I'm going to add IPv6 then. Thank you for providing the motivation :)
> I let you know in private when it's done (allow me a few days since I've
> other
> things on the TODO list right now).
>
>

[-- Attachment #2: Type: text/html, Size: 929 bytes --]

Re: [Caml-list] IPv6 packet parsing

[Richard W.M. Jones]

On Fri, Oct 18, 2013 at 01:20:18PM +0100, Anil Madhavapeddy wrote:
> On Fri, Oct 18, 2013 at 02:16:12PM +0200, rixed@happyleptic.org wrote:
> > -[ Fri, Oct 18, 2013 at 12:59:55PM +0100, Anil Madhavapeddy ]----
> > > One feature I'd really like to see in Bitstring is support for Bigarray,
> > > since that avoids a copy into the OCaml heap and lets us do quite high
> > > performance parsing.  If I remember right, there was a patch on the
> > > Bitstring issue tracker, but it wasn't parameterised (so it's either
> > > Bistring+string or Bitstring+bigarray, which isn't ideal).
> > 
> > Pardon my lack of familiarity with bigarrays, but I can't see what's the
> > difference between copying packets from pcap ring buffer into a bigarray
> > or into a string. Or do you mean using Bigarray.map_file on the whole
> > raw ring buffer and handle it without pcap help?
> 
> We have a number of use-cases that run OCaml in kernel mode, directly
> operating on packets read from a network driver that's also written in
> OCaml.  Bigarrays are used as the mechanism for passing around externally
> allocated memory (i.e. network card buffers) directly, whereas inspecting
> them with a string-based Bigarray requires an expensive data copy.

I'm late to the party here, but could you just Obj.magic the bitstring
string pointer (ie. the first in the 3-tuple) to point at the Bigarray
data?

Note that bitstring doesn't really move the string around (well, I
don't want to claim that never happens, but it shouldn't happen in the
common cases AFAIK).

Rich.

-- 
Richard Jones
Red Hat