[Caml-list] Operators for Int64 and Int32

[Caml-list] Operators for Int64 and Int32

[Michał Maciejewski]

Hello,

I'm not sure whether i should post this message here or in beginners
area (which is probably dead for a long time).

I'm quite new to OCaml but recently I had to write a wave file parsing
application. I've decided to try OCaml instead of C++. The first
problem I've encountered was of course too short native Integer type
for representing some values in wave file header. So I was forced to
use int64 type instead.

Why not to add real operators in another module for non-native types
like int64 or int32 to standard library? It's not too elegant to
write:

Int64.add (Int64.mul (Int64.add a b) b) c;;

especially when you try to avoid exceeding a limit of 79 characters in
line. Putting a part of expression in the next line isn't also a good
idea due to readability and conciseness of code. For me writing some
expressions in that way took even 5 too 7 lines. Opening Int64 could
help, but then it would be possible to write:

of_int someint;;

It's not clear what type someint is casted to.

Besides modifying application is very difficult now. Imagine that we
want to change type of a, b and c to int. We have to change previous
expression to (a+b) * b + c it can be very hard for more complex
expressions.

The best solutions to those problem would be in my opinion to add
something like this to standard library (to new module):

let ( +^^ ) a b = Int64.add a b
let ( -^^) a b = Int64.sub a b
let ( *^^ ) a b = Int64.mul a b
let ( /^^ ) a b = Int64.div a b
let ( !^^ ) a = Int64.neg a
let ( %^^ ) a b = Int64.sub a (Int64.mul (Int64.div a b) b)

let ( +^ ) a b = Int32.add a b
let ( -^ ) a b = Int32.sub a b
let ( *^ ) a b = Int32.mul a b
let ( /^ ) a b = Int32.div a b
let ( !^ ) a = Int32.neg a
let ( %^ ) a b = Int32.sub a (Int32.mul (Int32.div a b) b)

the same for big_int

Note that It also preserves backward compatibility of OCaml.

Best regards
Michal Maciejewski

RE: [Caml-list] Operators for Int64 and Int32

[David Allsopp]

I totally agree that expressions manipulating Int64 and Int32 quickly become
illegible, however I don't agree with your suggestions...

> Int64.add (Int64.mul (Int64.add a b) b) c;;
>
> especially when you try to avoid exceeding a limit of 79 characters in
> line. 

OT: May I politely suggest that perhaps your coding standards could do with
updating to mitigate part of this. Unless you're still on an 80x25 character
terminal limiting yourself to 79 characters per line is pretty archaic. We
use 150 character max line limit (and print in landscape if necessary)
because every display we work on has a resolution of at least 1280x1024.

> The best solutions to those problem would be in my opinion to add
> something like this to standard library (to new module):
>
> let ( +^^ ) a b = Int64.add a b
<snip>
>
> let ( +^ ) a b = Int32.add a b
<snip>

My problem with this, as someone who writes a lot of OCaml but uses Int64
and Int32 rarely, is that these operators aren't clearly anything to do with
Int64 or Int32 in terms of their "names" (symbols). Defining funny strings
of symbols to get around the (intentional) limitations of not having
operator overloading is IMHO not something that should be in the standard
library. If you're writing code that uses the operations so frequently that
clarity requires aliases then your code will be much more readable if you
have:

(* other, non-Int64/32 code *)

let ( +^^ ) a b = ...
and ( -^^ ) a b = ...
...
in
  (*
   * A block of code of reasonable length that makes extremely dense use of
   * the above operators.
   *)

(* Perhaps more non-Int64/32 code *)

That said, you could do it your way and my way with the aid of pa_openin
which allows you to open a module just within a block of code which would
save writing out the operators every time - for *your* code.

cout << "Adding lots of strange-looking operators (even in sub-modules \
that would need to be opened) to the StdLib is one step down the slippery \
slope to C(++) operator hell..."

Just my two pennyworth...


David

Re: [Caml-list] Operators for Int64 and Int32

[Bünzli Daniel]

Le 3 avr. 08 à 17:24, David Allsopp a écrit :

> OT: May I politely suggest that perhaps your coding standards could  
> do with
> updating to mitigate part of this. Unless you're still on an 80x25  
> character
> terminal limiting yourself to 79 characters per line is pretty  
> archaic. We
> use 150 character max line limit (and print in landscape if necessary)

Limiting oneself to 80 character per line is not an archaism. It has  
to do with typographic rules which were designed to cope with how our  
vision works, you'll find this in any book about typesetting. Going  
beyond ~80 characters, it becomes hard to quickly jump from one line  
to the other when you read (you need to use your thumb).

Best,

Daniel

Re: [Caml-list] Operators for Int64 and Int32

[Richard Jones]

On Thu, Apr 03, 2008 at 04:08:00PM +0200, Michał Maciejewski wrote:
> I'm quite new to OCaml but recently I had to write a wave file parsing
> application. I've decided to try OCaml instead of C++. The first
> problem I've encountered was of course too short native Integer type
> for representing some values in wave file header. So I was forced to
> use int64 type instead.

Yes, this is indeed an area where you have to be careful.  I'm writing
similar C-structure-parsing code and it is painful having to convert
between int/int32/int64 although of course there are good reasons for
it.

> The best solutions to those problem would be in my opinion to add
> something like this to standard library (to new module):
> 
> let ( +^^ ) a b = Int64.add a b
[etc.]

Yes, my code defines these operators too, although I picked slightly
different names for them, so perhaps it would be worth having these in
the stdlib if only because at least everyone would have the same name
for them :-)

Another thing you can do is to use functors to simplify writing code
that must work on int, int32 and int64 types.  eg take a basic
function like this:

  (* Check a value is in range 0 .. 2^bits-1. *)
  let range_unsigned v bits =
    let mask = lnot (mask bits) in
    (v land mask) = zero

The above function only works for ints.  But the same source code
could work for int32 as well, if you defined a few functions that you
need.  Note that the source of 'range_unsigned' is exactly the same
below as above:

  module I32 = struct
    let (<<) = Int32.shift_left
    let (>>) = Int32.shift_right_logical
    let (land) = Int32.logand
    let (lor) = Int32.logor
    let lnot = Int32.lognot
    let pred = Int32.pred
    let max_int = Int32.max_int
    let to_int = Int32.to_int
    let zero = Int32.zero
    let one = Int32.one
    let minus_one = Int32.minus_one
    let ff = 0xff_l
  
    (* Create a mask so many bits wide. *)
    let mask bits =
      if bits < 31 then
        pred (one << bits)
      else if bits = 31 then
        max_int
      else if bits = 32 then
        minus_one
      else
        invalid_arg "Bitmatch.I32.mask"

    (* Check a value is in range 0 .. 2^bits-1. *)
    let range_unsigned v bits =
      let mask = lnot (mask bits) in
      (v land mask) = zero
  end

(There's a few extra base functions in there because this example is
pulled from some real code which does the same thing over a collection
of bit-operating functions).

This can be converted into a functor relatively easily:

  module type IntegerType = sig
    type t
    val (<<) : t -> int -> t
    val (>>) : t -> int -> t
    val (land) : t -> t -> t
    val (lor) : t -> t -> t
    val lnot : t -> t
    val pred : t -> t
    val max_int : t
    val to_int : t -> int
    val zero : t
    val one : t
    val minus_one : t
    val ff : t
    val mask : int -> t
  end

  module type S = sig
    type t
    val range_unsigned : t -> int -> bool
  end

  module Make (I : IntegerType) : S with type t = I.t = struct
    include I
  
    (* Check a value is in range 0 .. 2^bits-1. *)
    let range_unsigned v bits =
      let mask = lnot (mask bits) in
      (v land mask) = zero
  end

And now you can use the functor to build int/int32/int64 versions of
the "range_unsigned" function automatically.  Of course this example
is a lot more worthwhile if you have a whole lot of these functions,
not just one :-)

  (* Make the module for int *)
  module I = Make (struct
    type t = int
    include Pervasives
    let (<<) = (lsl)
    let (>>) = (lsr)
    external to_int : int -> int = "%identity"
    let zero = 0
    let one = 1
    let minus_one = -1
    let ff = 0xff
  
    (* Create a mask so many bits wide. *)
    let mask bits =
      if bits < 30 then
        pred (one << bits)
      else if bits = 30 then
        max_int
      else if bits = 31 then
        minus_one
      else
        invalid_arg "Bitmatch.I.mask"
  end)

  (* Make the module for int32 *)
  module I32 = Make (struct
    include Int32
    let (<<) = Int32.shift_left
    let (>>) = Int32.shift_right_logical
    let (land) = Int32.logand
    let (lor) = Int32.logor
    let lnot = Int32.lognot
    let ff = 0xff_l
  
    (* Create a mask so many bits wide. *)
    let mask bits =
      if bits < 31 then
        pred (one << bits)
      else if bits = 31 then
        max_int
      else if bits = 32 then
        minus_one
      else
        invalid_arg "Bitmatch.I32.mask"
  end)

To check the functions have been defined:

  let () =
    ignore (I.range_unsigned 1 10);	(* the int version *)
    ignore (I32.range_unsigned 1_l 10); (* the int32 version *)

Rich.

-- 
Richard Jones
Red Hat

RE: [Caml-list] Operators for Int64 and Int32

[David Thomas]

For that matter, if the section is small enough, and
thus the declarations obviously associated, you could
even reuse +, -, *, etc.  Of course, this forbids
doing any operations on regular ints in that scope.

--- David Allsopp <dra-news@metastack.com> wrote:

> (* other, non-Int64/32 code *)
> 
> let ( +^^ ) a b = ...
> and ( -^^ ) a b = ...
> ...
> in
>   (*
>    * A block of code of reasonable length that makes
>    * extremely dense use of the above operators.
>    *)
> 
> (* Perhaps more non-Int64/32 code *)



      ____________________________________________________________________________________
You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost.  
http://tc.deals.yahoo.com/tc/blockbuster/text5.com

Re: [Caml-list] Operators for Int64 and Int32

[Michał Maciejewski]

2008/4/3, David Allsopp <dra-news@metastack.com>:
>
>  My problem with this, as someone who writes a lot of OCaml but uses Int64
>  and Int32 rarely, is that these operators aren't clearly anything to do with
>  Int64 or Int32 in terms of their "names" (symbols). Defining funny strings
>  of symbols to get around the (intentional) limitations of not having
>  operator overloading is IMHO not something that should be in the standard
>  library.
>

That was just an example, but I think that names of those operators
are as funny as the symbol  '@' . I don't see any relation between "@"
and lists. Just like between '!' and references, or '^' and strings.

In C you have the same operator '+' for ints and floats. OCaml
provides you with '+' and '+.' . So why not to have corresponding
operators for other types? It's better to have one rule than set of
rules to learn. Besides mathematical expressions in functional
languages should be IMHO as compact as it's possible.

Best regards,
Mlchal Maciejewski

Re: [Caml-list] Operators for Int64 and Int32

[Erik de Castro Lopo]

Michał Maciejewski wrote:

> I'm quite new to OCaml but recently I had to write a wave file parsing
> application.

If you need to read WAV files, why not just use libsndfile:

    http://www.mega-nerd.com/libsndfile/

and the libsndfile Ocaml wrapper:

    http://www.mega-nerd.com/tmp/libsndfile-ocaml-20071120-0622.tgz

It's basically working, but there might still be some slight
API changes.

The reason I suggest this is that WAV file parsing is probably
much, much harder than you immagine. Here's why:

 - WAV files allow the audio data to be encoded in dozens of
   different formants. Supporting even a small subset of them
   is a PITA.
 - There are numerous programs that create malformed files.
   Adding workarounds to these is a lot of work.
 - libsndfile already supports a whole bunch of other file
   formats.

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
With 22,100,000 legitimate businesses in the US alone,
allowing each to send only one UCE per *year* gets every
mailbox 60,547 emails per day. There will either be email
without UCE or there will be no email.

Re: [Caml-list] Operators for Int64 and Int32

[Michał Maciejewski]

2008/4/3, Erik de Castro Lopo <mle+ocaml@mega-nerd.com>:
> Michał Maciejewski wrote:
>
>  > I'm quite new to OCaml but recently I had to write a wave file parsing
>  > application.
>
>
> If you need to read WAV files, why not just use libsndfile:
>
>     http://www.mega-nerd.com/libsndfile/
>
>  and the libsndfile Ocaml wrapper:
>
>     http://www.mega-nerd.com/tmp/libsndfile-ocaml-20071120-0622.tgz
>

I have already finished the wave part, but thanks. I didn't know about
OCaml wrapper. Parsing a standard RIFF 8/16bit stereo/mono file isn't
so hard, actually.

Michal

Re: [Caml-list] Operators for Int64 and Int32

[Erik de Castro Lopo]

Michał Maciejewski wrote:

> I have already finished the wave part, but thanks. I didn't know about
> OCaml wrapper. Parsing a standard RIFF 8/16bit stereo/mono file isn't
> so hard, actually.

Exactly, parsing the most simple example is trivially easy.

However, being robust to malformed files, working around the broken-ness
where possible and supporting even a small subset of things other than
8/16 bit PCM is a huge task. From where you are now you will be tempted
to fix all these issues one by one. Down that path lines insanity.

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"If the thought of switching editors doesn't fill you with quite a bit
of dread, what you're using now is almost certainly underpowered, and
you definitely haven't customized it enough."
-- http://codeulate.com/?p=12

Re: [Caml-list] Operators for Int64 and Int32

[Richard Jones]

On Fri, Apr 04, 2008 at 07:17:29AM +1100, Erik de Castro Lopo wrote:
> If you need to read WAV files, why not just use libsndfile:
> 
>     http://www.mega-nerd.com/libsndfile/
> 
> and the libsndfile Ocaml wrapper:
> 
>     http://www.mega-nerd.com/tmp/libsndfile-ocaml-20071120-0622.tgz

I know that libsndfile is written in C, but I'd be interested to know
how much bitmatch
(http://et.redhat.com/~rjones/bitmatch/html/Bitmatch.html) might or
might not make a putative OCaml version easier to write.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Operators for Int64 and Int32

[Jon Harrop]

On Thursday 03 April 2008 20:50:40 Michał Maciejewski wrote:
> In C you have the same operator '+' for ints and floats. OCaml
> provides you with '+' and '+.' . So why not to have corresponding
> operators for other types?

That solution does not scale because you end up with an enormous number of 
different operators.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Operators for Int64 and Int32

[Erik de Castro Lopo]

Richard Jones wrote:

> I know that libsndfile is written in C, but I'd be interested to know
> how much bitmatch
> (http://et.redhat.com/~rjones/bitmatch/html/Bitmatch.html) might or
> might not make a putative OCaml version easier to write.

The hard bit has almost nothing to do with programming language
and almost everything to do with WAVE (and AIFF and all the other
formats) being extremely poorly and/or loosely specified, with
over 100 data encoding variants and dozens of broken implementations
in the field.

Very early in the libsndfile development process I came up with
a good way of packing and unpacking the headers (ie effectively 
the same  functionality as bitmatch) and everything has been 
built on top of that.

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"You can have quality software, or you can have pointer arithmetic;
but you cannot have both at the same time." -- Bertrand Meyer, 1989

Re: [Caml-list] Operators for Int64 and Int32

[Adrien]

2008/4/4, Erik de Castro Lopo <mle+ocaml@mega-nerd.com>:
> Richard Jones wrote:
>
> > I know that libsndfile is written in C, but I'd be interested to know
> > how much bitmatch
> > (http://et.redhat.com/~rjones/bitmatch/html/Bitmatch.html) might or
> > might not make a putative OCaml version easier to write.
>
> The hard bit has almost nothing to do with programming language
> and almost everything to do with WAVE (and AIFF and all the other
> formats) being extremely poorly and/or loosely specified, with
> over 100 data encoding variants and dozens of broken implementations
> in the field.
>
> Very early in the libsndfile development process I came up with
> a good way of packing and unpacking the headers (ie effectively
> the same  functionality as bitmatch) and everything has been
> built on top of that.
>
> Erik
>

I have to complain too about RIFF which is the format for the wave
container which is usually used for pcm. I, with two friends have been
working on a project involving sound and we needed to decode wave
files (to get their Fourier transform). I too have written a parsing
library. It took me some time to get the idea right but in the end, it
boiled down to six or seven lines of code.
However, writing the format description has been a real nightmare : I
found three different documents, started to implement from one which
turned out to be incomplete and incorrect, then moved on to the second
which too had problem (though less) and only the third, which was by
far the most complex of the three and complained about the format all
the time, gave me a proper reference.

But if you want the worst of the worst, have a look at the
edonkey-emule protocol. It's basically a reversed-engineered protocol
made for mldonkey, reused in emule, modified in the most inconsistent
way by tens of people pretending to keep backward compatibility with
clients that can't connect anymore to the network.
I tried to implement this but completely gave up after several days
without writing anything useful.
And by the way, I wanted to congratulate the mldonkey devels who
managed to implement these protocols (with the best algorithms too ;)
).


As for my library (I'm sure someone will ask about it), it is
basically used like Bitmatch. When I saw the Bitmatch release, I
translated Richard's examples to my library. I took me a few minutes.
It would benefit from some sugar but I don't know camlp4 (yet).

If you want you can have a look at the examples here :
http://ocaml.yaxm.org/wiki/doku.php?id=binary_parse#examples
I'll probably make some release this week-end if I find enough time (I
already have to get a new ID card as mine has just been stolen).


 ---

Adrien Nader

Re: [Caml-list] Operators for Int64 and Int32

[Richard Jones]

On Fri, Apr 04, 2008 at 07:58:00PM +0200, Adrien wrote:
> As for my library (I'm sure someone will ask about it), it is
> basically used like Bitmatch. When I saw the Bitmatch release, I
> translated Richard's examples to my library. I took me a few minutes.
> It would benefit from some sugar but I don't know camlp4 (yet).
> 
> If you want you can have a look at the examples here :
> http://ocaml.yaxm.org/wiki/doku.php?id=binary_parse#examples

I guess the advantage of bitmatch is matching against several cases.
In fact the pa_bitmatch extension generates a really large amount of
code in some cases.  The EXT3 case generates this:

http://www.annexia.org/tmp/ext3.txt

1814 lines of code from just 76 lines of source, ouch :-)  Well, most
of it is optional debugging statements ...

> I'll probably make some release this week-end if I find enough time (I
> already have to get a new ID card as mine has just been stolen).

But our Home Secretary "Wacky" Jacqui Smith assured us that we'll all
be safe with ID cards and all crime will cease!?!

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Operators for Int64 and Int32

[Adrien]

2008/4/4, Richard Jones <rich@annexia.org>:
> On Fri, Apr 04, 2008 at 07:58:00PM +0200, Adrien wrote:
>  > As for my library (I'm sure someone will ask about it), it is
>  > basically used like Bitmatch. When I saw the Bitmatch release, I
>  > translated Richard's examples to my library. I took me a few minutes.
>  > It would benefit from some sugar but I don't know camlp4 (yet).
>  >
>  > If you want you can have a look at the examples here :
>  > http://ocaml.yaxm.org/wiki/doku.php?id=binary_parse#examples
>
>
> I guess the advantage of bitmatch is matching against several cases.
>  In fact the pa_bitmatch extension generates a really large amount of
>  code in some cases.  The EXT3 case generates this:
>
>  http://www.annexia.org/tmp/ext3.txt
>
>  1814 lines of code from just 76 lines of source, ouch :-)  Well, most
>  of it is optional debugging statements ...

Well the approach is different as I'm not generating code but I'm
pretty sure our two libraries are similar in their use.
This is an excerpt from the wave parsing example :

  let rec parsing_list_of_chunk_id hashtbl=
    match Hashtbl.find hashtbl "next_chunk_id" with
      |S "fmt "->wav_format_chunk
      |S "data"->wav_data_chunk
      |_->assert false

  and type_of_riff_chunk hashtbl=
    match Hashtbl.find hashtbl "type" with
      |S "WAVE"->[input_string 4, "next_chunk_id", Some
(parsing_list_of_chunk_id)]
      |_->raise Unknown_type

Wave (1) has many is composed of chunks, among which "fmt " and
"data". The first function decides which chunk comes next according to
the ID written in the file and the second decides whether the riff
file contains a wave part or not (i.e. are we really parsing a wave
file) . So it is also possible to match against several cases.

(1) : in fact RIFF, the names were chosen when I did not really know
what belonged to riff and what belonged to wave
And btw, this has been enough to successfully parse tens of wave files
over the six past months. Fortunately most wave files are simple.

>  > I'll probably make some release this week-end if I find enough time (I
>  > already have to get a new ID card as mine has just been stolen).
>
>
> But our Home Secretary "Wacky" Jacqui Smith assured us that we'll all
>  be safe with ID cards and all crime will cease!?!

Don't forget the security cameras, except when they're pointed in the
other direction...


 ---

Adrien Nader