Documentation request: allocation of bigarrays in C

Documentation request: allocation of bigarrays in C

[sayan]

Hi,

Please add the following fact to the Bigarray module documentation (I 
had to look at the c source header) and/or correct it as necessary :

A pointer allocated from C using malloc is not freed by the OCaml 
garbage collector. If you want to be GC-friendly, use the bigarray 
allocation macros with a NULL pointer to allocate memory then get a 
pointer to the data array.

Example:

Instead of
float* carray;
carray = malloc(sizeof(float)*size);
CAMLlocal1(res);
res = alloc_bigarray_dims(BIGARRAY_FLOAT32 | 
BIGARRAY_C_LAYOUT,1,carray,size);
CAMLreturn(res);

Write
CAMLlocal1(res);
res = alloc_bigarray_dims(BIGARRAY_FLOAT32 | BIGARRAY_C_LAYOUT,1,NULL,size);
float* carray = Data_bigarray_val(res);
CAMLreturn(res);

Regards,

-- 
Li-Thiao-Té Sébastien

Re: [Caml-list] Documentation request: allocation of bigarrays in C

[Dmitry Bely]

sayan@crans.org writes:

> Please add the following fact to the Bigarray module documentation (I
> had to look at the c source header) and/or correct it as necessary :
>
> A pointer allocated from C using malloc is not freed by the OCaml
> garbage collector.

Of course - it's not possible in general.

> If you want to be GC-friendly, use the bigarray
> allocation macros with a NULL pointer to allocate memory then get a
> pointer to the data array.

If you need GC-controlled bigarray, why simply not to use
Bigarray.Genarray.create?

- Dmitry Bely

Re: [Caml-list] Documentation request: allocation of bigarrays in C

[Li-Thiao-Té Sébastien]

Dmitry Bely wrote:
> sayan@crans.org writes:
> 
>>If you want to be GC-friendly, use the bigarray
>>allocation macros with a NULL pointer to allocate memory then get a
>>pointer to the data array.
> 
> If you need GC-controlled bigarray, why simply not to use
> Bigarray.Genarray.create?
> 
Why is there a C bigarray allocation function at all then ? :)

In my case, I want to read a relatively big file (50MB) containing 
arrays of floats . I use C functions for the raw speed then pass the 
bigarrays to OCaml to do some reordering of the data, and higher-level 
processing.

I do not open the file in Ocaml then send a string to C for parsing, but 
rather open the file in C and return the bigarray list to Ocaml. (cf my 
previous mistake on creating Ocaml lists in C on this mailing-list)

I have hundreds of such files that I may want to load and compare 
interactively, so reclaiming unused memory is a big bonus.

-- 
Li-Thiao-Té Sébastien

Re: [Caml-list] Documentation request: allocation of bigarrays in C

[Jonathan Roewen]

> In my case, I want to read a relatively big file (50MB) containing
> arrays of floats . I use C functions for the raw speed then pass the
> bigarrays to OCaml to do some reordering of the data, and higher-level
> processing.
>
> I do not open the file in Ocaml then send a string to C for parsing, but
> rather open the file in C and return the bigarray list to Ocaml. (cf my
> previous mistake on creating Ocaml lists in C on this mailing-list)

Does Bigarray.X.map_file not serve the purpose?

Kindest Regards,

Jonathan

Re: [Caml-list] Documentation request: allocation of bigarrays in C

[Li-Thiao-Té Sébastien]

Jonathan Roewen wrote:
>>In my case, I want to read a relatively big file (50MB) containing
>>arrays of floats . I use C functions for the raw speed then pass the
>>bigarrays to OCaml to do some reordering of the data, and higher-level
>>processing.
>>
>>I do not open the file in Ocaml then send a string to C for parsing, but
>>rather open the file in C and return the bigarray list to Ocaml. (cf my
>>previous mistake on creating Ocaml lists in C on this mailing-list)
> 
> Does Bigarray.X.map_file not serve the purpose?
> 
Not really. If I am not mistaken, map_file reads a file into a bigarray 
in binary mode. The file format is not as straightforward as that, I 
need to parse strings into floats and output a list of bigarrays 
embedded in a record with additional info. See the following excerpt :
1796.6221 3
1797.4133 250143
1798.0955 2401147
1799.1106 483314
1799.8252 192082
SPEC#: 2   SIZE: 140   TIME: 16.300000   MSLEVEL: 2  PRECURSOR: 531.552002
156.1387 7265
158.9650 1234
161.0317 2536
162.7167 5898
164.6776 12305
165.7509 4636
166.8494 64993
168.6152 35544
169.3650 1798

FYI, this is data from mass spectrometry experiments.

Regards,

-- 
Li-Thiao-Té Sébastien

Re: [Caml-list] Documentation request: allocation of bigarrays in C

[Dmitry Bely]

Li-Thiao-Té Sébastien <sayan@crans.org> writes:

>>>If you want to be GC-friendly, use the bigarray
>>>allocation macros with a NULL pointer to allocate memory then get a
>>>pointer to the data array.
>> If you need GC-controlled bigarray, why simply not to use
>> Bigarray.Genarray.create?
>>
> Why is there a C bigarray allocation function at all then ? :)

alloc_bigarray_xxx()? Their primary purpose is to allocate a proxy
structure for an external C-allocated data (that cannot be finalized).
data==NULL behaviour IMHO is not for the final user but only for
Bigarray.Genarray.create internal use.

> In my case, I want to read a relatively big file (50MB) containing
> arrays of floats . I use C functions for the raw speed then pass the
> bigarrays to OCaml to do some reordering of the data, and higher-level
> processing.

You should use Bigarray.Genarray.create then pass the created bigarray to C
side using Data_bigarray_val(). Note that the bigarray is allocated with
malloc() outside the Ocaml heap.

> I do not open the file in Ocaml then send a string to C for parsing,
> but rather open the file in C and return the bigarray list to
> Ocaml. (cf my previous mistake on creating Ocaml lists in C on this
> mailing-list)
>
> I have hundreds of such files that I may want to load and compare
> interactively, so reclaiming unused memory is a big bonus.

Sure - Ocaml-allocated bigarrays will be finalized as usual.

- Dmitry Bely

Re: [Caml-list] Documentation request: allocation of bigarrays in C

[Li-Thiao-Té Sébastien]

Dmitry Bely wrote:
> Li-Thiao-Té Sébastien <sayan@crans.org> writes:
> 
>>>>If you want to be GC-friendly, use the bigarray
>>>>allocation macros with a NULL pointer to allocate memory then get a
>>>>pointer to the data array.
>>>
>>>If you need GC-controlled bigarray, why simply not to use
>>>Bigarray.Genarray.create?
>>>
>>Why is there a C bigarray allocation function at all then ? :)
> 
> alloc_bigarray_xxx()? Their primary purpose is to allocate a proxy
> structure for an external C-allocated data (that cannot be finalized).
> data==NULL behaviour IMHO is not for the final user but only for
> Bigarray.Genarray.create internal use.
> 
>>In my case, I want to read a relatively big file (50MB) containing
>>arrays of floats . I use C functions for the raw speed then pass the
>>bigarrays to OCaml to do some reordering of the data, and higher-level
>>processing.
> 
> You should use Bigarray.Genarray.create then pass the created bigarray to C
> side using Data_bigarray_val(). Note that the bigarray is allocated with
> malloc() outside the Ocaml heap.
> 
I do not know in advance the size of the multiple bigarrays to allocate. 
This is determined by parsing the file. To speed up the process, the 
file is parsed on the C side, which is why I want to do GC-friendly 
memory allocation directly from C.
I do not believe that calling Bigarray.Genarray.create within a C parser 
is a nice way to do things, especially when there is the opportunity to 
use alloc_bigarray_xxx to allocate memory.
NB: I do not return a single bigarray to ocaml, but a list of bigarrays.

-- 
Li-Thiao-Té Sébastien

Re: [Caml-list] Documentation request: allocation of bigarrays in C

[Dmitry Bely]

Li-Thiao-Té Sébastien <sayan@crans.org> writes:

> I do not know in advance the size of the multiple bigarrays to
> allocate. This is determined by parsing the file. To speed up the
> process, the file is parsed on the C side, which is why I want to do
> GC-friendly memory allocation directly from C.
> I do not believe that calling Bigarray.Genarray.create within a C
> parser is a nice way to do things, especially when there is the
> opportunity to use alloc_bigarray_xxx to allocate memory.

OK, probably the NULL case from alloc_bigarray() comment

/* [alloc_bigarray] will allocate a new bigarray object in the heap.
   If [data] is NULL, the memory for the contents is also allocated
   (with [malloc]) by [alloc_bigarray].
   [data] cannot point into the Caml heap.
   [dim] may point into an object in the Caml heap.
*/

should also be mentioned in the docs. If you believe it's necessary, file
the bug report at http://caml.inria.fr/mantis.

> NB: I do not return a single bigarray to ocaml, but a list of bigarrays.

It's hard to imagine when it's really needed, but probably you have ample
grounds for such design.

- Dmitry Bely

Re: [Caml-list] Documentation request: allocation of bigarrays in C

[Nathaniel Gray]

On 4/2/06, Li-Thiao-Té Sébastien <sayan@crans.org> wrote:
> I do not know in advance the size of the multiple bigarrays to allocate.
> This is determined by parsing the file. To speed up the process, the
> file is parsed on the C side, which is why I want to do GC-friendly
> memory allocation directly from C.

Do you have evidence that the parsing would be significantly slower in
ocaml?  I'm not attacking you, I'm just curious to know if you've made
measurements.

Cheers,
-n8

--
>>>-- Nathaniel Gray -- Caltech Computer Science ------>
>>>-- Mojave Project -- http://mojave.cs.caltech.edu -->