[Caml-list] memory mapped files with bigarray on windows

[Caml-list] memory mapped files with bigarray on windows

[Bauer, Christoph]

[-- Attachment #1: Type: text/plain, Size: 1373 bytes --]

Dear All,

my program maps  several blocks of a file into memory with Bigarray.Array1.map_file for writing.
Finally the file handle is closed with Unix.close.

On local files system everything works as expected. But if the file is on a network drive,
the resulting file is after the close still  locked [1] and it contains just zeros.

My workaround is, to call  Gc.full_major() before the close. Then all remaining bigarrays are
finalized (the finalizer calls caml_ba_unmap_file(), see bigarray_stubs.c).

Calling Gc.full_major () is somewhat dissatisfying. But I’m not sure how it could be solved more cleanly.
I would expect that Unix.close should unmap all remaining views of the file. But Unix.close knows nothing
about the bigarray mapping.

Maybe a special Bigarray.unmap is the solution.

Does anyone has a better idea?

Thanks,
Christoph Bauer

[1] You can’t delete it in the windows explorer.



-----------------
Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str. 10, 50823 Köln;
Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRA 28227;
Geschäftsführung und persönlich haftender Gesellschafter: Siemens Industry Software Management GmbH;
Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 70858

[-- Attachment #2: Type: text/html, Size: 3499 bytes --]

Re: [Caml-list] memory mapped files with bigarray on windows

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 2438 bytes --]

There were experiments adding an explicit "unmap this bigarray" operation
to the runtime: see Apr 2012 commit

https://github.com/ocaml/ocaml/commit/2330a3b30fe5b910f4776bf9c41f35ce2b980fb1
However, this change was reverted shortly thereafter because of performance
concerns: it changes the invariant that the size of a live bigarray never
changes, which makes it easier to optimize away bound checks.

Could you please create a bug report ( http://caml.inria.fr/mantis/ ) to
keep track of your particular issue with lack of explicit bigarray closure?
I haven't found traces of it in the bugtracker ( the rather old bug
http://caml.inria.fr/mantis/view.php?id=3571 is about the need to call
msync() in a NFS setting, but doesn't touch explicit closure ), so it may
not have been previously known.

On Thu, Sep 18, 2014 at 10:04 AM, Bauer, Christoph <
bauerchristoph@siemens.com> wrote:

>  Dear All,
>
>
>
> my program maps  several blocks of a file into memory with
> Bigarray.Array1.map_file for writing.
>
> Finally the file handle is closed with Unix.close.
>
>
>
> On local files system everything works as expected. But if the file is on
> a network drive,
>
> the resulting file is after the close still  locked [1] and it contains
> just zeros.
>
>
>
> My workaround is, to call  Gc.full_major() before the close. Then all
> remaining bigarrays are
>
> finalized (the finalizer calls caml_ba_unmap_file(), see
> bigarray_stubs.c).
>
>
>
> Calling Gc.full_major () is somewhat dissatisfying. But I’m not sure how
> it could be solved more cleanly.
>
> I would expect that Unix.close should unmap all remaining views of the
> file. But Unix.close knows nothing
>
> about the bigarray mapping.
>
>
>
> Maybe a special Bigarray.unmap is the solution.
>
>
>
> Does anyone has a better idea?
>
>
>
> Thanks,
>
> Christoph Bauer
>
>
>
> [1] You can’t delete it in the windows explorer.
>
>
>
>
>
> -----------------
> Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str. 10,
> 50823 Köln;
> Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht:
> Amtsgericht Köln, HRA 28227;
> Geschäftsführung und persönlich haftender Gesellschafter: Siemens Industry
> Software Management GmbH;
> Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln;
> Registergericht: Amtsgericht Köln, HRB 70858
>

[-- Attachment #2: Type: text/html, Size: 4185 bytes --]

Re: [Caml-list] memory mapped files with bigarray on windows

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 2914 bytes --]

Am Donnerstag, den 18.09.2014, 08:04 +0000 schrieb Bauer, Christoph:
> Dear All, 
> 
>  
> 
> my program maps  several blocks of a file into memory with
> Bigarray.Array1.map_file for writing.
> 
> Finally the file handle is closed with Unix.close.

You can close the file handle immediately after establishing the
mapping. The handle is only needed here as a reference of the file to
map, but it doesn't influence the way the file is written. In
particular, a close doesn't flush the contents to disk.

> On local files system everything works as expected.

Which is not guaranteed by POSIX, but normally works for all OS that
deeply integrate memory mapping. There is the msync() system call for
ensuring that all changes are finally reflected by the file.

>  But if the file is on a network drive,
> 
> the resulting file is after the close still  locked [1] and it
> contains just zeros.

Right, this is a known problem. Call msync(). I suggest it would be
easiest if msync() was part of the Bigarray module, but you can also
create your own wrapper.

Note that unmapping usually triggers the synchronization, but it doesn't
wait until the data is written (like calling msync() with the MS_ASYNC
flag).

> My workaround is, to call  Gc.full_major() before the close. Then all
> remaining bigarrays are
> 
> finalized (the finalizer calls caml_ba_unmap_file(), see
> bigarray_stubs.c).
> 

As noted, you cannot be sure that the data is really written at this
point. There is no way around msyncing the file.

Gerd

>  
> 
> Calling Gc.full_major () is somewhat dissatisfying. But I’m not sure
> how it could be solved more cleanly.
> 
> I would expect that Unix.close should unmap all remaining views of the
> file. But Unix.close knows nothing
> 
> about the bigarray mapping. 
> 
>  
> 
> Maybe a special Bigarray.unmap is the solution.
> 
>  
> 
> Does anyone has a better idea?
> 
>  
> 
> Thanks,
> 
> Christoph Bauer
> 
>  
> 
> [1] You can’t delete it in the windows explorer.
> 
>  
> 
>  
> 
> 
> -----------------
> Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str.
> 10, 50823 Köln;
> Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht:
> Amtsgericht Köln, HRA 28227;
> Geschäftsführung und persönlich haftender Gesellschafter: Siemens
> Industry Software Management GmbH;
> Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft:
> Köln; Registergericht: Amtsgericht Köln, HRB 70858
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

AW: [Caml-list] memory mapped files with bigarray on windows

[Bauer, Christoph]

Thanks for all the valuable answers.
I submitted a bug report: http://caml.inria.fr/mantis/view.php?id=6567

PR 3571 is somewhat different. The msync() is called also in the finalizer.
My problem was, that the finalizer was called too late (therefore the Gc.full_major ()).

To solve my problem I wrote as Gerd suggested my own Bigarray.Array1.flush.
It calls msync() on linux an FlushViewOfFile() on Windows.
I added the code to the bug  report.

Christoph Bauer



-----------------
Siemens Industry Software GmbH & Co. KG; Anschrift: Franz-Geuer-Str. 10, 50823 Köln;
Kommanditgesellschaft: Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRA 28227;
Geschäftsführung und persönlich haftender Gesellschafter: Siemens Industry Software Management GmbH;
Geschäftsführer: Urban August, Daniel Trebes; Sitz der Gesellschaft: Köln; Registergericht: Amtsgericht Köln, HRB 70858