Re: [Caml-list] appending data to a mmap-ed file

[Goswin von Brederlow <goswin-v-b@web.de>]

"Richard W.M. Jones" <rich@annexia.org> writes:

> On Thu, Dec 16, 2010 at 11:31:16AM +0000, Joel Reymont wrote:
>> I'm constantly appending to a file of stock quotes (ints, longs,
>> doubles, etc.). I have this file mapped into memory with mmap.
>>
>> What's the most efficient way to make newly appended data available
>> as part of the memory mapping?
>
> Unfortunately it's hard to reliably extend an mmap'd area.  The reason
> is not that you can't do it, but that you might overrun another memory
> mapping after it, where that other mapping could be something
> important like your program or a shared library.  The other mapping
> might not even be present at the time you initially map your file, but
> might appear as the result of an innocuous operation such as printing
> a string or allocating memory.
>
> Now you can, with a bunch of work, avoid this: parse /proc/self/maps,
> select a suitable base address for your mapping, move the mapping if
> it gets too large for the selected area or if another library is
> mapped in above it, etc. but this quickly gets very difficult.
>
> I would suggest a simpler way to solve your problem is simply to open
> the data file and append to it.  If you need to reference the values,
> keep them in memory structures.
>
> Rich.

Or avoid the whole issue and make the file large enough to begin
with. Thanks to sparse files you can create a huge file that only uses 1
block on disk. Then you can mmap that and it will use up more disk space
as you fill in data automatically.

MfG    Goswin