Re: [Caml-list] Early GC'ing

[oliver <oliver@first.in-berlin.de>]

On Mon, Aug 19, 2013 at 12:43:56AM -0600, Anthony Tavener wrote:
> What I was hinting at with Gc.full_major (), is that if you still had a
> large amount of memory allocated after calling that, I think that means
> your program is still holding on to the values somewhere.
> 
> In your loop, when you read in the data each time, is there any way
> something might leak? A hashtable holding reference to buffers? Or files
> left open?
[...]

I close the files after reading them and also select the data of the
file directly, evfore working on the next file.

For time-reasons I called the Gc-cleanup after 100 files;
I can try calling it immediately after each file, or after
a smaller number of files.

Maybe thats, why the Gc-call did not had that huge effect...
...but on average I would await that the needed size would be somewhat stable
after a while.


[...]
> Sorry I don't have much more help on this!
[...]

You already helped with the hint to the full-major cleanup.

I just tried to cleanup after every file.
The mem usage is about 30% after 19 minutes running time.
So Gc used that often consumes a lot of time...
If the decreased mem usage is from the effect of the Gc or
because it just needs longer until the program reaches
the 50% mem usage I don't know so far.

So, maybe I will just accept the mem usage.
There will be some overhead for storing the data
either way (not sure how much, possibly the used library
uses OOP and therefore has just some overhead).


As the used data is much more than the usual amount,
I think further optimization is not necessary.
I looked for a quick fix. Exploring in detail
would need more effort, and possibly the results will
not justify exploring in more depth.
(Or will do it if I have more time for it.)


Thanks so far for your support.

Ciao,
   Oliver

P.S.: Hmhh, at first look, cleaning up after 10 files seems to be a good
      compromise of speed and mem-usage.
      Maybe I can collect data from some more runs in a batched way to decide
      it...