Performance-question

Performance-question

[Oliver Bandel]

Hello,

I have a concrete (non philosophical) question
on the Performance of one of my scripts.

The general behaviour: it reads data fast first,
then slows down a lot (possibly after about 1/3 of the
data; it seems to completely have stopped then), then later speeds up
again.
Ths script needs nearly complete CPU-time and the system load goes up.


Now here some data:

=====================================================

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls   s/call   s/call  name
 44.45    163.08   163.08    26627     0.01     0.01  mark_slice
 22.31    244.92    81.84    38447     0.00     0.00  sweep_slice
 14.40    297.74    52.82      131     0.40     0.61  caml_compact_heap
  3.49    310.54    12.80 372416920     0.00     0.00  invert_pointer_at
  3.40    323.02    12.48 560604006     0.00     0.00  compact_allocate
  3.12    334.45    11.43  9725772     0.00     0.00  caml_fl_allocate
  0.92    337.81     3.36    25607     0.00     0.00  caml_add_to_heap

----------------------------------------------------------------------

minor_words: 218494242
promoted_words: 38303811
major_words: 3546606791
minor_collections: 64811
major_collections: 720
heap_words: 1044480
heap_chunks: 17
top_heap_words: 47322112
live_words: 666290
live_blocks: 103159
free_words: 378186
free_blocks: 360
largest_free: 60034
fragments: 4
compactions: 131

=====================================================


For me it looks like a lot of GC-actions, but I'm not sure in that
point. What do you think is slowing down here, and how to avoid
this behaviour?

Ciao,
   Oliver

Re: [Caml-list] Performance-question

[Jon Harrop]

On Tuesday 05 February 2008 13:33:53 Oliver Bandel wrote:
> For me it looks like a lot of GC-actions, but I'm not sure in that
> point. What do you think is slowing down here, and how to avoid
> this behaviour?

You need to run a memory profiler on your code to work out where it is 
allocating all of that data.

I've had the same problem and wrote my own memory profiler. I may 
commercialize it in the future...

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Performance-question

[Oliver Bandel]

Zitat von Jon Harrop <jon@ffconsultancy.com>:

> On Tuesday 05 February 2008 13:33:53 Oliver Bandel wrote:
> > For me it looks like a lot of GC-actions, but I'm not sure in that
> > point. What do you think is slowing down here, and how to avoid
> > this behaviour?
>
> You need to run a memory profiler on your code to work out where it
> is
> allocating all of that data.
[...]

Oh, I hope I can solve that problem otherwise.

I have some suspicious functions...
...possibly I should now switch to better datastructures/algorithms.

I hope I can solve it this way.

But thanks for your hint.

Ciao,
   Oliver

Re: [Caml-list] Performance-question

[Damien Doligez]

On 2008-02-05, at 14:33, Oliver Bandel wrote:

> Each sample counts as 0.01 seconds.
>  %   cumulative   self              self     total
> time   seconds   seconds    calls   s/call   s/call  name
> 44.45    163.08   163.08    26627     0.01     0.01  mark_slice
> 22.31    244.92    81.84    38447     0.00     0.00  sweep_slice
> 14.40    297.74    52.82      131     0.40     0.61  caml_compact_heap
>  3.49    310.54    12.80 372416920     0.00     0.00   
> invert_pointer_at
>  3.40    323.02    12.48 560604006     0.00     0.00  compact_allocate
>  3.12    334.45    11.43  9725772     0.00     0.00  caml_fl_allocate
>  0.92    337.81     3.36    25607     0.00     0.00  caml_add_to_heap

That's a lot of time spent in the compactor, indicating that your
program has a memory fragmentation problem.

I would be very interested in getting a copy of your program and data,
as I'm currently experimenting with new allocation policies designed
to reduce fragmentation.  This would make a good benchmark for me.

More generally, I'm looking for (small) programs that exhibit this
kind of behaviour: they call the compactor frequently and their memory
usage grows without bound when the compactor is disabled.

-- Damien

Re: [Caml-list] Performance-question

[Jon Harrop]

On Thursday 07 February 2008 16:13:25 Damien Doligez wrote:
> More generally, I'm looking for (small) programs that exhibit this
> kind of behaviour: they call the compactor frequently and their memory
> usage grows without bound when the compactor is disabled.

The "n"th-nearest neighbor example from OCaml for Scientists is certainly very 
allocation intensive because it makes extensive use of the Set module:

  http://www.ffconsultancy.com/products/ocaml_for_scientists/complete/

but I'm not sure that its memory use will grow unbounded if compaction is 
disabled though.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e