GC hangs application

GC hangs application

[Paul Steckler]

I have an OCaml application that does a lot of string manipulations, generating a
lot of garbage.  After running awhile, it seems not to make any progress.  The exact
point where it hangs varies.  The strings aren't very large, but there a lot of them.

By increasing the size of the minor heap, the application runs much longer.
And if I turn on Gc.verbose, I see output from the garbage collector just before
the app hangs.  I mean, there's a print diagnostic in my code on one line, I see
the Gc output, and the print diagnostic on the next line of my code never appears.

The problem occurs whether I compile to byte-code or native (Linux x64) code.

At first, I thought  the hanging problem was due to network code blocking.  But I rewrote
that to be multithreaded -- no difference.  I changed some other code to use Buffers
instead of strings, and by generating less garbage, the code went farther.

Are there known issues with the OCaml gc?  I'm using OCaml 3.11 on Fedora 12.

-- Paul
--
Paul Steckler
National ICT Australia
paul DOT steckler AT nicta.com.au

The information in this e-mail may be confidential and subject to legal professional privilege and/or copyright. National ICT Australia Limited accepts no liability for any damage caused by this email or its attachments.

Re: [Caml-list] GC hangs application

[Goswin von Brederlow]

Paul Steckler <Paul.Steckler@nicta.com.au> writes:

> I have an OCaml application that does a lot of string manipulations, generating a
> lot of garbage.  After running awhile, it seems not to make any progress.  The exact
> point where it hangs varies.  The strings aren't very large, but there a lot of them.
>
> By increasing the size of the minor heap, the application runs much longer.
> And if I turn on Gc.verbose, I see output from the garbage collector just before
> the app hangs.  I mean, there's a print diagnostic in my code on one line, I see
> the Gc output, and the print diagnostic on the next line of my code never appears.
>
> The problem occurs whether I compile to byte-code or native (Linux x64) code.
>
> At first, I thought  the hanging problem was due to network code blocking.  But I rewrote
> that to be multithreaded -- no difference.  I changed some other code to use Buffers
> instead of strings, and by generating less garbage, the code went farther.
>
> Are there known issues with the OCaml gc?  I'm using OCaml 3.11 on Fedora 12.
>
> -- Paul

Since ocaml doesn't do concurrent multithreading you can still block the
app with one thread blocking. If you have a blocking function you need
to enter/leave_blocking_section() around it. The Unix.read/write stubs
do that already but if you have your own stubs they might need that too.

What does strace say?

MfG
        Goswin

Re: [Caml-list] GC hangs application

[Basile Starynkevitch]

On Wed, 2010-08-11 at 19:30 +1000, Paul Steckler wrote:
> I have an OCaml application that does a lot of string manipulations, generating a
> lot of garbage.  After running awhile, it seems not to make any progress.  The exact
> point where it hangs varies.  The strings aren't very large, but there a lot of them.

Is your application a well behaved purely Ocaml code, or do you have
Obj.magic or external functions written in C? Perhaps you are violating
some GC invariant.

You might still occasionally (i.e. every 1000 steps or requests, or
every few seconds) call Gc.compact.

> The information in this e-mail may be confidential and subject to legal professional privilege and/or copyright. National ICT Australia Limited accepts no liability for any damage caused by this email or its attachments.

I understand you don't customize that kind of [stupid company policy]
mail footer, but it is really inappropriate on a public & archived
mailing list. The usual advice is to use a public email account (e.g. on
gmail) to avoid that kind of nonsense. Nothing on a publicly archived
list can be confidential!

Regards.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mine, sont seulement les miennes} ***

Re: [Caml-list] GC hangs application

[Richard Jones]

On Wed, Aug 11, 2010 at 07:30:44PM +1000, Paul Steckler wrote:
> Are there known issues with the OCaml gc?

Are you exceeding available RAM?  I have observed the GC slowing to a
crawl once we start to swap.

If your strings are mainly read and rarely/never written, then using
ocaml-ancient is one potential solution to this.

> I'm using OCaml 3.11 on Fedora 12.

There are no specific issues with OCaml in Fedora 12 that I know of.

Rich.

-- 
Richard Jones
Red Hat