Being aware of memory overuse

Being aware of memory overuse

[David Rajchenbach-Teller]

       Dear list,
 I'm writing some code that needs to be able to cope nicely with memory exhaustion. That is, it should be able to detect at some point if it is getting close to exhausting memory, and take the necessary course of action (e.g. bailing out nicely). I'm wondering what's the best way of doing this. I've been thinking about installing a [Gc.alarm] and checking [Gc.free_words], [Gc.free_blocks], [Gc.largest_free] just after each major cycle, but I'm wondering if it's the best technique.

Any suggestion?

Thanks in advance,
 David


-- 
David Rajchenbach-Teller
 Head of R&D
 MLstate

Re: Being aware of memory overuse

[Sylvain Le Gall]

On 05-02-2010, David Rajchenbach-Teller <David.Teller@mlstate.com> wrote:
>        Dear list,
>  I'm writing some code that needs to be able to cope nicely with
>  memory exhaustion. That is, it should be able to detect at some point
>  if it is getting close to exhausting memory, and take the necessary
>  course of action (e.g. bailing out nicely). I'm wondering what's the
>  best way of doing this. I've been thinking about installing a
>  [Gc.alarm] and checking [Gc.free_words], [Gc.free_blocks],
>  [Gc.largest_free] just after each major cycle, but I'm wondering if
>  it's the best technique.
>
> Any suggestion?
>

I worked on this subject, for a tool that need to use as much memory as
possible. I started by trying to use Gc module. I ended by reading
/proc/meminfo and getting/checking data from there. It is fast, simple
and very precise. In particular, it can take into account other
processes that take memory.

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: Being aware of memory overuse

[Goswin von Brederlow]

Sylvain Le Gall <sylvain@le-gall.net> writes:

> On 05-02-2010, David Rajchenbach-Teller <David.Teller@mlstate.com> wrote:
>>        Dear list,
>>  I'm writing some code that needs to be able to cope nicely with
>>  memory exhaustion. That is, it should be able to detect at some point
>>  if it is getting close to exhausting memory, and take the necessary
>>  course of action (e.g. bailing out nicely). I'm wondering what's the
>>  best way of doing this. I've been thinking about installing a
>>  [Gc.alarm] and checking [Gc.free_words], [Gc.free_blocks],
>>  [Gc.largest_free] just after each major cycle, but I'm wondering if
>>  it's the best technique.
>>
>> Any suggestion?
>>
>
> I worked on this subject, for a tool that need to use as much memory as
> possible. I started by trying to use Gc module. I ended by reading
> /proc/meminfo and getting/checking data from there. It is fast, simple
> and very precise. In particular, it can take into account other
> processes that take memory.
>
> Regards,
> Sylvain Le Gall

I always wanted to have kernel spport for this. Some way for aplication
to tell the kernel about freeable memory and for the kernel to request
some memory to be freeed instead of swapping it out.

This would be usefull for e.g. a web browser that caches images and
rendered pages in memory for quick redraws and many other applications.

MfG
        Goswin

Re: [Caml-list] Re: Being aware of memory overuse

[Dario Teixeira]

Hi,

> I always wanted to have kernel spport for this. Some way for aplication
> to tell the kernel about freeable memory and for the kernel to request
> some memory to be freeed instead of swapping it out.

If I recall correctly, there was an Lwn.net article reporting a lkml
(the Linux kernel mailing list) discussion on that subject.  One of
the proposals was for the kernel to send processes a signal (SIGFREE?)
requesting they free up memory (by running a major GC, for example)
whenever memory was running low.  In theory this could in some cases
avoid the invocation of the draconian OOM killer.

Question: just how effective such a feature would be in the Ocaml case?

Best regards,
Dario Teixeira



 

Re: Being aware of memory overuse

[Sylvain Le Gall]

On 10-02-2010, Dario Teixeira <darioteixeira@yahoo.com> wrote:
> Hi,
>
>> I always wanted to have kernel spport for this. Some way for aplication
>> to tell the kernel about freeable memory and for the kernel to request
>> some memory to be freeed instead of swapping it out.
>
> If I recall correctly, there was an Lwn.net article reporting a lkml
> (the Linux kernel mailing list) discussion on that subject.  One of
> the proposals was for the kernel to send processes a signal (SIGFREE?)
> requesting they free up memory (by running a major GC, for example)
> whenever memory was running low.  In theory this could in some cases
> avoid the invocation of the draconian OOM killer.
>
> Question: just how effective such a feature would be in the Ocaml case?
>
>

I forgot to tell about other paths I have followed: setrlimit/getrlimit.
There are limits like RLIMIT_DATA, RLIMIT_RSS et al (CPU...). setrlimit
limits memory by generating ENOMEM when limits are reached and getrlimit
should give information about the current level of use of these limits.
Unfortunately, the memory consumption I get with getrlimit is not
accurate (in fact not updated).

I have OCaml bindings for this, I should published in a near future. 

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: Being aware of memory overuse

[Goswin von Brederlow]

Dario Teixeira <darioteixeira@yahoo.com> writes:

> Hi,
>
>> I always wanted to have kernel spport for this. Some way for aplication
>> to tell the kernel about freeable memory and for the kernel to request
>> some memory to be freeed instead of swapping it out.
>
> If I recall correctly, there was an Lwn.net article reporting a lkml
> (the Linux kernel mailing list) discussion on that subject.  One of
> the proposals was for the kernel to send processes a signal (SIGFREE?)
> requesting they free up memory (by running a major GC, for example)
> whenever memory was running low.  In theory this could in some cases
> avoid the invocation of the draconian OOM killer.
>
> Question: just how effective such a feature would be in the Ocaml case?
>
> Best regards,
> Dario Teixeira

The danger of this on the other hand is that starting a mjor GC cycle
will most likely eat even more memory at first.

A signal would probably not be so helpfull. I was thinking more about a
memory preassure value. For example the kernel could provide a variable
in the vdso block saying:

0-100: You should shrink your memory usage to that many % of current
100+:  You could grow your memory to that many % of current

The kernel could consider the processes memory use, amount used for
caching in the process, other processes, general caching and the number
of hits to the pages to get a fair value for each process. and so on.

There could also be a SIGFREE signal to tell processes they must clean
up now or face the OOM killer but that would be secondary for me. I
don't ever want to see the OOM killer anyway.

MfG
        Goswin