[Caml-list] a bad value detector

[Caml-list] a bad value detector

[Frederic Perriot]

Hello caml-list,

I'd like to propose a detector to help in the detection of incorrect C
bindings that do not follow the GC rules.

The idea is rather simple:

1. after a minor collection, mprotect the pages of the minor heap to
disallow reads and writes
2. install a SEGV handler to catch the ensuing faults
3. if the faulting address is above caml_young_ptr - Max_young_whsize,
unprotect the page and carry on
4. otherwise, the program has no business accessing a value in the
unallocated part of the minor heap, so let it crash

I've hacked up a prototype that protects a single page at
caml_young_start, and it catches the bug I mention in my other message
entitled "an implicit GC rule".

Such a change surely degrades performance, but maybe it would be
useful as a runtime option available through CAMLRUNPARAM, to detect
misbehaved C bindings.

Does it sound like a viable technique?

I'm curious to hear what you think.

thanks,
Frédéric Perriot

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] a bad value detector

[Perry E. Metzger]

On Wed, 2 May 2018 18:19:39 +0200 Frederic Perriot
<fperriot@gmail.com> wrote:
> Hello caml-list,
> 
> I'd like to propose a detector to help in the detection of
> incorrect C bindings that do not follow the GC rules.

Once, I built a precise garbage collector for a language I'd
implemented in C. It was a simple Cheney collector with two
semispaces. I munmapped the unused semispace after each gc. In order
to debug C code that didn't properly register or protect GC roots, I
ran tests during every code change where I forced a collection after
every alloc -- this caused an almost immediate segfault in any C code
that was still pointing at the wrong semispace after the collection
because it would try to touch unmapped pages.

Generally I think tricks like this are a good idea. They allow one to
rapidly find code that doesn't correctly follow the rules at a fairly
low price. The only thing is you can only run such things while
testing because otherwise the performance hit is too high.

Perry

> 
> The idea is rather simple:
> 
> 1. after a minor collection, mprotect the pages of the minor heap to
> disallow reads and writes
> 2. install a SEGV handler to catch the ensuing faults
> 3. if the faulting address is above caml_young_ptr -
> Max_young_whsize, unprotect the page and carry on
> 4. otherwise, the program has no business accessing a value in the
> unallocated part of the minor heap, so let it crash
> 
> I've hacked up a prototype that protects a single page at
> caml_young_start, and it catches the bug I mention in my other
> message entitled "an implicit GC rule".
> 
> Such a change surely degrades performance, but maybe it would be
> useful as a runtime option available through CAMLRUNPARAM, to detect
> misbehaved C bindings.
> 
> Does it sound like a viable technique?
> 
> I'm curious to hear what you think.
> 
> thanks,
> Frédéric Perriot
> 



-- 
Perry E. Metzger		perry@piermont.com

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] a bad value detector

[Chet Murthy]

[-- Attachment #1: Type: text/plain, Size: 1802 bytes --]

If I remember correctly, a company once started with Boehm's conservative
GC, and by doing things somewhat like you suggest, built a
memory-leak-detector product.  So you're not alone in going down this sort
of path.

On Wed, May 2, 2018 at 9:19 AM, Frederic Perriot <fperriot@gmail.com> wrote:

> Hello caml-list,
>
> I'd like to propose a detector to help in the detection of incorrect C
> bindings that do not follow the GC rules.
>
> The idea is rather simple:
>
> 1. after a minor collection, mprotect the pages of the minor heap to
> disallow reads and writes
> 2. install a SEGV handler to catch the ensuing faults
> 3. if the faulting address is above caml_young_ptr - Max_young_whsize,
> unprotect the page and carry on
> 4. otherwise, the program has no business accessing a value in the
> unallocated part of the minor heap, so let it crash
>
> I've hacked up a prototype that protects a single page at
> caml_young_start, and it catches the bug I mention in my other message
> entitled "an implicit GC rule".
>
> Such a change surely degrades performance, but maybe it would be
> useful as a runtime option available through CAMLRUNPARAM, to detect
> misbehaved C bindings.
>
> Does it sound like a viable technique?
>
> I'm curious to hear what you think.
>
> thanks,
> Frédéric Perriot
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[-- Attachment #2: Type: text/html, Size: 2309 bytes --]

Re: [Caml-list] a bad value detector

[Frederic Perriot]

Perry E. Metzger  answered:
> Once, I built a precise garbage collector for a language I'd
> implemented in C. It was a simple Cheney collector with two
> semispaces. I munmapped the unused semispace after each gc.
> [...]

I only now realize your very trick is applicable to Ocaml's generational GC too:
just alternate the young generation between two virtual address ranges.

Then well-behaved programs should incur no page faults.

I'll give it a try, thanks.

Chet Murthy said:
>If I remember correctly, a company once started with Boehm's conservative GC, and by doing things somewhat like you suggest, built a memory-leak-detector product.

Interesting.

thank you guys,
regards,
FP

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

RE: [Caml-list] a bad value detector

[Stijn Devriendt]

> -----Original Message-----
> From: caml-list-request@inria.fr [mailto:caml-list-request@inria.fr] On Behalf
> Of Frederic Perriot
> Sent: woensdag 9 mei 2018 10:40
> To: Perry E. Metzger <perry@piermont.com>; Chet Murthy
> <murthy.chet@gmail.com>
> Cc: caml-list <caml-list@inria.fr>
> Subject: Re: [Caml-list] a bad value detector
> 
> Perry E. Metzger  answered:
> > Once, I built a precise garbage collector for a language I'd
> > implemented in C. It was a simple Cheney collector with two
> > semispaces. I munmapped the unused semispace after each gc.
> > [...]
> 
> I only now realize your very trick is applicable to Ocaml's generational GC too:
> just alternate the young generation between two virtual address ranges.
> 

I've been thinking about a similar scheme, but for a different purpose.
Currently it is to be expected that some of the memory allocated last on the minor
heap is not long-lived. Yet it will still be moved to the major heap on minor GC.

A scheme with 2 minor heaps that alternate could give those allocations the
time they need to become stale, avoiding cluttering the major heap with all
expected beneficial side-effects.

Regards,
Stijn

> Then well-behaved programs should incur no page faults.
> 
> I'll give it a try, thanks.
> 
> Chet Murthy said:
> >If I remember correctly, a company once started with Boehm's conservative
> GC, and by doing things somewhat like you suggest, built a memory-leak-
> detector product.
> 
> Interesting.
> 
> thank you guys,
> regards,
> FP
> 
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] a bad value detector

[Frederic Perriot]

On Fri, May 11, 2018 at 7:54 AM, Stijn Devriendt
<Stijn.Devriendt@wdc.com> wrote:

> A scheme with 2 minor heaps that alternate could give those allocations the
> time they need to become stale, avoiding cluttering the major heap with all
> expected beneficial side-effects.

Hello Stijn,

I believe there's already work in this direction, you may be interested in:

https://blog.janestreet.com/building-a-lower-latency-gc/

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs