[Caml-list] Segmentation fault right after caml_alloc_custom

[Caml-list] Segmentation fault right after caml_alloc_custom

[Lin Hong]

[-- Attachment #1: Type: text/plain, Size: 1235 bytes --]

Hi all,

we have a weird problem here. In a c file, right after one call of 'caml_alloc_custom', some old information we have in memory is gone, then we have the segmentation fault because the code try to access those informations later.

 turn on debug message with OCAMLRUNPARAM, get this right before it crash.

......
$Starting new major GC cycle
Marking 9223372036854775807 words
Subphase = 10
Sweeping 9223372036854775807 words
Compacting heap...
Shrinking heap to 37696k bytes
Shrinking heap to 36704k bytes
Shrinking heap to 35712k bytes
Shrinking heap to 34720k bytes
Shrinking heap to 33728k bytes
Shrinking heap to 32736k bytes
Shrinking heap to 31744k bytes
Shrinking heap to 30752k bytes
done.

is it possible that the ocaml garbage collection free memory that is still in use?

the data cause this is kind of big, it doesn't happen right away,and it doesn't happen on every machine (linux with ocaml4 is fine, ocaml3.xx is not. mac with ocaml4 still shows the problem though).

any suggestion is appreciated.


Thanks,
Lin


=======================
American Museum of Natural History
POY website :
https://code.google.com/p/poy/
http://research.amnh.org/scicomp/scripts/download.php

[-- Attachment #2: Type: text/html, Size: 3635 bytes --]

AW: [Caml-list] Segmentation fault right after caml_alloc_custom

[Gerd Stolpmann]

You did not send code, so we can only speculate. A reason for this  
behavior could be that a value is not registered as local root, so the  
GC thinks it is unused.

That you see the problem only on some platforms can be explained by the  
different implementations of malloc, and how quickly freed memory is  
really deallocated (i.e. given back to the kernel). I wouldn't give  
much on this observation.

Gerd

Am 01.08.2012 22:05:19 schrieb(en) Lin Hong:
> Hi all,
> 
> we have a weird problem here. In a c file, right after one call of  
> 'caml_alloc_custom', some old information we have in memory is gone,  
> then we have the segmentation fault because the code try to access  
> those informations later.
> 
>  turn on debug message with OCAMLRUNPARAM, get this right before it  
> crash.
> 
> ......
> $Starting new major GC cycle
> Marking 9223372036854775807 words
> Subphase = 10
> Sweeping 9223372036854775807 words
> Compacting heap...
> Shrinking heap to 37696k bytes
> Shrinking heap to 36704k bytes
> Shrinking heap to 35712k bytes
> Shrinking heap to 34720k bytes
> Shrinking heap to 33728k bytes
> Shrinking heap to 32736k bytes
> Shrinking heap to 31744k bytes
> Shrinking heap to 30752k bytes
> done.
> 
> is it possible that the ocaml garbage collection free memory that is  
> still in use?
> 
> the data cause this is kind of big, it doesn't happen right away,and  
> it doesn't happen on every machine (linux with ocaml4 is fine,  
> ocaml3.xx is not. mac with ocaml4 still shows the problem though).
> 
> any suggestion is appreciated.
> 
> 
> Thanks,
> Lin
> 
> 
> =======================
> American Museum of Natural History
> POY website :
> https://code.google.com/p/poy/
> http://research.amnh.org/scicomp/scripts/download.php
> 
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

RE: [Caml-list] Segmentation fault right after caml_alloc_custom

[Lin Hong]

Hi Gerd,

thank you for the reply.

how do we know if some value is registered as local root or not?

the code is long so I will just paste some of it first

in that c file, the value we tried to access but failed was also allocated by caml_alloc_custom(ops, size, used, max).  

caml_alloc_custom (&sankoff_custom_operations_eltarr,sizeof (struct elt_arr)+sz, 1,alloc_custom_max);

the ops was custom_operations with all standard options as follow:

static struct custom_operations sankoff_custom_operations_eltarr = {
    "http://www.amnh.org/poy/",
    custom_finalize_default,
    custom_compare_default,
    custom_hash_default,
    custom_serialize_default, 
    custom_deserialize_default
};


the size in caml_alloc_custom is set to the full size , that is the data structure itself, plus everything its pointers point to.


we  also debug with valgrind,  the feed back suggest the memory we tried to access is indeed freed by caml gc 

==64563== Invalid read of size 4
==64563==    at 0x38477B: sankoff_median_3 (in ../poy.native)
==64563==    by 0x386125: sankoff_CAML_median_3 (in ../poy.native)
==64563==    by 0x13177D: camlSankCS__median_3_196 (in ../poy.native)
==64563==  Address 0x8bb0c04 is 81,108 bytes inside a block of size 512,016 free'd
==64563==    at 0xAFD7BF: free (vg_replace_malloc.c:325)
==64563==    by 0x3E5AC2: caml_free_for_heap (in ../poy.native)
==64563==    by 0x3E5C31: caml_shrink_heap (in ../poy.native)
==64563==    by 0x3F16FC: caml_compact_heap (in ../poy.native)
==64563==    by 0x3F1A89: caml_compact_heap_maybe (in ../poy.native)
==64563==    by 0x3E4EBC: caml_major_collection_slice (in ../poy.native)
==64563==    by 0x3E5692: caml_minor_collection (in ../poy.native)
==64563==    by 0x3E57E0: caml_check_urgent_gc (in ../poy.native)
==64563==    by 0x3F209C: caml_alloc_custom (in ../poy.native)
 


Cheers,
Lin Hong
American Museum of Natural History
POY website :
https://code.google.com/p/poy/
http://research.amnh.org/scicomp/scripts/download.php

________________________________________
From: Gerd Stolpmann [gerd@edgespring.com]
Sent: Wednesday, August 01, 2012 4:13 PM
To: Lin Hong
Cc: Mailing OCaML
Subject: Re: [Caml-list] Segmentation fault right after caml_alloc_custom

You did not send code, so we can only speculate. A reason for this
behavior could be that a value is not registered as local root, so the
GC thinks it is unused.

That you see the problem only on some platforms can be explained by the
different implementations of malloc, and how quickly freed memory is
really deallocated (i.e. given back to the kernel). I wouldn't give
much on this observation.

Gerd

Am 01.08.2012 22:05:19 schrieb(en) Lin Hong:
> Hi all,
>
> we have a weird problem here. In a c file, right after one call of
> 'caml_alloc_custom', some old information we have in memory is gone,
> then we have the segmentation fault because the code try to access
> those informations later.
>
>  turn on debug message with OCAMLRUNPARAM, get this right before it
> crash.
>
> ......
> $Starting new major GC cycle
> Marking 9223372036854775807 words
> Subphase = 10
> Sweeping 9223372036854775807 words
> Compacting heap...
> Shrinking heap to 37696k bytes
> Shrinking heap to 36704k bytes
> Shrinking heap to 35712k bytes
> Shrinking heap to 34720k bytes
> Shrinking heap to 33728k bytes
> Shrinking heap to 32736k bytes
> Shrinking heap to 31744k bytes
> Shrinking heap to 30752k bytes
> done.
>
> is it possible that the ocaml garbage collection free memory that is
> still in use?
>
> the data cause this is kind of big, it doesn't happen right away,and
> it doesn't happen on every machine (linux with ocaml4 is fine,
> ocaml3.xx is not. mac with ocaml4 still shows the problem though).
>
> any suggestion is appreciated.
>
>
> Thanks,
> Lin
>
>
> =======================
> American Museum of Natural History
> POY website :
> https://code.google.com/p/poy/
> http://research.amnh.org/scicomp/scripts/download.php
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

AW: [Caml-list] Segmentation fault right after caml_alloc_custom

[Gerd Stolpmann]

Am 01.08.2012 23:08:12 schrieb(en) Lin Hong:
> Hi Gerd,
> 
> thank you for the reply.
> 
> how do we know if some value is registered as local root or not?

Easy: It is your task as developer of a C external to register values,  
so if you don't do it, it is not done.

> the code is long so I will just paste some of it first
> 
> in that c file, the value we tried to access but failed was also  
> allocated by caml_alloc_custom(ops, size, used, max).
> 
> caml_alloc_custom (&sankoff_custom_operations_eltarr,sizeof (struct  
> elt_arr)+sz, 1,alloc_custom_max);
> 
> the ops was custom_operations with all standard options as follow:
> 
> static struct custom_operations sankoff_custom_operations_eltarr = {
>     "http://www.amnh.org/poy/",
>     custom_finalize_default,
>     custom_compare_default,
>     custom_hash_default,
>     custom_serialize_default,
>     custom_deserialize_default
> };
> 
> 
> the size in caml_alloc_custom is set to the full size , that is the  
> data structure itself, plus everything its pointers point to.

This is probably all ok. Did you use the CAMLparam and CAMLlocal macros?

Gerd


> 
> we  also debug with valgrind,  the feed back suggest the memory we  
> tried to access is indeed freed by caml gc
> 
> ==64563== Invalid read of size 4
> ==64563==    at 0x38477B: sankoff_median_3 (in ../poy.native)
> ==64563==    by 0x386125: sankoff_CAML_median_3 (in ../poy.native)
> ==64563==    by 0x13177D: camlSankCS__median_3_196 (in ../poy.native)
> ==64563==  Address 0x8bb0c04 is 81,108 bytes inside a block of size  
> 512,016 free'd
> ==64563==    at 0xAFD7BF: free (vg_replace_malloc.c:325)
> ==64563==    by 0x3E5AC2: caml_free_for_heap (in ../poy.native)
> ==64563==    by 0x3E5C31: caml_shrink_heap (in ../poy.native)
> ==64563==    by 0x3F16FC: caml_compact_heap (in ../poy.native)
> ==64563==    by 0x3F1A89: caml_compact_heap_maybe (in ../poy.native)
> ==64563==    by 0x3E4EBC: caml_major_collection_slice (in  
> ../poy.native)
> ==64563==    by 0x3E5692: caml_minor_collection (in ../poy.native)
> ==64563==    by 0x3E57E0: caml_check_urgent_gc (in ../poy.native)
> ==64563==    by 0x3F209C: caml_alloc_custom (in ../poy.native)
> 
> 
> 
> Cheers,
> Lin Hong
> American Museum of Natural History
> POY website :
> https://code.google.com/p/poy/
> http://research.amnh.org/scicomp/scripts/download.php
> 
> ________________________________________
> From: Gerd Stolpmann [gerd@edgespring.com]
> Sent: Wednesday, August 01, 2012 4:13 PM
> To: Lin Hong
> Cc: Mailing OCaML
> Subject: Re: [Caml-list] Segmentation fault right after  
> caml_alloc_custom
> 
> You did not send code, so we can only speculate. A reason for this
> behavior could be that a value is not registered as local root, so the
> GC thinks it is unused.
> 
> That you see the problem only on some platforms can be explained by  
> the
> different implementations of malloc, and how quickly freed memory is
> really deallocated (i.e. given back to the kernel). I wouldn't give
> much on this observation.
> 
> Gerd
> 
> Am 01.08.2012 22:05:19 schrieb(en) Lin Hong:
> > Hi all,
> >
> > we have a weird problem here. In a c file, right after one call of
> > 'caml_alloc_custom', some old information we have in memory is gone,
> > then we have the segmentation fault because the code try to access
> > those informations later.
> >
> >  turn on debug message with OCAMLRUNPARAM, get this right before it
> > crash.
> >
> > ......
> > $Starting new major GC cycle
> > Marking 9223372036854775807 words
> > Subphase = 10
> > Sweeping 9223372036854775807 words
> > Compacting heap...
> > Shrinking heap to 37696k bytes
> > Shrinking heap to 36704k bytes
> > Shrinking heap to 35712k bytes
> > Shrinking heap to 34720k bytes
> > Shrinking heap to 33728k bytes
> > Shrinking heap to 32736k bytes
> > Shrinking heap to 31744k bytes
> > Shrinking heap to 30752k bytes
> > done.
> >
> > is it possible that the ocaml garbage collection free memory that is
> > still in use?
> >
> > the data cause this is kind of big, it doesn't happen right away,and
> > it doesn't happen on every machine (linux with ocaml4 is fine,
> > ocaml3.xx is not. mac with ocaml4 still shows the problem though).
> >
> > any suggestion is appreciated.
> >
> >
> > Thanks,
> > Lin
> >
> >
> > =======================
> > American Museum of Natural History
> > POY website :
> > https://code.google.com/p/poy/
> > http://research.amnh.org/scicomp/scripts/download.php
> >
> > --
> > Caml-list mailing list.  Subscription management and archives:
> > https://sympa-roc.inria.fr/wws/info/caml-list
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
> >
> >
> 
> 
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

RE: [Caml-list] Segmentation fault right after caml_alloc_custom

[Lin Hong]

Cheers,
Lin Hong
American Museum of Natural History
POY website :
https://code.google.com/p/poy/
http://research.amnh.org/scicomp/scripts/download.php

________________________________________
From: caml-list-request@inria.fr [caml-list-request@inria.fr] on behalf of Gerd Stolpmann [info@gerd-stolpmann.de]
Sent: Wednesday, August 01, 2012 5:24 PM
To: Mailing OCaML
Subject: AW: [Caml-list] Segmentation fault right after caml_alloc_custom

Am 01.08.2012 23:08:12 schrieb(en) Lin Hong:
> Hi Gerd,
>
> thank you for the reply.
>
> how do we know if some value is registered as local root or not?

Easy: It is your task as developer of a C external to register values,
so if you don't do it, it is not done.

> the code is long so I will just paste some of it first
>
> in that c file, the value we tried to access but failed was also
> allocated by caml_alloc_custom(ops, size, used, max).
>
> caml_alloc_custom (&sankoff_custom_operations_eltarr,sizeof (struct
> elt_arr)+sz, 1,alloc_custom_max);
>
> the ops was custom_operations with all standard options as follow:
>
> static struct custom_operations sankoff_custom_operations_eltarr = {
>     "http://www.amnh.org/poy/",
>     custom_finalize_default,
>     custom_compare_default,
>     custom_hash_default,
>     custom_serialize_default,
>     custom_deserialize_default
> };
>
>
> the size in caml_alloc_custom is set to the full size , that is the
> data structure itself, plus everything its pointers point to.

This is probably all ok. Did you use the CAMLparam and CAMLlocal macros?


yes,  we do, just like following one

value
sankoff_CAML_median_3(value a, value n, value l, value r) {
   CAMLparam4(a,n,l,r);
   CAMLlocal1(res);

   //load a,n,l,r to eapA,eapN,eapL and eapR
   eltarr_p eapA;
   eltarr_p eapN;
   eltarr_p eapL;
   eltarr_p eapR;
   Sankoff_eltarr_custom_val(eapA,a);
   sankoff_init_eltarr(eapA,eapA->num_states,eapA->num_elts,-1,-1,-1,-1,NULL,0);
   Sankoff_eltarr_custom_val(eapN,n);
   sankoff_init_eltarr(eapN,eapN->num_states,eapN->num_elts,-1,-1,-1,-1,NULL,0);
   Sankoff_eltarr_custom_val(eapL,l);
   sankoff_init_eltarr(eapL,eapL->num_states,eapL->num_elts,-1,-1,-1,-1,NULL,0);
   Sankoff_eltarr_custom_val(eapR,r);
   sankoff_init_eltarr(eapR,eapR->num_states,eapR->num_elts,-1,-1,-1,-1,NULL,0);


//we can print everything in eapA~eapR here, they look fine

   //create a new one for return
   res = caml_alloc_custom (&sankoff_custom_operations_eltarr,sizeof (struct elt_arr)+sz, 1,alloc_custom_max);

//if we print eapA~eapR at this point again, they are totally crap.

   neweltarr = Sankoff_eltarr_pointer(res);
   sankoff_median_3(eapA,eapN,eapL,eapR,neweltarr);

   CAMLreturn(res);
}

some explanations on functions:

[Sankoff_eltarr_custom_val] is defined in head file like this:

#define Sankoff_eltarr_pointer(a) ((struct elt_arr *) Data_custom_val(a))
#define Sankoff_eltarr_custom_val(to_asgn,a) to_asgn = Sankoff_eltarr_pointer(a)

[sankoff_init_eltarr] set pointers to their right position, since we are loading the full chunk of memory. not sure it's necessary, but there is no harm to make sure those pointers are correct.



Gerd


>
> we  also debug with valgrind,  the feed back suggest the memory we
> tried to access is indeed freed by caml gc
>
> ==64563== Invalid read of size 4
> ==64563==    at 0x38477B: sankoff_median_3 (in ../poy.native)
> ==64563==    by 0x386125: sankoff_CAML_median_3 (in ../poy.native)
> ==64563==    by 0x13177D: camlSankCS__median_3_196 (in ../poy.native)
> ==64563==  Address 0x8bb0c04 is 81,108 bytes inside a block of size
> 512,016 free'd
> ==64563==    at 0xAFD7BF: free (vg_replace_malloc.c:325)
> ==64563==    by 0x3E5AC2: caml_free_for_heap (in ../poy.native)
> ==64563==    by 0x3E5C31: caml_shrink_heap (in ../poy.native)
> ==64563==    by 0x3F16FC: caml_compact_heap (in ../poy.native)
> ==64563==    by 0x3F1A89: caml_compact_heap_maybe (in ../poy.native)
> ==64563==    by 0x3E4EBC: caml_major_collection_slice (in
> ../poy.native)
> ==64563==    by 0x3E5692: caml_minor_collection (in ../poy.native)
> ==64563==    by 0x3E57E0: caml_check_urgent_gc (in ../poy.native)
> ==64563==    by 0x3F209C: caml_alloc_custom (in ../poy.native)
>
>
>
> Cheers,
> Lin Hong
> American Museum of Natural History
> POY website :
> https://code.google.com/p/poy/
> http://research.amnh.org/scicomp/scripts/download.php
>
> ________________________________________
> From: Gerd Stolpmann [gerd@edgespring.com]
> Sent: Wednesday, August 01, 2012 4:13 PM
> To: Lin Hong
> Cc: Mailing OCaML
> Subject: Re: [Caml-list] Segmentation fault right after
> caml_alloc_custom
>
> You did not send code, so we can only speculate. A reason for this
> behavior could be that a value is not registered as local root, so the
> GC thinks it is unused.
>
> That you see the problem only on some platforms can be explained by
> the
> different implementations of malloc, and how quickly freed memory is
> really deallocated (i.e. given back to the kernel). I wouldn't give
> much on this observation.
>
> Gerd
>
> Am 01.08.2012 22:05:19 schrieb(en) Lin Hong:
> > Hi all,
> >
> > we have a weird problem here. In a c file, right after one call of
> > 'caml_alloc_custom', some old information we have in memory is gone,
> > then we have the segmentation fault because the code try to access
> > those informations later.
> >
> >  turn on debug message with OCAMLRUNPARAM, get this right before it
> > crash.
> >
> > ......
> > $Starting new major GC cycle
> > Marking 9223372036854775807 words
> > Subphase = 10
> > Sweeping 9223372036854775807 words
> > Compacting heap...
> > Shrinking heap to 37696k bytes
> > Shrinking heap to 36704k bytes
> > Shrinking heap to 35712k bytes
> > Shrinking heap to 34720k bytes
> > Shrinking heap to 33728k bytes
> > Shrinking heap to 32736k bytes
> > Shrinking heap to 31744k bytes
> > Shrinking heap to 30752k bytes
> > done.
> >
> > is it possible that the ocaml garbage collection free memory that is
> > still in use?
> >
> > the data cause this is kind of big, it doesn't happen right away,and
> > it doesn't happen on every machine (linux with ocaml4 is fine,
> > ocaml3.xx is not. mac with ocaml4 still shows the problem though).
> >
> > any suggestion is appreciated.
> >
> >
> > Thanks,
> > Lin
> >
> >
> > =======================
> > American Museum of Natural History
> > POY website :
> > https://code.google.com/p/poy/
> > http://research.amnh.org/scicomp/scripts/download.php
> >
> > --
> > Caml-list mailing list.  Subscription management and archives:
> > https://sympa-roc.inria.fr/wws/info/caml-list
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
> >
> >
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>
>

--
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------
--
Caml-list mailing list.  Subscription management and archives:
https://sympa-roc.inria.fr/wws/info/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

AW: [Caml-list] Segmentation fault right after caml_alloc_custom

[Gerd Stolpmann]

Hmmm, this looks all correct. You mention that there are pointers  
inside eltarr. Note that memory allocated with caml_alloc_custom can be  
moved around by the GC. So if you have pointers in the custom block  
pointing to another part of the block, these will become meaningless  
after such a move.

The more standard solution is to put eltarr_p into the custom block,  
not eltarr as a whole.

Gerd

Am 01.08.2012 23:44:45 schrieb(en) Lin Hong:
> 
> Cheers,
> Lin Hong
> American Museum of Natural History
> POY website :
> https://code.google.com/p/poy/
> http://research.amnh.org/scicomp/scripts/download.php
> 
> ________________________________________
> From: caml-list-request@inria.fr [caml-list-request@inria.fr] on  
> behalf of Gerd Stolpmann [info@gerd-stolpmann.de]
> Sent: Wednesday, August 01, 2012 5:24 PM
> To: Mailing OCaML
> Subject: AW: [Caml-list] Segmentation fault right after  
> caml_alloc_custom
> 
> Am 01.08.2012 23:08:12 schrieb(en) Lin Hong:
> > Hi Gerd,
> >
> > thank you for the reply.
> >
> > how do we know if some value is registered as local root or not?
> 
> Easy: It is your task as developer of a C external to register values,
> so if you don't do it, it is not done.
> 
> > the code is long so I will just paste some of it first
> >
> > in that c file, the value we tried to access but failed was also
> > allocated by caml_alloc_custom(ops, size, used, max).
> >
> > caml_alloc_custom (&sankoff_custom_operations_eltarr,sizeof (struct
> > elt_arr)+sz, 1,alloc_custom_max);
> >
> > the ops was custom_operations with all standard options as follow:
> >
> > static struct custom_operations sankoff_custom_operations_eltarr = {
> >     "http://www.amnh.org/poy/",
> >     custom_finalize_default,
> >     custom_compare_default,
> >     custom_hash_default,
> >     custom_serialize_default,
> >     custom_deserialize_default
> > };
> >
> >
> > the size in caml_alloc_custom is set to the full size , that is the
> > data structure itself, plus everything its pointers point to.
> 
> This is probably all ok. Did you use the CAMLparam and CAMLlocal  
> macros?
> 
> 
> yes,  we do, just like following one
> 
> value
> sankoff_CAML_median_3(value a, value n, value l, value r) {
>    CAMLparam4(a,n,l,r);
>    CAMLlocal1(res);
> 
>    //load a,n,l,r to eapA,eapN,eapL and eapR
>    eltarr_p eapA;
>    eltarr_p eapN;
>    eltarr_p eapL;
>    eltarr_p eapR;
>    Sankoff_eltarr_custom_val(eapA,a);
>     
> sankoff_init_eltarr(eapA,eapA->num_states,eapA->num_elts,-1,-1,-1,-1,NULL,0);
>    Sankoff_eltarr_custom_val(eapN,n);
>     
> sankoff_init_eltarr(eapN,eapN->num_states,eapN->num_elts,-1,-1,-1,-1,NULL,0);
>    Sankoff_eltarr_custom_val(eapL,l);
>     
> sankoff_init_eltarr(eapL,eapL->num_states,eapL->num_elts,-1,-1,-1,-1,NULL,0);
>    Sankoff_eltarr_custom_val(eapR,r);
>     
> sankoff_init_eltarr(eapR,eapR->num_states,eapR->num_elts,-1,-1,-1,-1,NULL,0);
> 
> 
> //we can print everything in eapA~eapR here, they look fine
> 
>    //create a new one for return
>    res = caml_alloc_custom (&sankoff_custom_operations_eltarr,sizeof  
> (struct elt_arr)+sz, 1,alloc_custom_max);
> 
> //if we print eapA~eapR at this point again, they are totally crap.
> 
>    neweltarr = Sankoff_eltarr_pointer(res);
>    sankoff_median_3(eapA,eapN,eapL,eapR,neweltarr);
> 
>    CAMLreturn(res);
> }
> 
> some explanations on functions:
> 
> [Sankoff_eltarr_custom_val] is defined in head file like this:
> 
> #define Sankoff_eltarr_pointer(a) ((struct elt_arr *)  
> Data_custom_val(a))
> #define Sankoff_eltarr_custom_val(to_asgn,a) to_asgn =  
> Sankoff_eltarr_pointer(a)
> 
> [sankoff_init_eltarr] set pointers to their right position, since we  
> are loading the full chunk of memory. not sure it's necessary, but  
> there is no harm to make sure those pointers are correct.
> 
> 
> 
> Gerd
> 
> 
> >
> > we  also debug with valgrind,  the feed back suggest the memory we
> > tried to access is indeed freed by caml gc
> >
> > ==64563== Invalid read of size 4
> > ==64563==    at 0x38477B: sankoff_median_3 (in ../poy.native)
> > ==64563==    by 0x386125: sankoff_CAML_median_3 (in ../poy.native)
> > ==64563==    by 0x13177D: camlSankCS__median_3_196 (in  
> ../poy.native)
> > ==64563==  Address 0x8bb0c04 is 81,108 bytes inside a block of size
> > 512,016 free'd
> > ==64563==    at 0xAFD7BF: free (vg_replace_malloc.c:325)
> > ==64563==    by 0x3E5AC2: caml_free_for_heap (in ../poy.native)
> > ==64563==    by 0x3E5C31: caml_shrink_heap (in ../poy.native)
> > ==64563==    by 0x3F16FC: caml_compact_heap (in ../poy.native)
> > ==64563==    by 0x3F1A89: caml_compact_heap_maybe (in ../poy.native)
> > ==64563==    by 0x3E4EBC: caml_major_collection_slice (in
> > ../poy.native)
> > ==64563==    by 0x3E5692: caml_minor_collection (in ../poy.native)
> > ==64563==    by 0x3E57E0: caml_check_urgent_gc (in ../poy.native)
> > ==64563==    by 0x3F209C: caml_alloc_custom (in ../poy.native)
> >
> >
> >
> > Cheers,
> > Lin Hong
> > American Museum of Natural History
> > POY website :
> > https://code.google.com/p/poy/
> > http://research.amnh.org/scicomp/scripts/download.php
> >
> > ________________________________________
> > From: Gerd Stolpmann [gerd@edgespring.com]
> > Sent: Wednesday, August 01, 2012 4:13 PM
> > To: Lin Hong
> > Cc: Mailing OCaML
> > Subject: Re: [Caml-list] Segmentation fault right after
> > caml_alloc_custom
> >
> > You did not send code, so we can only speculate. A reason for this
> > behavior could be that a value is not registered as local root, so  
> the
> > GC thinks it is unused.
> >
> > That you see the problem only on some platforms can be explained by
> > the
> > different implementations of malloc, and how quickly freed memory is
> > really deallocated (i.e. given back to the kernel). I wouldn't give
> > much on this observation.
> >
> > Gerd
> >
> > Am 01.08.2012 22:05:19 schrieb(en) Lin Hong:
> > > Hi all,
> > >
> > > we have a weird problem here. In a c file, right after one call of
> > > 'caml_alloc_custom', some old information we have in memory is  
> gone,
> > > then we have the segmentation fault because the code try to access
> > > those informations later.
> > >
> > >  turn on debug message with OCAMLRUNPARAM, get this right before  
> it
> > > crash.
> > >
> > > ......
> > > $Starting new major GC cycle
> > > Marking 9223372036854775807 words
> > > Subphase = 10
> > > Sweeping 9223372036854775807 words
> > > Compacting heap...
> > > Shrinking heap to 37696k bytes
> > > Shrinking heap to 36704k bytes
> > > Shrinking heap to 35712k bytes
> > > Shrinking heap to 34720k bytes
> > > Shrinking heap to 33728k bytes
> > > Shrinking heap to 32736k bytes
> > > Shrinking heap to 31744k bytes
> > > Shrinking heap to 30752k bytes
> > > done.
> > >
> > > is it possible that the ocaml garbage collection free memory that  
> is
> > > still in use?
> > >
> > > the data cause this is kind of big, it doesn't happen right  
> away,and
> > > it doesn't happen on every machine (linux with ocaml4 is fine,
> > > ocaml3.xx is not. mac with ocaml4 still shows the problem though).
> > >
> > > any suggestion is appreciated.
> > >
> > >
> > > Thanks,
> > > Lin
> > >
> > >
> > > =======================
> > > American Museum of Natural History
> > > POY website :
> > > https://code.google.com/p/poy/
> > > http://research.amnh.org/scicomp/scripts/download.php
> > >
> > > --
> > > Caml-list mailing list.  Subscription management and archives:
> > > https://sympa-roc.inria.fr/wws/info/caml-list
> > > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > > Bug reports: http://caml.inria.fr/bin/caml-bugs
> > >
> > >
> >
> >
> > --
> > Caml-list mailing list.  Subscription management and archives:
> > https://sympa-roc.inria.fr/wws/info/caml-list
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
> >
> >
> >
> 
> --
> ------------------------------------------------------------
> Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
> Creator of GODI and camlcity.org.
> Contact details:        http://www.camlcity.org/contact.html
> Company homepage:       http://www.gerd-stolpmann.de
> ------------------------------------------------------------
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 
> 
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 
> 
> 



-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

RE: [Caml-list] Segmentation fault right after caml_alloc_custom

[Lin Hong]

if we reload custom block again after caml_alloc_custom for the new one, it went fine. 

looks like you are right. the free() call is just because GC is moving memory around. I will try allocate pointer only and see what happen.

thank you very much.


Cheers,
Lin Hong
American Museum of Natural History
POY website :
https://code.google.com/p/poy/
http://research.amnh.org/scicomp/scripts/download.php

________________________________________
From: Gerd Stolpmann [info@gerd-stolpmann.de]
Sent: Wednesday, August 01, 2012 6:11 PM
To: Lin Hong
Cc: Mailing OCaML
Subject: AW: [Caml-list] Segmentation fault right after caml_alloc_custom

Hmmm, this looks all correct. You mention that there are pointers
inside eltarr. Note that memory allocated with caml_alloc_custom can be
moved around by the GC. So if you have pointers in the custom block
pointing to another part of the block, these will become meaningless
after such a move.

The more standard solution is to put eltarr_p into the custom block,
not eltarr as a whole.

Gerd

Am 01.08.2012 23:44:45 schrieb(en) Lin Hong:
>
> Cheers,
> Lin Hong
> American Museum of Natural History
> POY website :
> https://code.google.com/p/poy/
> http://research.amnh.org/scicomp/scripts/download.php
>
> ________________________________________
> From: caml-list-request@inria.fr [caml-list-request@inria.fr] on
> behalf of Gerd Stolpmann [info@gerd-stolpmann.de]
> Sent: Wednesday, August 01, 2012 5:24 PM
> To: Mailing OCaML
> Subject: AW: [Caml-list] Segmentation fault right after
> caml_alloc_custom
>
> Am 01.08.2012 23:08:12 schrieb(en) Lin Hong:
> > Hi Gerd,
> >
> > thank you for the reply.
> >
> > how do we know if some value is registered as local root or not?
>
> Easy: It is your task as developer of a C external to register values,
> so if you don't do it, it is not done.
>
> > the code is long so I will just paste some of it first
> >
> > in that c file, the value we tried to access but failed was also
> > allocated by caml_alloc_custom(ops, size, used, max).
> >
> > caml_alloc_custom (&sankoff_custom_operations_eltarr,sizeof (struct
> > elt_arr)+sz, 1,alloc_custom_max);
> >
> > the ops was custom_operations with all standard options as follow:
> >
> > static struct custom_operations sankoff_custom_operations_eltarr = {
> >     "http://www.amnh.org/poy/",
> >     custom_finalize_default,
> >     custom_compare_default,
> >     custom_hash_default,
> >     custom_serialize_default,
> >     custom_deserialize_default
> > };
> >
> >
> > the size in caml_alloc_custom is set to the full size , that is the
> > data structure itself, plus everything its pointers point to.
>
> This is probably all ok. Did you use the CAMLparam and CAMLlocal
> macros?
>
>
> yes,  we do, just like following one
>
> value
> sankoff_CAML_median_3(value a, value n, value l, value r) {
>    CAMLparam4(a,n,l,r);
>    CAMLlocal1(res);
>
>    //load a,n,l,r to eapA,eapN,eapL and eapR
>    eltarr_p eapA;
>    eltarr_p eapN;
>    eltarr_p eapL;
>    eltarr_p eapR;
>    Sankoff_eltarr_custom_val(eapA,a);
>
> sankoff_init_eltarr(eapA,eapA->num_states,eapA->num_elts,-1,-1,-1,-1,NULL,0);
>    Sankoff_eltarr_custom_val(eapN,n);
>
> sankoff_init_eltarr(eapN,eapN->num_states,eapN->num_elts,-1,-1,-1,-1,NULL,0);
>    Sankoff_eltarr_custom_val(eapL,l);
>
> sankoff_init_eltarr(eapL,eapL->num_states,eapL->num_elts,-1,-1,-1,-1,NULL,0);
>    Sankoff_eltarr_custom_val(eapR,r);
>
> sankoff_init_eltarr(eapR,eapR->num_states,eapR->num_elts,-1,-1,-1,-1,NULL,0);
>
>
> //we can print everything in eapA~eapR here, they look fine
>
>    //create a new one for return
>    res = caml_alloc_custom (&sankoff_custom_operations_eltarr,sizeof
> (struct elt_arr)+sz, 1,alloc_custom_max);
>
> //if we print eapA~eapR at this point again, they are totally crap.
>
>    neweltarr = Sankoff_eltarr_pointer(res);
>    sankoff_median_3(eapA,eapN,eapL,eapR,neweltarr);
>
>    CAMLreturn(res);
> }
>
> some explanations on functions:
>
> [Sankoff_eltarr_custom_val] is defined in head file like this:
>
> #define Sankoff_eltarr_pointer(a) ((struct elt_arr *)
> Data_custom_val(a))
> #define Sankoff_eltarr_custom_val(to_asgn,a) to_asgn =
> Sankoff_eltarr_pointer(a)
>
> [sankoff_init_eltarr] set pointers to their right position, since we
> are loading the full chunk of memory. not sure it's necessary, but
> there is no harm to make sure those pointers are correct.
>
>
>
> Gerd
>
>
> >
> > we  also debug with valgrind,  the feed back suggest the memory we
> > tried to access is indeed freed by caml gc
> >
> > ==64563== Invalid read of size 4
> > ==64563==    at 0x38477B: sankoff_median_3 (in ../poy.native)
> > ==64563==    by 0x386125: sankoff_CAML_median_3 (in ../poy.native)
> > ==64563==    by 0x13177D: camlSankCS__median_3_196 (in
> ../poy.native)
> > ==64563==  Address 0x8bb0c04 is 81,108 bytes inside a block of size
> > 512,016 free'd
> > ==64563==    at 0xAFD7BF: free (vg_replace_malloc.c:325)
> > ==64563==    by 0x3E5AC2: caml_free_for_heap (in ../poy.native)
> > ==64563==    by 0x3E5C31: caml_shrink_heap (in ../poy.native)
> > ==64563==    by 0x3F16FC: caml_compact_heap (in ../poy.native)
> > ==64563==    by 0x3F1A89: caml_compact_heap_maybe (in ../poy.native)
> > ==64563==    by 0x3E4EBC: caml_major_collection_slice (in
> > ../poy.native)
> > ==64563==    by 0x3E5692: caml_minor_collection (in ../poy.native)
> > ==64563==    by 0x3E57E0: caml_check_urgent_gc (in ../poy.native)
> > ==64563==    by 0x3F209C: caml_alloc_custom (in ../poy.native)
> >
> >
> >
> > Cheers,
> > Lin Hong
> > American Museum of Natural History
> > POY website :
> > https://code.google.com/p/poy/
> > http://research.amnh.org/scicomp/scripts/download.php
> >
> > ________________________________________
> > From: Gerd Stolpmann [gerd@edgespring.com]
> > Sent: Wednesday, August 01, 2012 4:13 PM
> > To: Lin Hong
> > Cc: Mailing OCaML
> > Subject: Re: [Caml-list] Segmentation fault right after
> > caml_alloc_custom
> >
> > You did not send code, so we can only speculate. A reason for this
> > behavior could be that a value is not registered as local root, so
> the
> > GC thinks it is unused.
> >
> > That you see the problem only on some platforms can be explained by
> > the
> > different implementations of malloc, and how quickly freed memory is
> > really deallocated (i.e. given back to the kernel). I wouldn't give
> > much on this observation.
> >
> > Gerd
> >
> > Am 01.08.2012 22:05:19 schrieb(en) Lin Hong:
> > > Hi all,
> > >
> > > we have a weird problem here. In a c file, right after one call of
> > > 'caml_alloc_custom', some old information we have in memory is
> gone,
> > > then we have the segmentation fault because the code try to access
> > > those informations later.
> > >
> > >  turn on debug message with OCAMLRUNPARAM, get this right before
> it
> > > crash.
> > >
> > > ......
> > > $Starting new major GC cycle
> > > Marking 9223372036854775807 words
> > > Subphase = 10
> > > Sweeping 9223372036854775807 words
> > > Compacting heap...
> > > Shrinking heap to 37696k bytes
> > > Shrinking heap to 36704k bytes
> > > Shrinking heap to 35712k bytes
> > > Shrinking heap to 34720k bytes
> > > Shrinking heap to 33728k bytes
> > > Shrinking heap to 32736k bytes
> > > Shrinking heap to 31744k bytes
> > > Shrinking heap to 30752k bytes
> > > done.
> > >
> > > is it possible that the ocaml garbage collection free memory that
> is
> > > still in use?
> > >
> > > the data cause this is kind of big, it doesn't happen right
> away,and
> > > it doesn't happen on every machine (linux with ocaml4 is fine,
> > > ocaml3.xx is not. mac with ocaml4 still shows the problem though).
> > >
> > > any suggestion is appreciated.
> > >
> > >
> > > Thanks,
> > > Lin
> > >
> > >
> > > =======================
> > > American Museum of Natural History
> > > POY website :
> > > https://code.google.com/p/poy/
> > > http://research.amnh.org/scicomp/scripts/download.php
> > >
> > > --
> > > Caml-list mailing list.  Subscription management and archives:
> > > https://sympa-roc.inria.fr/wws/info/caml-list
> > > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > > Bug reports: http://caml.inria.fr/bin/caml-bugs
> > >
> > >
> >
> >
> > --
> > Caml-list mailing list.  Subscription management and archives:
> > https://sympa-roc.inria.fr/wws/info/caml-list
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
> >
> >
> >
>
> --
> ------------------------------------------------------------
> Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
> Creator of GODI and camlcity.org.
> Contact details:        http://www.camlcity.org/contact.html
> Company homepage:       http://www.gerd-stolpmann.de
> ------------------------------------------------------------
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>
>



--
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------