Causes for segfaults

Causes for segfaults

[Jamie Brandon]

Is there an authoritative list anywhere of the possible causes for
segfaults in ocaml? I'm not using the FFI or -unsafe and I have ruled
out overflowing the c stack and allocating large strings/arrays. I'm
using ocaml 3.09.3 with extlib, netclient, json-static, xml-light, str
and no compilation options.

Cheers

Jamie

Re: [Caml-list] Causes for segfaults

[Raphael Proust]

Le jeudi 04 novembre 2010 21:19:33, Jamie Brandon a écrit :
> Is there an authoritative list anywhere of the possible causes for
> segfaults in ocaml? I'm not using the FFI or -unsafe and I have ruled
> out overflowing the c stack and allocating large strings/arrays. I'm
> using ocaml 3.09.3 with extlib, netclient, json-static, xml-light, str
> and no compilation options.

Unmarshalling can cause segfaults and Obj.magic too.

-- 
_______
Raphaël

Re: [Caml-list] Causes for segfaults

[Jamie Brandon]

I'm not using Obj. I am unmarshalling. I have 'ulimit -s unlimited'
set and I'm unmarshalling with the exact same binary that did the
marshalling. The unmarshalling itself finishes ok but I guess the
segfault could be caused when accessing the resulting data structure?

I'm marshalling data which contains a DynArray which in turn contains
functional values, but I'm using [Marshal.No_sharing;
Marshal.Closures] and I haven't changed the code between marshalling
and unmarshalling.


On Thu, Nov 4, 2010 at 8:25 PM, Raphael Proust <raphlalou@gmail.com> wrote:
> Le jeudi 04 novembre 2010 21:19:33, Jamie Brandon a écrit :
>> Is there an authoritative list anywhere of the possible causes for
>> segfaults in ocaml? I'm not using the FFI or -unsafe and I have ruled
>> out overflowing the c stack and allocating large strings/arrays. I'm
>> using ocaml 3.09.3 with extlib, netclient, json-static, xml-light, str
>> and no compilation options.
>
> Unmarshalling can cause segfaults and Obj.magic too.
>
> --
> _______
> Raphaël
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Re: [Caml-list] Causes for segfaults

[David MENTRE]

Hello,

2010/11/4 Jamie Brandon <jamie@scattered-thoughts.net>:
> The unmarshalling itself finishes ok but I guess the
> segfault could be caused when accessing the resulting data structure?

You could frequently call major collection in the GC, in order to be
as close as possible to the source of the segfault (e.g. after your
unmarshalling).

Sincerely yours,
david

Re: [Caml-list] Causes for segfaults

[oliver]

On Thu, Nov 04, 2010 at 08:39:27PM +0000, Jamie Brandon wrote:
> I'm not using Obj. I am unmarshalling. I have 'ulimit -s unlimited'
[...]


I once have used Marshall-Module, when exploring Ocaml the first time.
I changed the saved marshalled files by hand and read them back,
doing unmarshalling and this yielded a crash.

Then I decided to never again use this module and also asked to remove
such unsafe technics/modules from the standard lib, but as you can see,
it was not removed.

Should be an external add-on, or better, made safe.


Ciao,
   Oliver

Re: [Caml-list] Causes for segfaults

[Till Varoquaux]

I have a personnal marked dislike for xml-light; one of the reason
being that it is tied together using duct-tape and generous sprinkling
Obj.magic. Probably not the cause of your problem but, and again, who
knows.

We (Jane street) have a patched version that does not use Obj.magic;
the main reason that I haven't pushed for us to upstream these patches
is that I'd rather see Xml-light die in favour of something that
adheres better to the xml standards (like xmlm).

It might be intersting for you to run your app in gdb and sent the
backtrace of the segfault. Valgrind could also be very useful here but
it is not for the faint of heart.

very quick guide to gdb (although I am sure you know):

_ 1) Make sure your program is compiled with -g
_ 2) gdb ./my_pgm
       > run
       .... (boom!)
       > bt

On Thu, Nov 4, 2010 at 4:19 PM, Jamie Brandon
<jamie@scattered-thoughts.net> wrote:
> Is there an authoritative list anywhere of the possible causes for
> segfaults in ocaml? I'm not using the FFI or -unsafe and I have ruled
> out overflowing the c stack and allocating large strings/arrays. I'm
> using ocaml 3.09.3 with extlib, netclient, json-static, xml-light, str
> and no compilation options.
>
> Cheers
>
> Jamie
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Re: [Caml-list] Causes for segfaults

[Jamie Brandon]

> It might be intersting for you to run your app in gdb and sent the
> backtrace of the segfault. Valgrind could also be very useful here but
> it is not for the faint of heart.
>
> very quick guide to gdb (although I am sure you know):

Not at all - spent my whole life in HLL land. Learning to use gdb is
probably long overdue.

> _ 1) Make sure your program is compiled with -g

Ocaml 3.9 can't compile native code with -g :(

Installing newer versions of ocaml on old installations of centOS is
... entertaining. Despite the delay, I did eventually get this to
work.

[root@senldogo0183 texsearch-development2]# gdb ./index
GNU gdb Red Hat Linux (6.5-25.el5rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...(no debugging
symbols found)
Using host libthread_db library "/lib64/libthread_db.so.1".

(gdb) run
Starting program: /data/texsearch-development2/index
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
[New Thread 46912496271328 (LWP 30551)]
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 46912496271328 (LWP 30551)]
0x00000000004466e5 in camlIndex__run_query_1442 ()
(gdb) bt
#0  0x00000000004466e5 in camlIndex__run_query_1442 ()
#1  0x0000000000004e21 in ?? ()
#2  0x000000000056050c in caml_ml_close_channel ()
#3  0x000000000078d0a0 in camlIndex ()
#4  0x0000000000000001 in ?? ()
#5  0x7fffffffffffffff in ?? ()
#6  0x00002aaaaab04900 in ?? ()
#7  0x0000000000447e98 in camlIndex__entry ()
#8  0x00002aaaaab04900 in ?? ()
#9  0x000000000078d890 in camlIndex__142 ()
#10 0x000000000078d808 in camlIndex__140 ()
#11 0x000000000078d928 in camlIndex__144 ()
#12 0x000000000078d8c0 in camlIndex__143 ()
#13 0x00002aaaaab03a58 in ?? ()
#14 0x000000000078d7c0 in camlIndex__123 ()
#15 0x000000000078d828 in camlIndex__141 ()
#16 0x000000000078dab0 in camlIndex__146 ()
#17 0x00002aaaaab03918 in ?? ()
#18 0x000000000078d7e8 in camlIndex__139 ()
#19 0x00002aaaaab038e8 in ?? ()
#20 0x000000000078da78 in camlIndex__145 ()
#21 0x00002aaaaab038c0 in ?? ()
#22 0x00002aaaaab03a40 in ?? ()
#23 0x00002aaaaab03a28 in ?? ()
#24 0x00002aaaaab03a10 in ?? ()
#25 0x00002aaaaab039f8 in ?? ()
#26 0x00002aaaaab039b0 in ?? ()
#27 0x00002aaaaab03960 in ?? ()
#28 0x0000000000000201 in ?? ()
#29 0x000000000043fdc9 in caml_startup__code_begin ()
#30 0x00000000002c5123 in ?? ()
#31 0x000000000056996e in caml_start_program ()

I'm not sure how to interpret this. run_query doesnt *directly* do
anything but it calls into other modules (suffix_array etc) that are
in the same directory. It looks to me like I screwed up the build
process and there are no debugging symbols for suffix_array and co.
This is what I did:

ocamlfind ocamlopt -linkpkg -g \
    -package extlib \
    -package netclient \
    -package json-wheel \
    -package json-static \
    -package xml-light \
    -package str \
    -syntax camlp4o \
    -package json-static \
    util.mli util.ml \
    pid.mli pid.ml \
    latex.mli latex.ml \
    edit.ml \
    suffix_array.mli suffix_array.ml \
    query.ml \
    index.ml \
-o index

Any ideas?

Re: [Caml-list] Causes for segfaults

[Daniel de Rauglaudre]

Hi,

On Sat, Nov 06, 2010 at 12:14:31AM +0100, Jamie Brandon wrote:

I had a Segmentation Fault some days ago in native code. Compiling
with bytecode, it raised Stack_overflow. The bug was that I called
a non tail-recursive function.

Is your stack size unlimited ? (command: ulimit -a)

If it is, perhaps try to limit it and try again in bytecode to see
if the Segmentation Fault is transformed into a Stack Overflow.

-- 
Daniel de Rauglaudre
http://pauillac.inria.fr/~ddr/

Re: [Caml-list] Causes for segfaults

[Jamie Brandon]

> If it is, perhaps try to limit it and try again in bytecode to see
> if the Segmentation Fault is transformed into a Stack Overflow.

Marshalling is not tail-recursive, so I will always get a stack
overflow marshalling this much data. Setting ulimit -s unlimited
prevents this but also obscures the current problem.

Re: [Caml-list] Causes for segfaults

[Till Varoquaux]

That's an odd backtrace.
As far as I can tell you have compiled everything fine yet it tells me
that run_query is blowing up in a subcall to caml_ml_close_channel
(which is the underlying c call for close_in and close_out).

I might be reading this bt wrong. Otherwise there's something fishy
with your stack. Whatever it is, I am very confused by this all. Would
you mind posting your code to see if other people can reproduce the
problem?

Cheers,
Till

On Mon, Nov 8, 2010 at 9:26 AM, Jamie Brandon
<jamie@scattered-thoughts.net> wrote:
>> If it is, perhaps try to limit it and try again in bytecode to see
>> if the Segmentation Fault is transformed into a Stack Overflow.
>
> Marshalling is not tail-recursive, so I will always get a stack
> overflow marshalling this much data. Setting ulimit -s unlimited
> prevents this but also obscures the current problem.
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Re: [Caml-list] Causes for segfaults

[Jamie Brandon]

> Would you mind posting your code to see if other people can reproduce the problem?

I can post the code (
http://scattered-thoughts.net/code/texsearch-0.4/ ) but unfortunately
I can't post the (large) dataset that causes the segfault. I will see
if I can persuade it to blow up on a randomly generated dataset and
post the code for that.

Re: [Caml-list] Causes for segfaults

[Jamie Brandon]

I finally fixed this. The difficulty was down to the sheer number of
different causes:

During incremental updates (but not bulk updates) the wrong data
structure was marshaled to disk so the next process to load from the
disk would segfault

DynArray contains functional values so compiling with -g breaks
previously marshaled values

Marshal segfaults on large data structures unless 'ulimit -s unlimited' is set

List.merge is not tail recursive and segfaults on large lists unless
'ulimit -s unlimited' is set (this only happens during bulk updates)

On 32 bit machines the arrays used in the suffix array are too large,
in older versions of ocaml this causes a segfault

I'm still not sure why gdb reliably tracked a segfault to
caml_ml_close_channel but the very next instruction was accessing the
data structure marshaled from disk.

I do feel that this would have been much easier to fix if ocaml raised
more informative errros. The change to Array.make and String.make to
raise exceptions instead of segfaulting is welcome. Is there a way to
cause stack overflows to raise exceptions in native compiled ocaml as
well?

Re: [Caml-list] Causes for segfaults

[John Whitington]

Hi,

On 15/11/2010 17:58, Jamie Brandon wrote:
> I do feel that this would have been much easier to fix if ocaml raised
> more informative errros. The change to Array.make and String.make to
> raise exceptions instead of segfaulting is welcome. Is there a way to
> cause stack overflows to raise exceptions in native compiled ocaml as
> well?

Excessive recursion of a function does raise an exception, on most 
platforms, doesn't it?

However, this doesn't seem to be the case with excessive nested 
exceptions. I got caught by this the other day:

let rec lex_stream i acc =
   try lex_stream i (lex_next i::acc) with LexingEnd -> rev acc

(I had been replacing option types with exceptions in some places for 
speed and wasn't concentrating...)

This segfaults on OS X Intel with OCaml 3.12.0 on large inputs.

The workaround is to add a null constructor for the type e.g LexNone. In 
the absence of one, I think you have to do:

let rec lex_stream i acc =
   match try Some (lex_stream i) with LexingEnd -> None with
   | None -> rev acc
   | Some s -> lex_stream i (a::acc)

Hopefully that option type doesn't slow things down too much.

Cheers,

-- 
John Whitington
Director, Coherent Graphics Ltd
http://www.coherentpdf.com/

Re: [Caml-list] Causes for segfaults

[Jamie Brandon]

> Excessive recursion of a function does raise an exception, on most
> platforms, doesn't it?

jamie@jamie-aspire:~$ uname -a
Linux jamie-aspire 2.6.35-22-generic #35-Ubuntu SMP Sat Oct 16
20:36:48 UTC 2010 i686 GNU/Linux
jamie@jamie-aspire:~$ ocaml -version
The Objective Caml toplevel, version 3.11.2
jamie@jamie-aspire:~$ cat > segfault.ml
let rec ints n = n :: ints (n+1)
let _ = ints 0
jamie@jamie-aspire:~$ ocamlopt segfault.ml
jamie@jamie-aspire:~$ ./a.out
Segmentation fault

[root@senldogo0183 texsearch-development2]# uname -a
Linux senldogo0183 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007
x86_64 x86_64 x86_64 GNU/Linux
[root@senldogo0183 texsearch-development2]# ocaml -version
The Objective Caml toplevel, version 3.12.0
[root@senldogo0183 texsearch-development2]# cat > segfault.ml
let rec ints n = n :: ints (n+1)
let _ = ints 0
[root@senldogo0183 texsearch-development2]# ocamlopt segfault.ml
[root@senldogo0183 texsearch-development2]# ./a.out
Segmentation fault

Re: [Caml-list] Causes for segfaults

[Török Edwin]

On Mon, 15 Nov 2010 18:38:18 +0000
Jamie Brandon <jamie@scattered-thoughts.net> wrote:

> > Excessive recursion of a function does raise an exception, on most
> > platforms, doesn't it?
> 
> [root@senldogo0183 texsearch-development2]# uname -a
> Linux senldogo0183 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007
> x86_64 x86_64 x86_64 GNU/Linux
> [root@senldogo0183 texsearch-development2]# ocaml -version
> The Objective Caml toplevel, version 3.12.0
> [root@senldogo0183 texsearch-development2]# cat > segfault.ml
> let rec ints n = n :: ints (n+1)
> let _ = ints 0
> [root@senldogo0183 texsearch-development2]# ocamlopt segfault.ml
> [root@senldogo0183 texsearch-development2]# ./a.out
> Segmentation fault

I get a proper exception:
Fatal error: exception Stack_overflow

Linux debian 2.6.36-phenom #107 SMP PREEMPT Sat Oct 23 10:30:01 EEST
2010 x86_64 GNU/Linux

Although you have to be careful with recursion and calls to C code,
calling the C code may be the one overflowing the stack, and the ocaml
runtime doesn't generate stack overflow exception for that (only for
OCaml code). You will probably need the patch from
http://caml.inria.fr/mantis/view.php?id=5064 for that.

Best regards,
--Edwin

Re: [Caml-list] Causes for segfaults

[Daniel de Rauglaudre]

On Mon, Nov 15, 2010 at 07:38:25PM +0100, Jamie Brandon wrote:

> jamie@jamie-aspire:~$ cat > segfault.ml
> let rec ints n = n :: ints (n+1)
> let _ = ints 0
> jamie@jamie-aspire:~$ ocamlopt segfault.ml
> jamie@jamie-aspire:~$ ./a.out
> Segmentation fault

Stack overflow in native code triggers a segmentation fault, indeed.
I think in C, it is the same thing. Not tested in the OCaml runtime
for efficiency purposes, I think.

-- 
Daniel de Rauglaudre
http://pauillac.inria.fr/~ddr/

Re: [Caml-list] Causes for segfaults

[Anil Madhavapeddy]

On 15 Nov 2010, at 18:50, Daniel de Rauglaudre wrote:

> On Mon, Nov 15, 2010 at 07:38:25PM +0100, Jamie Brandon wrote:
> 
>> jamie@jamie-aspire:~$ cat > segfault.ml
>> let rec ints n = n :: ints (n+1)
>> let _ = ints 0
>> jamie@jamie-aspire:~$ ocamlopt segfault.ml
>> jamie@jamie-aspire:~$ ./a.out
>> Segmentation fault
> 
> Stack overflow in native code triggers a segmentation fault, indeed.
> I think in C, it is the same thing. Not tested in the OCaml runtime
> for efficiency purposes, I think.

The runtime can detect native stack overflow if the OS supports it. See the HAS_STACK_OVERFLOW_DETECTION define in the configure script :

case "$arch,$system" in
  i386,linux_elf|amd64,linux|power,rhapsody|amd64,macosx|i386,macosx|amd64,macosx)
    echo "System stack overflow can be detected."
    echo "#define HAS_STACK_OVERFLOW_DETECTION" >> s.h;;
  *)
    echo "Cannot detect system stack overflow.";;
esac

This works great for me on both MacOS X 10.6 and x86_64 Linux.

avsm@brolga-0:~$ uname -a
Linux brolga-0 2.6.27.29-0.1.1.xs0.1.0.669.1028xen-xcixen #3 SMP Thu Dec 24 23:39:24 GMT 2009 x86_64 GNU/Linux
avsm@click$ uname -a
Darwin click.local 10.5.0 Darwin Kernel Version 10.5.0: Fri Nov  5 23:19:13 PDT 2010; root:xnu-1504.9.17~1/RELEASE_X86_64 x86_64

avsm@brolga-0:~$ cat > segfault.ml
let rec ints n = n :: ints (n+1)
let _ = ints 0
avsm@brolga-0:~$ ocamlopt segfault.ml 
avsm@brolga-0:~$ ./a.out 
Fatal error: exception Stack_overflow

Take a look at your configure script output to see if it isn't being detected I guess. There may also be some interaction with your shell ulimit that forces a SIGSEGV, perhaps.

Anil

Re: [Caml-list] Causes for segfaults

[Norman Hardy]

On 2010 Nov 15, at 10:50 , Daniel de Rauglaudre wrote:

> On Mon, Nov 15, 2010 at 07:38:25PM +0100, Jamie Brandon wrote:
> 
>> jamie@jamie-aspire:~$ cat > segfault.ml
>> let rec ints n = n :: ints (n+1)
>> let _ = ints 0
>> jamie@jamie-aspire:~$ ocamlopt segfault.ml
>> jamie@jamie-aspire:~$ ./a.out
>> Segmentation fault
> 
> Stack overflow in native code triggers a segmentation fault, indeed.
> I think in C, it is the same thing. Not tested in the OCaml runtime
> for efficiency purposes, I think.
> 
> --

My 2 cents worth of ideas for efficient and safe stack frame allocation:
http://cap-lore.com/code/BigStack.html

On Mac:
172% uname -a
Darwin apple-s-Computer.local 10.5.0 Darwin Kernel Version 10.5.0: Fri Nov  5 23:20:39 PDT 2010; root:xnu-1504.9.17~1/RELEASE_I386 i386
172% ocaml -version
The Objective Caml toplevel, version 3.12.0
172% cat > segfault.ml
let rec ints n = n :: ints (n+1)
let _ = ints 0
172% ocamlopt segfault.ml
ld: in /usr/local/lib/ocaml/std_exit.o, indirect symbol index out of range
collect2: ld returned 1 exit status
File "caml_startup", line 1, characters 0-1:
Error: Error during linking

Re: [Caml-list] Causes for segfaults

[Daniel de Rauglaudre]

On Mon, Nov 15, 2010 at 08:00:05PM +0100, Anil Madhavapeddy wrote:

> The runtime can detect native stack overflow if the OS supports
> it. See the HAS_STACK_OVERFLOW_DETECTION define in the configure
> script :

$ grep HAS_STA config/s.h
#define HAS_STACK_OVERFLOW_DETECTION

$ cat > segfault.ml
let rec ints n = n :: ints (n+1)
let _ = ints 0

$ ocamlopt segfault.ml 

$ ./a.out
Segmentation fault

-- 
Daniel de Rauglaudre
http://pauillac.inria.fr/~ddr/