Stack_overflow

Stack_overflow

[mulhern]

Hi!

I'm trying to compile an automatically generated list definition with
approximately 12,000 elements. The list has type (record_type *
record_type list) list where record_type is a simple record with 4
fields.

I'm doing this because I'm trying to achieve a poor man's version of
staged compilation.

Unfortunately, I get a Stack_overflow exception. If I roughly half the
number of elements in the list the Stack_overflow exception goes away.

Does anybody have an suggestions for me to get around the stack
overflow problem? What if I want to make my data an order of magnitude
larger?

Thanks!

-mulhern

Re: [Caml-list] Stack_overflow

[Jon Harrop]

On Friday 31 March 2006 21:44, mulhern wrote:
> Unfortunately, I get a Stack_overflow exception. If I roughly half the
> number of elements in the list the Stack_overflow exception goes away.

Try increasing the stack size using:

  export CAMLRUNPARAM='...'

where you get "..." from the manual. Something like 'l=100M', IIRC.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists

Re: [Caml-list] Stack_overflow

[Eric Cooper]

On Fri, Mar 31, 2006 at 02:44:26PM -0600, mulhern wrote:
> I'm trying to compile an automatically generated list definition with
> approximately 12,000 elements.
> [...]
> Unfortunately, I get a Stack_overflow exception.

Try using the natively-compiled compilers, ocamlc.opt or ocamlopt.opt.

-- 
Eric Cooper             e c c @ c m u . e d u

Re: Stack_overflow

[Stefan Monnier]

> Does anybody have an suggestions for me to get around the stack
> overflow problem?

Chunkify your list: turn it into a bunch of lists (each limited to size
predefined maximum length) and then concat them all at the end?


        Stefan

Re: Stack_overflow

[mulhern]

Thanks to everybody who responded.

To clarify things:
I'm trying to _compile_ a list definition.
So, my .ml file looks like this:

---
let myList = [("first", ["some"; more]);
                   ("second", ["more"; "still"])]
---
except that there are 12,000 elements in the list instead of two as in
the example.
_ocamlc_ throws a Stack_overflow error while compiling this list. So,
I want to know how to influence _ocamlc_ to be able to compile this or
larger lists.

The suggestion of chunkifying the list into smaller lists is a
practical one; I may be forced to try it.

ocamlc.opt compiles the list fine but that reduces portability.

-mulhern

On 3/31/06, mulhern <mulhern@gmail.com> wrote:
> Hi!
>
> I'm trying to compile an automatically generated list definition with
> approximately 12,000 elements. The list has type (record_type *
> record_type list) list where record_type is a simple record with 4
> fields.
>
> I'm doing this because I'm trying to achieve a poor man's version of
> staged compilation.
>
> Unfortunately, I get a Stack_overflow exception. If I roughly half the
> number of elements in the list the Stack_overflow exception goes away.
>
> Does anybody have an suggestions for me to get around the stack
> overflow problem? What if I want to make my data an order of magnitude
> larger?
>
> Thanks!
>
> -mulhern
>

Re: [Caml-list] Re: Stack_overflow

[Ingo Bormuth]

Hi, 

I recently had a very similar problem, trying to build a simple 
in-binary database holding 80.000 records (12 MB).

In my case even ocamlc.opt couldn't cope. Maybe there are non tail 
recursive calls in the compiler. Eventually recompiling ocaml
itself with greater stack size could do the job.

Cutting the list into pieces and merging at runtime worked fine.

What I did, was using a CSV file to auto generate ocaml code similar to:

  let table = Hashtbl.create 80000 ;;

  Hashtbl.add table "Key" [ Value1 , Value2, Value3 ] ;; (* 80.000 times *)

  Iterate over table and build some index hash tables ValueX -> Key
 
Compilation takes a while and uses plenty of RAM, but building the table
and looking up 80000 values takes just a second on my old laptop.


Ingo



On 2006-03-31 18:23, mulhern wrote:
> To clarify things:
> I'm trying to _compile_ a list definition.
> So, my .ml file looks like this:
> 
> ---
> let myList = [("first", ["some"; more]);
>                    ("second", ["more"; "still"])]
> ---
> except that there are 12,000 elements in the list instead of two as in
> the example.
> _ocamlc_ throws a Stack_overflow error while compiling this list. So,
> I want to know how to influence _ocamlc_ to be able to compile this or
> larger lists.
> The suggestion of chunkifying the list into smaller lists is a
> practical one; I may be forced to try it.
> 
> ocamlc.opt compiles the list fine but that reduces portability.
> 
> Odn 3/31/06, mulhern <mulhern@gmail.com> wrote:
> >
> > I'm trying to compile an automatically generated list definition with
> > approximately 12,000 elements. The list has type (record_type *
> > record_type list) list where record_type is a simple record with 4
> > fields.
> > I'm doing this because I'm trying to achieve a poor man's version of
> > staged compilation.
> > Unfortunately, I get a Stack_overflow exception. If I roughly half the
> > number of elements in the list the Stack_overflow exception goes away.
> > Does anybody have an suggestions for me to get around the stack
> > overflow problem? What if I want to make my data an order of magnitude
> > larger?

-- 
Ingo Bormuth, voicebox & telefax: +49-12125-10226517       '(~o-o~)'
public key 86326EC9, http://ibormuth.efil.de/contact   --ooO--(.)--Ooo--

Re: [Caml-list] Re: Stack_overflow

[Christophe TROESTLER]

On Fri, 31 Mar 2006, mulhern <mulhern@gmail.com> wrote:
> 
> The suggestion of chunkifying the list into smaller lists is a
> practical one; I may be forced to try it.

Maybe you want to do this in an auxiliary program that saves the
compiled list in a file (output_value) and the main program just reads
it (input_value)?

My 0.02€,
ChriS

Re: [Caml-list] Re: Stack_overflow

[Till Varoquaux]

[-- Attachment #1: Type: text/plain, Size: 1312 bytes --]

On 4/1/06, mulhern <mulhern@gmail.com> wrote:
>
> Thanks to everybody who responded.
>
> To clarify things:
> I'm trying to _compile_ a list definition.
> So, my .ml file looks like this:
>
> ---
> let myList = [("first", ["some"; more]);
>                    ("second", ["more"; "still"])]
> ---
> except that there are 12,000 elements in the list instead of two as in
> the example.
> _ocamlc_ throws a Stack_overflow error while compiling this list. So,
> I want to know how to influence _ocamlc_ to be able to compile this or
> larger lists.
>
> The suggestion of chunkifying the list into smaller lists is a
> practical one; I may be forced to try it.
>
> ocamlc.opt compiles the list fine but that reduces portability.
>
> -mulhern


Note that ocamlc.opt should produce exactly the same code than ocamlc....
The difference is that the compiler is natively compiled instead of
byte-compiled. Therefor your program will run in as many places it just
won't compile on plateforms without ocamlc.opt. You could put your list in a
separate module, byte compile it and "include" it in the reste of your
source. This *should* solve the problem (I might be wrong) and you could
still bundle the rest of your sources in a way that would allow to compile
them with ocamlc.


Till

[-- Attachment #2: Type: text/html, Size: 1743 bytes --]