[Caml-list] troubleshooting problem related to garbage collection

[Caml-list] troubleshooting problem related to garbage collection

[james woodyatt]

friends--

Last night I spent a lot of time trying to figure out why an assertion 
was failing in my code when I use it with large data sets.  I'm still 
puzzled.

This morning, I inserted the following at the beginning of my main 
module:

> Gc.set {
>     (Gc.get ()) with
>     Gc.major_heap_increment = 64000
> }

This made the problem go away.

What's even more strange is that there are multiple failure modes to 
choose from depending on which number I pick for the 
major_heap_increment value.  The number I am using here just happens to 
be the one that makes the code work.

Can anyone advise me on how to find the bug here?


--
j h woodyatt <jhw@wetware.com>
"...the antidote to misinformation is more information, not less."
                                                      --vinton cerf

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] troubleshooting problem related to garbage collection

[james woodyatt]

On Friday, March 1, 2002, at 10:52 AM, james woodyatt wrote:
>
> Last night I spent a lot of time trying to figure out why an assertion 
> was failing in my code when I use it with large data sets.  I'm still 
> puzzled.
>
> This morning, I inserted the following at the beginning of my main 
> module:
>
>> Gc.set {
>>     (Gc.get ()) with
>>     Gc.major_heap_increment = 64000
>> }
>
> This made the problem go away.

Further investigation reveals that it does not make the problem go away, 
it changes the failure mode into one that only makes it look like the 
problem has gone away.  The process terminates prematurely (but there 
was no more output expected anyway).

> What's even more strange is that there are multiple failure modes to 
> choose from depending on which number I pick for the 
> major_heap_increment value.  The number I am using here just happens to 
> be the one that makes the code work.
>
> Can anyone advise me on how to find the bug here?

I've found the problem.  It was a function trying too hard not to blit a 
string.  The bug only manifested when writing to a non-blocking TCP 
socket would return a value less than the number of octets to send.

This makes me think that when similar problems arise-- i.e. when 
changing the parameters of the GC seems to make always-reproduceable 
misbehavior change in some remarkable way-- the first place to start 
looking for the cause is code that blits strings.


--
j h woodyatt <jhw@wetware.com>
"...the antidote to misinformation is more information, not less."
                                                      --vinton cerf

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

[Caml-list] The DLL-hell of O'Caml

[Mattias Waldau]

(THIS IS NOT A BUG REPORT ON WDIALOG, WDIALOG IS JUST AN EXAMPLE!)

I wanted to try WDialog made by Gerd Stolpmann.

Gerd is one of the more productive Ocaml programmers, 
who generates good and documentated libraries.

On my linux-machine I use the CDK with ocaml-3.03ALPHA.

This is the installation history:
1. Needs Ocaml 3.03 -> updated CDK using CVS (there are no tarballs
but I have a broadband-connection).
2. Ocamlnet missing -> downloaded ocamlnet-0.92
3. Failed to install ocamlnet, since the PCRE in the CDK seems to
be to old.
4. Installed a new PCRE.
5. Tried to install ocamlnet again, however failed with 
	"unbound type constructor Mimestring.s_param"

   Probably something else is too old...I got bored...
   ...never got back to try to install WDialog....

---------

The above history is the reason I started using the CDK,
in many cases using libraries that depend on other
libraries is almost impossible without the CDK. Too much 
time is spent on downloading and compiling.

I know 3 solutions to the problem:
1. CDK
2. Gerd invented findlib to solve the above problem.
3. Adding good packages to the standard distribution,
   so that mostly packages doesn't depend on other
   packages, but on packages in the standard
   distribution. (But very few if any new packages
   are added to the distribution.)


There are probably more solutions. I would call this 
problem a show stopper, since it prevents user to use
good libraries. 

We need a solution. Maybe a CPAN-like solution? 
In the Ocaml-CPAN it could either be source code, or
compile binaries (I can live without native code, at
least when experimenting with other peoples libraries.)

/mattias


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Markus Mottl]

On Sat, 02 Mar 2002, Mattias Waldau wrote:
> We need a solution. Maybe a CPAN-like solution?  In the Ocaml-CPAN it
> could either be source code, or compile binaries (I can live without
> native code, at least when experimenting with other peoples libraries.)

This raises the old question of "social tools" again. Given that our
community is still comparatively small, it is absolutely mandatory
that there be tools and standards that make us as productive as
possible. Not only will this allow us to use our scarce human resources
more efficiently, it is also more likely to attract more people when they
see that contributing and benefiting from other contributions is easy.

Some standard packaging tools + automated means of resolving dependencies,
e.g. by downloading contributions from some central archive, and a
documentation standard (ocamldoc?) would surely help. Any concrete plans?

Regards,
Markus Mottl

-- 
Markus Mottl                                             markus@oefai.at
Austrian Research Institute
for Artificial Intelligence                  http://www.oefai.at/~markus
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Alexander V. Voinov]

Hi Markus et al.

Markus Mottl wrote:
> Some standard packaging tools + automated means of resolving dependencies,
> e.g. by downloading contributions from some central archive, and a
> documentation standard (ocamldoc?) would surely help. Any concrete plans?

What CDK is? How is it related to the 'mainstream' OCaml? Why it's
always aside and out of date w.r.t. official releases?

Alexander
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Alain Frisch]

On Sat, 2 Mar 2002, Mattias Waldau wrote:

> The above history is the reason I started using the CDK,
> in many cases using libraries that depend on other
> libraries is almost impossible without the CDK. Too much
> time is spent on downloading and compiling.

A related issue is independance w.r.t OCaml version; if you want to try a
new OCaml release or CVS version, it is necessary to rebuild all the
libraries.  For instance, I now have three OCaml versions installed (3.04,
3.04+7 and 3.04+6 polymorphic methods), and it is a pain to maintain three
library trees.

Similarly, when you upgrade a library, you also have to rebuild all
the libraries that depend on it. Some kind of automatization in the
process would be great.

> I know 3 solutions to the problem:
> 1. CDK
> 2. Gerd invented findlib to solve the above problem.
> 3. Adding good packages to the standard distribution,
>    so that mostly packages doesn't depend on other
>    packages, but on packages in the standard
>    distribution. (But very few if any new packages
>    are added to the distribution.)

The monolithic and centralized approach of CDK does not seem to scale
well. Findlib does a good job. It would be great to have such a tool
(findlib or a clone, but why not findlib ?) in the standard
distribution...


> We need a solution. Maybe a CPAN-like solution?

Indeed, the new packaging facility in OCaml (-pack option, which glues
together a set of modules and allows decent namespace management) opens
the door to this kind of project.

> In the Ocaml-CPAN it could either be source code, or
> compile binaries (I can live without native code, at
> least when experimenting with other peoples libraries.)

An intermediate solution would be to distribute so-called "lambda-code",
that is the internal representation used in OCaml compilers just before
code production. The advantage is that it is common to bytecode and
native back-end, and it should be possible to design an upward-compatible
(w.r.t OCaml version) concrete representation of this code.


-- Alain

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Chris Hecker]

I think a CPAN-like-thing would be excellent and is sorely needed!  I'm 
super busy right now (who isn't?), but I'd be willing to commit some time 
to testing it at the least, and maybe contributing code after May.

I've used CPAN a little bit, but we should do a survey of other archive 
managers and learn from what works and what doesn't.  Maybe there's a web 
page somewhere.  There's also apt-get from Debian and the Redhat one to 
look at.

I wonder if we could just use one of the existing managers...although it 
seems like it should be written in caml for karmic reasons, at least.  :)

 >> In the Ocaml-CPAN it could either be source code, or
> > compile binaries (I can live without native code, at
> > least when experimenting with other peoples libraries.)

Why wouldn't it be source-only?  So people don't have to have the compiler 
installed?  It seems that there are two related but different projects:  an 
archive/package manager for developers, and one for users.  Maybe those can 
be the same thing, at least for bytecode.  But maybe the intermediate rep 
idea would work for native as well.  Or some server that builds all the 
supported platforms or something.

Chris


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] The DLL-hell of O'Caml

[Mattias Waldau]

> From: Chris Hecker [mailto:checker@d6.com] 
> I've used CPAN a little bit, but we should do a survey of 
> other archive 
> managers and learn from what works and what doesn't.  Maybe 
> there's a web 
> page somewhere.  There's also apt-get from Debian and the 

apt-get, interesting idea. (does it work for windows?)

>  >> In the Ocaml-CPAN it could either be source code, or
> > > compile binaries (I can live without native code, at
> > > least when experimenting with other peoples libraries.)
> 
> Why wouldn't it be source-only?  So people don't have to have 

compilation and windows is a problem. even if many windows users
use cygwin, I think the native port is very important. If there
would be commercial applications written in O'Caml, I think
they will be written for Windows (Anyone know any commercial
applications for *nix? :-) 

I got the impression that the byte-code is compatible between
different platforms (Is this true?), and then this would
simplify distribution, since especially windows-users wouldn't
have the compilationsproblems (unless they want to change the
code or use the native compiler).


/mattias

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] The DLL-hell of O'Caml

[Chris Hecker]

>apt-get, interesting idea. (does it work for windows?)

I doubt it, but it probably wouldn't be that hard to port (he says, 
cluelessly).

>compilation and windows is a problem.

Right, good point, I forgot about that.  Although, that does imply we're 
targeting both developers and users, which will make things harder.

 > I got the impression that the byte-code is compatible between
>different platforms (Is this true?)

I believe this is true.

Chris

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] The DLL-hell of O'Caml

[Vitaly Lugovsky]

On Sat, 2 Mar 2002, Chris Hecker wrote:

> >apt-get, interesting idea. (does it work for windows?)
> 
> I doubt it, but it probably wouldn't be that hard to port (he says, 
> cluelessly).

 It's already ported to RPM (e.g. in AltLinux distribution),
and RPM works with Cygwin32, so, it's not a problem.


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Sven]

On Sat, Mar 02, 2002 at 08:42:40PM +0100, Mattias Waldau wrote:
> > From: Chris Hecker [mailto:checker@d6.com] 
> > I've used CPAN a little bit, but we should do a survey of 
> > other archive 
> > managers and learn from what works and what doesn't.  Maybe 
> > there's a web 
> > page somewhere.  There's also apt-get from Debian and the 
> 
> apt-get, interesting idea. (does it work for windows?)

Mmm, i don't think so, altough i know the darwin project use a variation of
dpkg and apt-get for their package management. You are free to port it though,
if you want, it is open source anyway.

Also notice, that appart from the nice apt-get tool, much work has been done
to package the new ocaml 3.04 and to solve the problems related to shared libs
on debian systems. We even use a small tool ocaml-ldconf (quickly hacked by
me, so it is not optimal, but does its work) which uses 3 databases to store
the ld.conf files, one is the normal ld.conf from ocaml, and the other two
being the dpkg handled /var/lib/ocaml/ld.conf, and a system administrator
override in /etc/ocaml/ld.conf.

I would gladly contribute it if it seems usefull and people want it, but i am
not sure what help it would be without the integration permitted by apt-get,
dpkg and the debian packaging system (and the work of the debian/ocaml
maintainer team :)))

Friendly,

Sven Luther
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

possible solution to [Caml-list] The DLL-hell of O'Caml

[Dave Mason]

I have an undergrad thesis student who is working on a ``dynamic load
via http (including https) with local cache''.  She will be done by
May 1 (in order to graduate :-).

It will attempt to load byte-code versions if running in byte-code,
and the appropriate native-code versions if natively compiled.

We actually have even more exciting plans for this, but will hold the
announcement until we have things working.

Quite apart from the Windows compile problem, I don't think source
code is the right way to distribute code.  Distributing compiled
versions opens the potential for commercial (proprietary source)
libraries.

../Dave
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: possible solution to [Caml-list] The DLL-hell of O'Caml

[Mattias Waldau]

> I have an undergrad thesis student who is working on a 
> ``dynamic load via http (including https) with local cache''. 
>  She will be done by May 1 (in order to graduate :-).
> 
> It will attempt to load byte-code versions if running in 
> byte-code, and the appropriate native-code versions if 
> natively compiled.

That is a simple enough idea that could work for pure O'Caml code.

Will it handle different versions of the same library? Can I have two
versions of the PCRE simultaneously?

How can we handle Ocaml-code that depend on C-libraries, like PCRE? Any
ideas?

/mattias

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: possible solution to [Caml-list] The DLL-hell of O'Caml

[Dave Mason]

>>>>> On Tue, 5 Mar 2002 08:58:31 +0100, "Mattias Waldau" <mattias.waldau@abc.se> said:

>> ``dynamic load via http (including https) with local cache''

> That is a simple enough idea that could work for pure O'Caml code.

> How can we handle Ocaml-code that depend on C-libraries, like PCRE? 
> Any ideas?

For our purposes, the Ocaml code could call the infrastructure to get
it to load the C library before it tries to access the C code, or to
build the original executable with the libraries we need.

> Will it handle different versions of the same library? Can I have
> two versions of the PCRE simultaneously?

I think that would be very difficult to arrange.  Again, for our
purposes, the downloadable modules would come from single site, so we
will arrange for the libraries to be consistent.

../Dave
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] The DLL-hell of O'Caml

[Jacques Garrigue]

Hi,

> On my linux-machine I use the CDK with ocaml-3.03ALPHA.

Then you should upgrade. As shown by its name, this is an alpha release,
whose compatibility with future releases is not guaranteed.
If I remember correctly, it names DLLs differently from 3.04,
and you should scrape it away before installing 3.04, lest you get
into lots of trouble.

> This is the installation history:
> 1. Needs Ocaml 3.03 -> updated CDK using CVS (there are no tarballs
> but I have a broadband-connection).
> 2. Ocamlnet missing -> downloaded ocamlnet-0.92
> 3. Failed to install ocamlnet, since the PCRE in the CDK seems to
> be to old.
> 4. Installed a new PCRE.
> 5. Tried to install ocamlnet again, however failed with 
> 	"unbound type constructor Mimestring.s_param"
> 
>    Probably something else is too old...I got bored...
>    ...never got back to try to install WDialog....

I sympathize...

> I know 3 solutions to the problem:
> 1. CDK
> 2. Gerd invented findlib to solve the above problem.
> 3. Adding good packages to the standard distribution,
>    so that mostly packages doesn't depend on other
>    packages, but on packages in the standard
>    distribution. (But very few if any new packages
>    are added to the distribution.)

Personally, I think that the combination of (1) and (2) could help a
lot. Make the powerful findlib a standard. Then it would probably end
up one day in the standard distribution...
I would add also the role of the O'Caml link database, which is
particularly nice as authors can update themselve their information.

Adding packages to the standard distribution could easily make it
huge. It is already not so small. Note also the tendancy to have n
libraries for the same goal, which just reflects how people are
interested just as much in how it is done as what is done.

(I know, I should start by using findlib for my libraries.
But like everybody else, I'm lazy...)

It will still be difficult to get things as smooth as in Perl: strong
typing is good at detecting problem early, so early that it won't let
you compile some code because of a single function you don't need
anyway.
Here is a more theoretical problem: how to determine the cause of a
compilation error, whether this is some change in the language, or in
a library, and which library.

Cheers,

Jacques Garrigue
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners