ocamlbuild

ocamlbuild

[skaller]

hi .. I've got an ocamlbuild project which is hanging
forever. Top shows 100% CPU and 2% memory. Any ideas how 
this could happen?

I have a suspicion ocamlbuild can't handle directories
properly..

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] ocamlbuild

[Nicolas Pouillard]

Excerpts from skaller's message of Wed Sep 05 10:36:46 +0200 2007:
> hi .. I've got an ocamlbuild project which is hanging
> forever. Top shows 100% CPU and 2% memory. Any ideas how 
> this could happen?
> 
> I have a suspicion ocamlbuild can't handle directories
> properly..
> 

Are  you  on  windows? If yes what kind of environment you have and what OCaml
you have.

If  it's  Unix,  have  you symbolic links that make cycles (this case has been
handled, but I cannot see something else).

Can you also try it with -no-sanitize.

-- 
Nicolas Pouillard aka Ertai

Re: [Caml-list] ocamlbuild

[skaller]

On Mon, 2007-09-17 at 13:56 +0200, Nicolas Pouillard wrote:
> Excerpts from skaller's message of Wed Sep 05 10:36:46 +0200 2007:
> > hi .. I've got an ocamlbuild project which is hanging
> > forever. Top shows 100% CPU and 2% memory. Any ideas how 
> > this could happen?
> > 
> > I have a suspicion ocamlbuild can't handle directories
> > properly..
> > 
> 
> Are  you  on  windows? If yes what kind of environment you have and what OCaml
> you have.
> 
> If  it's  Unix,  have  you symbolic links that make cycles (this case has been
> handled, but I cannot see something else).
> 
> Can you also try it with -no-sanitize.

Our build system is portable, we need Ocamlbuild code to also be
entirely portable. We know at present it doesn't work on Windows.
But the problem is on Linux.

Now, I have no idea what the problem was because I deleted the entire
workspace and re-created it from the repository and it went away.
We don't make any symbolic links because the script is portable,
and you can't make them on Windows.

So just keep an eye out for someone having a similar problem.

Erick reports another issue: ocamlbuild tries to rebuild 
libraries a code depends on -- but not the system libraries.

The idea of building a library is to *avoid* having to do
any dependency checking on the module level. Is there some
way to fix this? I.e, to 'make' a library, and then have
it treated as a 'system' library.


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] ocamlbuild

[Nicolas Pouillard]

Excerpts from skaller's message of Mon Sep 17 18:21:34 +0200 2007:
> On Mon, 2007-09-17 at 13:56 +0200, Nicolas Pouillard wrote:
> > Excerpts from skaller's message of Wed Sep 05 10:36:46 +0200 2007:
> > > hi .. I've got an ocamlbuild project which is hanging
> > > forever. Top shows 100% CPU and 2% memory. Any ideas how 
> > > this could happen?
> > > 
> > > I have a suspicion ocamlbuild can't handle directories
> > > properly..
> > > 
> > 
> > Are  you  on  windows? If yes what kind of environment you have and what OCaml
> > you have.
> > 
> > If  it's  Unix,  have  you symbolic links that make cycles (this case has been
> > handled, but I cannot see something else).
> > 
> > Can you also try it with -no-sanitize.

Have you try this, since this part can be very time consuming.

> Our build system is portable, we need Ocamlbuild code to also be
> entirely portable. We know at present it doesn't work on Windows.
> But the problem is on Linux.
> 
> Now, I have no idea what the problem was because I deleted the entire
> workspace and re-created it from the repository and it went away.
> We don't make any symbolic links because the script is portable,
> and you can't make them on Windows.
> 
> So just keep an eye out for someone having a similar problem.

Is you source tree very big?

> Erick reports another issue: ocamlbuild tries to rebuild 
> libraries a code depends on -- but not the system libraries.

Yes it even don't know that's system libraries.

> The idea of building a library is to *avoid* having to do
> any dependency checking on the module level. Is there some
> way to fix this? I.e, to 'make' a library, and then have
> it treated as a 'system' library.

Yes you can do it by showing it the directory with compiled files.
Then you have to explain it that all these .cmo,.cmi... are not junk files.

Example of using a library in /usr/lib/ocaml/foolib

$ ln -s /usr/lib/ocaml/foolib foolib

$ cat _tags
"foolib": include, precious

The first tag (include) is equivalent to a `-I foolib' given to ocamlbuild. The second tag (precious, also known as not_hygienic) tell him to don't report these files as junk and propose their deletion.

Regards,
-- 
Nicolas Pouillard aka Ertai

Re: [Caml-list] ocamlbuild

[skaller]

On Tue, 2007-09-25 at 13:45 +0200, Nicolas Pouillard wrote:
> Excerpts from skaller's message of Mon Sep 17 18:21:34 +0200 2007:

> Have you try this, since this part can be very time consuming.

Since the problem went away when I deleted the whole directory
and recreated it, I can't debug the problem now, sorry ;(

> > So just keep an eye out for someone having a similar problem.
> 
> Is you source tree very big?

No, not particularly: 10 subdirectories, the biggest contains
95 *.ml files and similar number of *.mli files, others
probably 5-10 files each.

> > Erick reports another issue: ocamlbuild tries to rebuild 
> > libraries a code depends on -- but not the system libraries.
> 
> Yes it even don't know that's system libraries.

Sorry, I can't parse that. Ocamlbuild doesn't try to build system
libraries .. how does it know what libraries are system libraries
which are not to be rebuilt, and which one are application 
libraries which require rebuilding?


> > The idea of building a library is to *avoid* having to do
> > any dependency checking on the module level. Is there some
> > way to fix this? I.e, to 'make' a library, and then have
> > it treated as a 'system' library.
> 
> Yes you can do it by showing it the directory with compiled files.
> Then you have to explain it that all these .cmo,.cmi... are not junk files.

They are though, and indeed the *.ml and *.mli files are also junk.
We just want the *.cm(x)a library to be used, not rebuilt.

To explain better: we go to directory liba, and we use Ocamlbuild
to build 

	liba.cmxa

Ok, now we go to directory prog, which needs modules from liba,
and we tell it to build

	prog

In this second step, ocamlbuild is not allowed to check if the
library is up to date. It just uses the binary library, without
examining any source code.

Because, we already used Ocamlbuild to build it, and it is
expensive to rescan all the source code for liba, and even
if it is found to be out of date, Ocamlbuild's current invocation
cannot rebuild it. It could be installed in read-only location
for example.

So even though ocamldep will say prog depends on modules from
liba, we don't care. They're excluded from consideration.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] ocamlbuild

[Nicolas Pouillard]

Excerpts from skaller's message of Tue Sep 25 16:19:44 +0200 2007:
> On Tue, 2007-09-25 at 13:45 +0200, Nicolas Pouillard wrote:
> > Excerpts from skaller's message of Mon Sep 17 18:21:34 +0200 2007:
> 
> > Have you try this, since this part can be very time consuming.
> 
> Since the problem went away when I deleted the whole directory
> and recreated it, I can't debug the problem now, sorry ;(
> 
> > > So just keep an eye out for someone having a similar problem.
> > 
> > Is you source tree very big?
> 
> No, not particularly: 10 subdirectories, the biggest contains
> 95 *.ml files and similar number of *.mli files, others
> probably 5-10 files each.
> 
> > > Erick reports another issue: ocamlbuild tries to rebuild 
> > > libraries a code depends on -- but not the system libraries.
> > 
> > Yes it even don't know that's system libraries.
> 
> Sorry, I can't parse that. Ocamlbuild doesn't try to build system
> libraries .. how does it know what libraries are system libraries
> which are not to be rebuilt, and which one are application 
> libraries which require rebuilding?

There is an exception for stdlib.cma, otherwise it's rebuilt if needed.

Note  also  that  ocamlbuild  behave  differently  than  make, since when make
encounter  the  target file, it tries to build it with other rules if there is
no  such  rule it keeps the file. However when ocamlbuild encounter the target
file  (in  the source dir) it stops here. This approach works because there is
a  separation  between  the  source dir and the build dir. So in conclusion if
ocamlbuild  found  a .cmo that is needed in the source dir (tagged precious to
avoid the sanitize step) it will use it.

> > > The idea of building a library is to *avoid* having to do
> > > any dependency checking on the module level. Is there some
> > > way to fix this? I.e, to 'make' a library, and then have
> > > it treated as a 'system' library.
> > 
> > Yes you can do it by showing it the directory with compiled files.
> > Then you have to explain it that all these .cmo,.cmi... are not junk files.
> 
> They are though, and indeed the *.ml and *.mli files are also junk.
> We just want the *.cm(x)a library to be used, not rebuilt.
> 
> To explain better: we go to directory liba, and we use Ocamlbuild
> to build 
> 
>     liba.cmxa
> 
> Ok, now we go to directory prog, which needs modules from liba,
> and we tell it to build
> 
>     prog
> 
> In this second step, ocamlbuild is not allowed to check if the
> library is up to date. It just uses the binary library, without
> examining any source code.
> 
> Because, we already used Ocamlbuild to build it, and it is
> expensive to rescan all the source code for liba, and even
> if it is found to be out of date, Ocamlbuild's current invocation
> cannot rebuild it. It could be installed in read-only location
> for example.
> 
> So even though ocamldep will say prog depends on modules from
> liba, we don't care. They're excluded from consideration.

To do so, in prog you can type...

$ ln -s /path/to/liba/_build liba
$ echo '"liba": include, precious' >> _tags

HTH,
-- 
Nicolas Pouillard aka Ertai

ocamlbuild

[skaller]

Does Ocamlbuild work on Windows *transparently*?

To do this, it must read filenames written with / in it,
but use filenames with \ replacing the /. I'd say this is
mandatory, so the same build control file will work on
all platforms. Then you need to document that Unix filenames
must be used, even on Windows.

Note: although *Ocaml* basenames cannot have spaces in them,
the directories that contain them can. So Ocamlbuild must be
sure to quote all filenames, in case they have spaces.
This includes allowing quotes in the input specs.
Unix file systems allow spaces in filenames too, and many
modern Desktops will have 'folders' with spaces in
(if a 'doze or fruit user gets Ubuntu in their office).

I warn this is a serious PITA if you're building Windows
*Native* code, but hosting the build on Cygwin, because
executables have to use Unix filenames but arguments have
to use Windows filenames, so you have to carefully keep
track of which tool is running what.

It helps if standard tools translate internally (eg ocamlc).
However any build script has to allow for foreign tools
(eg Felix uses Dypgen as a parser).

A serious nasty here is if you have to use silly names
on windows for the tools, eg building Felix:

	C:\Program Files (x86)\Ocaml\ocamlc

and here the spaces, parentheses and C: drive
letter are mandatory and must be handled correctly
as a native filename. On Windows, Ocaml isn't necessarily
even in the path -- because on Windows XP64 I have multiple
Ocaml versions and I need to *specify* which one to use,
no accidents from PATH please! (Well actually, we do use
the PATH but it leads to a mess).


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net