ocamlbuild

ocamlbuild

[skaller]

hi .. I've got an ocamlbuild project which is hanging
forever. Top shows 100% CPU and 2% memory. Any ideas how 
this could happen?

I have a suspicion ocamlbuild can't handle directories
properly..

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] ocamlbuild

[Nicolas Pouillard]

Excerpts from skaller's message of Wed Sep 05 10:36:46 +0200 2007:
> hi .. I've got an ocamlbuild project which is hanging
> forever. Top shows 100% CPU and 2% memory. Any ideas how 
> this could happen?
> 
> I have a suspicion ocamlbuild can't handle directories
> properly..
> 

Are  you  on  windows? If yes what kind of environment you have and what OCaml
you have.

If  it's  Unix,  have  you symbolic links that make cycles (this case has been
handled, but I cannot see something else).

Can you also try it with -no-sanitize.

-- 
Nicolas Pouillard aka Ertai

Re: [Caml-list] ocamlbuild

[skaller]

On Mon, 2007-09-17 at 13:56 +0200, Nicolas Pouillard wrote:
> Excerpts from skaller's message of Wed Sep 05 10:36:46 +0200 2007:
> > hi .. I've got an ocamlbuild project which is hanging
> > forever. Top shows 100% CPU and 2% memory. Any ideas how 
> > this could happen?
> > 
> > I have a suspicion ocamlbuild can't handle directories
> > properly..
> > 
> 
> Are  you  on  windows? If yes what kind of environment you have and what OCaml
> you have.
> 
> If  it's  Unix,  have  you symbolic links that make cycles (this case has been
> handled, but I cannot see something else).
> 
> Can you also try it with -no-sanitize.

Our build system is portable, we need Ocamlbuild code to also be
entirely portable. We know at present it doesn't work on Windows.
But the problem is on Linux.

Now, I have no idea what the problem was because I deleted the entire
workspace and re-created it from the repository and it went away.
We don't make any symbolic links because the script is portable,
and you can't make them on Windows.

So just keep an eye out for someone having a similar problem.

Erick reports another issue: ocamlbuild tries to rebuild 
libraries a code depends on -- but not the system libraries.

The idea of building a library is to *avoid* having to do
any dependency checking on the module level. Is there some
way to fix this? I.e, to 'make' a library, and then have
it treated as a 'system' library.


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] ocamlbuild

[Nicolas Pouillard]

Excerpts from skaller's message of Mon Sep 17 18:21:34 +0200 2007:
> On Mon, 2007-09-17 at 13:56 +0200, Nicolas Pouillard wrote:
> > Excerpts from skaller's message of Wed Sep 05 10:36:46 +0200 2007:
> > > hi .. I've got an ocamlbuild project which is hanging
> > > forever. Top shows 100% CPU and 2% memory. Any ideas how 
> > > this could happen?
> > > 
> > > I have a suspicion ocamlbuild can't handle directories
> > > properly..
> > > 
> > 
> > Are  you  on  windows? If yes what kind of environment you have and what OCaml
> > you have.
> > 
> > If  it's  Unix,  have  you symbolic links that make cycles (this case has been
> > handled, but I cannot see something else).
> > 
> > Can you also try it with -no-sanitize.

Have you try this, since this part can be very time consuming.

> Our build system is portable, we need Ocamlbuild code to also be
> entirely portable. We know at present it doesn't work on Windows.
> But the problem is on Linux.
> 
> Now, I have no idea what the problem was because I deleted the entire
> workspace and re-created it from the repository and it went away.
> We don't make any symbolic links because the script is portable,
> and you can't make them on Windows.
> 
> So just keep an eye out for someone having a similar problem.

Is you source tree very big?

> Erick reports another issue: ocamlbuild tries to rebuild 
> libraries a code depends on -- but not the system libraries.

Yes it even don't know that's system libraries.

> The idea of building a library is to *avoid* having to do
> any dependency checking on the module level. Is there some
> way to fix this? I.e, to 'make' a library, and then have
> it treated as a 'system' library.

Yes you can do it by showing it the directory with compiled files.
Then you have to explain it that all these .cmo,.cmi... are not junk files.

Example of using a library in /usr/lib/ocaml/foolib

$ ln -s /usr/lib/ocaml/foolib foolib

$ cat _tags
"foolib": include, precious

The first tag (include) is equivalent to a `-I foolib' given to ocamlbuild. The second tag (precious, also known as not_hygienic) tell him to don't report these files as junk and propose their deletion.

Regards,
-- 
Nicolas Pouillard aka Ertai

Re: [Caml-list] ocamlbuild

[skaller]

On Tue, 2007-09-25 at 13:45 +0200, Nicolas Pouillard wrote:
> Excerpts from skaller's message of Mon Sep 17 18:21:34 +0200 2007:

> Have you try this, since this part can be very time consuming.

Since the problem went away when I deleted the whole directory
and recreated it, I can't debug the problem now, sorry ;(

> > So just keep an eye out for someone having a similar problem.
> 
> Is you source tree very big?

No, not particularly: 10 subdirectories, the biggest contains
95 *.ml files and similar number of *.mli files, others
probably 5-10 files each.

> > Erick reports another issue: ocamlbuild tries to rebuild 
> > libraries a code depends on -- but not the system libraries.
> 
> Yes it even don't know that's system libraries.

Sorry, I can't parse that. Ocamlbuild doesn't try to build system
libraries .. how does it know what libraries are system libraries
which are not to be rebuilt, and which one are application 
libraries which require rebuilding?


> > The idea of building a library is to *avoid* having to do
> > any dependency checking on the module level. Is there some
> > way to fix this? I.e, to 'make' a library, and then have
> > it treated as a 'system' library.
> 
> Yes you can do it by showing it the directory with compiled files.
> Then you have to explain it that all these .cmo,.cmi... are not junk files.

They are though, and indeed the *.ml and *.mli files are also junk.
We just want the *.cm(x)a library to be used, not rebuilt.

To explain better: we go to directory liba, and we use Ocamlbuild
to build 

	liba.cmxa

Ok, now we go to directory prog, which needs modules from liba,
and we tell it to build

	prog

In this second step, ocamlbuild is not allowed to check if the
library is up to date. It just uses the binary library, without
examining any source code.

Because, we already used Ocamlbuild to build it, and it is
expensive to rescan all the source code for liba, and even
if it is found to be out of date, Ocamlbuild's current invocation
cannot rebuild it. It could be installed in read-only location
for example.

So even though ocamldep will say prog depends on modules from
liba, we don't care. They're excluded from consideration.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] ocamlbuild

[Nicolas Pouillard]

Excerpts from skaller's message of Tue Sep 25 16:19:44 +0200 2007:
> On Tue, 2007-09-25 at 13:45 +0200, Nicolas Pouillard wrote:
> > Excerpts from skaller's message of Mon Sep 17 18:21:34 +0200 2007:
> 
> > Have you try this, since this part can be very time consuming.
> 
> Since the problem went away when I deleted the whole directory
> and recreated it, I can't debug the problem now, sorry ;(
> 
> > > So just keep an eye out for someone having a similar problem.
> > 
> > Is you source tree very big?
> 
> No, not particularly: 10 subdirectories, the biggest contains
> 95 *.ml files and similar number of *.mli files, others
> probably 5-10 files each.
> 
> > > Erick reports another issue: ocamlbuild tries to rebuild 
> > > libraries a code depends on -- but not the system libraries.
> > 
> > Yes it even don't know that's system libraries.
> 
> Sorry, I can't parse that. Ocamlbuild doesn't try to build system
> libraries .. how does it know what libraries are system libraries
> which are not to be rebuilt, and which one are application 
> libraries which require rebuilding?

There is an exception for stdlib.cma, otherwise it's rebuilt if needed.

Note  also  that  ocamlbuild  behave  differently  than  make, since when make
encounter  the  target file, it tries to build it with other rules if there is
no  such  rule it keeps the file. However when ocamlbuild encounter the target
file  (in  the source dir) it stops here. This approach works because there is
a  separation  between  the  source dir and the build dir. So in conclusion if
ocamlbuild  found  a .cmo that is needed in the source dir (tagged precious to
avoid the sanitize step) it will use it.

> > > The idea of building a library is to *avoid* having to do
> > > any dependency checking on the module level. Is there some
> > > way to fix this? I.e, to 'make' a library, and then have
> > > it treated as a 'system' library.
> > 
> > Yes you can do it by showing it the directory with compiled files.
> > Then you have to explain it that all these .cmo,.cmi... are not junk files.
> 
> They are though, and indeed the *.ml and *.mli files are also junk.
> We just want the *.cm(x)a library to be used, not rebuilt.
> 
> To explain better: we go to directory liba, and we use Ocamlbuild
> to build 
> 
>     liba.cmxa
> 
> Ok, now we go to directory prog, which needs modules from liba,
> and we tell it to build
> 
>     prog
> 
> In this second step, ocamlbuild is not allowed to check if the
> library is up to date. It just uses the binary library, without
> examining any source code.
> 
> Because, we already used Ocamlbuild to build it, and it is
> expensive to rescan all the source code for liba, and even
> if it is found to be out of date, Ocamlbuild's current invocation
> cannot rebuild it. It could be installed in read-only location
> for example.
> 
> So even though ocamldep will say prog depends on modules from
> liba, we don't care. They're excluded from consideration.

To do so, in prog you can type...

$ ln -s /path/to/liba/_build liba
$ echo '"liba": include, precious' >> _tags

HTH,
-- 
Nicolas Pouillard aka Ertai

ocamlbuild

[skaller]

Does Ocamlbuild work on Windows *transparently*?

To do this, it must read filenames written with / in it,
but use filenames with \ replacing the /. I'd say this is
mandatory, so the same build control file will work on
all platforms. Then you need to document that Unix filenames
must be used, even on Windows.

Note: although *Ocaml* basenames cannot have spaces in them,
the directories that contain them can. So Ocamlbuild must be
sure to quote all filenames, in case they have spaces.
This includes allowing quotes in the input specs.
Unix file systems allow spaces in filenames too, and many
modern Desktops will have 'folders' with spaces in
(if a 'doze or fruit user gets Ubuntu in their office).

I warn this is a serious PITA if you're building Windows
*Native* code, but hosting the build on Cygwin, because
executables have to use Unix filenames but arguments have
to use Windows filenames, so you have to carefully keep
track of which tool is running what.

It helps if standard tools translate internally (eg ocamlc).
However any build script has to allow for foreign tools
(eg Felix uses Dypgen as a parser).

A serious nasty here is if you have to use silly names
on windows for the tools, eg building Felix:

	C:\Program Files (x86)\Ocaml\ocamlc

and here the spaces, parentheses and C: drive
letter are mandatory and must be handled correctly
as a native filename. On Windows, Ocaml isn't necessarily
even in the path -- because on Windows XP64 I have multiple
Ocaml versions and I need to *specify* which one to use,
no accidents from PATH please! (Well actually, we do use
the PATH but it leads to a mess).


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] ocamlbuild

[Adrien]

Hi,

I can't answer your question on ocamlbuild but in order to install
some modules under mingw+msys, I had to make ocamlfind work correctly
under windows and learned a few things.


 > To do this, it must read filenames written with / in it,
 > but use filenames with \ replacing the /. I'd say this is
 > mandatory, so the same build control file will work on
 > all platforms. Then you need to document that Unix filenames
 > must be used, even on Windows.

It seems mingw doesn't care that much as I've seen it use paths with
mixed forward and backward slashes.
For the paths, it handles both windows and unix paths.
In mingw, I use paths such as these : /e/GNU/ocaml/lib
But in the application I am developping (as a toy), I need real
windows paths and there is no problem. You only have to change '\' to
"\\" to avoid invalid escapes (though I prefer this to a valid
one...).

I've launched a toplevel and tried Unix.stat on some files:

# Unix.stat "E:\\DocsAndSettings\\Administrateur\\.sancho\\sancho.pref";;
- : Unix.stats =
{Unix.st_dev = 4; Unix.st_ino = 0; Unix.st_kind = Unix.S_REG;
 Unix.st_perm = 438; Unix.st_nlink = 1; Unix.st_uid = 0; Unix.st_gid = 0;
 Unix.st_rdev = 4; Unix.st_size = 5640; Unix.st_atime = 1172179361.;
 Unix.st_mtime = 1172187293.; Unix.st_ctime = 1172179361.}

# Unix.stat "E:\\DocsAndSettings\\Administrateur\\Application
Data\\K-Meleon\\registry.dat";;
- : Unix.stats =
{Unix.st_dev = 4; Unix.st_ino = 0; Unix.st_kind = Unix.S_REG;
 Unix.st_perm = 438; Unix.st_nlink = 1; Unix.st_uid = 0; Unix.st_gid = 0;
 Unix.st_rdev = 4; Unix.st_size = 1181; Unix.st_atime = 1175886880.;
 Unix.st_mtime = 1186381757.; Unix.st_ctime = 1175886880.}


> Note: although *Ocaml* basenames cannot have spaces in them,
> the directories that contain them can. So Ocamlbuild must be
> sure to quote all filenames, in case they have spaces.
> This includes allowing quotes in the input specs.
> Unix file systems allow spaces in filenames too, and many
> modern Desktops will have 'folders' with spaces in
> (if a 'doze or fruit user gets Ubuntu in their office).

When making ocamlfind work under windows, on of the problems was with
Filename.quote. In fact, not exactly but when I removed the calls to
it, the problem disappeared. I tried to quote manually and realized it
required one more quotation and I ended with several \"\".
I think it would be better if Unix.command quoted on its own.

> I warn this is a serious PITA if you're building Windows
> *Native* code, but hosting the build on Cygwin, because
> executables have to use Unix filenames but arguments have
> to use Windows filenames, so you have to carefully keep
> track of which tool is running what.

I've been a mldonkey user for years and always wanted to compile
mldonkey myself for windows.

I've started with (pure, i.e. no mingw) Cygwin and everything built
quickly. However I had a complete unix system inside my computer :
hundreds of MBs used and most important tens of thousand of files
which made everything slower. And then I got the problem with unix vs.
windows filenames and when I saw it, it was a terrible mess.

I later tried with mingw and it wasn't good at then. I tried again
with mingw a few weeks ago and everything works.
I know ocaml's documentation says mingw+msys doesn't work because of
the limited command-line length of msys but this is untrue : the
limitation comes from windows's cmd.exe which is etwa 2000 under
windows 2000 but around 8000 for XP (see the very end of this page
which is really great btw http://mldonkey.sourceforge.net/Windows )
In fact, what's so great with mingw is that you can use what has been
coded for unix while keeping the windows features (see dev-cpp which
is a C IDE on and for windows).


> A serious nasty here is if you have to use silly names
> on windows for the tools, eg building Felix:
>
>        C:\Program Files (x86)\Ocaml\ocamlc
>
> and here the spaces, parentheses and C: drive
> letter are mandatory and must be handled correctly
> as a native filename. On Windows, Ocaml isn't necessarily
> even in the path -- because on Windows XP64 I have multiple
> Ocaml versions and I need to *specify* which one to use,
> no accidents from PATH please! (Well actually, we do use
> the PATH but it leads to a mess).

Thanks to the wiki page spiralvoice made for mldonkey compilation
under windows, I created a .profile file in msys's ~ with this:
  export PATH=/e/GNU/ocaml/bin:/e/GNU/ocaml/lib:$PATH
  export CAMLLIB=/e/GNU/ocaml/lib
  export OCAMLLIB=/e/GNU/ocaml/lib
  export CAMLP4LIB=/e/GNU/ocaml/lib/camlp4

And had not problem. Adding other entries solved another problem with
ocamlfind as it relies on environment variables.
OCaml's "make install" should update them itself.


Btw, am I wrong thinking you use the MSVC port ?



As a side note, if you are concerned about spaces on windows, you can
rename "Program Files", "Documents And Settings" and move the Temp
folder.
Here I have "Programs", "DocsAndSettings" and temp is in
"LocalSettings\Temp" (though "Local Settings" still exist ans is being
used but at least by everything).
You have to edit your windows CDs with a tool such as nlite
(nliteos.com) (actually you can do it after windows installation but I
wouldn't really advise it ;) ).



---

Adrien Nader

Re: [Caml-list] ocamlbuild

[Adrien]

[-- Attachment #1: Type: text/plain, Size: 343 bytes --]

I was debugguing a regular-expression and noticed that \" in a string
remained as-is and this somehow seems linked to my quotation problem :
  # let a="\"";;
  val a : string = "\""

  # let a="\034";;
  val a : string = "\""

Is it expected behaviour? :o
I can't get such a quote in any string.

(self-compiled ocaml 3.09.3 under mingw+msys)

[-- Attachment #2: Type: text/html, Size: 529 bytes --]

RE: [Caml-list] ocamlbuild

[David Allsopp]

> I was debugguing a regular-expression and noticed that \" in a string
> remained as-is and this somehow seems linked to my quotation problem :
>  # let a="\"";;
>  val a : string = "\""

What you're seeing here is the pretty-printer in the toploop which passes
string values through String.escaped:

	# let a = "\"";;
	val a : string = "\""
	# String.length a;;
	- : int = 1
	# Printf.printf "%s\n" a;;
	"
	- : unit = ()

I had the same problem with findlib compiling under the MinGW port and had
to remove Filename.quote in the same way...


David

Re: [Caml-list] ocamlbuild

[Christophe TROESTLER]

On Mon, 6 Aug 2007 11:34:01 +0200, Adrien wrote:
> 
> I was debugguing a regular-expression and noticed that \" in a string
> remained as-is and this somehow seems linked to my quotation problem :
>   # let a="\"";;
>   val a : string = "\""
> 
>   # let a="\034";;
>   val a : string = "\""
> 
> Is it expected behaviour? :o

Yes.  Your misunderstanding seems to come from the fact that it is
displayed as \" because it is quoted, this the usual quoting rules
apply.  But

# "\"".[0];;
- : char = '"'

ChriS

Re: [Caml-list] ocamlbuild

[Aleksey Nogin]

On 05.08.2007 16:19, skaller wrote:

> To do this, it must read filenames written with / in it,
> but use filenames with \ replacing the /. I'd say this is
> mandatory, so the same build control file will work on
> all platforms. Then you need to document that Unix filenames
> must be used, even on Windows.
> 
> Note: although *Ocaml* basenames cannot have spaces in them,
> the directories that contain them can. So Ocamlbuild must be
> sure to quote all filenames, in case they have spaces.
> This includes allowing quotes in the input specs.
> Unix file systems allow spaces in filenames too, and many
> modern Desktops will have 'folders' with spaces in
> (if a 'doze or fruit user gets Ubuntu in their office).

I do not know if ocamlbuild addresses these, but I know that OMake does ;-).

Aleksey

Re: [Caml-list] ocamlbuild

[Nicolas Pouillard]

Excerpts from skaller's message of Mon Aug 06 01:19:37 +0200 2007:
> Does Ocamlbuild work on Windows *transparently*?

We  tried  to  do our best at the design step of ocamlbuild to avoid as much as
possible problems with windows and especially with pathname quoting.

Ocamlbuild  helps  you  to  keep  your  command structured to enforce a correct
escaping.  But  it's still up to the user that customize ocamlbuild (by writing
a  plugin;  this  is  needed  for  adding  new rules, flags...). There is still
problems with some standard rules that use some unix commands.

So,  ocamlbuild  is  still  partially supported on windows and therefore is not
ideal  solution  for  building  ocaml  code under windows. However we hope that
this kind of problems will be solved in a future version.

> To do this, it must read filenames written with / in it,
> but use filenames with \ replacing the /. I'd say this is
> mandatory, so the same build control file will work on
> all platforms. Then you need to document that Unix filenames
> must be used, even on Windows.
> 
> Note: although *Ocaml* basenames cannot have spaces in them,
> the directories that contain them can. So Ocamlbuild must be
> sure to quote all filenames, in case they have spaces.
> This includes allowing quotes in the input specs.
> Unix file systems allow spaces in filenames too, and many
> modern Desktops will have 'folders' with spaces in
> (if a 'doze or fruit user gets Ubuntu in their office).
> 
> I warn this is a serious PITA if you're building Windows
> *Native* code, but hosting the build on Cygwin, because
> executables have to use Unix filenames but arguments have
> to use Windows filenames, so you have to carefully keep
> track of which tool is running what.
> 
> It helps if standard tools translate internally (eg ocamlc).
> However any build script has to allow for foreign tools
> (eg Felix uses Dypgen as a parser).
> 
> A serious nasty here is if you have to use silly names
> on windows for the tools, eg building Felix:
> 
>     C:\Program Files (x86)\Ocaml\ocamlc
> 
> and here the spaces, parentheses and C: drive
> letter are mandatory and must be handled correctly
> as a native filename. On Windows, Ocaml isn't necessarily
> even in the path -- because on Windows XP64 I have multiple
> Ocaml versions and I need to *specify* which one to use,
> no accidents from PATH please! (Well actually, we do use
> the PATH but it leads to a mess).
> 

We're  aware  of  that.  However  if  someone have a clear specification of the
quoting  system  on  windows  (the recomended one at least) and the one used by
Cygwin I'm interested.


-- 
Nicolas Pouillard aka Ertai