[Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[Soegtrop, Michael]

Dear David,

I just managed now to test the flexdll dependency fix in 4.03.0 you mentioned in your post from March 15th:

>  1. Ensure FlexDLL sources are in flexdll/ in the OCaml build tree
>	1a. If you're building from a git clone, issue git submodule update --init
>	1b. If you're building from a sources tarball, issue git clone > https://github.com/alainfrisch/flexdll.git
> 2. Follow usual configuration steps in config/
> 3. make -f Makefile.nt flexdll world opt opt.opt install

It is much better than it was, but I think there is still a minor flaw in it. When compiling byterun/win32.c (from asmrun) in the make opt phase, it doesn't find flexdll.h because gcc doesn't have an -I ../flexdll. If I copy flexdll.h to a system include directory it works fine. Is it possible that you had a flexdll.h somewhere and the build took it from there rather than from the flexdll subfolder when you tried this?

Also I think specific versions of OCaml should copy specific versions of flexdll. A git clone of the trunk version might lead to unexpected effects in the future and it might be hard to reconstruct older OCaml versions later.

Best regards,

Michael
Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

RE: [Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[David Allsopp]

Michael Soegtrop wrote:
> Dear David,
> 
> I just managed now to test the flexdll dependency fix in 4.03.0 you
> mentioned in your post from March 15th:
> 
> >  1. Ensure FlexDLL sources are in flexdll/ in the OCaml build tree
> >	1a. If you're building from a git clone, issue git submodule update
> --init
> >	1b. If you're building from a sources tarball, issue git clone >
> >https://github.com/alainfrisch/flexdll.git
> > 2. Follow usual configuration steps in config/  3. make -f Makefile.nt
> >flexdll world opt opt.opt install
> 
> It is much better than it was, but I think there is still a minor flaw in
> it. When compiling byterun/win32.c (from asmrun) in the make opt phase, it
> doesn't find flexdll.h because gcc doesn't have an -I ../flexdll. If I
> copy flexdll.h to a system include directory it works fine. Is it possible
> that you had a flexdll.h somewhere and the build took it from there rather
> than from the flexdll subfolder when you tried this?

I'm not convinced this is a fundamental flaw - I think it more likely that you have an existing flexlink in PATH, which will cause config/Makefile to pick up the wrong include directory. I think it likely if you study the failing command during make -f Makefile.nt opt you will see a -I pointing to your PATH'd flexlink.exe directory.

Does it work if you remove your system included flexdll.h and alter config/Makefile to delete the line FLEXDIR:= (a la https://github.com/dra27/opam-repository/blob/next-windows/packages/ocaml/ocaml.4.03.0/files/win.sh#L9)?

> Also I think specific versions of OCaml should copy specific versions of
> flexdll.

Yes and no - this is a bit like insisting that a given OCaml version be built with a specific version of gcc. At present, OCaml 3.11 (when flexlink was introduced) still builds just fine with trunk flexdll.

> A git clone of the trunk version might lead to unexpected effects
> in the future and it might be hard to reconstruct older OCaml versions
> later.

A git clone of trunk will use the version of flexdll configured at the time - that's how git submodules work (because it stores a commit SHA, not a branch).

Best,


D

RE: [Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[Soegtrop, Michael]

Dear David,

> I'm not convinced this is a fundamental flaw - I think it more likely that you have an existing flexlink
> in PATH, which will cause config/Makefile to pick up the wrong include directory. I think it likely if
> you study the failing command during make -f Makefile.nt opt you will see a -I pointing to your
> PATH'd flexlink.exe directory.

I have no system installation of flexdll. The compilation fails in my case, because there is no flexdll.h at all, except the one in the flexdll folder of ocaml. The only -I option when win32.c is compiled in asmrun is -I../byterun.

Does the flexdll make in ocamlc a local build of flexdll, or does it install flexdll somewhere globally? In the first case I think a -I../flexdll is missing, in the second case I guess I might miss some install prefixes.

> A git clone of trunk will use the version of flexdll configured at the time - that's how git submodules > work (because it stores a commit SHA, not a branch).

Does this also work if I get the ocaml sources as a tar ball rather than via a git clone?

Best regards,

Michael
Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

RE: [Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[David Allsopp]

Michael Soegtrop wrote:
> Dear David,
> 
> > I'm not convinced this is a fundamental flaw - I think it more likely
> > that you have an existing flexlink in PATH, which will cause
> > config/Makefile to pick up the wrong include directory. I think it
> > likely if you study the failing command during make -f Makefile.nt opt
> you will see a -I pointing to your PATH'd flexlink.exe directory.
> 
> I have no system installation of flexdll. The compilation fails in my
> case, because there is no flexdll.h at all, except the one in the flexdll
> folder of ocaml. The only -I option when win32.c is compiled in asmrun is
> -I../byterun.

Something weird is going on - you should not be seeing that! As https://github.com/ocaml/ocaml/blob/trunk/asmrun/Makefile.nt#L71-L74 states, there should definitely not be -I../byterun and there should be $(IFLEXDIR) which, as long as no system flexlink.exe is found, should be -I../flexdll.

What's your precise sequence of commands (right from untarring or git clone)? Is this from a git clone or a released tarball?

> Does the flexdll make in ocamlc a local build of flexdll, or does it
> install flexdll somewhere globally? In the first case I think a -
> I../flexdll is missing, in the second case I guess I might miss some
> install prefixes.
> 
> > A git clone of trunk will use the version of flexdll configured at the
> time - that's how git submodules > work (because it stores a commit SHA,
> not a branch).
> 
> Does this also work if I get the ocaml sources as a tar ball rather than
> via a git clone?

No, you'd need either to get a specific sources tarball or checkout the correct commit after cloning the *flexdll* repository. I agree that in future it should state the version - however, it doesn't at the moment because you need "0.35" (i.e. master) which isn't actually released (hence the instruction to use a git clone).

Relatedly, my work packaging it all up with native Windows OPAM does install the version of flexdll which was current at the time of release (see https://github.com/dra27/opam-repository/blob/next-windows/packages/ocaml/ocaml.4.03.0/opam#L7 vs https://github.com/dra27/opam-repository/blob/next-windows/packages/ocaml/ocaml.4.02.0/opam#L7, for example)


David

RE: [Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[Soegtrop, Michael]

Dear David,

> What's your precise sequence of commands (right from untarring or git clone)?
> Is this from a git clone or a released tarball?

I use this tarbal: 

https://github.com/ocaml/ocaml/archive/4.03.0.tar.gz

I try to create a mingw cross ocaml on Cygwin, which I guess is a bit uncommon. My command sequence is:

    < download and untar https://github.com/ocaml/ocaml/archive/4.03.0.tar.gz >
    git clone https://github.com/alainfrisch/flexdll.git
    cp config/m-nt.h config/m.h
    cp config/s-nt.h config/s.h
    cp config/Makefile.mingw64 config/Makefile
    make -f Makefile.nt flexdll
    make -f Makefile.nt world
    make -f Makefile.nt opt
    make -f Makefile.nt opt.opt

I already have a cygqin ocaml installed in the usual paths, I guess this leads to confusion. Probably I should give a PREFIX=something argument to make.

If you are interested, I can send you two scripts to setup a clean Cygwin and reproduce the issue, but I guess with the pointers you gave me, I should be able to figure it out myself.

Just two questions:

1.) in your .comp files I saw a win32.sh. Where is this from? It doesn't appear in the 4.03.0 tar ball mentioned above.

2.) How far is the windows opam and what build environment do you use for this? Cygwin mingw cross? Should I give it a try?

Best regards,

Michael
Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

RE: [Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[David Allsopp]

Michael Soegtrop wrote:
> Dear David,
> 
> > What's your precise sequence of commands (right from untarring or git
> clone)?
> > Is this from a git clone or a released tarball?
> 
> I use this tarbal:
> 
> https://github.com/ocaml/ocaml/archive/4.03.0.tar.gz
> 
> I try to create a mingw cross ocaml on Cygwin, which I guess is a bit
> uncommon.

That's the official way of doing it!

> My command sequence is:
> 
>     < download and untar
> https://github.com/ocaml/ocaml/archive/4.03.0.tar.gz >
>     git clone https://github.com/alainfrisch/flexdll.git
>     cp config/m-nt.h config/m.h
>     cp config/s-nt.h config/s.h
>     cp config/Makefile.mingw64 config/Makefile
>     make -f Makefile.nt flexdll
>     make -f Makefile.nt world
>     make -f Makefile.nt opt
>     make -f Makefile.nt opt.opt
> 
> I already have a cygqin ocaml installed in the usual paths, I guess this
> leads to confusion. Probably I should give a PREFIX=something argument to
> make.

Indeed - it means you *do* have a system installation of flexdll! It almost certainly means IFLEXDIR is wrong (though I'm still surprised that you get -I../byterun from make -f Makefile.nt opt). Presumably that means which flexlink from your Cygwin prompt returns something like /usr/bin/flexlink?

> If you are interested, I can send you two scripts to setup a clean Cygwin
> and reproduce the issue, but I guess with the pointers you gave me, I
> should be able to figure it out myself.
> 
> Just two questions:
> 
> 1.) in your .comp files I saw a win32.sh. Where is this from? It doesn't
> appear in the 4.03.0 tar ball mentioned above.

I'm working on OPAM 2.0, so it's a package rather than a .comp file. win.sh is a helper script in that package, since embedding the commands for all four ports in a single OPAM file gets unreadable very quickly! 

> 2.) How far is the windows opam and what build environment do you use for
> this? Cygwin mingw cross? Should I give it a try?

I'm very nearly ready to do an alpha release, but I've been delayed by emergency eye surgery this week, which has severely limited the number of hours I can spend in front of a computer! The version I'm working supports OCaml 3.07-4.03 using all the native ports (32/64-bit msvc and mingw) built the official way using Cygwin.


David

RE: [Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[Soegtrop, Michael]

Dear David,

> Indeed - it means you *do* have a system installation of flexdll!

You are right:

$ which flexlink
/usr/bin/flexlink

$ find /usr -name "flexdll.h"
/usr/include/flexdll.h

I was just checking the flexdll.h, and it is at a place where the mingw gcc doesn't find it, but of cause flexlink itself is in the path, so it is found by the cross build and messes it up.

The following "fixes" it:

    sed -i "s|FLEXLINK_CMD=flexlink|FLEXLINK_CMD=../flexdll/flexlink|" config/Makefile

I think the makefile should be made such, that the build local flexlink is prefered, if it is there.


One more question: how is this supposed to work with a native Cygwin build? For doing my initial native Cygwin build of ocaml, which I then use to build opam,  I use a normal configure make process, but I do it twice and build flexlink in between. I didn't find a way to convince configure that flexlink is there, when it isn't, and obviously I need to do configure before I do make. Also a make flexlink results in a "nothing to do" after configure.

    ./configure -no-shared-libs
    make world.opt
    make install
   < make + install flexlink >
    ./configure
    make world
    make opt
    make opt.opt
    make install

Thanks for the help!

Best regards,

Michael
Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

RE: [Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[David Allsopp]

Michael Soegtrop wrote:
> Dear David,
> 
> > Indeed - it means you *do* have a system installation of flexdll!
> 
> You are right:
> 
> $ which flexlink
> /usr/bin/flexlink
> 
> $ find /usr -name "flexdll.h"
> /usr/include/flexdll.h
> 
> I was just checking the flexdll.h, and it is at a place where the mingw
> gcc doesn't find it, but of cause flexlink itself is in the path, so it is
> found by the cross build and messes it up.
> 
> The following "fixes" it:
> 
>     sed -i "s|FLEXLINK_CMD=flexlink|FLEXLINK_CMD=../flexdll/flexlink|"
> config/Makefile

This isn't a great way - better is to override the detection of FLEXDIR a few lines later (the FLEXDIR:= line):

sed -I "s|FLEXDIR:=.*|FLEXDIR=|" config/Makefile

> I think the makefile should be made such, that the build local flexlink is
> prefered, if it is there.

Actually, I reviewed the original PR (https://github.com/ocaml/ocaml/pull/388) and contrary to what I'd said previously, I think this is a bug. The solution, however, is to ensure that IFLEXDIR is better set-up if the submodule has been initialised (see, in particular, https://github.com/ocaml/ocaml/blob/trunk/Makefile.nt#L24-L30). I'll have a proper look and open a PR.

> One more question: how is this supposed to work with a native Cygwin
> build? For doing my initial native Cygwin build of ocaml, which I then use
> to build opam,  I use a normal configure make process, but I do it twice
> and build flexlink in between. I didn't find a way to convince configure
> that flexlink is there, when it isn't, and obviously I need to do
> configure before I do make. Also a make flexlink results in a "nothing to
> do" after configure.
> 
>     ./configure -no-shared-libs
>     make world.opt
>     make install
>    < make + install flexlink >
>     ./configure
>     make world
>     make opt
>     make opt.opt
>     make install
> 
> Thanks for the help!

The bootstrapping stuff is for the native ports only - it was far too much (risky) work to put it into the Unix Makefiles as well (I also, even with the recent change of licence to LGPL, have no interest in the Cygwin port of OCaml, I'm afraid!)

My hope was that when I (or someone else) finally unify the two build processes so that the native Windows ports begin with ./configure that Cygwin would pick-up the bootstrapping of flexdll "for free", but for now it would involve maintaining the changes on two separate build paths, which runs a high risk of bitrot...


David

RE: [Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[Soegtrop, Michael]

Dear David,

> This isn't a great way - better is to override the detection of FLEXDIR a few lines
> later (the FLEXDIR:= line):

> sed -I "s|FLEXDIR:=.*|FLEXDIR=|" config/Makefile

I thought about this, but I wonder if this would use the correct flexdll.h, but still the wrong flexlink. I thought it would be safer to point it to the right flexlink and derive the flexdll.h location from there. 

The mingw cross compiler + standard library builds fine, but then when I use it, it doesn't find pervasives, although the pervasives.cmx is in the folder ocamlc -config says it should be. I guess something went wrong with the flexlink process. Is there some way to see from ocamlc - config if flexlink support is there? Or is there another way to check if flexlink is the problem?

Regarding the Cygwin build: I have no issue with this, it just takes a few minutes. I just wanted to make sure I don't do this in an unusual way. 

Best regards,

Michael
Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

RE: [Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[David Allsopp]

Michael Soegtrop wrote:
> > This isn't a great way - better is to override the detection of
> > FLEXDIR a few lines later (the FLEXDIR:= line):
> 
> > sed -I "s|FLEXDIR:=.*|FLEXDIR=|" config/Makefile
> 
> I thought about this, but I wonder if this would use the correct
> flexdll.h, but still the wrong flexlink. I thought it would be safer to
> point it to the right flexlink and derive the flexdll.h location from
> there.

If it's working correctly, each individual Makefile.nt selects flexlink based on the presence of flexdll/Makefile (e.g. https://github.com/ocaml/ocaml/blob/trunk/Makefile.nt#L24-L30). I think what's happened here is a piece of laziness/regression, because it's hard for config/Makefile to know what level it's been included at (the use of ROOTDIR, for example in https://github.com/ocaml/ocaml/blob/trunk/otherlibs/Makefile#L18, is not consistent across the entire build tree).

However, I was going to verify that it never picks up the wrong one by putting a dummy flexlink.exe into PATH which simply does `exit 2`.

> The mingw cross compiler + standard library builds fine, but then when I
> use it, it doesn't find pervasives, although the pervasives.cmx is in the
> folder ocamlc -config says it should be. I guess something went wrong with
> the flexlink process. Is there some way to see from ocamlc - config if
> flexlink support is there? Or is there another way to check if flexlink is
> the problem?

They should be identical, but possibly worth checking ocamlopt -config, rather than ocamlc -config. Being unable to find a .cmx is unlikely to be a flexlink problem (flexlink handles .o files, not .cmx).

Running ocamlopt -verbose allows you to see the flexlink command invocation and, IIRC, adding -ccopt -v should cause flexlink to display the commands it's running too.


David

RE: [Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[Soegtrop, Michael]

Dear David,

> Being unable to find a .cmx is unlikely to be a flexlink problem (flexlink handles .o files, not .cmx).

You are right. I used the MinGW ocaml compiler under Cygwin for so long without issues, that I forgot that they are not host=cygwin. It was a simple path incompatibility.

This was easy to fix, but this didn't get me that much further. With cross opam I get tiny path issues (a backslash here and there) in every other module.

I guess I need to build a real host=Cygwin target=MinGW OCaml. Let's see ...

Best regards,

Michael


Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

RE: [Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[David Allsopp]

Michael Soegtrop wrote:
> Dear David,
> 
> > Being unable to find a .cmx is unlikely to be a flexlink problem
> (flexlink handles .o files, not .cmx).
> 
> You are right. I used the MinGW ocaml compiler under Cygwin for so long
> without issues, that I forgot that they are not host=cygwin. It was a
> simple path incompatibility.
> 
> This was easy to fix, but this didn't get me that much further. With cross
> opam I get tiny path issues (a backslash here and there) in every other
> module.
> 
> I guess I need to build a real host=Cygwin target=MinGW OCaml. Let's see
> ...

Indeed - which is why, for now, my native Windows opam doesn't allow installation of the Cygwin ports (because these issues are too easy to hit).

In my opinion, this is all part of fixing the native Windows build system. Once the native Windows ports are built "properly" using configure, it should be relatively easy to have an ocamlopt with host=Cygwin target=MinGW (which is an ocamlopt producing native Windows code, but accepting Cygwin-style paths, etc.) and also more exotic combinations such as host=MinGW target=Cygwin


David 

RE: [Caml-list] Flexdll circular dependency fix uses system provided flexdll.h [was: opam-cross-windows]

[Soegtrop, Michael]

Dear David,
 
>> I guess I need to build a real host=Cygwin target=MinGW OCaml. Let's 
>> see ...

>Indeed - which is why, for now, my native Windows opam doesn't allow installation of the Cygwin >ports (because these issues are too easy to hit).

>In my opinion, this is all part of fixing the native Windows build system. Once the native Windows  >ports are built "properly" using configure, it should be relatively easy to have an ocamlopt with
>host=Cygwin target=MinGW (which is an ocamlopt producing native Windows code, but accepting >Cygwin-style paths, etc.) and also more exotic combinations such as host=MinGW target=Cygwin

The main issue I see is that OCaml is not an application but a toolchain. Tools and PPX need to be compiled for the host. So some libraries need to be compiled for both, target and host. I guess this will only work if one targets a naming scheme as used for gcc, so that two OCaml compilers can coexist in the path.

But I think it would be highly desirable to look into this. Looking at platforms like phones, tablets, Raspberry Pi, the internet of things in general: all these rely on proper cross compilation tool chains. PCs and PC OSes become less and less important. Meanwhile I would think the majority of Linux derived devices uses cross compiled SW.

Best regards,

Michael
Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928