[Caml-list] flexdll circular dependency on ocamlc: Impossible to built ocaml for mingw without using some prebuilt binaries?

[Caml-list] flexdll circular dependency on ocamlc: Impossible to built ocaml for mingw without using some prebuilt binaries?

[Soegtrop, Michael]

[-- Attachment #1: Type: text/plain, Size: 2046 bytes --]

Dear Ocaml Users,

building Ocaml for Mingw on Cygwin I came across a strange effect: flexdll (or flexlink) depends on the Ocaml compiler and the Ocaml compiler depends on flexdll. I wonder how one is supposed to build Ocaml for Mingw from scratch? One either needs prebuilt flexdll/flexlink or Ocaml. What does work is to first build a native Cygwin Ocaml, then flexlink using this one and then the MingW Ocaml, but then flexlink depends on the Cygwin dlls. I am not sure how bad this is but I think ocamlc needs flexlink later as well when linking MingW apps. Also the effort of this is rather high. So I used a prebuilt flexlink/flexdll but I don't really like this.

Also I wonder why flexdll/flexlink is required. The documentation of flexdll states:

Windows DLL cannot refer to symbols defined in the main application or in previously loaded DLLs.

In my experience this is not true. At least when using MSVC one can declare functions in the main executable as DLL-export. Then when linking the main executable an import library is created in the same way as when building a DLL by the linker. The DLL can then link to this import library and can access the functions  in the main executable. The same is true for any other dlls already loaded. I don't do this every day but I did this more than once. I am not sure if this also works on old Windows OSes. I think the oldest one on which I tested this is Windows 7. Also I don't know if gcc does support creating an import library when linking an executable.

If this is of interest, I could provide a small demo for this and also test it with gcc for mingw and, if someone thinks this is useful, also on XP and Vista.

Best regards,

Michael

Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Prof. Dr. Hermann Eul
Chairperson of the Supervisory Board: Tiffany Doon Silva
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

[-- Attachment #2: Type: text/html, Size: 4303 bytes --]

RE: [Caml-list] flexdll circular dependency on ocamlc: Impossible to built ocaml for mingw without using some prebuilt binaries?

[David Allsopp]

[-- Attachment #1: Type: text/plain, Size: 3950 bytes --]

This is indeed a strange effect - bluntly, flexlink should be bootstrapped with the OCaml sources (i.e. a bytecode image of flexlink should be in /boot).

Note that your Cygwin build-route really is missing a step - you're bootstrapping, so the final stage should be recompiling flexlink using your new ocamlopt (the compilers do indeed need flexlink), then the dependency on Cygwin disappears. For my own builds, I extract flexlink.exe from Alain's binary distribution, build the FlexDLL objects from sources, compile OCaml and then recompile flexlink.exe using the that fresh compiler and replace the earlier binary copy. See, for example, https://github.com/dra27/OCaml-Windows-Build/blob/master/build.cmd#L527-L570

At some point, I was going to alter things so that FlexDLL is the only requirement for pre-installation (as that can be built from source) and have the ocaml process use boot/flexlink if a binary flexlink isn't available... but as that's only to satisfy a religious fervour to remove the circular dependency, it's right down at the bottom of my personal task list, especially as I doubt it would be merged :o)

Regarding the reason for FlexDLL, the solution for tricking the linker does also work for gcc - see http://edll.sourceforge.net/#def_a (which is referenced from FlexDLL's webpage). I expect Alain's motivation for FlexDLL is that the "side-effects" referred to in http://alain.frisch.fr/flexdll.html#examples were known about beforehand which the .def/.a solution doesn't solve, especially as it makes the build process more complicated.


David


From: caml-list-request@inria.fr [mailto:caml-list-request@inria.fr] On Behalf Of Soegtrop, Michael
Sent: 24 October 2015 10:51
To: caml-list@inria.fr
Subject: [Caml-list] flexdll circular dependency on ocamlc: Impossible to built ocaml for mingw without using some prebuilt binaries?

Dear Ocaml Users,

building Ocaml for Mingw on Cygwin I came across a strange effect: flexdll (or flexlink) depends on the Ocaml compiler and the Ocaml compiler depends on flexdll. I wonder how one is supposed to build Ocaml for Mingw from scratch? One either needs prebuilt flexdll/flexlink or Ocaml. What does work is to first build a native Cygwin Ocaml, then flexlink using this one and then the MingW Ocaml, but then flexlink depends on the Cygwin dlls. I am not sure how bad this is but I think ocamlc needs flexlink later as well when linking MingW apps. Also the effort of this is rather high. So I used a prebuilt flexlink/flexdll but I don't really like this.

Also I wonder why flexdll/flexlink is required. The documentation of flexdll states:

Windows DLL cannot refer to symbols defined in the main application or in previously loaded DLLs.

In my experience this is not true. At least when using MSVC one can declare functions in the main executable as DLL-export. Then when linking the main executable an import library is created in the same way as when building a DLL by the linker. The DLL can then link to this import library and can access the functions  in the main executable. The same is true for any other dlls already loaded. I don't do this every day but I did this more than once. I am not sure if this also works on old Windows OSes. I think the oldest one on which I tested this is Windows 7. Also I don't know if gcc does support creating an import library when linking an executable.

If this is of interest, I could provide a small demo for this and also test it with gcc for mingw and, if someone thinks this is useful, also on XP and Vista.

Best regards,

Michael


Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de<http://www.intel.de>
Managing Directors: Christin Eisenschmid, Prof. Dr. Hermann Eul
Chairperson of the Supervisory Board: Tiffany Doon Silva
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

[-- Attachment #2: Type: text/html, Size: 8449 bytes --]

Re: [Caml-list] flexdll circular dependency on ocamlc: Impossible to built ocaml for mingw without using some prebuilt binaries?

[Alain Frisch]

On 24/10/2015 11:50, Soegtrop, Michael wrote:
> Also I wonder why flexdll/flexlink is required. The documentation of
> flexdll states:
>
> Windows DLL cannot refer to symbols defined in the main application or
> in previously loaded DLLs.
>
> In my experience this is not true. At least when using MSVC one can
> declare functions in the main executable as DLL-export. Then when
> linking the main executable an import library is created in the same way
> as when building a DLL by the linker. The DLL can then link to this
> import library and can access the functions  in the main executable.

Dynlink follows a different model: dynlinked units are not tied to a 
specific main executable.  A myplugin.cmxs can be dynamically linked by 
any application that provides the required interfaces.

As for the dependency between OCaml and flexdll: the simplest approach 
is to consider flexdll as any other external tool required to build 
OCaml (do you recompile "gcc", "make", etc  from sources?). It just 
happens to be implemented in OCaml.  Alternatively, you could create a 
bytecode version of flexlink, which would only require ocamlrun.


Alain

RE: [Caml-list] flexdll circular dependency on ocamlc: Impossible to built ocaml for mingw without using some prebuilt binaries?

[David Allsopp]

Alain Frisch wrote:
> On 24/10/2015 11:50, Soegtrop, Michael wrote:
> > Also I wonder why flexdll/flexlink is required. The documentation of
> > flexdll states:
> >
> > Windows DLL cannot refer to symbols defined in the main application or
> > in previously loaded DLLs.
> >
> > In my experience this is not true. At least when using MSVC one can
> > declare functions in the main executable as DLL-export. Then when
> > linking the main executable an import library is created in the same
> > way as when building a DLL by the linker. The DLL can then link to
> > this import library and can access the functions  in the main
> executable.
> 
> Dynlink follows a different model: dynlinked units are not tied to a
> specific main executable.  A myplugin.cmxs can be dynamically linked by
> any application that provides the required interfaces.

Wouldn't that still be true with the .def/.a approach, though?

> As for the dependency between OCaml and flexdll: the simplest approach is
> to consider flexdll as any other external tool required to build OCaml (do
> you recompile "gcc", "make", etc  from sources?). It just happens to be
> implemented in OCaml.  Alternatively, you could create a bytecode version
> of flexlink, which would only require ocamlrun.

It's not about *do* you, it's *can* you! Ignoring the fact that these days I think gcc has to be built with gcc, historically all those tools (gcc, make, etc.) could be built using *someone else's* C compiler and make implementation and then bootstrapped to create a "pure" one. There is no other implementation of flexlink (or ocaml) to achieve that, which is what makes classing flexlink (rather than the core of FlexDLL) as "external" taste weird, even if only in a philosophical manner.


David 

RE: [Caml-list] flexdll circular dependency on ocamlc: Impossible to built ocaml for mingw without using some prebuilt binaries?

[Soegtrop, Michael]

[-- Attachment #1: Type: text/plain, Size: 1847 bytes --]

Dear David,

Note that your Cygwin build-route really is missing a step - you're bootstrapping, so the final stage should be recompiling flexlink using your new ocamlopt (the compilers do indeed need flexlink), then the dependency on Cygwin disappears. For my own builds, I extract flexlink.exe from Alain's binary distribution, build the FlexDLL objects from sources, compile OCaml and then recompile flexlink.exe using the that fresh compiler and replace the earlier binary copy. See, for example, https://github.com/dra27/OCaml-Windows-Build/blob/master/build.cmd#L527-L570

Sure, if I go this way, I should do this. But it looks like a bit of overkill to build a native ocaml compiler to bootstrap flexdll. Also Ocaml is not very good at handling different compiler variants, so I would have to uninstall the native Ocaml after building flexdll. So for the time being I am using the binary version. I think about compiling flexlink from sources after building Ocaml.

At some point, I was going to alter things so that FlexDLL is the only requirement for pre-installation (as that can be built from source) and have the ocaml process use boot/flexlink if a binary flexlink isn't available... but as that's only to satisfy a religious fervour to remove the circular dependency, it's right down at the bottom of my personal task list, especially as I doubt it would be merged :o)

True, it is not a showstopper. I just mentioned it here to make sure I didn't overlook something.

Best regards,

Michael
Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Prof. Dr. Hermann Eul
Chairperson of the Supervisory Board: Tiffany Doon Silva
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

[-- Attachment #2: Type: text/html, Size: 5606 bytes --]

RE: [Caml-list] flexdll circular dependency on ocamlc: Impossible to built ocaml for mingw without using some prebuilt binaries?

[Soegtrop, Michael]

Dear Alain,

> Dynlink follows a different model: dynlinked units are not tied to a specific
> main executable.  A myplugin.cmxs can be dynamically linked by any
> application that provides the required interfaces.

Can you explain what you mean by Dynlink (or point me to some documents)? I though we are talking about loading DLLs with LoadLibrary programmatically (rather than by the image loader). I am quite sure calling main executable functions from a DLL loaded this way does work. But I am not sure we are talking about this mechanism or something else.

> As for the dependency between OCaml and flexdll: the simplest approach is
> to consider flexdll as any other external tool required to build OCaml (do you
> recompile "gcc", "make", etc  from sources?).

The idea of open source is that you have the possibility to fix bugs yourself. This can be an important factor for maintainability. I am quite sure I can compile gcc from sources if I want to. Everything where I am not so sure where it comes from and how it is build I actually do build from sources. This includes things which are as nasty to build as GTK.

Best regards,

Michael

Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Prof. Dr. Hermann Eul
Chairperson of the Supervisory Board: Tiffany Doon Silva
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

Re: [Caml-list] flexdll circular dependency on ocamlc: Impossible to built ocaml for mingw without using some prebuilt binaries?

[Alain Frisch]

On 26/10/2015 10:13, Soegtrop, Michael wrote:
> Dear Alain,
>
>> Dynlink follows a different model: dynlinked units are not tied to a specific
>> main executable.  A myplugin.cmxs can be dynamically linked by any
>> application that provides the required interfaces.
>
> Can you explain what you mean by Dynlink (or point me to some documents)? I though we are talking about loading DLLs with LoadLibrary programmatically (rather than by the image loader). I am quite sure calling main executable functions from a DLL loaded this way does work. But I am not sure we are talking about this mechanism or something else.

I'm talking about the Dynlink library which allows OCaml code to 
dynamically load plugins (made of OCaml + perhaps native code).  The 
loaded code is compiled in the same way as if it were to be linked in 
the main program, but then it is linked into a .cmxs file (which is 
really a dll).

http://caml.inria.fr/pub/docs/manual-ocaml/libref/Dynlink.html

Alain

Re: [Caml-list] flexdll circular dependency on ocamlc: Impossible to built ocaml for mingw without using some prebuilt binaries?

[Alain Frisch]

On 24/10/2015 14:51, David Allsopp wrote:
> Alain Frisch wrote:
>> On 24/10/2015 11:50, Soegtrop, Michael wrote:
>>> Also I wonder why flexdll/flexlink is required. The documentation of
>>> flexdll states:
>>>
>>> Windows DLL cannot refer to symbols defined in the main application or
>>> in previously loaded DLLs.
>>>
>>> In my experience this is not true. At least when using MSVC one can
>>> declare functions in the main executable as DLL-export. Then when
>>> linking the main executable an import library is created in the same
>>> way as when building a DLL by the linker. The DLL can then link to
>>> this import library and can access the functions  in the main
>> executable.
>>
>> Dynlink follows a different model: dynlinked units are not tied to a
>> specific main executable.  A myplugin.cmxs can be dynamically linked by
>> any application that provides the required interfaces.
>
> Wouldn't that still be true with the .def/.a approach, though?

I don't see how this would work without changing the compilation model.
Would you generate an import library for each .cmi file?

Also keep in mind that currently, dynlinked code can access directly 
data symbols from the main program (not just function calls that can 
more easily go through an import library).


Alain

RE: [Caml-list] flexdll circular dependency on ocamlc: Impossible to built ocaml for mingw without using some prebuilt binaries?

[David Allsopp]

Alain Frisch wrote:
> On 24/10/2015 14:51, David Allsopp wrote:
> > Alain Frisch wrote:
> >> On 24/10/2015 11:50, Soegtrop, Michael wrote:
> >>> Also I wonder why flexdll/flexlink is required. The documentation of
> >>> flexdll states:
> >>>
> >>> Windows DLL cannot refer to symbols defined in the main application
> >>> or in previously loaded DLLs.
> >>>
> >>> In my experience this is not true. At least when using MSVC one can
> >>> declare functions in the main executable as DLL-export. Then when
> >>> linking the main executable an import library is created in the same
> >>> way as when building a DLL by the linker. The DLL can then link to
> >>> this import library and can access the functions  in the main
> >> executable.
> >>
> >> Dynlink follows a different model: dynlinked units are not tied to a
> >> specific main executable.  A myplugin.cmxs can be dynamically linked
> >> by any application that provides the required interfaces.
> >
> > Wouldn't that still be true with the .def/.a approach, though?
> 
> I don't see how this would work without changing the compilation model.
> Would you generate an import library for each .cmi file?

Yes - but I wasn't taking the compilation model as fixed. I was (purely hypothetically, obviously!) imagining that flexlink (or some such) linker was still there and that it would generate the appropriate .def/.a files to allow the dll to be compiled, as it would also have access to the required .cmi files. That's quite similar to what flexlink already does, isn't it - just generating an import library to trick the actual/underlying linker rather than building separate tables embedded into .o/.obj files which get linked as well. 

But... 

> Also keep in mind that currently, dynlinked code can access directly data
> symbols from the main program (not just function calls that can more
> easily go through an import library).

... that sounds like the real "elevator explanation" for why things are as they are!


David

RE: [Caml-list] flexdll circular dependency on ocamlc: Impossible to built ocaml for mingw without using some prebuilt binaries?

[Soegtrop, Michael]

Dear Alain,

> I'm talking about the Dynlink library which allows OCaml code to dynamically
> load plugins (made of OCaml + perhaps native code).  The loaded code is
> compiled in the same way as if it were to be linked in the main program, but
> then it is linked into a .cmxs file (which is really a dll).
> 
> http://caml.inria.fr/pub/docs/manual-ocaml/libref/Dynlink.html

I see. Well I can't say I fully understand what the problem is, but then I don't need to understand everything. I just asked because flexdll results, as described, in some build trouble and I wanted to make sure that what you get from it is worth this trouble. I take your word that this is the case.

Best regards,

Michael
Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Prof. Dr. Hermann Eul
Chairperson of the Supervisory Board: Tiffany Doon Silva
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928