[Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Adrien Nader]

Hi,

Roughly one year ago, late Wojciech Meyer and I started working on the
integration of a set of patches aimed at enabling the OCaml compiler to
be used for cross-compilation.

Some of the patches so far have caused issues to some people. The goal
of this RFC and email thread is to raise awareness of the upcoming work
and its implications and also to start getting feedback as early as
possible. The patches should be handled through the github interface
which Gabriel Scherer mentionned a few weeks ago.


******************************** Note ********************************
The words "target", "host" and "build" below have the meaning that the
GNU project gives them when it comes to compilation.
When compiling an application or library:
- build: the system the compilation process runs on
- host: the system that will run the binaries that are built
- target: meaningless
When compiling a cross-compiler:
- build: the system the compilation process runs on
- host: the system that will run the binaries that are built (i.e. the
  cross-compiler)
- target: the system for which the cross-compiler builds binaries


*************************** Immediate Goals ***************************
- build "${target}-ocamlc", "${target}-ocamlopt", ..., i.e. a
  cross-compiler for ${target}
- typical cross-compilation of projects is expected to go through
  ocamlfind; in practice, oasis-based projects can be cross-compiled
  with no change


**************************** Further Goals ****************************
- cross-compile the compiler: build a native Windows toolchain on Linux
- canadian cross support: ${build}, ${target}, ${host} are all
  different, i.e. on Linux, build a toolchain that runs on Windows 64b
  but targets Windows 32b or a different architecture/OS like ARM Linux.


*************************** Current status ***************************
- the compiler makes few assumptions about the target; this is very
  good.
- build system is fairly large, has been in flux because of the
  separation of labltk and camlp4, supports both GNU make and BSD's
  makes (i.e. it uses no extension beyong POSIX), some things are split
  between "Makefile" and "Makefile.nt" (and for the ".nt" variant, only
  GNU make is supported and these files do use GNU make extensions).
- most of the patches I ended up with have been upstreamed; a couple
  hairy ones are left and one in particular caused issues when
  bootstrapping.


****************************** Approach ******************************

*** A cross-compiler requires a native toolchain of the same version ***
A working _native_ compiler toolchain of the _exact_ same version as the
cross-toolchain is needed, both when building the cross-toolchain and
when using it; the main reason was camlp4 but I believe it is still a
valid restriction


*** Give up the restriction to POSIX for makefiles ***
Either remove compatibility with BSD makes, or move GNU-make-isms to
"GNUMakefile", BSD-makes-isms to "BSDMakefile" and common data to
"Makefile.common" with the GNU/BSDMakefile files being the
"entrypoints".

The manual page for GNU make states (and mk is similar but with
BSDMakefile iirc):
  "If no -f option is  present, make  will  look for the makefiles
  GNUmakefile, makefile, and Makefile, in that order."

However, separating the code which benefits from the extension requires
that:
- the amount of non-shared code (i.e. the size of GNUmakefile and
  BSDMakefile files) is small and at least the data can be shared; this
  is meant for a few specific functions that are currently shelled-out
  because they are too difficult to write in POSIX make
- most BSDs, if not all, can be supported without workarounds, kludges
  or additional hurdles caused by their fragmentation (for instance,
  OpenBSD's pmake is 10-years late compared to NetBSD's and has
  several bugs that are long solved in NetBSD)


*** Use configure-defined variables instead of "ocaml{c,opt}"... ***
The creation of the cross-compiler will rely on a native toolchain on
${build}. As such, ./configure will locate that toolchain and store its
paths. This forbids any call like "./ocamlopt"; everything must rely on
the paths found by configure.


*** Use Int64.t in the compiler instead of Nativeint ***
Currently, OCaml uses Nativeint to handle all the integers in the code
that is being compiled. The "native" is for ${build}, not for ${target}.

With a 64b host and 32b target, OCaml will output its values over 64
bits in the assembly and ld will then complain and truncate them.

Move to Int64.t instead and delay the conversion to the right bitness
until code emiting. The following types change: Cmm.expression,
Cmm.data_item, Arch.specific_operation, Mach.operation. After that it's
mostly a matter of fixing the type errors (around 50); since I do not
have all the platforms that OCaml supports, I need to devise a way to
compile the files for the other architectures too as a quick test.

(types to change: Cmm.{expression,data_item}, Arch.specific_operation,
Mach.operation)

*** For bytecode, assume C libraries contain the required primitives ***

When doing linking for bytecode with -custom, OCaml will dlopen() the
corresponding library and dlsym() the primitive as an early check for
their availability at runtime.

Quite obviously, this fails for cross-compilation and the only solution
I can think of is to disable the check completely. I have already
written the corresponding patch; it adds 4 trivial lines and changes
one.

*** Devil is in the details ***

While I wish the list above was exhaustive, I know for sure that it
isn't, partly because some of the changes needed are implementation
details more than general approaches. Do not hesitate to ask for
clarification on anything you might wonder.

-- 
Adrien Nader

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support

[oleg]

> *** For bytecode, assume C libraries contain the required primitives ***
>
> When doing linking for bytecode with -custom, OCaml will dlopen() the
> corresponding library and dlsym() the primitive as an early check for
> their availability at runtime.
>
> Quite obviously, this fails for cross-compilation and the only solution
> I can think of is to disable the check completely. I have already
> written the corresponding patch; it adds 4 trivial lines and changes
> one.

I think the following message is related
        https://sympa.inria.fr/sympa/arc/caml-list/2013-09/msg00090.html
(please also see the follow-ups)

        and so your patch will also solve my problem. Solving several
problems seems to make a good case for a patch.

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Goswin von Brederlow]

On Tue, Feb 18, 2014 at 07:50:32PM +0100, Adrien Nader wrote:
> Hi,
> 
> Roughly one year ago, late Wojciech Meyer and I started working on the
> integration of a set of patches aimed at enabling the OCaml compiler to
> be used for cross-compilation.
> 
> Some of the patches so far have caused issues to some people. The goal
> of this RFC and email thread is to raise awareness of the upcoming work
> and its implications and also to start getting feedback as early as
> possible. The patches should be handled through the github interface
> which Gabriel Scherer mentionned a few weeks ago.

Seeing as you want to use the github interface to submit patches I
assume you also have a fully functioning branch on github that people
could try. Would be nice to give the URL for that (or to make such a
branch).

MfG
	Goswin

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Adrien Nader]

On Thu, Feb 20, 2014, Goswin von Brederlow wrote:
> On Tue, Feb 18, 2014 at 07:50:32PM +0100, Adrien Nader wrote:
> > Hi,
> > 
> > Roughly one year ago, late Wojciech Meyer and I started working on the
> > integration of a set of patches aimed at enabling the OCaml compiler to
> > be used for cross-compilation.
> > 
> > Some of the patches so far have caused issues to some people. The goal
> > of this RFC and email thread is to raise awareness of the upcoming work
> > and its implications and also to start getting feedback as early as
> > possible. The patches should be handled through the github interface
> > which Gabriel Scherer mentionned a few weeks ago.
> 
> Seeing as you want to use the github interface to submit patches I
> assume you also have a fully functioning branch on github that people
> could try. Would be nice to give the URL for that (or to make such a
> branch).

No, no public branch right now.
In the past, I have had to play catch up with the trunk.
I started with a set of patches, two months later I could start having
some integrated, meanwhile trunk had changed and most of the patches had
to be updated; by the time I could update the patches, trunk would have
moved again enough to require a new update and so on. Moreover, some
patches require a new approach and this is reason I sent this RFC.

I'm not yet sure how I will be pushing them. I think I will probably
push the WIP stuff on http://cgit.notk.org and mostly use github for the
code review stuff. That WIP stuff will also be pushed to and rebased
with no consideration for anyone: it will be a copy of my local repo, no
matter which state it is in.
(NB: rebases is very useful because bootstrapping creates new binaries
of several MBs)

In any case, I'll send a message to the list when the first code is
there and slightly usable.

-- 
Adrien Nader

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Xavier Leroy]

Hi Adrien,

> Roughly one year ago, late Wojciech Meyer and I started working on the
> integration of a set of patches aimed at enabling the OCaml compiler to
> be used for cross-compilation.
> 
> Some of the patches so far have caused issues to some people. 

s/some people/the core OCaml dev team/

> **************************** Further Goals ****************************
> - cross-compile the compiler: build a native Windows toolchain on Linux
> - canadian cross support: ${build}, ${target}, ${host} are all
>   different, i.e. on Linux, build a toolchain that runs on Windows 64b
>   but targets Windows 32b or a different architecture/OS like ARM Linux.

That could be a nice touch if it's not too much extra work.  The issue
was discussed at the last OCaml dev team meeting that Wojciech
attended, and we agreed that some scenarios were less important than
others, e.g. nobody really needs to compile from a 32-bit host for a
64-bit target.

> ****************************** Approach ******************************
> *** A cross-compiler requires a native toolchain of the same version ***
> A working _native_ compiler toolchain of the _exact_ same version as the
> cross-toolchain is needed, both when building the cross-toolchain and
> when using it; the main reason was camlp4 but I believe it is still a
> valid restriction

Perhaps less so now that camlp4 is split off.  The alternative is to
configure for the host, populate boot/ with a ocamlrun and standard
library appropriate for the host, then reconfigure for the target, and
finish the build.

> *** Give up the restriction to POSIX for makefiles ***
> Either remove compatibility with BSD makes, or move GNU-make-isms to
> "GNUMakefile", BSD-makes-isms to "BSDMakefile" and common data to
> "Makefile.common" with the GNU/BSDMakefile files being the
> "entrypoints".

The unanimous message from the OCaml dev team is to commit on GNU make
and forget about compatibility with other variants of make.  BSD
people will forgive us, and it looks like the only sane way to merge
the Unix and Windows makefiles and support all the cross-compilation
scenarios.

> *** Use configure-defined variables instead of "ocaml{c,opt}"... ***
> The creation of the cross-compiler will rely on a native toolchain on
> ${build}. As such, ./configure will locate that toolchain and store its
> paths. This forbids any call like "./ocamlopt"; everything must rely on
> the paths found by configure.

See above: this might not be necessary.

> *** Use Int64.t in the compiler instead of Nativeint ***
> Currently, OCaml uses Nativeint to handle all the integers in the code
> that is being compiled. The "native" is for ${build}, not for ${target}.
> With a 64b host and 32b target, OCaml will output its values over 64
> bits in the assembly and ld will then complain and truncate them.
> Move to Int64.t instead and delay the conversion to the right bitness
> until code emiting.

Constant propagation and maybe other passes of the compiler also need to
take bitsize into account.

If you're going this way, it's not Int64 the appropriate substitute
for Nativeint: it's Targetint, defined as Int32 or Int64 depending on
target bitsize.  (This can be expressed with first-class modules.)

There might be other ways if we assume bitsize(host) >= bitsize(target).

> *** For bytecode, assume C libraries contain the required primitives ***
> When doing linking for bytecode with -custom, OCaml will dlopen() the
> corresponding library and dlsym() the primitive as an early check for
> their availability at runtime.
> Quite obviously, this fails for cross-compilation and the only solution
> I can think of is to disable the check completely. 

Probably there is no choice.  But it pains me to move link-time errors
to run-time (more exactly, program startup time).  If we program in
OCaml rather than Python, it's also for the early error detection.

- Xavier Leroy

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Anil Madhavapeddy]

On 20 Feb 2014, at 16:53, Xavier Leroy <Xavier.Leroy@inria.fr> wrote:

>> 
>> **************************** Further Goals ****************************
>> - cross-compile the compiler: build a native Windows toolchain on Linux
>> - canadian cross support: ${build}, ${target}, ${host} are all
>>  different, i.e. on Linux, build a toolchain that runs on Windows 64b
>>  but targets Windows 32b or a different architecture/OS like ARM Linux.
> 
> That could be a nice touch if it's not too much extra work.  The issue
> was discussed at the last OCaml dev team meeting that Wojciech
> attended, and we agreed that some scenarios were less important than
> others, e.g. nobody really needs to compile from a 32-bit host for a
> 64-bit target.

This would actually be quite useful for Mirage compilation, since we've
had quite a few (~10) reports of people attempting to compile up OCaml
kernels intended for Xen (x86_64) using x86_32 host machines, with one
brave chap even trying on an ARM32 Chromebook.  The output kernels are
spun up on a remote cloud service like Amazon or Rackspace, and so being
flexible about the host machine is helpful.

(This scenario is certainly less important than the 32-bit target for
a 64-bit host, but I just thought I'd point it out)

-anil

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Adrien Nader]

On Fri, Feb 21, 2014, Anil Madhavapeddy wrote:
> On 20 Feb 2014, at 16:53, Xavier Leroy <Xavier.Leroy@inria.fr> wrote:
> 
> >> 
> >> **************************** Further Goals ****************************
> >> - cross-compile the compiler: build a native Windows toolchain on Linux
> >> - canadian cross support: ${build}, ${target}, ${host} are all
> >>  different, i.e. on Linux, build a toolchain that runs on Windows 64b
> >>  but targets Windows 32b or a different architecture/OS like ARM Linux.
> > 
> > That could be a nice touch if it's not too much extra work.  The issue
> > was discussed at the last OCaml dev team meeting that Wojciech
> > attended, and we agreed that some scenarios were less important than
> > others, e.g. nobody really needs to compile from a 32-bit host for a
> > 64-bit target.
> 
> This would actually be quite useful for Mirage compilation, since we've
> had quite a few (~10) reports of people attempting to compile up OCaml
> kernels intended for Xen (x86_64) using x86_32 host machines, with one
> brave chap even trying on an ARM32 Chromebook.  The output kernels are
> spun up on a remote cloud service like Amazon or Rackspace, and so being
> flexible about the host machine is helpful.
> 
> (This scenario is certainly less important than the 32-bit target for
> a 64-bit host, but I just thought I'd point it out)

My own use-case is to build Windows binaries of the compiler from Linux
so I can add them to win-builds ( http://win-builds.org ).

I really mention it as a _further_ goal however. I think it could be
achieved without too much work and current changes should keep this
possibility in mind but not try to handle it.
Realistically, the current changes first have to be finished, polished,
checked, released, settled, their bugs fixed...
(well, if the changes are simple, it should be possible to have a set of
"canonical" patches quickly and only merge them later on)

-- 
Adrien Nader

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Francois Berenger]

On 02/21/2014 01:53 AM, Xavier Leroy wrote:
> Hi Adrien,
>
>> Roughly one year ago, late Wojciech Meyer and I started working on the
>> integration of a set of patches aimed at enabling the OCaml compiler to
>> be used for cross-compilation.
>>
>> Some of the patches so far have caused issues to some people.
>
> s/some people/the core OCaml dev team/
>
>> **************************** Further Goals ****************************
>> - cross-compile the compiler: build a native Windows toolchain on Linux
>> - canadian cross support: ${build}, ${target}, ${host} are all
>>    different, i.e. on Linux, build a toolchain that runs on Windows 64b
>>    but targets Windows 32b or a different architecture/OS like ARM Linux.
>
> That could be a nice touch if it's not too much extra work.  The issue
> was discussed at the last OCaml dev team meeting that Wojciech
> attended, and we agreed that some scenarios were less important than
> others, e.g. nobody really needs to compile from a 32-bit host for a
> 64-bit target.
>
>> ****************************** Approach ******************************
>> *** A cross-compiler requires a native toolchain of the same version ***
>> A working _native_ compiler toolchain of the _exact_ same version as the
>> cross-toolchain is needed, both when building the cross-toolchain and
>> when using it; the main reason was camlp4 but I believe it is still a
>> valid restriction
>
> Perhaps less so now that camlp4 is split off.  The alternative is to
> configure for the host, populate boot/ with a ocamlrun and standard
> library appropriate for the host, then reconfigure for the target, and
> finish the build.
>
>> *** Give up the restriction to POSIX for makefiles ***
>> Either remove compatibility with BSD makes, or move GNU-make-isms to
>> "GNUMakefile", BSD-makes-isms to "BSDMakefile" and common data to
>> "Makefile.common" with the GNU/BSDMakefile files being the
>> "entrypoints".
>
> The unanimous message from the OCaml dev team is to commit on GNU make
> and forget about compatibility with other variants of make.  BSD
> people will forgive us, and it looks like the only sane way to merge
> the Unix and Windows makefiles and support all the cross-compilation
> scenarios.

Moreover, I think BSDs can install GNU make and that the corresponding 
command is simply gmake instead of make.

>> *** Use configure-defined variables instead of "ocaml{c,opt}"... ***
>> The creation of the cross-compiler will rely on a native toolchain on
>> ${build}. As such, ./configure will locate that toolchain and store its
>> paths. This forbids any call like "./ocamlopt"; everything must rely on
>> the paths found by configure.
>
> See above: this might not be necessary.
>
>> *** Use Int64.t in the compiler instead of Nativeint ***
>> Currently, OCaml uses Nativeint to handle all the integers in the code
>> that is being compiled. The "native" is for ${build}, not for ${target}.
>> With a 64b host and 32b target, OCaml will output its values over 64
>> bits in the assembly and ld will then complain and truncate them.
>> Move to Int64.t instead and delay the conversion to the right bitness
>> until code emiting.
>
> Constant propagation and maybe other passes of the compiler also need to
> take bitsize into account.
>
> If you're going this way, it's not Int64 the appropriate substitute
> for Nativeint: it's Targetint, defined as Int32 or Int64 depending on
> target bitsize.  (This can be expressed with first-class modules.)
>
> There might be other ways if we assume bitsize(host) >= bitsize(target).
>
>> *** For bytecode, assume C libraries contain the required primitives ***
>> When doing linking for bytecode with -custom, OCaml will dlopen() the
>> corresponding library and dlsym() the primitive as an early check for
>> their availability at runtime.
>> Quite obviously, this fails for cross-compilation and the only solution
>> I can think of is to disable the check completely.
>
> Probably there is no choice.  But it pains me to move link-time errors
> to run-time (more exactly, program startup time).  If we program in
> OCaml rather than Python, it's also for the early error detection.
>
> - Xavier Leroy
>


-- 
Best regards,
Francois Berenger.

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Adrien Nader]

Hi Xavier,

On Thu, Feb 20, 2014, Xavier Leroy wrote:
> > ****************************** Approach ******************************
> > *** A cross-compiler requires a native toolchain of the same version ***
> > A working _native_ compiler toolchain of the _exact_ same version as the
> > cross-toolchain is needed, both when building the cross-toolchain and
> > when using it; the main reason was camlp4 but I believe it is still a
> > valid restriction
> 
> Perhaps less so now that camlp4 is split off.  The alternative is to
> configure for the host, populate boot/ with a ocamlrun and standard
> library appropriate for the host, then reconfigure for the target, and
> finish the build.

Camlp4 was clearly the main component to require this but I have also
been wondering about ocamldoc and ocamldep which can break because of
new syntax.

For now this requirements is probably the safe approach. Note that this
restriction only checks for x.y.z versions, not for the tags than can
come after 'z'; it made working with trunk too difficult.

Also, as a packager, I prefer this. I find that the other approach makes
packaging more difficult.

> > *** Give up the restriction to POSIX for makefiles ***
> > Either remove compatibility with BSD makes, or move GNU-make-isms to
> > "GNUMakefile", BSD-makes-isms to "BSDMakefile" and common data to
> > "Makefile.common" with the GNU/BSDMakefile files being the
> > "entrypoints".
> 
> The unanimous message from the OCaml dev team is to commit on GNU make
> and forget about compatibility with other variants of make.  BSD
> people will forgive us, and it looks like the only sane way to merge
> the Unix and Windows makefiles and support all the cross-compilation
> scenarios.

OK; I liked pmake and its descendants, partly because of their clean
syntax with true flow control and thought it could be left as a
possibility. Anyway, I will take that route.

> > *** Use configure-defined variables instead of "ocaml{c,opt}"... ***
> > The creation of the cross-compiler will rely on a native toolchain on
> > ${build}. As such, ./configure will locate that toolchain and store its
> > paths. This forbids any call like "./ocamlopt"; everything must rely on
> > the paths found by configure.
> 
> See above: this might not be necessary.

I believe you refer to configuring for host, populating boot and then
configuring again for target. It's something I hadn't thought of and
could work nicely. I need to experiment with this a bit and see how
things would work out.

> > *** Use Int64.t in the compiler instead of Nativeint ***
> > Currently, OCaml uses Nativeint to handle all the integers in the code
> > that is being compiled. The "native" is for ${build}, not for ${target}.
> > With a 64b host and 32b target, OCaml will output its values over 64
> > bits in the assembly and ld will then complain and truncate them.
> > Move to Int64.t instead and delay the conversion to the right bitness
> > until code emiting.
> 
> Constant propagation and maybe other passes of the compiler also need to
> take bitsize into account.
> 
> If you're going this way, it's not Int64 the appropriate substitute
> for Nativeint: it's Targetint, defined as Int32 or Int64 depending on
> target bitsize.  (This can be expressed with first-class modules.)
> 
> There might be other ways if we assume bitsize(host) >= bitsize(target).

I wish everyone (at least everyone doing devlopment) would have a 64b
machine but it's not the case unfortunately. ='( 

This is not an area in the compiler I know much, if any, about. This can
most probably be worked on separately from the other patches; if someone
wants to contribute and can do the changes, it will be much
appreciated. :) 

> > *** For bytecode, assume C libraries contain the required primitives ***
> > When doing linking for bytecode with -custom, OCaml will dlopen() the
> > corresponding library and dlsym() the primitive as an early check for
> > their availability at runtime.
> > Quite obviously, this fails for cross-compilation and the only solution
> > I can think of is to disable the check completely. 
> 
> Probably there is no choice.  But it pains me to move link-time errors
> to run-time (more exactly, program startup time).  If we program in
> OCaml rather than Python, it's also for the early error detection.

I see no cross-platform solution either. However, I expect most people
to build natively before doing cross-compilation and that should catch
most link errors hopefully (that's a small consolation).

-- 
Adrien Nader

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Benoît Vaugon]

Hi,

For cross-compilation, have you seen opam cross-compiler packages for 
win32 and win64: https://github.com/vouillon/opam-windows-repository, 
and for android: https://github.com/vouillon/opam-android-repository? 
These repositories also provide opam packages for some cross-compiled 
libraries, which is not simple because it requires both the host ocaml 
compilers and the cross-ones. Itmight be interesting to combineour 
efforts...

As with your compiler, linking bytecode with -custom doesn't work with 
these compilers too, due to the same reason.

Probably just a naive question: to solve this problem, don't you think 
it would be possible to write a piece of OCaml code replacing the call 
to dlopen()/dlsym() in the OCaml compiler, that manually reads the C 
libraries and list the exported symbolsat compile time without loading them?

Benoît.

Le 21/02/2014 08:53, Adrien Nader a écrit :
> Hi Xavier,
>
> On Thu, Feb 20, 2014, Xavier Leroy wrote:
>>> ****************************** Approach ******************************
>>> *** A cross-compiler requires a native toolchain of the same version ***
>>> A working _native_ compiler toolchain of the _exact_ same version as the
>>> cross-toolchain is needed, both when building the cross-toolchain and
>>> when using it; the main reason was camlp4 but I believe it is still a
>>> valid restriction
>> Perhaps less so now that camlp4 is split off.  The alternative is to
>> configure for the host, populate boot/ with a ocamlrun and standard
>> library appropriate for the host, then reconfigure for the target, and
>> finish the build.
> Camlp4 was clearly the main component to require this but I have also
> been wondering about ocamldoc and ocamldep which can break because of
> new syntax.
>
> For now this requirements is probably the safe approach. Note that this
> restriction only checks for x.y.z versions, not for the tags than can
> come after 'z'; it made working with trunk too difficult.
>
> Also, as a packager, I prefer this. I find that the other approach makes
> packaging more difficult.
>
>>> *** Give up the restriction to POSIX for makefiles ***
>>> Either remove compatibility with BSD makes, or move GNU-make-isms to
>>> "GNUMakefile", BSD-makes-isms to "BSDMakefile" and common data to
>>> "Makefile.common" with the GNU/BSDMakefile files being the
>>> "entrypoints".
>> The unanimous message from the OCaml dev team is to commit on GNU make
>> and forget about compatibility with other variants of make.  BSD
>> people will forgive us, and it looks like the only sane way to merge
>> the Unix and Windows makefiles and support all the cross-compilation
>> scenarios.
> OK; I liked pmake and its descendants, partly because of their clean
> syntax with true flow control and thought it could be left as a
> possibility. Anyway, I will take that route.
>
>>> *** Use configure-defined variables instead of "ocaml{c,opt}"... ***
>>> The creation of the cross-compiler will rely on a native toolchain on
>>> ${build}. As such, ./configure will locate that toolchain and store its
>>> paths. This forbids any call like "./ocamlopt"; everything must rely on
>>> the paths found by configure.
>> See above: this might not be necessary.
> I believe you refer to configuring for host, populating boot and then
> configuring again for target. It's something I hadn't thought of and
> could work nicely. I need to experiment with this a bit and see how
> things would work out.
>
>>> *** Use Int64.t in the compiler instead of Nativeint ***
>>> Currently, OCaml uses Nativeint to handle all the integers in the code
>>> that is being compiled. The "native" is for ${build}, not for ${target}.
>>> With a 64b host and 32b target, OCaml will output its values over 64
>>> bits in the assembly and ld will then complain and truncate them.
>>> Move to Int64.t instead and delay the conversion to the right bitness
>>> until code emiting.
>> Constant propagation and maybe other passes of the compiler also need to
>> take bitsize into account.
>>
>> If you're going this way, it's not Int64 the appropriate substitute
>> for Nativeint: it's Targetint, defined as Int32 or Int64 depending on
>> target bitsize.  (This can be expressed with first-class modules.)
>>
>> There might be other ways if we assume bitsize(host) >= bitsize(target).
> I wish everyone (at least everyone doing devlopment) would have a 64b
> machine but it's not the case unfortunately. ='(
>
> This is not an area in the compiler I know much, if any, about. This can
> most probably be worked on separately from the other patches; if someone
> wants to contribute and can do the changes, it will be much
> appreciated. :)
>
>>> *** For bytecode, assume C libraries contain the required primitives ***
>>> When doing linking for bytecode with -custom, OCaml will dlopen() the
>>> corresponding library and dlsym() the primitive as an early check for
>>> their availability at runtime.
>>> Quite obviously, this fails for cross-compilation and the only solution
>>> I can think of is to disable the check completely.
>> Probably there is no choice.  But it pains me to move link-time errors
>> to run-time (more exactly, program startup time).  If we program in
>> OCaml rather than Python, it's also for the early error detection.
> I see no cross-platform solution either. However, I expect most people
> to build natively before doing cross-compilation and that should catch
> most link errors hopefully (that's a small consolation).
>

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Mark Shinwell]

On 21 February 2014 11:56, Benoît Vaugon <benoit.vaugon@gmail.com> wrote:
> Probably just a naive question: to solve this problem, don't you think it
> would be possible to write a piece of OCaml code replacing the call to
> dlopen()/dlsym() in the OCaml compiler, that manually reads the C libraries
> and list the exported symbolsat compile time without loading them?

You can use a cross binutils to do this, no?

Mark

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Adrien Nader]

You probably don't even need a cross binutils. Binutils is often built
for several formats. It then boils down to packaging politics.
In any case, building a binutils with support for additional
architectures is not an issue.

I'm a bit worried about the additional complexity to handle this case
and possibly the slowdown.

-- 
Adrien Nader

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Adrien Nader]

On Fri, Feb 21, 2014, Benoît Vaugon wrote:
> Hi,
> 
> For cross-compilation, have you seen opam cross-compiler packages
> for win32 and win64:
> https://github.com/vouillon/opam-windows-repository, and for
> android: https://github.com/vouillon/opam-android-repository? These
> repositories also provide opam packages for some cross-compiled
> libraries, which is not simple because it requires both the host
> ocaml compilers and the cross-ones. Itmight be interesting to
> combineour efforts...

I had been pointed to them a few months ago. From what I can tell, these
patches to the compiler cannot be integrated upstream; they require
"manual" steps like "make -C dir/ foo && mv bar baz && make -C dir2" and
a few changes which can only apply to windows targets.

By the way, the patches I've worked on were not started by me. However I
cannot trace their path properly; as far as I can tell, they were
started in Fedora, then went through debian and then mxe.cc.

Looking at these pages, I notice that I forgot one component for the
patches: ocamlfind and build system integration. IOW, how to cross-build
libraries. I'll add that to the RFC.

> As with your compiler, linking bytecode with -custom doesn't work
> with these compilers too, due to the same reason.

Actually, the patches I have make it possible to link bytecode with
-custom; it's simply that the test that would fail for cross-compilation
is skipped.

-- 
Adrien Nader

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Gabriel Kerneis]

On Sat, Feb 22, 2014 at 03:30:35PM +0100, Adrien Nader wrote:
> By the way, the patches I've worked on were not started by me. However I
> cannot trace their path properly; as far as I can tell, they were
> started in Fedora, then went through debian and then mxe.cc.

As far as I can tell, Richard Jones, from Red Hat, is the original
author:
http://caml.inria.fr/pub/ml-archives/caml-list/2008/11/48e7c6b3a40e5fa1d9555a3447820161.fr.html

Best,
-- 
Gabriel

Re: [Caml-list] [RFC] Remaining changes for cross-compilation support in OCaml

[Richard W.M. Jones]

On Sat, Feb 22, 2014 at 03:16:40PM +0000, Gabriel Kerneis wrote:
> On Sat, Feb 22, 2014 at 03:30:35PM +0100, Adrien Nader wrote:
> > By the way, the patches I've worked on were not started by me. However I
> > cannot trace their path properly; as far as I can tell, they were
> > started in Fedora, then went through debian and then mxe.cc.
> 
> As far as I can tell, Richard Jones, from Red Hat, is the original
> author:
> http://caml.inria.fr/pub/ml-archives/caml-list/2008/11/48e7c6b3a40e5fa1d9555a3447820161.fr.html

Quite a while back I hacked the OCaml compiler to enable cross-
compilation.  It kinda-sorta worked, but it was obvious [at that time,
2008 apparently] that it would take a lot of work upstream to make the
compiler friendly to cross-compilation.

I haven't look at cross-compilation of OCaml since, although of course
I'm still working on Fedora packaging, Aarch64 porting &c so I am
still fairly familiar with the compiler build system & internals.

Anyway, not sure what you were hoping I could do, except wish you good luck!

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW