[Caml-list] Library recompilation with OCamljava

[Caml-list] Library recompilation with OCamljava

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 2253 bytes --]

So, I'm attempting a large library recompile with ocamljava. It's pretty
audacious, because some people in my workplace are rather unwilling to
learn ocaml, yet a very well respected and needed library is authored in
it. Everyone knows java, so we hope very much to recompile the library just
with ocamljava.

Two important things that make me nervous in pursuing this effort are:

1) once fully recompiled, will the library work in java as it did with
native/ocaml byte code?

I've already started modifying the popular OCamlMakefile project to add a
java-byte-code target, and I've found that a threaded example is admitted
by the compiler produces an uncaught exception when run on the JVM... That
example may fall out of what the ocamljava docs has been safely implemented
under ocamljava, I'm not sure...

2) I've noticed that the rather large library that I need to compile, as my
final target, and aside from the toy targets I'm testing ocamljava with,
actually consumes some other C libraries and functions in it's dependency
path...

This is difficult; the traditional OCamlMakefile builds traditional c stubs
to .o files, correctly compiled with ocamlc. But when put on the ocamljava
command line, but ocamljava doesn't know what to do about it. Would there
be any way that I can have support for this ocaml feature as well, or
facilitate some way to link in or enable ocaml code that calls into C being
compiled down to java?

3) What if the answer to 2) is no/no?

If I can't use ocamljava, which is the most desired and elegant way,
allowing beautiful language inter-operation, how can I *best* facilitate
calls to the ocaml library? Is there a fast way to generate callbacks to
ocaml in java or any other language? It's very highly preferable not to
have to delegate back through the JNI due to type safety and fragility.
Alternatively, I looked at the OCaml library Restful, and wondered to
myself if there could be any kind of fast definition between ocaml types
and an exchange language, like json or something. Ideally, I'd like to be
able to generate a url per function in a very very simple declarative
manner, so that I can take ocaml libraries, and make them operate as a
service, where ocaml library functions correspond to URLs.

[-- Attachment #2: Type: text/html, Size: 2702 bytes --]

Re: [Caml-list] Library recompilation with OCamljava

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 2640 bytes --]

So, after doing some more work, I think the answer to 2) is no/no. After
looking more into piqi, I now just need to find a way to do protocol
buffers based RPC to an ocaml service...

Anybody know how to do that?

On Thu, Oct 16, 2014 at 4:08 AM, Kenneth Adam Miller <
kennethadammiller@gmail.com> wrote:

> So, I'm attempting a large library recompile with ocamljava. It's pretty
> audacious, because some people in my workplace are rather unwilling to
> learn ocaml, yet a very well respected and needed library is authored in
> it. Everyone knows java, so we hope very much to recompile the library just
> with ocamljava.
>
> Two important things that make me nervous in pursuing this effort are:
>
> 1) once fully recompiled, will the library work in java as it did with
> native/ocaml byte code?
>
> I've already started modifying the popular OCamlMakefile project to add a
> java-byte-code target, and I've found that a threaded example is admitted
> by the compiler produces an uncaught exception when run on the JVM... That
> example may fall out of what the ocamljava docs has been safely implemented
> under ocamljava, I'm not sure...
>
> 2) I've noticed that the rather large library that I need to compile, as
> my final target, and aside from the toy targets I'm testing ocamljava with,
> actually consumes some other C libraries and functions in it's dependency
> path...
>
> This is difficult; the traditional OCamlMakefile builds traditional c
> stubs to .o files, correctly compiled with ocamlc. But when put on the
> ocamljava command line, but ocamljava doesn't know what to do about it.
> Would there be any way that I can have support for this ocaml feature as
> well, or facilitate some way to link in or enable ocaml code that calls
> into C being compiled down to java?
>
> 3) What if the answer to 2) is no/no?
>
> If I can't use ocamljava, which is the most desired and elegant way,
> allowing beautiful language inter-operation, how can I *best* facilitate
> calls to the ocaml library? Is there a fast way to generate callbacks to
> ocaml in java or any other language? It's very highly preferable not to
> have to delegate back through the JNI due to type safety and fragility.
> Alternatively, I looked at the OCaml library Restful, and wondered to
> myself if there could be any kind of fast definition between ocaml types
> and an exchange language, like json or something. Ideally, I'd like to be
> able to generate a url per function in a very very simple declarative
> manner, so that I can take ocaml libraries, and make them operate as a
> service, where ocaml library functions correspond to URLs.
>
>

[-- Attachment #2: Type: text/html, Size: 3333 bytes --]

Re: [Caml-list] Library recompilation with OCamljava

[forum]

[-- Attachment #1: Type: text/plain, Size: 3861 bytes --]

Hi Kenneth,


Le 17 oct. 2014 à 23:33, Kenneth Adam Miller <kennethadammiller@gmail.com> a écrit :

> So, after doing some more work, I think the answer to 2) is no/no. After looking more into piqi, I now just need to find a way to do protocol buffers based RPC to an ocaml service...
> 
> Anybody know how to do that?
> 
> On Thu, Oct 16, 2014 at 4:08 AM, Kenneth Adam Miller <kennethadammiller@gmail.com> wrote:
> So, I'm attempting a large library recompile with ocamljava. It's pretty audacious, because some people in my workplace are rather unwilling to learn ocaml, yet a very well respected and needed library is authored in it. Everyone knows java, so we hope very much to recompile the library just with ocamljava. 
> 
> Two important things that make me nervous in pursuing this effort are:
> 
> 1) once fully recompiled, will the library work in java as it did with native/ocaml byte code?
> I've already started modifying the popular OCamlMakefile project to add a java-byte-code target, and I've found that a threaded example is admitted by the compiler produces an uncaught exception when run on the JVM... That example may fall out of what the ocamljava docs has been safely implemented under ocamljava, I'm not sure...

This sounds  like a bug... It would be nice to report it at https://github.com/xclerc/ocamljava,
particularly if you have a (small) reproduction case. The "thread" library is indeed only lightly
tested, mainly because the "concurrent" library (which is specific to OCaml-Java) includes
another implementation of threads, that is much closer to Java threads.


> 2) I've noticed that the rather large library that I need to compile, as my final target, and aside from the toy targets I'm testing ocamljava with, actually consumes some other C libraries and functions in it's dependency path...
> This is difficult; the traditional OCamlMakefile builds traditional c stubs to .o files, correctly compiled with ocamlc. But when put on the ocamljava command line, but ocamljava doesn't know what to do about it. Would there be any way that I can have support for this ocaml feature as well, or facilitate some way to link in or enable ocaml code that calls into C being compiled down to java? 

OCaml-Java does not know how to handle C files, as additional primitives should be implemented
through Java files. As a consequence, when porting a project from ocamlc/ocamlopt to ocamljava,
you have to port C files to Java files.


> 
> 3) What if the answer to 2) is no/no?
> If I can't use ocamljava, which is the most desired and elegant way, allowing beautiful language inter-operation, how can I best facilitate calls to the ocaml library? Is there a fast way to generate callbacks to ocaml in java or any other language? It's very highly preferable not to have to delegate back through the JNI due to type safety and fragility. Alternatively, I looked at the OCaml library Restful, and wondered to myself if there could be any kind of fast definition between ocaml types and an exchange language, like json or something. Ideally, I'd like to be able to generate a url per function in a very very simple declarative manner, so that I can take ocaml libraries, and make them operate as a service, where ocaml library functions correspond to URLs.

I think a neat way to use a C library from an ocamljava-compiled program would be to have a
Java "backend" for Jeremy Yallop's ctypes (https://github.com/ocamllabs/ocaml-ctypes).
I never had the time to implement that, but toyed with this idea and think the best way to
implement it would be to go through JNA (https://github.com/twall/jna) rather than JNI.
JNA includes a "dlopen"-like mechanism, and automatically maps simple types from Java
to C. My knowledge of ctypes is quite limited, but I see no showstopper.


Regards,

Xavier

[-- Attachment #2: Type: text/html, Size: 6263 bytes --]

Re: [Caml-list] Library recompilation with OCamljava

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 6225 bytes --]

Ok, to replicate the bug, just go to
https://github.com/KennethAdamMiller/ocaml-makefile where I'm updating the
fork of ocamlmakefile in order to allow a target to compile for ocamljava.
Once I update (I've committed, but haven't pushed yet, will within a matter
of hours hours), you can clone or pull, and cd to the threads subdirectory,
which is an example project that consumes the ocamlmakefile. Then just do

make jbc  #or make java-byte-code

ex1 and ex2 crashe with uncaught under/overflow exception and some other
overflow errors (not reliable); I think it's because the synchronization
that is expressed in the ocaml code isn't very well followed at java's
level; there seems to be some very great imbalance in thread scheduling
(possibly slow signal handling). In java, there's a long pause where the
output doesn't make it to the screen until the uncaught exception hits,
when it does, the numbers being passed have a huge difference. In contrast,
ocamlopt or ocamlc compiled code executes rather evenly, with each event
receiving scheduler attention pretty much equally. That's my jest.

Agreed! I'd much much rather construct some method of automatic (or even
just expedient manual intervention!) that offers ocaml -> C sublibrary :
java -> C sublibrary support! I've already started down the path of
ocamljava, and right now, even ocaml-RPC with protocol buffers looks like a
long haul as well! ocamljava is just a much better solution.

Precisely! ocaml-ctypes is exactly what's being used by the library that
I'm porting to call into C sub libraries. It would be really sweet if the
ocamljava compiler could detect the ocaml-ctypes and generate these
mappings automatically. This would eliminate a lot of error prone code,
since C code tends to interpret data raw at some point... How might I got
about writing this to boot? I'm moderately new to ocaml, lacking deep
expertise in it, but I'm an aggressive learner. Please explain the best
path forward, I want to create a robust and reusable solution.

On Sat, Oct 18, 2014 at 5:09 AM, forum@x9c.fr <forum@x9c.fr> wrote:

> Hi Kenneth,
>
>
> Le 17 oct. 2014 à 23:33, Kenneth Adam Miller <kennethadammiller@gmail.com>
> a écrit :
>
> So, after doing some more work, I think the answer to 2) is no/no. After
> looking more into piqi, I now just need to find a way to do protocol
> buffers based RPC to an ocaml service...
>
> Anybody know how to do that?
>
> On Thu, Oct 16, 2014 at 4:08 AM, Kenneth Adam Miller <
> kennethadammiller@gmail.com> wrote:
>
>> So, I'm attempting a large library recompile with ocamljava. It's pretty
>> audacious, because some people in my workplace are rather unwilling to
>> learn ocaml, yet a very well respected and needed library is authored in
>> it. Everyone knows java, so we hope very much to recompile the library just
>> with ocamljava.
>>
>> Two important things that make me nervous in pursuing this effort are:
>>
>> 1) once fully recompiled, will the library work in java as it did with
>> native/ocaml byte code?
>>
>> I've already started modifying the popular OCamlMakefile project to add a
>> java-byte-code target, and I've found that a threaded example is admitted
>> by the compiler produces an uncaught exception when run on the JVM... That
>> example may fall out of what the ocamljava docs has been safely implemented
>> under ocamljava, I'm not sure...
>>
>>
> This sounds  like a bug... It would be nice to report it at
> https://github.com/xclerc/ocamljava,
> particularly if you have a (small) reproduction case. The "thread" library
> is indeed only lightly
> tested, mainly because the "concurrent" library (which is specific to
> OCaml-Java) includes
> another implementation of threads, that is much closer to Java threads.
>
>
> 2) I've noticed that the rather large library that I need to compile, as
>> my final target, and aside from the toy targets I'm testing ocamljava with,
>> actually consumes some other C libraries and functions in it's dependency
>> path...
>>
>> This is difficult; the traditional OCamlMakefile builds traditional c
>> stubs to .o files, correctly compiled with ocamlc. But when put on the
>> ocamljava command line, but ocamljava doesn't know what to do about it.
>> Would there be any way that I can have support for this ocaml feature as
>> well, or facilitate some way to link in or enable ocaml code that calls
>> into C being compiled down to java?
>>
>>
> OCaml-Java does not know how to handle C files, as additional primitives
> should be implemented
> through Java files. As a consequence, when porting a project from
> ocamlc/ocamlopt to ocamljava,
> you have to port C files to Java files.
>
>
>
>> 3) What if the answer to 2) is no/no?
>>
>> If I can't use ocamljava, which is the most desired and elegant way,
>> allowing beautiful language inter-operation, how can I *best* facilitate
>> calls to the ocaml library? Is there a fast way to generate callbacks to
>> ocaml in java or any other language? It's very highly preferable not to
>> have to delegate back through the JNI due to type safety and fragility.
>> Alternatively, I looked at the OCaml library Restful, and wondered to
>> myself if there could be any kind of fast definition between ocaml types
>> and an exchange language, like json or something. Ideally, I'd like to be
>> able to generate a url per function in a very very simple declarative
>> manner, so that I can take ocaml libraries, and make them operate as a
>> service, where ocaml library functions correspond to URLs.
>>
>>
> I think a neat way to use a C library from an ocamljava-compiled program
> would be to have a
> Java "backend" for Jeremy Yallop's ctypes (
> https://github.com/ocamllabs/ocaml-ctypes).
> I never had the time to implement that, but toyed with this idea and think
> the best way to
> implement it would be to go through JNA (https://github.com/twall/jna)
> rather than JNI.
> JNA includes a "dlopen"-like mechanism, and automatically maps simple
> types from Java
> to C. My knowledge of ctypes is quite limited, but I see no showstopper.
>
>
> Regards,
>
> Xavier
>

[-- Attachment #2: Type: text/html, Size: 8783 bytes --]

Re: [Caml-list] Library recompilation with OCamljava

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 6718 bytes --]

Ok, I've pushed to master-you can now pull and appreciate the new ocaml
target, jbc, which compiles for ocamljava. From there, recompiling each of
the examples provided (except calc, since it uses C, and that's under dev).

On Sat, Oct 18, 2014 at 6:04 AM, Kenneth Adam Miller <
kennethadammiller@gmail.com> wrote:

> Ok, to replicate the bug, just go to
> https://github.com/KennethAdamMiller/ocaml-makefile where I'm updating
> the fork of ocamlmakefile in order to allow a target to compile for
> ocamljava. Once I update (I've committed, but haven't pushed yet, will
> within a matter of hours hours), you can clone or pull, and cd to the
> threads subdirectory, which is an example project that consumes the
> ocamlmakefile. Then just do
>
> make jbc  #or make java-byte-code
>
> ex1 and ex2 crashe with uncaught under/overflow exception and some other
> overflow errors (not reliable); I think it's because the synchronization
> that is expressed in the ocaml code isn't very well followed at java's
> level; there seems to be some very great imbalance in thread scheduling
> (possibly slow signal handling). In java, there's a long pause where the
> output doesn't make it to the screen until the uncaught exception hits,
> when it does, the numbers being passed have a huge difference. In contrast,
> ocamlopt or ocamlc compiled code executes rather evenly, with each event
> receiving scheduler attention pretty much equally. That's my jest.
>
> Agreed! I'd much much rather construct some method of automatic (or even
> just expedient manual intervention!) that offers ocaml -> C sublibrary :
> java -> C sublibrary support! I've already started down the path of
> ocamljava, and right now, even ocaml-RPC with protocol buffers looks like a
> long haul as well! ocamljava is just a much better solution.
>
> Precisely! ocaml-ctypes is exactly what's being used by the library that
> I'm porting to call into C sub libraries. It would be really sweet if the
> ocamljava compiler could detect the ocaml-ctypes and generate these
> mappings automatically. This would eliminate a lot of error prone code,
> since C code tends to interpret data raw at some point... How might I got
> about writing this to boot? I'm moderately new to ocaml, lacking deep
> expertise in it, but I'm an aggressive learner. Please explain the best
> path forward, I want to create a robust and reusable solution.
>
> On Sat, Oct 18, 2014 at 5:09 AM, forum@x9c.fr <forum@x9c.fr> wrote:
>
>> Hi Kenneth,
>>
>>
>> Le 17 oct. 2014 à 23:33, Kenneth Adam Miller <kennethadammiller@gmail.com>
>> a écrit :
>>
>> So, after doing some more work, I think the answer to 2) is no/no. After
>> looking more into piqi, I now just need to find a way to do protocol
>> buffers based RPC to an ocaml service...
>>
>> Anybody know how to do that?
>>
>> On Thu, Oct 16, 2014 at 4:08 AM, Kenneth Adam Miller <
>> kennethadammiller@gmail.com> wrote:
>>
>>> So, I'm attempting a large library recompile with ocamljava. It's pretty
>>> audacious, because some people in my workplace are rather unwilling to
>>> learn ocaml, yet a very well respected and needed library is authored in
>>> it. Everyone knows java, so we hope very much to recompile the library just
>>> with ocamljava.
>>>
>>> Two important things that make me nervous in pursuing this effort are:
>>>
>>> 1) once fully recompiled, will the library work in java as it did with
>>> native/ocaml byte code?
>>>
>>> I've already started modifying the popular OCamlMakefile project to add
>>> a java-byte-code target, and I've found that a threaded example is admitted
>>> by the compiler produces an uncaught exception when run on the JVM... That
>>> example may fall out of what the ocamljava docs has been safely implemented
>>> under ocamljava, I'm not sure...
>>>
>>>
>> This sounds  like a bug... It would be nice to report it at
>> https://github.com/xclerc/ocamljava,
>> particularly if you have a (small) reproduction case. The "thread"
>> library is indeed only lightly
>> tested, mainly because the "concurrent" library (which is specific to
>> OCaml-Java) includes
>> another implementation of threads, that is much closer to Java threads.
>>
>>
>> 2) I've noticed that the rather large library that I need to compile, as
>>> my final target, and aside from the toy targets I'm testing ocamljava with,
>>> actually consumes some other C libraries and functions in it's dependency
>>> path...
>>>
>>> This is difficult; the traditional OCamlMakefile builds traditional c
>>> stubs to .o files, correctly compiled with ocamlc. But when put on the
>>> ocamljava command line, but ocamljava doesn't know what to do about it.
>>> Would there be any way that I can have support for this ocaml feature as
>>> well, or facilitate some way to link in or enable ocaml code that calls
>>> into C being compiled down to java?
>>>
>>>
>> OCaml-Java does not know how to handle C files, as additional primitives
>> should be implemented
>> through Java files. As a consequence, when porting a project from
>> ocamlc/ocamlopt to ocamljava,
>> you have to port C files to Java files.
>>
>>
>>
>>> 3) What if the answer to 2) is no/no?
>>>
>>> If I can't use ocamljava, which is the most desired and elegant way,
>>> allowing beautiful language inter-operation, how can I *best*
>>> facilitate calls to the ocaml library? Is there a fast way to generate
>>> callbacks to ocaml in java or any other language? It's very highly
>>> preferable not to have to delegate back through the JNI due to type safety
>>> and fragility. Alternatively, I looked at the OCaml library Restful, and
>>> wondered to myself if there could be any kind of fast definition between
>>> ocaml types and an exchange language, like json or something. Ideally, I'd
>>> like to be able to generate a url per function in a very very simple
>>> declarative manner, so that I can take ocaml libraries, and make them
>>> operate as a service, where ocaml library functions correspond to URLs.
>>>
>>>
>> I think a neat way to use a C library from an ocamljava-compiled program
>> would be to have a
>> Java "backend" for Jeremy Yallop's ctypes (
>> https://github.com/ocamllabs/ocaml-ctypes).
>> I never had the time to implement that, but toyed with this idea and
>> think the best way to
>> implement it would be to go through JNA (https://github.com/twall/jna)
>> rather than JNI.
>> JNA includes a "dlopen"-like mechanism, and automatically maps simple
>> types from Java
>> to C. My knowledge of ctypes is quite limited, but I see no showstopper.
>>
>>
>> Regards,
>>
>> Xavier
>>
>
>

[-- Attachment #2: Type: text/html, Size: 9463 bytes --]

Re: [Caml-list] Library recompilation with OCamljava

[Jeremy Yallop]

[+cc ctypes@lists.ocaml.org]

On Sat, Oct 18, 2014 at 5:09 AM, forum@x9c.fr <forum@x9c.fr> wrote:
>> I think a neat way to use a C library from an ocamljava-compiled
>> program would be to have a Java "backend" for Jeremy Yallop's
>> ctypes (https://github.com/ocamllabs/ocaml-ctypes).  I never had
>> the time to implement that, but toyed with this idea and think the
>> best way to implement it would be to go through JNA
>> (https://github.com/twall/jna) rather than JNI.  JNA includes a
>> "dlopen"-like mechanism, and automatically maps simple types from
>> Java to C. My knowledge of ctypes is quite limited, but I see no
>> showstopper.

On 18/10/2014, Kenneth Adam Miller <kennethadammiller@gmail.com> wrote:
> Precisely! ocaml-ctypes is exactly what's being used by the library
> that I'm porting to call into C sub libraries. It would be really
> sweet if the ocamljava compiler could detect the ocaml-ctypes and
> generate these mappings automatically. This would eliminate a lot of
> error prone code, since C code tends to interpret data raw at some
> point... How might I got about writing this to boot? I'm moderately
> new to ocaml, lacking deep expertise in it, but I'm an aggressive
> learner. Please explain the best path forward, I want to create a
> robust and reusable solution.

OCaml-Java support is on the wish list for ctypes

    https://github.com/ocamllabs/ocaml-ctypes/issues/13

but we don't have the resources to implement it at present.  Adding
support would mostly likely only involve changing ctypes itself, not
OCaml-Java, and is likely to involve writing OCaml-Java
implementations of the following components:

  (1) memory access, i.e. functions for allocating blocks, for reading
and writing scalar values to arbitrary addresses, and for viewing C
objects as bigarrays:
     https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes/memory_stubs.ml
     https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes/bigarray_stubs.ml

  (2) functions for printing primitive (scalar) values:
     https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes/value_printing_stubs.ml

  (3) implementations of signed and unsigned integer types of various sizes:
     https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes/signed.ml
     https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes/unsigned.ml

  (4) functions for converting between OCaml and C string representations
     https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes/std_view_stubs.ml

plus one of the following approaches for calling functions

  (a) a "dynamic" approach, which resolves symbols and constructs call
frames at runtime, like the ctypes Foreign module.  This involves two
components: a dynamic loading interface along the
      following lines
      https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes-foreign-base/dl.mli
      and primitives for dynamically constructing and making calls:
      https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes-foreign-base/ffi_stubs.ml

  (b) a "static" approach, which generates code to be compiled by the
standard toolchain:
      https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/cstubs/cstubs.mli

Note that most of the links above are to internal Ctypes modules, not
to the interface, which I'd expect to remain largely unchanged.

As Xavier suggests, JNA may be a good starting point for some or all
of the above.  If someone would like to look at adding OCaml-Java
support, please feel free to ask questions, either on the GitHub issue
tracker, or on the ctypes mailing list.

Jeremy.