RE: [Caml-list] Dynamic linking

RE: [Caml-list] Dynamic linking

[Alexander Bottema]

But the problem with a DLL is that you never now what address will be
allocated for your executable. It can be located anywhere between 0 and
2^64-1. Thus, when a DLL calls another DLL a 64-bit call is potentially
required. AMD64 supports global address tables that enable you to
translate a 32-bit call into a 64-bit one (likewise with data access).
If you compile a C file with gcc (-fPIC -S) you'll get code that looks
like this:

The C file (amd64.c)
====================
extern int foo();

int xyzzy = 42;

int main(int argc, char *argv)
{
    xyzzy = 4711;
    return foo();
}

The assembly output (amd64.S) for gcc -O2 -fPIC -S
==================================================

	.file	"amd64.c"
.globl xyzzy
	.data
	.align 4
	.type	xyzzy,@object
	.size	xyzzy,4
xyzzy:
	.long	42
	.text
	.p2align 2,,3
.globl main
	.type	main,@function
main:
	pushl	%ebp
	movl	%esp, %ebp
	pushl	%ebx
	pushl	%eax
	call	.L2
.L2:
	popl	%ebx
	addl	$_GLOBAL_OFFSET_TABLE_+[.-.L2], %ebx
	movl	xyzzy@GOT(%ebx), %eax
	andl	$-16, %esp
	movl	$4711, (%eax)
	call	foo@PLT
	movl	-4(%ebp), %ebx
	leave
	ret
.Lfe1:
	.size	main,.Lfe1-main
	.ident	"GCC: (GNU) 3.2.3"

For OCaml to work you need to emit instructions of type 'call foo@PLT'
and 'movl xyzzy@GOT(%ebx)'. Currently, the OCaml does not do this for
AMD64, which is the heart of the problem. If you think this is trivial
to fix, please go head and do it; I'd be very happy.

Alexander Bottema
The MathWorks

-----Original Message-----
From: skaller [mailto:skaller@users.sourceforge.net] 
Sent: Tuesday, October 04, 2005 10:48 AM
To: Alexander Bottema
Cc: caml-list@yquem.inria.fr
Subject: RE: [Caml-list] Dynamic linking

On Tue, 2005-10-04 at 08:47 -0400, Alexander Bottema wrote:

> Now, as a developer it would be nice if it was possible to write
certain
> parts in OCaml. The foreign language interface is good enough to be
used
> in conjunction with C/C++, but the lack of support to use it in DLLs
> makes it impossible for us to have any OCaml code in our leaf product.
> 
> Nevertheless, I really tried to make it work (and I almost did). It's
> only Linux/AMD64 that I couldn't solve and I really tried to modify
the
> AMD64 emitter.

I actually find this very strange. The x86 is incapable of
supporting position independent code without hackery or 
segmentation:

(a) there is no IP relative call instruction (only short branches)
(b) there is no IP relative data access (at all)

The amd64 (x86_64) on the other hand supports IP relative
calls and data access (with 32 bit offsets only, but
usually enough for all code and global data).


> > The only platform which I couldn't get to work was Linux AMD64. I
> tried
> > to modify the OCaml emitter for AMD64 to accommodate position
> > independent code using the offset tables, but it was too hard.

There is no need for offset tables. Just make sure all
'absolute' addresses accesses are changed to be RIP relative
and the assembler will do the rest automatically.

[There may be a problem on Windows, apparently the linker
can't add two offsets together due to a constraint in COFF
format, not sure about that though]

See section 2.2.5 of the AMD document 24592.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

RE: [Caml-list] Dynamic linking

[skaller]

On Tue, 2005-10-04 at 11:56 -0400, Alexander Bottema wrote:
> But the problem with a DLL is that you never now what address will be
> allocated for your executable. It can be located anywhere between 0 and
> 2^64-1. 

Not on current versions of Linux/x86_64. They only support 48 bit
code addresses (I think it is 48 bits), this is the so-called
'small model'.

> Thus, when a DLL calls another DLL a 64-bit call is potentially
> required. AMD64 supports global address tables that enable you to
> translate a 32-bit call into a 64-bit one (likewise with data access).
> If you compile a C file with gcc (-fPIC -S) you'll get code that looks
> like this:

I know ..but it has NOTHING to do with position independent code.

> For OCaml to work you need to emit instructions of type 'call foo@PLT'
> and 'movl xyzzy@GOT(%ebx)'. Currently, the OCaml does not do this for
> AMD64, which is the heart of the problem. If you think this is trivial
> to fix, please go head and do it; I'd be very happy.

This has nothing to do with making the *code* position independent.
It is to do with supporting elf/ld.so dynamic loading. This is
another issue entirely. 

Clearly, to access a symbol of an 'unknown' address,
you put the address into a known table slot instead,
when the library is loaded, and access indirectly.

In any case, this doesn't solve the problem, but it does
identify it correctly: the problem is to emit instructions
to bind to load-time linked libraries on Linux/Elf/ld.so 
platforms. You can be sure Windows -- for the same processor --
uses a different dynamic linkage model .. :)


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Dynamic linking

[Jon Harrop]

On Friday 15 June 2007 17:03:33 Alain Frisch wrote:
> Enjoy and send some feedback!

I love the native-code top-level!

Any chance you could get MetaOCaml running on AMD64?

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
The OCaml Journal
http://www.ffconsultancy.com/products/ocaml_journal/?e

Re: [Caml-list] Dynamic linking

[Alain Frisch]

Hello again,

It is now possible to try the natdynlink branch with GODI:

- check that you are on section 3.10 (GODI_SECTION = 3.10 in your
godi.conf);

- "update the list of available packages"

- in the configuration screen for godi-ocaml-src, set these variables:
  OCAML_CVS_CHECKOUT = yes
  OCAML_CVS_REVISION = natdynlink

- ask GODI to rebuild the two packages godi-ocaml-src and godi-ocaml.


  Alain

Re: [Caml-list] Dynamic linking

[Jeffrey Loren Shaw]

Philippe, here's a little program I wrote that you might find useful. 

compile: 

ocamlopt -o keepsource keepsource.ml 

usage: 

./keepsource foo.ml 

creates a new file foo_keepsource.ml. If you compile it to the executable 
foo you can do ./foo -source to get the original source back. 

./keepsource foo.ml -compile 

does the same thing bug gives you an executable named foo in addition to 
foo_keepsource.ml. 

It will only work on single-file programs, and I haven't tested it much. 

(* keepsource.ml *) 

let replace_quote str =
 let result = ref [] in
 for i = 0 to String.length str - 1 do
   match str.[i] with
     '\\' -> result := '\\'::'\\'::!result
   | '\"' -> result := '\"'::'\\'::!result
   | a -> result := a::!result
 done;
 let buffer = " " in
 List.fold_left
   (fun accum a ->
      buffer.[0] <- a;
      buffer ^ accum
   )
   ""
   !result;; 

let print_help () =
   print_endline
   "usage: keepsource infile.ml (-compile)";; 

if Array.length Sys.argv = 1 then print_help ();; 

let sourcefile = Sys.argv.(1);; 

let sourcefile_keepsource = String.sub sourcefile 0 (String.length 
sourcefile - 3) ^ "_keepsource.ml";; 

if Sys.file_exists sourcefile then
 let inchan = open_in sourcefile in
 let source = ref [] in
 (
   try
     while true do
	source := (input_line inchan)::!source
     done
   with
     End_of_file -> ()
 );
 close_in inchan;
 source := List.rev !source;
 let outchan = open_out sourcefile_keepsource in
 let output = output_string outchan in
 output "let sourcecode = \"";
 List.iter (fun s -> output (replace_quote s^"\n")) !source;
 output "\"\n\nlet main () = (";
 List.iter (fun s -> output (s^"\n")) !source;
 output ");; 

if Array.length Sys.argv > 1 then
 if Sys.argv.(1) = \"-source\" then
   (
     print_endline sourcecode;
     exit 0
   )
 else
   main ()
 else
   main ()
";
 close_out outchan;; 

if Array.length Sys.argv > 2 then
 if Sys.argv.(2) = "-compile" then
   let compile_command =
     "ocamlopt -o " ^ String.sub sourcefile 0 (String.length sourcefile - 3) 
^ " "^ sourcefile_keepsource in
   ignore
     (
	Sys.command compile_command
     ) 

Re: [Caml-list] Dynamic linking

[Philippe Wang]

skaller wrote:
> On Sat, 2007-06-16 at 18:33 +0100, Jon Harrop wrote:
>   
>> On Friday 15 June 2007 19:24:20 Alain Frisch wrote:
>>     
>>> Except for the native toplevel which will probably not be included.
>>>       
>> The native top-level is the bit that interests me the most, as this combines 
>> interactivity with OCaml's superb native-code performance.
>>     
>
> I do not understand how the toplevel is even remotely useful:
> it isn't "interactive" in any real sense of the word.
>
> [...]

A native toplevel can be (very) useful when using ocaml as a scripting 
language.

I like to write (web) scripts with ocaml when I don't want to spend a 
lot of time on searching for trivial security holes (i.e. code 
injection) with languages such as bash or perl.
And compiling a file and using the binaries is a risk of loosing the 
source...
I know, this may sound weird...

--
Philippe Wang

Re: [Caml-list] Dynamic linking

[Christophe TROESTLER]

On Sun, 17 Jun 2007, skaller <skaller@users.sourceforge.net> wrote:
> 
> So exactly what is the advantage of the top level?
> 
> Yes, I use it occasionally for testing 1-2 line
> fragments, mainly the typing, but i just can't
> envisage how a ray tracer could be use productively
> in such an environment, as compared to just making
> a suitable file and compiling and running it.

I generally also type code in a file and use C-x C-e if I need to get
an immediate feedback.  However, native performance can be useful in
order (say) to perform interactive data analysis and use pure ocaml
libraries (as opposed to C wrappers).  Data analysis requires some
kind of fast feeback because you generally only discover what you want
to do during the analysis itself...

ChriS

Re: [Caml-list] Dynamic linking

[skaller]

On Sat, 2007-06-16 at 18:33 +0100, Jon Harrop wrote:
> On Friday 15 June 2007 19:24:20 Alain Frisch wrote:
> > Except for the native toplevel which will probably not be included.
> 
> The native top-level is the bit that interests me the most, as this combines 
> interactivity with OCaml's superb native-code performance.

I do not understand how the toplevel is even remotely useful:
it isn't "interactive" in any real sense of the word.

Natively you can't even edit lines, it's very lame.
Even with ledit, what you can do is very limited.

I fail to see how this is any better than preparing
a file and compiling it -- that way you get a proper
editor. If you're using emacs or some other souped up
editor it works much like an interactive IDE anyhow,
only better..

So exactly what is the advantage of the top level?

Yes, I use it occasionally for testing 1-2 line
fragments, mainly the typing, but i just can't
envisage how a ray tracer could be use productively
in such an environment, as compared to just making
a suitable file and compiling and running it.

It IS a bit of a pain, setting up the build environment
to do that, but I can't imagine it's that hard to automate
with something like emacs, or one of the make systems.


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Dynamic linking

[Jon Harrop]

On Friday 15 June 2007 19:24:20 Alain Frisch wrote:
> Except for the native toplevel which will probably not be included.

The native top-level is the bit that interests me the most, as this combines 
interactivity with OCaml's superb native-code performance.

I just benchmarked the new top-level and it runs my ray tracer 17x faster:

$ time ocaml unix.cma ray.ml >image.pgm

real    1m11.656s
user    2m21.106s
sys     0m0.331s

$ time ocamlnat unix.cmxa ray.ml >image.pgm

real    0m4.294s
user    0m6.265s
sys     0m0.521s

Interactive compilation to native code was one of the main benefits of F# over 
OCaml. So I, for one, would dearly love to see this in the mainstream OCaml!

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
OCaml for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists/?e

Re: [Caml-list] Dynamic linking

[Alain Frisch]

Quôc Peyrot wrote:
>> For Linux x86, the change is rather small and straightforward.
> 
> Is it working (or going to work) on PPC (Mac OS X)?

It is not currently working. The code generator must be adapted to
produce PIC code. I've started to do that but it doesn't work and I
won't be able to finish it. If someone is interested to help, please
tell me.

-- Alain

Re: [Caml-list] Dynamic linking

[skaller]

On Sat, 2007-06-16 at 00:45 +0200, Christophe TROESTLER wrote:
> On Sat, 16 Jun 2007, skaller <skaller@users.sourceforge.net> wrote:
> > 
> > A question, probably not directed at Alain: will there be support
> > for AMD64/64 bit Ocaml for Windows XP64/VC2005 +/- Vista?
> > Windows ports are all 32 bit at the moment aren't they?
> 
> Extract from README.win32 :
> 
>                                       Native MS     Native MinGW        Cygwin
> 
> 64 bits?                            Win32 or Win64    Win32 only      Win32 only

Ooops, OK! Sweet!

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Dynamic linking

[Christophe TROESTLER]

On Sat, 16 Jun 2007, skaller <skaller@users.sourceforge.net> wrote:
> 
> A question, probably not directed at Alain: will there be support
> for AMD64/64 bit Ocaml for Windows XP64/VC2005 +/- Vista?
> Windows ports are all 32 bit at the moment aren't they?

Extract from README.win32 :

                                      Native MS     Native MinGW        Cygwin

64 bits?                            Win32 or Win64    Win32 only      Win32 only

Third-party software required
  - for base bytecode system            none            none            none
  - for ocamlc -custom                  MSVC            Cygwin          Cygwin
  - for native-code generation          MSVC+MASM       Cygwin          Cygwin

Speed of bytecode interpreter           70%             100%            100%

Replay debugger                         no              no              yes

The Unix library                        partial         partial         full

The Threads library                     yes             yes             yes

The Graphics library                    yes             yes             no

Restrictions on generated executables?  none            none            yes (*)


ChriS

Re: [Caml-list] Dynamic linking

[Quôc Peyrot]

On Jun 15, 2007, at 8:24 PM, Alain Frisch wrote:

> Brian Hurt wrote:
>> B) How extensive were the changes to the Ocaml source code?
>
> For Linux x86, the change is rather small and straightforward.

Is it working (or going to work) on PPC (Mac OS X)?

-- 
Best Regards,
Quôc

Re: [Caml-list] Dynamic linking

[skaller]

On Fri, 2007-06-15 at 18:03 +0200, Alain Frisch wrote:
> Hello,
> 
> I'm pleased to announce the existence of an experimental branch in
> OCaml's CVS. Main changes:
> 
> * Dynamic loading of OCaml code in native code programs, through
>   the Dynlink module.
> 
> * Simplified dynamic linking of C code under Windows.
> 
> 
> More info: http://alain.frisch.fr/natdynlink.html
> 
> Enjoy and send some feedback!

A question, probably not directed at Alain: will there be support
for AMD64/64 bit Ocaml for Windows XP64/VC2005 +/- Vista?
Windows ports are all 32 bit at the moment aren't they?

Also a question: how does the Windows native (MSVC based) Ocaml
code find C libraries? The VC2003 style DLL-hell approach is now
deprecated on Windows platforms: you have to use Assemblies,
that is, ship applications with manifests and related components
which are installed in system caches to get around DLL-hell.

The resulting system is horrible to prepare for vendors I think,
I haven't figured it out, but it does support multi-version and
multi-arch, and seems more sophisticated the Linux versioning
technology.

The bottom line, however, is you simply cannot link to a DLL
by just plonking it in the PATH anymore.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Dynamic linking

[Alain Frisch]

Jon Harrop wrote:
> On Friday 15 June 2007 19:24:20 Alain Frisch wrote:
>> ... (Except for the native toplevel which will probably not be included.)
> 
> Is that "native" as in we're going to get an OCaml top-level with native-code 
> performance? If so, this is hugely important for me...

What kind of application do you have in mind?

It's "native" as in "we have an OCaml top-level with native-code
performance and a large latency". Each phrase is compiled with
ocamlopt's backend to an assembler file, which is then assembled, linked
into a shared library, dl-opened and then run. (Shared libraries will
never be closed so your OS must be strong enough to support that.)
Performances will likely be equal to ocamlopt's generated code (in
particular, we have cross-phrase optimizations).

Feel free to play with it ("make ocamlnat") and tell us whether this is
adequate for what you want to do.

  Alain

Re: [Caml-list] Dynamic linking

[Jon Harrop]

This is incredible. Thank you so much!

On Friday 15 June 2007 19:24:20 Alain Frisch wrote:
> ... (Except for the native toplevel which will probably not be included.)

Is that "native" as in we're going to get an OCaml top-level with native-code 
performance? If so, this is hugely important for me...

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
OCaml for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists/?e

Re: [Caml-list] Dynamic linking

[Alain Frisch]

Brian Hurt wrote:
> B) How extensive were the changes to the Ocaml source code?

For Linux x86, the change is rather small and straightforward.

For Windows ports, it is more tricky, because a Windows DLL cannot
reference symbols from the main program. I wrote the FlexDLL tool to
simulate a POSIX-ish dlopen API, so that the patch to OCaml itself
mostly reduces to removing code or sharing Makefiles with Unix. All the
trouble is encapsulated into FlexDLL (which is itself a quite small
piece of code, but a tricky one).

>From Linux AMD64, the ABI/dynamic linker does not allow to dynamically
load position dependant code, so I had to modify the last stage of
ocamlopt's backend in order to produce PIC code (when the option -dlcode
is given; note that this is independant from the existing -fpic option,
details on demand). Again, this is a rather small and unintrusive change.

You can use CVS to get a diff between the ocaml3100 tag and the
natdynlink branch (but it won't show you new files; you can always "cvs
checkout" both and do a local diff).

> How likely
> is it that this will get included in some future version of the main trunk?

It is likely to be included in OCaml 3.11, not before. (Except for the
native toplevel which will probably not be included.) Of course,
feedback can only help for the integration.

-- Alain

Re: [Caml-list] Dynamic linking

[Brian Hurt]

Alain Frisch wrote:

>Hello,
>
>I'm pleased to announce the existence of an experimental branch in
>OCaml's CVS. Main changes:
>
>* Dynamic loading of OCaml code in native code programs, through
>  the Dynlink module.
>
>* Simplified dynamic linking of C code under Windows.
>
>
>More info: http://alain.frisch.fr/natdynlink.html
>
>Enjoy and send some feedback!
>
>Alain
>
>  
>

A) Thanks, this is usefull.

B) How extensive were the changes to the Ocaml source code?  How likely 
is it that this will get included in some future version of the main trunk?

Brian

Re: [Caml-list] Dynamic linking

[Joel Reymont]

Alain,

On Jun 15, 2007, at 5:03 PM, Alain Frisch wrote:

> * Dynamic loading of OCaml code in native code programs, through
>   the Dynlink module.

When will this make it into the main tree?

	Thanks, Joel

--
http://topdog.cc      - EasyLanguage to C# translator
http://wagerlabs.com  - Blog

RE: [Caml-list] Dynamic linking

[Alexander Bottema]

I think we may attach different meanings to "position independent code."
For me PIC includes PIC suitable for dynamic linking. (This is what the
-fPIC option tells GCC to do.)

-- Alexander

-----Original Message-----
From: skaller [mailto:skaller@users.sourceforge.net] 
Sent: Wednesday, October 05, 2005 4:15 AM
To: Alexander Bottema
Cc: caml-list@yquem.inria.fr
Subject: RE: [Caml-list] Dynamic linking

On Tue, 2005-10-04 at 11:56 -0400, Alexander Bottema wrote:
> But the problem with a DLL is that you never now what address will be
> allocated for your executable. It can be located anywhere between 0
and
> 2^64-1. 

Not on current versions of Linux/x86_64. They only support 48 bit
code addresses (I think it is 48 bits), this is the so-called
'small model'.

> Thus, when a DLL calls another DLL a 64-bit call is potentially
> required. AMD64 supports global address tables that enable you to
> translate a 32-bit call into a 64-bit one (likewise with data access).
> If you compile a C file with gcc (-fPIC -S) you'll get code that looks
> like this:

I know ..but it has NOTHING to do with position independent code.

> For OCaml to work you need to emit instructions of type 'call foo@PLT'
> and 'movl xyzzy@GOT(%ebx)'. Currently, the OCaml does not do this for
> AMD64, which is the heart of the problem. If you think this is trivial
> to fix, please go head and do it; I'd be very happy.

This has nothing to do with making the *code* position independent.
It is to do with supporting elf/ld.so dynamic loading. This is
another issue entirely. 

Clearly, to access a symbol of an 'unknown' address,
you put the address into a known table slot instead,
when the library is loaded, and access indirectly.

In any case, this doesn't solve the problem, but it does
identify it correctly: the problem is to emit instructions
to bind to load-time linked libraries on Linux/Elf/ld.so 
platforms. You can be sure Windows -- for the same processor --
uses a different dynamic linkage model .. :)


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

RE: [Caml-list] Dynamic linking

[skaller]

On Tue, 2005-10-04 at 08:47 -0400, Alexander Bottema wrote:

> Now, as a developer it would be nice if it was possible to write certain
> parts in OCaml. The foreign language interface is good enough to be used
> in conjunction with C/C++, but the lack of support to use it in DLLs
> makes it impossible for us to have any OCaml code in our leaf product.
> 
> Nevertheless, I really tried to make it work (and I almost did). It's
> only Linux/AMD64 that I couldn't solve and I really tried to modify the
> AMD64 emitter.

I actually find this very strange. The x86 is incapable of
supporting position independent code without hackery or 
segmentation:

(a) there is no IP relative call instruction (only short branches)
(b) there is no IP relative data access (at all)

The amd64 (x86_64) on the other hand supports IP relative
calls and data access (with 32 bit offsets only, but
usually enough for all code and global data).


> > The only platform which I couldn't get to work was Linux AMD64. I
> tried
> > to modify the OCaml emitter for AMD64 to accommodate position
> > independent code using the offset tables, but it was too hard.

There is no need for offset tables. Just make sure all
'absolute' addresses accesses are changed to be RIP relative
and the assembler will do the rest automatically.

[There may be a problem on Windows, apparently the linker
can't add two offsets together due to a constraint in COFF
format, not sure about that though]

See section 2.2.5 of the AMD document 24592.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

RE: [Caml-list] Dynamic linking

[Alexander Bottema]

As a developer of the MathWorks we are shipping a product containing
C/C++ code, Java code, Fortran, etc. The infrastructure that drives
everything is written in C++ and we're talking about a code base that
has been developed for more than 20 years. Every module that is based on
top of the core is written as a DLL and that's the way everything is
organized. Changing this infrastructure would be a task that would not
be reasonable by any means. It would cost the company a tremendous
amount of money and there would be no clear benefits from it. Actually,
I like the way it is being organized where every "leaf" product is just
a DLL.

Now, as a developer it would be nice if it was possible to write certain
parts in OCaml. The foreign language interface is good enough to be used
in conjunction with C/C++, but the lack of support to use it in DLLs
makes it impossible for us to have any OCaml code in our leaf product.

Nevertheless, I really tried to make it work (and I almost did). It's
only Linux/AMD64 that I couldn't solve and I really tried to modify the
AMD64 emitter.

-- Alexander Bottema
   The MathWorks

-----Original Message-----
From: Jon Harrop [mailto:jon@ffconsultancy.com] 
Sent: Tuesday, October 04, 2005 6:57 AM
To: Alexander Bottema
Subject: Re: [Caml-list] Dynamic linking

On Wednesday 28 September 2005 13:44, you wrote:
> I managed to get dynamic linking for OCaml (i.e. making OCaml
producing
> DLLs that could then be used by other native code/binaries) to work on
> almost all our supported platforms (Linux x86, Windows (2000/XP),
> Solaris 2, Mac OS X).
> The only platform which I couldn't get to work was Linux AMD64. I
tried
> to modify the OCaml emitter for AMD64 to accommodate position
> independent code using the offset tables, but it was too hard. If
anyone
> picks up this task, please let us know. As it stands today we cannot
use
> OCaml for production code.

May I ask what other approaches you have tried? In particular, why not 
statically link in the OCaml code?

I'd have thought it would then be a case of either calling everything
else 
from OCaml, or initialising the OCaml runtime and calling OCaml from 
"everything else".

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists

RE: [Caml-list] Dynamic linking

[Alexander Bottema]

I managed to get dynamic linking for OCaml (i.e. making OCaml producing
DLLs that could then be used by other native code/binaries) to work on
almost all our supported platforms (Linux x86, Windows (2000/XP),
Solaris 2, Mac OS X).
The only platform which I couldn't get to work was Linux AMD64. I tried
to modify the OCaml emitter for AMD64 to accommodate position
independent code using the offset tables, but it was too hard. If anyone
picks up this task, please let us know. As it stands today we cannot use
OCaml for production code.

Alexander Bottema
The MathWorks

-----Original Message-----
From: caml-list-bounces@yquem.inria.fr
[mailto:caml-list-bounces@yquem.inria.fr] On Behalf Of Jacques Garrigue
Sent: Wednesday, September 28, 2005 5:06 AM
To: jonathan.roewen@gmail.com
Cc: caml-list@yquem.inria.fr
Subject: Re: [Caml-list] Dynamic linking

Follow-up on dynamic linking for native code:

I believed this didn't exist for ocaml, but I just learned that
MetaOCaml does it, at least on x86.
So this is not only possible, but an implementation is available.
(I understand this is partly work in progress.)

Jacques Garrigue

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Dynamic linking

[Jacques Garrigue]

Follow-up on dynamic linking for native code:

I believed this didn't exist for ocaml, but I just learned that
MetaOCaml does it, at least on x86.
So this is not only possible, but an implementation is available.
(I understand this is partly work in progress.)

Jacques Garrigue

Re: [Caml-list] Dynamic linking

[Jonathan Roewen]

> No document that I know, but the code is clear enough.
> You may have a look at Env.read_pers_struct for .cmi and
> Compilenv.read_unit_info for .cmx.
> However I'm not sure it would help you very much in practice.
> There are at least two other problems before you can do dynamic linking
> of native code in ocaml:
> * the code generator is geared toward static code, which means that it
>  cannot be handled by the runtime linker on many architectures
>  (I only know of a patch for x86)
> * some glue code is generated at link time
> So this looks like a rather difficult task...

Well, we'd be building our very own custom runtime linker for starters.

The idea goes something like:

- a single [kernel mode] address space everything is run in.
- somehow, necessary kernel symbols are exported (basically, what can
be seen in interface files)
- a runtime linker in the ocaml kernel loads ocaml code, and resolves
symbols to kernel functions so can call into the kernel directly (no
syscalls or IPC)
- probably have some way of having a defined entry point for
executables in ocaml code that's also exported, so kernel+linker can
run the code without needing some C layer.

Of course, the only problem here is: is it possible to do? ;-) I have
no idea myself... =/

Jonathan

Re: [Caml-list] Dynamic linking

[Jacques Garrigue]

From: Jonathan Roewen <jonathan.roewen@gmail.com>

> We're trying to figure out how to provide some sort of dynamic linker
> for DST to load ocaml native code at runtime on an ocaml native code
> kernel.
> 
> Is there any document that details the format of .cmi/.cmx files;
> also, are these (esp .cmi) able to provide the necessary details on
> the symbols in the kernel for building a dynamic linker that can
> resolve symbols to kernel functions in ocaml code loaded at runtime?

No document that I know, but the code is clear enough.
You may have a look at Env.read_pers_struct for .cmi and
Compilenv.read_unit_info for .cmx.
However I'm not sure it would help you very much in practice.
There are at least two other problems before you can do dynamic linking
of native code in ocaml:
* the code generator is geared toward static code, which means that it
  cannot be handled by the runtime linker on many architectures
  (I only know of a patch for x86)
* some glue code is generated at link time
So this looks like a rather difficult task...

> I know Dynlink module doesn't work with native code, and if I'm
> correct, the unsupported Asmdynlink module provided a runtime
> environment for bytecode ocaml code in native code programs right?
> Which is not what we'd be looking for.

Yet another approach would be to use bytecode + JIT.
But this won't give you the performance of native code.

    Jacques

Re: [Caml-list] Dynamic linking

[skaller]

On Mon, 2005-09-26 at 15:09 +1200, Jonathan Roewen wrote:
> Hi,
> 
> We're trying to figure out how to provide some sort of dynamic linker
> for DST to load ocaml native code at runtime on an ocaml native code
> kernel.
> 
> Is there any document that details the format of .cmi/.cmx files;
> also, are these (esp .cmi) able to provide the necessary details on
> the symbols in the kernel for building a dynamic linker that can
> resolve symbols to kernel functions in ocaml code loaded at runtime?
> 
> I know Dynlink module doesn't work with native code, and if I'm
> correct, the unsupported Asmdynlink module provided a runtime
> environment for bytecode ocaml code in native code programs right?
> Which is not what we'd be looking for.
> 
> Sorry if my questions come across a bit weird: it's not an easy
> subject for me to think about ;-)

If you find out let me know .. since I've spent over 5 years
designing a new programming language primarily to solve this problem.
Had ocaml supported generating shared libraries (dlls) and 
dynamic loading of them, at the native code level, I would
have used Ocaml.

The bottom line is that basically it can 'just work', since
native code is just native code, all you really need is
that the library code link to the right symbols to
provide access to the run time (and of course you need
to initialise it).


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

[Caml-list] Dynamic linking

[Jonathan Roewen]

Hi,

We're trying to figure out how to provide some sort of dynamic linker
for DST to load ocaml native code at runtime on an ocaml native code
kernel.

Is there any document that details the format of .cmi/.cmx files;
also, are these (esp .cmi) able to provide the necessary details on
the symbols in the kernel for building a dynamic linker that can
resolve symbols to kernel functions in ocaml code loaded at runtime?

I know Dynlink module doesn't work with native code, and if I'm
correct, the unsupported Asmdynlink module provided a runtime
environment for bytecode ocaml code in native code programs right?
Which is not what we'd be looking for.

Sorry if my questions come across a bit weird: it's not an easy
subject for me to think about ;-)

Kindest Regards,

Jonathan

Re: [Caml-list] Dynamic linking

[Igor Pechtchanski]

On Tue, 5 Apr 2005, Emir Pasalic wrote:

> We are writing a program that generates a C file, compiles it to a
> dynamic library, and uses dlopen (and such) to load it, execute it and
> bring its value into ocaml (bytecode) runtime. To do this, we need to
> use some of the functionality of the ocaml runtime (e.g., caml_alloc,
> caml_update) so we can marshall values from the C world into the world
> of ocaml. Our solution works on linux and macos platforms, but we have a
> problem trying to make it run on windows with Cygwin.
>
> So, we're trying to create a shared library on Cygwin that contains
> symbols such as "caml_alloc" and "caml_update".
> We do not know of a way to easily incorporate these symbols in the
> linking process, and so they remain undefined when we try to create a
> library, and undefined symbols are not allowed in Cygwin shared
> libraries.

The curent version of O'Caml under Cygwin doesn't support dynamic linking
in any structured way.  I was able to build an ad-hoc set of dynamic
libraries for standard modules, but I'm still in the process of modifying
O'Caml tools to do this seamlessly.

That said, there is still a limitation in Windows that any unresolved
symbols in a DLL have to have a *statically* known target, i.e., the
loader has to know what DLL to load the symbols from.  The two possible
workarounds are to a) extract the unresolved symbols from the executable
into a helper DLL that both the executable and the library are linked
with, or b) use dlopen/dlsym, as you did.

> Therefore we tried to resort to another method, where the calls to
> caml_alloc and caml_update are replaced by calls to dlopen and dlsym
> functions, i.e., we were trying to do this:
>
>        h = dlopen ("<the library name>", RLTD_NOW);
>        /* process error */
>        s = dlsym (h, "caml_alloc");
>        /* process error */
>        my_alloc = /* proper casting */ s;
>        result = my_alloc ( /* arguments */ );
>
> Assuming that this is possible, what is the name that should be given to
> the library?

Any name will do, as long as LD_LIBRARY_PATH contains the directory of
your library (yes, it *is* used on Cygwin for dlopen calls).  It doesn't
even have to end in ".dll".

> Else, is it possible to build a shared library on Cygwin that contains
> references to these symbols?

It is.  You'll need to create a helper DLL and an import library for it.
Then link your executable and the library DLL with the helper.  I would
have referred you to the (experimental) ocaml-3.08.1-2 cygwin package, but
it apparently wasn't uploaded to the main Cygwin package repository.  I
can send you the source/patches if interested.

> Note that all this works perfectly fine on MacOS and linux which allow
> unresolved symbols in dynamic libraries, but Cygwin simply dies. Any
> Windows/Cygwin experts out there who can help us?

I would be willing to help you build a small example app to get you
started.  Let me know where to get the source.

Igor Pechtchanski, the volunteer O'Caml maintainer for Cygwin
-- 
				http://cs.nyu.edu/~pechtcha/
      |\      _,,,---,,_		pechtcha@cs.nyu.edu
ZZZzz /,`.-'`'    -.  ;-;;,_		igor@watson.ibm.com
     |,4-  ) )-,_. ,\ (  `'-'		Igor Pechtchanski, Ph.D.
    '---''(_/--'  `-'\_) fL	a.k.a JaguaR-R-R-r-r-r-.-.-.  Meow!

"The Sun will pass between the Earth and the Moon tonight for a total
Lunar eclipse..." -- WCBS Radio Newsbrief, Oct 27 2004, 12:01 pm EDT