caml-list - the Caml user's mailing list
 help / color / mirror / Atom feed
* Re: [Caml-list] Dynamic linking
@ 2007-06-18  4:18 Jeffrey Loren Shaw
  0 siblings, 0 replies; 29+ messages in thread
From: Jeffrey Loren Shaw @ 2007-06-18  4:18 UTC (permalink / raw)
  To: lists; +Cc: caml-list

Philippe, here's a little program I wrote that you might find useful. 

compile: 

ocamlopt -o keepsource keepsource.ml 

usage: 

./keepsource foo.ml 

creates a new file foo_keepsource.ml. If you compile it to the executable 
foo you can do ./foo -source to get the original source back. 

./keepsource foo.ml -compile 

does the same thing bug gives you an executable named foo in addition to 
foo_keepsource.ml. 

It will only work on single-file programs, and I haven't tested it much. 

(* keepsource.ml *) 

let replace_quote str =
 let result = ref [] in
 for i = 0 to String.length str - 1 do
   match str.[i] with
     '\\' -> result := '\\'::'\\'::!result
   | '\"' -> result := '\"'::'\\'::!result
   | a -> result := a::!result
 done;
 let buffer = " " in
 List.fold_left
   (fun accum a ->
      buffer.[0] <- a;
      buffer ^ accum
   )
   ""
   !result;; 

let print_help () =
   print_endline
   "usage: keepsource infile.ml (-compile)";; 

if Array.length Sys.argv = 1 then print_help ();; 

let sourcefile = Sys.argv.(1);; 

let sourcefile_keepsource = String.sub sourcefile 0 (String.length 
sourcefile - 3) ^ "_keepsource.ml";; 

if Sys.file_exists sourcefile then
 let inchan = open_in sourcefile in
 let source = ref [] in
 (
   try
     while true do
	source := (input_line inchan)::!source
     done
   with
     End_of_file -> ()
 );
 close_in inchan;
 source := List.rev !source;
 let outchan = open_out sourcefile_keepsource in
 let output = output_string outchan in
 output "let sourcecode = \"";
 List.iter (fun s -> output (replace_quote s^"\n")) !source;
 output "\"\n\nlet main () = (";
 List.iter (fun s -> output (s^"\n")) !source;
 output ");; 

if Array.length Sys.argv > 1 then
 if Sys.argv.(1) = \"-source\" then
   (
     print_endline sourcecode;
     exit 0
   )
 else
   main ()
 else
   main ()
";
 close_out outchan;; 

if Array.length Sys.argv > 2 then
 if Sys.argv.(2) = "-compile" then
   let compile_command =
     "ocamlopt -o " ^ String.sub sourcefile 0 (String.length sourcefile - 3) 
^ " "^ sourcefile_keepsource in
   ignore
     (
	Sys.command compile_command
     ) 



^ permalink raw reply	[flat|nested] 29+ messages in thread
* Dynamic linking
@ 2007-06-15 16:03 Alain Frisch
  2007-06-15 16:47 ` [Caml-list] " Joel Reymont
                   ` (4 more replies)
  0 siblings, 5 replies; 29+ messages in thread
From: Alain Frisch @ 2007-06-15 16:03 UTC (permalink / raw)
  To: caml-list

Hello,

I'm pleased to announce the existence of an experimental branch in
OCaml's CVS. Main changes:

* Dynamic loading of OCaml code in native code programs, through
  the Dynlink module.

* Simplified dynamic linking of C code under Windows.


More info: http://alain.frisch.fr/natdynlink.html

Enjoy and send some feedback!

Alain


^ permalink raw reply	[flat|nested] 29+ messages in thread
* RE: [Caml-list] Dynamic linking
@ 2005-10-05 13:11 Alexander Bottema
  0 siblings, 0 replies; 29+ messages in thread
From: Alexander Bottema @ 2005-10-05 13:11 UTC (permalink / raw)
  To: caml-list

I think we may attach different meanings to "position independent code."
For me PIC includes PIC suitable for dynamic linking. (This is what the
-fPIC option tells GCC to do.)

-- Alexander

-----Original Message-----
From: skaller [mailto:skaller@users.sourceforge.net] 
Sent: Wednesday, October 05, 2005 4:15 AM
To: Alexander Bottema
Cc: caml-list@yquem.inria.fr
Subject: RE: [Caml-list] Dynamic linking

On Tue, 2005-10-04 at 11:56 -0400, Alexander Bottema wrote:
> But the problem with a DLL is that you never now what address will be
> allocated for your executable. It can be located anywhere between 0
and
> 2^64-1. 

Not on current versions of Linux/x86_64. They only support 48 bit
code addresses (I think it is 48 bits), this is the so-called
'small model'.

> Thus, when a DLL calls another DLL a 64-bit call is potentially
> required. AMD64 supports global address tables that enable you to
> translate a 32-bit call into a 64-bit one (likewise with data access).
> If you compile a C file with gcc (-fPIC -S) you'll get code that looks
> like this:

I know ..but it has NOTHING to do with position independent code.

> For OCaml to work you need to emit instructions of type 'call foo@PLT'
> and 'movl xyzzy@GOT(%ebx)'. Currently, the OCaml does not do this for
> AMD64, which is the heart of the problem. If you think this is trivial
> to fix, please go head and do it; I'd be very happy.

This has nothing to do with making the *code* position independent.
It is to do with supporting elf/ld.so dynamic loading. This is
another issue entirely. 

Clearly, to access a symbol of an 'unknown' address,
you put the address into a known table slot instead,
when the library is loaded, and access indirectly.

In any case, this doesn't solve the problem, but it does
identify it correctly: the problem is to emit instructions
to bind to load-time linked libraries on Linux/Elf/ld.so 
platforms. You can be sure Windows -- for the same processor --
uses a different dynamic linkage model .. :)


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net


^ permalink raw reply	[flat|nested] 29+ messages in thread
* RE: [Caml-list] Dynamic linking
@ 2005-10-04 15:56 Alexander Bottema
  2005-10-05  8:14 ` skaller
  0 siblings, 1 reply; 29+ messages in thread
From: Alexander Bottema @ 2005-10-04 15:56 UTC (permalink / raw)
  To: caml-list

But the problem with a DLL is that you never now what address will be
allocated for your executable. It can be located anywhere between 0 and
2^64-1. Thus, when a DLL calls another DLL a 64-bit call is potentially
required. AMD64 supports global address tables that enable you to
translate a 32-bit call into a 64-bit one (likewise with data access).
If you compile a C file with gcc (-fPIC -S) you'll get code that looks
like this:

The C file (amd64.c)
====================
extern int foo();

int xyzzy = 42;

int main(int argc, char *argv)
{
    xyzzy = 4711;
    return foo();
}

The assembly output (amd64.S) for gcc -O2 -fPIC -S
==================================================

	.file	"amd64.c"
.globl xyzzy
	.data
	.align 4
	.type	xyzzy,@object
	.size	xyzzy,4
xyzzy:
	.long	42
	.text
	.p2align 2,,3
.globl main
	.type	main,@function
main:
	pushl	%ebp
	movl	%esp, %ebp
	pushl	%ebx
	pushl	%eax
	call	.L2
.L2:
	popl	%ebx
	addl	$_GLOBAL_OFFSET_TABLE_+[.-.L2], %ebx
	movl	xyzzy@GOT(%ebx), %eax
	andl	$-16, %esp
	movl	$4711, (%eax)
	call	foo@PLT
	movl	-4(%ebp), %ebx
	leave
	ret
.Lfe1:
	.size	main,.Lfe1-main
	.ident	"GCC: (GNU) 3.2.3"

For OCaml to work you need to emit instructions of type 'call foo@PLT'
and 'movl xyzzy@GOT(%ebx)'. Currently, the OCaml does not do this for
AMD64, which is the heart of the problem. If you think this is trivial
to fix, please go head and do it; I'd be very happy.

Alexander Bottema
The MathWorks

-----Original Message-----
From: skaller [mailto:skaller@users.sourceforge.net] 
Sent: Tuesday, October 04, 2005 10:48 AM
To: Alexander Bottema
Cc: caml-list@yquem.inria.fr
Subject: RE: [Caml-list] Dynamic linking

On Tue, 2005-10-04 at 08:47 -0400, Alexander Bottema wrote:

> Now, as a developer it would be nice if it was possible to write
certain
> parts in OCaml. The foreign language interface is good enough to be
used
> in conjunction with C/C++, but the lack of support to use it in DLLs
> makes it impossible for us to have any OCaml code in our leaf product.
> 
> Nevertheless, I really tried to make it work (and I almost did). It's
> only Linux/AMD64 that I couldn't solve and I really tried to modify
the
> AMD64 emitter.

I actually find this very strange. The x86 is incapable of
supporting position independent code without hackery or 
segmentation:

(a) there is no IP relative call instruction (only short branches)
(b) there is no IP relative data access (at all)

The amd64 (x86_64) on the other hand supports IP relative
calls and data access (with 32 bit offsets only, but
usually enough for all code and global data).


> > The only platform which I couldn't get to work was Linux AMD64. I
> tried
> > to modify the OCaml emitter for AMD64 to accommodate position
> > independent code using the offset tables, but it was too hard.

There is no need for offset tables. Just make sure all
'absolute' addresses accesses are changed to be RIP relative
and the assembler will do the rest automatically.

[There may be a problem on Windows, apparently the linker
can't add two offsets together due to a constraint in COFF
format, not sure about that though]

See section 2.2.5 of the AMD document 24592.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net


^ permalink raw reply	[flat|nested] 29+ messages in thread
* RE: [Caml-list] Dynamic linking
@ 2005-10-04 12:47 Alexander Bottema
  2005-10-04 14:47 ` skaller
  0 siblings, 1 reply; 29+ messages in thread
From: Alexander Bottema @ 2005-10-04 12:47 UTC (permalink / raw)
  To: caml-list

As a developer of the MathWorks we are shipping a product containing
C/C++ code, Java code, Fortran, etc. The infrastructure that drives
everything is written in C++ and we're talking about a code base that
has been developed for more than 20 years. Every module that is based on
top of the core is written as a DLL and that's the way everything is
organized. Changing this infrastructure would be a task that would not
be reasonable by any means. It would cost the company a tremendous
amount of money and there would be no clear benefits from it. Actually,
I like the way it is being organized where every "leaf" product is just
a DLL.

Now, as a developer it would be nice if it was possible to write certain
parts in OCaml. The foreign language interface is good enough to be used
in conjunction with C/C++, but the lack of support to use it in DLLs
makes it impossible for us to have any OCaml code in our leaf product.

Nevertheless, I really tried to make it work (and I almost did). It's
only Linux/AMD64 that I couldn't solve and I really tried to modify the
AMD64 emitter.

-- Alexander Bottema
   The MathWorks

-----Original Message-----
From: Jon Harrop [mailto:jon@ffconsultancy.com] 
Sent: Tuesday, October 04, 2005 6:57 AM
To: Alexander Bottema
Subject: Re: [Caml-list] Dynamic linking

On Wednesday 28 September 2005 13:44, you wrote:
> I managed to get dynamic linking for OCaml (i.e. making OCaml
producing
> DLLs that could then be used by other native code/binaries) to work on
> almost all our supported platforms (Linux x86, Windows (2000/XP),
> Solaris 2, Mac OS X).
> The only platform which I couldn't get to work was Linux AMD64. I
tried
> to modify the OCaml emitter for AMD64 to accommodate position
> independent code using the offset tables, but it was too hard. If
anyone
> picks up this task, please let us know. As it stands today we cannot
use
> OCaml for production code.

May I ask what other approaches you have tried? In particular, why not 
statically link in the OCaml code?

I'd have thought it would then be a case of either calling everything
else 
from OCaml, or initialising the OCaml runtime and calling OCaml from 
"everything else".

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists


^ permalink raw reply	[flat|nested] 29+ messages in thread
* RE: [Caml-list] Dynamic linking
@ 2005-09-28 12:44 Alexander Bottema
  0 siblings, 0 replies; 29+ messages in thread
From: Alexander Bottema @ 2005-09-28 12:44 UTC (permalink / raw)
  To: caml-list

I managed to get dynamic linking for OCaml (i.e. making OCaml producing
DLLs that could then be used by other native code/binaries) to work on
almost all our supported platforms (Linux x86, Windows (2000/XP),
Solaris 2, Mac OS X).
The only platform which I couldn't get to work was Linux AMD64. I tried
to modify the OCaml emitter for AMD64 to accommodate position
independent code using the offset tables, but it was too hard. If anyone
picks up this task, please let us know. As it stands today we cannot use
OCaml for production code.

Alexander Bottema
The MathWorks

-----Original Message-----
From: caml-list-bounces@yquem.inria.fr
[mailto:caml-list-bounces@yquem.inria.fr] On Behalf Of Jacques Garrigue
Sent: Wednesday, September 28, 2005 5:06 AM
To: jonathan.roewen@gmail.com
Cc: caml-list@yquem.inria.fr
Subject: Re: [Caml-list] Dynamic linking

Follow-up on dynamic linking for native code:

I believed this didn't exist for ocaml, but I just learned that
MetaOCaml does it, at least on x86.
So this is not only possible, but an implementation is available.
(I understand this is partly work in progress.)

Jacques Garrigue

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs


^ permalink raw reply	[flat|nested] 29+ messages in thread
* [Caml-list] Dynamic linking
@ 2005-09-26  3:09 Jonathan Roewen
  2005-09-26  4:26 ` skaller
  2005-09-26  5:14 ` Jacques Garrigue
  0 siblings, 2 replies; 29+ messages in thread
From: Jonathan Roewen @ 2005-09-26  3:09 UTC (permalink / raw)
  To: caml-list

Hi,

We're trying to figure out how to provide some sort of dynamic linker
for DST to load ocaml native code at runtime on an ocaml native code
kernel.

Is there any document that details the format of .cmi/.cmx files;
also, are these (esp .cmi) able to provide the necessary details on
the symbols in the kernel for building a dynamic linker that can
resolve symbols to kernel functions in ocaml code loaded at runtime?

I know Dynlink module doesn't work with native code, and if I'm
correct, the unsupported Asmdynlink module provided a runtime
environment for bytecode ocaml code in native code programs right?
Which is not what we'd be looking for.

Sorry if my questions come across a bit weird: it's not an easy
subject for me to think about ;-)

Kindest Regards,

Jonathan


^ permalink raw reply	[flat|nested] 29+ messages in thread
* Dynamic linking
@ 2005-04-05 21:12 Emir Pasalic
  2005-04-05 22:02 ` [Caml-list] " Igor Pechtchanski
  0 siblings, 1 reply; 29+ messages in thread
From: Emir Pasalic @ 2005-04-05 21:12 UTC (permalink / raw)
  To: caml-list

We are writing a program that generates a C file, compiles it to a 
dynamic library, and uses dlopen (and such) to load it, execute it and 
bring its value into ocaml (bytecode) runtime. To do this, we need to 
use some of the functionality of the ocaml runtime (e.g., caml_alloc, 
caml_update) so we can marshall values from the C world into the world 
of ocaml. Our solution works on linux and macos platforms, but we have 
a problem trying to make it run on windows with Cygwin.

So, we're trying to create a shared library on Cygwin that contains 
symbols such as "caml_alloc" and "caml_update".
  We do not know of a way to easily incorporate these symbols in the 
linking process, and so they
remain undefined when we try to create a library, and undefined symbols 
are not allowed in Cygwin shared libraries.

Therefore we tried to resort to another method, where the calls to  
caml_alloc and caml_update are replaced by
calls to dlopen and dlsym functions, i.e., we were trying to do this:

        h = dlopen ("<the library name>", RLTD_NOW);
        /* process error */
        s = dlsym (h, "caml_alloc");
        /* process error */
        my_alloc = /* proper casting */ s;
        result = my_alloc ( /* arguments */ );

Assuming that this is possible, what is the name that should be given 
to the library?
Else, is it possible to build a shared library on Cygwin that contains 
references to these symbols?

Note that all this works perfectly fine on MacOS and linux which allow 
unresolved symbols in dynamic libraries, but Cygwin simply dies. Any 
Windows/Cygwin experts out there who can help us?


^ permalink raw reply	[flat|nested] 29+ messages in thread

end of thread, other threads:[~2007-06-24 11:41 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2007-06-18  4:18 [Caml-list] Dynamic linking Jeffrey Loren Shaw
  -- strict thread matches above, loose matches on Subject: below --
2007-06-15 16:03 Alain Frisch
2007-06-15 16:47 ` [Caml-list] " Joel Reymont
2007-06-15 16:52 ` Brian Hurt
2007-06-15 18:24   ` Alain Frisch
2007-06-15 18:59     ` Jon Harrop
2007-06-15 20:57       ` Alain Frisch
2007-06-15 22:38     ` Quôc Peyrot
2007-06-16  7:23       ` Alain Frisch
2007-06-16 17:33     ` Jon Harrop
2007-06-16 17:59       ` skaller
2007-06-16 20:13         ` Christophe TROESTLER
2007-06-16 23:10         ` Philippe Wang
2007-06-15 22:37 ` skaller
2007-06-15 22:45   ` Christophe TROESTLER
2007-06-15 22:58     ` skaller
2007-06-20 13:40 ` Alain Frisch
2007-06-24 11:35 ` Jon Harrop
2005-10-05 13:11 Alexander Bottema
2005-10-04 15:56 Alexander Bottema
2005-10-05  8:14 ` skaller
2005-10-04 12:47 Alexander Bottema
2005-10-04 14:47 ` skaller
2005-09-28 12:44 Alexander Bottema
2005-09-26  3:09 Jonathan Roewen
2005-09-26  4:26 ` skaller
2005-09-26  5:14 ` Jacques Garrigue
2005-09-26  5:20   ` Jonathan Roewen
2005-09-28  9:05   ` Jacques Garrigue
2005-04-05 21:12 Emir Pasalic
2005-04-05 22:02 ` [Caml-list] " Igor Pechtchanski

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).