Dynamic linking

Dynamic linking

[Alain Frisch]

Hello,

I'm pleased to announce the existence of an experimental branch in
OCaml's CVS. Main changes:

* Dynamic loading of OCaml code in native code programs, through
  the Dynlink module.

* Simplified dynamic linking of C code under Windows.


More info: http://alain.frisch.fr/natdynlink.html

Enjoy and send some feedback!

Alain

Re: [Caml-list] Dynamic linking

[Joel Reymont]

Alain,

On Jun 15, 2007, at 5:03 PM, Alain Frisch wrote:

> * Dynamic loading of OCaml code in native code programs, through
>   the Dynlink module.

When will this make it into the main tree?

	Thanks, Joel

--
http://topdog.cc      - EasyLanguage to C# translator
http://wagerlabs.com  - Blog

Re: [Caml-list] Dynamic linking

[Brian Hurt]

Alain Frisch wrote:

>Hello,
>
>I'm pleased to announce the existence of an experimental branch in
>OCaml's CVS. Main changes:
>
>* Dynamic loading of OCaml code in native code programs, through
>  the Dynlink module.
>
>* Simplified dynamic linking of C code under Windows.
>
>
>More info: http://alain.frisch.fr/natdynlink.html
>
>Enjoy and send some feedback!
>
>Alain
>
>  
>

A) Thanks, this is usefull.

B) How extensive were the changes to the Ocaml source code?  How likely 
is it that this will get included in some future version of the main trunk?

Brian

Re: [Caml-list] Dynamic linking

[Alain Frisch]

Brian Hurt wrote:
> B) How extensive were the changes to the Ocaml source code?

For Linux x86, the change is rather small and straightforward.

For Windows ports, it is more tricky, because a Windows DLL cannot
reference symbols from the main program. I wrote the FlexDLL tool to
simulate a POSIX-ish dlopen API, so that the patch to OCaml itself
mostly reduces to removing code or sharing Makefiles with Unix. All the
trouble is encapsulated into FlexDLL (which is itself a quite small
piece of code, but a tricky one).

>From Linux AMD64, the ABI/dynamic linker does not allow to dynamically
load position dependant code, so I had to modify the last stage of
ocamlopt's backend in order to produce PIC code (when the option -dlcode
is given; note that this is independant from the existing -fpic option,
details on demand). Again, this is a rather small and unintrusive change.

You can use CVS to get a diff between the ocaml3100 tag and the
natdynlink branch (but it won't show you new files; you can always "cvs
checkout" both and do a local diff).

> How likely
> is it that this will get included in some future version of the main trunk?

It is likely to be included in OCaml 3.11, not before. (Except for the
native toplevel which will probably not be included.) Of course,
feedback can only help for the integration.

-- Alain

Re: [Caml-list] Dynamic linking

[Jon Harrop]

This is incredible. Thank you so much!

On Friday 15 June 2007 19:24:20 Alain Frisch wrote:
> ... (Except for the native toplevel which will probably not be included.)

Is that "native" as in we're going to get an OCaml top-level with native-code 
performance? If so, this is hugely important for me...

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
OCaml for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists/?e

Re: [Caml-list] Dynamic linking

[Alain Frisch]

Jon Harrop wrote:
> On Friday 15 June 2007 19:24:20 Alain Frisch wrote:
>> ... (Except for the native toplevel which will probably not be included.)
> 
> Is that "native" as in we're going to get an OCaml top-level with native-code 
> performance? If so, this is hugely important for me...

What kind of application do you have in mind?

It's "native" as in "we have an OCaml top-level with native-code
performance and a large latency". Each phrase is compiled with
ocamlopt's backend to an assembler file, which is then assembled, linked
into a shared library, dl-opened and then run. (Shared libraries will
never be closed so your OS must be strong enough to support that.)
Performances will likely be equal to ocamlopt's generated code (in
particular, we have cross-phrase optimizations).

Feel free to play with it ("make ocamlnat") and tell us whether this is
adequate for what you want to do.

  Alain

Re: [Caml-list] Dynamic linking

[skaller]

On Fri, 2007-06-15 at 18:03 +0200, Alain Frisch wrote:
> Hello,
> 
> I'm pleased to announce the existence of an experimental branch in
> OCaml's CVS. Main changes:
> 
> * Dynamic loading of OCaml code in native code programs, through
>   the Dynlink module.
> 
> * Simplified dynamic linking of C code under Windows.
> 
> 
> More info: http://alain.frisch.fr/natdynlink.html
> 
> Enjoy and send some feedback!

A question, probably not directed at Alain: will there be support
for AMD64/64 bit Ocaml for Windows XP64/VC2005 +/- Vista?
Windows ports are all 32 bit at the moment aren't they?

Also a question: how does the Windows native (MSVC based) Ocaml
code find C libraries? The VC2003 style DLL-hell approach is now
deprecated on Windows platforms: you have to use Assemblies,
that is, ship applications with manifests and related components
which are installed in system caches to get around DLL-hell.

The resulting system is horrible to prepare for vendors I think,
I haven't figured it out, but it does support multi-version and
multi-arch, and seems more sophisticated the Linux versioning
technology.

The bottom line, however, is you simply cannot link to a DLL
by just plonking it in the PATH anymore.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Dynamic linking

[Quôc Peyrot]

On Jun 15, 2007, at 8:24 PM, Alain Frisch wrote:

> Brian Hurt wrote:
>> B) How extensive were the changes to the Ocaml source code?
>
> For Linux x86, the change is rather small and straightforward.

Is it working (or going to work) on PPC (Mac OS X)?

-- 
Best Regards,
Quôc

Re: [Caml-list] Dynamic linking

[Christophe TROESTLER]

On Sat, 16 Jun 2007, skaller <skaller@users.sourceforge.net> wrote:
> 
> A question, probably not directed at Alain: will there be support
> for AMD64/64 bit Ocaml for Windows XP64/VC2005 +/- Vista?
> Windows ports are all 32 bit at the moment aren't they?

Extract from README.win32 :

                                      Native MS     Native MinGW        Cygwin

64 bits?                            Win32 or Win64    Win32 only      Win32 only

Third-party software required
  - for base bytecode system            none            none            none
  - for ocamlc -custom                  MSVC            Cygwin          Cygwin
  - for native-code generation          MSVC+MASM       Cygwin          Cygwin

Speed of bytecode interpreter           70%             100%            100%

Replay debugger                         no              no              yes

The Unix library                        partial         partial         full

The Threads library                     yes             yes             yes

The Graphics library                    yes             yes             no

Restrictions on generated executables?  none            none            yes (*)


ChriS

Re: [Caml-list] Dynamic linking

[skaller]

On Sat, 2007-06-16 at 00:45 +0200, Christophe TROESTLER wrote:
> On Sat, 16 Jun 2007, skaller <skaller@users.sourceforge.net> wrote:
> > 
> > A question, probably not directed at Alain: will there be support
> > for AMD64/64 bit Ocaml for Windows XP64/VC2005 +/- Vista?
> > Windows ports are all 32 bit at the moment aren't they?
> 
> Extract from README.win32 :
> 
>                                       Native MS     Native MinGW        Cygwin
> 
> 64 bits?                            Win32 or Win64    Win32 only      Win32 only

Ooops, OK! Sweet!

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Dynamic linking

[Alain Frisch]

Quôc Peyrot wrote:
>> For Linux x86, the change is rather small and straightforward.
> 
> Is it working (or going to work) on PPC (Mac OS X)?

It is not currently working. The code generator must be adapted to
produce PIC code. I've started to do that but it doesn't work and I
won't be able to finish it. If someone is interested to help, please
tell me.

-- Alain

Re: [Caml-list] Dynamic linking

[Jon Harrop]

On Friday 15 June 2007 19:24:20 Alain Frisch wrote:
> Except for the native toplevel which will probably not be included.

The native top-level is the bit that interests me the most, as this combines 
interactivity with OCaml's superb native-code performance.

I just benchmarked the new top-level and it runs my ray tracer 17x faster:

$ time ocaml unix.cma ray.ml >image.pgm

real    1m11.656s
user    2m21.106s
sys     0m0.331s

$ time ocamlnat unix.cmxa ray.ml >image.pgm

real    0m4.294s
user    0m6.265s
sys     0m0.521s

Interactive compilation to native code was one of the main benefits of F# over 
OCaml. So I, for one, would dearly love to see this in the mainstream OCaml!

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
OCaml for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists/?e

Re: [Caml-list] Dynamic linking

[skaller]

On Sat, 2007-06-16 at 18:33 +0100, Jon Harrop wrote:
> On Friday 15 June 2007 19:24:20 Alain Frisch wrote:
> > Except for the native toplevel which will probably not be included.
> 
> The native top-level is the bit that interests me the most, as this combines 
> interactivity with OCaml's superb native-code performance.

I do not understand how the toplevel is even remotely useful:
it isn't "interactive" in any real sense of the word.

Natively you can't even edit lines, it's very lame.
Even with ledit, what you can do is very limited.

I fail to see how this is any better than preparing
a file and compiling it -- that way you get a proper
editor. If you're using emacs or some other souped up
editor it works much like an interactive IDE anyhow,
only better..

So exactly what is the advantage of the top level?

Yes, I use it occasionally for testing 1-2 line
fragments, mainly the typing, but i just can't
envisage how a ray tracer could be use productively
in such an environment, as compared to just making
a suitable file and compiling and running it.

It IS a bit of a pain, setting up the build environment
to do that, but I can't imagine it's that hard to automate
with something like emacs, or one of the make systems.


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Dynamic linking

[Christophe TROESTLER]

On Sun, 17 Jun 2007, skaller <skaller@users.sourceforge.net> wrote:
> 
> So exactly what is the advantage of the top level?
> 
> Yes, I use it occasionally for testing 1-2 line
> fragments, mainly the typing, but i just can't
> envisage how a ray tracer could be use productively
> in such an environment, as compared to just making
> a suitable file and compiling and running it.

I generally also type code in a file and use C-x C-e if I need to get
an immediate feedback.  However, native performance can be useful in
order (say) to perform interactive data analysis and use pure ocaml
libraries (as opposed to C wrappers).  Data analysis requires some
kind of fast feeback because you generally only discover what you want
to do during the analysis itself...

ChriS

Re: [Caml-list] Dynamic linking

[Philippe Wang]

skaller wrote:
> On Sat, 2007-06-16 at 18:33 +0100, Jon Harrop wrote:
>   
>> On Friday 15 June 2007 19:24:20 Alain Frisch wrote:
>>     
>>> Except for the native toplevel which will probably not be included.
>>>       
>> The native top-level is the bit that interests me the most, as this combines 
>> interactivity with OCaml's superb native-code performance.
>>     
>
> I do not understand how the toplevel is even remotely useful:
> it isn't "interactive" in any real sense of the word.
>
> [...]

A native toplevel can be (very) useful when using ocaml as a scripting 
language.

I like to write (web) scripts with ocaml when I don't want to spend a 
lot of time on searching for trivial security holes (i.e. code 
injection) with languages such as bash or perl.
And compiling a file and using the binaries is a risk of loosing the 
source...
I know, this may sound weird...

--
Philippe Wang

Re: [Caml-list] Dynamic linking

[Alain Frisch]

Hello again,

It is now possible to try the natdynlink branch with GODI:

- check that you are on section 3.10 (GODI_SECTION = 3.10 in your
godi.conf);

- "update the list of available packages"

- in the configuration screen for godi-ocaml-src, set these variables:
  OCAML_CVS_CHECKOUT = yes
  OCAML_CVS_REVISION = natdynlink

- ask GODI to rebuild the two packages godi-ocaml-src and godi-ocaml.


  Alain

Re: [Caml-list] Dynamic linking

[Jon Harrop]

On Friday 15 June 2007 17:03:33 Alain Frisch wrote:
> Enjoy and send some feedback!

I love the native-code top-level!

Any chance you could get MetaOCaml running on AMD64?

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
The OCaml Journal
http://www.ffconsultancy.com/products/ocaml_journal/?e

Dynamic linking

[Emir Pasalic]

We are writing a program that generates a C file, compiles it to a 
dynamic library, and uses dlopen (and such) to load it, execute it and 
bring its value into ocaml (bytecode) runtime. To do this, we need to 
use some of the functionality of the ocaml runtime (e.g., caml_alloc, 
caml_update) so we can marshall values from the C world into the world 
of ocaml. Our solution works on linux and macos platforms, but we have 
a problem trying to make it run on windows with Cygwin.

So, we're trying to create a shared library on Cygwin that contains 
symbols such as "caml_alloc" and "caml_update".
  We do not know of a way to easily incorporate these symbols in the 
linking process, and so they
remain undefined when we try to create a library, and undefined symbols 
are not allowed in Cygwin shared libraries.

Therefore we tried to resort to another method, where the calls to  
caml_alloc and caml_update are replaced by
calls to dlopen and dlsym functions, i.e., we were trying to do this:

        h = dlopen ("<the library name>", RLTD_NOW);
        /* process error */
        s = dlsym (h, "caml_alloc");
        /* process error */
        my_alloc = /* proper casting */ s;
        result = my_alloc ( /* arguments */ );

Assuming that this is possible, what is the name that should be given 
to the library?
Else, is it possible to build a shared library on Cygwin that contains 
references to these symbols?

Note that all this works perfectly fine on MacOS and linux which allow 
unresolved symbols in dynamic libraries, but Cygwin simply dies. Any 
Windows/Cygwin experts out there who can help us?