[Caml-list] Ocaml shared libraries

[Caml-list] Ocaml shared libraries

[Eric Stokes]

Hello All,
	As the director of a shop who is using Ocaml to do real work (yes I 
know, research is more important :P),
I would really like to be able to build a shared library out of code 
that I have written in Ocaml,
and link other Ocaml programs to it. There are practical reasons for 
wanting to do this, I write and maintain
some rather large systems written in Ocaml. Currently, whenever I 
update a library (not changing its interface),
I need to recompile and reinstall the entire system. These problems I 
can live with for now.
	But... I also have "delusions" (or so I'm told). IMHO, Ocaml is fast 
enough, and has enough good libraries
in existence to create a climate where Ocaml software will slowly start 
replacing software written in C.
I've been watching the building blocks go into place for six months 
now, and things look like they
are accelerating. As the number of libraries, and applications using 
them increases, static linking starts
to become a nightmare very quickly. If you were, for example, to build 
a desktop environment in Ocaml,
you'd have to rebuild the entire system every time you changed a core 
library. And, all the little executables
in the desktop would be HUGE. I think that Ocaml is a very good 
language (understatement) for building
large reliable systems, and I would hate to see its growth be hampered 
artificially by its lack of shared
libraries.


Eric Stokes
Middleware Technical Lead
California State University, Northridge

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml shared libraries

[skaller]

On Mon, 2004-05-17 at 16:09, Eric Stokes wrote:
> Hello All,
> 	As the director of a shop who is using Ocaml to do real work (yes I 
> know, research is more important :P),
> I would really like to be able to build a shared library out of code 
> that I have written in Ocaml,
> and link other Ocaml programs to it. 

You and me both. The Ocaml team is aware of this desire.
At least one obstacle appears to be that the x86 backend
does not generate relocatable code. I'm curious if the
'C' backend could be used for this purpose though??

> There are practical reasons for 
> wanting to do this, I write and maintain
> some rather large systems written in Ocaml. Currently, whenever I 
> update a library (not changing its interface),
> I need to recompile and reinstall the entire system. These problems I 
> can live with for now.

I cannot. My system has a fundamental requirement for self-extension
and high performance. Extensibility is possible with bytecode
but not native code.

> 	But... I also have "delusions" (or so I'm told). IMHO, Ocaml is fast 
> enough, and has enough good libraries
> in existence to create a climate where Ocaml software will slowly start 
> replacing software written in C.

	Yeah, you're deluded if you think performance and
quality have much to do with this .. just think about Java ..

>  I think that Ocaml is a very good 
> language (understatement) for building
> large reliable systems, and I would hate to see its growth be hampered 
> artificially by its lack of shared
> libraries.

	See Felix. I started off *mandating* the target
as a shared library. Static linkage was only added recently.
To a large extent this whole project is inspired by the inability
of Ocaml to build shared libraries and interface easily to C.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml shared libraries

[Xavier Leroy]

> 	As the director of a shop who is using Ocaml to do real work
> (yes I know, research is more important :P), I would really like to
> be able to build a shared library out of code that I have written in
> Ocaml, and link other Ocaml programs to it. [...]

I'm not sure which aspect of shared libraries you're interested in.
Unix shared libraries or Windows DLL offer the following features:
1- Smaller executables through code sharing.
2- Upgradeability: upgrade the shared library and (with luck) all
   executables that refer to it are automatically upgraded.
3- Dynamic loading of code in a running program.

There is however one big difference between C and OCaml, which is that
OCaml has type-safe linking.  The linker checks that clients of a
library were typed and compiled against the interface and cross-module
optimization information of the library.  

As it is done today, this check is rather brittle: any change in the
library interface or optimization information cause it to fail and
require a recompile of the client modules.

For static linking, this is not too bad: a package management
framework such as GODI can automate the recompilation of clients when
the library changes.  More importantly, you need to bring things in
sync only when you're rebuilding your client: already compiled
programs continue to work, since they embark their own version of the
library.

With shared libraries, any update on the shared lib would immediately
invalidate all executables that use it.  This is the well-known "DLL
hell" problem, just exacerbated by the very strict consistency
checkings done by the OCaml linker.  So, feature 2- above is really
not applicable to OCaml as it is today, and static linking is much
more usable than dynamic linking of shared libs.

As for feature 1- (smaller executables), I'm not convinced this is a
major issue.  I'm old enough to remember the introduction of shared
libraries in the Unix world (in SunOS 4).  At that time, the saving in
disk space was significant: disks were small (a complete SunOS 4
install could fit in as little as 100 Mb) and the size of executables
wasn't negligible compared to the size of data files.  Times have
changed, however: disk space has increased much faster than executable
sizes, and the disks on a modern machine are mostly filled with data
(think MP3 and movies :-), making executable sizes a non-issue.

Feature 3- (dynamic code loading) is already available in bytecode
through the Dynlink API.  I understand there's a demand for having it
in native-code as well, and that might be possible without too much fuss,
at least on selected operating systems.

So, in summary: shared libraries are simply too fragile, especially
when combined with OCaml's type-safe linking.  This is such a big
problem that the drawbacks of static linking (bigger executables)
appear very minor in comparison.

- Xavier Leroy

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml shared libraries

[Michal Moskal]

On Mon, May 17, 2004 at 10:22:54AM +0200, Xavier Leroy wrote:
> As for feature 1- (smaller executables), I'm not convinced this is a
> major issue.  I'm old enough to remember the introduction of shared
> libraries in the Unix world (in SunOS 4).  At that time, the saving in
> disk space was significant: disks were small (a complete SunOS 4
> install could fit in as little as 100 Mb) and the size of executables
> wasn't negligible compared to the size of data files.  Times have
> changed, however: disk space has increased much faster than executable
> sizes, and the disks on a modern machine are mostly filled with data
> (think MP3 and movies :-), making executable sizes a non-issue.

Disk is one thing. Memory is another. Under unix each shared library
resides in memory once.  Of course each executable is also loaded once,
so this is only an issue when there are multiple ocaml programs (different
executables) run simultaneously. Nevertheless this is sometimes the
case.

-- 
: Michal Moskal :: http://www.kernel.pl/~malekith :: GCS !tv h e>+++ b++
: When in doubt, use brute force. -- Ken Thompson :: UL++++$ C++ E--- a?

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml shared libraries

[John Goerzen]

On Mon, May 17, 2004 at 10:22:54AM +0200, Xavier Leroy wrote:
> With shared libraries, any update on the shared lib would immediately
> invalidate all executables that use it.  This is the well-known "DLL
> hell" problem, just exacerbated by the very strict consistency
> checkings done by the OCaml linker.  So, feature 2- above is really
> not applicable to OCaml as it is today, and static linking is much
> more usable than dynamic linking of shared libs.

I'm not quite sure that conclusion follows from your premise.

Let's say we have an OCaml executable and needs to link with certain
OCaml libraries.  Couldn't the dynamic loader embedded in that
executable, whatever form it may take, check with the libraries it's
dynamically linking with to make sure the interfaces remain the same?

> As for feature 1- (smaller executables), I'm not convinced this is a
> major issue.  I'm old enough to remember the introduction of shared
> libraries in the Unix world (in SunOS 4).  At that time, the saving in
> disk space was significant: disks were small (a complete SunOS 4
> install could fit in as little as 100 Mb) and the size of executables
> wasn't negligible compared to the size of data files.  Times have
> changed, however: disk space has increased much faster than executable
> sizes, and the disks on a modern machine are mostly filled with data
> (think MP3 and movies :-), making executable sizes a non-issue.

There are plenty of situations where this is not the case.  For
instance:

1. Handheld devices (think Sharp Zaurus, re-flashed iPAQ, or other
   units that run Linux)

2. Other embedded systems

3. Limited-storage systems (thin clients, etc)

4. Limited-RAM systems

A little more on #4... on *nix platforms, at least, the code of a .so is
usually shared among all processes that use it.

> Feature 3- (dynamic code loading) is already available in bytecode
> through the Dynlink API.  I understand there's a demand for having it
> in native-code as well, and that might be possible without too much fuss,
> at least on selected operating systems.

It exists, but it's not exactly to the point where you can use it to
load in any arbitrary library; specifically, you can only load libraries
that are specifically written to register their functions with your app
somehow.

-- John

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml shared libraries

[Eric Stokes]

Yes, I realized when I started this thread that all of these issues 
would arise. I know
it is a nasty problem to do type safe shared libraries. The feature I 
am most interested
in is #2, upgradeability, #3 is a close second. So building a less 
brittle but still type safe linker
would be necessary. My question really is, do you all have any interest 
in doing so. It
seems like it would be an interesting project with practical benefits.

On May 17, 2004, at 1:22 AM, Xavier Leroy wrote:

>> 	As the director of a shop who is using Ocaml to do real work
>> (yes I know, research is more important :P), I would really like to
>> be able to build a shared library out of code that I have written in
>> Ocaml, and link other Ocaml programs to it. [...]
>
> I'm not sure which aspect of shared libraries you're interested in.
> Unix shared libraries or Windows DLL offer the following features:
> 1- Smaller executables through code sharing.
> 2- Upgradeability: upgrade the shared library and (with luck) all
>    executables that refer to it are automatically upgraded.
> 3- Dynamic loading of code in a running program.
>
> There is however one big difference between C and OCaml, which is that
> OCaml has type-safe linking.  The linker checks that clients of a
> library were typed and compiled against the interface and cross-module
> optimization information of the library.
>
> As it is done today, this check is rather brittle: any change in the
> library interface or optimization information cause it to fail and
> require a recompile of the client modules.
>
> For static linking, this is not too bad: a package management
> framework such as GODI can automate the recompilation of clients when
> the library changes.  More importantly, you need to bring things in
> sync only when you're rebuilding your client: already compiled
> programs continue to work, since they embark their own version of the
> library.
>
> With shared libraries, any update on the shared lib would immediately
> invalidate all executables that use it.  This is the well-known "DLL
> hell" problem, just exacerbated by the very strict consistency
> checkings done by the OCaml linker.  So, feature 2- above is really
> not applicable to OCaml as it is today, and static linking is much
> more usable than dynamic linking of shared libs.
>
> As for feature 1- (smaller executables), I'm not convinced this is a
> major issue.  I'm old enough to remember the introduction of shared
> libraries in the Unix world (in SunOS 4).  At that time, the saving in
> disk space was significant: disks were small (a complete SunOS 4
> install could fit in as little as 100 Mb) and the size of executables
> wasn't negligible compared to the size of data files.  Times have
> changed, however: disk space has increased much faster than executable
> sizes, and the disks on a modern machine are mostly filled with data
> (think MP3 and movies :-), making executable sizes a non-issue.
>
> Feature 3- (dynamic code loading) is already available in bytecode
> through the Dynlink API.  I understand there's a demand for having it
> in native-code as well, and that might be possible without too much 
> fuss,
> at least on selected operating systems.
>
> So, in summary: shared libraries are simply too fragile, especially
> when combined with OCaml's type-safe linking.  This is such a big
> problem that the drawbacks of static linking (bigger executables)
> appear very minor in comparison.
>
> - Xavier Leroy
>
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: 
> http://caml.inria.fr
> Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: 
> http://caml.inria.fr/FAQ/
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Ocaml shared libraries

[Eric Stokes]

Hah, I scoured the list archives for weeks looking for just that 
message :P,
and I didn't find it. I guess my search skills are unl33t. Anyway, that 
answers
a lot of my questions. It seems the trade offs do not favor shared 
libraries at all.

On May 17, 2004, at 7:51 AM, Olivier Andrieu wrote:

>  Eric Stokes [Mon, 17 May 2004]:
>> Yes, I realized when I started this thread that all of these issues
>> would arise. I know it is a nasty problem to do type safe shared
>> libraries. The feature I am most interested in is #2,
>> upgradeability, #3 is a close second. So building a less brittle
>> but still type safe linker would be necessary. My question really
>> is, do you all have any interest in doing so. It seems like it
>> would be an interesting project with practical benefits.
>
> Hi,
>
> these questions have already been raised on the list (it does not mean
> they're not worthy of being mentionned again :), so you may be
> interested by this in-depth reply from Xavier:
>
>   http://tinyurl.com/2ywsu
>
> -- 
>    Olivier

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners