Portability of applications written in OCAML

Portability of applications written in OCAML

[Claude Marche]

Bonjour,

[long message, english translation follows]

J'ai des remarques et des questions à propos de la façon dont on peut
distribuer une application écrite en Objective CAML, et sur la
portabilité à laquelle on peut s'attendre.

Voici mon problème : supposons que l'on ait programmé une application
en ocaml, que l'on a envie de distribuer (gratuitement) sur le
Web, comment doit-on procéder pour la distribuer ? Supposons que cette
application n'ait en elle-meme rien à voir avec CAML, et que l'on
souhaite atteindre un public qui a priori ne connait pas ce langage,
et ne dispose pas du compilateur ni de l'interpréteur.

Pour distribuer, on peut proposer a priori les sources, et aussi des
exécutables (compilés avec ocamlopt pour que ça aille
vite). Proposer les sources pose un problème car une personne
intéressée par cette application devra au préalable installer ocaml
sur son ordinateur, et cela rebuterait je pense une grande partie des
personnes intéressées au départ. Proposer des binaires contourne ce
problème mais on entre dans de multiples problemes de configuration,
car l'exécutable ne pourra fonctionner que sur la bonne architecture,
avec la bonne version du bon OS, sans oublier les problèmes de
versions de bibliothèques dynamiques (curses/ncurses, libc/glibc,
etc.) 

Le juste milieu me semble être d'utiliser le bytecode : Est-ce que
c'est vrai qu'il est possible d'executer n'importe quel bytecode sur
n'importe quel archi et OS à partir du moment où on a le ocamlrun
approprié pour cette architecture et cet OS ? (Meme sous des OS non
unix comme Microsoft-Windows ou Mac-OS ?) Si oui, un utilisateur
potentiel de l'application n'aurait pas besoin d'installer tout ocaml
mais seulement une version de ocamlrun appropriée pour sa combinaison
architecture/OS. Cela permettrait a cet utilisateur potentiel de
facilement essayer l'application qui l'interesse, et s'il la trouve
utile pour lui, il sera alors motivé pour faire l'effort d'installer
ocaml pour compiler une version en code natif de cette application
pour son archi et son OS.

Pour que cela fonctionne, il faudrait qu'il existe quelque part sur le
Web une sorte d'archive de versions de ocamlrun, permettant de choisir
son architecture, son OS, la version ocaml requise, peut-etre aussi
certaines variantes de bibliotheques comme libc 5 ou 6, glibc.

Je pense qu'il y a potentiellement un nombre non negligeable
d'applications concernées, il suffit de se promener un peu sur la «
bosse » : HeVeA, GeneWeb, Bibtex2html, et bien d'autres.

Il y a néanmoins un problème ennuyeux: dès qu'une application devient
un peu conséquente, il arrive que l'on commence à utiliser des
bibliothèques comme unix, str, nums, threads, qui nécessitent une
compilation en -custom. Si j'ai bien compris, les binaires obtenus
contiennent le code natif de l'interpréteur de bytecode lui-même, et
ne sont plus portables. Est-il techniquement possible de changer cela
en utilisant un ocamlrun linké avec ces bibliothèques ?

Pour résumer mon propos, voici donc les questions que je me pose :

1) Un fichier de bytecode est-il exécutable par ocamlrun sur n'importe
   quelle combinaison d'une architecture et d'un OS ? Y compris OS non
   unix ?

2) Est-il possible de fabriquer des versions d'ocamlrun contenant en
   plus des bibliothèques linkées habituellement avec -custom : unix,
   str, nums, threads ; et par conséquent pourraient exécuter des
   bytecode portables utilisant ces bibliotheques ?

Si ceci est techniquement faisable, cela m'intéresserait de connaitre
les personnes intéressées par l'utiliser, pour distribuer leur propre
application sous forme de bytecode portable. De plus, pour constituer
cette archive de binaires ocamlrun, il faudrait la participation du
plus grand nombre de personnes possible afin de couvrir largement les
combinaisons d'architectures et d'OS. Personnellement j'ai accès à des
PC sous Linux (Distribution Redhat en général) et des SPARC sous
Solaris 2.7, mais c'est tout.

Toutes remarques, toutes suggestions et tous commentaires sont les
bienvenus. Mon but est de pouvoir distribuer le plus largement
possible une application écrite en Ocaml, en évitant des remarques du
genre « cette application m'intéresse mais je ne peux pas compiler le
Ocaml et vous ne fournissez pas un binaire adapté à ma configuration
», et je serais heureux d'entendre toute suggestion permettant
d'atteindre ce but.

[English translation]

I have a few remarks and questions about how one may distribute an
application written in Objective CAML, and how much portability one
may expect.

Here is my problem: suppose one has written an application in ocaml,
and wants to distribute it (freely) on the Web, how should he proceed?
Suppose that the application itself has nothing to do with CAML, and
one wants to reach people that do not know a priori this langage, so
do not have caml installed on their computer.

Distribution could be achieve by making the source files available,
and also some binary executables (compiled with ocamlopt for
efficiency). Installing from source is not nice because anyone
interested must first install ocaml on his computer, and I think that
many people in that case would not do it, and will better try to find
another application doing the same thing but easier to install.
Distributing binaries is not nice also, because one may encounter many
configuration problems, the program will run only for the right
architecture, the right version of the right OS, not to mention
problems with versions of dynamic libraries (curses/ncurses,
libc/glibc, etc.)

One idea could be distributing bytecode: is it true the any bytecode
can be executed on any architecture and OS as soon as the right
ocamlrun is used? (Even under non unix OSs like Microsoft-Windows or
Mac-OS?) If yes, anyone interested in using the application would
need only to install the suitable ocamlrun for its architecture/OS
combination.  It would allow any potentially interested user to try
the application, and in case he finds it suitable for its need, he
would be motivated to install ocaml for compiling from the sources and
getting a faster application.

For this to work, a kind of archive of various versions of ocamlrun
should exist somewhere on the web, allowing to choose the right
architecture/OS, also the right ocaml version, the right libraries
like libc 5 or 6 or glibc.

I think there is a significant number of applications that could be
interested in being distributed like this, like the ones that are on
the ``hump'': HeVeA, GeneWeb, Bibtex2html, and many others.

Nevertheless, there is a problem for applications that use other
libraries like unix, str, nums, threads, that need to be compiled in
-custom mode. If I understand well, binaries compiled with -custom
contain the native code of the interpreter itself, so are not portable
anymore. Techically speaking, is it possible to build an ocamlrun
interpreter that contains those libraries statically linked?

So to conclude, I would like to know two things:

1) Can a bytecode file be run using any ocamlrun version, on any
   combination of architecture and OS? Even a non-unix OS?

2) Is it possible to build an ocamlrun interpreter containing also
   some librairies usually linked with -custom: unix, str, nums,
   threads ; hence being able to run portable bytecode files using
   such libraries ? 

If this is technically feasible, I would like to know people who are
interested in using this for distribution of their own application in
bytecode form. Moreover, for building this archive of ocamlrun
binaries, many people are needed to cover a wide range of
architecture/OS combinations. I have access myself to PCs under Linux
(Redhat distribution) and SPARC/Solaris 2.7, that's all.

Any comments, remarks and suggestions will be welcome. My goal is be
able to distribute as widely as possible an application written in
Ocaml, avoiding remarks like ``I'm interested in this application but
I cannot compile Ocaml sources, and you do not offer a suitable binary
for my configuration », and I would be glad to hear any suggestion for
achieving this goal.

- Claude



-- 
| Claude Marché           | mailto:Claude.Marche@lri.fr |
| LRI - Bât. 490          | http://www.lri.fr/~marche/  |
| Université de Paris-Sud | phoneto: +33 1 69 15 64 85  |
| F-91405 ORSAY Cedex     | faxto: +33 1 69 15 65 86    |

Re: Portability of applications written in OCAML

[Jean-Francois Monin]

> Mon but est de pouvoir distribuer le plus largement possible une
> application écrite en Ocaml, en évitant des remarques du genre «cette
> application m'intéresse mais je ne peux pas compiler le Ocaml et vous
> ne fournissez pas un binaire adapté à ma configuration»

Tiens, cela arrive qu'on ne puisse pas compiler les sources de Ocaml ?
Peut-être sur une config très exotique ?  De plus on trouve
facilement des .deb et .rpm tout prêts (sans compter les binaires pour
Wxx), ce qui assure, me semble-t-il, une bonne couverture actuellement.

> My goal is be able to distribute as widely as possible an application
> written in Ocaml, avoiding remarks like ``I'm interested in this
> application but I cannot compile Ocaml sources, and you do not offer
> a suitable binary for my configuration'',

I didn't imagine that Ocaml compiling sources could be difficult
excepted, may be, on exotic architectures. Then why care ?

  Jean-Francois

Re: Portability of applications written in OCAML

[Claude Marche]

Je me répond en partie a moi-meme car je viens de découvrir l'option
-make-runtime de ocamlc, qui me semble repondre a ma demande, je n'ai
pas encore testé. Je répond aussi a Jean-Francois ci-dessous.

[English]

It seems that option -make-runtime of ocamlc may answer to my
problem. Not tested yet.

- Claude


>>>>> "Jean-Francois" == Jean-Francois Monin <JeanFrancois.Monin@cnet.francetelecom.fr> writes:

    >> Mon but est de pouvoir distribuer le plus largement possible
    >> une application écrite en Ocaml, en évitant des remarques du
    >> genre « cette application m'intéresse mais je ne peux pas
    >> compiler le Ocaml et vous ne fournissez pas un binaire adapté à
    >> ma configuration »

    Jean-Francois> Tiens, cela arrive qu'on ne puisse pas compiler les
    Jean-Francois> sources de Ocaml ?  Peut-être sur une config très
    Jean-Francois> exotique ?  De plus on trouve facilement des .deb
    Jean-Francois> et .rpm tout prêts (sans compter les binaires pour
    Jean-Francois> Wxx), ce qui assure, me semble-t-il, une bonne
    Jean-Francois> couverture actuellement.

Le piege c'est que quand un utilisateur dit « je ne peux pas
compiler le Ocaml » il veut dire qu'il ne peut pas compiler des
fichiers .ml, et soit il ne s'est pas rendu compte qu'il lui fallait
installer le compilateur, soit il s'en est rendu compte et a décidé de
ne pas le faire en pensant que ce sera la galere. On ne peut pas
savoir que ocaml est facile a installer tant qu'on ne l'a pas fait !

    >> My goal is be able to distribute as widely as possible an
    >> application written in Ocaml, avoiding remarks like ``I'm
    >> interested in this application but I cannot compile Ocaml
    >> sources, and you do not offer a suitable binary for my
    >> configuration »,

    Jean-Francois> I didn't imagine that Ocaml compiling sources could
    Jean-Francois> be difficult excepted, may be, on exotic
    Jean-Francois> architectures. Then why care ?

How do you say "Le client est roi" in english ?




-- 
| Claude Marché           | mailto:Claude.Marche@lri.fr |
| LRI - Bât. 490          | http://www.lri.fr/~marche/  |
| Université de Paris-Sud | phoneto: +33 1 69 15 64 85  |
| F-91405 ORSAY Cedex     | faxto: +33 1 69 15 65 86    |

Re: Portability of applications written in OCAML

[Jean-Francois Monin]

> On ne peut pas savoir que ocaml est facile a installer tant qu'on ne
> l'a pas fait !

C'est juste. Il faudrait mettre dans la license Ocaml :
"Tout logiciel diffusé sous forme de sources Ocaml doit
préciser en grosses lettres que Ocaml est facile a installer" :-)

  JF

Re: Portability of applications written in OCAML

[Gerd Stolpmann]

On Tue, 15 Feb 2000, Claude Marche wrote:

>One idea could be distributing bytecode: is it true the any bytecode
>can be executed on any architecture and OS as soon as the right
>ocamlrun is used? (Even under non unix OSs like Microsoft-Windows or
>Mac-OS?) 

As far as I know: Yes. I did it already, and it worked even for systems
with different endianess. But I did it only for Unixes.

>If yes, anyone interested in using the application would
>need only to install the suitable ocamlrun for its architecture/OS
>combination.  It would allow any potentially interested user to try
>the application, and in case he finds it suitable for its need, he
>would be motivated to install ocaml for compiling from the sources and
>getting a faster application.
>
>For this to work, a kind of archive of various versions of ocamlrun
>should exist somewhere on the web, allowing to choose the right
>architecture/OS, also the right ocaml version, the right libraries
>like libc 5 or 6 or glibc.

Good idea.

>Nevertheless, there is a problem for applications that use other
>libraries like unix, str, nums, threads, that need to be compiled in
>-custom mode. If I understand well, binaries compiled with -custom
>contain the native code of the interpreter itself, so are not portable
>anymore. Techically speaking, is it possible to build an ocamlrun
>interpreter that contains those libraries statically linked?

This is possible by using the -make-runtime option.

But: This is not the best solution, as the number of libraries is growing,
and every time a new library (even a new version) would be needed by an
application, also new bytecode interpreters for all platforms would be
necessary. With two consequences:

1) The interpreters become very large, and there will be certainly people
   saying that OCaml is a big fat all-memory-eating language

2) The interpreters for platforms which are not everybody's darling will
   rapidly become too old to execute many applications.

My suggestion: Every modern operating system can link libraries dynamically.
This is also possible for the parts of the OCaml libraries written in C
which need to be available in the runtime system. The runtime system consists
of three parts:

1) The bytecode interpreter
2) The additional C-libraries
3) A startup routine which initializes the interpreter and configures the
   additional primitives available by 2)

There are no problems with parts 1 and 2 and dynamic libraries. Unfortunately,
part 3 is a generated C function, i.e. a different version is needed for every
combination of C-libraries. If we want dynamic libraries part 3 must be
replaced by a configurable dynamic loader. This is not too difficult, I think
I can write one (at least for Unix); the task of the dynamic loader is 
reading the list of necessary primitives from the bytecode file, loading the
associated libraries, and setting up the table that maps the primitives to
object-code symbols.

If this is done, we can link part 1 and the new loader that replaces part 3 to
the new, generic main executable which is able to run EVERY application for
which the necessary dynamic libraries are available.

So:

- For every platform, we need only one generic bytecode interpreter and one
  set of shared libraries

- Only libraries needed for the application are loaded into memory; the memory
  footprints become much smaller

- If new libraries are available, it is possible to compile them independently
  of the other libraries

- Multiple versions of the same library can easily be supported.

- It is often not necessary to recompile the libraries if a new version
  of the OCaml interpreter is available (at least if symbol-level 
  compatibility is given; the C FFI did not change often in the past).

>If this is technically feasible, I would like to know people who are
>interested in using this for distribution of their own application in
>bytecode form. Moreover, for building this archive of ocamlrun
>binaries, many people are needed to cover a wide range of
>architecture/OS combinations. I have access myself to PCs under Linux
>(Redhat distribution) and SPARC/Solaris 2.7, that's all.

Interested. I have some Web applications, and some TK applications.

I can offer 
 - Linux with libc5
 - Linux with glibc6.0
 - Linux with glibc6.1
   (Yes, I had problems to run OCaml applications compiled for glibc6.0 under
    glibc6.1!)
 - FreeBSD 3.3
 - Solaris 2.6
 - AIX 4.2

Gerd
-- 
----------------------------------------------------------------------------
Gerd Stolpmann      Telefon: +49 6151 997705 (privat)
Viktoriastr. 100             
64293 Darmstadt     EMail:   Gerd.Stolpmann@darmstadt.netsurf.de (privat)
Germany                     
----------------------------------------------------------------------------

Re: Portability of applications written in OCAML

[skaller]

Claude Marche wrote:
> Pour résumer mon propos, voici donc les questions que je me pose :
[]
> another application doing the same thing but easier to install.

 
> If this is technically feasible, I would like to know people who are
> interested in using this for distribution of their own application in
> bytecode form. 

> Any comments, remarks and suggestions will be welcome. My goal is be
> able to distribute as widely as possible an application written in
> Ocaml, avoiding remarks like ``I'm interested in this application but
> I cannot compile Ocaml sources, and you do not offer a suitable binary
> for my configuration », and I would be glad to hear any suggestion for
> achieving this goal.

I believe one fundamental obstacle to configuring ocaml is the lack of
dynamic loading.
Ideally, it should be possible for ocamlopt to build 'extensions' to the
bytecode
interpreter, efficively doing '-custom' linkage at load time.

The reason this is necessary is that if you have multiple ocaml
applications,
you may not want a special 'run time' for each one: each application
might
use the core, and a single separate extension module. 

I have no idea how to implement this. However, one of the things that
you can do -- albiet clumbsily -- in python, is generate C code,
run the compiler, and then load the resulting shared library as a python
extension module.

-- 
John (Max) Skaller, mailto:skaller@maxtal.com.au
10/1 Toxteth Rd Glebe NSW 2037 Australia voice: 61-2-9660-0850
homepage: http://www.maxtal.com.au/~skaller
download: ftp://ftp.cs.usyd.edu/au/jskaller

Re: Portability of applications written in OCAML

[Patrick Goldbronn - SYSCO]

Claude Marche wrote:
> >>>>> "Jean-Francois" == Jean-Francois Monin <JeanFrancois.Monin@cnet.francetelecom.fr> writes:
> 
>     >> Mon but est de pouvoir distribuer le plus largement possible
>     >> une application écrite en Ocaml, en évitant des remarques du
>     >> genre « cette application m'intéresse mais je ne peux pas
>     >> compiler le Ocaml et vous ne fournissez pas un binaire adapté à
>     >> ma configuration »
> 
>     Jean-Francois> Tiens, cela arrive qu'on ne puisse pas compiler les
>     Jean-Francois> sources de Ocaml ?  Peut-être sur une config très
>     Jean-Francois> exotique ?  De plus on trouve facilement des .deb
>     Jean-Francois> et .rpm tout prêts (sans compter les binaires pour
>     Jean-Francois> Wxx), ce qui assure, me semble-t-il, une bonne
>     Jean-Francois> couverture actuellement.
> 
> Le piege c'est que quand un utilisateur dit « je ne peux pas
> compiler le Ocaml » il veut dire qu'il ne peut pas compiler des
> fichiers .ml, et soit il ne s'est pas rendu compte qu'il lui fallait
> installer le compilateur, soit il s'en est rendu compte et a décidé de
> ne pas le faire en pensant que ce sera la galere. On ne peut pas
> savoir que ocaml est facile a installer tant qu'on ne l'a pas fait !
> 

C'est encore plus vrai si l'utilisateur potentiel est de mauvaise fois :
toutes les excuses sont bonnes pour ne pas utiliser un produit, meme
s'il a été fait pour lui s'il a décidé de ne pas entendre parlé de ocaml
(meme s'il n'a pas regardé ;-( .



>     >> My goal is be able to distribute as widely as possible an
>     >> application written in Ocaml, avoiding remarks like ``I'm
>     >> interested in this application but I cannot compile Ocaml
>     >> sources, and you do not offer a suitable binary for my
>     >> configuration »,
> 
>     Jean-Francois> I didn't imagine that Ocaml compiling sources could
>     Jean-Francois> be difficult excepted, may be, on exotic
>     Jean-Francois> architectures. Then why care ?
> 
> How do you say "Le client est roi" in english ?
> 

"Customers are kings", aren't they ?

-- 
#####################################
# Patrick GOLDBRONN                 #
# CEA - DRN/DMT/SYSCO               #
# CE-Saclay, Bâtiment 460           #
# 91191 GIF/YVETTE CEDEX (FRANCE)   #
#                                   #
# Tél : 01 69 08 73 55              #
# Fax : 01 69 08 96 96              #
#####################################

Re: Portability of applications written in OCAML

[Xavier Leroy]

Claude Marche asks:

>One idea could be distributing bytecode: is it true the any bytecode
>can be executed on any architecture and OS as soon as the right
>ocamlrun is used? (Even under non unix OSs like Microsoft-Windows or
>Mac-OS?) 

For MacOS, there are some problems with end-of-line being represented
by different control characters in MacOS and in Unix and Windows.  So,
newlines in program output are broken.

For MacOS and Windows, there is also the fact that file path names are
represented differently.  So, if your program manipulates file names,
even though the Filename module, the bytecode may not work well under
a different OS.

I think the easiest way to distribute an OCaml application is as
sources, plus binaries for a couple of popular architectures
(typically, Linux/Intel and Windows).  Novice users can pick the
binaries, and experienced users should have no problems installing
OCaml for compiling your application.

Gerd Stolpmann wrote:
> My suggestion: Every modern operating system can link libraries dynamically.
> This is also possible for the parts of the OCaml libraries written in C
> which need to be available in the runtime system.

Right, I have been considering dynamic loading of C libraries as an
alternative to -custom.  This would allow, among other benefits,
dynamic loading of Caml libraries that contain some C code.

There are some portability issues with old or exotic versions of Unix,
but I think it can be made to work under, say, Linux, Solaris,
Digital/Tru64 Unix, and Windows.  There are some issues still to be
resolved, however, such as how to build shared C libraries in a
portable fashion, perhaps by using the GNU libtool package.

> - Only libraries needed for the application are loaded into memory;
>   the memory footprints become much smaller

Yes and no, because static linking under C is able to remove members
of the .a archive that are not referenced, while dynamic loading
typically loads everything.  But memory footprint of code is not much
of an issue these days.

- Xavier Leroy

Re: Portability of applications written in OCAML

[skaller]

Xavier Leroy wrote:
 
> I think the easiest way to distribute an OCaml application is as
> sources, plus binaries for a couple of popular architectures
> (typically, Linux/Intel and Windows).  Novice users can pick the
> binaries, and experienced users should have no problems installing
> OCaml for compiling your application.

I have no problem compiling Ocaml on my Linux box but I CANNOT
do so on my WinNT box -- I don't have a C compiler for Windows
[except gcc for Cygwin]

-- 
John (Max) Skaller, mailto:skaller@maxtal.com.au
10/1 Toxteth Rd Glebe NSW 2037 Australia voice: 61-2-9660-0850
checkout Vyper http://Vyper.sourceforge.net
download Interscript http://Interscript.sourceforge.net

Re: Portability of applications written in OCAML

[Sven LUTHER]

On Fri, Feb 18, 2000 at 10:36:03AM +0100, Xavier Leroy wrote:
> Claude Marche asks:
> 
> >One idea could be distributing bytecode: is it true the any bytecode
> >can be executed on any architecture and OS as soon as the right
> >ocamlrun is used? (Even under non unix OSs like Microsoft-Windows or
> >Mac-OS?) 
> 
> For MacOS, there are some problems with end-of-line being represented
> by different control characters in MacOS and in Unix and Windows.  So,
> newlines in program output are broken.
> 
> For MacOS and Windows, there is also the fact that file path names are
> represented differently.  So, if your program manipulates file names,
> even though the Filename module, the bytecode may not work well under
> a different OS.
> 
> I think the easiest way to distribute an OCaml application is as
> sources, plus binaries for a couple of popular architectures
> (typically, Linux/Intel and Windows).  Novice users can pick the
> binaries, and experienced users should have no problems installing
> OCaml for compiling your application.
> 
> Gerd Stolpmann wrote:
> > My suggestion: Every modern operating system can link libraries dynamically.
> > This is also possible for the parts of the OCaml libraries written in C
> > which need to be available in the runtime system.
> 
> Right, I have been considering dynamic loading of C libraries as an
> alternative to -custom.  This would allow, among other benefits,
> dynamic loading of Caml libraries that contain some C code.

Would that make -custom bytecode arch independent again ? how do you handle
libraries with different exported symbols per arch ?

> There are some portability issues with old or exotic versions of Unix,
> but I think it can be made to work under, say, Linux, Solaris,
> Digital/Tru64 Unix, and Windows.  There are some issues still to be
> resolved, however, such as how to build shared C libraries in a
> portable fashion, perhaps by using the GNU libtool package.
> 
> > - Only libraries needed for the application are loaded into memory;
> >   the memory footprints become much smaller
> 
> Yes and no, because static linking under C is able to remove members
> of the .a archive that are not referenced, while dynamic loading
> typically loads everything.  But memory footprint of code is not much
> of an issue these days.

Euh, ...

is the member removal not done using the strip program ? stripping ocaml
bytecode executable is a very bad idea, as can be seen when trying to strip
ocamldebug for example. Notice that ocamlc, ocamlopt and ocaml don't seem to
suffer from this problem.

Friendly,

Sven LUTHER

Re: Portability of applications written in OCAML

[Xavier Leroy]

> Would that make -custom bytecode arch independent again ?

That's the idea, yes.

> how do you handle libraries with different exported symbols per arch ?

Typically, when interfacing an existing C library with Caml, you have
two libraries: the C library and a library containing the stub
functions for communication with Caml.  The latter must export the
same stub functions on all platforms, indeed, but it can then adapt to
variations in the underlying C library using #ifdefs and so on.

> Euh, ...
> is the member removal not done using the strip program ? 

Not at all.  I was talking about the following feature of Unix
linkers: when a ".a" library is statically linked, not all ".o" object
files contained in the library are linked and put in the executable,
but only those that define symbols actually referenced by the program.

> stripping ocaml
> bytecode executable is a very bad idea, as can be seen when trying to strip
> ocamldebug for example. Notice that ocamlc, ocamlopt and ocaml don't seem to
> suffer from this problem.

Yes, all bytecode executables produced with the -custom option must
not be stripped.  This is even mentioned in the manual, I think.

- Xavier Leroy

Re: Portability of applications written in OCAML: C stuff

[Max Skaller]

I have a problem using some third party\x03C codes with ocaml,
this includes pcre (defintely) and possibly mlgtk.
Some work needs to be done figuring out the right way to do this.
Here's the problem:

I am distributing an ocaml/C tool which uses third party
components like mlgtk and pcre which require C code.

The way these packages are distributed is a 'build in separate
directory and install' model. This is NOT acceptable for my
package. I require all components to be built and usable
WITHOUT installing them (with one exception: the standard ocaml (and
python)
distributions). 

What I do is copy relevant source files
from distributed packages, and, if necessary, edit them,
then I build them as part of my system, directly from these
sources.

[In other words I must distribute IN MY PACKAGE everything
which is not stock standard in the official ocaml distribution,
and a single command must build the whole system]

This is a pain. I would like to ask suppliers of third party
packages please

	1. DO NOT USE A FILE CALLED 'config.h'
	  
	Such a file will clobber someone elses config.h
	Pleae call these files 'pcre_config.h' or 'gdk_config.h'
	or whatever.

	PLEASE use filenames that are specialised to your package,
	do NOT use generic names on the assumption your code will
	be built with your Makefile in a separate directory.

	2. Do NOT require you package be 'main'.
	Only ONE main is allowed, and it is MINE.
	Do NOT supply 'main' in ANY C libraries.
	(you can supply a sample 'main.c', but is must be that: a sample)

	[This also seems to affect 'numerix']


Some of these problems will be alieviated when Ocaml gets a proper
mechanism to add components to the system. This is non-trivial,
but it is handled much better in Python than Ocaml at present.

I have made quite extensive additions to mlgtk, but I cannot
supply a 'patch' to the original authors because I have been
forced to rename files, and make edits which break that packages
build model.

-- 
John (Max) Skaller at OTT [Open Telecommications Ltd]
mailto:maxs@in.ot.com.au      -- at work
mailto:skaller@maxtal.com.au  -- at home

Re: Portability of applications written in OCAML: C stuff

[Markus Mottl]

> The way these packages are distributed is a 'build in separate
> directory and install' model. This is NOT acceptable for my
> package. I require all components to be built and usable
> WITHOUT installing them (with one exception: the standard ocaml (and
> python)
> distributions). 

Hm, I don't know about "mlgtk", but what concerns the PCRE, you need not
install it anywhere to use it. The only thing you have to do is place
directory "{whatever}/pcre_ocaml/pcre-OCaml" on the include- and the
library path when you compile/link your project.

In case you also use my "OcamlMakefile", this should be fairly easy, e.g.
add something like:

  INCDIRS = ../../pcre-OCaml
  LIBDIRS = ../../pcre-OCaml
  LIBS    = pcre

> [In other words I must distribute IN MY PACKAGE everything
> which is not stock standard in the official ocaml distribution,
> and a single command must build the whole system]

I don't see any problem here - just "entering" into the directory of the
"third-party" distribution, "making" and including the directory with the
generated libraries should work fine.

> 	1. DO NOT USE A FILE CALLED 'config.h'
> 	  
> 	Such a file will clobber someone elses config.h
> 	Pleae call these files 'pcre_config.h' or 'gdk_config.h'
> 	or whatever.

"config.h" is generated by "configure" automatically and is actually part
of the C-library "pcre", which I always copy verbatim from its author.
Unless you also copy this "configure"-script out of its assigned directory
into your source dir, it won't do any harm to your files.

> 	PLEASE use filenames that are specialised to your package,
> 	do NOT use generic names on the assumption your code will
> 	be built with your Makefile in a separate directory.

This is impossible. How do you want to guarantee that files of other people
don't get overwritten if you mix some distribution with theirs? That's what
directories are for!

> 	2. Do NOT require you package be 'main'.
> 	Only ONE main is allowed, and it is MINE.
> 	Do NOT supply 'main' in ANY C libraries.
> 	(you can supply a sample 'main.c', but is must be that: a sample)

I always prefix C-identifiers with very unique names so as to avoid
name-clashes. I also don't see any "main"-routine in the underlying
PCRE-library other than in example files.

> Some of these problems will be alieviated when Ocaml gets a proper
> mechanism to add components to the system. This is non-trivial,
> but it is handled much better in Python than Ocaml at present.

I agree on this: a standard way for adding libraries, etc., would probably
boost productivity...

Regards,
Markus Mottl

-- 
Markus Mottl, mottl@miss.wu-wien.ac.at, http://miss.wu-wien.ac.at/~mottl

Re: Portability of applications written in OCAML: C stuff

[Max Skaller]

Markus Mottl wrote:
> 
> > The way these packages are distributed is a 'build in separate
> > directory and install' model. This is NOT acceptable for my
> > package. I require all components to be built and usable
> > WITHOUT installing them (with one exception: the standard ocaml (and
> > python)
> > distributions).
> 
> Hm, I don't know about "mlgtk", but what concerns the PCRE, you need not
> install it anywhere to use it. The only thing you have to do is place
> directory "{whatever}/pcre_ocaml/pcre-OCaml" on the include- and the
> library path when you compile/link your project.


	This model doesn't really work for me. Consider that typically
this requires

	1) run .configure
	2) run make

This is not acceptable. My rules are: after the client untars the
distribution of
my product, they may have to edit part of the build script, 'maker'
which is 
written in Python. Then they say 'python maker' and the build has to
work
without any other intervention. Make and autoconf are not permitted: any
work
these do must be done by ME in the 'maker' script IN PYTHON.

In particular, while my product is currently Unix only, the build
process
must work under Windows too, which rules out make, autoconf, and other
such silliness. :-)

> "config.h" is generated by "configure" automatically and is actually part
> of the C-library "pcre", which I always copy verbatim from its author.
> Unless you also copy this "configure"-script out of its assigned directory
> into your source dir, it won't do any harm to your files.

	pcre cannot build without config.h, and neither can mlgtk.
So there is a conflict, since both require a file called 'config.h'.

> >       PLEASE use filenames that are specialised to your package,
> >       do NOT use generic names on the assumption your code will
> >       be built with your Makefile in a separate directory.
> 
> This is impossible. 

	It is not impossible to try.

> How do you want to guarantee that files of other people
> don't get overwritten if you mix some distribution with theirs? 

	I guarrantee it, since I am combining the sources.

> That's what directories are for!

	I do not wish to build and mess around with subpackages
in directories, which amount to a small part of my overall package:
pcre, for example, corresponds to a single library module with a few
functions in it. There are hundreds of functions to build, and I want
the sources all in the SAME directory.

	When ocaml supports packages with subdirectory structures natively,
THEN I will require the client install third party packages, rather than
distributing them as an integral part of my source.
 
> I agree on this: a standard way for adding libraries, etc., would probably
> boost productivity...

	Enormously -- but I will be happy to wait until it can be done
right, since it is VERY hard to 'fix' a broken design here. {The python 
model is OK, but not perfect}

-- 
John (Max) Skaller at OTT [Open Telecommications Ltd]
mailto:maxs@in.ot.com.au      -- at work
mailto:skaller@maxtal.com.au  -- at home

Re: Portability of applications written in OCAML: C stuff

[Sven LUTHER]

On Thu, Feb 24, 2000 at 01:55:57PM +1100, Max Skaller wrote:
> Markus Mottl wrote:
> > 
> > > The way these packages are distributed is a 'build in separate
> > > directory and install' model. This is NOT acceptable for my
> > > package. I require all components to be built and usable
> > > WITHOUT installing them (with one exception: the standard ocaml (and
> > > python)
> > > distributions).
> > 
> > Hm, I don't know about "mlgtk", but what concerns the PCRE, you need not

About mlgtk, ...

you can use it without installing it anywhere, look at the examples for
guidance on how to do it (mostly you haveto set the right -I and -L options,
so ocaml will find the libraries).

Friendly,

Sven LUTHER

Re: Portability of applications written in OCAML: C stuff

[Alan Schmitt]

..
>In particular, while my product is currently Unix only, the build
>process
>must work under Windows too, which rules out make, autoconf, and other
>such silliness. :-)

I gather this rules out using a compiler too ... ;-)
In particular, when you say "must work under windows", do you mean out
of the box window, or one with some tools installed (ocaml, python,
cygwin thus bash, make ...)

>	pcre cannot build without config.h, and neither can mlgtk.
>So there is a conflict, since both require a file called 'config.h'.

I have many "config.h" in many source distributions in my /usr/src
directory. I have never had such a problem. Actually, most of the
time, to build an application you don't need to build the
library. You actually require that it is there (and the nice thing is
you can make your ./configure check for it). Librairies and include
files are installed in some canonical places which can be looked up,
or non-canonical which can be explicitely given to the installation
script.

>> >       PLEASE use filenames that are specialised to your package,
>> >       do NOT use generic names on the assumption your code will
>> >       be built with your Makefile in a separate directory.
>> 
>> This is impossible. 
>
>	It is not impossible to try.

True, but since there is a nice mechanism to deal with this (ie
Makefile and directories), is this really worth it ?
Do you think we should adopt the java convention for naming ? This
would be a pain ...

Alan Schmitt
--
The hacker: someone who figured things out and made something cool happen.

Re: Portability of applications written in OCAML: C stuff

[Gerd Stolpmann]

On Thu, 24 Feb 2000, Max Skaller wrote:
>This is not acceptable. My rules are: after the client untars the
>distribution of
>my product, they may have to edit part of the build script, 'maker'
>which is 
>written in Python. Then they say 'python maker' and the build has to
>work
>without any other intervention. Make and autoconf are not permitted: any
>work
>these do must be done by ME in the 'maker' script IN PYTHON.
>
>In particular, while my product is currently Unix only, the build
>process
>must work under Windows too, which rules out make, autoconf, and other
>such silliness. :-)

We all know that Windows is not a serious operating system because it lacks
support for many tasks we all EXPECT from an operating system. This is one of
the reasons why I do not offer Windows support for my components. You have only
two possibilities: (A) reinventing the wheel by writing tools yourself; (B) not
offering special Windows support.

Please don't blame us that it is difficult to implement (A).

I would say that your build process is ... mhhh ... strange. If I did it
with only limited means, I would still distinguish between source directories,
and a (temporary) installation directory, say:

lib: the installation directory
source1, source2, ...: source directories

Simply compile the sources one after another (in a fixed order), and install
the results in lib. While compiling the sources, include -I ../lib such that
the previous results can be used in the next compilation step. This is simple
enough to be implemented even without Makefiles and autoconf.

>> "config.h" is generated by "configure" automatically and is actually part
>> of the C-library "pcre", which I always copy verbatim from its author.
>> Unless you also copy this "configure"-script out of its assigned directory
>> into your source dir, it won't do any harm to your files.
>
>	pcre cannot build without config.h, and neither can mlgtk.
>So there is a conflict, since both require a file called 'config.h'.

As far as I know you cannot choose the name of "config.h"; this is hard-wired
in autoconf.

>> That's what directories are for!
>
>	I do not wish to build and mess around with subpackages
>in directories, which amount to a small part of my overall package:
>pcre, for example, corresponds to a single library module with a few
>functions in it. There are hundreds of functions to build, and I want
>the sources all in the SAME directory.

Your argumentation is ridiculous. I often have directories with only very few
files if there good reasons to create them. In general I would argue that
directories reduce the mess.

>	When ocaml supports packages with subdirectory structures natively,
>THEN I will require the client install third party packages, rather than
>distributing them as an integral part of my source.

Why do you need a "native" package implementation? I think there are good
reasons to have one (e.g. a uniform installation procedure), but in your case
you can use ANY package mechanism (I have one), simply distribute it with your
sources.

>> I agree on this: a standard way for adding libraries, etc., would probably
>> boost productivity...
>
>	Enormously -- but I will be happy to wait until it can be done
>right, since it is VERY hard to 'fix' a broken design here. {The python 
>model is OK, but not perfect}

What does "right" mean? It depends on your expectations. For example, I
think a package mechanism should not include any build functionality (some kind
of "make") because there are already good solutions (even for Windows (cygwin)).

Gerd
-- 
----------------------------------------------------------------------------
Gerd Stolpmann      Telefon: +49 6151 997705 (privat)
Viktoriastr. 100             
64293 Darmstadt     EMail:   Gerd.Stolpmann@darmstadt.netsurf.de (privat)
Germany                     
----------------------------------------------------------------------------

Re: Portability of applications written in OCAML: C stuff

[Max Skaller]

Alan Schmitt wrote:
> 
> ..
> >In particular, while my product is currently Unix only, the build
> >process
> >must work under Windows too, which rules out make, autoconf, and other
> >such silliness. :-)
> 
> I gather this rules out using a compiler too ... ;-)


	I wish. The client must configure commands to compile
and link C. This cannot be avoided.

> In particular, when you say "must work under windows", do you mean out
> of the box window, or one with some tools installed (ocaml, python,
> cygwin thus bash, make ...)

	The current requirements are for Python and Ocaml installed.
Plus you need the same C compiler used to build Python and Ocaml.
[You also need GTK]

> I have many "config.h" in many source distributions in my /usr/src
> directory. I have never had such a problem. 

> True, but since there is a nice mechanism to deal with this (ie
> Makefile and directories), is this really worth it ?

	What? Nice mechanism? Are you joking or ignorant?

-- 
John (Max) Skaller at OTT [Open Telecommications Ltd]
mailto:maxs@in.ot.com.au      -- at work
mailto:skaller@maxtal.com.au  -- at home

Re: Portability of applications written in OCAML: C stuff

[Max Skaller]

Gerd Stolpmann wrote:

> We all know that Windows is not a serious operating system because it lacks
> support for many tasks we all EXPECT from an operating system. 

	? I do not understand. I run NT at home, and it supports
a LOT of functionality I expect from an operating system that my
Linux box does not.

	It runs games and office products like Word and IE. 
It has a clean, snappy, GUI. It has a reasonable scheduler, and a 
superior method of program construction (DLLs).


-- 
John (Max) Skaller at OTT [Open Telecommications Ltd]
mailto:maxs@in.ot.com.au      -- at work
mailto:skaller@maxtal.com.au  -- at home

Re: Portability of applications written in OCAML: C stuff

[Alan Schmitt]

>> In particular, when you say "must work under windows", do you mean out
>> of the box window, or one with some tools installed (ocaml, python,
>> cygwin thus bash, make ...)
>
>	The current requirements are for Python and Ocaml installed.
>Plus you need the same C compiler used to build Python and Ocaml.
>[You also need GTK]

I have never compiled ocaml under windows, but I thought there was
some kind of nmake to do it.

And some day, it should be possible to compile it using cygwin, which
is slightly cheaper than some m$ compilers ;-) and which makes
available many nice command line tools.

>> True, but since there is a nice mechanism to deal with this (ie
>> Makefile and directories), is this really worth it ?
>
>	What? Nice mechanism? Are you joking or ignorant?
>

I must say I am ignorant. I've only compiled software under Linux, and
installed it under Linux and Windows.

The only way to get really rid of software under windows is to
reinstall the whole system from scratch (the garbage collecting of
dlls is not that good ;-). Most of the time, when I uninstall
something that decided to go live in common directories, uninstall
fails. Under Linux, I think makefiles work well, and it is possible to
specify an install directory during the configuration process (and
when I say that, I mean there will be nothing that will go in another
directory, as it can happen for some other os).

Now if you would be so kind as to explain what better solution you
have compared to makefiles, I'm all ears.

Alan Schmitt
--
The hacker: someone who figured things out and made something cool happen.

Re: Portability of applications written in OCAML: C stuff

[STARYNKEVITCH Basile]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 1028 bytes --]

>>>>> "Max" == Max Skaller <maxs@in.ot.com.au> writes:

    Max> Gerd Stolpmann wrote:
    >> We all know that Windows is not a serious operating system
    >> because it lacks support for many tasks we all EXPECT from an
    >> operating system.

    Max> 	? I do not understand. I run NT at home, and it
    Max> supports a LOT of functionality I expect from an operating
    Max> system that my Linux box does not.

Although I do prefer Linux to Microsoft systems I really think that OS
flamewars definitely DO NOT belong to the Ocaml list.

N.B. Any opinions expressed here are only mine, and not of my organization.
N.B. Les opinions exprimees ici me sont personnelles et n engagent pas le CEA.

---------------------------------------------------------------------
Basile STARYNKEVITCH   ----  Commissariat à l Energie Atomique 
DTA/LETI/DEIN/SLA * CEA/Saclay b.528 (p111f) * 91191 GIF/YVETTE CEDEX * France
phone: 1,69.08.60.55; fax: 1.69.08.83.95 home: 1,46.65.45.53
email: Basile point Starynkevitch at cea point fr 

Re: Portability of applications written in OCAML: C stuff

[skaller]

Alan Schmitt wrote:

> Now if you would be so kind as to explain what better solution you
> have compared to makefiles, I'm all ears.

See my previous email. I have no end of problems installing
things (in general) as source. Ocaml is a pleasant exception,
it is very easy to build.

BTW: please think about configuration management and installation for
a second: it is well known to be a nightmare on all systems.
Specialists are employed by most companies with more than few boxes
just to keep them running and maintain the software configuration.
I think it's obvious on reflection that this is a really difficult
problem, and there really has to be something a lot better than
make and friends.

-- 
John (Max) Skaller, mailto:skaller@maxtal.com.au
10/1 Toxteth Rd Glebe NSW 2037 Australia voice: 61-2-9660-0850
checkout Vyper http://Vyper.sourceforge.net
download Interscript http://Interscript.sourceforge.net

Re: Portability of applications written in OCAML: C stuff

[Juergen Pfitzenmaier]

Sorry, I forgot that URL last time. Here they are.
   http://www.baldmt.com/cons/
   ftp://ftp.cs.colorado.edu/pub/distribs/odin/

pfitzen

Re: Portability of applications written in OCAML: C stuff

[skaller]

Juergen Pfitzenmaier wrote:
> 
> Max Skaller wrote on Wed, 23 Feb 2000 10:43:46 +1100
> >         1. DO NOT USE A FILE CALLED 'config.h'
> ....
> >        do NOT use generic names on the assumption your code will
> >        be built with your Makefile in a separate directory.
> 
> I have seen some big projects (> 100 MB code) and several smaller ones. In each of
> them people had to fight with the configuration. They all settled for what seems to
> be the least pain: built and distribute (!) in small blocks; keep each block in its
> own directory; use the same general frame for each block if possible (yes: use the
> name config.h over and over again).
> Even if its leading away from OCaml: I think that your real problem is with "make"
> and its way of building project. Look for "configuration management" and "make replacement"
> on the net.

no thanks. you see, I have a vastly superior system.
It is called interscript. (http://interscript.sourceforge.net).
Interscript is a literate programming tool written in Python.
Interscript 'script' is Python too. This replaces _completely_
all configuration control, takes responsibility not only for
compiling target software -- and of course, documenting it --
but also testing and installing it.

Unlike like 'make', 'autoconf' and various lame shell scripts,
Python is a full scale general purpose programming language.

Interscript, being experimental, does not yet provide the uniform
language independent (and that includes human languages as well as
programming
languages) build platform I'd like it to, and it is a bit slow
generating
large documents.

For this reason, I an writing a fast, fully featured, Python compatible
interpreter call Vyper (http://Vyper.sourceforge.net), which is designed
to support later building of a compiler. Vyper is written in Ocaml.

The current version of Vyper is not literate programmed, and it won't be
until Vyper executes interscript satisfactorily. [One of the main hang
ups
is finalising Python class instance objects -- which means executing
their __del__ methods, in the correct order(ocaml 2.999 finalisers work,
but finalise the objects in the wrong order at present).

The combination of Vyper and Interscript will be a rather amazingly
powerful
tool -- compared with make, autoconf, and shell script.

Interscript programs are distributed as sets of text files. Rarely are
subdirectories needed, mainly because an interscript source is a level
of abstraction above a standard source code source file. [Interscript
itself,
for example, has a tree like directory structure of Python packages,
but the original sources typically contain a whole directory contents
in a few files (interscript is 'verging' on needing subdirectories for
the source).]

In a sense, interscript sources are 'self extracting archives'.
And unlike other methods of packaging 'free' or other software sources,
you _always_ get documentation (even if it is only a program listing :-)

As to autoconf -- it may well make sense to run it _once_ to build
a configuration library interscript can use to control the building
of software. But typically, the ability of interscript to _generate_
code provides a vastly superior method of configuration to hackery
like C macros, conditional compilation, and other weird C'isms
(that won't work with other languages).

So my aim is to eventually distribute Vyper as interscript source
(as interscript itself is distributed, together with a prebuild
version for bootstrapping). And that means I need to get rid of 
subpackage makes, autoconfs, and other things, so as to unify
the distribution mechanism.

So you see, you're quite right:

> Even if its leading away from OCaml: I think that your real problem is with "make"
> and its way of building project. 

but

>Look for "configuration management" and "make replacement" on the net.

I'be been programming for quite some time, and I know how the ludicrous
make
and configuration systems people use on systems, including Unix and
Windows,
work -- or should I say, _fail_ to work. This is well known. Why else
would Redhat, for example, have created rpm? (that doesn't work so well
either ..)

-- 
John (Max) Skaller, mailto:skaller@maxtal.com.au
10/1 Toxteth Rd Glebe NSW 2037 Australia voice: 61-2-9660-0850
checkout Vyper http://Vyper.sourceforge.net
download Interscript http://Interscript.sourceforge.net

Re: Portability of applications written in OCAML: C stuff

[Juergen Pfitzenmaier]

Alan Schmitt wrote on Fri, 25 Feb 2000 08:37:43 +0000
>  Now if you would be so kind as to explain what better solution you
>  have compared to makefiles, I'm all ears.
I don't know what others have to say, but take a look at tools like odin and cons.

pfitzen

Re: Portability of applications written in OCAML: C stuff

[Juergen Pfitzenmaier]

Max Skaller wrote on Wed, 23 Feb 2000 10:43:46 +1100
>         1. DO NOT USE A FILE CALLED 'config.h'
....
>        do NOT use generic names on the assumption your code will
>        be built with your Makefile in a separate directory.

I have seen some big projects (> 100 MB code) and several smaller ones. In each of
them people had to fight with the configuration. They all settled for what seems to
be the least pain: built and distribute (!) in small blocks; keep each block in its
own directory; use the same general frame for each block if possible (yes: use the
name config.h over and over again).
Even if its leading away from OCaml: I think that your real problem is with "make"
and its way of building project. Look for "configuration management" and "make replacement"
on the net.

pfitzen