[Caml-list] OCaml packaging problems: findlib helps!

[Caml-list] OCaml packaging problems: findlib helps!

[Gerd Stolpmann]

Hello list,

I have prepared an experimental version of findlib (download see below)
that allows users to store DLLs into a shared directory, called "libexec".
As you may know, findlib organizes packages by directory. For example,
an installation with three packages X, Y, and Z could look like:

/usr/local/lib/ocaml/site-lib/X/<files>
/usr/local/lib/ocaml/site-lib/Y/<files>
/usr/local/lib/ocaml/site-lib/Z/<files>

Previous versions of findlib put DLLs simply into the package directories,
and findlib tried to update the ld.conf file (which may fail because of
missing file permissions). A similar solution is also used by the Debian
packagers.

There are some problems with it:
- There are scalability problems when really many directories are in ld.conf
- Package management systems like dpkg and rpm must use postinstall scripts
  which makes packaging more difficult
- There is a conflict between dpkg/rpm packages and additional manual
  installations: Who is responsible for which line in ld.conf?
- Name clashes are hard to detect (e.g. X and Y use both dllfoo.so)

The experimental version of findlib recognizes a special directory libexec:

/usr/local/lib/ocaml/site-lib/libexec/<DLL files>

The simple existence of this directory causes that all files with the suffixes
.so and .dll are copied into this directory (no symlinks). Of course, this breaks
the current way of representing the ownership of files. In previous versions,
all files in the package directory for X are owned by X. This is simple,
and works for everything else very well. In libexec, there are now .owner files,
e.g. for dllfoo.so findlib creates a second file dllfoo.so.owner, containing
only one line with the name of the owner.

Knowing the owner of files is necessary to delete a package.

The advantages of this solution are:
- It is not necessary to update ld.conf when a package is installed or deleted.
  Usually, there are very few such libexec directories, and they are known in
  advance.
- dpkg and rpm do no longer need postinstall scripts.
- The lookup of DLLs is faster.
- findlib checks whether there are name clashes.

Nothing is perfect:
- Recognizing a DLL by its file extension may be wrong
- The .owner files are ugly, and this functionality overlaps with dpkg/rpm.
- It is still difficult to install several versions of the same library.
  Although findlib supports several package repositories, this does
  not help, because all libexec directories must be mentioned in ld.conf,
  and the order in this file determines which library is loaded. The workaround
  is to set CAML_LD_LIBRARY_PATH to influence the search order manually. 

If you want to look at this experiment, you can download it from
http://www.ocaml-programming.de/packages/findlib-0.7-pre.tar.gz

Gerd
-- 
----------------------------------------------------------------------------
Gerd Stolpmann      Telefon: +49 6151 997705 (privat)
Viktoriastr. 45             
64293 Darmstadt     EMail:   gerd@gerd-stolpmann.de
Germany                     
----------------------------------------------------------------------------
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

[Caml-list] RFC [Was: ... findlib helps!]

[Stefano Zacchiroli]

On Sat, May 18, 2002 at 08:23:16PM +0200, Gerd Stolpmann wrote:
> The experimental version of findlib recognizes a special directory libexec:

I'd like comments about this schema from the OCaml INRIA's guys.

We are going to discuss this solution between we debian ocaml
maintainers and, before choose a way for the debian ocaml installation,
we really appreciate upstream comments.

TIA,
Cheers.

-- 
Stefano Zacchiroli - undergraduate student of CS @ Univ. Bologna, Italy
zack@cs.unibo.it | ICQ# 33538863 | http://www.cs.unibo.it/~zacchiro
"I know you believe you understood what you think I said, but I am not
sure you realize that what you heard is not what I meant!" -- G.Romney
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] OCaml packaging problems: findlib helps!

[Blair Zajac]

Gerd Stolpmann wrote:
> 
> Hello list,
> 
> I have prepared an experimental version of findlib (download see below)
> that allows users to store DLLs into a shared directory, called "libexec".
> As you may know, findlib organizes packages by directory. For example,
> an installation with three packages X, Y, and Z could look like:
> 
> /usr/local/lib/ocaml/site-lib/X/<files>
> /usr/local/lib/ocaml/site-lib/Y/<files>
> /usr/local/lib/ocaml/site-lib/Z/<files>
> 
> Previous versions of findlib put DLLs simply into the package directories,
> and findlib tried to update the ld.conf file (which may fail because of
> missing file permissions). A similar solution is also used by the Debian
> packagers.
> 
> There are some problems with it:
> - There are scalability problems when really many directories are in ld.conf
> - Package management systems like dpkg and rpm must use postinstall scripts
>   which makes packaging more difficult
> - There is a conflict between dpkg/rpm packages and additional manual
>   installations: Who is responsible for which line in ld.conf?
> - Name clashes are hard to detect (e.g. X and Y use both dllfoo.so)
> 
> The experimental version of findlib recognizes a special directory libexec:
> 
> /usr/local/lib/ocaml/site-lib/libexec/<DLL files>
> 
> The simple existence of this directory causes that all files with the suffixes
> .so and .dll are copied into this directory (no symlinks). Of course, this breaks
> the current way of representing the ownership of files. In previous versions,
> all files in the package directory for X are owned by X. This is simple,
> and works for everything else very well. In libexec, there are now .owner files,
> e.g. for dllfoo.so findlib creates a second file dllfoo.so.owner, containing
> only one line with the name of the owner.

Gerd,

What about the possibility of having multiple owners for the same library?
I can see the case (in fact I'm going to build a module that does this)
that segments a single shared library into multiple modules.  Then I'd
want multiple owners for the same library.

Are there any other tricks I need to be careful of to make sure that
this works?

On an unrelated point, can you modify the Makefile to define a new
variable, say DEBUG, that is set to either -g or nothing and replace
all the -g's with $(DEBUG)?  This would trim down the size of the
generated files.  This could also be added to configure as a -debug
command line option.  I can send in a patch if requested.

Best,
Blair

--
Blair Zajac <blair@orcaware.com>
Web and OS performance plots - http://www.orcaware.com/orca/
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] OCaml packaging problems: findlib helps!

[Gerd Stolpmann]

On 2002.05.22 01:25 Blair Zajac wrote:
> Gerd Stolpmann wrote:
> > 
> > Hello list,
> > 
> > I have prepared an experimental version of findlib (download see below)
> > [...]
> > The simple existence of this directory causes that all files with the suffixes
> > .so and .dll are copied into this directory (no symlinks). Of course, this breaks
> > the current way of representing the ownership of files. In previous versions,
> > all files in the package directory for X are owned by X. This is simple,
> > and works for everything else very well. In libexec, there are now .owner files,
> > e.g. for dllfoo.so findlib creates a second file dllfoo.so.owner, containing
> > only one line with the name of the owner.
> 
> Gerd,
> 
> What about the possibility of having multiple owners for the same library?
> I can see the case (in fact I'm going to build a module that does this)
> that segments a single shared library into multiple modules.  Then I'd
> want multiple owners for the same library.
> 
> Are there any other tricks I need to be careful of to make sure that
> this works?

You could have a special package L that only contains the library, and a minimum
cma/cmxa archive that loads the library, but that does not access it. The rest
can be placed in other packages that list L as predecessor. Ensure that
you call at least one O'Caml function from L (it can be empty), otherwise
the linker will remove the libraries from L from any executable you try
to build.

> On an unrelated point, can you modify the Makefile to define a new
> variable, say DEBUG, that is set to either -g or nothing and replace
> all the -g's with $(DEBUG)?  This would trim down the size of the
> generated files.  This could also be added to configure as a -debug
> command line option.  I can send in a patch if requested.

I can remove it from the command that builds "ocamlfind", but I do not think
that the debugging symbols should be missing in any cma library. They are
removed anyway if you build an executable without -g, so it is only a matter
of disk space (on the developer's system), and disks become larger all
the time.

Gerd
-- 
----------------------------------------------------------------------------
Gerd Stolpmann      Telefon: +49 6151 997705 (privat)
Viktoriastr. 45             
64293 Darmstadt     EMail:   gerd@gerd-stolpmann.de
Germany                     
----------------------------------------------------------------------------
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners