Re: [Caml-list] [OSR] Ports-like package management system

[Berke Durak <berke.durak@exalead.com>]

Hello all,

Here is the kind of thing I would like to be able to do with an
integrated build/packaging/VCS system.  I think this explains why I want
to base the system on a VCS.

We have a collection of software components C1, C2, ...  A version
control system VCS is selected and agreed upon.  Each usable version
of a software component Ci is stored under a VCS repository Si.  A
single command, say OPKG, does all.  (I think we'll have OPKG =
ocamlbuild, but it could be OMake or something else).

   % cd src
   % mkdir myproject
   % OPKG init           # Initialize the project
   % cat >foo.ml
   Printf.printf "Hello, world!\n" ^D
   % OPKG build foo.native
   (Compiling using ocamlbuild)
   % ./foo.native
   Hello, world!

Now let's say we want to use a regexp library.  Assume there exist
multiple implementations with the same API, say
OSRStandardRegexpLibraryAPI, that we want to use:

   % cat >_tags
   <foo.native>:
   require(some_regexp_lib_with_osr_standard_regexp_library_api)

As pointed by others, it is a good idea to decouple the build rules
from the used versions or the repository URLs.

A file, _requirements, maps require strings (which are enclosed in a
require(..) construct in the _tags file) to actual requirements.

   % cat >_requirements
   some_regexp_lib_with_osr_standard_regexp_library_api: latest(pcre)

In a globally accessible place, there is a file that defines how to
resolve component names to actual repository URLs.  For instance, in
$HOME/.opkg/resolve:

   % cat ~/.opkg/resolve
   pcre latest: v3.4
   pcre updates: http://ocaml-pcre.org/opkg/updates
   pcre v3.4: http://ocaml-pcre.org/repo/ rev r381
   pcre v3.3: http://ocaml-pcre.org/repo/ rev r341
   pcre v3.2: http://ocaml-pcre.org/repo/ rev v3.2

   netclient latest: v9.99
   netclient updates: http://ocaml-pcre.org/opkg/updates
   netclient v9.99: http://netclient.com/repo/ rev r364
   netclient v9.98: http://netclient.com/repo/ rev r300
   netclient v9.97: http://netclient.com/repo/ rev r200

This resolution file can be overridden by creating a file _override_resolve in
the project directory.

We then start using Regexps in our foo.ml:

   % cat foo.ml
   open OSR.Regexp

   let _ = Printf.printf "Hello, %s!\n"
     (if regexp_match ~rex:"[Rr]obert" Sys.argv.(1) then
       "Bob"
     else
       Sys.argv.(1))

   % OPKG build foo.native
   OPKG: inside ~/src/myproject
   OPKG: building map
   OPKG: map: foo.native needs some_regexp_lib_with_osr_standard_regexp_library_api
   OPKG: map: some_regexp_lib_with_osr_standard_regexp_library_api configured here
              as latest(pcre) from _requirements
   OPKG: map: latest(pcre) maps to pcre v3.4 from ~/.opkg/map

OPKG has here created a file _map that maps requirements to actual
versions.

% cat _map
some_regexp_lib_with_osr_standard_regexp_library_api: pcre v3.4

OPKG continues building the target:

   OPKG: done building map OPKG: pcre v3.4 not already checked out
   OPKG: pcre v3.4 available from hg http://ocaml-pcre.org/ rev r381
   OPKG: checking out pcre v3.4 inside ~/src/opkg/pcre-v3.4 (path
         configured from from ~/.opkg.config)
   OPKG: building pcre.native v3.4 in ~/src/opkg/pcre-v3.4
   OPKG: continuing compilation of foo.native

Object files for pcre-v3.4 would reside in
~/src/opkg/pcre-v3.4/_build/ No need to build the bytecode version
unless explicitly needed.

   % OPKG build foo.byte
   OPKG: inside ~/src/myproject ...
   OPKG: building pcre.byte v3.4 in ~/src/opkg/pcre/v3.4
   OPKG: continuing compilation of foo.byte

Similarly, if we want to build a multithreaded version, OPKG will
automatically build multithreaded versions of all required libraries,
using the tag mechanism.

   % OPKG build foo.byte.mt
   OPKG: building pcre.byte.mt v3.4 in ~/src/opkg/pcre/v3.4

Now assume we take a vacation and when we come back, a new version of
PCRE is released, say v3.5.  Of course we need to ask OPKG to update
its resolution file.  We can invoke it anywhere.

   % OPKG update
   OPKG: fetching http://ocaml-pcre.org/opkg/updates
   OPKG: new pcre version v3.5 (also the latest)
   OPKG: fetching http://netclient.com/opkg/updates
   OPKG: netclient still at v9.99
   OPKG: updated ~/.opkg/resolve

If we want to use the newest PCRE in our project, we need to update
~/src/myproject/_map, since it is this file which defines the exact
version to be used.  We can edit that file by hand; but we can also
ask OPKG to regenerate it from the _requirements file:

   % OPKG remap
   OPKG: inside ~/src/myproject
   OPKG: building map
   OPKG: map: foo.native needs
         some_regexp_lib_with_osr_standard_regexp_library_api
   OPKG: map: some_regexp_lib_with_osr_standard_regexp_library_api
         configured here as latest(pcre) from _requirements
   OPKG: map: latest(pcre) maps to pcre v3.5 from ~/.opkg/map
   OPKG: done building map

We can check which versions have been used in building a target with
the command OPKG composition:

   % OPKG composition foo.native
   foo.native:
     made from map(0f9e8b9d8bbdae7bd9e78ab9d8e7ade83)(OUTDATED)
   map(0f9e8b9d8bbdae7bd9e78ab9d8e7ade83):
     ocaml 3.10.1(nativedynlinkpatch),
     PCRE v3.4(native)

Here OPKG tells us that the current foo.native was built using the a
map file whose digest was 0f9e8b9d8bbdae7bd9e78ab9d8e7ade83 which
instructed OPKG to use ocaml 3.10.1 configured with the
nativedynlinkpatch option, and with PCRE v3.5, in native code.
OUTDATED means that the map has changed.

Now let's rebuild our project:

   % OPKG build foo.native
   OPKG: inside ~/src/myproject
   OPKG: map 0f9e8b9d8bbdae7bd9e78ab9d8e7ade83 outdated by current map
         d78b279f78b3fb804983481834
   OPKG: pcre v3.5 not already checked out
   OPKG: pcre v3.5 available from hg http://ocaml-pcre.org/ rev r450
   ...

And OPKG will fetch pcre v3.5, and rebuild it.

   % OPKG composition foo.native
   foo.native:
     made from map(d78b279f78b3fb804983481834)(CURRENT)
   map(d78b279f78b3fb804983481834):
     ocaml 3.10.1(nativedynlinkpatch),
     PCRE v3.5(native)

What happens if we want to make an improvement to PCRE ?  For
instance, we might want to add a ?drm option to the match_regexp
function.

We simply go into ~/src/opkg/pcre and clone the v3.5 directory into
v3.5-drm:

   % cd ~/src/opkg/pcre VCS clone v3.5 v3.5-drm

We then start hacking into v3.5-drm.  Of course we want to use
myproject to debug it, so we override its map file by adding the
following line into _map_override:

   % cat _map_override
   some_regexp_lib_with_osr_standard_regexp_library_api: pcre v3.5-drm

Meanwhile, an important buffer overflow is fixed in PCRE.  As we have
been hacking the DRM extension under VCS, we can easily merge the
patch without much hassle.

Now we have finished myproject and want to release it.  Unfortunately,
it makes heavy use of the DRM extension.  So we have to release the
drmified PCRE too (maybe also because of licensing requirements).
No worries!

We simply commit our myproject at myproject.org/reopsitory/myproject
along with PCRE v3.5-drm at myproject.org/repository/pcredrm.  But wait!
When another user comiles myproject, how is her OPKG going to find PCRE v3.5-drm?
Simple.  We have added a file _requirements in the myproject directory,
which simply states that:

   pcre-drm updates: http://myproject.org/repository/pcre-drm

This file implicitly adds a repository source (some kind of
/apt/sources.list) to the user's ~/.opkg/resolve file.  Of course,
this line does not affect other projects, so if you override an
important library, it will only affect the program you are currently
compiling.

Later, when the DRM patch is integrated into PCRE v3.6 proper, changing
the requirements or resolution file will let you map PCRE v3.5-drm
onto PCRE v3.6.

The requirements resolution mechanism is incomplete, inconsistent and needs
refinements.  But my aim here was to give a general feeling of what
I have in mind.

One open question:

Do we want a mechanism for having the complete dependency graph,
instead of discovering it on-the-fly, in the process of checking out
and building discovered dependencies?

Dependencies may be dynamic and depend on particular
configuration options selected by the user (i.e. --with-gtk implies
Gtk).

Formalizing the dependency information will need to take configuration
into account.  This means a Gentoo-like system - however that still might
not be powerful enough...
--
Berke Durak


-- 
Berke DURAK