From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=disabled version=3.1.3 X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from mail1-relais-roc.national.inria.fr (mail1-relais-roc.national.inria.fr [192.134.164.82]) by yquem.inria.fr (Postfix) with ESMTP id 601EFBC6C for ; Wed, 30 Jan 2008 13:37:44 +0100 (CET) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgAAAIbrn0fBL1AZh2dsb2JhbACQJwEBAQgKKYEUni0 X-IronPort-AV: E=Sophos;i="4.25,277,1199660400"; d="scan'208";a="7427500" Received: from concorde.inria.fr ([192.93.2.39]) by mail1-smtp-roc.national.inria.fr with ESMTP; 30 Jan 2008 13:37:44 +0100 Received: from mail1-relais-roc.national.inria.fr (mail1-relais-roc.national.inria.fr [192.134.164.82]) by concorde.inria.fr (8.13.6/8.13.6) with ESMTP id m0UCbfEs027233 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=OK) for ; Wed, 30 Jan 2008 13:37:44 +0100 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgAAAIbrn0fBL1AZh2dsb2JhbACQJwEBAQgKKYEUni0 X-IronPort-AV: E=Sophos;i="4.25,277,1199660400"; d="scan'208";a="7427499" Received: from gw.exalead.com (HELO exalead.com) ([193.47.80.25]) by mail1-smtp-roc.national.inria.fr with ESMTP; 30 Jan 2008 13:37:41 +0100 Received: from [192.168.204.148] (madpc064.exalead.com [192.168.204.148]) (authenticated bits=0) by exalead.com (8.14.0/8.14.0) with ESMTP id m0UCbdGq017538 for ; Wed, 30 Jan 2008 13:37:39 +0100 Message-ID: <47A06F93.8060006@exalead.com> Date: Wed, 30 Jan 2008 13:37:39 +0100 From: Berke Durak User-Agent: Thunderbird 1.5.0.10 (X11/20070221) MIME-Version: 1.0 To: Caml-list List Subject: Re: [Caml-list] [OSR] Ports-like package management system References: <479F0664.2070706@exalead.com> In-Reply-To: <479F0664.2070706@exalead.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Miltered: at concorde with ID 47A06F95.000 by Joe's j-chkmail (http://j-chkmail . ensmp . fr)! X-Spam: no; 0.00; berke:01 durak:01 berke:01 durak:01 mkdir:01 foo:01 printf:01 printf:01 foo:01 regexp:01 regexp:01 lib:01 lib:01 pcre:01 pcre:01 Hello all, Here is the kind of thing I would like to be able to do with an integrated build/packaging/VCS system. I think this explains why I want to base the system on a VCS. We have a collection of software components C1, C2, ... A version control system VCS is selected and agreed upon. Each usable version of a software component Ci is stored under a VCS repository Si. A single command, say OPKG, does all. (I think we'll have OPKG = ocamlbuild, but it could be OMake or something else). % cd src % mkdir myproject % OPKG init # Initialize the project % cat >foo.ml Printf.printf "Hello, world!\n" ^D % OPKG build foo.native (Compiling using ocamlbuild) % ./foo.native Hello, world! Now let's say we want to use a regexp library. Assume there exist multiple implementations with the same API, say OSRStandardRegexpLibraryAPI, that we want to use: % cat >_tags : require(some_regexp_lib_with_osr_standard_regexp_library_api) As pointed by others, it is a good idea to decouple the build rules from the used versions or the repository URLs. A file, _requirements, maps require strings (which are enclosed in a require(..) construct in the _tags file) to actual requirements. % cat >_requirements some_regexp_lib_with_osr_standard_regexp_library_api: latest(pcre) In a globally accessible place, there is a file that defines how to resolve component names to actual repository URLs. For instance, in $HOME/.opkg/resolve: % cat ~/.opkg/resolve pcre latest: v3.4 pcre updates: http://ocaml-pcre.org/opkg/updates pcre v3.4: http://ocaml-pcre.org/repo/ rev r381 pcre v3.3: http://ocaml-pcre.org/repo/ rev r341 pcre v3.2: http://ocaml-pcre.org/repo/ rev v3.2 netclient latest: v9.99 netclient updates: http://ocaml-pcre.org/opkg/updates netclient v9.99: http://netclient.com/repo/ rev r364 netclient v9.98: http://netclient.com/repo/ rev r300 netclient v9.97: http://netclient.com/repo/ rev r200 This resolution file can be overridden by creating a file _override_resolve in the project directory. We then start using Regexps in our foo.ml: % cat foo.ml open OSR.Regexp let _ = Printf.printf "Hello, %s!\n" (if regexp_match ~rex:"[Rr]obert" Sys.argv.(1) then "Bob" else Sys.argv.(1)) % OPKG build foo.native OPKG: inside ~/src/myproject OPKG: building map OPKG: map: foo.native needs some_regexp_lib_with_osr_standard_regexp_library_api OPKG: map: some_regexp_lib_with_osr_standard_regexp_library_api configured here as latest(pcre) from _requirements OPKG: map: latest(pcre) maps to pcre v3.4 from ~/.opkg/map OPKG has here created a file _map that maps requirements to actual versions. % cat _map some_regexp_lib_with_osr_standard_regexp_library_api: pcre v3.4 OPKG continues building the target: OPKG: done building map OPKG: pcre v3.4 not already checked out OPKG: pcre v3.4 available from hg http://ocaml-pcre.org/ rev r381 OPKG: checking out pcre v3.4 inside ~/src/opkg/pcre-v3.4 (path configured from from ~/.opkg.config) OPKG: building pcre.native v3.4 in ~/src/opkg/pcre-v3.4 OPKG: continuing compilation of foo.native Object files for pcre-v3.4 would reside in ~/src/opkg/pcre-v3.4/_build/ No need to build the bytecode version unless explicitly needed. % OPKG build foo.byte OPKG: inside ~/src/myproject ... OPKG: building pcre.byte v3.4 in ~/src/opkg/pcre/v3.4 OPKG: continuing compilation of foo.byte Similarly, if we want to build a multithreaded version, OPKG will automatically build multithreaded versions of all required libraries, using the tag mechanism. % OPKG build foo.byte.mt OPKG: building pcre.byte.mt v3.4 in ~/src/opkg/pcre/v3.4 Now assume we take a vacation and when we come back, a new version of PCRE is released, say v3.5. Of course we need to ask OPKG to update its resolution file. We can invoke it anywhere. % OPKG update OPKG: fetching http://ocaml-pcre.org/opkg/updates OPKG: new pcre version v3.5 (also the latest) OPKG: fetching http://netclient.com/opkg/updates OPKG: netclient still at v9.99 OPKG: updated ~/.opkg/resolve If we want to use the newest PCRE in our project, we need to update ~/src/myproject/_map, since it is this file which defines the exact version to be used. We can edit that file by hand; but we can also ask OPKG to regenerate it from the _requirements file: % OPKG remap OPKG: inside ~/src/myproject OPKG: building map OPKG: map: foo.native needs some_regexp_lib_with_osr_standard_regexp_library_api OPKG: map: some_regexp_lib_with_osr_standard_regexp_library_api configured here as latest(pcre) from _requirements OPKG: map: latest(pcre) maps to pcre v3.5 from ~/.opkg/map OPKG: done building map We can check which versions have been used in building a target with the command OPKG composition: % OPKG composition foo.native foo.native: made from map(0f9e8b9d8bbdae7bd9e78ab9d8e7ade83)(OUTDATED) map(0f9e8b9d8bbdae7bd9e78ab9d8e7ade83): ocaml 3.10.1(nativedynlinkpatch), PCRE v3.4(native) Here OPKG tells us that the current foo.native was built using the a map file whose digest was 0f9e8b9d8bbdae7bd9e78ab9d8e7ade83 which instructed OPKG to use ocaml 3.10.1 configured with the nativedynlinkpatch option, and with PCRE v3.5, in native code. OUTDATED means that the map has changed. Now let's rebuild our project: % OPKG build foo.native OPKG: inside ~/src/myproject OPKG: map 0f9e8b9d8bbdae7bd9e78ab9d8e7ade83 outdated by current map d78b279f78b3fb804983481834 OPKG: pcre v3.5 not already checked out OPKG: pcre v3.5 available from hg http://ocaml-pcre.org/ rev r450 ... And OPKG will fetch pcre v3.5, and rebuild it. % OPKG composition foo.native foo.native: made from map(d78b279f78b3fb804983481834)(CURRENT) map(d78b279f78b3fb804983481834): ocaml 3.10.1(nativedynlinkpatch), PCRE v3.5(native) What happens if we want to make an improvement to PCRE ? For instance, we might want to add a ?drm option to the match_regexp function. We simply go into ~/src/opkg/pcre and clone the v3.5 directory into v3.5-drm: % cd ~/src/opkg/pcre VCS clone v3.5 v3.5-drm We then start hacking into v3.5-drm. Of course we want to use myproject to debug it, so we override its map file by adding the following line into _map_override: % cat _map_override some_regexp_lib_with_osr_standard_regexp_library_api: pcre v3.5-drm Meanwhile, an important buffer overflow is fixed in PCRE. As we have been hacking the DRM extension under VCS, we can easily merge the patch without much hassle. Now we have finished myproject and want to release it. Unfortunately, it makes heavy use of the DRM extension. So we have to release the drmified PCRE too (maybe also because of licensing requirements). No worries! We simply commit our myproject at myproject.org/reopsitory/myproject along with PCRE v3.5-drm at myproject.org/repository/pcredrm. But wait! When another user comiles myproject, how is her OPKG going to find PCRE v3.5-drm? Simple. We have added a file _requirements in the myproject directory, which simply states that: pcre-drm updates: http://myproject.org/repository/pcre-drm This file implicitly adds a repository source (some kind of /apt/sources.list) to the user's ~/.opkg/resolve file. Of course, this line does not affect other projects, so if you override an important library, it will only affect the program you are currently compiling. Later, when the DRM patch is integrated into PCRE v3.6 proper, changing the requirements or resolution file will let you map PCRE v3.5-drm onto PCRE v3.6. The requirements resolution mechanism is incomplete, inconsistent and needs refinements. But my aim here was to give a general feeling of what I have in mind. One open question: Do we want a mechanism for having the complete dependency graph, instead of discovering it on-the-fly, in the process of checking out and building discovered dependencies? Dependencies may be dynamic and depend on particular configuration options selected by the user (i.e. --with-gtk implies Gtk). Formalizing the dependency information will need to take configuration into account. This means a Gentoo-like system - however that still might not be powerful enough... -- Berke Durak -- Berke DURAK