[OSR] Ports-like package management system

[OSR] Ports-like package management system

[Berke Durak]

Hello,

Following Markus's message, discussions with Nicolas Pouillard and
Sylvain Le Gall and others, and of course my previous work in the EDOS
project, here are some thoughts about package management systems
(PMSs) for Ocaml.

First, the PMSs of Linux distributions are perfectly adequate for end
users and administrators having to deploy and manage identical
packages accross multiple machines.  These are very complex systems
including a substantial social part.

But they are not very suitable for developers.

As Markus pointed out, developers cannot go thru a packaging step to
test the result of a change, much less wait for the fine Debian team.

One reason is that most developers, including myself, cannot be
bothered to package software for a Linux distribution, because correct
packaging is complex and requires adherence to a set of rules must be
remembered and which cannot all be checked by software.  As packaging
is not done very often, you tend to forget those rules, and that is
why we have people who package often to not forget the rules:
maintainers.  Without maintainers, we would be in a world of pain to
install any piece of non-trivial software and we are thankful to them.

But developers absolutely need to be able to work on multiple versions
of the same software component at once, patch those versions or
compile them with unusual flags, and often use the absolutely latest
unpackaged version.  And that is the second reason why the Debian or
Red Hat PMSs are not adequate.  They have a single global state per
system, which includes the installed files, and the package database,
and cannot handle multiple versions of the same package, nor multiple
compilations of the same version.  (This also applies to Gentoo.)

We thus need versions, and lots of them!  We need to base our
developer packages on a version control system, in the style of BSD
ports.  BSD ports are usually based on CVS, sometimes on Subversion.
As we are looking to increase collaboration, having a single point of
contention is a serious limitations of these centralized systems;
we'll prefer more recent "distributed" version control system.

Of available distributed VCSs with a serious user base, we have Darcs,
Mercurial and Git.

Basing a PMS for Ocaml on a VCS written in Haskell would violate the
``Trading with the Enemy'' act.  Moreover Darcs has some performance
problems of its own.

Mercurial (Hg) is written in Python extended in C for performance.  It
is quite friendly and works well under Windows.  However, its
developers are not as elite as Git's, its merging features are less
advanced and Python sux0rz.

I have been using Hg for the past few months and been quite happy with
it, but then I was mostly working alone.  Git is certainly as good as
Mercurial for that kind of usage and, as it is written in pure C, I
advocate its use.  Until someone writes a VCS in Ocaml, that is.

Let's get back to the subject.  BSD ports are also based on make,
whose main limitation, the static dependency graph, has been addressed
in ocamlbuild.  I know there is Omake, but I think it suffers from the
``Yet Another Turing-Complete Language'' syndrome.

So I am calling for a solution based on a ports-like system but based
on a distributed VCS and on an improved ocamlbuild.

Assume you are writing a program FOO and want to use a package BAR
available from bar.org.  You tell ocamlbuild by adding some tag such
as

   <mytarget.native>: require(http://bar.org/repository/)

And when you run ocamlbuild, it automatically checks out a copy of
BAR, compiles and loads its myocamlbuild.ml module which adds the
required flags.  Of course it should be possible to specify a
particular revision...  And if BAR has itself dependencies, those too
would be checked out.

Note that Git has a nice option for cloning checked out repositories
using hard links; that could be used to maintain a cache of checkouts,
for instance in the user's ~/.ocamlbuild/checkouts/ directory.

So basically I propose that we improve ocamlbuild to allow for
multiple plugin files (using dynamic loading) and use that to define a
BSD ports-like system targeted at developers.

-- 
Berke DURAK

Re: [Caml-list] [OSR] Ports-like package management system

[David Teller]

I have created a Wiki for this discussion at
http://cocan.org/osr/ports-like_management_system 

I'll try and follow the discussion and update the page periodically. In
the meantime, Berke, please feel free to update your summary and the
"list of questions".


On Tue, 2008-01-29 at 11:56 +0100, Berke Durak wrote:

> So basically I propose that we improve ocamlbuild to allow for
> multiple plugin files (using dynamic loading) and use that to define a
> BSD ports-like system targeted at developers.
> 
-- 
David Teller
 Security of Distributed Systems
  http://www.univ-orleans.fr/lifo/Members/David.Teller
 Angry researcher: French Universities need reforms, but the LRU act
brings liquidations. 

Re: [Caml-list] [OSR] Ports-like package management system

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 2508 bytes --]

On Jan 29, 2008 5:56 AM, Berke Durak <berke.durak@exalead.com> wrote:

> Basing a PMS for Ocaml on a VCS written in Haskell would violate the
> ``Trading with the Enemy'' act.  Moreover Darcs has some performance
> problems of its own.


Come now, Haskell is a dear friend and relative, not an enemy at all.

Besides, darcs has some key advantages for this kind of use.  Cherry-picking
and flexible maintenance of patches on top of someone else's tree would be
very valuable for this kind of application, and neither hg nor git support
that use case well.  And I believe the darcs team is making real advances
towards fixing these problems.

If not darcs, I would choose hg next.  hg supports windows well, which is a
big deal, I think.  Its user interface was more pleasant than git's last I
checked.  And it has some support for renames (not as good as darc's or
bzr's, but still good.)  We've used hg very intensively at Jane Street and
have been very happy with the results.


> Let's get back to the subject.  BSD ports are also based on make,
> whose main limitation, the static dependency graph, has been addressed
> in ocamlbuild.  I know there is Omake, but I think it suffers from the
> ``Yet Another Turing-Complete Language'' syndrome.
>

Does anyone with experience with both omake and ocamlbuild have an opinion
on the matter?  I've used omake quite a bit, and ocamlbuild not at all.   In
my mind, omake has the advantage that I'm pretty sure it's up to the task.
ocamlbuild has the advantage of being in the standard distribution and
having OCaml as its extension language.  It would be great to get the
opinion of someone who knows both systems well.


> So I am calling for a solution based on a ports-like system but based
> on a distributed VCS and on an improved ocamlbuild.
>
> Assume you are writing a program FOO and want to use a package BAR
> available from bar.org.  You tell ocamlbuild by adding some tag such
> as
>
>   <mytarget.native>: require(http://bar.org/repository/)<http://caml.inria.fr/bin/caml-bugs>


It would also be nice to have a set of versions of the various libraries
that hang together, as GODI does.  Otherwise, problems in the case where
there are packages A, B and C where A depends on B and C and B depends on
C.  You need a version of C that works with your versions of A and B, or
you're sunk.  So some central repo where you can maintain a set of "safe"
versions would allow for a developer to ask for a easily pull a collection
of working libraries.

y

[-- Attachment #2: Type: text/html, Size: 3446 bytes --]

Re: [Caml-list] [OSR] Ports-like package management system

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 3037 bytes --]

Excerpts from Yaron Minsky's message of Tue Jan 29 14:11:55 +0100 2008:
> On Jan 29, 2008 5:56 AM, Berke Durak <berke.durak@exalead.com> wrote:
> 
> > Basing a PMS for Ocaml on a VCS written in Haskell would violate the
> > ``Trading with the Enemy'' act.  Moreover Darcs has some performance
> > problems of its own.
> 
> 
> Come now, Haskell is a dear friend and relative, not an enemy at all.

Exact, however I think that was Berke's humor :)

> Besides, darcs has some key advantages for this kind of use.  Cherry-picking
> and flexible maintenance of patches on top of someone else's tree would be
> very valuable for this kind of application, and neither hg nor git support
> that use case well.  And I believe the darcs team is making real advances
> towards fixing these problems.

That's fully right, darcs2 solve a lot of his formers issues.

> If not darcs, I would choose hg next.  hg supports windows well, which is a
> big deal, I think.  Its user interface was more pleasant than git's last I
> checked.  And it has some support for renames (not as good as darc's or
> bzr's, but still good.)  We've used hg very intensively at Jane Street and
> have been very happy with the results.

I'm largely in favor of darcs.

> > Let's get back to the subject.  BSD ports are also based on make,
> > whose main limitation, the static dependency graph, has been addressed
> > in ocamlbuild.  I know there is Omake, but I think it suffers from the
> > ``Yet Another Turing-Complete Language'' syndrome.
> >
> 
> Does anyone with experience with both omake and ocamlbuild have an opinion
> on the matter?  I've used omake quite a bit, and ocamlbuild not at all.   In
> my mind, omake has the advantage that I'm pretty sure it's up to the task.
> ocamlbuild has the advantage of being in the standard distribution and
> having OCaml as its extension language.  It would be great to get the
> opinion of someone who knows both systems well.

I think that dynamic dependency graph of ocamlbuild could really help.
 
> > So I am calling for a solution based on a ports-like system but based
> > on a distributed VCS and on an improved ocamlbuild.
> >
> > Assume you are writing a program FOO and want to use a package BAR
> > available from bar.org.  You tell ocamlbuild by adding some tag such
> > as
> >
> >   <mytarget.native>: require(http://bar.org/repository/)<http://caml.inria.fr/bin/caml-bugs>
> 
> 
> It would also be nice to have a set of versions of the various libraries
> that hang together, as GODI does.  Otherwise, problems in the case where
> there are packages A, B and C where A depends on B and C and B depends on
> C.  You need a version of C that works with your versions of A and B, or
> you're sunk.  So some central repo where you can maintain a set of "safe"
> versions would allow for a developer to ask for a easily pull a collection
> of working libraries.

Yes,  we  have  to  think  about a clean interface to specify versions without
fall into a too much complex system.

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: [Caml-list] [OSR] Ports-like package management system

[Berke Durak]

Yaron Minsky a écrit :
> On Jan 29, 2008 5:56 AM, Berke Durak <berke.durak@exalead.com 
> <mailto:berke.durak@exalead.com>> wrote:
> 
>     Basing a PMS for Ocaml on a VCS written in Haskell would violate the
>     ``Trading with the Enemy'' act.  Moreover Darcs has some performance
>     problems of its own.
> 
> 
> Come now, Haskell is a dear friend and relative, not an enemy at all.

No but seriously, what would we look like if the semi-official 
distribution depends on Haskell?

> Besides, darcs has some key advantages for this kind of use.  
> Cherry-picking and flexible maintenance of patches on top of someone 
> else's tree would be very valuable for this kind of application, and 
> neither hg nor git support that use case well.  And I believe the darcs 
> team is making real advances towards fixing these problems.

I'll rank required VCS features :

* Distributed nature (for easy collaboration)
* Portability (Windows)
* Speed (for fast checkouts)
* Reliability
* User community
* Ease of use (not a big concern since the VCS commands would be called
by the distribution management code anyway)
* Advanced merging features

> If not darcs, I would choose hg next.  hg supports windows well, which 
> is a big deal, I think.  Its user interface was more pleasant than git's 
> last I checked.  And it has some support for renames (not as good as 
> darc's or bzr's, but still good.)  We've used hg very intensively at 
> Jane Street and have been very happy with the results. 

I've been using hg daily for about a year now and I am quite pleased 
with it.  My only reservation is that it is written in Python... 
However, I did a sloccount on Git today and I got:

ansic:        70503 (48.98%)
sh:           36153 (25.12%)
perl:         17731 (12.32%)
tcl:          14805 (10.29%)
python:        2839 (1.97%)
lisp:          1682 (1.17%)
asm:            220 (0.15%)

whereas the same on hg gives:

python:       14610 (95.29%)
ansic:          722 (4.71%)

Mercurial is obviously cleaner and more portable.  It runs well under 
Windows and has only three small files written in .c ; git is an unholy 
mixture of mostly C and shell (the Perl is in the plugins) and it has a 
bad reputation under Windows (not very surprising...).

However Darcs really has some show-stopping performance problems. 
Proponents of Darcs can say that it'll get really better with Darcs 2.0 
or whatever but, in practice, full repository checkouts over the network 
are unacceptably slow, and this is probably the most important thing for 
the application we are envisioning: a ports-like system.

I checked out Darcs itself and it took 15 minutes; while checking out 
Git or Mercurial took only one or two minutes.  The slowness of Darcs is 
well-known, many people are talking about it.

It seems that the latest repository format doesn't help much:

http://lists.osuosl.org/pipermail/darcs-devel/2008-January/007057.html

So I'm leaning towards Hg (and my Aurochs & Jsure repositories are 
already under Hg), but could agree on Git.  Also note that Git stores
files as snapshots and not as a sequence of patches.

Git also has some nifty advanced features:

http://blog.moertel.com/articles/2007/12/10/how-i-stopped-missing-darcs-and-started-loving-git

Both Mercurial and Git are used by first-class open-source projects (the 
kernel, Mozilla ...).  Darcs has only GHC and Darcs (duh!).

Also Mercurial has been selected over Git by Sun for OpenSolaris:

   http://www.opensolaris.org/os/community/tools/scm/mercurial-eval.html

Darcs may be based on some abstract ideas having some sex-appeal to 
theoretical computer scientists, and Git may have some sex-appeal to C 
hackers.

But...

* In practice, Darcs, as is currently distributed, is really slow
* Darcs is written in Haskell which will give Haskellers ammunition in 
flame wars^W^W^W^W^W might be bad for Ocaml evangelism.
* Apparently Darcs had serious fundamental algorithmic problems due to 
exponential blowup with commuting patches, and some practical problems 
with repository corruption under concurrent access...
* Git is written mostly in C with lots of shell scripts, which doesn't 
seem very appropriate for such an application in 2008; it has 
portability problems under Windows and seems to require Cygwin.
* The advanced features of Darcs & Git (cherry-picking, bisection, 
stash...) may be all fine and dandy but we don't need them for Ports. 
Heck, we could almost use Subversion...

Hence I agree with Yaron and recommend Mercurial.

> It would also be nice to have a set of versions of the various libraries 
> that hang together, as GODI does.  Otherwise, problems in the case where 
> there are packages A, B and C where A depends on B and C and B depends 
> on C.  You need a version of C that works with your versions of A and B, 
> or you're sunk.  So some central repo where you can maintain a set of 
> "safe" versions would allow for a developer to ask for a easily pull a 
> collection of working libraries.

I recommend the use of a VCS precisely for that kind of thing:  to be 
able to depend on an exact revision.

-- 
Berke DURAK

Re: [Caml-list] [OSR] Ports-like package management system

[Bünzli Daniel]

Le 29 janv. 08 à 18:35, Berke Durak a écrit :

>> It would also be nice to have a set of versions of the various  
>> libraries that hang together, as GODI does.  Otherwise, problems in  
>> the case where there are packages A, B and C where A depends on B  
>> and C and B depends on C.  You need a version of C that works with  
>> your versions of A and B, or you're sunk.  So some central repo  
>> where you can maintain a set of "safe" versions would allow for a  
>> developer to ask for a easily pull a collection of working libraries.
>
> I recommend the use of a VCS precisely for that kind of thing:  to  
> be able to depend on an exact revision.

This won't solve the problem per se.

If you have :

A-v1 depends on C-v1
B-v1 depends on C-v2

You need a way to be able to tell A-v1 to use C-v2 (or B-v1 to use  C- 
v1). So that you can use both A-v1 and B-v1 toghether. In a  
distributed setting you need a way to override the dependencies and  
pray that it works and if it doesn't contact the A's developer to  
release a new version that works with C-v2.

Again this has nothing to do with the use of a vcs.

Daniel

Re: [Caml-list] [OSR] Ports-like package management system

[Paul Pelzl]

On Tue, Jan 29, 2008 at 06:35:08PM +0100, Berke Durak wrote:
> I'll rank required VCS features :
> 
> * Distributed nature (for easy collaboration)
> * Portability (Windows)
> * Speed (for fast checkouts)
> * Reliability
> * User community
> * Ease of use (not a big concern since the VCS commands would be called
> by the distribution management code anyway)
> * Advanced merging features
> 
> >If not darcs, I would choose hg next.  hg supports windows well, which 
> >is a big deal, I think.  Its user interface was more pleasant than git's 
> >last I checked.  And it has some support for renames (not as good as 
> >darc's or bzr's, but still good.)  We've used hg very intensively at 
> >Jane Street and have been very happy with the results. 
> 
> I've been using hg daily for about a year now and I am quite pleased 
> with it.  My only reservation is that it is written in Python... 
> However, I did a sloccount on Git today and I got:
> 
> ansic:        70503 (48.98%)
> sh:           36153 (25.12%)
> perl:         17731 (12.32%)
> tcl:          14805 (10.29%)
> python:        2839 (1.97%)
> lisp:          1682 (1.17%)
> asm:            220 (0.15%)
> 
> whereas the same on hg gives:
> 
> python:       14610 (95.29%)
> ansic:          722 (4.71%)
> 
> Mercurial is obviously cleaner and more portable.  It runs well under 
> Windows and has only three small files written in .c ; git is an unholy 
> mixture of mostly C and shell (the Perl is in the plugins) and it has a 
> bad reputation under Windows (not very surprising...).

If distributed version control is to be considered as a solution to the
"package management" problem, then any unbiased evaluation of DVCS
candidates must surely include Bazaar (bzr): http://bazaar-vcs.org/

Architecturally, the design is very similar to Mercurial (anyone who
likes hg will probably like bzr, and vice versa).  However, bzr also has
best-in-class support for history-preserving file and directory renames,
provides excellent built-in merging capabilities, and is flexible enough
to support more workflow models than any other tool I have used.  In
addition, at the time of my last side-by-side comparison bzr had
significantly better Windows support than Mercurial.

bzr has recently reached its 1.0 milestone, and is now quite fast and
suitable for use on very large trees (not yet quite as fast as Mercurial
for most operations, but close).  It is a very well-supported product,
with a core developer team primarily employed by Canonical (of Ubuntu
fame).

Paul

Re: [Caml-list] [OSR] Ports-like package management system

[Paolo Donadeo]

Just my two cents about the VCS topic, coming from my long experience
in using such softwares, also in very critical environments (I was the
"librarian" in a military project, with about 100 developers and
managers over 3 countries).


> I checked out Darcs itself and it took 15 minutes; while checking out
> Git or Mercurial took only one or two minutes.  The slowness of Darcs is
> well-known, many people are talking about it.

The last time I used Darcs in a distributed environment it was so
buggy and slow that I replaced it with tla/bazaar. Darcs took 48 hours
to merge two branches of 2Mb of sources each (I'm not kidding!) and I
decided to to throw it out. It's *AWFUL*.


> Both Mercurial and Git are used by first-class open-source projects (the
> kernel, Mozilla ...).  Darcs has only GHC and Darcs (duh!).

I dare ask why... ;-)


Now I'm using GIT, both in my personal projects (distributed
environment, but only three developers and a simple situation, without
deadlines) and for my job (distributed env., but many developers and a
complex situation, and customers deadlines every 15 days).

GIT pros:
 + speed: it's the fastest VCS I never used;
 + standard: I think GIT will become the new de-facto standard in the
next 12-24 months;
 + documentation: GIT has a *really* good and complete documentation;
 + simple: it is very simple compared with anything else, but this is
of course a personal opinion;
 + most important: it's rock solid, no doubt at all on this point.
Bugs are corrected in hours, sometimes by Torvalds.

GIT cons:
 + does not work properly under Windows, at this moment. The Cygwin
package works well (I personally use it), but AFAIK a native Windows
port is not available. This may be a problem.

Hg should be ok, but I never used it. It's similar to GIT and runs
under Windows.

And I consider any centralized VCS (SVN, CVS) as dinosaurs completely
useless nowadays.


-- 
                                            Paolo


Paolo Donadeo, Senior Software Engineer
Studio Associato 4Sigma
Email: p.donadeo@4sigma.it
~
~
:wq

Re: [Caml-list] [OSR] Ports-like package management system

[Bünzli Daniel]

This discussion is turning into useless vcs advocacy.

The funny thing is that we (or at least I) don't have  a clue how the  
system is going to work and yet we are already discussing  
implementations details, namely choosing a vcs that I don't know for  
what it will be used for.

Convince me first that a vcs is needed and then we can talk about  
which one. The OP should have make an architectural proposal, but  
there is no such thing in his message.

It is not because there is now a cute [OSR] tag to put on discussions,  
that they will automatically become interesting. Some quality in the  
formulation of proposals is needed.

Best,

Daniel

Re: [Caml-list] [OSR] Ports-like package management system

[Paolo Donadeo]

Just my two cents about the VCS topic, coming from my long experience
in using such softwares, also in very critical environments (I was the
"librarian" in a military project, with about 100 developers and
managers over 3 countries).


> I checked out Darcs itself and it took 15 minutes; while checking out
> Git or Mercurial took only one or two minutes.  The slowness of Darcs is
> well-known, many people are talking about it.

The last time I used Darcs in a distributed environment it was so
buggy and slow that I replaced it with tla/bazaar. Darcs took 48 hours
to merge two branches of 2Mb of sources each (I'm not kidding!) and I
decided to to throw it out. It's *AWFUL*.


> Both Mercurial and Git are used by first-class open-source projects (the
> kernel, Mozilla ...).  Darcs has only GHC and Darcs (duh!).

I dare ask why... ;-)


Now I'm using GIT, both in my personal projects (distributed
environment, but only three developers and a simple situation, without
deadlines) and for my job (distributed env., but many developers and a
complex situation, and customers deadlines every 15 days).

GIT pros:
 + speed: it's the fastest VCS I never used;
 + standard: I think GIT will become the new de-facto standard in the
next 12-24 months;
 + documentation: GIT has a *really* good and complete documentation;
 + simple: it is very simple compared with anything else, but this is
of course a personal opinion;
 + most important: it's rock solid, no doubt at all on this point.
Bugs are corrected in hours, sometimes by Torvalds.

GIT cons:
 + does not work properly under Windows, at this moment. The Cygwin
package works well (I personally use it), but AFAIK a native Windows
port is not available. This may be a problem.

Hg should be ok, but I never used it. It's similar to GIT and runs
under Windows.

And I consider any centralized VCS (SVN, CVS) as dinosaurs completely
useless nowadays.


-- 
                                            Paolo


Paolo Donadeo, Senior Software Engineer
Studio Associato 4Sigma
Email: p.donadeo@4sigma.it
~
~
:wq

Re: [Caml-list] [OSR] Ports-like package management system

[Yaron Minsky]

Another thought: the discussion here is essentially about cloning  
haskell's cabal system. Seems like a good place to look for inspiration.

Yaron Minsky

On Jan 29, 2008, at 5:56 AM, Berke Durak <berke.durak@exalead.com>  
wrote:

> Hello,
>
> Following Markus's message, discussions with Nicolas Pouillard and
> Sylvain Le Gall and others, and of course my previous work in the EDOS
> project, here are some thoughts about package management systems
> (PMSs) for Ocaml.
>
> First, the PMSs of Linux distributions are perfectly adequate for end
> users and administrators having to deploy and manage identical
> packages accross multiple machines.  These are very complex systems
> including a substantial social part.
>
> But they are not very suitable for developers.
>
> As Markus pointed out, developers cannot go thru a packaging step to
> test the result of a change, much less wait for the fine Debian team.
>
> One reason is that most developers, including myself, cannot be
> bothered to package software for a Linux distribution, because correct
> packaging is complex and requires adherence to a set of rules must be
> remembered and which cannot all be checked by software.  As packaging
> is not done very often, you tend to forget those rules, and that is
> why we have people who package often to not forget the rules:
> maintainers.  Without maintainers, we would be in a world of pain to
> install any piece of non-trivial software and we are thankful to them.
>
> But developers absolutely need to be able to work on multiple versions
> of the same software component at once, patch those versions or
> compile them with unusual flags, and often use the absolutely latest
> unpackaged version.  And that is the second reason why the Debian or
> Red Hat PMSs are not adequate.  They have a single global state per
> system, which includes the installed files, and the package database,
> and cannot handle multiple versions of the same package, nor multiple
> compilations of the same version.  (This also applies to Gentoo.)
>
> We thus need versions, and lots of them!  We need to base our
> developer packages on a version control system, in the style of BSD
> ports.  BSD ports are usually based on CVS, sometimes on Subversion.
> As we are looking to increase collaboration, having a single point of
> contention is a serious limitations of these centralized systems;
> we'll prefer more recent "distributed" version control system.
>
> Of available distributed VCSs with a serious user base, we have Darcs,
> Mercurial and Git.
>
> Basing a PMS for Ocaml on a VCS written in Haskell would violate the
> ``Trading with the Enemy'' act.  Moreover Darcs has some performance
> problems of its own.
>
> Mercurial (Hg) is written in Python extended in C for performance.  It
> is quite friendly and works well under Windows.  However, its
> developers are not as elite as Git's, its merging features are less
> advanced and Python sux0rz.
>
> I have been using Hg for the past few months and been quite happy with
> it, but then I was mostly working alone.  Git is certainly as good as
> Mercurial for that kind of usage and, as it is written in pure C, I
> advocate its use.  Until someone writes a VCS in Ocaml, that is.
>
> Let's get back to the subject.  BSD ports are also based on make,
> whose main limitation, the static dependency graph, has been addressed
> in ocamlbuild.  I know there is Omake, but I think it suffers from the
> ``Yet Another Turing-Complete Language'' syndrome.
>
> So I am calling for a solution based on a ports-like system but based
> on a distributed VCS and on an improved ocamlbuild.
>
> Assume you are writing a program FOO and want to use a package BAR
> available from bar.org.  You tell ocamlbuild by adding some tag such
> as
>
>  <mytarget.native>: require(http://bar.org/repository/)
>
> And when you run ocamlbuild, it automatically checks out a copy of
> BAR, compiles and loads its myocamlbuild.ml module which adds the
> required flags.  Of course it should be possible to specify a
> particular revision...  And if BAR has itself dependencies, those too
> would be checked out.
>
> Note that Git has a nice option for cloning checked out repositories
> using hard links; that could be used to maintain a cache of checkouts,
> for instance in the user's ~/.ocamlbuild/checkouts/ directory.
>
> So basically I propose that we improve ocamlbuild to allow for
> multiple plugin files (using dynamic loading) and use that to define a
> BSD ports-like system targeted at developers.
>
> -- 
> Berke DURAK
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] [OSR] Ports-like package management system

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 4934 bytes --]

Excerpts from Yaron Minsky's message of Tue Jan 29 14:47:03 +0100 2008:
> Another thought: the discussion here is essentially about cloning  
> haskell's cabal system. Seems like a good place to look for inspiration.

I  also  think  that  many  ideas  can be borrowed from cabal since it's quite
recent and have close requirements.

> On Jan 29, 2008, at 5:56 AM, Berke Durak <berke.durak@exalead.com>  
> wrote:
> 
> > Hello,
> >
> > Following Markus's message, discussions with Nicolas Pouillard and
> > Sylvain Le Gall and others, and of course my previous work in the EDOS
> > project, here are some thoughts about package management systems
> > (PMSs) for Ocaml.
> >
> > First, the PMSs of Linux distributions are perfectly adequate for end
> > users and administrators having to deploy and manage identical
> > packages accross multiple machines.  These are very complex systems
> > including a substantial social part.
> >
> > But they are not very suitable for developers.
> >
> > As Markus pointed out, developers cannot go thru a packaging step to
> > test the result of a change, much less wait for the fine Debian team.
> >
> > One reason is that most developers, including myself, cannot be
> > bothered to package software for a Linux distribution, because correct
> > packaging is complex and requires adherence to a set of rules must be
> > remembered and which cannot all be checked by software.  As packaging
> > is not done very often, you tend to forget those rules, and that is
> > why we have people who package often to not forget the rules:
> > maintainers.  Without maintainers, we would be in a world of pain to
> > install any piece of non-trivial software and we are thankful to them.
> >
> > But developers absolutely need to be able to work on multiple versions
> > of the same software component at once, patch those versions or
> > compile them with unusual flags, and often use the absolutely latest
> > unpackaged version.  And that is the second reason why the Debian or
> > Red Hat PMSs are not adequate.  They have a single global state per
> > system, which includes the installed files, and the package database,
> > and cannot handle multiple versions of the same package, nor multiple
> > compilations of the same version.  (This also applies to Gentoo.)
> >
> > We thus need versions, and lots of them!  We need to base our
> > developer packages on a version control system, in the style of BSD
> > ports.  BSD ports are usually based on CVS, sometimes on Subversion.
> > As we are looking to increase collaboration, having a single point of
> > contention is a serious limitations of these centralized systems;
> > we'll prefer more recent "distributed" version control system.
> >
> > Of available distributed VCSs with a serious user base, we have Darcs,
> > Mercurial and Git.
> >
> > Basing a PMS for Ocaml on a VCS written in Haskell would violate the
> > ``Trading with the Enemy'' act.  Moreover Darcs has some performance
> > problems of its own.
> >
> > Mercurial (Hg) is written in Python extended in C for performance.  It
> > is quite friendly and works well under Windows.  However, its
> > developers are not as elite as Git's, its merging features are less
> > advanced and Python sux0rz.
> >
> > I have been using Hg for the past few months and been quite happy with
> > it, but then I was mostly working alone.  Git is certainly as good as
> > Mercurial for that kind of usage and, as it is written in pure C, I
> > advocate its use.  Until someone writes a VCS in Ocaml, that is.
> >
> > Let's get back to the subject.  BSD ports are also based on make,
> > whose main limitation, the static dependency graph, has been addressed
> > in ocamlbuild.  I know there is Omake, but I think it suffers from the
> > ``Yet Another Turing-Complete Language'' syndrome.
> >
> > So I am calling for a solution based on a ports-like system but based
> > on a distributed VCS and on an improved ocamlbuild.
> >
> > Assume you are writing a program FOO and want to use a package BAR
> > available from bar.org.  You tell ocamlbuild by adding some tag such
> > as
> >
> >  <mytarget.native>: require(http://bar.org/repository/)
> >
> > And when you run ocamlbuild, it automatically checks out a copy of
> > BAR, compiles and loads its myocamlbuild.ml module which adds the
> > required flags.  Of course it should be possible to specify a
> > particular revision...  And if BAR has itself dependencies, those too
> > would be checked out.
> >
> > Note that Git has a nice option for cloning checked out repositories
> > using hard links; that could be used to maintain a cache of checkouts,
> > for instance in the user's ~/.ocamlbuild/checkouts/ directory.
> >
> > So basically I propose that we improve ocamlbuild to allow for
> > multiple plugin files (using dynamic loading) and use that to define a
> > BSD ports-like system targeted at developers.

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: [Caml-list] [OSR] Ports-like package management system

[Alain Frisch]

Hello,

It's great to see some discussion and thoughts about package management 
for OCaml software. Of course, various kind of users have different 
expectations. I'm quite happy with GODI as a package developer or 
maintainer or when I need to write small applications that uses external 
OCaml libraries. But I'd like to describe our current approach here at 
LexiFi concerning the use of external libraries. This is the point of 
view a company with a few "big" projects, several developers, and 
several source languages. I guess that many companies that use OCaml 
would face a similar situation.

We develop end-user applications and our main target and development 
platform is Windows (using the native Win32 and Mingw ports of OCaml), 
although we also compile everything under Linux. This unfortunately 
rules out any existing package management system for OCaml. We use a 
number of third-party libraries in OCaml, C, Fortran that come with 
source code and some .Net components that come in binary form.

We don't want to depend on too many external packages to be downloaded 
in order to build our projects. Currently, we only rely on a few tools 
like MS Visual Studio, omake, a Subversion client to be installed on the 
development machines. Everything else is considered part of our project 
tree and kept under the control of Subversion. In particular, we have 
our locally-patched version of OCaml and all the third-party libraries 
(either in source or binary form) in the repository. (Note that our 
membership to the Caml Consortium even allow use to redistribute 
directly this modified version of OCaml.)

Putting everything under the control of SVN has several advantages:

- the developers don't need to do anything special on their computer 
when a new library is installed or updated; we know they all have a 
uniform environment;

- it is trivial to add local patches;

- we have control over the precise version of the libraries we use (we 
cannot afford to depend on any external decision to upgrade a library);

- in particular, various development branches can have different 
versions, and it is trivial to go back in time and build an old version 
with exactly the same libraries as it used to be.

The drawback, of course, is that we need to take care manually of 
upgrading to new versions of external libraries. The OCaml libraries we 
use are quite stable and we rarely find a serious problem that cannot be 
resolved by a small local patch. The only project that we follow 
regularly is OCaml itself. We have a vendor branch in our SVN on which 
we simply copy a current version from INRIA's public CVS; then we merge 
the diffs from this vendor branch into our project.

Another point worth mentioning is that all our project tree, except the 
OCaml part, is managed as a single OMake project. Creating OMakefiles 
for external OCaml/C/Fortan libraries is usually very easy (something 
like 2-3 lines for each library). It is quite comfortable to be able to 
patch, add debug statements, etc in 3rd party components and see the 
changes quickly (without recompiling everything). Another motivation for 
writing our own OMakefiles is that the Makefiles provided in the 
upstream packages are often broken under Windows (which is not 
surprising given how hard it is to deal with Windows path names within a 
Makefile).

We don't use findlib but have recreated a very minimal version of it 
within omake. Although we haven't done it yet, I can imagine something 
more serious: omake functions to read findlib's META files (or something 
similar, maybe in OMake syntax so that they can be simply include'd) so 
that various parts of the project can declare in a simple way which 
OCaml libraries they use and that correct flags can be automatically 
computed and passed to the compilers.

We also use omake to do some local configuration (finding where some 
external tools are, deciding what to compile or not depending on the 
host system, ...), manage local installation, prepare directory 
structures to help installer builders and help manage the list of files 
that should be distributed. We still have some bash/Perl scripts around 
(using Cygwin under Windows), but we should be able to get rid of them 
quite easily (omake has its own scripting language). I believe that 
ocamlbuild is really a great tool but in its current form, it could not 
replace our use of omake. Since we need something like omake anyway, we 
don't see a compelling reason to switch to ocamlbuild and to write 
plugins to compile C#/Fortan/...

We are quite satisfied with the current situation. Things could be 
streamlined if upstream packages came with OMakefiles that could be 
integrated directly in such configurations (that would require some 
conventions to be created and followed). It is really a minor point for 
us, but it could allow to transpose our approach to smaller projects 
where writing OMakefiles by hand is not desirable. Also, I can imagine 
that some scripts could automate the task of downloading, integrating 
and tracking external libraries in a SVN repository, maybe with a nice 
GUI and a central list of  packages (a la GODI). Note that almost 
nothing here is really OCaml-specific (except maybe some omake functions 
/ conventions to emulate findlib).

-- Alain

Re: [Caml-list] [OSR] Ports-like package management system

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 6381 bytes --]

Jane Street has a setup quite similar to the one Alain describes.  The
differences are:

   - We use mercurial instead of SVN
   - We do most of our building on Linux rather than Windows.
   - We're less far along the path of bringing external libraries into
   our tree, and so we still use external package management with GODI and
   findlib for many of our libraries.  We are moving to a model closer to
   LexiFi's, though.

We have a very similar-sounding omake setup, and we have the same feelings
about the fact that ocamlbuild is not a complete make replacement.

y

On Jan 29, 2008 11:00 AM, Alain Frisch <alain@frisch.fr> wrote:

> Hello,
>
> It's great to see some discussion and thoughts about package management
> for OCaml software. Of course, various kind of users have different
> expectations. I'm quite happy with GODI as a package developer or
> maintainer or when I need to write small applications that uses external
> OCaml libraries. But I'd like to describe our current approach here at
> LexiFi concerning the use of external libraries. This is the point of
> view a company with a few "big" projects, several developers, and
> several source languages. I guess that many companies that use OCaml
> would face a similar situation.
>
> We develop end-user applications and our main target and development
> platform is Windows (using the native Win32 and Mingw ports of OCaml),
> although we also compile everything under Linux. This unfortunately
> rules out any existing package management system for OCaml. We use a
> number of third-party libraries in OCaml, C, Fortran that come with
> source code and some .Net components that come in binary form.
>
> We don't want to depend on too many external packages to be downloaded
> in order to build our projects. Currently, we only rely on a few tools
> like MS Visual Studio, omake, a Subversion client to be installed on the
> development machines. Everything else is considered part of our project
> tree and kept under the control of Subversion. In particular, we have
> our locally-patched version of OCaml and all the third-party libraries
> (either in source or binary form) in the repository. (Note that our
> membership to the Caml Consortium even allow use to redistribute
> directly this modified version of OCaml.)
>
> Putting everything under the control of SVN has several advantages:
>
> - the developers don't need to do anything special on their computer
> when a new library is installed or updated; we know they all have a
> uniform environment;
>
> - it is trivial to add local patches;
>
> - we have control over the precise version of the libraries we use (we
> cannot afford to depend on any external decision to upgrade a library);
>
> - in particular, various development branches can have different
> versions, and it is trivial to go back in time and build an old version
> with exactly the same libraries as it used to be.
>
> The drawback, of course, is that we need to take care manually of
> upgrading to new versions of external libraries. The OCaml libraries we
> use are quite stable and we rarely find a serious problem that cannot be
> resolved by a small local patch. The only project that we follow
> regularly is OCaml itself. We have a vendor branch in our SVN on which
> we simply copy a current version from INRIA's public CVS; then we merge
> the diffs from this vendor branch into our project.
>
> Another point worth mentioning is that all our project tree, except the
> OCaml part, is managed as a single OMake project. Creating OMakefiles
> for external OCaml/C/Fortan libraries is usually very easy (something
> like 2-3 lines for each library). It is quite comfortable to be able to
> patch, add debug statements, etc in 3rd party components and see the
> changes quickly (without recompiling everything). Another motivation for
> writing our own OMakefiles is that the Makefiles provided in the
> upstream packages are often broken under Windows (which is not
> surprising given how hard it is to deal with Windows path names within a
> Makefile).
>
> We don't use findlib but have recreated a very minimal version of it
> within omake. Although we haven't done it yet, I can imagine something
> more serious: omake functions to read findlib's META files (or something
> similar, maybe in OMake syntax so that they can be simply include'd) so
> that various parts of the project can declare in a simple way which
> OCaml libraries they use and that correct flags can be automatically
> computed and passed to the compilers.
>
> We also use omake to do some local configuration (finding where some
> external tools are, deciding what to compile or not depending on the
> host system, ...), manage local installation, prepare directory
> structures to help installer builders and help manage the list of files
> that should be distributed. We still have some bash/Perl scripts around
> (using Cygwin under Windows), but we should be able to get rid of them
> quite easily (omake has its own scripting language). I believe that
> ocamlbuild is really a great tool but in its current form, it could not
> replace our use of omake. Since we need something like omake anyway, we
> don't see a compelling reason to switch to ocamlbuild and to write
> plugins to compile C#/Fortan/...
>
> We are quite satisfied with the current situation. Things could be
> streamlined if upstream packages came with OMakefiles that could be
> integrated directly in such configurations (that would require some
> conventions to be created and followed). It is really a minor point for
> us, but it could allow to transpose our approach to smaller projects
> where writing OMakefiles by hand is not desirable. Also, I can imagine
> that some scripts could automate the task of downloading, integrating
> and tracking external libraries in a SVN repository, maybe with a nice
> GUI and a central list of  packages (a la GODI). Note that almost
> nothing here is really OCaml-specific (except maybe some omake functions
> / conventions to emulate findlib).
>
> -- Alain
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 7214 bytes --]

Re: [Caml-list] [OSR] Ports-like package management system

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 7376 bytes --]

Excerpts from Yaron Minsky's message of Wed Jan 30 07:58:15 +0100 2008:
> Jane Street has a setup quite similar to the one Alain describes.  The
> differences are:
> 
>    - We use mercurial instead of SVN
>    - We do most of our building on Linux rather than Windows.
>    - We're less far along the path of bringing external libraries into
>    our tree, and so we still use external package management with GODI and
>    findlib for many of our libraries.  We are moving to a model closer to
>    LexiFi's, though.
> 
> We have a very similar-sounding omake setup, and we have the same feelings
> about the fact that ocamlbuild is not a complete make replacement.

Conceptually  ocamlbuild engine is really a complete make replacement. However
in  surface,  its  quite incomparable since there is no specific language, but
an OCaml API and we write plain ocaml code.

Example of rule written in OCaml for ocamlbuild:

rule "LaTeX to PDF conversion rule"
  ~prod:"%.pdf"
  ~dep:"%.tex"
  begin fun env _build ->
    let tex = env "%.tex" in
    let tags = tags_of_pathname tex++"compile"++"LaTeX"++"pdf" in
    let cmd = Cmd(S[!pdflatex; T tags; P tex; Sh"< /dev/null"]) in
    (* Hoping that \LaTeX will converge in two iterations *)
    Seq[cmd; cmd]
  end;
flag ["compile"; "LaTeX"; "pdf"; "safe"] (A"-halt-on-error");

> On Jan 29, 2008 11:00 AM, Alain Frisch <alain@frisch.fr> wrote:
> 
> > Hello,
> >
> > It's great to see some discussion and thoughts about package management
> > for OCaml software. Of course, various kind of users have different
> > expectations. I'm quite happy with GODI as a package developer or
> > maintainer or when I need to write small applications that uses external
> > OCaml libraries. But I'd like to describe our current approach here at
> > LexiFi concerning the use of external libraries. This is the point of
> > view a company with a few "big" projects, several developers, and
> > several source languages. I guess that many companies that use OCaml
> > would face a similar situation.
> >
> > We develop end-user applications and our main target and development
> > platform is Windows (using the native Win32 and Mingw ports of OCaml),
> > although we also compile everything under Linux. This unfortunately
> > rules out any existing package management system for OCaml. We use a
> > number of third-party libraries in OCaml, C, Fortran that come with
> > source code and some .Net components that come in binary form.
> >
> > We don't want to depend on too many external packages to be downloaded
> > in order to build our projects. Currently, we only rely on a few tools
> > like MS Visual Studio, omake, a Subversion client to be installed on the
> > development machines. Everything else is considered part of our project
> > tree and kept under the control of Subversion. In particular, we have
> > our locally-patched version of OCaml and all the third-party libraries
> > (either in source or binary form) in the repository. (Note that our
> > membership to the Caml Consortium even allow use to redistribute
> > directly this modified version of OCaml.)
> >
> > Putting everything under the control of SVN has several advantages:
> >
> > - the developers don't need to do anything special on their computer
> > when a new library is installed or updated; we know they all have a
> > uniform environment;
> >
> > - it is trivial to add local patches;
> >
> > - we have control over the precise version of the libraries we use (we
> > cannot afford to depend on any external decision to upgrade a library);
> >
> > - in particular, various development branches can have different
> > versions, and it is trivial to go back in time and build an old version
> > with exactly the same libraries as it used to be.
> >
> > The drawback, of course, is that we need to take care manually of
> > upgrading to new versions of external libraries. The OCaml libraries we
> > use are quite stable and we rarely find a serious problem that cannot be
> > resolved by a small local patch. The only project that we follow
> > regularly is OCaml itself. We have a vendor branch in our SVN on which
> > we simply copy a current version from INRIA's public CVS; then we merge
> > the diffs from this vendor branch into our project.
> >
> > Another point worth mentioning is that all our project tree, except the
> > OCaml part, is managed as a single OMake project. Creating OMakefiles
> > for external OCaml/C/Fortan libraries is usually very easy (something
> > like 2-3 lines for each library). It is quite comfortable to be able to
> > patch, add debug statements, etc in 3rd party components and see the
> > changes quickly (without recompiling everything). Another motivation for
> > writing our own OMakefiles is that the Makefiles provided in the
> > upstream packages are often broken under Windows (which is not
> > surprising given how hard it is to deal with Windows path names within a
> > Makefile).
> >
> > We don't use findlib but have recreated a very minimal version of it
> > within omake. Although we haven't done it yet, I can imagine something
> > more serious: omake functions to read findlib's META files (or something
> > similar, maybe in OMake syntax so that they can be simply include'd) so
> > that various parts of the project can declare in a simple way which
> > OCaml libraries they use and that correct flags can be automatically
> > computed and passed to the compilers.
> >
> > We also use omake to do some local configuration (finding where some
> > external tools are, deciding what to compile or not depending on the
> > host system, ...), manage local installation, prepare directory
> > structures to help installer builders and help manage the list of files
> > that should be distributed. We still have some bash/Perl scripts around
> > (using Cygwin under Windows), but we should be able to get rid of them
> > quite easily (omake has its own scripting language). I believe that
> > ocamlbuild is really a great tool but in its current form, it could not
> > replace our use of omake. Since we need something like omake anyway, we
> > don't see a compelling reason to switch to ocamlbuild and to write
> > plugins to compile C#/Fortan/...
> >
> > We are quite satisfied with the current situation. Things could be
> > streamlined if upstream packages came with OMakefiles that could be
> > integrated directly in such configurations (that would require some
> > conventions to be created and followed). It is really a minor point for
> > us, but it could allow to transpose our approach to smaller projects
> > where writing OMakefiles by hand is not desirable. Also, I can imagine
> > that some scripts could automate the task of downloading, integrating
> > and tracking external libraries in a SVN repository, maybe with a nice
> > GUI and a central list of  packages (a la GODI). Note that almost
> > nothing here is really OCaml-specific (except maybe some omake functions
> > / conventions to emulate findlib).
> >
> > -- Alain
> >
> > _______________________________________________
> > Caml-list mailing list. Subscription management:
> > http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> > Archives: http://caml.inria.fr
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
> >

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: [Caml-list] [OSR] Ports-like package management system

[Bünzli Daniel]

Le 29 janv. 08 à 11:56, Berke Durak a écrit :

> We thus need versions, and lots of them!  We need to base our
> developer packages on a version control system, in the style of BSD
> ports.  BSD ports are usually based on CVS, sometimes on Subversion.

My experience with bsd-like port systems (at least darwinports) is  
that _port description files_ are versioned. But the bits they compile  
are tarball releases, they do not pull directly out of the projects'  
vcs to install your local copy.

I would be all in favor of using somehting like darcs2 because it is  
what I use. However I don't think it is a good idea to impose a vcs to  
developers, they should be free to use their own. I don't understand  
why you don't want to rely on tarballs, one per version, with a  
facility to quickly create and upload them.

> As we are looking to increase collaboration, having a single point of
> contention is a serious limitations of these centralized systems;
> we'll prefer more recent "distributed" version control system.

You are right everyone should be able to publish its package on its  
own server and mirrors (it also solves the problem of who is going to  
pay for the infrastructure). But this has nothing to do with  
distributed vcs.

> Assume you are writing a program FOO and want to use a package BAR
> available from bar.org.  You tell ocamlbuild by adding some tag such
> as
>
>  <mytarget.native>: require(http://bar.org/repository/)

This is good, FOO's dependencies are explicit. However I'm not sure a  
direct uri is a good thing, you'll need something to be able to  
specify alternate download sites. The package name + version  
specification should be decoupled from the uri you use to access it.  
Even if you do that nothing prevents you to implement this over plain  
http. E.g. you have download locations uri1 uri2, and a package-name  
and version-name then you try to download from uri1/package-name/ 
version-name and then if it fails uri2/... Of course in that case  
you'll need digests.

> Of course it should be possible to specify a particular revision...

For me this is too fine grained -- and this is also the reason why you  
want to integrate a vcs to your system. I would like to be able to  
specify a version that the developer of the package deemed stable  
enough to distribute, not a random revision. I strongly think that  
tarball releases are enough, if there are simple and efficient tools  
to produce them. Going down to the revision is overkill.

>   I know there is Omake,

Having used ocamlbuild for caml projects, for me it is out of question  
to return to something make-like.


Le 29 janv. 08 à 14:11, Yaron Minsky a écrit :

> It would also be nice to have a set of versions of the various  
> libraries that hang together, as GODI does.  Otherwise, problems in  
> the case where there are packages A, B and C where A depends on B  
> and C and B depends on C.  You need a version of C that works with  
> your versions of A and B, or you're sunk.  So some central repo  
> where you can maintain a set of "safe" versions would allow for a  
> developer to ask for a easily pull a collection of working libraries.

These are the kind problem you get when you go distributed. For  
emergencies you need at least an override mechanism (use this version  
instead of that one to build all packages). The rest should be solved  
by cooperation between developers. But nothing prevents someone to  
construct a cathedral, godi-like, collection of packages well tested  
to work toghether and that you will use as your only source. It is  
easy to build a centralized system on a distributed one, but the  
converse is not true.

At the risk of repeating myself I'll point again to my partial mod  
proposal [1], some of the ideas there could be recast more thightly  
with ocamlbuild as is  propose above. This would certainly also  
simplify the proposal (which was supposed to be a tool really  
separated from ocamlbuild).

Best,

Daniel

[1] http://erratique.ch/writings/mod.pdf

Re: [Caml-list] [OSR] Ports-like package management system

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 1761 bytes --]

Excerpts from daniel.buenzli's message of Tue Jan 29 18:56:37 +0100 2008:
> 
> Le 29 janv. 08 à 11:56, Berke Durak a écrit :
> 
> > We thus need versions, and lots of them!  We need to base our
> > developer packages on a version control system, in the style of BSD
> > ports.  BSD ports are usually based on CVS, sometimes on Subversion.
> 
> My experience with bsd-like port systems (at least darwinports) is  
> that _port description files_ are versioned. But the bits they compile  
> are tarball releases, they do not pull directly out of the projects'  
> vcs to install your local copy.

I  think  this  was  what  Berke  has in mind to. However the repository still
becomes very large even if there only a few files by package.

[...]

> For me this is too fine grained -- and this is also the reason why you  
> want to integrate a vcs to your system. I would like to be able to  
> specify a version that the developer of the package deemed stable  
> enough to distribute, not a random revision. I strongly think that  
> tarball releases are enough, if there are simple and efficient tools  
> to produce them. Going down to the revision is overkill.

Perhaps  not so overkill for developers, if you've just patched a project, you
need   to  update  the  package  quickly  and  perhaps  not  want  to  have  a
release/tarball  for  each  of  them.  I think that the upstream source can be
either  a tarball URL or a VCS URL. For upstream sources one can supports some
VCSs (CVS, SVN, darcs, git, hg) since one only need to checkout.

> >   I know there is Omake,
> 
> Having used ocamlbuild for caml projects, for me it is out of question  
> to return to something make-like.

Of course, I agree also on this :)

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: [Caml-list] [OSR] Ports-like package management system

[Bünzli Daniel]

Le 29 janv. 08 à 19:17, Nicolas Pouillard a écrit :

> I  think  this  was  what  Berke  has in mind to. However the  
> repository still
> becomes very large even if there only a few files by package.

Then I don't understand anymore. So the system is still centralized in  
the sense that all port files are under the same vcs at some location.  
That is exactly the kind of thing I'd like to avoid.

If you don't have a centralized system, then managing your package's  
port file is at your discretion. The port file itself should describe  
the various versions it provides of the package, their dependencies  
and where you can find them.

>> For me this is too fine grained -- and this is also the reason why  
>> you
>> want to integrate a vcs to your system. I would like to be able to
>> specify a version that the developer of the package deemed stable
>> enough to distribute, not a random revision. I strongly think that
>> tarball releases are enough, if there are simple and efficient tools
>> to produce them. Going down to the revision is overkill.
>
> Perhaps  not so overkill for developers, if you've just patched a  
> project, you
> need   to  update  the  package  quickly  and  perhaps  not  want   
> to  have  a
> release/tarball  for  each  of  them.

Frankly this is not the average case, please try to solve the average  
case, not the baroque ones. If you need to follow a project at the  
patch level deal directly with the developer's repository. Browse the  
hump and try to think about which projects you'd follow at that level  
to use in your own projects - dealing with moving targets is annoying.

Besides having such a fine grain will bother both package users (up to  
which patch should I pull ?) and package developers (transient issues  
due to a user that pulled up to a given patch).

Another thing you may want to consider is that a "released" package  
may need less (build) dependencies than a direct pull from a  
repository (e.g. files generated using custom tools, etc.).

> I think that the upstream source can be
> either  a tarball URL or a VCS URL. For upstream sources one can  
> supports some
> VCSs (CVS, SVN, darcs, git, hg) since one only need to checkout.

A system can always be complexified. I'd rather have something simple  
that works. I strongly think the vcs should be kept outside the  
package management system.

Best,

Daniel

Re: [Caml-list] [OSR] Ports-like package management system

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 2705 bytes --]

Excerpts from daniel.buenzli's message of Tue Jan 29 20:13:40 +0100 2008:
> 
> Le 29 janv. 08 à 19:17, Nicolas Pouillard a écrit :
> 
> > I  think  this  was  what  Berke  has in mind to. However the  
> > repository still
> > becomes very large even if there only a few files by package.
> 
> Then I don't understand anymore. So the system is still centralized in  
> the sense that all port files are under the same vcs at some location.  
> That is exactly the kind of thing I'd like to avoid.
> 
> If you don't have a centralized system, then managing your package's  
> port file is at your discretion. The port file itself should describe  
> the various versions it provides of the package, their dependencies  
> and where you can find them.

You  have a local branch of the whole port hierarchy, that's why we're talking
about DVCS.

> >> For me this is too fine grained -- and this is also the reason why  
> >> you
> >> want to integrate a vcs to your system. I would like to be able to
> >> specify a version that the developer of the package deemed stable
> >> enough to distribute, not a random revision. I strongly think that
> >> tarball releases are enough, if there are simple and efficient tools
> >> to produce them. Going down to the revision is overkill.
> >
> > Perhaps  not so overkill for developers, if you've just patched a  
> > project, you
> > need   to  update  the  package  quickly  and  perhaps  not  want   
> > to  have  a
> > release/tarball  for  each  of  them.
> 
> Frankly this is not the average case, please try to solve the average  
> case, not the baroque ones. If you need to follow a project at the  
> patch level deal directly with the developer's repository. Browse the  
> hump and try to think about which projects you'd follow at that level  
> to use in your own projects - dealing with moving targets is annoying.
> 
> Besides having such a fine grain will bother both package users (up to  
> which patch should I pull ?) and package developers (transient issues  
> due to a user that pulled up to a given patch).

That's  not  a  baroque  case,  I  mean  if  you are responsible of libFoo and
progBar,  you  perhaps  want to quickly package progBar using the last version
of libFoo.

[...]

> > I think that the upstream source can be
> > either  a tarball URL or a VCS URL. For upstream sources one can  
> > supports some
> > VCSs (CVS, SVN, darcs, git, hg) since one only need to checkout.
> 
> A system can always be complexified. I'd rather have something simple  
> that works. I strongly think the vcs should be kept outside the  
> package management system.

I also would like to keep things simple...

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: [Caml-list] [OSR] Ports-like package management system

[Bünzli Daniel]

Le 30 janv. 08 à 09:49, Nicolas Pouillard a écrit :

> You  have a local branch of the whole port hierarchy, that's why  
> we're talking
> about DVCS.

Ok I understand now. So the port hierarchy is centralized and  
versioned that's exactly what I don't like because it is not flexible  
enough, you have to  wait for things to update (e.g. like waiting for  
the 3.10 branch in godi) because package description files support  
only one version.

I don't like the idea of one version per description file, it is not  
flexible enough and having the port hierarchy under vcs doesn't help  
to select a particular version of a port.

The way I see it is

- a package is a list of versions
- a version is a list of files that build themselves with ocamlbuild  
and a specification of dependencies.

Now the tool I seek allows me to specify package dependencies in  
ocamlbuild and take care of their install.

> That's  not  a  baroque  case,  I  mean  if  you are responsible of  
> libFoo and
> progBar,  you  perhaps  want to quickly package progBar using the  
> last version
> of libFoo.

If you don't have a centralized port hierarchy, you don't get these  
problems because you manage your own package description file. In that  
case just add a new version to your package description file and  
publish it. Decentralize the system let anyone publish their own  
package descriptions on their website, use the web.


Le 30 janv. 08 à 10:53, Sylvain Le Gall a écrit :

> Lets choose a very simple storage system. Not a VCS. This too heavy  
> and
> create too many complicated case.

Agreed.

Daniel

Re: [Caml-list] [OSR] Ports-like package management system

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 3446 bytes --]

Excerpts from daniel.buenzli's message of Wed Jan 30 12:15:01 +0100 2008:
> 
> Le 30 janv. 08 à 09:49, Nicolas Pouillard a écrit :
> 
> > You  have a local branch of the whole port hierarchy, that's why  
> > we're talking
> > about DVCS.
> 
> Ok I understand now. So the port hierarchy is centralized and  
> versioned that's exactly what I don't like because it is not flexible  
> enough, you have to  wait for things to update (e.g. like waiting for  
> the 3.10 branch in godi) because package description files support  
> only one version.

Not  necessarily,  in  Gentoo,  there  is  all  installable  versions  in  the
directory of the project.

Here is the OCaml package in Gentoo:

$ find /usr/portage/dev-lang/ocaml
/usr/portage/dev-lang/ocaml
/usr/portage/dev-lang/ocaml/Manifest
/usr/portage/dev-lang/ocaml/files
/usr/portage/dev-lang/ocaml/files/ocaml-rebuild.sh
/usr/portage/dev-lang/ocaml/files/ocaml-3.09.3-exec-stack-fixes.patch
/usr/portage/dev-lang/ocaml/files/ocaml-3.09.3-Makefile.patch
/usr/portage/dev-lang/ocaml/files/ocaml-3.09.3-configure.patch
/usr/portage/dev-lang/ocaml/files/ocaml-3.10.0-exec-stack-fixes.patch
/usr/portage/dev-lang/ocaml/files/ocaml-3.09.3-execheap.patch
/usr/portage/dev-lang/ocaml/files/ocaml-3.09.3-automagic.patch
/usr/portage/dev-lang/ocaml/files/ocaml-3.09.3-hppa-optimize-for-size-ocamlp4.patch
/usr/portage/dev-lang/ocaml/files/ocaml-3.09.3-call_ld_with_proper_flags.patch
/usr/portage/dev-lang/ocaml/files/ocaml-3.10.0-automagic.patch
/usr/portage/dev-lang/ocaml/files/ocaml-3.10.0-configure.patch
/usr/portage/dev-lang/ocaml/files/ocaml-3.10.0-call-ld-with-proper-ldflags.patch
/usr/portage/dev-lang/ocaml/ocaml-3.08.4.ebuild
/usr/portage/dev-lang/ocaml/ocaml-3.09.2.ebuild
/usr/portage/dev-lang/ocaml/ChangeLog
/usr/portage/dev-lang/ocaml/metadata.xml
/usr/portage/dev-lang/ocaml/ocaml-3.09.3-r1.ebuild
/usr/portage/dev-lang/ocaml/ocaml-3.09.3.ebuild
/usr/portage/dev-lang/ocaml/ocaml-3.10.0.ebuild
/usr/portage/dev-lang/ocaml/ocaml-3.09.3-r2.ebuild
/usr/portage/dev-lang/ocaml/ocaml-3.10.1.ebuild

You  can  see  that  there  is  multiple  versions,  but  not  necessarily all
versions. There regularly garbage-collect some versions.

> I don't like the idea of one version per description file, it is not  
> flexible enough and having the port hierarchy under vcs doesn't help  
> to select a particular version of a port.

As you can see, that's wrong in the case of portage.

> The way I see it is
> 
> - a package is a list of versions
> - a version is a list of files that build themselves with ocamlbuild  
> and a specification of dependencies.
> 
> Now the tool I seek allows me to specify package dependencies in  
> ocamlbuild and take care of their install.

Yes and such a list of package, can be hosted in a hierarchy in a VCS.

> > That's  not  a  baroque  case,  I  mean  if  you are responsible of  
> > libFoo and
> > progBar,  you  perhaps  want to quickly package progBar using the  
> > last version
> > of libFoo.
> 
> If you don't have a centralized port hierarchy, you don't get these  
> problems because you manage your own package description file. In that  
> case just add a new version to your package description file and  
> publish it. Decentralize the system let anyone publish their own  
> package descriptions on their website, use the web.

In this system anyone can publish it's variant of the ports hierarchy.

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: [Caml-list] [OSR] Ports-like package management system

[Hezekiah M. Carty]

On Jan 29, 2008 5:56 AM, Berke Durak <berke.durak@exalead.com> wrote:
> We thus need versions, and lots of them!  We need to base our
> developer packages on a version control system, in the style of BSD
> ports.  BSD ports are usually based on CVS, sometimes on Subversion.
> As we are looking to increase collaboration, having a single point of
> contention is a serious limitations of these centralized systems;
> we'll prefer more recent "distributed" version control system.
>
> Of available distributed VCSs with a serious user base, we have Darcs,
> Mercurial and Git.

What about using Subversion for the main/central repository?  I
realize that it is not distributed itself, but Git, SVK and bzr all
allow for "distributed" connectivity to Subversion repositories.
There may be similar tools/plugins available for Mercurial but I have
not used them.  Subversion has the cross-platform issue covered, with
many CLI and GUI tools available.  It is fairly standard at this
point, which should help to lower the entry barrier for any new
developers.

I have personally found SVK and git-svn to work very well with
Subversion repositories.  This allows for each individual developer to
use the tool(s) they prefer, while still connecting to a central,
canonical repository.

Hez

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 29-01-2008, Berke Durak <berke.durak@exalead.com> wrote:
> Hello,
>
> Following Markus's message, discussions with Nicolas Pouillard and
> Sylvain Le Gall and others, and of course my previous work in the EDOS
> project, here are some thoughts about package management systems
> (PMSs) for Ocaml.
>
> First, the PMSs of Linux distributions are perfectly adequate for end
> users and administrators having to deploy and manage identical
> packages accross multiple machines.  These are very complex systems
> including a substantial social part.
>
> But they are not very suitable for developers.
>
> As Markus pointed out, developers cannot go thru a packaging step to
> test the result of a change, much less wait for the fine Debian team.
>
> One reason is that most developers, including myself, cannot be
> bothered to package software for a Linux distribution, because correct
> packaging is complex and requires adherence to a set of rules must be
> remembered and which cannot all be checked by software.  As packaging
> is not done very often, you tend to forget those rules, and that is
> why we have people who package often to not forget the rules:
> maintainers.  Without maintainers, we would be in a world of pain to
> install any piece of non-trivial software and we are thankful to them.
>
> But developers absolutely need to be able to work on multiple versions
> of the same software component at once, patch those versions or
> compile them with unusual flags, and often use the absolutely latest
> unpackaged version.  And that is the second reason why the Debian or
> Red Hat PMSs are not adequate.  They have a single global state per
> system, which includes the installed files, and the package database,
> and cannot handle multiple versions of the same package, nor multiple
> compilations of the same version.  (This also applies to Gentoo.)
>
> We thus need versions, and lots of them!  We need to base our
> developer packages on a version control system, in the style of BSD
> ports.  BSD ports are usually based on CVS, sometimes on Subversion.
> As we are looking to increase collaboration, having a single point of
> contention is a serious limitations of these centralized systems;
> we'll prefer more recent "distributed" version control system.
>
> Of available distributed VCSs with a serious user base, we have Darcs,
> Mercurial and Git.
>

[...]

>
> So I am calling for a solution based on a ports-like system but based
> on a distributed VCS and on an improved ocamlbuild.

Please don't go into this. If you want to talk about PM, don't talk
about VCS. My point is that if you want to build something that last you
should keep focus on PM, which is not really bound to VCS (like content
of files is not bound to filesystem). There is no best VCS for doing PM.
We just should handle a way to :
* let anyone use a different VCS
* be able to download at least a version of each different packages

The most simple way to handle this, to my mind:
* distribute METADATA for packaging into ftp/http
* put a link to the VCS inside METADATA, to tell where you can find most
 recent one

This way, you will be able to download the repository of METADATA using
a simple curl/wget/any other client that allow to download things. You
will also be able to modify it by either checking it out from the VCS
(can be automated using the VCS metadata) or by producing patches and
sending it to the metadata author.

We, in Debian, are using this way. It works pretty well and don't let
people going into endless discussion about Git being faster than Darcs
-- but less useful than Hg.

> Assume you are writing a program FOO and want to use a package BAR
> available from bar.org.  You tell ocamlbuild by adding some tag such
> as
>
>    <mytarget.native>: require(http://bar.org/repository/)
>
> And when you run ocamlbuild, it automatically checks out a copy of
> BAR, compiles and loads its myocamlbuild.ml module which adds the
> required flags.  Of course it should be possible to specify a
> particular revision...  And if BAR has itself dependencies, those too
> would be checked out.
>

This part is really about PM... and i agree.

> Note that Git has a nice option for cloning checked out repositories
> using hard links; that could be used to maintain a cache of checkouts,
> for instance in the user's ~/.ocamlbuild/checkouts/ directory.
>


This part is going back to classical VCS flameware. Please avoid it.

> So basically I propose that we improve ocamlbuild to allow for
> multiple plugin files (using dynamic loading) and use that to define a
> BSD ports-like system targeted at developers.
>

Without the VCS part, i agree.

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Berke Durak]

Sylvain Le Gall a écrit :
> Please don't go into this. If you want to talk about PM, don't talk
> about VCS. My point is that if you want to build something that last you
> should keep focus on PM, which is not really bound to VCS (like content
> of files is not bound to filesystem). There is no best VCS for doing PM.
> We just should handle a way to :
> * let anyone use a different VCS
> * be able to download at least a version of each different packages

 > The most simple way to handle this, to my mind:
 > * distribute METADATA for packaging into ftp/http
 > * put a link to the VCS inside METADATA, to tell where you can find most
 >  recent one

We want to guarantee to the user that if she has installed Ocaml and the 
Ocaml package system, she will be able to use any component packaged in 
the system without having to install extra software.

The Ocaml pakcage system must therefore include as a prerequisite all 
the tools required for fetching data.  Hence if we want to use a VCS as 
part of the system, we must agree on one.

The VCS selected, if any, doesn't have to be the one the developers use, 
of course.  (But there exist many VCS-to-VCS bridges, so it's not really 
a problem.)

Speed and portability are however important.

Now why would we want to use a VCS?

* As a data storage and distribution mechanism
* For efficient updates
* For the ability to check out any revision

These functions could be emulated by having a directory with a lot of 
tarball snapshots and using wget.  This may work for passive use of the 
software but even for this limited use case, use of a VCS is much more 
useful - efficient updates, storage of all intermediate revisions, 
branches, and the possibility for the upstream author to directly work 
on that VCS.

But the aim of the Ocaml packaging system should be to foster 
cooperation among Ocaml developers.  That is why having a recommended 
VCS and a standardized build system (or at least set of build commands)
is important.

Assume we agree on a distributed VCS system S.  Most Ocaml software will 
be developed in its own S repository, hosted by the author.  So we will 
have programs P1, P2... with respective repositories S1, S2...

When you develop a program P you will have use programs P_{i1}, P_{i2}, 
... and thus have checkouts of repositories S_{i1}, S_{i2}, and so on. 
If you find a bug in P_{i1} you can directly edit the checked out 
version, recompile everything automatically by launching ocaml in P's 
directory and thus debug P_{i1}.  You can then locally commit your 
changes to P_{i1} and then easily push the patch or send the diff to the 
upstream author.

If we don't use a VCS, this becomes much more difficult and error-prone.

The increase in collaboration and productivity could be tremendous. 
This requires close integration of the build and repository system and 
agreement on a common VCS.

Now being able to use the software of your choice, plurality and 
multiplicity of solutions are all good, but in this particular case this 
hinders collaboration.

It's as if everyone was using a different language!

My intent was not to provoke VCS flame wars.  But I think agreeing on a 
VCS is important and a flame war or two is acceptable collateral damage 
in the process of selecting one.

Now of course I don't have a precise idea of what the Ocaml packaging 
system would be like, but I think the way to go is to :

   - Use a common distributed VCS system for storage and distribution of 
source code, with multiple repositories

   - A build system integrated with the package management system,

   - A package management system able to manage arbitrarily many 
versions or copies of the same software component, and compile any 
program using an arbitrary selection of the required versions,
-- 
Berke DURAK

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 30-01-2008, Berke Durak <berke.durak@exalead.com> wrote:
> Sylvain Le Gall a écrit :
>> Please don't go into this. If you want to talk about PM, don't talk
>> about VCS. My point is that if you want to build something that last you
>> should keep focus on PM, which is not really bound to VCS (like content
>> of files is not bound to filesystem). There is no best VCS for doing PM.
>> We just should handle a way to :
>> * let anyone use a different VCS
>> * be able to download at least a version of each different packages
>
> > The most simple way to handle this, to my mind:
> > * distribute METADATA for packaging into ftp/http
> > * put a link to the VCS inside METADATA, to tell where you can find most
> >  recent one
>
> We want to guarantee to the user that if she has installed Ocaml and the 
> Ocaml package system, she will be able to use any component packaged in 
> the system without having to install extra software.
>
> The Ocaml pakcage system must therefore include as a prerequisite all 
> the tools required for fetching data.  Hence if we want to use a VCS as 
> part of the system, we must agree on one.
>
> The VCS selected, if any, doesn't have to be the one the developers use, 
> of course.  (But there exist many VCS-to-VCS bridges, so it's not really 
> a problem.)
>
> Speed and portability are however important.
>

Using a simple wget/rsync (you can redevelop it in OCaml) is far more simple
than to use a VCS. 

VCS-to-VCS is a dream. If have tried to use many, most of the times they
are complicated to setup and don't work well.

And of course wget/rsync are the fastest way to get data. 

> Now why would we want to use a VCS?
>
> * As a data storage and distribution mechanism
> * For efficient updates
> * For the ability to check out any revision
>
> These functions could be emulated by having a directory with a lot of 
> tarball snapshots and using wget.  This may work for passive use of the 
> software but even for this limited use case, use of a VCS is much more 
> useful - efficient updates, storage of all intermediate revisions, 
> branches, and the possibility for the upstream author to directly work 
> on that VCS.

Use rsync!

>
> But the aim of the Ocaml packaging system should be to foster 
> cooperation among Ocaml developers.  That is why having a recommended 
> VCS and a standardized build system (or at least set of build commands)
> is important.
>
> Assume we agree on a distributed VCS system S.  Most Ocaml software will 
> be developed in its own S repository, hosted by the author.  So we will 
> have programs P1, P2... with respective repositories S1, S2...
>

Assume that Mister Z don't want to use VCS S. Mister Z create a lot of
great OCaml libraries... What do you do?

> When you develop a program P you will have use programs P_{i1}, P_{i2}, 
> ... and thus have checkouts of repositories S_{i1}, S_{i2}, and so on. 
> If you find a bug in P_{i1} you can directly edit the checked out 
> version, recompile everything automatically by launching ocaml in P's 
> directory and thus debug P_{i1}.  You can then locally commit your 
> changes to P_{i1} and then easily push the patch or send the diff to the 
> upstream author.
>

Send a patch to author of P_{i1}. This is the easiest way. 

<hint>
diff -Nurd  P_{i1} P_{i1}.new
</hint>

> If we don't use a VCS, this becomes much more difficult and error-prone.
>
> The increase in collaboration and productivity could be tremendous. 
> This requires close integration of the build and repository system and 
> agreement on a common VCS.
>

This also lead to be stuck with one VCS for the rest of our lives. It
also prevents people who don't like/don't have time to learn VCS S, to
contribute.

> Now being able to use the software of your choice, plurality and 
> multiplicity of solutions are all good, but in this particular case this 
> hinders collaboration.
>
> It's as if everyone was using a different language!
>
> My intent was not to provoke VCS flame wars.  But I think agreeing on a 
> VCS is important and a flame war or two is acceptable collateral damage 
> in the process of selecting one.
>

Looking at all the mail of this thread, i mainly read people talking
about VCS R being faster than VCS S...

> Now of course I don't have a precise idea of what the Ocaml packaging 
> system would be like, but I think the way to go is to :
>
>    - Use a common distributed VCS system for storage and distribution of 
> source code, with multiple repositories
>

Lets choose a very simple storage system. Not a VCS. This too heavy and
create too many complicated case.

>    - A build system integrated with the package management system,
>

Yes, if you do not force every software developers to use it and still
provide a way to integrate their software into your package management
system.

>    - A package management system able to manage arbitrarily many 
> versions or copies of the same software component, and compile any 
> program using an arbitrary selection of the required versions,

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

Excerpts from Sylvain Le Gall's message of Wed Jan 30 10:53:34 +0100 2008:
> On 30-01-2008, Berke Durak <berke.durak@exalead.com> wrote:
> > Sylvain Le Gall a écrit :
[...]
> > When you develop a program P you will have use programs P_{i1}, P_{i2}, 
> > ... and thus have checkouts of repositories S_{i1}, S_{i2}, and so on. 
> > If you find a bug in P_{i1} you can directly edit the checked out 
> > version, recompile everything automatically by launching ocaml in P's 
> > directory and thus debug P_{i1}.  You can then locally commit your 
> > changes to P_{i1} and then easily push the patch or send the diff to the 
> > upstream author.
> >
> 
> Send a patch to author of P_{i1}. This is the easiest way. 
> 
> <hint>
> diff -Nurd  P_{i1} P_{i1}.new
> </hint>

And  what  if the author of P_{i1} have changed it's project (locally or not),
let's  say  some  file  renamings,  adding  some  code around the bug, do some
global replacing...

With such a diff file you end-up with a patch that is not applicable.

That's not a problem when using darcs!

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Bünzli Daniel]

Le 30 janv. 08 à 11:50, Nicolas Pouillard a écrit :

> And  what  if the author of P_{i1} have changed it's project  
> (locally or not),
> let's  say  some  file  renamings,  adding  some  code around the  
> bug, do some
> global replacing...
>
> With such a diff file you end-up with a patch that is not applicable.
>
> That's not a problem when using darcs!

The port system should focus on making it easy to use and share ocaml  
modules, not helping to develop them. If you want to help in the  
development of a module interact directly with the project's vcs.

Daniel

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

Excerpts from daniel.buenzli's message of Wed Jan 30 12:15:03 +0100 2008:
> 
> Le 30 janv. 08 à 11:50, Nicolas Pouillard a écrit :
> 
> > And  what  if the author of P_{i1} have changed it's project  
> > (locally or not),
> > let's  say  some  file  renamings,  adding  some  code around the  
> > bug, do some
> > global replacing...
> >
> > With such a diff file you end-up with a patch that is not applicable.
> >
> > That's not a problem when using darcs!
> 
> The port system should focus on making it easy to use and share ocaml  
> modules, not helping to develop them. If you want to help in the  
> development of a module interact directly with the project's vcs.

Of  course  you want to contribute to upstream. But I don't want to wait them,
so we need to modify them in a coherent way.

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 30-01-2008, Nicolas Pouillard <nicolas.pouillard@gmail.com> wrote:
>
> --===============1976516885==
> Content-Type: multipart/signed; protocol="application/pgp-signature";
> 	boundary="=-1201694054-349540-27517-1921-14-="; micalg="pgp-sha1"
>
>
> --=-1201694054-349540-27517-1921-14-=
> Content-Type: text/plain; charset=UTF-8
>
> Excerpts from daniel.buenzli's message of Wed Jan 30 12:15:03 +0100 2008:
>> 
>> Le 30 janv. 08 à 11:50, Nicolas Pouillard a écrit :
>> 
>> > And  what  if the author of P_{i1} have changed it's project  
>> > (locally or not),
>> > let's  say  some  file  renamings,  adding  some  code around the  
>> > bug, do some
>> > global replacing...
>> >
>> > With such a diff file you end-up with a patch that is not applicable.
>> >
>> > That's not a problem when using darcs!
>> 
>> The port system should focus on making it easy to use and share ocaml  
>> modules, not helping to develop them. If you want to help in the  
>> development of a module interact directly with the project's vcs.
>
> Of  course  you want to contribute to upstream. But I don't want to wait them,
> so we need to modify them in a coherent way.
>

Upstream should be the synchronization point. Because, if you make it
move (by sending him patches), the benefit of your patches will also go
into other distribution (GODI/Fedora/Debian). Debian try to push patches
to upstream because it makes other get the benefit from it...

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 1743 bytes --]

Excerpts from Sylvain Le Gall's message of Wed Jan 30 14:58:21 +0100 2008:
> On 30-01-2008, Nicolas Pouillard <nicolas.pouillard@gmail.com> wrote:
> >
> > --===============1976516885==
> > Content-Type: multipart/signed; protocol="application/pgp-signature";
> >     boundary="=-1201694054-349540-27517-1921-14-="; micalg="pgp-sha1"
> >
> >
> > --=-1201694054-349540-27517-1921-14-=
> > Content-Type: text/plain; charset=UTF-8
> >
> > Excerpts from daniel.buenzli's message of Wed Jan 30 12:15:03 +0100 2008:
> >> 
> >> Le 30 janv. 08 à 11:50, Nicolas Pouillard a écrit :
> >> 
> >> > And  what  if the author of P_{i1} have changed it's project  
> >> > (locally or not),
> >> > let's  say  some  file  renamings,  adding  some  code around the  
> >> > bug, do some
> >> > global replacing...
> >> >
> >> > With such a diff file you end-up with a patch that is not applicable.
> >> >
> >> > That's not a problem when using darcs!
> >> 
> >> The port system should focus on making it easy to use and share ocaml  
> >> modules, not helping to develop them. If you want to help in the  
> >> development of a module interact directly with the project's vcs.
> >
> > Of  course  you want to contribute to upstream. But I don't want to wait them,
> > so we need to modify them in a coherent way.
> >
> 
> Upstream should be the synchronization point. Because, if you make it
> move (by sending him patches), the benefit of your patches will also go
> into other distribution (GODI/Fedora/Debian). Debian try to push patches
> to upstream because it makes other get the benefit from it...

Of  course,  you send the patch upstream, then you don't want to wait for them
to answer in order to continue your task.

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Berke Durak]

Sylvain Le Gall a écrit :
> On 30-01-2008, Berke Durak <berke.durak@exalead.com> wrote:
>> When you develop a program P you will have use programs P_{i1}, P_{i2}, 
>> ... and thus have checkouts of repositories S_{i1}, S_{i2}, and so on. 
>> If you find a bug in P_{i1} you can directly edit the checked out 
>> version, recompile everything automatically by launching ocaml in P's 
>> directory and thus debug P_{i1}.  You can then locally commit your 
>> changes to P_{i1} and then easily push the patch or send the diff to the 
>> upstream author.
>>
> 
> Send a patch to author of P_{i1}. This is the easiest way. 
> 
> <hint>
> diff -Nurd  P_{i1} P_{i1}.new
> </hint>

So you need to get a separate copy of P_{i1}, unpack it, and run a diff 
by hand...  And what will happen during the process of developing the 
patch?  You will be working on your patch without version control 
system.  Which is very uncomfortable, especially as you will be working 
on unfamiliar code - you will make mistakes, erase or modify important 
stuff, won't be able to easily rollback, branch or review your changes.
You will have to play around with tar, diff, patch and merge by hand, 
make mistakes and create a mess.  Any VCS can handle all these tasks easily.

Unless your correction is very minor, you will have to either (a) import 
the project into your own favorite VCS, losing history and create a 
short-lived repository or be forced to track all future changes by hand; 
or (b) checkout the sources using the VCS the authors use - but then 
we're back at case one: you will have to potentially have all the 
version control systems used by all the software components you may want 
to contribute to, and also learn how to use them all.

>>    - A build system integrated with the package management system,
>>
> 
> Yes, if you do not force every software developers to use it and still
> provide a way to integrate their software into your package management
> system.

We are not forcing anyone to work using the selected VCS.  We are just 
asking them to regularly publish their software to the VCS.  If they 
directly work using the selected VCS, publishing commits will be easier 
for them - but they can still work on a day-to-day basis using their own 
VCS.

-- 
Berke DURAK

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Bünzli Daniel]

Le 30 janv. 08 à 12:15, Berke Durak a écrit :

> We are not forcing anyone to work using the selected VCS.  We are  
> just asking them to regularly publish their software to the VCS.  If  
> they directly work using the selected VCS, publishing commits will  
> be easier for them - but they can still work on a day-to-day basis  
> using their own VCS.

This complexifies the publishing procedure. Please, keep simple things  
simple, lower the bar to module sharing do not raise it.

Best,

Daniel

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 30-01-2008, Bünzli Daniel <daniel.buenzli@erratique.ch> wrote:
>
> Le 30 janv. 08 à 12:15, Berke Durak a écrit :
>
>> We are not forcing anyone to work using the selected VCS.  We are  
>> just asking them to regularly publish their software to the VCS.  If  
>> they directly work using the selected VCS, publishing commits will  
>> be easier for them - but they can still work on a day-to-day basis  
>> using their own VCS.
>
> This complexifies the publishing procedure. Please, keep simple things  
> simple, lower the bar to module sharing do not raise it.
>

Totally agree!

Regards,
Sylvain Le Gall

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 30-01-2008, Berke Durak <berke.durak@exalead.com> wrote:
> Sylvain Le Gall a écrit :
>> On 30-01-2008, Berke Durak <berke.durak@exalead.com> wrote:
>>> When you develop a program P you will have use programs P_{i1}, P_{i2}, 
>>> ... and thus have checkouts of repositories S_{i1}, S_{i2}, and so on. 
>>> If you find a bug in P_{i1} you can directly edit the checked out 
>>> version, recompile everything automatically by launching ocaml in P's 
>>> directory and thus debug P_{i1}.  You can then locally commit your 
>>> changes to P_{i1} and then easily push the patch or send the diff to the 
>>> upstream author.
>>>
>> 
>> Send a patch to author of P_{i1}. This is the easiest way. 
>> 
>> <hint>
>> diff -Nurd  P_{i1} P_{i1}.new
>> </hint>
>
> So you need to get a separate copy of P_{i1}, unpack it, and run a diff 
> by hand...  And what will happen during the process of developing the 
> patch?  You will be working on your patch without version control 
> system.  Which is very uncomfortable, especially as you will be working 
> on unfamiliar code - you will make mistakes, erase or modify important 
> stuff, won't be able to easily rollback, branch or review your changes.
> You will have to play around with tar, diff, patch and merge by hand, 
> make mistakes and create a mess.  Any VCS can handle all these tasks easily.
>
> Unless your correction is very minor, you will have to either (a) import 
> the project into your own favorite VCS, losing history and create a 
> short-lived repository or be forced to track all future changes by hand; 
> or (b) checkout the sources using the VCS the authors use - but then 
> we're back at case one: you will have to potentially have all the 
> version control systems used by all the software components you may want 
> to contribute to, and also learn how to use them all.

You are mixing software development and packaging. If you really want to
have something clean, you should avoid having changes into the source of
the upstream author (at least changes that should last).

Lets take an example:
* you commit source of project C, version V1 into your VCS
* you do everything required packaging, including some change to project
 C to make it compile (changeset P1)
* upstream author publish project C, version V2 (changeset P2) between V1
 and V2
* the changeset between P1 and P2 cannot merge
...

So what help do you get from your VCS here? You will have anyway to
rework it.

Forcing a VCS is just a way to loose people who won't have access to
your VCS... I don't see a real advantage of doing it...

If people want to put sources of anyone under their own VCS, they can do
it. You can do even something very clever: put a metadata about upstream
VCS into the packaging metadata. This way you will be able to checkout
upstream author VCS directly and be able to send them patches with their
own VCS... Better than to have to do a patch or to convert darcs/hg/git
changeset to send it...

Example:

File METADATA.opkg:
Upstream-Email: moi@chezmoi.com
Upstream-VCS: svn+ssh://chezmoi.com/mygreat-project/tags/1.0.0

or 

Upstream-VCS: darcs://chezmoi.com/mygreat-project@version_1.0.0

or 

Upstream-VCS: git://chezmoi.com/mygreat-project@version_1.0.0

(Upstream-VCS will be 

Of course, the file METADATA.opkg can and all the other packaging info
can be stored in any VCS (svn for GODI today).

>>>    - A build system integrated with the package management system,
>>>
>> 
>> Yes, if you do not force every software developers to use it and still
>> provide a way to integrate their software into your package management
>> system.
>
> We are not forcing anyone to work using the selected VCS.  We are just 
> asking them to regularly publish their software to the VCS.  If they 
> directly work using the selected VCS, publishing commits will be easier 
> for them - but they can still work on a day-to-day basis using their own 
> VCS.
>

Do they have to merge their changes with packager changes? In this case,
i really do think that upstream author won't publish their changes a
lot.

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Berke Durak]

Sylvain Le Gall a écrit :
> You are mixing software development and packaging. If you really want to
> have something clean, you should avoid having changes into the source of
> the upstream author (at least changes that should last).

I want to combine development and packaging because packaging does not work
well for developers.

> Lets take an example:
> * you commit source of project C, version V1 into your VCS
> * you do everything required packaging, including some change to project
>  C to make it compile (changeset P1)
> * upstream author publish project C, version V2 (changeset P2) between V1
>  and V2
> * the changeset between P1 and P2 cannot merge
> ...
> 
> So what help do you get from your VCS here? You will have anyway to
> rework it.

The VCS manages changesets, assists you in merging, keeps a logged history,
allows you to rollback, and so on.  Quite useful.

> Forcing a VCS is just a way to loose people who won't have access to
> your VCS... I don't see a real advantage of doing it...

Using a VCS as an infrastructure is not the same thing as forcing developers
to use it.  (See below).

>> We are not forcing anyone to work using the selected VCS.  We are just 
>> asking them to regularly publish their software to the VCS.  If they 
>> directly work using the selected VCS, publishing commits will be easier 
>> for them - but they can still work on a day-to-day basis using their own 
>> VCS.
>>
> 
> Do they have to merge their changes with packager changes? In this case,
> i really do think that upstream author won't publish their changes a
> lot.

If they are using VCS2 for working, they just have to commit their changeset to
some VCS repository from time to time.  This repository can be hosted by anyone.
There won't be a mandatory central repository.

Example 1: Alphonse uses Darcs to develop Nicemodule.  He sets up a VCS repository
(say, Mercurial, if we agree on mercurial) repository on his server.  He adds the
required opkg files in his source tree.  Once in a while he commits Nicemodule to
his Mercurial repository, but keeps working on Darcs.  Bernard checks out the
  Mercurial and sends a patch P against the Mercurial version.

Example 2: Alphonse doesn't want to use opkg or mercurial, but is the author of
a really nice module Nicemodule. Bernard sets up a Mercurial repository where he imports
Alphonse's Nicemodule and adds opkg information. Opkg users can then use Nicemodule
by simply writing require(nicemodule).  Bernard tracks Alphonse's source
code and sends him back patches against his code.

Example 3: Alphonse finally sees that using the same VCS as everyone else has more
benefits than the extra niceness his preferred VCS has.  His changes will be visible
immediately in the Opkg system and he will be able to pull patches from his friends
easily.  He therefore uses Opkg and its associated VCS right from the start for
his new project.  When he has a new stable release, he simply tags it and adds a line
in the release file.  When Bernard starts developing a feature Y, he can directly pull
Alphonse's changes and push back bug fixes.
-- 
Berke DURAK

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 30-01-2008, Berke Durak <berke.durak@exalead.com> wrote:
> Sylvain Le Gall a écrit :
>>> We are not forcing anyone to work using the selected VCS.  We are just 
>>> asking them to regularly publish their software to the VCS.  If they 
>>> directly work using the selected VCS, publishing commits will be easier 
>>> for them - but they can still work on a day-to-day basis using their own 
>>> VCS.
>>>
>> 
>> Do they have to merge their changes with packager changes? In this case,
>> i really do think that upstream author won't publish their changes a
>> lot.
>
> If they are using VCS2 for working, they just have to commit their changeset to
> some VCS repository from time to time.  This repository can be hosted by anyone.
> There won't be a mandatory central repository.
>
> Example 1: Alphonse uses Darcs to develop Nicemodule.  He sets up a VCS repository
> (say, Mercurial, if we agree on mercurial) repository on his server.  He adds the
> required opkg files in his source tree.  Once in a while he commits Nicemodule to
> his Mercurial repository, but keeps working on Darcs.  Bernard checks out the
>   Mercurial and sends a patch P against the Mercurial version.
>

Not sure it will work in most case.

> Example 2: Alphonse doesn't want to use opkg or mercurial, but is the author of
> a really nice module Nicemodule. Bernard sets up a Mercurial repository where he imports
> Alphonse's Nicemodule and adds opkg information. Opkg users can then use Nicemodule
> by simply writing require(nicemodule).  Bernard tracks Alphonse's source
> code and sends him back patches against his code.
>

I think this should be an option. The only things is that Bernard must
continue to send patches to Alphonse...

> Example 3: Alphonse finally sees that using the same VCS as everyone else has more
> benefits than the extra niceness his preferred VCS has.  His changes will be visible
> immediately in the Opkg system and he will be able to pull patches from his friends
> easily.  He therefore uses Opkg and its associated VCS right from the start for
> his new project.  When he has a new stable release, he simply tags it and adds a line
> in the release file.  When Bernard starts developing a feature Y, he can directly pull
> Alphonse's changes and push back bug fixes.

Stop dreaming ;-) (i.e. Alphonse and Bernard keep talking about VCS,
Alphonse and Bernard decide to both start a new VCS from scratch, in
OCaml, nobody works anymore on Nicemodule).

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Bünzli Daniel]

Le 30 janv. 08 à 15:24, Berke Durak a écrit :

> I want to combine development and packaging because packaging does  
> not work
> well for developers.

I think this is only true if you have a monolithic port tree.  
Monolithic in the sense that the whole tree is realeased now and then  
and you have no choice and have to use it as is. If each individual  
port is allowed to have its own release pace, then packaging works  
perfectly for developers.

The scenarios you describe are much too complicated to me. Let's keep  
it the caml way, rough but efficient.

Best,

Daniel

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Vlad Skvortsov]

Sylvain Le Gall wrote:
>>>> When you develop a program P you will have use programs P_{i1}, P_{i2}, 
>>>> ... and thus have checkouts of repositories S_{i1}, S_{i2}, and so on. 
>>>> If you find a bug in P_{i1} you can directly edit the checked out 
>>>> version, recompile everything automatically by launching ocaml in P's 
>>>> directory and thus debug P_{i1}.  You can then locally commit your 
>>>> changes to P_{i1} and then easily push the patch or send the diff to the 
>>>> upstream author.
>>>>
>>>>         
>>> Send a patch to author of P_{i1}. This is the easiest way. 
>>>
>>> <hint>
>>> diff -Nurd  P_{i1} P_{i1}.new
>>> </hint>
>>>       
>> So you need to get a separate copy of P_{i1}, unpack it, and run a diff 
>> by hand...  And what will happen during the process of developing the 
>> patch?  You will be working on your patch without version control 
>> system.  Which is very uncomfortable, especially as you will be working 
>> on unfamiliar code - you will make mistakes, erase or modify important 
>> stuff, won't be able to easily rollback, branch or review your changes.
>> You will have to play around with tar, diff, patch and merge by hand, 
>> make mistakes and create a mess.  Any VCS can handle all these tasks easily.
>>
>> Unless your correction is very minor, you will have to either (a) import 
>> the project into your own favorite VCS, losing history and create a 
>> short-lived repository or be forced to track all future changes by hand; 
>> or (b) checkout the sources using the VCS the authors use - but then 
>> we're back at case one: you will have to potentially have all the 
>> version control systems used by all the software components you may want 
>> to contribute to, and also learn how to use them all.
>>     
>
> You are mixing software development and packaging. If you really want to
> have something clean, you should avoid having changes into the source of
> the upstream author (at least changes that should last).
>   

I agree with this. The lower the entry barrier for the system, the 
better. The simplier the system, the better.

I wouldn't use a system that forces me to install a new sexy VCS on N of 
my machines and learn a new workflow just to use some packaged module.

My strong preference is that any package should be available for 
download via HTTP/FTP. The metadata (central package registry) can be 
versioned or not, I don't care, as long as I can publish my package with 
a single command.

I really like the way it's done in Python, here is a link to my previous 
email on this topic:

http://www.nabble.com/Re:-Re:-On-module-distribution-p14852553.html

-- 
Vlad Skvortsov, vss@73rus.com, http://vss.73rus.com

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Michael Ekstrand]

Sylvain Le Gall <sylvain@le-gall.net> writes:
> Using a simple wget/rsync (you can redevelop it in OCaml) is far
> more simple than to use a VCS.

Hear, hear.

I think that it's somewhat strange to incorporate the source for all
packages into one massive VCS, and it looks like that's what's been
proposed.

Forgive me if I'm rehashing things that have already been brought up,
but I think that some kind of hybrid thing is good, and that Sylvain
has some excellent points.  For the general case of package
management, you need 2 things, metadata and source, and those two
should probably not be managed together.  The Ports-like systems all
seem to do this well - you have a tree of metadata (usually managed
via some VCS [FreeBSD uses Perforce and CVS], but easily distributed
via rsync to end users).  Metadata references tarballs.  I don't see a
whole lot of merit for hooking into upstream VCS for the general case,
as most users will probably want to use released tarballs of
everything except the few modules they're working on.

Now, it makes a lot of sense to me to use a DVCS for managing the
metadata, but that's entirely orthogonal to the systems used for
managing sources.  The DVCS metadata has the benefit of allowing
developers to pull down a tree, version-control their changes as they
incorporate controls to build newer/CVS versions, and the resubmit
those to the master tree for distribution to users.  Administrators
can pull down a tree, version-control local adaptations (changing
compiler options or whatnot), and re-sync with upstream in a
controlled fashion.  Also, for this, providing the ability to pull
source from VCS would be a decent idea, so the developer can build on
their local system, but this is primarily for development purposes and
not for production or end-user distribution (in the general case).
This could possibly be implemented by hooks to generate tarballs from
the VCS -- yes, it has some build overhead, but it keeps the
developers honest by making the path of least resistance require that
their source stay in a quasi-releasable state.

Whether this metadata looks like Debian build files, or like
*BSD/Gentoo ports, or a tree of RPM SPEC files, doesn't much matter in
my mind.

To recap, my vision for this thing:
 - Tree of build control and dependency declaration files, managed
   with a DVCS, with end-user distribution (perhaps updated nightly
   from the VCS) via rsync.
 - Tarballs of upstream releases, published by upstream developers,
   and cached by the distribution project (i.e. a "distfiles"
   directory containing the sources for all software currently
   referenced by the tree, perhaps with old versions also).

Now, to test code against the tree, the developer checks out a local
copy of the metadata tree.  They then go to the control files for
their project, and modify them to say "get the source from over here"
(adjusting version numbers, controls, etc. as appropriate).  Then,
they build, and everything should work smoothly, with minimal hassle.

Of course, the CPAN-style proposal also sounds like a good idea.  I'm
just not so sure on this whole all-sources-in-one-tree thing - it
seems like a significant amount of wasted space and strangeness.

(Of course, if I'm misunderstanding what's being proposed, feel free
to correct me!)

- Michael

-- 
mouse, n: A device for pointing at the xterm in which you want to type.
Confused by the strange files?  I cryptographically sign my messages.
For more information see <http://www.elehack.net/resources/gpg>.

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 30-01-2008, Michael Ekstrand <michael+ocaml@elehack.net> wrote:
> Sylvain Le Gall <sylvain@le-gall.net> writes:
>> Using a simple wget/rsync (you can redevelop it in OCaml) is far
>> more simple than to use a VCS.
>
> To recap, my vision for this thing:
>  - Tree of build control and dependency declaration files, managed
>    with a DVCS, with end-user distribution (perhaps updated nightly
>    from the VCS) via rsync.
>  - Tarballs of upstream releases, published by upstream developers,
>    and cached by the distribution project (i.e. a "distfiles"
>    directory containing the sources for all software currently
>    referenced by the tree, perhaps with old versions also).
>
> Now, to test code against the tree, the developer checks out a local
> copy of the metadata tree.  They then go to the control files for
> their project, and modify them to say "get the source from over here"
> (adjusting version numbers, controls, etc. as appropriate).  Then,
> they build, and everything should work smoothly, with minimal hassle.
>

Yep, i like this idea. I think i will set up a ftp zone which should
accept upstream tarball and put them into a standard location (as
discussed in another post). This is a good first step, that makes
sense...

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 2177 bytes --]

Excerpts from Michael Ekstrand's message of Wed Jan 30 17:32:20 +0100 2008:
> Sylvain Le Gall <sylvain@le-gall.net> writes:
> > Using a simple wget/rsync (you can redevelop it in OCaml) is far
> > more simple than to use a VCS.
> 
> Hear, hear.
> 
> I think that it's somewhat strange to incorporate the source for all
> packages into one massive VCS, and it looks like that's what's been
> proposed.
> 
> Forgive me if I'm rehashing things that have already been brought up,
> but I think that some kind of hybrid thing is good, and that Sylvain
> has some excellent points.  For the general case of package
> management, you need 2 things, metadata and source, and those two
> should probably not be managed together.  The Ports-like systems all
> seem to do this well - you have a tree of metadata (usually managed
> via some VCS [FreeBSD uses Perforce and CVS], but easily distributed
> via rsync to end users).  Metadata references tarballs.  I don't see a
> whole lot of merit for hooking into upstream VCS for the general case,
> as most users will probably want to use released tarballs of
> everything except the few modules they're working on.

That's  what  I  have  in  mind,  but I think that Berke had the whole tree of
sources idea in mind.

But  what's  happen  when  you  want  to  improve  on  some  external package.
Generally  ports  like  systems  have  a  directory  where  they store patches
against  upstream. This kind of system is perhaps not the best one when you've
made a non trivial change on the upstream package.

However  you  can  point  the  URL  field  of the control file to your locally
hacked version of the upstream package.

Example, instead of:

$ cd dev-ml/libfoo
# make the patch
$ diff -u ... ... > files/libfoo-4.5_with_feature_bar.patch
# create a new control file, for this new version
$ cp libfoo-4.5.opkg libfoo-4.5_with_feature_bar.opkg
# add the patch to the list of patches in the .opkg

Rather do:

$ cd dev-ml/libfoo
$ cp libfoo-4.5.opkg libfoo-4.5_with_feature_bar.opkg
# replace the URL field in libfoo-4.5_with_feature_bar.opk
# by URL: http://myhost.net/darcs/repos/libfoo@with_feature

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Olivier Andrieu]

On Jan 30, 2008 5:32 PM, Michael Ekstrand <michael+ocaml@elehack.net> wrote:
> Sylvain Le Gall <sylvain@le-gall.net> writes:
> > Using a simple wget/rsync (you can redevelop it in OCaml) is far
> > more simple than to use a VCS.
>
> Hear, hear.
>
> I think that it's somewhat strange to incorporate the source for all
> packages into one massive VCS, and it looks like that's what's been
> proposed.
>
> Metadata references tarballs.

Having metadata point to the upstream VCS isn't necessarily difficult,
lots of build systems support that.
It can be as simple as using the right URI: http:// ftp:// for
tarballs, svn://, hg:// git://, etc. for a VCS.
The user doesn't need to know how to operate the VCS, the build system
does this for him.

> I don't see a
> whole lot of merit for hooking into upstream VCS for the general case,
> as most users will probably want to use released tarballs of
> everything except the few modules they're working on.

Speaking as the author of a caml binding library that has no published
tarball, I don't see a lot of merit for preventing access to those VCS
either :)
A revision id or tag on a repository is as good as a tarball for
referencing a precise state of some software component.

-- 
  Olivier

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Vlad Skvortsov]

Olivier Andrieu wrote:
>> I think that it's somewhat strange to incorporate the source for all
>> packages into one massive VCS, and it looks like that's what's been
>> proposed.
>>
>> Metadata references tarballs.
>>     
>
> Having metadata point to the upstream VCS isn't necessarily difficult,
> lots of build systems support that.
> It can be as simple as using the right URI: http:// ftp:// for
> tarballs, svn://, hg:// git://, etc. for a VCS.
> The user doesn't need to know how to operate the VCS, the build system
> does this for him.
>   


...meaning that the user has to install *all* possible VCSs on *all* 
machines where an application is to be built? Looks like a maintenance 
nightmare to me.

>> I don't see a
>> whole lot of merit for hooking into upstream VCS for the general case,
>> as most users will probably want to use released tarballs of
>> everything except the few modules they're working on.
>>     
>
> Speaking as the author of a caml binding library that has no published
> tarball, I don't see a lot of merit for preventing access to those VCS
> either :)
> A revision id or tag on a repository is as good as a tarball for
> referencing a precise state of some software component.
>   

This is about introducing yet another concept into the package 
management system, which doesn't simplify that at all.

I think the system needs to use the least common denominator as a basic 
transport if we want it to be widely adopted. Which is, I believe, 
HTTP/FTP in today's world.

-- 
Vlad Skvortsov, vss@73rus.com, http://vss.73rus.com

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 30-01-2008, Berke Durak <berke.durak@exalead.com> wrote:
> Sylvain Le Gall a écrit :
>> Please don't go into this. If you want to talk about PM, don't talk
>> about VCS. My point is that if you want to build something that last you
>> should keep focus on PM, which is not really bound to VCS (like content
>> of files is not bound to filesystem). There is no best VCS for doing PM.
>> We just should handle a way to :
>> * let anyone use a different VCS
>> * be able to download at least a version of each different packages
>
> > The most simple way to handle this, to my mind:
> > * distribute METADATA for packaging into ftp/http
> > * put a link to the VCS inside METADATA, to tell where you can find most
> >  recent one
>
>
> Now why would we want to use a VCS?
>
> * As a data storage and distribution mechanism
> * For efficient updates
> * For the ability to check out any revision
>

[...]

I forget to tell you: even if i am in favor of a simple FTP repository
and not about forcing people to use a VCS, i will provide you with
resources for both (i.e. a VCS or a FTP repository) in the name of
OCamlCore. 

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Jon Harrop]

On Wednesday 30 January 2008 09:06:29 Sylvain Le Gall wrote:
> We, in Debian, are using this way. It works pretty well...

Speaking as a Debian user: the Debian model works well for all of my other 
software but not for OCaml.

OCaml has brittle binding problems and, consequently, its Debian packages are 
constantly in a state of flux as all minor minor updates cause avalanches of 
broken dependencies and packages gradually get recompiled and redistributed. 
This is presumably why Ubuntu's snapshots don't work either.

I'm often recompiling from Debian's source packages in a desperate attempt to 
get things to work. I wasted a *lot* of time doing this when OCaml 3.10 was 
first released on Debian because it broke so much code. That release caused a 
lot of problems for me and solved basically nothing. I can sympathize with 
the Debian user who recently said they moved to GODI.

OCaml currently has lots of problems and many are quite severe. INRIA have 
made it clear that they will never fix these (it isn't their job). Lots of 
people want to fix OCaml and are more than capable of doing so. They just 
need an OCaml distribution they can contribute to. The solution seems quite 
obvious to me: fork OCaml, fix it, include a package management system 
specifically for OCaml in its distribution (alleviating distro package 
maintainers) and then users can contribute third party libraries for it 
easily, recompiled from source as required. If anyone does this, I for one 
will gladly migrate.

> and don't let people going into endless discussion about Git being faster
> than Darcs -- but less useful than Hg. 

Debian have also failed to resolve this.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 30-01-2008, Jon Harrop <jon@ffconsultancy.com> wrote:
> On Wednesday 30 January 2008 09:06:29 Sylvain Le Gall wrote:
>> We, in Debian, are using this way. It works pretty well...
>
> Speaking as a Debian user: the Debian model works well for all of my other 
> software but not for OCaml.
>

[...]

OCaml in Debian has a lot of problems. I have talked about openly
during the meeting. You can find my slides on the wiki, if you want more
information about it.
http://wiki.cocan.org/events/europe/ocamlmeetingparis2008

In this case, i was talking about dealing with repository/metadata.
Which is something about package management and not OCaml in Debian. I
just suggest using a ftp repository and putting a VCS field in the
metadata.

Please read me twice before answering. 

>
>> and don't let people going into endless discussion about Git being faster
>> than Darcs -- but less useful than Hg. 
>
> Debian have also failed to resolve this.
>

I never participate to flameware. But i have seen many on Debian lists,
and think it is a waste of time. Just like people on this list when they
also go into flameware...

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Jon Harrop]

On Wednesday 30 January 2008 10:18:14 Sylvain Le Gall wrote:
> OCaml in Debian has a lot of problems. I have talked about openly
> during the meeting. You can find my slides on the wiki, if you want more
> information about it.
> http://wiki.cocan.org/events/europe/ocamlmeetingparis2008

Yes. Thanks for the link.

> In this case, i was talking about dealing with repository/metadata.
> Which is something about package management and not OCaml in Debian. I
> just suggest using a ftp repository and putting a VCS field in the
> metadata.

Sure. I don't see a clear preference for either myself.

> >> and don't let people going into endless discussion about Git being
> >> faster than Darcs -- but less useful than Hg.
> >
> > Debian have also failed to resolve this.
>
> I never participate to flameware. But i have seen many on Debian lists,
> and think it is a waste of time. Just like people on this list when they
> also go into flameware...

Flamewars may be a waste of time but I don't see that as a flamewar. The idea 
of using software written in Haskell (darcs) because OCaml has none and 
Haskell is a related language seems nice but several darcs users have said 
here that it is really broken and unusable. I've never used it myself but 
I've certainly heard that a lot elsewhere and few people use darcs anyway.

Given that many people want to use the vanilla OCaml distribution from INRIA 
but many other people want to use improved versions, I only ask that future 
work provide for both. If the distribution is to be source-based with 
automatic recompilation then this should be quite easy: we have two roots for 
the distribution, one providing INRIA's raw OCaml and the other providing a 
dressed up alternative that the community can develop.

Would everyone be happy with that?

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 1742 bytes --]

Excerpts from jon's message of Wed Jan 30 11:43:55 +0100 2008:
> On Wednesday 30 January 2008 10:18:14 Sylvain Le Gall wrote:
> > OCaml in Debian has a lot of problems. I have talked about openly
> > during the meeting. You can find my slides on the wiki, if you want more
> > information about it.
> > http://wiki.cocan.org/events/europe/ocamlmeetingparis2008
> 
> Yes. Thanks for the link.
> 
> > In this case, i was talking about dealing with repository/metadata.
> > Which is something about package management and not OCaml in Debian. I
> > just suggest using a ftp repository and putting a VCS field in the
> > metadata.
> 
> Sure. I don't see a clear preference for either myself.
> 
> > >> and don't let people going into endless discussion about Git being
> > >> faster than Darcs -- but less useful than Hg.
> > >
> > > Debian have also failed to resolve this.
> >
> > I never participate to flameware. But i have seen many on Debian lists,
> > and think it is a waste of time. Just like people on this list when they
> > also go into flameware...
> 
> Flamewars may be a waste of time but I don't see that as a flamewar. The idea 
> of using software written in Haskell (darcs) because OCaml has none and 
> Haskell is a related language seems nice but several darcs users have said 
> here that it is really broken and unusable. I've never used it myself but 
> I've certainly heard that a lot elsewhere and few people use darcs anyway.

Users  that  consider  darcs  broken and/or unusable are not darcs users, they
perhaps  tried  it, that's all.

IMHO  they've  applied  the  "correction  before  optimization"  principle.  A
principle that we often spread in functional programming community.

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Jon Harrop]

On Wednesday 30 January 2008 12:00:01 Nicolas Pouillard wrote:
> Users  that  consider  darcs  broken and/or unusable are not darcs users,
> they perhaps  tried  it, that's all.

So you have not personally experienced performance problems with darcs.

> IMHO  they've  applied  the  "correction  before  optimization"  principle.
> A principle that we often spread in functional programming community.

If darcs is unusably slow then its theoretical correctness is irrelevant. 
Regardless, its actual correctness is still in question because it has so few 
users that it is basically untested software.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Nicolas Pouillard]

[-- Attachment #1: Type: text/plain, Size: 2530 bytes --]

Excerpts from jon's message of Wed Jan 30 14:25:28 +0100 2008:
> On Wednesday 30 January 2008 12:00:01 Nicolas Pouillard wrote:
> > Users  that  consider  darcs  broken and/or unusable are not darcs users,
> > they perhaps  tried  it, that's all.
> 
> So you have not personally experienced performance problems with darcs.

For day-to-day operations... no.

* darcs get URL:
  The  initial "darcs get" can takes some time, because like any other DVCS
  it has to checkout all versions of the project.

  However  having  multiple  branches  of the same project don't imply running
  more  than  one  "darcs  get"  of the upstream URL. One rather clone our our
  local  copy  to  make  another branch. Hard links are used in order to share
  common patches.

  Moreover  darcs  supports  partial  checkout  up  to a checkpoint. Basically
  upstream  makes  checkpoints  when regularly tagging the upstream repo. Then
  if  you  do  a  partial  get,  you only take the last checkpoint and patches
  after  this checkpoint. The problem with partial repository is when you want
  to  access  the  history  before  the  checkpoint,  in  darcs1 you can't. In
  darcs2, it will lazily download missing patches when needed.

  Then  darcs2  have  made  a  lot of progress for improving "darcs get", like
  http pipelining.

* When having a lot of conflicts:
  darcs1  suffer  from  a flawed conflict-resolution algorithm (an exponential
  one). darcs2 handle much better this kind of situations.

* There   is some other cases where darcs can be very slow. However that's not
  when  doing  day-to-day  tasks  but  when  using advanced repository hacking
  things.

> > IMHO  they've  applied  the  "correction  before  optimization"  principle.
> > A principle that we often spread in functional programming community.
> 
> If darcs is unusably slow then its theoretical correctness is irrelevant. 

But  that's  not the case. I use it all the day long and I'm far from being alone
in this case.

> Regardless, its actual correctness is still in question because it has so few 
> users that it is basically untested software.

I  really  doubt  that.  There is an incomplete and certainly outdated list of
projects  using  it  [1].  I also that darcs is far more mature than git or hg
(original announcement in 2003).

Moreover  as  you  know, some programming techniques and languages can help to
gain robustness without huge mass user testing.

[1]: http://wiki.darcs.net/DarcsWiki/ProjectsUsingDarcs

-- 
Nicolas Pouillard aka Ertai

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Pietro Abate]

Hi all

On Wed, Jan 30, 2008 at 09:06:29AM +0000, Sylvain Le Gall wrote:
> On 29-01-2008, Berke Durak <berke.durak@exalead.com> wrote:
> > Assume you are writing a program FOO and want to use a package BAR
> > available from bar.org.  You tell ocamlbuild by adding some tag such
> > as
> >
> >    <mytarget.native>: require(http://bar.org/repository/)
> >
> > And when you run ocamlbuild, it automatically checks out a copy of
> > BAR, compiles and loads its myocamlbuild.ml module which adds the
> > required flags.  Of course it should be possible to specify a
> > particular revision...  And if BAR has itself dependencies, those too
> > would be checked out.

My two cents here:

I'd like to follow the lead from the idea above. I've the impression we
are not converging. I think if we want to succeed in this packaging
revolution we need to proceed is small steps. 

The basic and foremost important requirement we have is to try to
standardize the way ocaml libraries/applications are built. One of the
main problem package maintainers have is that everybody seems to write
their own build system and to follow different rules in their projects.
We have a plethora of building system, but in the end what we really
need is an interface to use them. Using a simple makefile for example
with make {opt,byte,install,clean} and to agree to what these rules do
would be a huge step forward. I'm not saying everybody must use make,
but that all projects should ship a makefile with these 4 rules. What is
then effectively used to build the package is not my problem (except for
building dependencies with exotic tools). In the future we might think
of proposing a standard building tool for everybody as ocamlbuild... but
I think this is premature now.

Ok, at this point we have the package, and we know how to build it. Next
step is to have one place to upload all tarballs. Godi has its own
repository, debian has its own repository on alioth. Here I'm thinking
something like CPAN, with a light weight registration and upload
mechanism based on pgp (Register, sign the package, upload the package).
The infrastructure to build this kind of service is not rocket science.
We could even imagine some king of validation process. This way all
developer could just work as they do, using whatever vcs they like,
adding just one simple step whan they want to release a new package into
the wild. Dependencies could be addressed at this level or at the next.
Anyway a complex project with many dependencies must find a way to
resolve these dependencies somehow: either in their makefile (see
require(http://bar.org/repository/) as above) or just by assuming that all
dependencies are already satisfied and leave the burden to Package
maintainers (as it is at the moment).

>From this point on the sky. Debian, godi, fedora,... can just keep going
as they do with their system but using a single upstream repository with
standardized build system. People with other needs could use a system as
describe above with ocamlbuild. In the future we can even think to some
kind of convergence of different distributions to handle dependencies,
versioning, etc ...

In this way I tried to chance as small as possible of the current
infrastructure, still trying to give people the possibility of easily
recompiling their projects and all dependencies.

Maybe this view it too naive, but it might be a good way to start.

:)
pietro

-- 
++ 
++ "All great truths begin as blasphemies." -George Bernard Shaw
++ Please avoid sending me Word or PowerPoint attachments.
   See http://www.gnu.org/philosophy/no-word-attachments.html

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Stefano Zacchiroli]

On Wed, Jan 30, 2008 at 11:37:05PM +1100, Pietro Abate wrote:
> The basic and foremost important requirement we have is to try to
> standardize the way ocaml libraries/applications are built. One of the

Full ack on this.

Of course now the point is how to agree on a standard API, and how to
make all projects out there start switching to it ...

> Next step is to have one place to upload all tarballs. Godi has its
> own repository, debian has its own repository on alioth.

Just to be clear here: Debian does not have a repository of OCaml
libraries and programs on alioth (the GForge instance used by Debian).
It only has a repository with the Debian specificities used to build
debian packages out of those libraries and programs. Then we also cache,
for the ease of our workflow, the original tarballs there, but they are
not the point of that repository.

> Here I'm thinking something like CPAN, with a light weight
> registration and upload mechanism based on pgp (Register, sign the
> package, upload the package).  The infrastructure to build this kind

And this is something which is hopefully coming, provided by Sylvain and
a couple of helpers (including myself). His technical proposal is not
the same as yours, but has the advantages of being based on pieces of
software which already exists and do not need to be written from
scratch. Just wait a bit for the setup ...

-- 
Stefano Zacchiroli -*- PhD in Computer Science ............... now what?
zack@{upsilon.cc,cs.unibo.it,debian.org}  -<%>-  http://upsilon.cc/zack/
(15:56:48)  Zack: e la demo dema ?    /\    All one has to do is hit the
(15:57:15)  Bac: no, la demo scema    \/    right keys at the right time

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Gerd Stolpmann]

Hi,

first of all sorry for my last message where I lost my temper. This is
not the tone I like to read either. Let's better stick to the real
facts, and ignore messages where we have the impression that it is only
complained without giving any insight.

Am Mittwoch, den 30.01.2008, 23:37 +1100 schrieb Pietro Abate:
> Hi all
> 
> On Wed, Jan 30, 2008 at 09:06:29AM +0000, Sylvain Le Gall wrote:
> > On 29-01-2008, Berke Durak <berke.durak@exalead.com> wrote:
> > > Assume you are writing a program FOO and want to use a package BAR
> > > available from bar.org.  You tell ocamlbuild by adding some tag such
> > > as
> > >
> > >    <mytarget.native>: require(http://bar.org/repository/)
> > >
> > > And when you run ocamlbuild, it automatically checks out a copy of
> > > BAR, compiles and loads its myocamlbuild.ml module which adds the
> > > required flags.  Of course it should be possible to specify a
> > > particular revision...  And if BAR has itself dependencies, those too
> > > would be checked out.
> 
> My two cents here:
> 
> I'd like to follow the lead from the idea above. I've the impression we
> are not converging. I think if we want to succeed in this packaging
> revolution we need to proceed is small steps. 

I agree that less is more in this area. Software distribution is a quite
complex problem, and it is unlikely that we can find a consensus in the
community unless we reduce it to the minimum where cooperation makes
sense. My advice is to see this not mainly as a technical but as a
political challenge.

I've followed the VCS-based versus central debate now for a while, and
my impression is that it is too theoretical. I would like to recall that
GODI implements a mixed system - it is a central system by default, but
you can overlay the centrally distributed packages with your own (I
showed it on a slide in Paris, and note that GODI differs in this
respect from BSD ports), so it is already possible to distribute patched
versions. Of course, it is a bit more work to publish patched versions
than in a VCS-based system - they have to be tared up, and put onto a
web server, but I'm sure this additional overhead can be technically
addressed. But anyway, whoever wants to build a VCS-based distribution
system, go ahead and make your own experience.

The more interesting thing is the question: what is the unit of
distribution? GODI and Debian have the concept of "packages" that reside
in a dependency tree, and the packages have a certain interface for
interaction. I think now that as a common unit for upstream authors such
a concept goes too far. We need something simpler - a descriptive
distribution format so that the distributed units can be plugged in to
existing systems like GODI or Debian, but that the units can also be
built manually for the users who need more control.
 
> The basic and foremost important requirement we have is to try to
> standardize the way ocaml libraries/applications are built. One of the
> main problem package maintainers have is that everybody seems to write
> their own build system and to follow different rules in their projects.
> We have a plethora of building system, but in the end what we really
> need is an interface to use them. Using a simple makefile for example
> with make {opt,byte,install,clean} and to agree to what these rules do
> would be a huge step forward. I'm not saying everybody must use make,
> but that all projects should ship a makefile with these 4 rules. What is
> then effectively used to build the package is not my problem (except for
> building dependencies with exotic tools). In the future we might think
> of proposing a standard building tool for everybody as ocamlbuild... but
> I think this is premature now.

I don't think we will reach a consensus about the build tools.
Currently, the most widely used is make+Unix utilities. I've not yet
seen much omake, and ocamlbuild neither. Suggestions to move away from
make are often driven by better Windows support.

My suggestion is this: A driver file "Obuild" that descriptively says
which build tools (out of some generally accepted options) are supported
by this piece of software. Example for Obuild:

configure_script: configure
build_tool: GNU make
make_bytecode_target: all
make_native_target: opt
make_install_target: install
supports_prefix: true

Obuild could also list several build tools, so the user has choice.

Obuild could also describe build options (with/without a certain
component), and how to pass in paths for C libraries etc.

What's left out are dependencies. They make only sense as a system, and
managing a system is a different story. That should be left to packagers
like GODI or Debian.

> Ok, at this point we have the package, and we know how to build it. Next
> step is to have one place to upload all tarballs. Godi has its own
> repository, debian has its own repository on alioth. Here I'm thinking
> something like CPAN, with a light weight registration and upload
> mechanism based on pgp (Register, sign the package, upload the package).
> The infrastructure to build this kind of service is not rocket science.
> We could even imagine some king of validation process. This way all
> developer could just work as they do, using whatever vcs they like,
> adding just one simple step whan they want to release a new package into
> the wild. Dependencies could be addressed at this level or at the next.
> Anyway a complex project with many dependencies must find a way to
> resolve these dependencies somehow: either in their makefile (see
> require(http://bar.org/repository/) as above) or just by assuming that all
> dependencies are already satisfied and leave the burden to Package
> maintainers (as it is at the moment).

Well, as said, I suggest to leave out dependencies. A dependency error
in a single uploaded package can make the whole tree unbuildable. Deps
are outside the scope of loose cooperation.

> >From this point on the sky. Debian, godi, fedora,... can just keep going
> as they do with their system but using a single upstream repository with
> standardized build system. People with other needs could use a system as
> describe above with ocamlbuild. In the future we can even think to some
> kind of convergence of different distributions to handle dependencies,
> versioning, etc ...
> 
> In this way I tried to chance as small as possible of the current
> infrastructure, still trying to give people the possibility of easily
> recompiling their projects and all dependencies.
> 
> Maybe this view it too naive, but it might be a good way to start.

I think this is more realistic than all the other approaches to start
another complex distribution system from scratch, without seeing that
this is a big undertaking.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Stefano Zacchiroli]

On Wed, Jan 30, 2008 at 03:07:14PM +0100, Gerd Stolpmann wrote:
> My suggestion is this: A driver file "Obuild" that descriptively says
> which build tools (out of some generally accepted options) are supported
> by this piece of software. Example for Obuild:
> 
> configure_script: configure
> build_tool: GNU make
> make_bytecode_target: all
> make_native_target: opt
> make_install_target: install
> supports_prefix: true

I like this proposal in general, and the above example targets sound
reasonable. However, I would like to stress that some semantics
associated to the above name should be clarified somewhere. For example,
here is a sampling of questions I've found differently answered in my
experience of Debian packages of OCaml software:
- does the make_install_target above installs bytecode, nativecode,
  both, the "better" of the two?
- does the make_native_target above requires that make_bytecode_target
  has been invoked first?

The answers might seem obvious, but I assure you that there are an
unbelievable number of different combination of them in the OCaml
distributed softwares out there. The compilation of random C libraries
is far more standardized that OCaml's; this is probably our chance to do
the same.

Also, you would need to support "custom" build tools in which you are
just told to run a given command. Extlib for example needs you to run
"./install.ml" in order to be installed ...

> What's left out are dependencies. They make only sense as a system,
> and managing a system is a different story. That should be left to
> packagers like GODI or Debian.
<snip>
> Well, as said, I suggest to leave out dependencies. A dependency error
> in a single uploaded package can make the whole tree unbuildable. Deps
> are outside the scope of loose cooperation.

I don't follow you here.

First of all the OCaml namespace of distributed software ATM is quite
well defined even if we have had so far no common place where to upload
stuff. So we can rely in general on names being unique. Then, I would
like to have in an Obuild file the declaration of which other OCaml
softwares I need to build that one (i.e. I'm talking about *build*
dependencies here, not runtime/development time dependencies). Why such
a requirement isn't reasonable? You are basically proposing a
declarative specification of the build process of a distributed
software, I consider build dependencies a fundamental part of such a
specification.

Or where you talking about runtime/development time dependencies?

Cheers.

-- 
Stefano Zacchiroli -*- PhD in Computer Science ............... now what?
zack@{upsilon.cc,cs.unibo.it,debian.org}  -<%>-  http://upsilon.cc/zack/
(15:56:48)  Zack: e la demo dema ?    /\    All one has to do is hit the
(15:57:15)  Bac: no, la demo scema    \/    right keys at the right time

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Gerd Stolpmann]

Am Mittwoch, den 30.01.2008, 14:37 +0100 schrieb Stefano Zacchiroli:
> On Wed, Jan 30, 2008 at 03:07:14PM +0100, Gerd Stolpmann wrote:
> > My suggestion is this: A driver file "Obuild" that descriptively says
> > which build tools (out of some generally accepted options) are supported
> > by this piece of software. Example for Obuild:
> > 
> > configure_script: configure
> > build_tool: GNU make
> > make_bytecode_target: all
> > make_native_target: opt
> > make_install_target: install
> > supports_prefix: true
> 
> I like this proposal in general, and the above example targets sound
> reasonable. However, I would like to stress that some semantics
> associated to the above name should be clarified somewhere. For example,
> here is a sampling of questions I've found differently answered in my
> experience of Debian packages of OCaml software:
> - does the make_install_target above installs bytecode, nativecode,
>   both, the "better" of the two?
> - does the make_native_target above requires that make_bytecode_target
>   has been invoked first?

Fully agreed that we need a reference document for this.

> The answers might seem obvious, but I assure you that there are an
> unbelievable number of different combination of them in the OCaml
> distributed softwares out there. The compilation of random C libraries
> is far more standardized that OCaml's; this is probably our chance to do
> the same.
> 
> Also, you would need to support "custom" build tools in which you are
> just told to run a given command. Extlib for example needs you to run
> "./install.ml" in order to be installed ...
> 
> > What's left out are dependencies. They make only sense as a system,
> > and managing a system is a different story. That should be left to
> > packagers like GODI or Debian.
> <snip>
> > Well, as said, I suggest to leave out dependencies. A dependency error
> > in a single uploaded package can make the whole tree unbuildable. Deps
> > are outside the scope of loose cooperation.
> 
> I don't follow you here.
> 
> First of all the OCaml namespace of distributed software ATM is quite
> well defined even if we have had so far no common place where to upload
> stuff. So we can rely in general on names being unique. Then, I would
> like to have in an Obuild file the declaration of which other OCaml
> softwares I need to build that one (i.e. I'm talking about *build*
> dependencies here, not runtime/development time dependencies). Why such
> a requirement isn't reasonable? You are basically proposing a
> declarative specification of the build process of a distributed
> software, I consider build dependencies a fundamental part of such a
> specification.
> 
> Or where you talking about runtime/development time dependencies?

See, this question already demonstrates that dependencies are a not so
easy concept. It is at least difficult to reach a common understanding
what would have to be listed as dep and what not. We have dependencies
that come into play at different times, and there are also conditional
dependencies (i.e. you need this software only if you want to build this
special feature). Then there is the question of how detailed the
dependency relation is (versions, or about allowing to have only parts
of software units as dependency).

A formalization of deps needs some complexity in order to be useful, but
that makes a common understanding of them more difficult. I have some
doubts that we find a balance here, so my advice to leave a formal
description of deps out.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Stefano Zacchiroli]

On Wed, Jan 30, 2008 at 04:12:15PM +0100, Gerd Stolpmann wrote:
> > Or where you talking about runtime/development time dependencies?
> 
> See, this question already demonstrates that dependencies are a not so
> easy concept. It is at least difficult to reach a common understanding
> what would have to be listed as dep and what not. We have dependencies
> that come into play at different times, and there are also conditional
> dependencies (i.e. you need this software only if you want to build this
> special feature). Then there is the question of how detailed the
> dependency relation is (versions, or about allowing to have only parts
> of software units as dependency).
> 
> A formalization of deps needs some complexity in order to be useful, but
> that makes a common understanding of them more difficult. I have some
> doubts that we find a balance here, so my advice to leave a formal
> description of deps out.

That it is trickier than listing a handful of make targets if out of
discussion. That it is *so* tricky that should not be addressed I
disagree with.

All the required information you mention above are already a part of the
build instruction of whatever program library you find out there. The
INSTALL (or whatever) file should list the set of libraries which are
required to build properly that library; if some of them are optional,
they should be flagged as such.  My question is: given that this kind of
information are already available, why shouldn't we refrain to describe
it with a common syntax which can help in understanding mechanically
interrelationships?

Note that I'm not requesting to actually mandate the *use* of this kind
of information in whatever build system, but only to use "morale
suasion" on library/program authors to write them down in a parseable
syntax.

Regarding the kind of relationships which can be described and how
(alternatives, version requirements), the major GNU/Linux distribution
families have a quite large subset of supported feature which are not so
tricky and are agreed upon.

Cheers.

-- 
Stefano Zacchiroli -*- PhD in Computer Science ............... now what?
zack@{upsilon.cc,cs.unibo.it,debian.org}  -<%>-  http://upsilon.cc/zack/
(15:56:48)  Zack: e la demo dema ?    /\    All one has to do is hit the
(15:57:15)  Bac: no, la demo scema    \/    right keys at the right time

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Gerd Stolpmann]

Am Donnerstag, den 31.01.2008, 10:02 +0100 schrieb Stefano Zacchiroli:
> On Wed, Jan 30, 2008 at 04:12:15PM +0100, Gerd Stolpmann wrote:
> > > Or where you talking about runtime/development time dependencies?
> > 
> > See, this question already demonstrates that dependencies are a not so
> > easy concept. It is at least difficult to reach a common understanding
> > what would have to be listed as dep and what not. We have dependencies
> > that come into play at different times, and there are also conditional
> > dependencies (i.e. you need this software only if you want to build this
> > special feature). Then there is the question of how detailed the
> > dependency relation is (versions, or about allowing to have only parts
> > of software units as dependency).
> > 
> > A formalization of deps needs some complexity in order to be useful, but
> > that makes a common understanding of them more difficult. I have some
> > doubts that we find a balance here, so my advice to leave a formal
> > description of deps out.
> 
> That it is trickier than listing a handful of make targets if out of
> discussion. That it is *so* tricky that should not be addressed I
> disagree with.
> 
> All the required information you mention above are already a part of the
> build instruction of whatever program library you find out there. The
> INSTALL (or whatever) file should list the set of libraries which are
> required to build properly that library; if some of them are optional,
> they should be flagged as such.  My question is: given that this kind of
> information are already available, why shouldn't we refrain to describe
> it with a common syntax which can help in understanding mechanically
> interrelationships?
> 
> Note that I'm not requesting to actually mandate the *use* of this kind
> of information in whatever build system, but only to use "morale
> suasion" on library/program authors to write them down in a parseable
> syntax.
> 
> Regarding the kind of relationships which can be described and how
> (alternatives, version requirements), the major GNU/Linux distribution
> families have a quite large subset of supported feature which are not so
> tricky and are agreed upon.

Interesting view, but it still doesn't convince me that including a
formalization of deps would be worth the effort. I would like to hear
more opinions about this:

- Are software maintainers willing to provide the information in a
  formalized way? That would mean they had to understand the details
  of the formalization, and also had to maintain these data (e.g.
  respond to users who find errors etc.). Depending on how much detail
  the formalization can express, this can be very easy or tricky.

- Is this so useful for the users that they really need it? I mean
  the primary users would be the existing distribution systems
  like GODI or Debian that already have formalizations for deps. 
  I expect that most end users won't directly install software,
  and if so, it is likely that they run manually through the
  the build package by package, so a description of deps in English
  would be almost as useful as a formalization.
  
_If_ we agree that a formalization is useful, I am still of the opinion
the simpler the better. So maybe only a list of build requirements,
where some are flagged as optional. That would be a simple string

"pgk1 pkg2 pkg3? pkg4?"

(where "?" signals an optional dependency).

Including version requirements has the disadvantage that it enforces a
certain syntax of the package names, and that there are several
solutions for comparing version strings.

For even more structure like alternatives, or mutual incompatibilities I
don't see how this could be processed in a uniform way. The package
systems are very different in this point.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Stefano Zacchiroli]

On Fri, Feb 01, 2008 at 04:03:42PM +0100, Gerd Stolpmann wrote:
> > Regarding the kind of relationships which can be described and how
> > (alternatives, version requirements), the major GNU/Linux distribution
> > families have a quite large subset of supported feature which are not so
> > tricky and are agreed upon.
> Interesting view, but it still doesn't convince me that including a
> formalization of deps would be worth the effort. I would like to hear
> more opinions about this:

Fair enough. Socially however, I doubt you will be able to seek several
opinions in this sub-sub-sub thread :), you should probably start a new
thread or something such.

I will answer for myself.

> - Are software maintainers willing to provide the information in a
>   formalized way? That would mean they had to understand the details

Wearing my hat of (not so widespread) software maintainer, it wouldn't
make any difference. If I care about proper installation instruction I
already have to maintainer a proper INSTALL file, maintaining a more
formal dependency expression in a file won't be much of a difference.
Actually, if the format spreads and the community reach an agreement on
it, it would be even better since I can avoid writing useless prose to
describe the deps in INSTALL, and just care of the dependency
expression.

(Note that in my case, since I'm also the Debian maintainer of my
software, I'm also already maintaining the dependency expression, but I
presume this is not a common case.)

> - Is this so useful for the users that they really need it? I mean
>   the primary users would be the existing distribution systems
>   like GODI or Debian that already have formalizations for deps. 
>   I expect that most end users won't directly install software,
>   and if so, it is likely that they run manually through the
>   the build package by package, so a description of deps in English
>   would be almost as useful as a formalization.

In general, the usefulness for the users depends on the available tools
using the dependency formalization. This is a like a chicken and egg
problem: until we have the formalization we won't be able to show
usefulness. I'm trying to catch the bus which is passing: if we are
going to formalize software installation related stuff, I'll try to get
it right without loosing information.

Anyhow, regarding your end users point, I don't know for GODI since I'm
not using it, but in Debian we have always kept in mind that users
should be able to install OCaml libraries which are not packaged in
Debian. So, for example, our default findlib configuration supports
/usr/local/... The reason is that no packaging framework supports every
piece of OCaml software out there.  In this sense, end users do install
software, and a dependency formalization would give us possibilities
such as:

- have a single system (ocamlfind based) to check if the required deps
  to compile a library or program are available

In a word, it would get us nearer to CPAN's "perl Makefile.PL" 1-click
installation procedure.

> _If_ we agree that a formalization is useful, I am still of the
> opinion the simpler the better. So maybe only a list of build
> requirements, where some are flagged as optional. That would be a
> simple string
> 
> "pgk1 pkg2 pkg3? pkg4?"

Full ACK: the simpler, the better. Something like that is fine with me,
though I would like to see at least >= version checking. See below ...

> Including version requirements has the disadvantage that it enforces a
> certain syntax of the package names, and that there are several
> solutions for comparing version strings.

I'm less convinced about that (though we are getting into details here,
whereas your bold "If" above is still to be settled). It is true that
there are various syntaxes for versions, but they do entail different
comparison results only in very corner cases. Take pkg-config, AFAIK it
supports >= version comparisons supporting just one syntax for versions.
It does work fine with 99.9999% of the software libraries out there.

> For even more structure like alternatives, or mutual incompatibilities I
> don't see how this could be processed in a uniform way. The package
> systems are very different in this point.

Agreed, also it is seldomly needed.

Cheers.

-- 
Stefano Zacchiroli -*- PhD in Computer Science ............... now what?
zack@{upsilon.cc,cs.unibo.it,debian.org}  -<%>-  http://upsilon.cc/zack/
(15:56:48)  Zack: e la demo dema ?    /\    All one has to do is hit the
(15:57:15)  Bac: no, la demo scema    \/    right keys at the right time

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Matthew Hannigan]

Maintaining a bunch of dependencies and
patches is a bit old school.

Have a look at 
https://wiki.ubuntu.com/NoMoreSourcePackages

It's about ubuntu and bzr, but the approach
is not specific to those two.

Matt

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Nathaniel Gray]

On Feb 1, 2008 7:03 AM, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
>
> - Are software maintainers willing to provide the information in a
>   formalized way? That would mean they had to understand the details
>   of the formalization, and also had to maintain these data (e.g.
>   respond to users who find errors etc.). Depending on how much detail
>   the formalization can express, this can be very easy or tricky.

Personally, I would be enthusiastically happy to provide such a description.

> - Is this so useful for the users that they really need it? I mean
>   the primary users would be the existing distribution systems
>   like GODI or Debian that already have formalizations for deps.
>   I expect that most end users won't directly install software,
>   and if so, it is likely that they run manually through the
>   the build package by package, so a description of deps in English
>   would be almost as useful as a formalization.

I have seen many, many cases where the dependency information compiled
by non-developers is just awful.  I've seen packages that depend on
Firefox (and want me to build it from source!) because the software
uses HTML help files.  I recently installed KOffice on OS X using
MacPorts and got a chuckle when I learned that gtk2 was a dependency.
It would be much better if dependencies were specified by the
developers directly and the packagers used that info.

Cheers,
-n8

-- 
>>>-- Nathaniel Gray -- Caltech Computer Science ------>
>>>-- Mojave Project -- http://mojave.cs.caltech.edu -->

RE: [Caml-list] Re: [OSR] Ports-like package management system

[David Allsopp]

> I don't think we will reach a consensus about the build tools.
> Currently, the most widely used is make+Unix utilities. I've not yet
> seen much omake, and ocamlbuild neither. Suggestions to move away from
> make are often driven by better Windows support.

Windows has ample support for make and friends at
http://unxutils.sourceforge.net/ - I've happily used make under Windows for
years without resorting to Cygwin. As long as you keep your scripts sane you
can write "portable" OCaml Makefiles that work correctly under *NIX and
Windows using all your favourite shell commands.

I recall that RH Linux (a long time ago, Fedora may be different!), by
default, didn't install make or even a C compiler so a developer using RH
Linux had to choose install these things. There's no conceivable reason why
a Windows developer should expect not to have to install the odd development
tool too in addition to the OCaml compiler.

There of course other off-topic arguments for trying to adopt build systems
other than make.

Just to add a bit of Windows seasoning to this debate - if GODI (or
whatever) becomes a "de facto" standard for OCaml packaging, then it's going
to need to support Windows.


David

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 30-01-2008, Pietro Abate <Pietro.Abate@anu.edu.au> wrote:
> Hi all
>
> On Wed, Jan 30, 2008 at 09:06:29AM +0000, Sylvain Le Gall wrote:
>> On 29-01-2008, Berke Durak <berke.durak@exalead.com> wrote:
>> > Assume you are writing a program FOO and want to use a package BAR
>> > available from bar.org.  You tell ocamlbuild by adding some tag such
>> > as
>> >
>> >    <mytarget.native>: require(http://bar.org/repository/)
>> >
>> > And when you run ocamlbuild, it automatically checks out a copy of
>> > BAR, compiles and loads its myocamlbuild.ml module which adds the
>> > required flags.  Of course it should be possible to specify a
>> > particular revision...  And if BAR has itself dependencies, those too
>> > would be checked out.
>
> My two cents here:
>
> I'd like to follow the lead from the idea above. I've the impression we
> are not converging. I think if we want to succeed in this packaging
> revolution we need to proceed is small steps. 
>

[...]

To make a short summary:

* have a makefile with 4 targets in upstream tarball: opt, byte, clean, install
* have a central repository with all the upstream tarball

That is "keep it simple and stupide" (KISS)! That is great, easy and direct. I
like it.

I think we should add some variable to the makefile to define target
installation et al, but that is technical details. 

The main idea fit to a progressive way of doing. Maybe at the beginning
we should provide the Makefile as and external things (just like GODI).

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Bünzli Daniel]

Le 30 janv. 08 à 15:13, Sylvain Le Gall a écrit :

> To make a short summary:
>
> * have a makefile with 4 targets in upstream tarball: opt, byte,  
> clean, install

Add 'doc', it's important.

As long as the makefile can transfer the hard work to ocamlbuild I  
have no objection.

However I would just like to make a case for using ocamlbuild instead  
of make. First it removes a dependency from the port system since  
ocamlbuild is distributed with the bare caml distribution -- it is  
also makes it more Windows friendly altough I'm not particularly  
sensitive to that. Secondly provided you use ocamlbuild as your build  
system it may significantly simplify the _usage_ of ports in our  
projects. I guess most of the time it will be about just adding a _tag  
to the executable to produce, like

exec.native : use_my_package_with_c_bindings

No more tedious link errors, the flags are automatically setup  
properly etc. ocamlbuild's tag system is the right way to manage build  
exceptions, it is declarative and not spagheti like. I urge you to  
have a serious look at it.

Best,

Daniel

Re: [Caml-list] Re: [OSR] Ports-like package management system

[N. Owen Gunden]

On Wed, Jan 30, 2008 at 02:13:26PM +0000, Sylvain Le Gall wrote:
> To make a short summary:
> 
> * have a makefile with 4 targets in upstream tarball: opt, byte, clean, install
> * have a central repository with all the upstream tarball
> 
> That is "keep it simple and stupide" (KISS)! That is great, easy and direct. I
> like it.
> 
> I think we should add some variable to the makefile to define target
> installation et al, but that is technical details. 

When we designed godiva, we made up a "packaging policy" that details
upstream requirements, much like you're discussing.  You might look at
it for ideas, though I'm sure it's not exactly what you want:

http://projects.phauna.org/GODIVA/docs/policy.html

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Jon Harrop]

On Wednesday 30 January 2008 12:37:05 Pietro Abate wrote:
> I'd like to follow the lead from the idea above. I've the impression we
> are not converging. I think if we want to succeed in this packaging
> revolution we need to proceed is small steps.

Yes.

> The basic and foremost important requirement we have is to try to
> standardize the way ocaml libraries/applications are built.

... and installed!

> One of the 
> main problem package maintainers have is that everybody seems to write
> their own build system and to follow different rules in their projects.
> We have a plethora of building system, but in the end what we really
> need is an interface to use them. Using a simple makefile for example
> with make {opt,byte,install,clean} and to agree to what these rules do
> would be a huge step forward.

I would like to have core and enhanced OCaml distributions installed and 
simultaneously usable on my system with packages for each fetched, built and 
installed independently. This requires completely different compiled 
libraries (because the stdlibs will be different) and, therefore, different 
installed locations.

Perhaps the current system can already support this by giving the enhanced 
distro a version starting with "e":

  /usr/lib/ocaml/3.10.0
  /usr/lib/ocaml/e3.10.0

and so on. The OCaml compilers etc. should install to "ocamlopt-3.10.0" 
and "ocamlopt-e3.10.0" so the user can provide defaults, e.g. using Debian's 
alternatives.

The core and enhanced OCaml distributions could be separate Debian packages 
with libraries coming entirely from OCaml and nothing else coming from 
Debian.

I think this would solve a lot of headaches. This is orthogonal to VCS vs 
wget, of course.

> I'm not saying everybody must use make, 
> but that all projects should ship a makefile with these 4 rules.

I think that was actually an excellent idea. Can it get "ocamlc" 
and "ocamlopt" from environment variables and install to their default 
location with `ocamlc -where`?

> What is 
> then effectively used to build the package is not my problem (except for
> building dependencies with exotic tools). In the future we might think
> of proposing a standard building tool for everybody as ocamlbuild... but
> I think this is premature now.

Agreed.

> Ok, at this point we have the package, and we know how to build it. Next
> step is to have one place to upload all tarballs. Godi has its own
> repository, debian has its own repository on alioth. Here I'm thinking
> something like CPAN, with a light weight registration and upload
> mechanism based on pgp (Register, sign the package, upload the package).
> The infrastructure to build this kind of service is not rocket science.
> We could even imagine some king of validation process. This way all
> developer could just work as they do, using whatever vcs they like,
> adding just one simple step whan they want to release a new package into
> the wild. Dependencies could be addressed at this level or at the next.
> Anyway a complex project with many dependencies must find a way to
> resolve these dependencies somehow: either in their makefile (see
> require(http://bar.org/repository/) as above) or just by assuming that all
> dependencies are already satisfied and leave the burden to Package
> maintainers (as it is at the moment).
>
> >From this point on the sky. Debian, godi, fedora,... can just keep going
>
> as they do with their system but using a single upstream repository with
> standardized build system. People with other needs could use a system as
> describe above with ocamlbuild. In the future we can even think to some
> kind of convergence of different distributions to handle dependencies,
> versioning, etc ...

... licensing. The system should warn when licenses are broken. For example, I 
might like to release Smoke under the GPL for such a distribution and I'd 
like users to know when they're breaking the license agreement so they can 
buy the commercial license for Smoke (now only £199 ;-).

> In this way I tried to chance as small as possible of the current
> infrastructure, still trying to give people the possibility of easily
> recompiling their projects and all dependencies.
>
> Maybe this view it too naive, but it might be a good way to start.

I think these are really excellent ideas. I agree completely.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] [OSR] Ports-like package management system

[Berke Durak]

Hello all,

Here is the kind of thing I would like to be able to do with an
integrated build/packaging/VCS system.  I think this explains why I want
to base the system on a VCS.

We have a collection of software components C1, C2, ...  A version
control system VCS is selected and agreed upon.  Each usable version
of a software component Ci is stored under a VCS repository Si.  A
single command, say OPKG, does all.  (I think we'll have OPKG =
ocamlbuild, but it could be OMake or something else).

   % cd src
   % mkdir myproject
   % OPKG init           # Initialize the project
   % cat >foo.ml
   Printf.printf "Hello, world!\n" ^D
   % OPKG build foo.native
   (Compiling using ocamlbuild)
   % ./foo.native
   Hello, world!

Now let's say we want to use a regexp library.  Assume there exist
multiple implementations with the same API, say
OSRStandardRegexpLibraryAPI, that we want to use:

   % cat >_tags
   <foo.native>:
   require(some_regexp_lib_with_osr_standard_regexp_library_api)

As pointed by others, it is a good idea to decouple the build rules
from the used versions or the repository URLs.

A file, _requirements, maps require strings (which are enclosed in a
require(..) construct in the _tags file) to actual requirements.

   % cat >_requirements
   some_regexp_lib_with_osr_standard_regexp_library_api: latest(pcre)

In a globally accessible place, there is a file that defines how to
resolve component names to actual repository URLs.  For instance, in
$HOME/.opkg/resolve:

   % cat ~/.opkg/resolve
   pcre latest: v3.4
   pcre updates: http://ocaml-pcre.org/opkg/updates
   pcre v3.4: http://ocaml-pcre.org/repo/ rev r381
   pcre v3.3: http://ocaml-pcre.org/repo/ rev r341
   pcre v3.2: http://ocaml-pcre.org/repo/ rev v3.2

   netclient latest: v9.99
   netclient updates: http://ocaml-pcre.org/opkg/updates
   netclient v9.99: http://netclient.com/repo/ rev r364
   netclient v9.98: http://netclient.com/repo/ rev r300
   netclient v9.97: http://netclient.com/repo/ rev r200

This resolution file can be overridden by creating a file _override_resolve in
the project directory.

We then start using Regexps in our foo.ml:

   % cat foo.ml
   open OSR.Regexp

   let _ = Printf.printf "Hello, %s!\n"
     (if regexp_match ~rex:"[Rr]obert" Sys.argv.(1) then
       "Bob"
     else
       Sys.argv.(1))

   % OPKG build foo.native
   OPKG: inside ~/src/myproject
   OPKG: building map
   OPKG: map: foo.native needs some_regexp_lib_with_osr_standard_regexp_library_api
   OPKG: map: some_regexp_lib_with_osr_standard_regexp_library_api configured here
              as latest(pcre) from _requirements
   OPKG: map: latest(pcre) maps to pcre v3.4 from ~/.opkg/map

OPKG has here created a file _map that maps requirements to actual
versions.

% cat _map
some_regexp_lib_with_osr_standard_regexp_library_api: pcre v3.4

OPKG continues building the target:

   OPKG: done building map OPKG: pcre v3.4 not already checked out
   OPKG: pcre v3.4 available from hg http://ocaml-pcre.org/ rev r381
   OPKG: checking out pcre v3.4 inside ~/src/opkg/pcre-v3.4 (path
         configured from from ~/.opkg.config)
   OPKG: building pcre.native v3.4 in ~/src/opkg/pcre-v3.4
   OPKG: continuing compilation of foo.native

Object files for pcre-v3.4 would reside in
~/src/opkg/pcre-v3.4/_build/ No need to build the bytecode version
unless explicitly needed.

   % OPKG build foo.byte
   OPKG: inside ~/src/myproject ...
   OPKG: building pcre.byte v3.4 in ~/src/opkg/pcre/v3.4
   OPKG: continuing compilation of foo.byte

Similarly, if we want to build a multithreaded version, OPKG will
automatically build multithreaded versions of all required libraries,
using the tag mechanism.

   % OPKG build foo.byte.mt
   OPKG: building pcre.byte.mt v3.4 in ~/src/opkg/pcre/v3.4

Now assume we take a vacation and when we come back, a new version of
PCRE is released, say v3.5.  Of course we need to ask OPKG to update
its resolution file.  We can invoke it anywhere.

   % OPKG update
   OPKG: fetching http://ocaml-pcre.org/opkg/updates
   OPKG: new pcre version v3.5 (also the latest)
   OPKG: fetching http://netclient.com/opkg/updates
   OPKG: netclient still at v9.99
   OPKG: updated ~/.opkg/resolve

If we want to use the newest PCRE in our project, we need to update
~/src/myproject/_map, since it is this file which defines the exact
version to be used.  We can edit that file by hand; but we can also
ask OPKG to regenerate it from the _requirements file:

   % OPKG remap
   OPKG: inside ~/src/myproject
   OPKG: building map
   OPKG: map: foo.native needs
         some_regexp_lib_with_osr_standard_regexp_library_api
   OPKG: map: some_regexp_lib_with_osr_standard_regexp_library_api
         configured here as latest(pcre) from _requirements
   OPKG: map: latest(pcre) maps to pcre v3.5 from ~/.opkg/map
   OPKG: done building map

We can check which versions have been used in building a target with
the command OPKG composition:

   % OPKG composition foo.native
   foo.native:
     made from map(0f9e8b9d8bbdae7bd9e78ab9d8e7ade83)(OUTDATED)
   map(0f9e8b9d8bbdae7bd9e78ab9d8e7ade83):
     ocaml 3.10.1(nativedynlinkpatch),
     PCRE v3.4(native)

Here OPKG tells us that the current foo.native was built using the a
map file whose digest was 0f9e8b9d8bbdae7bd9e78ab9d8e7ade83 which
instructed OPKG to use ocaml 3.10.1 configured with the
nativedynlinkpatch option, and with PCRE v3.5, in native code.
OUTDATED means that the map has changed.

Now let's rebuild our project:

   % OPKG build foo.native
   OPKG: inside ~/src/myproject
   OPKG: map 0f9e8b9d8bbdae7bd9e78ab9d8e7ade83 outdated by current map
         d78b279f78b3fb804983481834
   OPKG: pcre v3.5 not already checked out
   OPKG: pcre v3.5 available from hg http://ocaml-pcre.org/ rev r450
   ...

And OPKG will fetch pcre v3.5, and rebuild it.

   % OPKG composition foo.native
   foo.native:
     made from map(d78b279f78b3fb804983481834)(CURRENT)
   map(d78b279f78b3fb804983481834):
     ocaml 3.10.1(nativedynlinkpatch),
     PCRE v3.5(native)

What happens if we want to make an improvement to PCRE ?  For
instance, we might want to add a ?drm option to the match_regexp
function.

We simply go into ~/src/opkg/pcre and clone the v3.5 directory into
v3.5-drm:

   % cd ~/src/opkg/pcre VCS clone v3.5 v3.5-drm

We then start hacking into v3.5-drm.  Of course we want to use
myproject to debug it, so we override its map file by adding the
following line into _map_override:

   % cat _map_override
   some_regexp_lib_with_osr_standard_regexp_library_api: pcre v3.5-drm

Meanwhile, an important buffer overflow is fixed in PCRE.  As we have
been hacking the DRM extension under VCS, we can easily merge the
patch without much hassle.

Now we have finished myproject and want to release it.  Unfortunately,
it makes heavy use of the DRM extension.  So we have to release the
drmified PCRE too (maybe also because of licensing requirements).
No worries!

We simply commit our myproject at myproject.org/reopsitory/myproject
along with PCRE v3.5-drm at myproject.org/repository/pcredrm.  But wait!
When another user comiles myproject, how is her OPKG going to find PCRE v3.5-drm?
Simple.  We have added a file _requirements in the myproject directory,
which simply states that:

   pcre-drm updates: http://myproject.org/repository/pcre-drm

This file implicitly adds a repository source (some kind of
/apt/sources.list) to the user's ~/.opkg/resolve file.  Of course,
this line does not affect other projects, so if you override an
important library, it will only affect the program you are currently
compiling.

Later, when the DRM patch is integrated into PCRE v3.6 proper, changing
the requirements or resolution file will let you map PCRE v3.5-drm
onto PCRE v3.6.

The requirements resolution mechanism is incomplete, inconsistent and needs
refinements.  But my aim here was to give a general feeling of what
I have in mind.

One open question:

Do we want a mechanism for having the complete dependency graph,
instead of discovering it on-the-fly, in the process of checking out
and building discovered dependencies?

Dependencies may be dynamic and depend on particular
configuration options selected by the user (i.e. --with-gtk implies
Gtk).

Formalizing the dependency information will need to take configuration
into account.  This means a Gentoo-like system - however that still might
not be powerful enough...
--
Berke Durak


-- 
Berke DURAK

Re: [Caml-list] [OSR] Ports-like package management system

[David Teller]

Ok, I see numerous interesting ideas (I particularly like the idea of
ocamlbuild extended with the ability to download source code from
repositories), but no clear conclusion.

Is there anything clear enough to go forward ? Because if there is, we
can start drafting volunteers to finalize specifications and/or work on
the implementation.

Cheers,
 David

On Tue, 2008-01-29 at 11:56 +0100, Berke Durak wrote:
> Hello,
> 
> Following Markus's message, discussions with Nicolas Pouillard and
> Sylvain Le Gall and others, and of course my previous work in the EDOS
> project, here are some thoughts about package management systems
> (PMSs) for Ocaml.

[ ... ]

> So basically I propose that we improve ocamlbuild to allow for
> multiple plugin files (using dynamic loading) and use that to define a
> BSD ports-like system targeted at developers.

-- 
David Teller
 Security of Distributed Systems
  http://www.univ-orleans.fr/lifo/Members/David.Teller
 Angry researcher: French Universities need reforms, but the LRU act
brings liquidations. 

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 13-02-2008, David Teller <David.Teller@univ-orleans.fr> wrote:
> Ok, I see numerous interesting ideas (I particularly like the idea of
> ocamlbuild extended with the ability to download source code from
> repositories), but no clear conclusion.
>
> Is there anything clear enough to go forward ? Because if there is, we
> can start drafting volunteers to finalize specifications and/or work on
> the implementation.
>

I am not sure there is a clear conclusion about this thread. Well, in
fact, there is ONE solution simple and direct which is to create a
FTP/HTTP area where all upstream tarball can be stored. This is a good
first step for beginning. It will serve all current distributions
(Debian/GODI) by providing an alternate download site. It could also be
the first step to make something more complex.

Concerning doing something better, i am not sure we can reach a
consensus for now.

However, just storing tarball somewhere is not a Package Management
System.

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Berke Durak]

Sylvain Le Gall a écrit :
> On 13-02-2008, David Teller <David.Teller@univ-orleans.fr> wrote:
>> Ok, I see numerous interesting ideas (I particularly like the idea of
>> ocamlbuild extended with the ability to download source code from
>> repositories), but no clear conclusion.
>>
>> Is there anything clear enough to go forward ? Because if there is, we
>> can start drafting volunteers to finalize specifications and/or work on
>> the implementation.
>>
> 
> I am not sure there is a clear conclusion about this thread. Well, in
> fact, there is ONE solution simple and direct which is to create a
> FTP/HTTP area where all upstream tarball can be stored. This is a good
> first step for beginning. It will serve all current distributions
> (Debian/GODI) by providing an alternate download site. It could also be
> the first step to make something more complex.
> 
> Concerning doing something better, i am not sure we can reach a
> consensus for now.
> 
> However, just storing tarball somewhere is not a Package Management
> System.

It is not, but while we try to figure out what a reasonable system might be,
if you can set up a permanent repository for Ocaml code, that would
be already quite useful!  Most links on the Humps are dead.  For instance,
I can't find a copy of Baire.

-- 
Berke DURAK

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 13-02-2008, Berke Durak <berke.durak@exalead.com> wrote:
> Sylvain Le Gall a écrit :
>> 
>> However, just storing tarball somewhere is not a Package Management
>> System.
>
> It is not, but while we try to figure out what a reasonable system might be,
> if you can set up a permanent repository for Ocaml code, that would
> be already quite useful!  Most links on the Humps are dead.  For instance,
> I can't find a copy of Baire.
>

Exactly my point, i think that the most agreed proposition is to have a
kind of central place to store sources. This could be the first step
(storing tarballs at a central place). This will help to "secure"
dependencies by providing an alternate place to find tarball...

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Richard Jones]

On Wed, Feb 13, 2008 at 11:48:24AM +0100, Berke Durak wrote:
> It is not, but while we try to figure out what a reasonable system might be,
> if you can set up a permanent repository for Ocaml code, that would
> be already quite useful!  Most links on the Humps are dead.  For instance,
> I can't find a copy of Baire.

Exactly ... And the lesson here is that this tarball repository must
be mirrored around the world.  There are existing mirrors who will do
this for you, but they won't mirror anything that isn't flat files
accessible by rsync.

Rich.

-- 
Richard Jones
Red Hat

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 13-02-2008, Richard Jones <rich@annexia.org> wrote:
> On Wed, Feb 13, 2008 at 11:48:24AM +0100, Berke Durak wrote:
>> It is not, but while we try to figure out what a reasonable system might be,
>> if you can set up a permanent repository for Ocaml code, that would
>> be already quite useful!  Most links on the Humps are dead.  For instance,
>> I can't find a copy of Baire.
>
> Exactly ... And the lesson here is that this tarball repository must
> be mirrored around the world.  There are existing mirrors who will do
> this for you, but they won't mirror anything that isn't flat files
> accessible by rsync.
>

Don't start again the SCM thread. This is useless. For now there will be
an access through FTP. I think it is fully acceptable to also have a
rsync access, but i need to check this with my host provided. OVH
specify that their dedicated servers should not be used for mirroring
except... (if you subscribe to an option AFAIK) 

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Richard Jones]

On Wed, Feb 13, 2008 at 02:22:59PM +0000, Sylvain Le Gall wrote:
> On 13-02-2008, Richard Jones <rich@annexia.org> wrote:
> > On Wed, Feb 13, 2008 at 11:48:24AM +0100, Berke Durak wrote:
> >> It is not, but while we try to figure out what a reasonable system might be,
> >> if you can set up a permanent repository for Ocaml code, that would
> >> be already quite useful!  Most links on the Humps are dead.  For instance,
> >> I can't find a copy of Baire.
> >
> > Exactly ... And the lesson here is that this tarball repository must
> > be mirrored around the world.  There are existing mirrors who will do
> > this for you, but they won't mirror anything that isn't flat files
> > accessible by rsync.
> >
> 
> Don't start again the SCM thread. This is useless. For now there will be
> an access through FTP. I think it is fully acceptable to also have a
> rsync access, but i need to check this with my host provided. OVH
> specify that their dedicated servers should not be used for mirroring
> except... (if you subscribe to an option AFAIK) 

I wasn't talking about SCM :-)

Here's the University of Kent's policy on mirroring:

  http://www.mirrorservice.org/Help/FeedBack/Mirror
  and some here: http://www.mirrorservice.org/Help/Contact_Us

They'll mirror from HTTP or FTP too, but I suspect they prefer rsync.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Maxence Guesdon]

[-- Attachment #1: Type: text/plain, Size: 1339 bytes --]

Hello,

On Wed, 13 Feb 2008 11:48:24 +0100
Berke Durak <berke.durak@exalead.com> wrote:

> It is not, but while we try to figure out what a reasonable system might be,
> if you can set up a permanent repository for Ocaml code, that would
> be already quite useful!  Most links on the Humps are dead.  For instance,
> I can't find a copy of Baire.

Sorry, buy I can't let you say that, because it's wrong. I made a quick
check (using wget) of the urls associated to each contributions of the caml
humps.

The result[1] is that 84.4% of the urls were found (487 on 577 contribs).
The remaining 15.6% (90 contribs) do not exist any more or did not respond
in time (-T 20 option for wget). So I wouldn't say that "most" links are
dead.

The quality of the links of the hump depends on the information I can gather
from announces on caml-list and mails to caml-hump@inria.fr. I don't crawl
the web all the day to find new or updated contributions.

To improve this quality, the best is to send to caml-hump@inria.fr
information about new/updated/moved contributions, and other information
like licence, topic classification, etc.

Thanks.

Maxence, Caml Hump maintainer.

[1] See the attached file for the links wget failed to retrieve.
Damned, there are even contribs of mine whose urls are deprecated ! I will
send me an email about those...

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: test_contribs.log --]
[-- Type: text/x-log; name=test_contribs.log, Size: 6632 bytes --]

A Concise Introduction to Objective Caml [6] at http://www.csc.vill.edu/~dmatusze/resources/ocaml/ocaml.html failed
AlphaProlog [11] at http://www.cs.cornell.edu/people/jcheney/aprolog/ failed
An example of OCaml/Fortran interface [12] at http://www.dai.ed.ac.uk/homes/williamc/ocaml-fortran/ failed
Automli [18] at http://farrand.net/automli.shtml failed
Baire [19] at http://www.edite-de-paris.com.fr/~fpons/Caml/Baire/index_en.html failed
BNC [31] at http://amaranth.homelinux.org/ocaml/BNC/ failed
XSPP [34] at http://www.kernel.pl/~malekith/xspp/ failed
Camllisp [75] at http://ontil.ihep.su/~vsl/ failed
Hashtbl syntax extension [78] at http://bononia.it/zack/myprograms.en.html#hashtbl_ext failed
Trx [82] at http://pauillac.inria.fr/~chaudhur/ontrx.html failed
Tony [91] at http://www.st.cs.uni-sb.de/~lindig/src/ failed
ChartPak [95] at http://www.havenrock.com/software/chartpak failed
Taglet [103] at http://redwood.ucdavis.edu/~issac/software/ failed
taggage [105] at http://redwood.ucdavis.edu/~issac/taggage-1.1.ml failed
Confluence [107] at http://www.launchbird.com/ failed
Stred [126] at http://pauillac.inria.fr/~chaudhur/ failed
SoCaml [130] at http://redwood.ucdavis.edu/~issac/software/ failed
SCaml [139] at http://www.boblycat.org/~malc/scaml/ failed
Dependent ML [148] at http://www.ececs.uc.edu/~hwxi/DML/DML.html failed
rpc [153] at http://www.ocaml-programming.de/packages/documentation/rpc failed
rmwd [154] at http://ncc.csun.edu/opensource/rmwd/rmwd-description.html failed
Practical hints for using OCaml [164] at http://www.ai.univie.ac.at/~markus/home/ocaml_hints.html failed
Polygen [167] at http://polygen.org/web/Home.444.0.html failed
Polycopié de Programmation Fonctionnelle [168] at http://www-rocq.inria.fr/~pecquet/download/teach/Licence/Programmation_Fonctionnelle/poly.pdf failed
Programming Languages [169] at http://www.cs.jhu.edu/~scott/plbook/book/html/main.html failed
PLAN [172] at http://www.cis.upenn.edu/~switchware/PLAN/ failed
ECaml [182] at http://redwood.ucdavis.edu/~issac/software/ failed
EFuns [183] at http://pauillac.inria.fr/para/cdrom/prog/unix/efuns/eng.htm failed
Ensemble Juke Box [189] at http://www.cs.caltech.edu/~jyh/ejb/index.html failed
FoC [204] at http://www-spi.lip6.fr/foc failed
Fresh Objective Caml [207] at http://www.freshml.org/foc/ failed
GeOCaml [215] at https://gna.org/projects/geocaml/ failed
GwML [225] at http://pauillac.inria.fr/para/cdrom/prog/unix/efuns/eng.htm failed
Hlins [231] at http://www.lsv.ens-cachan.fr/~treinen/hlins/ failed
IceDock [237] at http://www.boblycat.org/~malc/code/ failed
Iox [242] at http://www.wetware.com/jhw/src/ failed
Jabbr [245] at http://mikelin.mit.edu/xmpp/jabbr/ failed
Katmake [248] at http://casse.hugues.free.fr/projects/katmake.html failed
lablglut [251] at http://redwood.ucdavis.edu/~issac/software/ failed
Libml [258] at http://libml.org failed
Lightweight records [259] at http://www.ens-lyon.fr/~noury/lw_records.tgz failed
mail-smtp [264] at http://www.silve.net/caml/smtp-mail failed
ML-Postscript [272] at http://www.eleves.ens.fr:8080/home/george/info/prg/postscript.html failed
Mp3tag [283] at http://www.maxence-g.net/Tools/mp3tag/ failed
NAB (Network in A Box) [284] at http://nab.epfl.ch/ failed
Nativize [285] at http://redwood.ucdavis.edu/~issac/software/ failed
nML [287] at http://ropas.kaist.ac.kr/n/ failed
OOoWrite [298] at http://www.merjis.com/developers/ooowrite/ failed
Missinglib [300] at http://quux.org/devel/missinglib failed
LablGTK2 Tree View Tutorial [322] at http://compiler.kaist.ac.kr/~shoh/ocaml/lablgtk2/treeview-tutorial/ failed
LablGTK2 Tutorial [323] at http://compiler.kaist.ac.kr/~shoh/ocaml/lablgtk2/lablgtk2-tutorial/ failed
Amble [327] at http://home.gna.org/amble/ failed
Surreal [332] at http://www15.ocn.ne.jp/~rodinia/surreal.html failed
GikiWiki [336] at http://wiki.woozle.org/GikiWiki failed
OCaml mode for NEdit [348] at http://floating.caltech.edu/~n8gray/code/index.html failed
Ocaml-MinGW-Maxi [359] at http://ibgs.christoph-bauer.net/~fridolin/omm/ failed
OCaml-TDL [369] at http://home.gna.org/ocamltdl/ failed
ocaml.vim [372] at http://www.ai.univie.ac.at/~markus/home/ocaml_sources.html failed
OCamlBDB [375] at http://osdn.org.ua/~yrashk/ocamlbdb-0.0.1.tar.gz failed
Ocamlcl [377] at ftp://ftp.inria.fr/INRIA/Projects/cristal/Daniel.de_Rauglaudre/ocamlcl/ failed
OcamlConf [378] at http://kenn.frap.net/ocamlconf/ failed
ocamldelaunay [383] at http://redwood.ucdavis.edu/~issac/software/ failed
OcamlDoom [384] at http://pauillac.inria.fr/~pessaux/creations_fr.html failed
OCamlexc [389] at http://caml.inria.fr/ocamlexc/ocamlexc.htm failed
OCamlGettext [393] at http://osdn.org.ua/~yrashk/ocamlgettext-0.0.1.tar.gz\n failed
OCamllex tutorial [398] at http://pllab.kaist.ac.kr/~shoh/ocaml/ocamllex-ocamlyacc/ocamllex-tutorial/index.html failed
OCamlWinPlus [411] at http://dorm.tunkeymicket.com/OCamlWinPlus/Release/ failed
OCamlyacc tutorial [412] at http://pllab.kaist.ac.kr/~shoh/ocaml/ocamllex-ocamlyacc/ocamlyacc-tutorial/index.html failed
odoc_check [418] at http://www.maxence-g.net/odoc_generators/ failed
odoc_fhtml [419] at http://www.maxence-g.net/odoc_generators/ failed
odoc_tdl [420] at http://home.gna.org/ocamltdl/ failed
CoWNFS [428] at http://www.russross.com/CoWNFS.html failed
Pocengine [431] at http://williamkramps.rashitoul.net/pocengine failed
OCaml-ast-analyze [435] at http://webperso.easyconnect.fr/gildor/ocaml-ast-analyze.html failed
Indexy [445] at http://capricorn.dnsalias.org/mike/index/ failed
Caml2tex [450] at http://meunier.pe.free.fr/caml2tex.php failed
Dynaml [465] at http://farrand.net/dynaml.shtml failed
Fjavac [469] at http://www.cis.upenn.edu/~stse/javac/ failed
IMT [485] at http://www.boblycat.org/~malc/imt/ failed
ocfgc [492] at http://www.cis.upenn.edu/~stse/javac/ failed
Fjavac [493] at http://www.cis.upenn.edu/~stse/javac/ failed
OCaml for AIX [494] at http://ibgs.christoph-bauer.net/~fridolin/ocaml-aix.html failed
Kdialog2OCaml [499] at http://www.limsi.fr/Individu/sejourne/distrib.php failed
Xdialog2Ocaml [500] at http://www.limsi.fr/Individu/sejourne/distrib.php failed
Cairo-OCaml [518] at http://www.cairographics.org/cairo_2docaml failed
Cis [519] at http://www.irisa.fr/lande/ferre/software.en.html failed
APC [536] at http://www.boblycat.org/~malc/apc/ failed
Dypgen [539] at http://perso.ens-lyon.fr/emmanuel.onzon/ failed
SOSS [543] at http://www.caterpillarjones.org/soss/ failed
EasyLanguage to C# translator [575] at http://algokit.com/ failed
contribs ok = 487, contribs ko = 90

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Berke Durak]

Maxence Guesdon a écrit :
> Berke Durak <berke.durak@exalead.com> wrote:
>> Most links on the Humps are dead.  For instance,
>> I can't find a copy of Baire.
> 
> Sorry, buy I can't let you say that, because it's wrong. I made a quick
> check (using wget) of the urls associated to each contributions of the caml
> humps.

I should have said "many" instead of "most" but then you wouldn't bother
writing to the list :)

 > The quality of the links of the hump depends on the information I can gather
 > from announces on caml-list and mails to caml-hump@inria.fr. I don't crawl
 > the web all the day to find new or updated contributions.

I appreciate the humps for being a nicely organized and aesthetically-
pleasant official place to put Ocaml stuff, and I know it's your work that made it that way.

However, the model where one person updates the entries for hundreds of contributions can't
and doesn't scale and you just gave numbers proving it.  Do you really want to spend your
days updating link entries?  I don't want you to.  You have better things to do.

Why not just allow contributors to update their fields, in the LinkDB way?
You can of course keep moderating and ensuring that there is no spam or misclassifications.

Having that permanent archive we are talking about is also part of the solution.
-- 
Berke DURAK

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Maxence Guesdon]

On Fri, 15 Feb 2008 10:47:02 +0100
Berke Durak <berke.durak@exalead.com> wrote:

> Maxence Guesdon a écrit :
> > Berke Durak <berke.durak@exalead.com> wrote:
> >> Most links on the Humps are dead.  For instance,
> >> I can't find a copy of Baire.
> > 
> > Sorry, buy I can't let you say that, because it's wrong. I made a quick
> > check (using wget) of the urls associated to each contributions of the caml
> > humps.
> 
> I should have said "many" instead of "most" but then you wouldn't bother
> writing to the list :)

It was a good occasion to pass the message "do not forget to warn caml-hump
about modifications" ;-)

>  > The quality of the links of the hump depends on the information I can gather
>  > from announces on caml-list and mails to caml-hump@inria.fr. I don't crawl
>  > the web all the day to find new or updated contributions.
> 
> I appreciate the humps for being a nicely organized and aesthetically-
> pleasant official place to put Ocaml stuff, and I know it's your work that made it that way.
> 
> However, the model where one person updates the entries for hundreds of contributions can't
> and doesn't scale and you just gave numbers proving it.  Do you really want to spend your
> days updating link entries?  I don't want you to.  You have better things to do.
> 
> Why not just allow contributors to update their fields, in the LinkDB way?
> You can of course keep moderating and ensuring that there is no spam or misclassifications.

It's just to keep it consistent, and the number of adds/updates is not
high, so the time spent on administrating the hump is by far shorter than
the time it would take to develop a registering system and moderate the
modifications.

> Having that permanent archive we are talking about is also part of the solution.

I agree. The hump does not aim to be any kind of repository, just a list of
links with some order. If/when an official place with files and
classification where to find all contribs existed/exists, the hump
would/will be useless.

Maxence

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Stefano Zacchiroli]

On Fri, Feb 15, 2008 at 11:24:01AM +0100, Maxence Guesdon wrote:
> It's just to keep it consistent, and the number of adds/updates is not
> high, so the time spent on administrating the hump is by far shorter than
> the time it would take to develop a registering system and moderate the
> modifications.
> 
> > Having that permanent archive we are talking about is also part of the solution.
> 
> I agree. The hump does not aim to be any kind of repository, just a list of
> links with some order. If/when an official place with files and
> classification where to find all contribs existed/exists, the hump
> would/will be useless.

I agree with your motivation. In the meantime, a quick solution which
can help having more "lively" update notifications can be to just keep
an online version of the list of broken links you've just sent to the
list.

I'm thinking at a link on the main page "broken links", containing a
daily generated list of broken links. That way people, including both
contributors and users, can look at the list and send you an email with
the up to date link. It won't be a wiki, but at least it would enable
users to improve the content and diminish the overall update time of
links .... I think :)

Thanks for your work on The Humps.
Cheers.

-- 
Stefano Zacchiroli -*- PhD in Computer Science ............... now what?
zack@{upsilon.cc,cs.unibo.it,debian.org}  -<%>-  http://upsilon.cc/zack/
(15:56:48)  Zack: e la demo dema ?    /\    All one has to do is hit the
(15:57:15)  Bac: no, la demo scema    \/    right keys at the right time

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Maxence Guesdon]

On Fri, 15 Feb 2008 11:59:37 +0100
Stefano Zacchiroli <zack@upsilon.cc> wrote:

> On Fri, Feb 15, 2008 at 11:24:01AM +0100, Maxence Guesdon wrote:
> > It's just to keep it consistent, and the number of adds/updates is not
> > high, so the time spent on administrating the hump is by far shorter than
> > the time it would take to develop a registering system and moderate the
> > modifications.
> > 
> > > Having that permanent archive we are talking about is also part of the solution.
> > 
> > I agree. The hump does not aim to be any kind of repository, just a list of
> > links with some order. If/when an official place with files and
> > classification where to find all contribs existed/exists, the hump
> > would/will be useless.
> 
> I agree with your motivation. In the meantime, a quick solution which
> can help having more "lively" update notifications can be to just keep
> an online version of the list of broken links you've just sent to the
> list.
> 
> I'm thinking at a link on the main page "broken links", containing a
> daily generated list of broken links. That way people, including both
> contributors and users, can look at the list and send you an email with
> the up to date link. It won't be a wiki, but at least it would enable
> users to improve the content and diminish the overall update time of
> links .... I think :)

It seems a good idea, with some warnings about the result being possibly
wrong because a website can be temporary down... or even the inria network
being down at the bad moment.

I'll add this page soon and announce it on the lis.

Regards,

Maxence

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Ralph Douglass]

[-- Attachment #1: Type: text/plain, Size: 1913 bytes --]

Can you post the list of dead links?  Some people on the list might be able
to provide updated information.

On 2/15/08, Maxence Guesdon <maxence.guesdon@inria.fr> wrote:
>
> Hello,
>
> On Wed, 13 Feb 2008 11:48:24 +0100
>
> Berke Durak <berke.durak@exalead.com> wrote:
>
>
> > It is not, but while we try to figure out what a reasonable system might
> be,
> > if you can set up a permanent repository for Ocaml code, that would
> > be already quite useful!  Most links on the Humps are dead.  For
> instance,
> > I can't find a copy of Baire.
>
>
> Sorry, buy I can't let you say that, because it's wrong. I made a quick
> check (using wget) of the urls associated to each contributions of the
> caml
> humps.
>
> The result[1] is that 84.4% of the urls were found (487 on 577 contribs).
> The remaining 15.6% (90 contribs) do not exist any more or did not respond
> in time (-T 20 option for wget). So I wouldn't say that "most" links are
> dead.
>
> The quality of the links of the hump depends on the information I can
> gather
> from announces on caml-list and mails to caml-hump@inria.fr. I don't crawl
> the web all the day to find new or updated contributions.
>
> To improve this quality, the best is to send to caml-hump@inria.fr
> information about new/updated/moved contributions, and other information
> like licence, topic classification, etc.
>
> Thanks.
>
> Maxence, Caml Hump maintainer.
>
> [1] See the attached file for the links wget failed to retrieve.
> Damned, there are even contribs of mine whose urls are deprecated ! I will
> send me an email about those...
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>
>


-- 
Ralph

[-- Attachment #2: Type: text/html, Size: 2728 bytes --]

Re: [Caml-list] Re: [OSR] Ports-like package management system

[Christophe TROESTLER]

On Fri, 15 Feb 2008 08:35:28 -0500, Ralph Douglass wrote:
> 
> Can you post the list of dead links?  Some people on the list might be able
> to provide updated information.

It was attached to his mail.

ChriS

Re: [Caml-list] Re: [OSR] Ports-like package management system

[David Teller]

On Wed, 2008-02-13 at 10:02 +0000, Sylvain Le Gall wrote:
> I am not sure there is a clear conclusion about this thread. Well, in
> fact, there is ONE solution simple and direct which is to create a
> FTP/HTTP area where all upstream tarball can be stored. This is a good
> first step for beginning. It will serve all current distributions
> (Debian/GODI) by providing an alternate download site. It could also be
> the first step to make something more complex.

Do I understand that this would be combined with dependencies written in
OCamlBuild ?

If you're reading this, Nicolas, how difficult would it be to add a tag
"tgz" and be able to write, say

<my_file.ml>: use_tgz(http://www.ocamlcore.org/global_repository/)

To automatically http-download and unpack a tgz in some central
directory, create a link to whichever directories are contained inside
the tgz and then consider them as subdirectories ? 

That wouldn't be a package management system but, well, it would start
to look like one. At a later stage, this could be completed with
accessing more complex kinds of repositories.

Cheers,
 David

-- 
David Teller
 Security of Distributed Systems
  http://www.univ-orleans.fr/lifo/Members/David.Teller
 Angry researcher: French Universities need reforms, but the LRU act brings liquidations. 

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 13-02-2008, David Teller <David.Teller@univ-orleans.fr> wrote:
> On Wed, 2008-02-13 at 10:02 +0000, Sylvain Le Gall wrote:
>> I am not sure there is a clear conclusion about this thread. Well, in
>> fact, there is ONE solution simple and direct which is to create a
>> FTP/HTTP area where all upstream tarball can be stored. This is a good
>> first step for beginning. It will serve all current distributions
>> (Debian/GODI) by providing an alternate download site. It could also be
>> the first step to make something more complex.
>
> Do I understand that this would be combined with dependencies written in
> OCamlBuild ?
>

Indeed, this could be. But what i call "ocaml-archives" won't be
ocamlbuild related. It will just be archives organized such as that
making possible to use them in an automated way.

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[David Teller]

On Wed, 2008-02-13 at 13:48 +0000, Sylvain Le Gall wrote:
> Indeed, this could be. But what i call "ocaml-archives" won't be
> ocamlbuild related. It will just be archives organized such as that
> making possible to use them in an automated way.
> 
> Regards,
> Sylvain Le Gall

Unless I'm mistaken, this "use_tgz" tag would be sufficient to start and
would be innocuous enough that it wouldn't get in the way of
ocaml-archives, whenever their format gets specified, wouldn't it ?

Cheers,
 David

-- 
David Teller
 Security of Distributed Systems
  http://www.univ-orleans.fr/lifo/Members/David.Teller
 Angry researcher: French Universities need reforms, but the LRU act brings liquidations. 

Re: [OSR] Ports-like package management system

[Sylvain Le Gall]

On 13-02-2008, David Teller <David.Teller@univ-orleans.fr> wrote:
> On Wed, 2008-02-13 at 13:48 +0000, Sylvain Le Gall wrote:
>> Indeed, this could be. But what i call "ocaml-archives" won't be
>> ocamlbuild related. It will just be archives organized such as that
>> making possible to use them in an automated way.
>> 
>> Regards,
>> Sylvain Le Gall
>
> Unless I'm mistaken, this "use_tgz" tag would be sufficient to start and
> would be innocuous enough that it wouldn't get in the way of
> ocaml-archives, whenever their format gets specified, wouldn't it ?
>

I think if Nicolas Pouillard wants/has the time to implement "use_tgz"
this sound a good idea. For now, i think ocaml-archives should only take
care of downloading/making accessible tarballs. Ocaml-archives won't go
into distribution specific details -- for now. 

If we succeed in doing so, we will see what can be done with it.

Regards,
Sylvain Le Gall

Re: [Caml-list] Re: [OSR] Ports-like package management system

[David Teller]

Hint, hint, wink, wink.

... Nicolas ?

Cheers,
 David


On Wed, 2008-02-13 at 14:20 +0000, Sylvain Le Gall wrote:
> I think if Nicolas Pouillard wants/has the time to implement "use_tgz"
> this sound a good idea. For now, i think ocaml-archives should only take
> care of downloading/making accessible tarballs. Ocaml-archives won't go
> into distribution specific details -- for now. 
> 
> If we succeed in doing so, we will see what can be done with it.
> 
> Regards,
> Sylvain Le Gall

-- 
David Teller
 Security of Distributed Systems
  http://www.univ-orleans.fr/lifo/Members/David.Teller
 Angry researcher: French Universities need reforms, but the LRU act brings liquidations.