[Caml-list] opam and versions

[Caml-list] opam and versions

[Jon Ludlam]

Hi all,

Firstly I'd like to add another voice in support of opam - it's been a 
very pleasant experience using it despite it's beta status, and so many 
of the developers here have become dependent on it that we're changing 
our XenServer/XCP build system to build everything using it.

Before opam is officially announced (!) I'd like to raise an issue with 
versioning which people may like to consider. Currently opam has several 
ways of specifying the location of the source tarball for a package. By 
custom, the packages are broadly categorised into rapidly developing 
packages and stable packages. For rapidly developing packages, the 
source is often set to be a pointer to github to the master branch, 
often because no 'stable' version exists. For stable packages, the url 
is often a pointer to a tarball on the developers website, or sometimes 
a tagged release from github.

For our purposes, we need to be able to ensure that our builds are 
reproducible, and hence need to know exactly which versions are 
installed. We had hoped to achieve this by removing or disabling 
packages whose sources were got directly from a master branch in github. 
However, it turns out that some packages that are 'stable' are dependent 
upon these packages, which seems brittle. The question is how to fix it? 
Should the opam repository maintainers require that 'stable' packages 
aren't dependent on 'unstable' ones? Should opam itself be aware of the 
difference and enforce this policy? If someone really wants to release a 
stable version of their thing and it's dependent upon an upstream 
project with only a github repo, should the developer engage the 
upstream devs and request at least a tag, or should they make their own 
tarball/github fork?

Many of these questions are social rather than technical, hence I'm 
posting to the general ocaml list.

Cheers,

Jon

Re: [Caml-list] opam and versions

[Gabriel Kerneis]

On Mon, Oct 22, 2012 at 02:42:12PM +0100, Jon Ludlam wrote:
>  If someone really wants to release a stable version of their thing and it's
>  dependent upon an upstream project with only a github repo, should the
>  developer engage the upstream devs and request at least a tag, or should they
>  make their own tarball/github fork?

If you have developed and tested your build with some specific upstream
snapshot, but prefer not to embed it, you do not even need a tag: you can depend
directly on the relevant upstream commit.  If opam does not support this, it
should definitely be added since it solves the "unresponsive upstream" issue.
Whether it is a good idea to depend on unstable version (and carve this
dependency in stone!) is another issue…

Best,
-- 
Gabriel

Re: [Caml-list] opam and versions

[Anil Madhavapeddy]

Hi Jon,

I ran into exactly this problem with Mirage, and have solved it for now by having two separate OPAM remotes for the stable and development systems.  A continuous build system (Jenkins) installs all packages with and without the development remote, and highlights any dependencies that have crept in on every push.

I would like the ability to specify remote refs/tags/branches too, as Gabriel suggests. This is worth a feature request at http://github.com/OCamlPro/opam, preferably with an intended workflow of what you'd like.

For the purposes of reproducible builds, you may find that 'opam-mk-repo' comes in useful on a checkout out package repository. It populates the archives/ directory of a repository with a tarball containing the exact archive snapshot. If an archive tarball is missing, OPAM attempts to create it (via rsync/git), but if present, is treated as the most current tarball. You could modify the XCP build system to run `opam-mk-repo` regularly and preserve the contents of archives/.

(the latter may well be an undocumented feature of OPAM. Thomas Gazagnaire is on vacation sailing the high seas at the moment. Unless we can get a friendly seagull to deliver this query, it will probably have to wait for his return next week).

Finally, for development purposes, you can also use 'opam pin' to force a particular package to a development version:

$ opam pin mirage-net git://github.com/mirage/mirage-net
$ opam update
# this will refresh all the pinned remotes, either via rsync or git fetch
$ opam upgrade
# this will recompile all dependent libraries with the newest interface
$ opam pin mirage-net none
# this will unpin the package and go back to normal versioning.

With sufficiently cunning pinning, you may not need to use the development remotes for day-to-day use at all.

-anil

On 22 Oct 2012, at 08:42, Jon Ludlam <jonathan.ludlam@eu.citrix.com> wrote:

> Hi all,
> 
> Firstly I'd like to add another voice in support of opam - it's been a very pleasant experience using it despite it's beta status, and so many of the developers here have become dependent on it that we're changing our XenServer/XCP build system to build everything using it.
> 
> Before opam is officially announced (!) I'd like to raise an issue with versioning which people may like to consider. Currently opam has several ways of specifying the location of the source tarball for a package. By custom, the packages are broadly categorised into rapidly developing packages and stable packages. For rapidly developing packages, the source is often set to be a pointer to github to the master branch, often because no 'stable' version exists. For stable packages, the url is often a pointer to a tarball on the developers website, or sometimes a tagged release from github.
> 
> For our purposes, we need to be able to ensure that our builds are reproducible, and hence need to know exactly which versions are installed. We had hoped to achieve this by removing or disabling packages whose sources were got directly from a master branch in github. However, it turns out that some packages that are 'stable' are dependent upon these packages, which seems brittle. The question is how to fix it? Should the opam repository maintainers require that 'stable' packages aren't dependent on 'unstable' ones? Should opam itself be aware of the difference and enforce this policy? If someone really wants to release a stable version of their thing and it's dependent upon an upstream project with only a github repo, should the developer engage the upstream devs and request at least a tag, or should they make their own tarball/github fork?
> 
> Many of these questions are social rather than technical, hence I'm posting to the general ocaml list.
> 
> Cheers,
> 
> Jon
> 
> 
> -- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 

Re: [Caml-list] opam and versions

[Thomas Gazagnaire]

Hi Jon,

> For our purposes, we need to be able to ensure that our builds are reproducible, and hence need to know exactly which versions are installed. We had hoped to achieve this by removing or disabling packages whose sources were got directly from a master branch in github. However, it turns out that some packages that are 'stable' are dependent upon these packages, which seems brittle. The question is how to fix it? Should the opam repository maintainers require that 'stable' packages aren't dependent on 'unstable' ones? Should opam itself be aware of the difference and enforce this policy? If someone really wants to release a stable version of their thing and it's dependent upon an upstream project with only a github repo, should the developer engage the upstream devs and request at least a tag, or should they make their own tarball/github fork?

Before the 1.0 release my plan is:
* to remove the unstable packages in the main opam-repository (ie. every packages should have a stable tarball with a fixed checksum) [1]
* add a way to specify commits/branches for unstable packages if needed. [2]

The current workaround is, as Anil pointed out, to clone opam-repository, use 'opam-mk-repo' at its root to generate a local mirror of opam.ocamlpro.com, and tell opam to add the local repository as a remote:  'opam remote -add local /path/to/your/local/repository'

Then 'opam remote -list' should display the list of repositories and their respective priority (higher is better). You can also tweak ~/.opam/repo/index manually to tell opam to use your local repository only for some packages, for instance the unstable ones (don't forget to run 'opam update' after changing the index file).

For [1], I'm gladly accepting external contributions (for [2] as well actually if someone really wants to hack into opam).

--
Thomas

[1] https://github.com/OCamlPro/opam-repository/issues/171
[2] https://github.com/OCamlPro/opam/issues/267