Re: [Caml-list] GODI is shutting down

[Anil Madhavapeddy <anil@recoil.org>]

On 24 Jul 2013, at 08:58, Markus Mottl <markus.mottl@gmail.com> wrote:

> My guess is that you are not totally happy with Oasis at this point so
> this may be a topic to address.  Being an Oasis user, I would say that
> the Oasis tools may not be as mature yet as one would hope, but the
> Cabal-like package specification syntax seems decent enough to be
> supported by OPAM.  It really shouldn't be that hard to automate the
> process of updating OPAM metadata (including version constraints)
> based on changes in Oasis package specifications.

I think that's fair, but the devil is in the details. The core of OASIS
is sound, but is hampered by its need to build on top of ocamlfind
and ocamlbuild, and the ensuing impedance mismatches that follow (e.g.
manually editing _tags files, and trying to debug which tool a failure
happens in is an advanced operation). OPAM packages represent an output
collection of (several optionally built) ocamlfind packages, and 
OASIS sits awkwardly in between these two right now, with ocamlbuild
crashing the party and getting drunk in the kitchen.

OASIS is structured very well to help get around this situation though.
It has a library that takes care of parsing the _oasis file, and then a
plugin generator that currently uses ocamlbuild to drive this specification
into concrete outputs.  Meanwhile, the OCaml compiler has been exposing
more of its innards via compiler-libs. Leo and I are taking advantage of
this by trying to folding all of these layers into a new "packaging-friendly"
OCaml driver that statically links:

- all the camlp4 libraries required for a project, and directly calls it
  instead of going via -pp (avoiding a fork/marshal).
- links to OPAM/findlib to find dependencies directly, avoiding a shell
  out to ocamlfind.
- can publish a static schedule of its activities to help Makefiles
  figure out what actually got built (similar to the recent ocamlbuild
  request).

The idea is to have a "ocamlmkcompiler" that builds a binary for-purpose,
as "ocamlmktop" does today for toplevels.  It shouldn't even need any
patches to the official compiler, thanks to compiler-libs.  More to the
point, it should make it far easier to build a camlp4-aware js_of_ocaml
toplevel, to fit into the experiments that OCamlPro is doing with web
IDE interfaces.

Anyway, these are still experiments and quite likely to run into unexpected
issues as we build them, and we'll have to consider all the alternatives
before folding them all into OPAM.

Right now, OPAM dictates the *minimal* possible set of requirements for
a package since it just uses shell commands.  It should be possible to
specialise this for common cases such as a OASIS library, but it does
require some careful thought first.  Luckily, we can use OPAM itself to
conduct such experiments [1].

All such discussions and experiments are very welcome on the mailing
list (http://lists.ocaml.org/listinfo/opam-devel) to help establish
everyone's use cases.  Daniel Buenzli's input has been particularly
interesting to date, as his libraries adopt a very minimal shell-script
driven approach that doesn't use OASIS, but still distributes META files.
That's just one example of someone that doesn't use OASIS directly.
The Core OASIS files are also a bit special due to their configuration
requirements.  Xen/Mirage is also quite demanding in terms of the 
compilation environment.

Here's my goal: I want to have near instant-rebuilds of Mirage packages.
I know it's possible to pull this off with the existing compiler tools,
since the monolithic Mirage tree (with lovingly crafted ocamlbuild rules
and judicious use of native code compiler binaries) could rebuild the
entire webserver in ~10s in parallel.  Nowadays, with OPAM, ocamlbuild,
ocamlfind and camlp4, it takes around 2-3 minutes, which just isn't
good enough.

-anil

[1] http://opam.ocamlpro.com/pkg/debug-camlp4-log.base.html: I hacked
up a debug camlp4 to trace all the uses of camlp4 in the OPAM repository,
during the wg-camlp4 discussion on the topic of where to go next with it.
This just required doing a mass OPAM install, drinking some coffee, and
reading the resulting trace logs.