Re: Problems to get larger user base ... (Re: [Caml-list] OCaml's variables)

[Chet Murthy <murthy.chet@gmail.com>]

I'm glad we're having this discussion.

On Thursday, May 30, 2013 06:52:25 AM Malcolm Matalka wrote:
> I think out would be wrong for opam to try to solve this problem.  There
> are already many tools available for deploying (Ansible, Puppet, Chef,
> Fabric, Capistrano).  Such a later can be build on top of opam of need be.

I think this is incorrect.  Let me explain.

(1) when we look at deploying complex collections of code/libs/data
onto multiple machines, usually we assume that the code has already
been built.

(2) but let's first dispatch the case where the code has -not- been
built.  In such a case, I presume you're proposing that the code be
built on each machine, yes?

  (a) this drastically increases the CPU required to perform upgrades
  and deploys

  (b) but far, far, far more importantly, it means that on each
  machine, a nontrivially complex script runs that builds the actual
  installed binaries.  If that script contains -any- nondeterminism or
  environmental sensitivity, it could produce different results on
  different machines.  The technical term is "version skew".

In scale-out systems, this sort of "skew" is absolutely fatal, because
it means that machines/nodes are not a priori interchangeable.  And
all of fast-fail fault-tolerance depends on nodes being
interchangeable.

(3) But let's say that what you really mean is that we should use
tools like puppet/chef/capistrano to copy collections of
binaries/libs/data to target machines and install them.  These
scripts/recipes are written by some person.  You could have equally
well suggested that that person build Debian packages (or RPMs) of
each OPAM package, writing out all the descriptions and manifests.

And manually specifying all dependencies and requiremeents.

Either way, that person is doing a job that OPAM already does a lot
of, and does quite well.  Gosh, wouldn't it be nice if OPAM could
generate those RPMs?  Well, it's a little more complicated than that,
but really, not much more.  The complexity comes in that you -might-
(I'm not saying I have this part figured out yet) want ways to
-generalize- (say) the camlp5 package so that it could be installed on
many different base OPAM installations.

But setting aside that nice-to-have, imagine that OPAM knew how to
generate RPMs from each package it installed, and from the ocaml+opam
base itself.  You combine those, and you can:

  (i) install ocaml, opam, and a bunch of packages

  (ii) push a button, and out come a pile of RPMs, along with
  dependencies amongst them (and hopefully on the relevant
  environmental RPMs (e.g., libpcre-dev for pcre-ocaml, etc) so that
  you can just stuff those RPMs into a YUM repo, go to a second box,
  and say

    "yum install opam ocaml pcre-ocaml"

  and get everything slurped down and installed, just as if OPAM had
  installed it all, package-by-package.

--chet--

P.S. And this doesn't even get into the unsuitability of chef/puppet
for managing software package installation.  There's a reason that no
distro uses such schemes to install the large and complex sets of
packages needed to run amodern Linux box.  And why there is no Linux
version of Microsorft's "DLL Hell".  Linux distros by and large (and
esp Debian and Ubuntu) have worked hard to make package installation
foolproof -- and chef/puppet etc are anything but.