rixed@happyleptic.org writes:

> Hello!
>
> The more tools are added on top of each others to make life easier for newcomers, and the harder it is to understand what's happening.
>
> So let's get back to the beginning. I'm assuming you know already most of this but I'd rather give a picture of the whole stack so that you know which documentation you need to read to learn the details about various steps.
>
> OCaml compiler without any other helper tool knows very little about where to find the files it needs.
> Following the tradition, it just knows one directory, where the system libraries are located. And it takes everything else from the command line: the inputs (some .ml, .cmx, .cmxa, .o, .a, etc files, some additional paths where to look for libraries if more are needed), and the single output (-o somefile).
>
> You can work only with this, that's very simple, but also quickly cumbersome as the number of libraries you need grow, each requiring other libraries and special options for the compiler, recursively.
> So at some point, you want a package manager: something that organise libraries in your local disk with some additional metadata about what depends on what, so that you only have to say "I want to link with the foobar, preferably the lwt variant".
> This is ocamlfind. To play by the rules, you have to install all the package in a given directory, with an additional small META file that contains the dependencies and various options and other meta data such as a unique name and a description.
> If library authors play by the rules and provide that META file and install their libraries in a standardized manner (which is greatly simplified by ocamlfind command line tool) then everything become as easy as `ocamlfind ocamlc -package 'foo bar baz' my_source.ml`.
>
> This is great and nice, and by and large OCaml library authors have played by the rules for a very long time now (such a tool, to be successful, have to be reliable and _stable_ for a long time ; ocamlfind have been very successful and essential to OCaml popularity, because it has stayed the same for many years ; thank you so much to its maintainers to resist the need to break backward compatibility).
>
> There is still an important annoyance though: for the above to work, you still need libraries foo, bar and baz to be downloaded, compiled, and installed on your local machine. For this, you need a standard package repository. This is OPAM. Opam also have it's small file with metainfo about each package, except this one is called "opam" (or "$package.opam") and states where to get the package from, what command to run to build it, and how to install it.
> Sounds familiar? Yes that's exactly what your Linux distribution does, but it works also on non-linux systems and is also updated much quicker, as there are less bureaucracy involved (notice it solves a human problem not a technical problem here).
>
> There is a bit of redundancy between ocamlfind and opam, because, despite ocamlfind being used by >90% of packages, opam is ocamlfind agnostic and can work, in theory, for non-ocamlfind packages (indeed, it can also be used to deploy about anything not only OCaml packages). So you have to give names, description, dependencies both to ocamlfind and opam.
> But that's only a minor annoyance.
>
> If it really bothers you, then you can use another tool on top of all that, such as dune, that will not only build your project but also write the META and the opam file for you (and more). With the risk that, if something fails at this point, you have all the above stack to unfold to understand what's happening.
>
> If you have read this far then maybe you are interested in the topic of software packaging and deployment.
> Then it must be mentioned that better, more sophisticated package managers do exist, which solve this whole problem once and for all, such as nix or guix, and that hopefully one day we will all abandon our compartmentalised solutions that are popular today and move on to the next level.

Another option is to use pds and hll which I wrote a few years ago and
use for all of my own projects.  A tutorial can be found here:

http://blog.appliedcompscilab.com/2016-Q4/index.html

It's gained some features since that post.

The idea is that pds can transform a TOML file into a Makefile to build
and hll can turn the TOML file into an opam package.  Other tools to
create a .merlin file.

It's pretty simple (pds is implemented in about 800 lines of code
currently + a few dependencies vendored in the repo).  pds supports
parallel builds, incremental builds, building differently based on
environment, and works fine within a monorepo (I've only tested it
against my monorepo where I do all my development so YMMV).  It also
supports building non-ocaml sourcetrees assuming the sourcetree
implements some Makefile targets.  I use this to jump through the Ocaml
-> C -> Ocaml hoops needed to build OCaml bindings with ctypes.  It
definitely has some rough edges since it only works for what I care
about.

It looks like dune has and will have the majority of mindshare, but
pds/hll might be useful to someone who doesn't want to take that leap.
Presumably it should be fairly straightforwad to turn the pds TOML file
into a Dune file as well.  I haven't kept the opam package up-to-date
since I didn't think it'd be that interesting to anyone else but I can
push the latest package this week.

/Malcolm