Re: [Caml-list] Build-/Installation-Tools - not enogh of them?

[Louis Gesbert <louis.gesbert@ocamlpro.com>]

[-- Attachment #1: Type: text/plain, Size: 9128 bytes --]

Thanks all for this input. While working on the intrinsics and details of the tools, it's easy to lose the big picture, and the very important point of view of the newcomers.

So first, as the main developper of opam 2.0, I'd like to say that we have been putting a lot of work in it, and a large part of the effort was to improve for convenience, and the many use-cases that weren't supported before — including, but not limited to, reproducible build environments, and project-local sandboxes, a.k.a. switches.

The documentation, however, is severly lacking at the moment on all these new features, and the preferred and simplest ways to accomplish all the basic tasks. For all asking about the detailed formats, we have a fairly complete manual [1], and the API should be fairly well documented [2], but indeed, it's way too detailed to be the first documentation you would get to.

Let me assure you however that everything is slowly getting into place for an easier approach for everyone. I'll go below through the "typical workflow" you propose, checking what is here or not, but let's first focus on the users rather than the developers:

- installing ocaml:
  indeed, if not easily available for your system, the easiest is to install opam, then just run `opam init`. (Yes, we should be explicit in the doc that _this_ is the command that will install the compiler)

- installing a given package, and assuming opam is installed and initialised:

   * if in the repository, just one `opam install` command should be all you need

   * otherwise, if your source is available somewhere and contains an opam package definition file, `opam pin URL` is everything you need (URL pointing to an archive, or a git repository, etc.). We could merge this use-case into `opam install` for better discoverability.

   * if not and/or you want to build the project manually from a clone, the support has been much improved in opam 2, so that you can for example document specific pinned dependencies, or a "locked" development state (see opam-lock [3] to do that automatically). Then e.g. `opam switch create . --locked` will recreate a local switch with the exact same development configuration, and install the project in it. `opam install . --locked` also works, if you don't want the local sandbox.

- it has been mentionned already in this thread, but the `opam bundle` plugin can make distribution easier by including the whole OCaml + opam + package environment in a single, self-building self-extracting archive. At the cost of rebuilding everything on installation. A new release is pending [4]. Yes, it's yet another tool, but with its straight-forward interface and everyting explained in its 100-lines, included manpage, I find the criticism uncalled for. Not a silver bullet by any means, but it fits some use-cases.

As for using wrapping Makefiles, they are nice for simple build-system calls, and I like them if only to document the entry-point, but shouldn't IMHO mess with the packaging system. Note also that the main purpose of `opam` files is actually to document the building commands of any project, taking into account all OS specificities, and in an easily understandable format. I personally find that having clear and simple build instructions arout the top of the README is enough.

Once properly documented from the opam side (huh), I expect project maintainers to be able to put simple setup + installation instructions at the top of their READMEs, so that users who don't care about OCaml or opam just need to copy-paste them to get the environment setup and the project compiled. As far as I understand it now, this is where the problem really stands. To avoid having to look anything up or learning about exotic build system, this is the best compromise IMHO.

I'd also like to point out that this is not specific to OCaml, and I believe all language package managers / build systems suffer from these issues: I for one struggle every time I have to use something building with NPM, and they don't generally provide Makefiles. Of course, with a tool as popular as NPM, the problem is less visible because you have to go through it anyway. So we do need to improve documentation and simplify basic workflows as much as possible, but expecting people to work with OCaml without learning any of the tooling is unrealistic (unless they stick to an online IDE or e.g. Learn-OCaml, and even that is tooling in some form).


Let me now go through your "typical workflow":

---

> cd some-ocaml-proj
> opam install # Switches compiler if necessary and installs and locally
> caches package dependencies

You can do this with `opam switch create .`
Since "if necessary" is pretty subjective, just run `opam install .` if you prefer to share the environment with other projects.

> opam build

`opam install .`

> opam run # Automatically builds if necessary

there is no package←→executable bijection, so I don't see how this would work? (same as for OS-level packages)
see below, but this might be `dune exec <command>`

> opam test # Ditto

indeed here we enter the domain where the separation between build system and packaging system can hurt. You can run `opam install . --with-test`, but probably want `dune runtest` instead.

> opam package # Ditto; --upload option can immediately upload to opam

at this point you must already have a package definition available ? Or do you mean creating a release archive ?
If your source is hosted on Github, you only need to push a tag and run `opam publish` (you otherwise need to provide an URL for the source archive and that's it).

> opam doc # Builds documentation with ocamldoc or whatever
> opam login -u user -p password

I am not sure what you have in mind here. `opam publish` will go through Oauth authentification with Github for submitting your new package.

---


As one last note, let me mention that we are right now discussing:
- better integration of opam and dune
- integration of system dependency handling ("depexts") into opam


Hope this helps, feedback and questions welcome.

Louis Gesbert — OCamlPro




[1] https://opam.ocaml.org/doc/Manual.html
[2] https://opam.ocaml.org/doc/api/index.html
[3] https://github.com/AltGr/opam-lock
[4] https://github.com/ocaml/opam-repository/pull/13064

> - Yawar Amin, 26/11/2018 11:41 -
> If anyone would like to chime in and say that OCaml build and packaging
> system is not that complicated, I would recommend first looking at
> https://github.com/rizo/awesome-ocaml#package-management . IMHO we need to
> seriously look at consolidating efforts around OPAM for package management,
> packaging, building, testing and running. All the serious language-specific
> package managers do it, it's a proven strategy and it simplifies life for
> the developer.
> 
> This could be a typical workflow:
> 
> cd some-ocaml-proj
> opam install # Switches compiler if necessary and installs and locally
> caches package dependencies
> opam build
> opam run # Automatically builds if necessary
> opam test # Ditto
> opam package # Ditto; --upload option can immediately upload to opam
> opam doc # Builds documentation with ocamldoc or whatever
> opam login -u user -p password
> 
> Regards,
> 
> Yawar
> 
> On Mon, Nov 26, 2018 at 5:15 AM Oliver Bandel <oliver@first.in-berlin.de>
> wrote:
> 
> > Hello,
> >
> > a while ago it looked like there were not enough build- and
> > installation-tools
> > for OCaml. I remember some discussions about that.
> >
> > Now it seems to me that there are a lot of them.
> > So, developers can pick the one they know about.
> >
> > For all these tools there might be good reasons to use them, and those
> > developers who looked at these tools and choose them for their projects,
> > will
> > know them well enough.
> >
> > The situation differs, if one wants to package the written software,
> > and one needs to know many of those tools, just to compile the stuff.
> > So, when one just wants to compile and install some software,
> > just for that, it would take much effort to learn the different
> > build-tools.
> >
> > So, packaging has become more complicated, even though for the developers
> > these tools may save time.
> >
> > It would be nice if people who used one of the many new building tools
> > could provide a Makefile that allows just to type
> > "make" and "make install", instead of expecting everyone who wants to
> > compile
> > the software to first learn just-another-build-tool.
> >
> > Also it would be good, to mention early, which installation tools
> > (make-dependencies)
> > are in use, and too mention needed packages (opam or others) to just build
> > the stuff.
> >
> > Thanks and regards,
> >   Oliver Bandel
> >
> > --
> > Caml-list mailing list.  Subscription management and archives:
> > https://sympa.inria.fr/sympa/arc/caml-list
> > https://inbox.ocaml.org/caml-list
> > Forum: https://discuss.ocaml.org/
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
> >
> 
> 

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 488 bytes --]