[Caml-list] (dune/opam) Proper way of vendoring a library inside an application?

[Caml-list] (dune/opam) Proper way of vendoring a library inside an application?

[François Pottier]

Dear fellow OCaml users,

I am trying to figure out how to "vendor" (embark) a private
copy of a library inside a project. (Concretely, I would like
to vendor Fix inside Menhir.) I want this fact to remain
invisible, that is, installing Menhir should not result
in installing or upgrading the user's public copy of Fix.

I have copied the source tree for Fix inside the source tree
for Menhir, and added a (vendored_dirs fix) stanza in Menhir's
toplevel "dune" file.

Compilation works fine, but installation does not seem to work
the way I wish:

* "dune install" installs Menhir *and Fix*, though I would like
   it to install Menhir only.

* Attempting to pin Menhir using "opam pin" fails because opam
   cannot find Fix (inded, Fix is not globally installed).

I have tried removing (public_name fix) from fix/src/dune, but
then the compilation of Menhir fails (Fix cannot be found).

I am confused... what's the proper way of embarking a private
library inside an application without installing it globally?

Thanks!

-- 
François Pottier
francois.pottier@inria.fr
http://cambium.inria.fr/~fpottier/

Re: [Caml-list] (dune/opam) Proper way of vendoring a library inside an application?

[François Bobot]

Le 31/01/2020 à 09:48, François Pottier a écrit :
> I am trying to figure out how to "vendor" (embark) a private
> copy of a library inside a project. (Concretely, I would like
> to vendor Fix inside Menhir.) I want this fact to remain
> invisible, that is, installing Menhir should not result
> in installing or upgrading the user's public copy of Fix.
> 
> I have copied the source tree for Fix inside the source tree
> for Menhir, and added a (vendored_dirs fix) stanza in Menhir's
> toplevel "dune" file.

> I am confused... what's the proper way of embarking a private
> library inside an application without installing it globally?


Will Fix be used for MenhirLib? It is just to understand the impact on users of the library.

Best,

-- 
François

Re: [Caml-list] (dune/opam) Proper way of vendoring a library inside an application?

[François Pottier]

On 31/01/2020 14:08, François Bobot wrote:
> Will Fix be used for MenhirLib? It is just to understand the impact on users of the library.

No, Fix will be used only by the menhir executable,
not by MenhirLib.

-- 
François Pottier
francois.pottier@inria.fr
http://cambium.inria.fr/~fpottier/

Re: [Caml-list] (dune/opam) Proper way of vendoring a library inside an application?

[Jeremie Dimino]

Hi François,

On Fri, Jan 31, 2020 at 8:48 AM François Pottier
<francois.pottier@inria.fr> wrote:
> * "dune install" installs Menhir *and Fix*, though I would like
>    it to install Menhir only.
>
> * Attempting to pin Menhir using "opam pin" fails because opam
>    cannot find Fix (inded, Fix is not globally installed).

These look like bugs in Dune, we'll fix them for the next release.

Cheers,

Jeremie

Re: [Caml-list] (dune/opam) Proper way of vendoring a library inside an application?

[Jeremie Dimino]

Here are the PR fixing these:

https://github.com/ocaml/dune/pull/3074
https://github.com/ocaml/dune/pull/3075

On Mon, Feb 3, 2020 at 10:26 AM Jeremie Dimino <jdimino@janestreet.com> wrote:
>
> Hi François,
>
> On Fri, Jan 31, 2020 at 8:48 AM François Pottier
> <francois.pottier@inria.fr> wrote:
> > * "dune install" installs Menhir *and Fix*, though I would like
> >    it to install Menhir only.
> >
> > * Attempting to pin Menhir using "opam pin" fails because opam
> >    cannot find Fix (inded, Fix is not globally installed).
>
> These look like bugs in Dune, we'll fix them for the next release.
>
> Cheers,
>
> Jeremie



-- 
Jeremie

Re: [Caml-list] (dune/opam) Proper way of vendoring a library inside an application?

[François Pottier]

On 03/02/2020 15:16, Jeremie Dimino wrote:
> Here are the PR fixing these:
> 
> https://github.com/ocaml/dune/pull/3074
> https://github.com/ocaml/dune/pull/3075

Thanks! I will try these changes as soon as possible
(I guess I can create a private fork of dune, merge
these PRs, and test?).

-- 
François Pottier
francois.pottier@inria.fr
http://cambium.inria.fr/~fpottier/

Re: [Caml-list] (dune/opam) Proper way of vendoring a library inside an application?

[Jeremie Dimino]

On Mon, Feb 3, 2020 at 2:53 PM François Pottier
<francois.pottier@inria.fr> wrote:
> Thanks! I will try these changes as soon as possible
> (I guess I can create a private fork of dune, merge
> these PRs, and test?).

The PRs have now been merged, testing would be welcome!

-- 
Jeremie

Re: [Caml-list] (dune/opam) Proper way of vendoring a library inside an application?

[François Pottier]

On 04/02/2020 00:13, Jeremie Dimino wrote:
> The PRs have now been merged, testing would be welcome!

Tested, and seems to work -- both problems fixed. Thanks!
Looking forward to the next release (2.1.4?).

-- 
François Pottier
francois.pottier@inria.fr
http://cambium.inria.fr/~fpottier/

Re: [Caml-list] (dune/opam) Proper way of vendoring a library inside an application?

[Jeremie Dimino]

Thanks for testing!
We just submitted the release of Dune 2.2.0:
https://github.com/ocaml/opam-repository/pull/15795

On Tue, Feb 4, 2020 at 10:11 AM François Pottier
<francois.pottier@inria.fr> wrote:
>
> On 04/02/2020 00:13, Jeremie Dimino wrote:
> > The PRs have now been merged, testing would be welcome!
>
> Tested, and seems to work -- both problems fixed. Thanks!
> Looking forward to the next release (2.1.4?).
>
> --
> François Pottier
> francois.pottier@inria.fr
> http://cambium.inria.fr/~fpottier/



-- 
Jeremie

Re: [Caml-list] dune for OCaml < 4.07?

[François Pottier]

On 06/02/2020 23:33, Jeremie Dimino wrote:
> We just submitted the release of Dune 2.2.0:
> https://github.com/ocaml/opam-repository/pull/15795

Thanks Jérémie. This seems to work great.

However, I realize that recent versions of dune require OCaml 4.07.
Menhir is supposed to be compatible with OCaml all the way back to
4.02.3. Is there any way to install dune 2.2.0 with OCaml < 4.07?

--
François Pottier
francois.pottier@inria.fr
http://cambium.inria.fr/~fpottier/

RE: [Caml-list] dune for OCaml < 4.07?

[David Allsopp]

François Pottier wrote:
> On 06/02/2020 23:33, Jeremie Dimino wrote:
> > We just submitted the release of Dune 2.2.0:
> > https://github.com/ocaml/opam-repository/pull/15795
> 
> Thanks Jérémie. This seems to work great.
> 
> However, I realize that recent versions of dune require OCaml 4.07.
> Menhir is supposed to be compatible with OCaml all the way back to 4.02.3.
> Is there any way to install dune 2.2.0 with OCaml < 4.07?

TL;DR for installing Menhir on OCaml 4.02-4.06 using opam, you don't need to do anything.

Dune 2.0.0+ requires at least OCaml 4.07 to build, but the dune package in opam does not. More detail in case it's relevant to be able to do that without opam...

For OCaml 4.02-4.06, Dune depends on the ocaml-secondary-compiler package[1], which installs a separate copy of OCaml 4.08.1 in your switch which is accessed using `ocamlfind -toolchain secondary` and that's configured in the ocamlfind-secondary package. Dune's bootstrap natively looks for the ocamlfind secondary toolchain if the OCaml version is < 4.07, so opam is not required, you just have to configure ocamlfind manually if not using it. So installing without opam requires building (or having) a later of version of OCaml in a different path, and then creating the configuration file for ocamlfind so that it can use it.

HTH,


David


[1] https://github.com/ocaml/opam-repository/tree/master/packages/ocaml-secondary-compiler/ocaml-secondary-compiler.4.08.1
[2] https://github.com/ocaml/opam-repository/tree/master/packages/ocamlfind-secondary/ocamlfind-secondary.1.8.1

Re: [Caml-list] dune for OCaml < 4.07?

[François Pottier]

Hello David,

> TL;DR for installing Menhir on OCaml 4.02-4.06 using opam, you don't need to do anything.

Thanks for your prompt reply, which sounds reassuring!

However, attempting to install dune 2.2.0 on a pre-4.07 switch
(for instance, 4.02.3) results in the following failure:

   $ opam install dune.2.2.0
   The following dependencies couldn't be met:
     - dune → ocaml >= 4.07 → ocaml-variants >= 4.07.0
         conflict with the base packages of this switch
     - dune → ocaml >= 4.07 → ocaml-system >= 4.07.0
         unmet availability conditions, e.g. sys-ocaml-version = "4.07.0"
     - dune → ocaml >= 4.07 → ocaml-base-compiler >= 4.07.0
         base of this switch (use `--unlock-base' to force)

I have run "opam update" first. Is there something else I should do
to convince opam that it is possible to install dune?

--
François Pottier
francois.pottier@inria.fr
http://cambium.inria.fr/~fpottier/

RE: [Caml-list] dune for OCaml < 4.07?

[David Allsopp]

François Pottier wrote:
> > TL;DR for installing Menhir on OCaml 4.02-4.06 using opam, you don't
> > need to do anything.
> 
> Thanks for your prompt reply, which sounds reassuring!
> 
> However, attempting to install dune 2.2.0 on a pre-4.07 switch (for
> instance, 4.02.3) results in the following failure:
> 
>    $ opam install dune.2.2.0
>    The following dependencies couldn't be met:
>      - dune → ocaml >= 4.07 → ocaml-variants >= 4.07.0
>          conflict with the base packages of this switch
>      - dune → ocaml >= 4.07 → ocaml-system >= 4.07.0
>          unmet availability conditions, e.g. sys-ocaml-version = "4.07.0"
>      - dune → ocaml >= 4.07 → ocaml-base-compiler >= 4.07.0
>          base of this switch (use `--unlock-base' to force)
> 
> I have run "opam update" first. Is there something else I should do to
> convince opam that it is possible to install dune?

Oops :$ - https://github.com/ocaml/opam-repository/pull/15820 !

Best,


D

Re: [Caml-list] (dune/opam) Proper way of vendoring a library inside an application?

[Richard W.M. Jones]

On Fri, Jan 31, 2020 at 09:48:48AM +0100, François Pottier wrote:
> 
> Dear fellow OCaml users,
> 
> I am trying to figure out how to "vendor" (embark) a private
> copy of a library inside a project. (Concretely, I would like
> to vendor Fix inside Menhir.) I want this fact to remain
> invisible, that is, installing Menhir should not result
> in installing or upgrading the user's public copy of Fix.
> 
> I have copied the source tree for Fix inside the source tree
> for Menhir, and added a (vendored_dirs fix) stanza in Menhir's
> toplevel "dune" file.
> 
> Compilation works fine, but installation does not seem to work
> the way I wish:
> 
> * "dune install" installs Menhir *and Fix*, though I would like
>   it to install Menhir only.
> 
> * Attempting to pin Menhir using "opam pin" fails because opam
>   cannot find Fix (inded, Fix is not globally installed).
> 
> I have tried removing (public_name fix) from fix/src/dune, but
> then the compilation of Menhir fails (Fix cannot be found).
> 
> I am confused... what's the proper way of embarking a private
> library inside an application without installing it globally?

I wish you wouldn't do this.  Bundling libraries like this causes
problems for packagers because we can never be sure what (possibly out
of date or insecure) version of a library that we package elsewhere
has been bundled in another place.

As is this is already in opam, what are you trying to achieve here?

Rich.

Re: [Caml-list] (dune/opam) Proper way of vendoring a library inside an application?

[François Pottier]

Hi,

On 29/02/2020 09:41, Richard W.M. Jones wrote:
> As is this is already in opam, what are you trying to achieve here?

I want to maintain the property that Menhir depends on no external
libraries, as this makes it easier for some people (who do not use
opam) to install.

Also, this allows me to know exactly which version of fix I am
using, so two versions of Menhir that have the same version
number are guaranteed to work in the same way; that wouldn't
necessarily be true otherwise.

I don't see how it could cause any packaging problem; it should
be transparent. The copy of Fix embedded inside Menhir is used
when Menhir is installed and is immediately thrown away.

-- 
François Pottier
francois.pottier@inria.fr
http://cambium.inria.fr/~fpottier/

Re: [Caml-list] (dune/opam) Proper way of vendoring a library inside an application?

[François Bobot]

Le 29/02/2020 à 12:20, François Pottier a écrit :
> I don't see how it could cause any packaging problem; it should
> be transparent. The copy of Fix embedded inside Menhir is used
> when Menhir is installed and is immediately thrown away.
> 

Even if it is perhaps not applicable for Fix which is a small library, without attack surface.
Generally if there is a security bug in Fix, distributions don't want to need to patch it in all the
package which vendor Fix. Patching Fix once is simpler, more efficient and safer.

But for a distribution removing this vendor directory just mean to remove it, no other modifications
are needed; dune will then used the installed dependency. Package creator could look at
`(vendored_dirs vendor)` to find those directories. Of course the version can be different from the
last version of Fix. But to choose common version is usually the hurdle of packagers (which we
should strive not to burden more!).

Best,

-- 
François