[Caml-list] Library installation procedure (for use with OPAM)?

[Caml-list] Library installation procedure (for use with OPAM)?

[Anthony Tavener]

[-- Attachment #1: Type: text/plain, Size: 940 bytes --]

For local libraries, my "make install" amounts to copying the relevant
files into the system OCaml directory (ie. /usr/local/lib64/ocaml).
Terrible, I know.

Now I'm switching to using OPAM for general use, and packaging up local
libraries so that OPAM can be aware of them and install them in its managed
environment. But I figure OPAM will be expecting a more sane,
environment-sensitive install... right? Otherwise it must do some voodoo.

Is there a simple example of an appropriate install process suitable for
use with OPAM? I've tried looking at a few existing packages and the
build+install process, as usual, is convoluted... not as obscure as
autotools, but different for each project. Is findlib still applicable? Or
some other means to query the system install directory?

Maybe someone has blog/page covering "library build/install/packaging
suitable for the OCaml ecosystem, from .ml to OPAM"?

Thank-you for any help!
 -Tony

[-- Attachment #2: Type: text/html, Size: 1137 bytes --]

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[Stéphane Glondu]

Le 08/12/2013 21:55, Anthony Tavener a écrit :
> For local libraries, my "make install" amounts to copying the relevant
> files into the system OCaml directory (ie. /usr/local/lib64/ocaml).
> Terrible, I know.

I would recommend using "ocamlfind install". It figures out by itself
where to put files, and is easily customizable for package managers.


Cheers,

-- 
Stéphane

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[John Whitington]

Hi Anthony,

Anthony Tavener wrote:
> For local libraries, my "make install" amounts to copying the relevant
> files into the system OCaml directory (ie. /usr/local/lib64/ocaml).
> Terrible, I know.
>
> Now I'm switching to using OPAM for general use, and packaging up local
> libraries so that OPAM can be aware of them and install them in its
> managed environment. But I figure OPAM will be expecting a more sane,
> environment-sensitive install... right? Otherwise it must do some voodoo.
>
> Is there a simple example of an appropriate install process suitable for
> use with OPAM? I've tried looking at a few existing packages and the
> build+install process, as usual, is convoluted... not as obscure as
> autotools, but different for each project. Is findlib still applicable?
> Or some other means to query the system install directory?

OCamlMakefile knows how to use ocamlfind for installation. I assume the 
other popular build systems do too.

For example, here's the makefile for "cpdf", providing "make" and "make 
install" targets. The "PACKS" line tells OCamlMakefile which ocamlfind 
packages to depend on. "install : libinstall" tells it to install things 
as an ocamlfind package.


# Build the cpdf command line tools and top level
MODS = cpdfstrftime cpdf cpdfcommand

SOURCES = $(foreach x,$(MODS),$(x).ml $(x).mli) cpdfcommandrun.ml

RESULT = cpdf
ANNOTATE = true
PACKS = camlpdf

OCAMLNCFLAGS = -g
OCAMLBCFLAGS = -g
OCAMLLDFLAGS = -g

all : native-code native-code-library byte-code-library top htdoc

LIBINSTALL_FILES = cpdf.a cpdf.cma cpdf.cmxa \
$(foreach x,$(MODS),$x.mli) $(foreach x,$(MODS),$x.cmi)

install : libinstall

-include OCamlMakefile


The 'opam' file is then easy:


opam-version: "1"
maintainer: "contact@coherentgraphics.co.uk"
build: [
   [make]
   [make "install"]
]
remove: [["ocamlfind" "remove" "cpdf"]]
depends: [
   "ocamlfind"
   "camlpdf"
]

-- 
John Whitington

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[Daniel Bünzli]

Le dimanche, 8 décembre 2013 à 21:55, Anthony Tavener a écrit :
> Is there a simple example of an appropriate install process suitable for use with OPAM? I've tried looking at a few existing packages and the build+install process, as usual, is convoluted... not as obscure as autotools, but different for each project. Is findlib still applicable? Or some other means to query the system install directory?


For my packages I now generate an opam .install file at the root of the distribution. This .install file indicates the type of build artefacts you generated and where they are located. This lets opam figure out where it should put that. The advantage over using `findlib install` is that it also support installing binaries, man pages, documentation, configuration files, etc. Note that you still want your users to be able to use findlib for using your package so be sure to also install a META file (no need to use `findlib install` for that).  
  
At the moment for achieving this I use a very simple build script [2] that allows you to describe what you want (see [3]), builds what you describe with ocamlbuild and generates a corresponding .install file. The resulting opam file for this example is here [4] (the pkg-git invocation is for massaging the checkout when you pin the package).

Best,

Daniel

[1] See section 2.2.3 in https://github.com/OCamlPro/opam/raw/master/doc/dev-manual/dev-manual.pdf
[2] https://github.com/dbuenzli/xmlm/blob/master/pkg/pkg-builder
[3] https://github.com/dbuenzli/xmlm/blob/master/pkg/build
[4] https://github.com/dbuenzli/xmlm/blob/master/opam

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[Sebastien Mondet]

[-- Attachment #1: Type: text/plain, Size: 2450 bytes --]

On Mon, Dec 9, 2013 at 10:04 AM, Daniel Bünzli
<daniel.buenzli@erratique.ch>wrote:

> Le dimanche, 8 décembre 2013 à 21:55, Anthony Tavener a écrit :
> > Is there a simple example of an appropriate install process suitable for
> use with OPAM? I've tried looking at a few existing packages and the
> build+install process, as usual, is convoluted... not as obscure as
> autotools, but different for each project. Is findlib still applicable? Or
> some other means to query the system install directory?
>
>
> For my packages I now generate an opam .install file at the root of the
> distribution. This .install file indicates the type of build artefacts you
> generated and where they are located. This lets opam figure out where it
> should put that. The advantage over using `findlib install` is that it also
> support installing binaries, man pages, documentation, configuration files,
> etc. Note that you still want your users to be able to use findlib for
> using your package so be sure to also install a META file (no need to use
> `findlib install` for that).
>
>

hi

Opam's .install files should be used only to fix packages whose
installation is already broken or unsufficient, shouldn't it?

If you rely *only* on the .install, your package can be installed only with
opam, it will break for other users (GODI, WODI, Manual Package
Management™, etc.).
I think it's good hygiene to at least provide an ocamlfind-based
installation of the libraries.


Cheers
Seb






> At the moment for achieving this I use a very simple build script [2] that
> allows you to describe what you want (see [3]), builds what you describe
> with ocamlbuild and generates a corresponding .install file. The resulting
> opam file for this example is here [4] (the pkg-git invocation is for
> massaging the checkout when you pin the package).
>
> Best,
>
> Daniel
>
> [1] See section 2.2.3 in
> https://github.com/OCamlPro/opam/raw/master/doc/dev-manual/dev-manual.pdf
> [2] https://github.com/dbuenzli/xmlm/blob/master/pkg/pkg-builder
> [3] https://github.com/dbuenzli/xmlm/blob/master/pkg/build
> [4] https://github.com/dbuenzli/xmlm/blob/master/opam
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 3688 bytes --]

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[Daniel Bünzli]

Le lundi, 9 décembre 2013 à 16:23, Sebastien Mondet a écrit :

> Opam's .install files should be used only to fix packages whose installation is already broken or unsufficient, shouldn't it?

Where did you get that from ?  

A .install file is a good way of specifying what you want to install without having to replicate any installation logic in your package which is good when you develop and distribute many packages and need to scale.  
  
> If you rely *only* on the .install, your package can be installed only with opam, it will break for other users (GODI, WODI, Manual Package Management™, etc.).

It can be installed by anything that understands what a .install file is, there's nothing specific to opam here. Godi is shutting down, Wodi could be make to understand .install file. Manual Package Management™, the file format of .install is very simple you could perfectly develop have a command line tool that reads it and issues the appropriate cp commands, I'm personally no longer interested in that form of package management and don't have the time to support it.  
  
> I think it's good hygiene to at least provide an ocamlfind-based installation of the libraries.

As I said `ocamlfind install` doesn't allow you to install binaries, man pages, configuration files, documentation etc. With a .install file you can specify what you want to be installed to the classic unix paths (bin, etc, man, share, etc.).  

Best,

Daniel

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[Sebastien Mondet]

[-- Attachment #1: Type: text/plain, Size: 2467 bytes --]

On Mon, Dec 9, 2013 at 10:57 AM, Daniel Bünzli
<daniel.buenzli@erratique.ch>wrote:

>
>
> Le lundi, 9 décembre 2013 à 16:23, Sebastien Mondet a écrit :
>
> > Opam's .install files should be used only to fix packages whose
> installation is already broken or unsufficient, shouldn't it?
>
> Where did you get that from ?
>
> A .install file is a good way of specifying what you want to install
> without having to replicate any installation logic in your package which is
> good when you develop and distribute many packages and need to scale.
>
> > If you rely *only* on the .install, your package can be installed only
> with opam, it will break for other users (GODI, WODI, Manual Package
> Management™, etc.).
>
> It can be installed by anything that understands what a .install file is,
> there's nothing specific to opam here. Godi is shutting down, Wodi could be
> make to understand .install file. Manual Package Management™, the file
> format of .install is very simple you could perfectly develop have a
> command line tool that reads it and issues the appropriate cp commands, I'm
> personally no longer interested in that form of package management and
> don't have the time to support it.
>
>

So, if everybody else have to implement them, opam's install files become a
new standard for the OCaml cummunity? Do most of us agree? (I'm not against
it, I just don't want to rely on 50 different standards).

(BTW, since you generate those .install files you could have generated
install.sh files doing the 'cp's as easily ;) )


> I think it's good hygiene to at least provide an ocamlfind-based
> installation of the libraries.
>
> As I said `ocamlfind install` doesn't allow you to install binaries, man
> pages, configuration files, documentation etc. With a .install file you can
> specify what you want to be installed to the classic unix paths (bin, etc,
> man, share, etc.).
>
>
yes but in practice `ocamlfind install` is enough to make most dependencies
of a given package work (a few times a `cp exec $BIN` is also needed
indeed).

(A while ago, before opam, a new version of GODI broke on our main platform
because of some ocamlnet configuration bug, but I was able to reinstall
ocaml and about 25 third-party libraries and syntax extensions from scratch
without too much pain → we see great value in that package-management
independence)






> Best,
>
> Daniel
>
>
>

[-- Attachment #2: Type: text/html, Size: 3278 bytes --]

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[Daniel Bünzli]

Le lundi, 9 décembre 2013 à 17:39, Sebastien Mondet a écrit :
> So, if everybody else have to implement them, opam's install files become a new standard for the OCaml cummunity?  

Frankly besides maybe wodi on windows I'm not so sure I see something reasonably competing with opam at that point, so that "everybody" may not include a lot of persons. Personally I don't have time to care for alternatives and I do what minimizes my admin/maintenance time at most.
  
> Do most of us agree? (I'm not against it, I just don't want to rely on 50 different standards).

I don't think it's a question of agreeing, usage will tell. Besides I don't see another standard, maybe `ocamlfind install` but it's limited in what it can install (and I hope that eventually the annoying two-headed structure ocamlfind/opam will disapear, e.g. the notion of ocamlfind package could be merged in the compiler).
  
> (BTW, since you generate those .install files you could have generated install.sh (http://install.sh) files doing the 'cp's as easily ;) )

Sure but that's one more thing to maintain across all my packages. Besides I hope that eventually a good build tool will emerge for OCaml and can generate these .install files for me, I'm still not fond of having to maintain and copy this pkg-builder script across all my packages.  
  
> (a few times a `cp exec $BIN` is also needed indeed).

About half of my packages do that. Besides since with package managers we no longer see the tarballs I think it's a good practice to at least install the CHANGES, README and the sample code in the doc directory of a package.

> (A while ago, before opam, a new version of GODI broke on our main platform because of some ocamlnet configuration bug, but I was able to reinstall ocaml and about 25 third-party libraries and syntax extensions from scratch without too much pain → we see great value in that package-management independence)
Why not. I guess you can see the value of having each of these libraries generating a .install file that you can process with a simple command line tool rather than have to deal with the idiosyncrasies of each custom (usually semi-broken) install procedure.

Best,

Daniel

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[Stéphane Glondu]

Le 09/12/2013 18:37, Daniel Bünzli a écrit :
>> So, if everybody else have to implement them, opam's install files
>> become a new standard for the OCaml cummunity?
> 
> Frankly besides maybe wodi on windows I'm not so sure I see something
> reasonably competing with opam at that point, so that "everybody" may
> not include a lot of persons. Personally I don't have time to care
> for alternatives and I do what minimizes my admin/maintenance time at
> most.

You seem to forget all the system package managers (dpkg, rpm, ...).
While opam does have its advantages, it does not really compete with them.

>> Do most of us agree? (I'm not against it, I just don't want to rely
>> on 50 different standards).
> 
> I don't think it's a question of agreeing, usage will tell. Besides I
> don't see another standard, maybe `ocamlfind install` but it's
> limited in what it can install (and I hope that eventually the
> annoying two-headed structure ocamlfind/opam will disapear, e.g. the
> notion of ocamlfind package could be merged in the compiler).

The standard generic interface for package-manager-agnostic upstream
packages is:

  ./configure && make && sudo make install PREFIX=/foo

If people cannot agree on a standard implementation, at least they could
agree on the interface.

>> (A while ago, before opam, a new version of GODI broke on our main
>> platform because of some ocamlnet configuration bug, but I was able
>> to reinstall ocaml and about 25 third-party libraries and syntax
>> extensions from scratch without too much pain → we see great value
>> in that package-management independence)
> Why not. I guess you can see the value of having each of these
> libraries generating a .install file that you can process with a
> simple command line tool rather than have to deal with the
> idiosyncrasies of each custom (usually semi-broken) install
> procedure.

http://xkcd.com/927/


Cheers,

-- 
Stéphane

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[Daniel Bünzli]

Le lundi, 9 décembre 2013 à 20:41, Stéphane Glondu a écrit :
> You seem to forget all the system package managers (dpkg, rpm, ...).
> While opam does have its advantages, it does not really compete with them.

I don't forget them, I'm pretty sure the combination of opam/.install files can be used for automating the generation of such packages.  
  
> http://xkcd.com/927/

So what's your proposal then ?  

Daniel

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[Stéphane Glondu]

Le 09/12/2013 21:04, Daniel Bünzli a écrit :
>> You seem to forget all the system package managers (dpkg, rpm, ...).
>> While opam does have its advantages, it does not really compete with them.
> 
> I don't forget them, I'm pretty sure the combination of opam/.install files can be used for automating the generation of such packages.  

Yes, but the existing stuff too. And the packaging work is (should be)
mainly adding metadata and integrating with the other packages, which is
hardly automatable. What you are talking about is package building,
which can already be automated with the standard "./configure && make &&
sudo make install" interface.

>> http://xkcd.com/927/
> 
> So what's your proposal then ?  

I am not proposing anything new. You are.


Cheers,

-- 
Stéphane

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[Daniel Bünzli]

Le lundi, 9 décembre 2013 à 21:22, Stéphane Glondu a écrit :

> Yes, but the existing stuff too. And the packaging work is (should be)
> mainly adding metadata and integrating with the other packages, which is
> hardly automatable. What you are talking about is package building,
> which can already be automated with the standard "./configure && make &&
> sudo make install" interface.

Why not but then provide me *convenient and easily understandable* tools that allow me to implement this interface in a reliable way. I don't want to replicate install logic in my twelve and growing packages, I used to do that but it's too costly in terms of maintenance. I now found a way of delegating the install logic to a tool in what I suppose is the main distribution channel for my packages (and that shouldn't be too hard for other distribution channels to use), I'm not going back unless something better and as convenient emerges.

> > > http://xkcd.com/927/
> > So what's your proposal then ?
>  
> I am not proposing anything new. You are.

Joke for joke. Besides I'm not proposing anything, I responded to the original poster's question on how I proceed to install libraries with OPAM.

Best,  

Daniel

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[Anthony Tavener]

[-- Attachment #1: Type: text/plain, Size: 3152 bytes --]

Thanks for the input, everyone! I see there's not quite one-true-way. This
was part of my confusion when I tried searches and looking at existing
packages. It helps to know that OCamlMakefile or ocamlbuild use ocamlfind,
but hide this detail. From my perspective when this is hidden it just looks
like there are even more packaging mechanisms.

I can understand Daniel's tact: choose one sensible method. This helps to
force an evolution rather than building up cruft. Certainly making a META
file along with descr/opam/url is somewhat redundant. Isn't it?

For now though, I'll do my install steps with ocamlfind using a META file.

It wasn't clear to me that ocamlfind is appropriate for installation, or
even how you would use it. Searching, I mostly get matches for using
ocamlfind to install existing packages, not make something installable. The
"install" subsection of ocamlfind really gives the impression of an
end-user tool, not for a library creator. References are made to the META
file but no description of this file.

Searching for the META file related to OCaml doesn't fare too well either.
Then I realized "man META" actually gives me exactly what I was looking
for. I didn't expect that, since I'd expect other systems could have a
configuration file called META.

Anyway, thanks! If there isn't an explanation of packaging libraries
somewhere, maybe I'll make a blog post: "How to share your library". It's
the kind of thing which might seem like nothing once you know it.

 -Tony


On Mon, Dec 9, 2013 at 2:04 PM, Daniel Bünzli
<daniel.buenzli@erratique.ch>wrote:

>
>
> Le lundi, 9 décembre 2013 à 21:22, Stéphane Glondu a écrit :
>
> > Yes, but the existing stuff too. And the packaging work is (should be)
> > mainly adding metadata and integrating with the other packages, which is
> > hardly automatable. What you are talking about is package building,
> > which can already be automated with the standard "./configure && make &&
> > sudo make install" interface.
>
> Why not but then provide me *convenient and easily understandable* tools
> that allow me to implement this interface in a reliable way. I don't want
> to replicate install logic in my twelve and growing packages, I used to do
> that but it's too costly in terms of maintenance. I now found a way of
> delegating the install logic to a tool in what I suppose is the main
> distribution channel for my packages (and that shouldn't be too hard for
> other distribution channels to use), I'm not going back unless something
> better and as convenient emerges.
>
> > > > http://xkcd.com/927/
> > > So what's your proposal then ?
> >
> > I am not proposing anything new. You are.
>
> Joke for joke. Besides I'm not proposing anything, I responded to the
> original poster's question on how I proceed to install libraries with OPAM.
>
> Best,
>
> Daniel
>
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 4196 bytes --]

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[Francois Berenger]

On 12/10/2013 06:36 AM, Anthony Tavener wrote:
> Thanks for the input, everyone! I see there's not quite one-true-way.
> This was part of my confusion when I tried searches and looking at
> existing packages. It helps to know that OCamlMakefile or ocamlbuild use
> ocamlfind, but hide this detail. From my perspective when this is hidden
> it just looks like there are even more packaging mechanisms.
>
> I can understand Daniel's tact: choose one sensible method. This helps
> to force an evolution rather than building up cruft. Certainly making a
> META file along with descr/opam/url is somewhat redundant. Isn't it?
>
> For now though, I'll do my install steps with ocamlfind using a META file.

OASIS can automatically generate the META file for you, if I remember well.

> It wasn't clear to me that ocamlfind is appropriate for installation, or
> even how you would use it. Searching, I mostly get matches for using
> ocamlfind to install existing packages, not make something installable.
> The "install" subsection of ocamlfind really gives the impression of an
> end-user tool, not for a library creator. References are made to the
> META file but no description of this file.
>
> Searching for the META file related to OCaml doesn't fare too well
> either. Then I realized "man META" actually gives me exactly what I was
> looking for. I didn't expect that, since I'd expect other systems could
> have a configuration file called META.
>
> Anyway, thanks! If there isn't an explanation of packaging libraries
> somewhere, maybe I'll make a blog post: "How to share your library".
> It's the kind of thing which might seem like nothing once you know it.
>
>   -Tony
>
>
> On Mon, Dec 9, 2013 at 2:04 PM, Daniel Bünzli
> <daniel.buenzli@erratique.ch <mailto:daniel.buenzli@erratique.ch>> wrote:
>
>
>
>     Le lundi, 9 décembre 2013 à 21:22, Stéphane Glondu a écrit :
>
>      > Yes, but the existing stuff too. And the packaging work is
>     (should be)
>      > mainly adding metadata and integrating with the other packages,
>     which is
>      > hardly automatable. What you are talking about is package building,
>      > which can already be automated with the standard "./configure &&
>     make &&
>      > sudo make install" interface.
>
>     Why not but then provide me *convenient and easily understandable*
>     tools that allow me to implement this interface in a reliable way. I
>     don't want to replicate install logic in my twelve and growing
>     packages, I used to do that but it's too costly in terms of
>     maintenance. I now found a way of delegating the install logic to a
>     tool in what I suppose is the main distribution channel for my
>     packages (and that shouldn't be too hard for other distribution
>     channels to use), I'm not going back unless something better and as
>     convenient emerges.
>
>      > > > http://xkcd.com/927/
>      > > So what's your proposal then ?
>      >
>      > I am not proposing anything new. You are.
>
>     Joke for joke. Besides I'm not proposing anything, I responded to
>     the original poster's question on how I proceed to install libraries
>     with OPAM.
>
>     Best,
>
>     Daniel
>
>
>
>     --
>     Caml-list mailing list.  Subscription management and archives:
>     https://sympa.inria.fr/sympa/arc/caml-list
>     Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>     Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>


-- 
Best regards,
Francois Berenger.

Re: [Caml-list] Library installation procedure (for use with OPAM)?

[Daniel Bünzli]

Le lundi, 9 décembre 2013 à 16:57, Daniel Bünzli a écrit :
> Manual Package Management™, the file format of .install is very simple you could perfectly develop have a command line tool that reads it and issues the appropriate cp commands,

In fact, thanks to Louis Gesbert, this tool already seems to exist since 17 days:

https://github.com/ocaml/opam/pull/1026

Best,

Daniel

[Caml-list] [ANN] opam-installer (beta) (was Re: Library installation procedure (for use with OPAM)?)

[Louis Gesbert]

Hah, Daniel, you're too fast, it actually just landed into OPAM trunk just now !
So here is the official announcement :)

OPAM now installs together with a small opam-installer script designed just towards that goal: it handles .install files, and can be used to install or remove any package that generates them, independently of OPAM. Hopefully this will help the acceptation of this format as a standard, both easing the programmer's (and build-system writer's) work and providing better integration with the packaging system.

Additionally, opam-installer can generate shell-scripts that could be used, for example, for distributing to targets which don't have the tool.

This is still in beta stage, all reports welcome.


--
Louis Gesbert / OCamlPro

On Tuesday 10 December 2013 15:50:53 Daniel Bünzli wrote:
> Le lundi, 9 décembre 2013 à 16:57, Daniel Bünzli a écrit :
> > Manual Package Management™, the file format of .install is very simple you could perfectly develop have a command line tool that reads it and issues the appropriate cp commands,
> 
> In fact, thanks to Louis Gesbert, this tool already seems to exist since 17 days:
> 
> https://github.com/ocaml/opam/pull/1026
> 
> Best,
> 
> Daniel