[Caml-list] A (silly ?) question about opam and ocamlfind

[Caml-list] A (silly ?) question about opam and ocamlfind

[Jocelyn Sérot]

[-- Attachment #1: Type: text/plain, Size: 2714 bytes --]

Dear Camlers,

I’ve been fighting for one entire day with a problem without success and the more i dig, the more confused i am :(

Suppose i define a small opam package - let’s call it « dummy » :

opam-version: "2.0"
name: "dummy"
version: "0.0"
synopsis: "A dummy OPAM package"
maintainer: "JS"
authors: "JS"
license: "MIT"
depends: [ "ocaml" "ocamlfind" ]
build: [
  [make]
]

where [make] simply builds the library files [mylib.cma] and [mylib.cmxa] (by directly calling ocamlc/opt or using dune - this does not matter here).

What is the correct way to make this package available to other programs :

[opam pin add .] 

or 

[opam install .]

?

I’ve tried both solutions but systematically get :

ocamlfind ocamlc -package dummy -linkpkg -o main main.ml
ocamlfind: Package `dummy' not found

when trying to compile [main.ml], where this file contains, for example : « let _ = Mylib.dump () » (and where [dump] is a function defined in mylib.ml » 

When simply pinning the package, [opam info dummy] gives :

<><> dummy: information on all versions <><><><><><><><><><><><><><><><><><>  🐫 
name         dummy
all-versions 0.0

<><> Version-specific details <><><><><><><><><><><><><><><><><><><><><><><>  🐫 
version     0.0
pin         file:///Users/jserot/Dev/ml/opam/simple/build/raw
url.src:    "file:///Users/jserot/Dev/ml/opam/simple/build/raw"
authors:    "JS"
maintainer: "JS"
license:    "MIT"
depends:    "ocaml" "ocamlfind"
synopsis    A dummy OPAM package

which tends does not show any installed version of the package (and hence could explain why ocamlfind does not find it ?)

After installing the package ([opam install .]), [opam info dummy] gives :

$ opam info dummy

<><> dummy: information on all versions <><><><><><><><><><><><><><><><><><>  🐫 
name                   dummy
all-installed-versions 0.0 [4.06.0]
all-versions           0.0

<><> Version-specific details <><><><><><><><><><><><><><><><><><><><><><><>  🐫 
version     0.0
pin         file:///Users/jserot/Dev/ml/opam/simple/build/raw
url.src:    "file:///Users/jserot/Dev/ml/opam/simple/build/raw"
authors:    "JS"
maintainer: "JS"
license:    "MIT"
depends:    "ocaml" "ocamlfind"
synopsis    A dummy OPAM package
$ 

which now seems to indicates that the package is « installed » (though i do not fully understand the distinction between the former case..).
But i still get the message 

ocamlfind ocamlc -package dummy -linkpkg -o main main.ml
ocamlfind: Package `dummy' not found

There must be sth i’m doing wrong (or have not understood in the way opam works)..
Any help appreciated.

Jocelyn







[-- Attachment #2: Type: text/html, Size: 9305 bytes --]

Re: [Caml-list] A (silly ?) question about opam and ocamlfind

[Kakadu]

[-- Attachment #1: Type: text/plain, Size: 3697 bytes --]

Hello,

There is old school package manager for OCaml called `ocamlfind` which
gives in your case an error about missing package 'dummy'. To make your
package discoverable by ocamlfind you need to install your cmxa's with
right META file. It is usually done using command 'ocamlfind install dummy
META dummy.cmxa' and discoverable later using `ocamlfind query dummy`.
Something like this in opam file should help:

build: [
  [make]
]
install: ["ocamlfind" "install" "dummy" "META" file1file2 ]


You can also try new fancy way of installing by providing opam an explicit
list of files that should be copied but I'm not very familiar with it.
Documentation will probably help.

Happy hacking,
Dmitrii

On Wed, Aug 14, 2019 at 4:20 PM Jocelyn Sérot <jocelyn.serot@uca.fr> wrote:

> Dear Camlers,
>
> I’ve been fighting for one entire day with a problem without success and
> the more i dig, the more confused i am :(
>
> Suppose i define a small opam package - let’s call it « dummy » :
>
> opam-version: "2.0"
> name: "dummy"
> version: "0.0"
> synopsis: "A dummy OPAM package"
> maintainer: "JS"
> authors: "JS"
> license: "MIT"
> depends: [ "ocaml" "ocamlfind" ]
> build: [
>   [make]
> ]
>
> where [make] simply builds the library files [mylib.cma] and [mylib.cmxa]
> (by directly calling ocamlc/opt or using dune - this does not matter here).
>
> What is the correct way to make this package available to other programs :
>
> [opam pin add .]
>
> or
>
> [opam install .]
>
> ?
>
> I’ve tried both solutions but systematically get :
>
> ocamlfind ocamlc -package dummy -linkpkg -o main main.ml
> ocamlfind: Package `dummy' not found
>
> when trying to compile [main.ml], where this file contains, for example
> : « let _ = Mylib.dump () » (and where [dump] is a function defined in
> mylib.ml »
>
> When simply pinning the package, [opam info dummy] gives :
>
> <><> dummy: information on all versions
> <><><><><><><><><><><><><><><><><><>  🐫
> name         dummy
> all-versions 0.0
>
> <><> Version-specific details
> <><><><><><><><><><><><><><><><><><><><><><><>  🐫
> version     0.0
> pin         file:///Users/jserot/Dev/ml/opam/simple/build/raw
> url.src:    "file:///Users/jserot/Dev/ml/opam/simple/build/raw"
> authors:    "JS"
> maintainer: "JS"
> license:    "MIT"
> depends:    "ocaml" "ocamlfind"
> synopsis    A dummy OPAM package
>
> which tends does not show any installed version of the package (and hence
> could explain why ocamlfind does not find it ?)
>
> After installing the package ([opam install .]), [opam info dummy] gives :
>
> $ opam info dummy
>
> <><> dummy: information on all versions
> <><><><><><><><><><><><><><><><><><>  🐫
> name                   dummy
> all-installed-versions 0.0 [4.06.0]
> all-versions           0.0
>
> <><> Version-specific details
> <><><><><><><><><><><><><><><><><><><><><><><>  🐫
> version     0.0
> pin         file:///Users/jserot/Dev/ml/opam/simple/build/raw
> url.src:    "file:///Users/jserot/Dev/ml/opam/simple/build/raw"
> authors:    "JS"
> maintainer: "JS"
> license:    "MIT"
> depends:    "ocaml" "ocamlfind"
> synopsis    A dummy OPAM package
> $
>
> which now seems to indicates that the package is « installed » (though i
> do not fully understand the distinction between the former case..).
> But i still get the message
>
> ocamlfind ocamlc -package dummy -linkpkg -o main main.ml
> ocamlfind: Package `dummy' not found
>
> There must be sth i’m doing wrong (or have not understood in the way opam
> works)..
> Any help appreciated.
>
> Jocelyn
>
>
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 10206 bytes --]

Re: [Caml-list] A (silly ?) question about opam and ocamlfind

[rixed]

Hello!

The more tools are added on top of each others to make life easier for newcomers, and the harder it is to understand what's happening.

So let's get back to the beginning. I'm assuming you know already most of this but I'd rather give a picture of the whole stack so that you know which documentation you need to read to learn the details about various steps.

OCaml compiler without any other helper tool knows very little about where to find the files it needs.
Following the tradition, it just knows one directory, where the system libraries are located. And it takes everything else from the command line: the inputs (some .ml, .cmx, .cmxa, .o, .a, etc files, some additional paths where to look for libraries if more are needed), and the single output (-o somefile).

You can work only with this, that's very simple, but also quickly cumbersome as the number of libraries you need grow, each requiring other libraries and special options for the compiler, recursively.
So at some point, you want a package manager: something that organise libraries in your local disk with some additional metadata about what depends on what, so that you only have to say "I want to link with the foobar, preferably the lwt variant".
This is ocamlfind. To play by the rules, you have to install all the package in a given directory, with an additional small META file that contains the dependencies and various options and other meta data such as a unique name and a description.
If library authors play by the rules and provide that META file and install their libraries in a standardized manner (which is greatly simplified by ocamlfind command line tool) then everything become as easy as `ocamlfind ocamlc -package 'foo bar baz' my_source.ml`.

This is great and nice, and by and large OCaml library authors have played by the rules for a very long time now (such a tool, to be successful, have to be reliable and _stable_ for a long time ; ocamlfind have been very successful and essential to OCaml popularity, because it has stayed the same for many years ; thank you so much to its maintainers to resist the need to break backward compatibility).

There is still an important annoyance though: for the above to work, you still need libraries foo, bar and baz to be downloaded, compiled, and installed on your local machine. For this, you need a standard package repository. This is OPAM. Opam also have it's small file with metainfo about each package, except this one is called "opam" (or "$package.opam") and states where to get the package from, what command to run to build it, and how to install it.
Sounds familiar? Yes that's exactly what your Linux distribution does, but it works also on non-linux systems and is also updated much quicker, as there are less bureaucracy involved (notice it solves a human problem not a technical problem here).

There is a bit of redundancy between ocamlfind and opam, because, despite ocamlfind being used by >90% of packages, opam is ocamlfind agnostic and can work, in theory, for non-ocamlfind packages (indeed, it can also be used to deploy about anything not only OCaml packages). So you have to give names, description, dependencies both to ocamlfind and opam.
But that's only a minor annoyance.

If it really bothers you, then you can use another tool on top of all that, such as dune, that will not only build your project but also write the META and the opam file for you (and more). With the risk that, if something fails at this point, you have all the above stack to unfold to understand what's happening.

If you have read this far then maybe you are interested in the topic of software packaging and deployment.
Then it must be mentioned that better, more sophisticated package managers do exist, which solve this whole problem once and for all, such as nix or guix, and that hopefully one day we will all abandon our compartmentalised solutions that are popular today and move on to the next level.

Re: [Caml-list] A (silly ?) question about opam and ocamlfind

[Jocelyn Sérot]

Thanks for the answer and the « historical » perspective. 

In fact, I’ve started to use dune a few weeks ago and this is a problem with the transition from a package previously developed with ocamlfind to dune which triggers my initial question.

Anyway, after carefully re-reading the opam and dune documentation, i’ve finally fixed the problem.

I’m about to publish a small repository on github summerizing the solution(s) with the hope that it can  be useful to other programmers.

Jocelyn

Le 14 août 2019 à 21:08, rixed@happyleptic.org a écrit :

> Hello!
> 
> The more tools are added on top of each others to make life easier for newcomers, and the harder it is to understand what's happening.
> 
> So let's get back to the beginning. I'm assuming you know already most of this but I'd rather give a picture of the whole stack so that you know which documentation you need to read to learn the details about various steps.
> 
> OCaml compiler without any other helper tool knows very little about where to find the files it needs.
> Following the tradition, it just knows one directory, where the system libraries are located. And it takes everything else from the command line: the inputs (some .ml, .cmx, .cmxa, .o, .a, etc files, some additional paths where to look for libraries if more are needed), and the single output (-o somefile).
> 
> You can work only with this, that's very simple, but also quickly cumbersome as the number of libraries you need grow, each requiring other libraries and special options for the compiler, recursively.
> So at some point, you want a package manager: something that organise libraries in your local disk with some additional metadata about what depends on what, so that you only have to say "I want to link with the foobar, preferably the lwt variant".
> This is ocamlfind. To play by the rules, you have to install all the package in a given directory, with an additional small META file that contains the dependencies and various options and other meta data such as a unique name and a description.
> If library authors play by the rules and provide that META file and install their libraries in a standardized manner (which is greatly simplified by ocamlfind command line tool) then everything become as easy as `ocamlfind ocamlc -package 'foo bar baz' my_source.ml`.
> 
> This is great and nice, and by and large OCaml library authors have played by the rules for a very long time now (such a tool, to be successful, have to be reliable and _stable_ for a long time ; ocamlfind have been very successful and essential to OCaml popularity, because it has stayed the same for many years ; thank you so much to its maintainers to resist the need to break backward compatibility).
> 
> There is still an important annoyance though: for the above to work, you still need libraries foo, bar and baz to be downloaded, compiled, and installed on your local machine. For this, you need a standard package repository. This is OPAM. Opam also have it's small file with metainfo about each package, except this one is called "opam" (or "$package.opam") and states where to get the package from, what command to run to build it, and how to install it.
> Sounds familiar? Yes that's exactly what your Linux distribution does, but it works also on non-linux systems and is also updated much quicker, as there are less bureaucracy involved (notice it solves a human problem not a technical problem here).
> 
> There is a bit of redundancy between ocamlfind and opam, because, despite ocamlfind being used by >90% of packages, opam is ocamlfind agnostic and can work, in theory, for non-ocamlfind packages (indeed, it can also be used to deploy about anything not only OCaml packages). So you have to give names, description, dependencies both to ocamlfind and opam.
> But that's only a minor annoyance.
> 
> If it really bothers you, then you can use another tool on top of all that, such as dune, that will not only build your project but also write the META and the opam file for you (and more). With the risk that, if something fails at this point, you have all the above stack to unfold to understand what's happening.
> 
> If you have read this far then maybe you are interested in the topic of software packaging and deployment.
> Then it must be mentioned that better, more sophisticated package managers do exist, which solve this whole problem once and for all, such as nix or guix, and that hopefully one day we will all abandon our compartmentalised solutions that are popular today and move on to the next level.

Re: [Caml-list] A (silly ?) question about opam and ocamlfind

[Malcolm Matalka]

[-- Attachment #1: Type: text/plain, Size: 5448 bytes --]


rixed@happyleptic.org writes:

> Hello!
>
> The more tools are added on top of each others to make life easier for newcomers, and the harder it is to understand what's happening.
>
> So let's get back to the beginning. I'm assuming you know already most of this but I'd rather give a picture of the whole stack so that you know which documentation you need to read to learn the details about various steps.
>
> OCaml compiler without any other helper tool knows very little about where to find the files it needs.
> Following the tradition, it just knows one directory, where the system libraries are located. And it takes everything else from the command line: the inputs (some .ml, .cmx, .cmxa, .o, .a, etc files, some additional paths where to look for libraries if more are needed), and the single output (-o somefile).
>
> You can work only with this, that's very simple, but also quickly cumbersome as the number of libraries you need grow, each requiring other libraries and special options for the compiler, recursively.
> So at some point, you want a package manager: something that organise libraries in your local disk with some additional metadata about what depends on what, so that you only have to say "I want to link with the foobar, preferably the lwt variant".
> This is ocamlfind. To play by the rules, you have to install all the package in a given directory, with an additional small META file that contains the dependencies and various options and other meta data such as a unique name and a description.
> If library authors play by the rules and provide that META file and install their libraries in a standardized manner (which is greatly simplified by ocamlfind command line tool) then everything become as easy as `ocamlfind ocamlc -package 'foo bar baz' my_source.ml`.
>
> This is great and nice, and by and large OCaml library authors have played by the rules for a very long time now (such a tool, to be successful, have to be reliable and _stable_ for a long time ; ocamlfind have been very successful and essential to OCaml popularity, because it has stayed the same for many years ; thank you so much to its maintainers to resist the need to break backward compatibility).
>
> There is still an important annoyance though: for the above to work, you still need libraries foo, bar and baz to be downloaded, compiled, and installed on your local machine. For this, you need a standard package repository. This is OPAM. Opam also have it's small file with metainfo about each package, except this one is called "opam" (or "$package.opam") and states where to get the package from, what command to run to build it, and how to install it.
> Sounds familiar? Yes that's exactly what your Linux distribution does, but it works also on non-linux systems and is also updated much quicker, as there are less bureaucracy involved (notice it solves a human problem not a technical problem here).
>
> There is a bit of redundancy between ocamlfind and opam, because, despite ocamlfind being used by >90% of packages, opam is ocamlfind agnostic and can work, in theory, for non-ocamlfind packages (indeed, it can also be used to deploy about anything not only OCaml packages). So you have to give names, description, dependencies both to ocamlfind and opam.
> But that's only a minor annoyance.
>
> If it really bothers you, then you can use another tool on top of all that, such as dune, that will not only build your project but also write the META and the opam file for you (and more). With the risk that, if something fails at this point, you have all the above stack to unfold to understand what's happening.
>
> If you have read this far then maybe you are interested in the topic of software packaging and deployment.
> Then it must be mentioned that better, more sophisticated package managers do exist, which solve this whole problem once and for all, such as nix or guix, and that hopefully one day we will all abandon our compartmentalised solutions that are popular today and move on to the next level.

Another option is to use pds and hll which I wrote a few years ago and
use for all of my own projects.  A tutorial can be found here:

http://blog.appliedcompscilab.com/2016-Q4/index.html

It's gained some features since that post.

The idea is that pds can transform a TOML file into a Makefile to build
and hll can turn the TOML file into an opam package.  Other tools to
create a .merlin file.

It's pretty simple (pds is implemented in about 800 lines of code
currently + a few dependencies vendored in the repo).  pds supports
parallel builds, incremental builds, building differently based on
environment, and works fine within a monorepo (I've only tested it
against my monorepo where I do all my development so YMMV).  It also
supports building non-ocaml sourcetrees assuming the sourcetree
implements some Makefile targets.  I use this to jump through the Ocaml
-> C -> Ocaml hoops needed to build OCaml bindings with ctypes.  It
definitely has some rough edges since it only works for what I care
about.

It looks like dune has and will have the majority of mindshare, but
pds/hll might be useful to someone who doesn't want to take that leap.
Presumably it should be fairly straightforwad to turn the pds TOML file
into a Dune file as well.  I haven't kept the opam package up-to-date
since I didn't think it'd be that interesting to anyone else but I can
push the latest package this week.

/Malcolm

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]