[Caml-list] man, doc, … and opam

[Caml-list] man, doc, … and opam

[Dagnat Fabien]

Hello,
After reading the opam documentation, I'm not sure anything specific has been done about the extra resources of packages (man and doc files for example).

The only things, I found in the documentation is build-doc switch to indicate how to build the doc. For example, utop.1.5.opam contains
build-doc: [ "ocaml" "setup.ml" "-doc" ]

I was expecting to find the doc of utop but on my .opam repository, the doc directory of utop is empty. Furthermore, in the utop.install file there is no doc command.

Does it depends on how the install command have been done by the user? Have I missed something?

More generally, my question also apply to man pages not found anywhere in the opam documentation.

Fabien

Re: [Caml-list] man, doc, … and opam

[Daniel Bünzli]

Hello,

Incidentally I learned more about all this today.

> After reading the opam documentation, I'm not sure anything specific has been done about the extra resources of packages (man and doc files for example).
Some things have been done, read section 1.2.5 of the developer manual. An install file allow you to specify exactly what to install and where in `bin`, `doc` and `lib`.  

In fact this it can completely replace an `ocamlfind install` or `ocaml setup.ml -install`. Just make sure you add the `META` in `lib`, and `ocamlfind` will find your package.

I also discovered that the `remove:` section with the usual `ocamlfind remove PKG` is not needed. This is all due to the fact the IIUC  `ocamlfind` uses the file system as the database so there's no worry to bypass it in these circumstances, opam will just remove the lib/PKG directory et voilà.
  
> The only things, I found in the documentation is build-doc switch to indicate how to build the doc. For example, utop.1.5.opam contains
> build-doc: [ "ocaml" "setup.ml" "-doc" ]
>  
> I was expecting to find the doc of utop but on my .opam repository, the doc directory of utop is empty. Furthermore, in the utop.install file there is no doc command.
The problem is that apparently the utop package has a static .install file in the package description and that this file doesn't list the things generated by build-doc in the `doc:` section. As written in the manual section mentioned above it's much better if the .install file is actually generated by the underlying build system according to the artefacts it has built. But to the best of my knowledge none of the current build systems in use support the generation of such a file (you can easily do it yourself though).

So the idea is that the `build:` section has commands that eventually generate a file PKG.install in the root directory of your distribution and opam then uses that file to install the files associated to the package. (Unfortunately the example of the online documentation uses an static .install file added to the package description *itself* which makes it believe that this was the only way to use them).

> More generally, my question also apply to man pages not found anywhere in the opam documentation.
I have created an issue about this a few hours ago: https://github.com/OCamlPro/opam/issues/673

I think the support is mostly right, the only thing that is missing is install in man folders and a way to easily get to the doc directory of a particular package, instead of having to hunt in the .opam directory. This means that we can put the README and CHANGES in there aswell for people to read.

The conclusion of this is that the problem seems more to be that the install file of most packages is crap rather than a lack of support from opam itself. Though the documentation about all this could be improved on the opam side aswell.

Best,

Daniel

Re: [Caml-list] man, doc, … and opam

[Thomas Gazagnaire]

First of all, thanks Daniel for the precise answer. It seems that writing this damn developer manual was not so much of a waste of time after all :-)

>> After reading the opam documentation, I'm not sure anything specific has been done about the extra resources of packages (man and doc files for example).
> Some things have been done, read section 1.2.5 of the developer manual. An install file allow you to specify exactly what to install and where in `bin`, `doc` and `lib`.  
> 
> In fact this it can completely replace an `ocamlfind install` or `ocaml setup.ml -install`. Just make sure you add the `META` in `lib`, and `ocamlfind` will find your package.
> 
> I also discovered that the `remove:` section with the usual `ocamlfind remove PKG` is not needed. This is all due to the fact the IIUC  `ocamlfind` uses the file system as the database so there's no worry to bypass it in these circumstances, opam will just remove the lib/PKG directory et voilà.

This is not totally true. ocamlfind also install all the C stubs in `DESTDIR/stublibs/dllPKG.so` and this is not removed automatically by OPAM (as it currently has no good way to know the library name used by ocamlfind). So, in this case, `ocamlfind remove PKG` does a bit more than removing DESTDIR/PKG.

OPAM has some mechanics to define internal libraries name (the field `library`) -- this was an early experiment to simulate basic functionalities of ocamlfind, so I guess we could reuse that field to declare which ocamlfind libraries a given OPAM package contains.
> 
>> The only things, I found in the documentation is build-doc switch to indicate how to build the doc. For example, utop.1.5.opam contains
>> build-doc: [ "ocaml" "setup.ml" "-doc" ]
>> 
>> I was expecting to find the doc of utop but on my .opam repository, the doc directory of utop is empty. Furthermore, in the utop.install file there is no doc command.
> The problem is that apparently the utop package has a static .install file in the package description and that this file doesn't list the things generated by build-doc in the `doc:` section. As written in the manual section mentioned above it's much better if the .install file is actually generated by the underlying build system according to the artefacts it has built. But to the best of my knowledge none of the current build systems in use support the generation of such a file (you can easily do it yourself though).

Indeed, no build system currently supports this. At one point, we've started to discuss the integration with oasis, obuild and ocp-build (with Sylvain, Vincent and Fabrice), but currenlty none of these systems has implemented that feature yet. The `.install` mechanics is not perfect yet (the file needs to be at the root of the directory, and needs to be called PKG.install, where PKG is the OPAM package name) but I've waited for feedback before improving this.

> So the idea is that the `build:` section has commands that eventually generate a file PKG.install in the root directory of your distribution and opam then uses that file to install the files associated to the package. (Unfortunately the example of the online documentation uses an static .install file added to the package description *itself* which makes it believe that this was the only way to use them).

Indeed, the documentation is a bit misleading there, but to my defense, this was supposed to be an advance usage of OPAM :-)

>> More generally, my question also apply to man pages not found anywhere in the opam documentation.
> I think the support is mostly right, the only thing that is missing is install in man folders and a way to easily get to the doc directory of a particular package, instead of having to hunt in the .opam directory. This means that we can put the README and CHANGES in there aswell for people to read.

Indeed, this is clearly missing. I've created https://github.com/OCamlPro/opam/issues/674 to track this.
> 
> The conclusion of this is that the problem seems more to be that the install file of most packages is crap rather than a lack of support from opam itself. Though the documentation about all this could be improved on the opam side aswell.

Ack. We'll try to improve that before the next release (which unfortunately has already been postponed too much). I've created https://github.com/OCamlPro/opam/issues/675 to track this.

Thanks for the feedback!
Thomas

Re: [Caml-list] man, doc, … and opam

[Daniel Bünzli]

Le lundi, 8 juillet 2013 à 10:56, Thomas Gazagnaire a écrit :
> This is not totally true. ocamlfind also install all the C stubs in `DESTDIR/stublibs/dllPKG.so` and this is not removed automatically by OPAM (as it currently has no good way to know the library name used by ocamlfind). So, in this case, `ocamlfind remove PKG` does a bit more than removing DESTDIR/PKG.

Aha ok, spoke too fast.  

> Indeed, no build system currently supports this. At one point, we've started to discuss the integration with oasis, obuild and ocp-build (with Sylvain, Vincent and Fabrice), but currenlty none of these systems has implemented that feature yet.  
For myself, I do have simple requirements, my packages being mostly pure OCaml. However I do need to scale horizontally, I have many packages developed, documented, maintained and formally released and I have an absolute *hate* for administrative duties.

I now decided to eschew each of these solutions (tested all of them) as none in the current state is able to fullfill my needs in a satisfying way -- both technically and from a usability point of view. For the curious see here [1] for how I plan to proceed from now on, at least until something *really* good emerges.

However now that I decided to go the ocamlbuild only + install file route for pure ocaml packages it would be nice if there was a way for ocamlbuild to tell us, given an .mllib (or .itarget) file, all the installable artefacts it *might* generate as it does actually have that knowledge and I'd prefer not to reencode it my scripts. I do it purely manually for now (see the links from the message [1]) it's not too bad, but I'd prefer having a simpler way so that I'm sure I don't miss anything (also if new build artefacts may appear in the future like cmxs did).  
  
> The `.install` mechanics is not perfect yet (the file needs to be at the root of the directory, and needs to be called PKG.install, where PKG is the OPAM package name) but I've waited for feedback before improving this.

As I said on opam-devel I don't think these requirements are that bad.

Best,

Daniel

[1] http://lists.ocaml.org/pipermail/opam-devel/2013-July/000161.html

Re: [Caml-list] man, doc, … and opam

[Gabriel Kerneis]

On Mon, Jul 08, 2013 at 04:41:55PM +0100, Daniel Bünzli wrote:
> However now that I decided to go the ocamlbuild only + install file route for
> pure ocaml packages it would be nice if there was a way for ocamlbuild to tell
> us, given an .mllib (or .itarget) file, all the installable artefacts it
> *might* generate as it does actually have that knowledge and I'd prefer not to
> reencode it my scripts.

I fully agree with the request.  For CIL, I do this with a custom rule .mllib ->
.libfiles in myocamlbuild.ml, but it is an ugly hack I am not very proud of (and
in fact no quite sure how reliable it is):
https://github.com/kerneis/cil/blob/develop/myocamlbuild.ml#L7

I've seen reinvented in some Debian packages as well. A unified ocamlbuild +
ocamlfind mechanism would be great here.

Best,
-- 
Gabriel

Re: [Caml-list] man, doc, … and opam

[Daniel Bünzli]

Gabriel S. asked me to make an entry in the bugtracker: 

http://caml.inria.fr/mantis/view.php?id=6067

Gabriel K. you may want to join the discussion to ensure that if something gets done it also satisfies you (I stole the suffix name you used for the target). 

Best,

Daniel