[Caml-list] building and using a library in a subdirectory

[Caml-list] building and using a library in a subdirectory

[Martin DeMello]

[-- Attachment #1: Type: text/plain, Size: 1452 bytes --]

If I have my project set up like this:

.
├── file.ml
├── plugins
│   └── puz
│       ├── puz_bin.ml
│       ├── puz_match.ml
│       ├── puz.ml
│       ├── puz_plugin.mllib
│       ├── puz_types.ml
│       └── puz_utils.ml
└── _tags

$ cat plugins/puz/puz_plugin.mllib
Puz Puz_bin Puz_match

This works:

$ ocamlbuild -use-ocamlfind plugins/puz/puz_plugin.cmxa
Finished, 22 targets (3 cached) in 00:00:03.

My _tags file:
------------------------------------------------
$ cat _tags
true: thread,debug
true: package(core_kernel)

"plugins/puz": include

<gui.*>: package(labltk)
<**/puz.*>: package(unix), package(str), package(core_kernel),
package(bitstring), package(mikmatch_pcre)
<**/*_bin.*>: package(bitstring.syntax), syntax(bitstring)
<**/*_match.*>: package(mikmatch_pcre), syntax(camlp4o)
------------------------------------------------

But I can't figure out how to use this library in my main program. This is
a minimal example of what I'm trying to do, not working of course:

$ cat file.ml
open Core_kernel.Std

let read fname =
  let data = In_channel.read_all fname in
  Puz_plugin.read data

$ ocamlbuild -use-ocamlfind file.native
+ ocamlfind ocamlc -c -g -thread -package core_kernel -I plugins/puz -o
file.cmo file.ml
File "file.ml", line 5, characters 2-17:
Error: Unbound module Puz_plugin

martin

[-- Attachment #2: Type: text/html, Size: 2889 bytes --]

Re: [Caml-list] building and using a library in a subdirectory

[Gabriel Scherer]

A .cma or .cmxa archive does not pack its sub-modules under a common
module name: the module name Puz_plugin that you use does not refer to
anything. Accessing any of the constituent modules directly (eg.
Puz_match.foo) should work -- although it may not go through the .cmxa
but directly refer to the compilation unit.

Note that if "_plugin" is intended to refer to dynamic linking, you
may want to use a .mldylib file instead of .mllib, to be used to
generate a .cmxs file: .cmxa is not suitable for dynamic linking
(while, at the bytecode level, .cma work for both static and dynamic
linking).

On Tue, Sep 22, 2015 at 8:23 AM, Martin DeMello <martindemello@gmail.com> wrote:
> If I have my project set up like this:
>
> .
> ├── file.ml
> ├── plugins
> │   └── puz
> │       ├── puz_bin.ml
> │       ├── puz_match.ml
> │       ├── puz.ml
> │       ├── puz_plugin.mllib
> │       ├── puz_types.ml
> │       └── puz_utils.ml
> └── _tags
>
> $ cat plugins/puz/puz_plugin.mllib
> Puz Puz_bin Puz_match
>
> This works:
>
> $ ocamlbuild -use-ocamlfind plugins/puz/puz_plugin.cmxa
> Finished, 22 targets (3 cached) in 00:00:03.
>
> My _tags file:
> ------------------------------------------------
> $ cat _tags
> true: thread,debug
> true: package(core_kernel)
>
> "plugins/puz": include
>
> <gui.*>: package(labltk)
> <**/puz.*>: package(unix), package(str), package(core_kernel),
> package(bitstring), package(mikmatch_pcre)
> <**/*_bin.*>: package(bitstring.syntax), syntax(bitstring)
> <**/*_match.*>: package(mikmatch_pcre), syntax(camlp4o)
> ------------------------------------------------
>
> But I can't figure out how to use this library in my main program. This is a
> minimal example of what I'm trying to do, not working of course:
>
> $ cat file.ml
> open Core_kernel.Std
>
> let read fname =
>   let data = In_channel.read_all fname in
>   Puz_plugin.read data
>
> $ ocamlbuild -use-ocamlfind file.native
> + ocamlfind ocamlc -c -g -thread -package core_kernel -I plugins/puz -o
> file.cmo file.ml
> File "file.ml", line 5, characters 2-17:
> Error: Unbound module Puz_plugin
>
> martin

Re: [Caml-list] building and using a library in a subdirectory

[Martin DeMello]

[-- Attachment #1: Type: text/plain, Size: 2693 bytes --]

I finally found
http://ocaml.org/learn/tutorials/ocamlbuild/Using_internal_libraries.html
(which didn't come up earlier because I was searching for "subdirectory");
following that I added a myocamlbuild file:

 $ cat myocamlbuild.ml
open Ocamlbuild_plugin

let () =
  dispatch begin function
  | After_rules ->
     ocaml_lib "plugins/puz/Puz_plugin";
  | _ -> ()
  end

but then attempting to add to the _tags file:

<file.*>: use_puz_plugin

gave me the following:

Warning: the tag "use_puz_plugin" is not used in any flag or dependency
declaration, so it will have no effect; it may be a typo. Otherwise you can
use `mark_tag_used` in your myocamlbuild.ml to disable this warning.
+ ocamlfind ocamlc -c -g -thread -package core_kernel -I plugins/puz -o
file.cmo file.ml
File "file.ml", line 5, characters 2-17:
Error: Unbound module Puz_plugin


Feedback on the tutorial: It would be useful to have a complete repository
corresponding to each of the examples, that could be checked out and played
with.

martin



On Mon, Sep 21, 2015 at 11:23 PM, Martin DeMello <martindemello@gmail.com>
wrote:

> If I have my project set up like this:
>
> .
> ├── file.ml
> ├── plugins
> │   └── puz
> │       ├── puz_bin.ml
> │       ├── puz_match.ml
> │       ├── puz.ml
> │       ├── puz_plugin.mllib
> │       ├── puz_types.ml
> │       └── puz_utils.ml
> └── _tags
>
> $ cat plugins/puz/puz_plugin.mllib
> Puz Puz_bin Puz_match
>
> This works:
>
> $ ocamlbuild -use-ocamlfind plugins/puz/puz_plugin.cmxa
> Finished, 22 targets (3 cached) in 00:00:03.
>
> My _tags file:
> ------------------------------------------------
> $ cat _tags
> true: thread,debug
> true: package(core_kernel)
>
> "plugins/puz": include
>
> <gui.*>: package(labltk)
> <**/puz.*>: package(unix), package(str), package(core_kernel),
> package(bitstring), package(mikmatch_pcre)
> <**/*_bin.*>: package(bitstring.syntax), syntax(bitstring)
> <**/*_match.*>: package(mikmatch_pcre), syntax(camlp4o)
> ------------------------------------------------
>
> But I can't figure out how to use this library in my main program. This is
> a minimal example of what I'm trying to do, not working of course:
>
> $ cat file.ml
> open Core_kernel.Std
>
> let read fname =
>   let data = In_channel.read_all fname in
>   Puz_plugin.read data
>
> $ ocamlbuild -use-ocamlfind file.native
> + ocamlfind ocamlc -c -g -thread -package core_kernel -I plugins/puz -o
> file.cmo file.ml
> File "file.ml", line 5, characters 2-17:
> Error: Unbound module Puz_plugin
>
> martin
>

[-- Attachment #2: Type: text/html, Size: 5122 bytes --]

Re: [Caml-list] building and using a library in a subdirectory

[Martin DeMello]

[-- Attachment #1: Type: text/plain, Size: 3424 bytes --]

Thanks, I didn't realise that. Replacing Puz_plugin.read with Puz.read
compiled, but didn't link:

+ ocamlfind ocamlopt -linkpkg -g -thread -package core_kernel
plugins/puz/puz_types.cmx plugins/puz/puz_utils.cmx plugins/puz/puz_bin.cmx
plugins/puz/puz_match.cmx types.cmx xword.cmx plugins/puz/puz.cmx file.cmx
-o file.native
File "_none_", line 1:
Error: No implementations provided for the following modules:
         Bitstring referenced from plugins/puz/puz_bin.cmx
         Run_mikmatch_pcre referenced from plugins/puz/puz_match.cmx
         Str referenced from plugins/puz/puz.cmx
         Pcre referenced from plugins/puz/puz_match.cmx
Command exited with code 2.

I don't (yet) need dynamic linking; my current main aim with the .mllib
setup is to specify the package dependencies for each plugin in its own
section of the _tags file, and not have a long list of every plugin's
dependencies in the main module.

martin

On Mon, Sep 21, 2015 at 11:45 PM, Gabriel Scherer <gabriel.scherer@gmail.com
> wrote:

> A .cma or .cmxa archive does not pack its sub-modules under a common
> module name: the module name Puz_plugin that you use does not refer to
> anything. Accessing any of the constituent modules directly (eg.
> Puz_match.foo) should work -- although it may not go through the .cmxa
> but directly refer to the compilation unit.
>
> Note that if "_plugin" is intended to refer to dynamic linking, you
> may want to use a .mldylib file instead of .mllib, to be used to
> generate a .cmxs file: .cmxa is not suitable for dynamic linking
> (while, at the bytecode level, .cma work for both static and dynamic
> linking).
>
> On Tue, Sep 22, 2015 at 8:23 AM, Martin DeMello <martindemello@gmail.com>
> wrote:
> > If I have my project set up like this:
> >
> > .
> > ├── file.ml
> > ├── plugins
> > │   └── puz
> > │       ├── puz_bin.ml
> > │       ├── puz_match.ml
> > │       ├── puz.ml
> > │       ├── puz_plugin.mllib
> > │       ├── puz_types.ml
> > │       └── puz_utils.ml
> > └── _tags
> >
> > $ cat plugins/puz/puz_plugin.mllib
> > Puz Puz_bin Puz_match
> >
> > This works:
> >
> > $ ocamlbuild -use-ocamlfind plugins/puz/puz_plugin.cmxa
> > Finished, 22 targets (3 cached) in 00:00:03.
> >
> > My _tags file:
> > ------------------------------------------------
> > $ cat _tags
> > true: thread,debug
> > true: package(core_kernel)
> >
> > "plugins/puz": include
> >
> > <gui.*>: package(labltk)
> > <**/puz.*>: package(unix), package(str), package(core_kernel),
> > package(bitstring), package(mikmatch_pcre)
> > <**/*_bin.*>: package(bitstring.syntax), syntax(bitstring)
> > <**/*_match.*>: package(mikmatch_pcre), syntax(camlp4o)
> > ------------------------------------------------
> >
> > But I can't figure out how to use this library in my main program. This
> is a
> > minimal example of what I'm trying to do, not working of course:
> >
> > $ cat file.ml
> > open Core_kernel.Std
> >
> > let read fname =
> >   let data = In_channel.read_all fname in
> >   Puz_plugin.read data
> >
> > $ ocamlbuild -use-ocamlfind file.native
> > + ocamlfind ocamlc -c -g -thread -package core_kernel -I plugins/puz -o
> > file.cmo file.ml
> > File "file.ml", line 5, characters 2-17:
> > Error: Unbound module Puz_plugin
> >
> > martin
>

[-- Attachment #2: Type: text/html, Size: 4952 bytes --]

Re: [Caml-list] building and using a library in a subdirectory

[Gabriel Scherer]

> I don't (yet) need dynamic linking; my current main aim with the .mllib setup is to specify the package dependencies for each plugin in its > own section of the _tags file, and not have a long list of every plugin's dependencies in the main module.

Well, that doesn't quite work: you need to have the ocamlfind packages
passed to the command linking the final executable.

I suppose changing
  <**/puz.*>: ...
to
  <**/puz.*> or <file.*>: ...
could work, as this line seems to pass all the libraries flags.


On Tue, Sep 22, 2015 at 8:52 AM, Martin DeMello <martindemello@gmail.com> wrote:
> Thanks, I didn't realise that. Replacing Puz_plugin.read with Puz.read
> compiled, but didn't link:
>
> + ocamlfind ocamlopt -linkpkg -g -thread -package core_kernel
> plugins/puz/puz_types.cmx plugins/puz/puz_utils.cmx plugins/puz/puz_bin.cmx
> plugins/puz/puz_match.cmx types.cmx xword.cmx plugins/puz/puz.cmx file.cmx
> -o file.native
> File "_none_", line 1:
> Error: No implementations provided for the following modules:
>          Bitstring referenced from plugins/puz/puz_bin.cmx
>          Run_mikmatch_pcre referenced from plugins/puz/puz_match.cmx
>          Str referenced from plugins/puz/puz.cmx
>          Pcre referenced from plugins/puz/puz_match.cmx
> Command exited with code 2.
>
> I don't (yet) need dynamic linking; my current main aim with the .mllib
> setup is to specify the package dependencies for each plugin in its own
> section of the _tags file, and not have a long list of every plugin's
> dependencies in the main module.
>
> martin
>
> On Mon, Sep 21, 2015 at 11:45 PM, Gabriel Scherer
> <gabriel.scherer@gmail.com> wrote:
>>
>> A .cma or .cmxa archive does not pack its sub-modules under a common
>> module name: the module name Puz_plugin that you use does not refer to
>> anything. Accessing any of the constituent modules directly (eg.
>> Puz_match.foo) should work -- although it may not go through the .cmxa
>> but directly refer to the compilation unit.
>>
>> Note that if "_plugin" is intended to refer to dynamic linking, you
>> may want to use a .mldylib file instead of .mllib, to be used to
>> generate a .cmxs file: .cmxa is not suitable for dynamic linking
>> (while, at the bytecode level, .cma work for both static and dynamic
>> linking).
>>
>> On Tue, Sep 22, 2015 at 8:23 AM, Martin DeMello <martindemello@gmail.com>
>> wrote:
>> > If I have my project set up like this:
>> >
>> > .
>> > ├── file.ml
>> > ├── plugins
>> > │   └── puz
>> > │       ├── puz_bin.ml
>> > │       ├── puz_match.ml
>> > │       ├── puz.ml
>> > │       ├── puz_plugin.mllib
>> > │       ├── puz_types.ml
>> > │       └── puz_utils.ml
>> > └── _tags
>> >
>> > $ cat plugins/puz/puz_plugin.mllib
>> > Puz Puz_bin Puz_match
>> >
>> > This works:
>> >
>> > $ ocamlbuild -use-ocamlfind plugins/puz/puz_plugin.cmxa
>> > Finished, 22 targets (3 cached) in 00:00:03.
>> >
>> > My _tags file:
>> > ------------------------------------------------
>> > $ cat _tags
>> > true: thread,debug
>> > true: package(core_kernel)
>> >
>> > "plugins/puz": include
>> >
>> > <gui.*>: package(labltk)
>> > <**/puz.*>: package(unix), package(str), package(core_kernel),
>> > package(bitstring), package(mikmatch_pcre)
>> > <**/*_bin.*>: package(bitstring.syntax), syntax(bitstring)
>> > <**/*_match.*>: package(mikmatch_pcre), syntax(camlp4o)
>> > ------------------------------------------------
>> >
>> > But I can't figure out how to use this library in my main program. This
>> > is a
>> > minimal example of what I'm trying to do, not working of course:
>> >
>> > $ cat file.ml
>> > open Core_kernel.Std
>> >
>> > let read fname =
>> >   let data = In_channel.read_all fname in
>> >   Puz_plugin.read data
>> >
>> > $ ocamlbuild -use-ocamlfind file.native
>> > + ocamlfind ocamlc -c -g -thread -package core_kernel -I plugins/puz -o
>> > file.cmo file.ml
>> > File "file.ml", line 5, characters 2-17:
>> > Error: Unbound module Puz_plugin
>> >
>> > martin
>
>

Re: [Caml-list] building and using a library in a subdirectory

[Martin DeMello]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

On Mon, Sep 21, 2015 at 11:58 PM, Gabriel Scherer <gabriel.scherer@gmail.com
> wrote:

> > I don't (yet) need dynamic linking; my current main aim with the .mllib
> setup is to specify the package dependencies for each plugin in its > own
> section of the _tags file, and not have a long list of every plugin's
> dependencies in the main module.
>
> Well, that doesn't quite work: you need to have the ocamlfind packages
> passed to the command linking the final executable.
>
> I suppose changing
>   <**/puz.*>: ...
> to
>   <**/puz.*> or <file.*>: ...
> could work, as this line seems to pass all the libraries flags.


Isn't there any way with a .cmxa to compile the dependencies into a library
and then just have the library as a link-time dependency for the main
project? I can imagine things getting pretty messy once I have several
plugins and have to concatenate all their package deps into the file.* tags
line.

martin

[-- Attachment #2: Type: text/html, Size: 1358 bytes --]

Re: [Caml-list] building and using a library in a subdirectory

[Gabriel Scherer]

The compiler allows to store C compilation flags in library archives,
but not OCaml-level flags, and in any case that would be a dubious
idea as hardcoding the path to the ocamlfind packages on your system
would make the (bytecode) archive libraries less portable.

You can easily define the packages of the final executable in several
steps, one per plugin:

  <plugin1/**> or <main.*>: package(foo)
  ...
  <plugin2/**> or <main.*>: package(bar)

Finally, you could also distribute each of your plugin as a separate
ocamlfind package (allowing to combine both the link to the archive(s)
and information about ocamlfind dependencies in a single place, the
package's META file), but that requires re-installing a package for
its changes to be visible from the main program -- so it is probably
more useful for libraries that are less tightly coupled.

On Tue, Sep 22, 2015 at 9:09 AM, Martin DeMello <martindemello@gmail.com> wrote:
> On Mon, Sep 21, 2015 at 11:58 PM, Gabriel Scherer
> <gabriel.scherer@gmail.com> wrote:
>>
>> > I don't (yet) need dynamic linking; my current main aim with the .mllib
>> > setup is to specify the package dependencies for each plugin in its > own
>> > section of the _tags file, and not have a long list of every plugin's
>> > dependencies in the main module.
>>
>> Well, that doesn't quite work: you need to have the ocamlfind packages
>> passed to the command linking the final executable.
>>
>> I suppose changing
>>   <**/puz.*>: ...
>> to
>>   <**/puz.*> or <file.*>: ...
>> could work, as this line seems to pass all the libraries flags.
>
>
> Isn't there any way with a .cmxa to compile the dependencies into a library
> and then just have the library as a link-time dependency for the main
> project? I can imagine things getting pretty messy once I have several
> plugins and have to concatenate all their package deps into the file.* tags
> line.
>
> martin

Re: [Caml-list] building and using a library in a subdirectory

[Martin DeMello]

[-- Attachment #1: Type: text/plain, Size: 2358 bytes --]

Okay, thanks, that does make sense. I'm curious now as to what the use case
is for a .mllib file that just defines an internal library. It doesn't seem
to be buying me anything if I don't want to compile the library separately
and use it in a different project.

martin

On Tue, Sep 22, 2015 at 12:45 AM, Gabriel Scherer <gabriel.scherer@gmail.com
> wrote:

> The compiler allows to store C compilation flags in library archives,
> but not OCaml-level flags, and in any case that would be a dubious
> idea as hardcoding the path to the ocamlfind packages on your system
> would make the (bytecode) archive libraries less portable.
>
> You can easily define the packages of the final executable in several
> steps, one per plugin:
>
>   <plugin1/**> or <main.*>: package(foo)
>   ...
>   <plugin2/**> or <main.*>: package(bar)
>
> Finally, you could also distribute each of your plugin as a separate
> ocamlfind package (allowing to combine both the link to the archive(s)
> and information about ocamlfind dependencies in a single place, the
> package's META file), but that requires re-installing a package for
> its changes to be visible from the main program -- so it is probably
> more useful for libraries that are less tightly coupled.
>
> On Tue, Sep 22, 2015 at 9:09 AM, Martin DeMello <martindemello@gmail.com>
> wrote:
> > On Mon, Sep 21, 2015 at 11:58 PM, Gabriel Scherer
> > <gabriel.scherer@gmail.com> wrote:
> >>
> >> > I don't (yet) need dynamic linking; my current main aim with the
> .mllib
> >> > setup is to specify the package dependencies for each plugin in its >
> own
> >> > section of the _tags file, and not have a long list of every plugin's
> >> > dependencies in the main module.
> >>
> >> Well, that doesn't quite work: you need to have the ocamlfind packages
> >> passed to the command linking the final executable.
> >>
> >> I suppose changing
> >>   <**/puz.*>: ...
> >> to
> >>   <**/puz.*> or <file.*>: ...
> >> could work, as this line seems to pass all the libraries flags.
> >
> >
> > Isn't there any way with a .cmxa to compile the dependencies into a
> library
> > and then just have the library as a link-time dependency for the main
> > project? I can imagine things getting pretty messy once I have several
> > plugins and have to concatenate all their package deps into the file.*
> tags
> > line.
> >
> > martin
>

[-- Attachment #2: Type: text/html, Size: 3173 bytes --]

Re: [Caml-list] building and using a library in a subdirectory

[Christian Lindig]

> On 22 Sep 2015, at 21:02, Martin DeMello <martindemello@gmail.com> wrote:
> 
> Okay, thanks, that does make sense. I'm curious now as to what the use case is for a .mllib file that just defines an internal library. It doesn't seem to be buying me anything if I don't want to compile the library separately and use it in a different project.

Like you have observed, libraries are good for distribution but are not useful within a project where you have all source files of that library available and where ocamlbuild makes sure, everything fits together.

(Conceptually, each source *.ml defines a top-level module and these modules all live in flat name space. Packaging them into libraries or distributing them over directories doesn’t introduce name spaces on the language level but might be still useful as a tool for organising source and object files.)

— C