[Caml-list] Installing library with hidden modules

[Caml-list] Installing library with hidden modules

[Alexey Egorov]

Hello,

I'm trying to build a library where most of the modules are hidden and only exported as aliases via "main" file.
Simple example here -  https://gist.github.com/anonymous/2f39bbc4999670cecf22f369f83e1eb3
So I don't want for Private module to be exposed to the end user directly, but I need it to be available as Public.A module.

How do I install such package? When I install (using ocamlfind) only 'public.*' files, it gives me an error 'This is an alias for module Private, which is missing'. If I install all of compiled files - module 'Private' still accessible via it's name...

Re: [Caml-list] Installing library with hidden modules

[Daniel Bünzli]

On Sunday, 7 May 2017 at 07:24, Alexey Egorov wrote:

> How do I install such package? When I install (using ocamlfind) only 'public.*' files, it gives me an error 'This is an alias for module Private, which is missing'. If I install all of compiled files - module 'Private' still accessible via it's name...

You can't hide toplevel names with module aliases. 

D

Re: [Caml-list] Installing library with hidden modules

[Ivan Gotovchits]

[-- Attachment #1: Type: text/plain, Size: 1219 bytes --]

Hi Alexey,

The generic recipe is to hide (not install) the cmi file, while still
installing cmx and other object files of the private module. An absence of
the cmi file, will prevent users from accessing the module directly. The
OASIS system provides an easy way to hide modules, with the
`InternalModules` field. OASIS will install all the necessary parts of the
module (i.e., cmxs, cma, o, a, etc), but will not install the interface
part.

Best wishes,
Ivan Gotovchits

On Mon, May 8, 2017 at 6:56 AM, Daniel Bünzli <daniel.buenzli@erratique.ch>
wrote:

>
>
> On Sunday, 7 May 2017 at 07:24, Alexey Egorov wrote:
>
> > How do I install such package? When I install (using ocamlfind) only
> 'public.*' files, it gives me an error 'This is an alias for module
> Private, which is missing'. If I install all of compiled files - module
> 'Private' still accessible via it's name...
>
> You can't hide toplevel names with module aliases.
>
> D
>
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 2045 bytes --]

Re: [Caml-list] Installing library with hidden modules

[Daniel Bünzli]

On Monday, 8 May 2017 at 14:49, Ivan Gotovchits wrote:
> An absence of the cmi file, will prevent users from accessing the module directly. The OASIS system provides an easy way to hide modules, with the `InternalModules` field. OASIS will install all the necessary parts of the module (i.e., cmxs, cma, o, a, etc), but will not install the interface part.


If I'm not mistaken you can't do that with module aliases, you'll need the cmi files of the right hand side of the alias (`module M : sig end = M` is invalid syntax). 

Besides it's not because you hide the cmi files that the names won't show up at link time. If your "hidden" toplevel names are generic (let's say `Config`), they will still prevent linking with other libraries that define the same generic names.

Again, you can't hide toplevel names using module aliases.

Best, 

Daniel

Re: [Caml-list] Installing library with hidden modules

[Ivan Gotovchits]

[-- Attachment #1: Type: text/plain, Size: 3920 bytes --]

Hi Daniel,

You're absolutely right in all details. And yes the devil is in details. I
should be more precise and disclose all the ingredients of my recipe.
Indeed, we are not using module aliases, and yes hiding the cmi file, of
course, will not fix the flat namespace of compilation units. So, the
solution that we are using for our libraries in CMU is pretty simple, for a
library/project named LIB do the following:

1. prefix all internal modules with the `LIB_`
2. provide a public module `LIB.ml` that reexports some of the internals
3. describe the public interface in `LIB.mli`
4. do not install `cmi` and `mli` files for the internal modules.

The public interface, of course, should not mention any internals.

Given a concrete example, suppose we want to implement a Monad library and
name it `monad`. The project structure would be the following:

```
monad_types.ml
monad_common.ml
monad_state.ml
monad_state.mli
monad_reader.ml
monad_reader.mli
...
monad.ml
monad.mli
```

A possible contents of the `monad.ml`:

```
include Monad_types
module State = Monad_state
module Reader = Monad_reader
...
```

The contents of the `monad.mli`

```
(** Lot's of docs *)
module type S = sig
   type 'a t
   val map : 'a t -> ('a -> 'b) -> 'b t
   val bind : 'a t -> ('a -> 'b t) -> 'b t
   val return : 'a -> 'a t
   ...
end

...

module State : sig
    type ('a,'e) t
    include S2 with type ('a,'e) t := ('a,'e) t
    ...
    val run : ('a,'e) t -> 'e -> 'a * 'e
end

...
```



The most controversial part is the duplication of the `mli` files. Of
course, one can omit `mli` for the internals, but I myself is a strong
proponent of writing mli files, and will not go that way. Thus, we decided
to support the two sets of mli files. The internal set, describes our
internal developer's interface, that is sometimes richer and less
stable than the external. The external defines the public API, that is
stable and versioned, it also must be well-documented. The internal
interfaces usually do not contain documents, except technical notes and
implementation details.


For a real life example, look at the graphlib library:

library structure:
https://github.com/BinaryAnalysisPlatform/bap/tree/master/lib/graphlib
oasis file:
https://github.com/BinaryAnalysisPlatform/bap/blob/master/oasis/graphlib


P.S. This is not the only solution. Another, probably better, solution,
that I was using before, is to use packing. It is much nicer in the sense,
that it solves the flat namespace issue. However, OASIS messes with the
packed modules, so we were forced to use this, more ad-hoc approach.

P.P.S. Strictly speaking we do not need to install even the `cmx` of the
internal modules, however, their absence will prevent cross-module
optimization if my understanding of the latter is correct.

Best wishes,
Ivan


On Tue, May 9, 2017 at 5:01 PM, Daniel Bünzli <daniel.buenzli@erratique.ch>
wrote:

> On Monday, 8 May 2017 at 14:49, Ivan Gotovchits wrote:
> > An absence of the cmi file, will prevent users from accessing the module
> directly. The OASIS system provides an easy way to hide modules, with the
> `InternalModules` field. OASIS will install all the necessary parts of the
> module (i.e., cmxs, cma, o, a, etc), but will not install the interface
> part.
>
>
> If I'm not mistaken you can't do that with module aliases, you'll need the
> cmi files of the right hand side of the alias (`module M : sig end = M` is
> invalid syntax).
>
> Besides it's not because you hide the cmi files that the names won't show
> up at link time. If your "hidden" toplevel names are generic (let's say
> `Config`), they will still prevent linking with other libraries that define
> the same generic names.
>
> Again, you can't hide toplevel names using module aliases.
>
> Best,
>
> Daniel
>
>
>

[-- Attachment #2: Type: text/html, Size: 5600 bytes --]

Re: [Caml-list] Installing library with hidden modules

[Daniel Bünzli]

On Wednesday, 10 May 2017 at 14:17, Ivan Gotovchits wrote:
> So, the solution that we are using for our libraries in CMU is pretty simple, for a library/project named LIB do the following:
> 
> 1. prefix all internal modules with the `LIB_` 
> 2. provide a public module `LIB.ml` that reexports some of the internals 
> 3. describe the public interface in `LIB.mli` 
> 4. do not install `cmi` and `mli` files for the internal modules. 
> 
> The public interface, of course, should not mention any internals.

Yes that's exactly what I systematically do as well. 

Leaving aside the prefixing, the only drawback is that for large libraries this will always link in all the internal modules even if there are some your program doesn't use. Module aliases can solve this problem but I dislike the plethora of redundant names it introduces to the end-user of the API and you are still left prefixing your internal modules with `LIB_`.

It's funny (or sad) to have that powerful module system with abstraction and name hiding capabilities in your hands and you end up having to do what you do in C to ensure naming isolation among libraries. 

I hope one day we can solve this problem without having to prefix the internal modules, without force linking all the internal modules, without multiplying the names that are exposed to the end user of the API and with generated documentation navigation that exactly matches how module items are accessed in your API.

Best, 

Daniel