[Caml-list] Libraries exporting external definitions in modules

[Caml-list] Libraries exporting external definitions in modules

[David Allsopp]

I've hit an unexpected gotcha using external declarations in .mli files.

I have a module where the functions exported are all C stubs and so, knowing
that exposing the external definition in the .mli file allows the compiler
to generate more efficient function calls, I'd exported the functions using
external declarations in the .mli files.

The module also contains an important piece of initialisation code which
must be run before any of the other functions will work. So, simplified, it
looks like this:

Foo.ml
------
external bar : 'a -> unit = "my_c_implementation_of_bar"
external baz : unit -> unit = "very_important_initialisation_stub"

let _ =
  baz ()

Foo.mli
-------
external bar : 'a -> unit = "my_c_implementation_of_bar"

This is then wrapped up in a .cmxa file and the .cmxa, .cmx, .a and .cmi
files installed as normal. Now I produce a program using it:

Bar.ml
------
Foo.bar 42

and compile this program using ocamlopt -o fubar foo.cmxa Bar.ml

When I run it, the initialisation code in Foo.ml calling baz never takes
place (which I can tell because it means that the call to Foo.bar, which
does take place, fails).

After much head-scratching, I realised that the reason for this is that the
external declaration in Foo.mli means that the module Foo presumably isn't
linked as its code was deemed unnecessary (as it would be if Bar.ml only
referenced types in the module Foo, rather than values).

So, my question: is that really correct behaviour? I can see why referencing
a type only is not sufficient to cause a module to be linked, but surely
referencing a value (even when it's declared external) should not exempt the
linker from actually linking the module code - at least when the module
contains top-level code (rather than just static values or functions)?

It seems a shame to slow the calls down (however trivially) by removing the
external declarations from the .mli file (i.e. declaring val bar : 'a ->
unit in the .mli) simply to keep the linker happy.


David

PS I don't expect it's relevant, but for this I was running OCaml 4.01.0 on
Linux/armv6l

Re: [Caml-list] Libraries exporting external definitions in modules

[Markus Mottl]

This bug report also describes this problem:
http://caml.inria.fr/mantis/view.php?id=4166

I think this should get fixed.  Though there is an easy (though
slightly less efficient) workaround, this problem is pretty hard to
identify, especially for people who have never seen it.

Regards,
Markus

On Fri, Dec 27, 2013 at 8:27 AM, David Allsopp <dra-news@metastack.com> wrote:
> I've hit an unexpected gotcha using external declarations in .mli files.
>
> I have a module where the functions exported are all C stubs and so, knowing
> that exposing the external definition in the .mli file allows the compiler
> to generate more efficient function calls, I'd exported the functions using
> external declarations in the .mli files.
>
> The module also contains an important piece of initialisation code which
> must be run before any of the other functions will work. So, simplified, it
> looks like this:
>
> Foo.ml
> ------
> external bar : 'a -> unit = "my_c_implementation_of_bar"
> external baz : unit -> unit = "very_important_initialisation_stub"
>
> let _ =
>   baz ()
>
> Foo.mli
> -------
> external bar : 'a -> unit = "my_c_implementation_of_bar"
>
> This is then wrapped up in a .cmxa file and the .cmxa, .cmx, .a and .cmi
> files installed as normal. Now I produce a program using it:
>
> Bar.ml
> ------
> Foo.bar 42
>
> and compile this program using ocamlopt -o fubar foo.cmxa Bar.ml
>
> When I run it, the initialisation code in Foo.ml calling baz never takes
> place (which I can tell because it means that the call to Foo.bar, which
> does take place, fails).
>
> After much head-scratching, I realised that the reason for this is that the
> external declaration in Foo.mli means that the module Foo presumably isn't
> linked as its code was deemed unnecessary (as it would be if Bar.ml only
> referenced types in the module Foo, rather than values).
>
> So, my question: is that really correct behaviour? I can see why referencing
> a type only is not sufficient to cause a module to be linked, but surely
> referencing a value (even when it's declared external) should not exempt the
> linker from actually linking the module code - at least when the module
> contains top-level code (rather than just static values or functions)?
>
> It seems a shame to slow the calls down (however trivially) by removing the
> external declarations from the .mli file (i.e. declaring val bar : 'a ->
> unit in the .mli) simply to keep the linker happy.
>
>
> David
>
> PS I don't expect it's relevant, but for this I was running OCaml 4.01.0 on
> Linux/armv6l
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs



-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com

Re: [Caml-list] Libraries exporting external definitions in modules

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 3005 bytes --]

There is an easy workaround: Don't put the external declaration into the
mli, just a val. When installing the library, take care to also include
the cmx file. This way, the extra function wrapper can be eliminated by
the automatic inliner of ocamlopt.

Of course, this works only for native code.

Gerd

Am Freitag, den 27.12.2013, 13:27 +0000 schrieb David Allsopp:
> I've hit an unexpected gotcha using external declarations in .mli files.
> 
> I have a module where the functions exported are all C stubs and so, knowing
> that exposing the external definition in the .mli file allows the compiler
> to generate more efficient function calls, I'd exported the functions using
> external declarations in the .mli files.
> 
> The module also contains an important piece of initialisation code which
> must be run before any of the other functions will work. So, simplified, it
> looks like this:
> 
> Foo.ml
> ------
> external bar : 'a -> unit = "my_c_implementation_of_bar"
> external baz : unit -> unit = "very_important_initialisation_stub"
> 
> let _ =
>   baz ()
> 
> Foo.mli
> -------
> external bar : 'a -> unit = "my_c_implementation_of_bar"
> 
> This is then wrapped up in a .cmxa file and the .cmxa, .cmx, .a and .cmi
> files installed as normal. Now I produce a program using it:
> 
> Bar.ml
> ------
> Foo.bar 42
> 
> and compile this program using ocamlopt -o fubar foo.cmxa Bar.ml
> 
> When I run it, the initialisation code in Foo.ml calling baz never takes
> place (which I can tell because it means that the call to Foo.bar, which
> does take place, fails).
> 
> After much head-scratching, I realised that the reason for this is that the
> external declaration in Foo.mli means that the module Foo presumably isn't
> linked as its code was deemed unnecessary (as it would be if Bar.ml only
> referenced types in the module Foo, rather than values).
> 
> So, my question: is that really correct behaviour? I can see why referencing
> a type only is not sufficient to cause a module to be linked, but surely
> referencing a value (even when it's declared external) should not exempt the
> linker from actually linking the module code - at least when the module
> contains top-level code (rather than just static values or functions)?
> 
> It seems a shame to slow the calls down (however trivially) by removing the
> external declarations from the .mli file (i.e. declaring val bar : 'a ->
> unit in the .mli) simply to keep the linker happy.
> 
> 
> David
> 
> PS I don't expect it's relevant, but for this I was running OCaml 4.01.0 on
> Linux/armv6l
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

RE: [Caml-list] Libraries exporting external definitions in modules

[David Allsopp]

Thanks for the replies!

Gerd Stolpmann wrote:
> There is an easy workaround: Don't put the external declaration into the
> mli, just a val. When installing the library, take care to also include
> the cmx file. This way, the extra function wrapper can be eliminated by
> the automatic inliner of ocamlopt.

Ah, I didn't realise that that was one of the optimisations which installing the .cmx file allows. In fact, that seems to be a good case for *never* using external in the .cmi file, as that way you have transparent control over a future version which just might need to put an OCaml wrapper around the stub.

> Of course, this works only for native code.

True, but the introduction of native dynlink (on most platforms, at least) does seem to make worrying about very small optimisations (i.e. non-algorithmic) for bytecode silly :o)

Markus Mottl wrote:
> This bug report also describes this problem:
> http://caml.inria.fr/mantis/view.php?id=4166

ocamldoc/trunk/manual/cmds/intf-c.etex was updated by Damien at the end of November with a warning in the explanation of the [external] keyword that this can happen (was glad I spotted that *before* attaching a documentation patch to your PR!)

> I think this should get fixed.  Though there is an easy (though slightly
> less efficient) workaround, this problem is pretty hard to identify,
> especially for people who have never seen it.

Judging by Jacques note connecting it to PR#6063 it will be soon - though given that installing the .cmx files enables the optimisation as well, I'm wondering if never using [external] in .mli files is actually the better route?


David