[Caml-list] Native dynlink and reloading modules

[Caml-list] Native dynlink and reloading modules

[Richard W.M. Jones]

[-- Attachment #1: Type: text/plain, Size: 738 bytes --]


I'm a bit surprised to find that native dynlink doesn't work in the
same way as bytecode dynlink in respect to reloading the same module.
(See attached test program)

In bytecode dynlink, reloading (ie. Dynlink.loadfile) the same module
causes the new module code to override the old module code:

  $ ./dynlink_test
  testing bytecode ...
  a
  b

But in native dynlink, the new module is silently discarded and the
old code appears to run:

  $ ./dynlink_test
  testing native ...
  a
  a

I would classify this as a bug, but I'm not quite sure what is
expected to happen.  Is there some other way to override a module as
in bytecode?

  $ ocaml -version
  The Objective Caml toplevel, version 3.12.1

Rich.

-- 
Richard Jones
Red Hat

[-- Attachment #2: dynlink_test.ml --]
[-- Type: text/plain, Size: 1142 bytes --]

(* Save this file as 'dynlink_test.ml'
 * and compile it with:
 *   ocamlc dynlink.cma dynlink_value.ml dynlink_test.ml -o dynlink_test
 * or:
 *   ocamlopt dynlink.cmxa dynlink_value.ml dynlink_test.ml -o dynlink_test
 *)

open Printf

let native = Dynlink.is_native
let suffix = if native then "cmxs" else "cmo"

let write_test_module v =
  let chan = open_out "dynlink_test_module.ml" in
  fprintf chan "Dynlink_value.value := \"%s\"\n" v;
  close_out chan;

  let cmd =
    if native then
      "ocamlopt -c dynlink_test_module.ml &&\nocamlopt -shared dynlink_test_module.cmx -o dynlink_test_module.cmxs"
    else
      "ocamlc -c dynlink_test_module.ml" in
  assert (Sys.command cmd = 0)

let main () =
  printf "testing %s ...\n" (if native then "native" else "bytecode");

  write_test_module "a";
  Dynlink.loadfile (sprintf "dynlink_test_module.%s" suffix);
  print_endline !Dynlink_value.value;

  write_test_module "b";
  Dynlink.loadfile (sprintf "dynlink_test_module.%s" suffix);
  print_endline !Dynlink_value.value

let () =
  try main ()
  with
  | Dynlink.Error err ->
    eprintf "dynlink: %s\n" (Dynlink.error_message err)

[-- Attachment #3: dynlink_value.ml --]
[-- Type: text/plain, Size: 67 bytes --]

(* Save this file as 'dynlink_value.ml' *)

let value = ref "none"

Re: [Caml-list] Native dynlink and reloading modules

[Alain Frisch]

On 03/22/2012 12:47 PM, Richard W.M. Jones wrote:
>
> I'm a bit surprised to find that native dynlink doesn't work in the
> same way as bytecode dynlink in respect to reloading the same module.
> (See attached test program)
>
> In bytecode dynlink, reloading (ie. Dynlink.loadfile) the same module
> causes the new module code to override the old module code:
>
>    $ ./dynlink_test
>    testing bytecode ...
>    a
>    b
>
> But in native dynlink, the new module is silently discarded and the
> old code appears to run:
>
>    $ ./dynlink_test
>    testing native ...
>    a
>    a
>
> I would classify this as a bug, but I'm not quite sure what is
> expected to happen.  Is there some other way to override a module as
> in bytecode?

natdynlink currenlty depends on the OS dynamic linker, we cannot control 
the semantics so precisely (you can try loadfile_private, but I'm not 
sure it would solve your issue). Note that doing Dynlink.loadfile 
several times on the same module name can break the soundness of the 
type system (if you change the implementation while keeping the same 
interface):

   http://caml.inria.fr/mantis/view.php?id=4229


So it's probably a bad idea anyway.

-- Alain

Re: [Caml-list] Native dynlink and reloading modules

[Richard W.M. Jones]

On Thu, Mar 22, 2012 at 01:51:57PM +0100, Alain Frisch wrote:
> On 03/22/2012 12:47 PM, Richard W.M. Jones wrote:
> >
> >I'm a bit surprised to find that native dynlink doesn't work in the
> >same way as bytecode dynlink in respect to reloading the same module.
> >(See attached test program)
> >
> >In bytecode dynlink, reloading (ie. Dynlink.loadfile) the same module
> >causes the new module code to override the old module code:
> >
> >   $ ./dynlink_test
> >   testing bytecode ...
> >   a
> >   b
> >
> >But in native dynlink, the new module is silently discarded and the
> >old code appears to run:
> >
> >   $ ./dynlink_test
> >   testing native ...
> >   a
> >   a
> >
> >I would classify this as a bug, but I'm not quite sure what is
> >expected to happen.  Is there some other way to override a module as
> >in bytecode?
> 
> natdynlink currenlty depends on the OS dynamic linker, we cannot
> control the semantics so precisely (you can try loadfile_private,
> but I'm not sure it would solve your issue).

This is all a bit, umm, unexpected.

Stepping back, the problem I'm trying to solve is how to reload a
configuration-like file into a daemon without restarting the daemon.
The "configuration-like file" is the whenjobs jobs script[1], and the
daemon is the whenjobs daemon[2].

Any ideas on how best to go about this?  Note that the whenjobs jobs
script is intentionally a Turing-complete OCaml program, so converting
it to another format is probably not going to be practical.

Rich.

[1] examples:
    http://git.annexia.org/?p=whenjobs.git;a=blob;f=tests/jobs/t201_ocaml_set_variable.ml;hb=HEAD
    http://people.redhat.com/~rjones/whenjobs/whenjobs.txt

[2] http://git.annexia.org/?p=whenjobs.git;a=blob;f=daemon/daemon.ml;h=bc4f51a2f1177e5be2692a233d7f2b850f9a55bc;hb=HEAD#l293

-- 
Richard Jones
Red Hat

Re: [Caml-list] Native dynlink and reloading modules

[Pierre Chambart]

On 03/22/2012 05:42 PM, Richard W.M. Jones wrote:
> On Thu, Mar 22, 2012 at 01:51:57PM +0100, Alain Frisch wrote:
>> On 03/22/2012 12:47 PM, Richard W.M. Jones wrote:
>>> I'm a bit surprised to find that native dynlink doesn't work in the
>>> same way as bytecode dynlink in respect to reloading the same module.
>>> (See attached test program)
>>>
>>> In bytecode dynlink, reloading (ie. Dynlink.loadfile) the same module
>>> causes the new module code to override the old module code:
>>>
>>>   $ ./dynlink_test
>>>   testing bytecode ...
>>>   a
>>>   b
>>>
>>> But in native dynlink, the new module is silently discarded and the
>>> old code appears to run:
>>>
>>>   $ ./dynlink_test
>>>   testing native ...
>>>   a
>>>   a
>>>
>>> I would classify this as a bug, but I'm not quite sure what is
>>> expected to happen.  Is there some other way to override a module as
>>> in bytecode?
>> natdynlink currenlty depends on the OS dynamic linker, we cannot
>> control the semantics so precisely (you can try loadfile_private,
>> but I'm not sure it would solve your issue).
> This is all a bit, umm, unexpected.
>
> Stepping back, the problem I'm trying to solve is how to reload a
> configuration-like file into a daemon without restarting the daemon.
> The "configuration-like file" is the whenjobs jobs script[1], and the
> daemon is the whenjobs daemon[2].
>
> Any ideas on how best to go about this?  Note that the whenjobs jobs
> script is intentionally a Turing-complete OCaml program, so converting
> it to another format is probably not going to be practical.
>
I did not really read your code, but it seams to me that you are doing
the compilation in your tool.
So you can choose the name of the module. Isn't it possible to change
the name of the file each
time you compile/reload it ?
-- 
Pierre

Re: [Caml-list] Native dynlink and reloading modules

[Richard W.M. Jones]

On Thu, Mar 22, 2012 at 06:14:03PM +0100, Pierre Chambart wrote:
> On 03/22/2012 05:42 PM, Richard W.M. Jones wrote:
> > Any ideas on how best to go about this?  Note that the whenjobs jobs
> > script is intentionally a Turing-complete OCaml program, so converting
> > it to another format is probably not going to be practical.
> >
> I did not really read your code, but it seams to me that you are doing
> the compilation in your tool.
> So you can choose the name of the module. Isn't it possible to change
> the name of the file each
> time you compile/reload it ?

Yes, I guess something involving -pack to pack all the files into
a randomly-named submodule, could be the way to go.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Native dynlink and reloading modules

[Richard W.M. Jones]

On Thu, Mar 22, 2012 at 08:12:08PM +0000, Richard W.M. Jones wrote:
> On Thu, Mar 22, 2012 at 06:14:03PM +0100, Pierre Chambart wrote:
> > On 03/22/2012 05:42 PM, Richard W.M. Jones wrote:
> > > Any ideas on how best to go about this?  Note that the whenjobs jobs
> > > script is intentionally a Turing-complete OCaml program, so converting
> > > it to another format is probably not going to be practical.
> > >
> > I did not really read your code, but it seams to me that you are doing
> > the compilation in your tool.
> > So you can choose the name of the module. Isn't it possible to change
> > the name of the file each
> > time you compile/reload it ?
> 
> Yes, I guess something involving -pack to pack all the files into
> a randomly-named submodule, could be the way to go.

Just to follow up on this.  I implemented this by packing all my files
into a randomly-named module.  ie. My files are compiled using the
-pack/-for-pack options as submodules of 'Jobs__<time>' where <time>
is the current time_t.  The top (randomly named) module is then loaded.

This works fine in both bytecode and native code.

The code is here:

http://git.annexia.org/?p=whenjobs.git;a=commitdiff;h=de72662854c3db9365296dd45cade2253910be7f

Rich.

-- 
Richard Jones
Red Hat