Lazy modules

Lazy modules

[Dario Teixeira]

Hi,

I've come across a problem which though novel to me, I presume must be
familiar to those with more ML baggage.  Namely, I need a module whose
values are not known at the initialisation stage, since they can only
be determined after reading a configuration file.  If this were a lone
value, I would declare it as a lazy value which when forced would read
from the configuration file.  But how can I achieve similar semantics
with a whole module? 

In a sense I need a lazy module.  Also, note that I find the solution
of declaring all values in that module as lazy a bit inelegant.

I do have a solution which though hackish and relying on a language
extension (local modules) seems to work: I create the module on demand
via a functor parameterised over an empty sig:

module Socket_maker (S: sig end) : Client.SOCKET =
struct
    let sockaddr = !Config.sockaddr
    let sockdomain = !Config.sockdomain
    let socktype = !Config.socktype
    let sockproto = !Config.sockproto
end

let foo =
    let module Socket = Socket_maker (struct end) in
    ...

But I wonder a) what's the penalty associated with the runtime application
of the functor, and b) if there's some better way of doing this.  Any thoughts?

Thanks for your time!
Best regards,
Dario Teixeira

Re: [Caml-list] Lazy modules

[Alain Frisch]

On 17/03/2010 18:04, Dario Teixeira wrote:
> In a sense I need a lazy module.  Also, note that I find the solution
> of declaring all values in that module as lazy a bit inelegant.
>
> I do have a solution which though hackish and relying on a language
> extension (local modules) seems to work: I create the module on demand
> via a functor parameterised over an empty sig:

Here is a variant of your version with first-class modules (which will 
be available in OCaml 3.12). Compared to your version, we get the 
benefit of laziness (the "functor body" is evaluated only once).


module Config =
   struct
     let x = ref "XXX"
   end

module type S =
   sig
     val x: string
   end

let lazy_module =
   lazy
     (
      let module M =
        struct
          let x = !Config.x
        end
      in
      (module M : S)
     )

let foo () =
   let module M = (val Lazy.force lazy_module : S) in
   M.x


-- Alain

Re: [Caml-list] Lazy modules

[Dario Teixeira]

Hi,

> Here is a variant of your version with first-class modules
> (which will be available in OCaml 3.12). Compared to your
> version, we get the benefit of laziness (the "functor body"
> is evaluated only once).

I see, thanks.  That is actually a very welcome feature; I have in the
past attempted to use modules as first-class values, and found the lack
thereof a bit disappointing, as it "breaks the symmetry" between values
and modules.

Anyway, 3.12 is looking interesting -- are there any other features we
should be aware of?  (I'm hoping GADTs will be in the pipeline eventually...)

Cheers,
Dario Teixeira

RE: [Caml-list] Lazy modules

[David Allsopp]

Dario Teixeira wrote:
> I've come across a problem which though novel to me, I presume must be
> familiar to those with more ML baggage.  Namely, I need a module whose
> values are not known at the initialisation stage, since they can only be
> determined after reading a configuration file.  If this were a lone
> value, I would declare it as a lazy value which when forced would read
> from the configuration file.  But how can I achieve similar semantics
> with a whole module?

I've hit this similarly with databases - if for example you have a signature DB and modules, say, OracleDB, MSSQLDB, PostgresDB. What I'd wanted in the past is a module DB which is one of those three but determined at runtime when a configuration file is loaded. I couldn't find a satisfactory solution with OCaml 3.09 at the time which didn't involve recompiling on each change.

> I do have a solution which though hackish and relying on a language
> extension (local modules) seems to work: I create the module on demand
> via a functor parameterised over an empty sig:

AFAIK local modules is a syntax extension not a compiler extension - I expect (not looked at it) that the syntax extension simply alpha renames all the local module declarations to make them unique and puts them globally... a very useful extension but no expressive power added. 

> module Socket_maker (S: sig end) : Client.SOCKET = struct
>     let sockaddr = !Config.sockaddr
>     let sockdomain = !Config.sockdomain
>     let socktype = !Config.socktype
>     let sockproto = !Config.sockproto
> end
> 
> let foo =
>     let module Socket = Socket_maker (struct end) in
>     ...
> 
> But I wonder a) what's the penalty associated with the runtime
> application of the functor, and

The module system at present is a compile time feature (I think that's universally true - even with weird things like recursive modules) - functors are simply a way of introducing more modules so there is no runtime overhead in using a functor. There are I believe some performance overheads in using modules if you care about speed as some optimisations in ocamlopt can't happen across module boundaries. My impression has been that if you're that worried about these slight performance hits then maybe OCaml is the wrong language for you :o)

> b) if there's some better way of doing this.  Any thoughts?

I believe that the module system due for OCaml 3.12.0 will allow this kind of runtime application of functors as modules are first class values.

Hope that's helpful - the inability to do what you're wanting to do is one of the reasons that I've never delved deeply into the module system - powerful as it may be, it didn't seem to be able to help performing a simple task (similar to yours) that I wanted it to do (I have also in the past wanted to exactly what you're doing - i.e. a module of loaded configuration values).


David

Re: [Caml-list] Lazy modules

[Alain Frisch]

On 3/17/2010 6:42 PM, David Allsopp wrote:
> AFAIK local modules is a syntax extension not a compiler extension - I
> expect (not looked at it) that the syntax extension simply alpha renames
> all the local module declarations to make them unique and puts them
> globally... a very useful extension but no expressive power added.

This is not true. Local modules are not lifted in any way. This is not 
simply a syntax extension. For instance, if the local module has 
toplevel side-effects (e.g. a structure item like: let () = 
print_endline "Hello"), then the side effect will occur every time the 
local module is evaluated.

At runtime, a structure is represented simply by a block with GC tag 0, 
exactly as a record or a tuple. The block contains dynamic components of 
the structure (values, sub-modules, exceptions, classes) in the order 
given by its signature. Evaluating a structure simply evaluates its 
runtime components a build the block.

A functor is represented as a function.

>The module system at present is a compile time feature (I think that's
> universally true - even with weird things like recursive modules) -
> functors are simply a way of introducing more modules so there is no
> runtime overhead in using a functor.

Modules and functors are much more dynamic than what you believe. The 
introduction of first-class module did not require any change in the way 
modules are compiled.

A local module which is a functor application really applies the functor 
at runtime and evaluates the functor body every time the local module 
expression is evaluated.


Alain

RE: [Caml-list] Lazy modules

[David Allsopp]

Alain Frisch wrote:
> On 3/17/2010 6:42 PM, David Allsopp wrote:
> > AFAIK local modules is a syntax extension not a compiler extension - I
> > expect (not looked at it) that the syntax extension simply alpha
> > renames all the local module declarations to make them unique and puts
> > them globally... a very useful extension but no expressive power
> > added.
> 
> This is not true. Local modules are not lifted in any way. This is not
> simply a syntax extension. For instance, if the local module has
> toplevel side-effects (e.g. a structure item like: let () =
> print_endline "Hello"), then the side effect will occur every time the
> local module is evaluated.

I've muddled this with something else! I seem to have muddled a lot in my post... :$

> At runtime, a structure is represented simply by a block with GC tag 0,
> exactly as a record or a tuple. The block contains dynamic components of
> the structure (values, sub-modules, exceptions, classes) in the order
> given by its signature. Evaluating a structure simply evaluates its
> runtime components a build the block.
> 
> A functor is represented as a function.
> 
> >The module system at present is a compile time feature (I think that's
> >universally true - even with weird things like recursive modules) -
> >functors are simply a way of introducing more modules so there is no
> >runtime overhead in using a functor.
> 
> Modules and functors are much more dynamic than what you believe. The
> introduction of first-class module did not require any change in the way
> modules are compiled.
> 
> A local module which is a functor application really applies the functor
> at runtime and evaluates the functor body every time the local module
> expression is evaluated.

Live 'n learn - thanks for the correction!


David

Re: [Caml-list] Lazy modules

[Yoann Padioleau]

On Mar 17, 2010, at 11:23 AM, Alain Frisch wrote:

> 
> On 3/17/2010 6:42 PM, David Allsopp wrote:
>> AFAIK local modules is a syntax extension not a compiler extension - I
>> expect (not looked at it) that the syntax extension simply alpha renames
>> all the local module declarations to make them unique and puts them
>> globally... a very useful extension but no expressive power added.
> 
> This is not true. Local modules are not lifted in any way. This is not simply a syntax extension. For instance, if the local module has toplevel side-effects (e.g. a structure item like: let () = print_endline "Hello"), then the side effect will occur every time the local module is evaluated.
> 
> At runtime, a structure is represented simply by a block with GC tag 0, exactly as a record or a tuple.

Which makes me wonder why people not use simply record or tuples to solve such problem ...
Many times I've found that passing a record of functions was as good as functors and simply
more flexible because they are in the same "value" world. 

> The block contains dynamic components of the structure (values, sub-modules, exceptions, classes) in the order given by its signature. Evaluating a structure simply evaluates its runtime components a build the block.
> 
> A functor is represented as a function.
> 
>> The module system at present is a compile time feature (I think that's
>> universally true - even with weird things like recursive modules) -
>> functors are simply a way of introducing more modules so there is no
>> runtime overhead in using a functor.
> 
> Modules and functors are much more dynamic than what you believe. The introduction of first-class module did not require any change in the way modules are compiled.
> 
> A local module which is a functor application really applies the functor at runtime and evaluates the functor body every time the local module expression is evaluated.
> 
> 
> Alain
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 

RE: [Caml-list] Lazy modules

[Dario Teixeira]

Hi,

> AFAIK local modules is a syntax extension not a compiler
> extension - I expect (not looked at it) that the syntax
> extension simply alpha renames all the local module
> declarations to make them unique and puts them globally... a
> very useful extension but no expressive power added.

But if that were true, wouldn't the functor instantiation happen at
initialisation time, thus preventing the delayed instantiation that
is key for this solution to work?


> I believe that the module system due for OCaml 3.12.0 will
> allow this kind of runtime application of functors as
> modules are first class values.

Again, I'm under the impression that functor application can already
(with 3.11 at least) occur at runtime when local modules are used.
(Or are you talking about different things?).  For example:

# module Foo (S: sig end) = struct let () = print_endline "foo!" end;;
module Foo : functor (S : sig  end) -> sig  end

# let hello () = let module F = Foo (struct end) in ();;
val hello : unit -> unit = <fun>

# hello ();;
foo!
- : unit = ()


> Hope that's helpful - the inability to do what you're
> wanting to do is one of the reasons that I've never delved
> deeply into the module system - powerful as it may be, it
> didn't seem to be able to help performing a simple task
> (similar to yours) that I wanted it to do (I have also in
> the past wanted to exactly what you're doing - i.e. a module
> of loaded configuration values).

Yep, Alain confirmed that modules as first-class values are indeed
coming for 3.12.  Ocaml's module system just got more interesting...

Cheers,
Dario Teixeira

RE: [Caml-list] Lazy modules

[David Allsopp]

Dario Teixeira wrote:
> Hi,
> 
> > AFAIK local modules is a syntax extension not a compiler extension - I
> > expect (not looked at it) that the syntax extension simply alpha
> > renames all the local module declarations to make them unique and puts
> > them globally... a very useful extension but no expressive power
> > added.
> 
> But if that were true, wouldn't the functor instantiation happen at
> initialisation time, thus preventing the delayed instantiation that is
> key for this solution to work?

Yup, somewhat embarrassingly I seemed to be barking in the wrong forest there, let alone up the wrong tree!!

> > I believe that the module system due for OCaml 3.12.0 will allow this
> > kind of runtime application of functors as modules are first class
> > values.
> 
> Again, I'm under the impression that functor application can already
> (with 3.11 at least) occur at runtime when local modules are used.
> (Or are you talking about different things?).  For example:
> 
> # module Foo (S: sig end) = struct let () = print_endline "foo!" end;;
> module Foo : functor (S : sig  end) -> sig  end
> 
> # let hello () = let module F = Foo (struct end) in ();; val hello :
> unit -> unit = <fun>
> 
> # hello ();;
> foo!
> - : unit = ()
> 
> 
> > Hope that's helpful - the inability to do what you're wanting to do is
> > one of the reasons that I've never delved deeply into the module
> > system - powerful as it may be, it didn't seem to be able to help
> > performing a simple task (similar to yours) that I wanted it to do (I
> > have also in the past wanted to exactly what you're doing - i.e. a
> > module of loaded configuration values).
> 
> Yep, Alain confirmed that modules as first-class values are indeed
> coming for 3.12.  Ocaml's module system just got more interesting...

Definitely!


David

Re: [Caml-list] Lazy modules

[Goswin von Brederlow]

"David Allsopp" <dra-news@metastack.com> writes:

> Dario Teixeira wrote:
>> Hi,
>> 
>> > AFAIK local modules is a syntax extension not a compiler extension - I
>> > expect (not looked at it) that the syntax extension simply alpha
>> > renames all the local module declarations to make them unique and puts
>> > them globally... a very useful extension but no expressive power
>> > added.
>> 
>> But if that were true, wouldn't the functor instantiation happen at
>> initialisation time, thus preventing the delayed instantiation that is
>> key for this solution to work?
>
> Yup, somewhat embarrassingly I seemed to be barking in the wrong forest there, let alone up the wrong tree!!
>
>> > I believe that the module system due for OCaml 3.12.0 will allow this
>> > kind of runtime application of functors as modules are first class
>> > values.
>> 
>> Again, I'm under the impression that functor application can already
>> (with 3.11 at least) occur at runtime when local modules are used.
>> (Or are you talking about different things?).  For example:
>> 
>> # module Foo (S: sig end) = struct let () = print_endline "foo!" end;;
>> module Foo : functor (S : sig  end) -> sig  end
>> 
>> # let hello () = let module F = Foo (struct end) in ();; val hello :
>> unit -> unit = <fun>
>> 
>> # hello ();;
>> foo!
>> - : unit = ()
>> 
>> 
>> > Hope that's helpful - the inability to do what you're wanting to do is
>> > one of the reasons that I've never delved deeply into the module
>> > system - powerful as it may be, it didn't seem to be able to help
>> > performing a simple task (similar to yours) that I wanted it to do (I
>> > have also in the past wanted to exactly what you're doing - i.e. a
>> > module of loaded configuration values).
>> 
>> Yep, Alain confirmed that modules as first-class values are indeed
>> coming for 3.12.  Ocaml's module system just got more interesting...
>
> Definitely!
>
>
> David

Wouldn't objects work here too? You define a virtual class with the
common interface the different configs share and define all your
different config classes and a loader function that dispatches the
creation of the right object. Then you do:

let config = load_config ()

config#say_hello;
config#open_socket;
...

MfG
        Goswin