[Caml-list] Delaying module initialization

[Caml-list] Delaying module initialization

[Richard Jones]

I have a problem with module initialization which manifests itself in
perl4caml (thanks to Alaine Frisch for pointing this one out).

Basically the perl4caml.cma / .cmxa library consists of lots of
wrapper modules such as Pl_Template which wraps Perl's Template, and
Pl_LWP_UserAgent which wraps LWP::UserAgent, etc.

Each of these wrappers has to 'use' the corresponding Perl library
when it is initialized.  Each wrapper module has code similar to this:

open Perl
let _ = eval "use LWP::UserAgent"

This is just a requirement of Perl, that you have to 'use' a library
before you can actually use it.

Now the problem is that any program which links with perl4caml.cma
appears to immediately 'use' every Perl module, even ones for wrappers
which the program doesn't actually need.

This has two bad consequences: (1) Large amount of memory is consumed
at start-up.  (2) If you move the Perl library onto another machine
(eg. if it's compiled and bundled into a .deb), then the library won't
work unless all the corresponding Perl modules have been installed,
which is very inconvenient.

One way around this would be to force users of modules to explicitly
initialize the wrapper, eg. by calling Pl_LWP_UserAgent.init () at the
start of their program.  But if they forget to do this, then bad
things can happen.  Another way would be to split the library into
individual .cmos, although this might be inconvenient, and there are
also tricky dependency issues - Eg. what if a wrapper depends on other
wrappers being around?  In Perl everything is dynamic so one library
can depend on another library which doesn't exist, and as long as the
other library isn't used, everything will be fine.  This is not the
case with OCaml.

Does anyone have any other ideas how to solve this?

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
C2LIB is a library of basic Perl/STL-like types for C. Vectors, hashes,
trees, string funcs, pool allocator: http://www.annexia.org/freeware/c2lib/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[Yamagata Yoriyuki]

From: Richard Jones <rich@annexia.org>
Subject: [Caml-list] Delaying module initialization
Date: Fri, 26 Mar 2004 16:59:43 +0000

> Now the problem is that any program which links with perl4caml.cma
> appears to immediately 'use' every Perl module, even ones for wrappers
> which the program doesn't actually need.

Actually, no.  Among the modules in the cma file, modules really
initialized and linked are modules which contain values used in the
main program.  Other modules are not initialized.

--
Yamagata Yoriyuki

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[Richard Jones]

On Sat, Mar 27, 2004 at 02:26:27AM +0900, Yamagata Yoriyuki wrote:
> From: Richard Jones <rich@annexia.org>
> Subject: [Caml-list] Delaying module initialization
> Date: Fri, 26 Mar 2004 16:59:43 +0000
> 
> > Now the problem is that any program which links with perl4caml.cma
> > appears to immediately 'use' every Perl module, even ones for wrappers
> > which the program doesn't actually need.
> 
> Actually, no.  Among the modules in the cma file, modules really
> initialized and linked are modules which contain values used in the
> main program.  Other modules are not initialized.

I had to go back and check this, but I'm pretty sure that it is
initializing all the modules.

To verify this, I did:

$ strace -o /tmp/log ocaml -I +perl
        Objective Caml version 3.07+2

# #load "perl4caml.cma";;
# <Ctrl-D>

I then looked at /tmp/log and verified that this loads libraries such
as LWP::UserAgent (I can see it opening those .pm file on disk).  This
must, I assume, mean that the 'use' statements are being run.

Is there something else that might be causing the modules to be
initialized even though they are never used?

Here is how the wrappers are compiled and linked:

ocamlc -w s -g -I wrappers -c wrappers/pl_Date_Calc.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_Date_Format.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_Date_Parse.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_Net_Google_Cache.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_Net_Google_Response.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_Net_Google_Search.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_Net_Google_Spelling.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_Net_Google.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_HTML_Element.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_HTML_Parser.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_HTML_TreeBuilder.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_URI.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_HTTP_Message.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_HTTP_Request.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_HTTP_Response.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_LWP_UserAgent.ml
ocamlc -w s -g -I wrappers -c wrappers/pl_Template.ml
ocamlmklib -o perl4caml -rdynamic  -L/usr/local/lib /usr/lib/perl/5.8.2/auto/DynaLoader/DynaLoader.a -L/usr/lib/perl/5.8.2/CORE -lperl -ldl -lm -lpthread -lc -lcrypt perl.cmo perl_c.o wrappers/pl_Date_Calc.cmo wrappers/pl_Date_Format.cmo wrappers/pl_Date_Parse.cmo wrappers/pl_Net_Google_Cache.cmo wrappers/pl_Net_Google_Response.cmo wrappers/pl_Net_Google_Search.cmo wrappers/pl_Net_Google_Spelling.cmo wrappers/pl_Net_Google.cmo wrappers/pl_HTML_Element.cmo wrappers/pl_HTML_Parser.cmo wrappers/pl_HTML_TreeBuilder.cmo wrappers/pl_URI.cmo wrappers/pl_HTTP_Message.cmo wrappers/pl_HTTP_Request.cmo wrappers/pl_HTTP_Response.cmo wrappers/pl_LWP_UserAgent.cmo wrappers/pl_Template.cmo
Unknown option -rdynamic

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
Learning Objective CAML for C, C++, Perl and Java programmers:
http://www.merjis.com/richj/computers/ocaml/tutorial/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[Remi Vanicat]

Richard Jones <rich@annexia.org> writes:

> On Sat, Mar 27, 2004 at 02:26:27AM +0900, Yamagata Yoriyuki wrote:
>> From: Richard Jones <rich@annexia.org>
>> Subject: [Caml-list] Delaying module initialization
>> Date: Fri, 26 Mar 2004 16:59:43 +0000
>> 
>> > Now the problem is that any program which links with perl4caml.cma
>> > appears to immediately 'use' every Perl module, even ones for wrappers
>> > which the program doesn't actually need.
>> 
>> Actually, no.  Among the modules in the cma file, modules really
>> initialized and linked are modules which contain values used in the
>> main program.  Other modules are not initialized.
>
> I had to go back and check this, but I'm pretty sure that it is
> initializing all the modules.
>
> To verify this, I did:
>
> $ strace -o /tmp/log ocaml -I +perl
>         Objective Caml version 3.07+2

Well, you are using the interpreter. The interpreter cannot know which
modules of the cma you use, so it load them all. The ocamlc compiler
do know, and the compiled prgm don't need to load all those cma.




>
> # #load "perl4caml.cma";;
> # <Ctrl-D>
>
> I then looked at /tmp/log and verified that this loads libraries such
> as LWP::UserAgent (I can see it opening those .pm file on disk).  This
> must, I assume, mean that the 'use' statements are being run.

I've done a :

moi@vanicat:~/lang/ocaml$ ocamlc -o prgm -I +perl perl4caml.cma test.ml
moi@vanicat:~/lang/ocaml$ strace  -o /tmp/log ./prgm

and the/tmp/log contain no trace of any .pm.


-- 
Rémi Vanicat

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[skaller]

On Sat, 2004-03-27 at 03:59, Richard Jones wrote:
> 
> Does anyone have any other ideas how to solve this?

Hassle the ocaml team for generation of shared libraries?

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[Richard Jones]

The actual specific problem I'm having is using perl4caml inside
Apache (with mod_caml).

I think this must have something to do with Dynlink.  The manual
states:

  All toplevel expressions in the loaded compilation units are
  evaluated.

The sequence of events is very complex.  The upshot, however, is that
I can't use perl4caml inside Apache, because it loads all the Perl
libraries into every httpd process.

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
http://www.winwinsales.co.uk/ - CRM improvement consultancy

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[Xavier Leroy]

> I think this must have something to do with Dynlink.  The manual
> states:
> 
>   All toplevel expressions in the loaded compilation units are
>   evaluated.

Yes, indeed.  Compilation units loaded at top-level or dynamically via
Dynlink are always loaded and their initialization code executed,
because there is no way for the system to determine whether they'll be
needed later.  Static linking can determine this, and therefore
removes unreferenced units from .cma libraries.

J. Skaller unhelpfully suggests:

> Hassle the ocaml team for generation of shared libraries?

Shared libs will definitely not help here (and elsewhere).  By
definition, linking with a shared lib is like dynamically loading all
units contained in this lib, hence no way to eliminate unneeded ones.
Only static linking can do that.

(As far as "hassling" goes, this will only reinforce my conviction
that shared libraries are useless.  So, don't do that.)

Coming back to Rich Jones' problem, there are two ways to go about
this:

1- Since you're dynamically loading the Caml modules anyway, couldn't
you arrange so that only those you really need are loaded?

2- Instead of doing

        let _ = eval "use LWP::UserAgent"
        let somefun x y = ...

you could implement on-demand evaluation of the "use" bit, e.g.

        let inited = ref false
        let init() =
          if not !inited then begin
            eval "use LWP::UserAgent";
            inited := true
          end
        let somefun x y = init(); ...

Every function needs to be protected by a call to init().  If you
export constants and not just functions, this won't work, of course,
but maybe this doesn't happen too often.

A nicer way to write the above is

        let init = lazy (eval "use LWP::UserAgent")
        let somefun x y = Lazy.force init; ...

Hope this helps,

- Xavier Leroy

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[Richard Jones]

On Sat, Mar 27, 2004 at 12:18:32PM +0100, Xavier Leroy wrote:
> A nicer way to write the above is
> 
>         let init = lazy (eval "use LWP::UserAgent")
>         let somefun x y = Lazy.force init; ...

Ah yes, that's nice!

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
MOD_CAML lets you run type-safe Objective CAML programs inside the Apache
webserver. http://www.merjis.com/developers/mod_caml/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[Benjamin Geer]

Xavier Leroy wrote:
> (As far as "hassling" goes, this will only reinforce my conviction
> that shared libraries are useless.  So, don't do that.)

I don't mean to hassle you, but could you explain this conviction a bit? 
  If shared libraries are useless, why do all Unix and Linux programs 
use libc as a shared library?  Why don't they statically link to it? 
I've always assumed that this is because nobody wants 'ls' and 'cp' to 
be over a megabyte in size.

The lead developer of GNOME and Evolution (which are written in C) 
started the Mono project (a free implementation of .NET) because, as he 
put it:

   There is a point in your life when you realize that
   you have written enough destructors, and have spent
   enough time tracking down a memory leak, and you have
   spend enough time tracking down memory corruption, and
   you have spent enough time using low-level insecure
   functions, and you have implemented way too many linked
   lists.[1]

Now there is debate about whether to write the next version of GNOME in 
C#.  If Caml could make shared libraries, perhaps those sorts of 
libraries (very large libraries, meant to be used by many small 
programs) would have a better chance of being written in Caml.

Ben

[1] http://mail.gnome.org/archives/gnome-hackers/2002-February/msg00031.html

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[skaller]

On Sat, 2004-03-27 at 22:18, Xavier Leroy wrote:

> J. Skaller unhelpfully suggests:
> 
> > Hassle the ocaml team for generation of shared libraries?
> 
> Shared libs will definitely not help here (and elsewhere).  

I beg to differ, and I have 2-3 years experience with this
in the Vyper Python interpreter written
in Ocaml .. which was abandoned in part due to the lack of
dynamic loading: in this case I needed dynamic loading AND
native code compilation. :(

> (As far as "hassling" goes, this will only reinforce my conviction
> that shared libraries are useless.  So, don't do that.)

"Hassling" is just a colloquial term of endearment. 
I don't get annoyed because I can't read French.
I speak a foreign language too, namely 'Strine. :D

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[skaller]

On Sat, 2004-03-27 at 22:21, Richard Jones wrote:
> On Sat, Mar 27, 2004 at 12:18:32PM +0100, Xavier Leroy wrote:
> > A nicer way to write the above is
> > 
> >         let init = lazy (eval "use LWP::UserAgent")
> >         let somefun x y = Lazy.force init; ...
> 
> Ah yes, that's nice!

Yes but it won't work.

The problem with apache plugins is that the
interpreted script in an HTML page can invoke
*arbitrary* scripts.

If those scripts are interpreted dynamically then
the Ocaml bytecodes for them must also be loaded dynamically,
and then the Perl being wrapped will also be
loaded dynamically.

So there is no use of this idiom: it is quite enough
to just write:

	let init = eval "use LWP::UserAgent"

in the dynamically loaded Ocaml bytecode.
The problem is NOT to dynamically load the Perl,
but to dynamically load the Ocaml bytecodes
required by a given HTML page (and no others).

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[skaller]

On Sat, 2004-03-27 at 23:38, Benjamin Geer wrote:
> Xavier Leroy wrote:
> > (As far as "hassling" goes, this will only reinforce my conviction
> > that shared libraries are useless.  So, don't do that.)
> 
> I don't mean to hassle you, but could you explain this conviction a bit? 
>   If shared libraries are useless, why do all Unix and Linux programs 
> use libc as a shared library? 

This isn't really what I call dynamic loading though, 
but lazy static binding.

By dynamic loading *I* mean the module to load is computed
at run time depending on inputs to the program.

> If Caml could make shared libraries, perhaps those sorts of 
> libraries (very large libraries, meant to be used by many small 
> programs) would have a better chance of being written in Caml.

Yes. There is a tension between dynamically loading which is
possible with bytecode, and generation of dynamically loadable
native code.

The former is interesting but not industrially satisfying
because it doesn't allow dynamic loading of interfaces
to C, which must be native code.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[Jacques Garrigue]

From: skaller <skaller@users.sourceforge.net>

> > If Caml could make shared libraries, perhaps those sorts of 
> > libraries (very large libraries, meant to be used by many small 
> > programs) would have a better chance of being written in Caml.
> 
> Yes. There is a tension between dynamically loading which is
> possible with bytecode, and generation of dynamically loadable
> native code.
> 
> The former is interesting but not industrially satisfying
> because it doesn't allow dynamic loading of interfaces
> to C, which must be native code.

But, this is plain wrong: since 3.04 you can dynamically load C stubs
through dynamically loading a bytecode library.
I think this is a huge improvement.
And if you're going to use .NET, shouldn't you be happy with bytecode :-)

The main reason I see for wanting dynamic loading of native code is
speed, and symmetry with bytecode. These are valid reasons.
But for one I'm perfectly happy with bytecode.

Jacques Garrigue

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[skaller]

On Sun, 2004-03-28 at 20:34, Jacques Garrigue wrote:

> > The former is interesting but not industrially satisfying
> > because it doesn't allow dynamic loading of interfaces
> > to C, which must be native code.
> 
> But, this is plain wrong: since 3.04 you can dynamically load C stubs
> through dynamically loading a bytecode library.

This I do not understanding but I seem to have missed something. 

Are you saying I can write a module X.ml which wraps a C library 
with function f(), and another Y.ml which wraps a different 
function f() of another  shared library, and then write a 
program which can be run like:

loadit X
loadit Y

where the first script invokes X's f() and the second Y's f()?

That wasn't my understanding, but that is exactly what Python
and Perl can do, and it's what is required. If you can do this
you have a clumbsy way of generating code at run time and
executing it, which is a fairly minimal condition
for an extensible program. Interscript, for example, requires
this facility (it uses the Python exec feature to do it
one way, and importing of generated files as another).

Another example which is less abstract: dynamically choose
which version of GTK to run, meaning both the C shared library
used *and* the wrapper code used are chosen at run time.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[skaller]

On Sun, 2004-03-28 at 20:34, Jacques Garrigue wrote:
> From: skaller <skaller@users.sourceforge.net>

> The main reason I see for wanting dynamic loading of native code is
> speed, 

.. although the bytecode interpreter is apparently quite fast,
and for my purposes (a compiler) probably adequate (except for the
too small stack size). However I've found the native code more
robust, and since the bytecode and native code compilers use
*different* binding interfaces to C I felt I had to chose one or 
the other, and for better or worse I've gone with the native
code system. Felix actually works both ways at the moment ..
but there is no extensibility yet.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[skaller]

If dynamic loading is as Jacques suggests, then it would
seem mod_caml has a strange design..

Given an HTML page containing some Caml script
which in invokes some functions which wrap
Perl modules .. the whole thing should be 
dynamic automatically.

I can't see how 'initialising all the Perl modules'
could happen. I'd have to try extremely hard to even
think about how to make that happen.

The core apache module shouldn't be linked with
any Perl other than core interpreter. So for example
invoking the database module of Perl through a Caml
interface would be entirely dependent on whether the
client HTML contained embedded script which loaded
the module or not.

Am I somehow misunderstanding how mod_caml operates?

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[Jacques Garrigue]

From: skaller <skaller@users.sourceforge.net>
> On Sun, 2004-03-28 at 20:34, Jacques Garrigue wrote:
> 
> > > The former is interesting but not industrially satisfying
> > > because it doesn't allow dynamic loading of interfaces
> > > to C, which must be native code.
> > 
> > But, this is plain wrong: since 3.04 you can dynamically load C stubs
> > through dynamically loading a bytecode library.
> 
> This I do not understanding but I seem to have missed something. 
> 
> Are you saying I can write a module X.ml which wraps a C library 
> with function f(), and another Y.ml which wraps a different 
> function f() of another  shared library, and then write a 
> program which can be run like:
> 
> loadit X
> loadit Y
> 
> where the first script invokes X's f() and the second Y's f()?

You are mixing here different problems, and I can interpret your
question in several ways.

If your question is: can I load simultaneously several shared
libraries containing some C functions with the same name, then the
answer is probably: only under windows. But I did not try, so maybe it
works on other systems too, or maybe it doesn't work on windows for
some reason.  This is a question of C binary namespace, flat or
hierarchical.  To simplify writing libraries, ocaml uses the flat
model (more usual on Unix), but windows happens to support only the
hierarchical model.

If it is: can I choose at runtime which shared library to load, and
provide a unified interface independent of the library, then the
answer is yes. IIRC, CamlGL uses that to allow choosing your openGL
library at runtime. There are some pitfalls, like the fact you need to
provide a different stub library (both C and caml part) for each
shared library you want to load, or the fact you cannot unload a
library. But in my opinion the main problem is that different
libraries shall probably have different signatures under the refined
ocaml type system, so that it will be very hard to use a unified
interface (we have already trouble with the different version of Gtk+-2.x).

If it is: can I dynamically generate stubs for a shared library,
compile them (both C and caml part) and load the library in the same
program, then this should be yes too.  Automatizing all this might be
a lot of work, but there is no theoretical limitation.  Again there
may be problems with name spaces, but if this is really your concern
you just have to change one line in ocamlrun sources to use the
hierarchical model.

> If dynamic loading is as Jacques suggests, then it would
> seem mod_caml has a strange design..
> 
> Given an HTML page containing some Caml script
> which in invokes some functions which wrap
> Perl modules .. the whole thing should be 
> dynamic automatically.
> 
> I can't see how 'initialising all the Perl modules'
> could happen. I'd have to try extremely hard to even
> think about how to make that happen.

Here we enter another problem: ocaml has dynamic loading, but not
auto-loading/initializing.  Looks like lots of people would want to
have that: a module would be initialized only when one of its members
is used. This is _not_ a problem of dynamic loading, and this could
actually change the semantics of the language, since initializing one
module might force initializing another one in its turn, and as a result
module initialization might not be sequential.  On the other hand, this
should not be too hard to do with bytecode, at least java style: parse
the .cma's, but do not load their dependencies immediately.

Without auto-loading, you end-up loading everything in your name
space; so that would result in initializing all Perl modules if you
are initializing them on load. Some workarounds have already been
suggested to delay this initialization.

Jacques Garrigue

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Delaying module initialization

[skaller]

On Mon, 2004-03-29 at 13:36, Jacques Garrigue wrote:
> From: skaller <skaller@users.sourceforge.net>
[]
Thanks for that long explanation!
Put in FAQ?

> > If dynamic loading is as Jacques suggests, then it would
> > seem mod_caml has a strange design..

> Here we enter another problem: ocaml has dynamic loading, but not
> auto-loading/initializing.  Looks like lots of people would want to
> have that: a module would be initialized only when one of its members
> is used. 

Yes: this seems to imply the modules are pre-linked instead
of loaded dynamically.

> This is _not_ a problem of dynamic loading, and this could
> actually change the semantics of the language,

Indeed, and Xaviers solution is the way to handle this, as in C++
where the trick originates and has an interesting encoding, but
the bottom line is: if you want to control 'initialisation order'
make the 'initialisation' a user defined function and call it.

> Without auto-loading, you end-up loading everything in your name
> space; so that would result in initializing all Perl modules if you
> are initializing them on load. Some workarounds have already been
> suggested to delay this initialization.

Indeed but it seems the wrong solution to pre-load every
module you might want to use. The only viable universal 
solution is to load dynamically the same way Perl itself does.

More precisely I'd envisage the following design:

(1) Core modules are linked statically. Example:
some module used for formatting HTML is likely
to be needed in every script.

(2) A user defined list of modules is loaded
at run time initially. This introduces a delay
processing user script in an HTML page, but it
will crash the processes early if there is a failure.

On some architectures, it may be possible to do this
ONCE and suspend the process, then fork the processs
for the client page, greatly speeding up service
of the client page by reducing initialisation
to copying on demand performed by the Virtual Memory
system. I don't know if Apache can do this ..

(3) Other modules are loaded on demand.
This should be used where there is some conditional
execution, and must be where there is dynamic module 
name determination. This technique is the least reliable,
but avoids loading modules that might not be used.

An interesting module set is: ones to handle differences
is the client browser. EG: a module to uniformly
handle generation of *different* HTML for Mozilla
or Internet Explorer.

Note by "load on demand" i do not mean 'pre-load
and initialise on use' .. that's a separable technique
that might be useful for (1) or (2) if there is a lot of 
'pieces' of initialisation needed for different functions, 
and a lot of functions, only some of which are likely to be used
in a given invocation.

The three kinds of module are specified:

(1) At mod_caml construction time in the
actual Ocaml namespace: static linkage.

(2) In a configuration file. Load time linkage.

(3) In the client HTML page. Run time linkage.


-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners