Binding the Lua library [was: adding a scripting language to an ocaml program]

Binding the Lua library [was: adding a scripting language to an ocaml program]

[Paolo Donadeo]

Some time ago I started to write an OCaml binding to the Lua API
library [1] and the so called auxiliary library [2]. My objective was
a tool to give the user of an application (in this case a web
application) the power to extend the application with plugins in a
simple and safe environment. I choose Lua because I don't want to
compel my user to learn a relatively difficult language like OCaml,
nor an esoteric one like Scheme. The user, in the setting I have in
mind, is supposed to be strange to programming languages. Moreover Lua
is a beautiful language, Scheme inspired, and the model of its
development is very similar to OCaml (academic and "cathedral", but
open to the community). And Lua has been designed from its start to be
an extension language, it's *very* easy to sandbox it.

So I wrote the binding to some basic functions and a stress test
designed to reveal memory leaks of the binding. The test passed and so
I consider the code quite stable, but I had to stop the development
due to the chronic lack of time and due to a specific problem: I don't
know how to handle the impedance mismatch between the two garbage
collectors.

Lua has the concept of "userdata": from the C side you can create
opaque values (malloc allocated void* objects) along with the
operations (C functions conforming a specific typedef) to handle
those. It's easy to push the userdatum and its operations on the stack
of the Lua state and build an object (a Lua table, similar to a
Javascript object). The Lua program can use the object and call
methods. This is the way libraries like LuaSocket [3] are written.

In an ideal world one want to define an OCaml type, create values of
that type and operations on it, push everything on the stack and the
use it from the Lua side. But what happen when the Lua GC decide to
collect the userdatum? If a "metamethod" __gc is provided (ideally an
OCaml function) the Lua state calls it before deallocating the object.
But that object could be still alive for the OCaml GC, so what is the
standard way (if any) to handle these situations?

And again: how to translate (read: how to bind) in OCaml functions like these:

void *lua_newuserdata (lua_State *L, size_t size); [4]
or
void *lua_touserdata (lua_State *L, int index); [5]

Is Obj.magic the only possible way? It's ugly and, of course,
"Obj.magic is not part of the OCaml language", but in this case you
use a cast in C, so why not cast a spell in OCaml?

If anyone is interested in my prototype I could clean up the source,
remove comments in Italian and publish it on GitHub or OCamlCore.

And, of course, any ideas or help on the garbage collector(s) issue are welcome.



[1] http://www.lua.org/manual/5.1/manual.html#3
[2] http://www.lua.org/manual/5.1/manual.html#4
[3] http://w3.impa.br/~diego/software/luasocket/
[4] http://www.lua.org/manual/5.1/manual.html#lua_newuserdata
[5] http://www.lua.org/manual/5.1/manual.html#lua_touserdata

-- 
Paolo
~
~
:wq

Re: [Caml-list] Binding the Lua library [was: adding a scripting language to an ocaml program]

[Guillaume Yziquel]

Paolo Donadeo a écrit :
> 
> So I wrote the binding to some basic functions and a stress test
> designed to reveal memory leaks of the binding. The test passed and so
> I consider the code quite stable, but I had to stop the development
> due to the chronic lack of time and due to a specific problem: I don't
> know how to handle the impedance mismatch between the two garbage
> collectors.

Yes. This kind of issue is tough. I also had it in the OCaml-R code. And 
the solution is far from perfect, but works. I do not know about Lua, 
though.

> And again: how to translate (read: how to bind) in OCaml functions like these:
> 
> void *lua_newuserdata (lua_State *L, size_t size); [4]
> or
> void *lua_touserdata (lua_State *L, int index); [5]
> 
> Is Obj.magic the only possible way? It's ugly and, of course,
> "Obj.magic is not part of the OCaml language", but in this case you
> use a cast in C, so why not cast a spell in OCaml?

You might want to check out the code sample I published here (at the 
bottom).

	http://ocaml.janestreet.com/?q=node/59

Unfortunately, using this recursive subtyping trick (which can be 
extended to more sophisticated recursive subtyping between more than 2 
type systems) raises issues in the presence of OCaml classes: The 
compiler goes in an infinite loop when doing type inference. The issue 
should be solved in 3.12.

It's not yet used in the OCaml-R binding because of compiler infinite 
loop problem. But it is used in my (unpublished) Python binding.

> If anyone is interested in my prototype I could clean up the source,
> remove comments in Italian and publish it on GitHub or OCamlCore.

I'm always interested in language bindings... And I think a few other 
people are interested in an Lua binding.

> And, of course, any ideas or help on the garbage collector(s) issue are welcome.

I do not know much about Lua's GC. My experience (for R, Python and 
Java) is that it's doable. The first shot is often imperfect and 
inefficient becauses GCs are not designed in the spirit of being binded. 
(R's API only provides a cumbersome stack-based mechanism to interact 
with R's GC, which is wholly inadequate).

I guess that with more available time, it is indeed possible to do clean 
tricks in order to get the GCs to cooperate efficiently... (I have a few 
tricks in mind for R that will have to wait.)

All the best,

-- 
      Guillaume Yziquel
http://yziquel.homelinux.org/

Re: Binding the Lua library [was: adding a scripting language to an ocaml program]

[Sylvain Le Gall]

On 06-07-2010, Guillaume Yziquel <guillaume.yziquel@citycable.ch> wrote:
> Paolo Donadeo a écrit :
>
>> If anyone is interested in my prototype I could clean up the source,
>> remove comments in Italian and publish it on GitHub or OCamlCore.
>
> I'm always interested in language bindings... And I think a few other 
> people are interested in an Lua binding.
>

I think Guillaume is refering to me. I am indeed quite interested in
this kind of binding, especially to measure the performance of the Lua
language interacting with OCaml. I would use Lua to create functions
to process huge amount of data and to replace a DSL I created.

So far, we have:
http://bitbucket.org/dpowers/luacaml by David Powers
a prototype by Paolo Donadeo
lua-ml by Christian Lindig

Maybe we can join our forces to create a common binding to Lua from
OCaml.

I propose to host it at http://forge.ocamlcore.org, we can even have an
hg repository if needed.

If anyone is interested, I think more forces to this effort are welcome.

I would be glad that at least David, Paolo and Christian join ;-)

Regards,
Sylvain Le Gall

ps: right now I can provide some time to setup the forge account and web
pages, but I will have to dedicate a lot of time to real life starting
on Friday (or before -- depending the time my daughter will choose to
born).

Re: [Caml-list] Re: Binding the Lua library [was: adding a scripting language to an ocaml program]

[Martin DeMello]

On Wed, Jul 7, 2010 at 7:20 PM, Sylvain Le Gall <sylvain@le-gall.net> wrote:
>
> I think Guillaume is refering to me. I am indeed quite interested in
> this kind of binding, especially to measure the performance of the Lua
> language interacting with OCaml. I would use Lua to create functions
> to process huge amount of data and to replace a DSL I created.

I'm very interested too, mainly because a hobby DSL of mine is stalled
for lack of a good way to take a user specified function f : String ->
String and map it over an OCaml list of strings. Depending on how the
bindings look, I might invert control altogether and write it as a lua
dsl tied to an OCaml engine.

martin

Re: [Caml-list] Binding the Lua library [was: adding a scripting language to an ocaml program]

[Paolo Donadeo]

On Tue, Jul 6, 2010 at 23:28, Guillaume Yziquel
<guillaume.yziquel@citycable.ch> wrote:
>> And, of course, any ideas or help on the garbage collector(s) issue are
>> welcome.
> I do not know much about Lua's GC. My experience (for R, Python and Java) is
> that it's doable.

The GC implemented in Lua [1] is an incremental mark-and-sweep
collector. Since Lua type system is simple (nil, boolean, number,
string, function, userdata, thread, for coroutines, and tables) it
automatically collects everything without problems. The strategy for
basic types is to copy values from the C (or OCaml) side to the Lua
stack: this copy decouples the two GC and sets them free to deallocate
values.

Problems arise when one tries to share userdata or closures between
OCaml and Lua. In this case it's not clear (at least to me) how to
approch the problem.



[1] http://www.lua.org/manual/5.1/manual.html#2.10


-- 
Paolo ⠠⠵

Re: [Caml-list] Binding the Lua library [was: adding a scripting language to an ocaml program]

[Török Edwin]

On Wed, 7 Jul 2010 22:36:28 +0200
Paolo Donadeo <p.donadeo@gmail.com> wrote:

> On Tue, Jul 6, 2010 at 23:28, Guillaume Yziquel
> <guillaume.yziquel@citycable.ch> wrote:
> >> And, of course, any ideas or help on the garbage collector(s)
> >> issue are welcome.
> > I do not know much about Lua's GC. My experience (for R, Python and
> > Java) is that it's doable.
> 
> The GC implemented in Lua [1] is an incremental mark-and-sweep
> collector. Since Lua type system is simple (nil, boolean, number,
> string, function, userdata, thread, for coroutines, and tables) it
> automatically collects everything without problems. The strategy for
> basic types is to copy values from the C (or OCaml) side to the Lua
> stack: this copy decouples the two GC and sets them free to deallocate
> values.
> 
> Problems arise when one tries to share userdata or closures between
> OCaml and Lua. In this case it's not clear (at least to me) how to
> approch the problem.
> 
> 
> 
> [1] http://www.lua.org/manual/5.1/manual.html#2.10

How about translating a program in lua (or lua bytecode) to OCaml
sourcecode automatically?
Would that be possible?

Best regards,
--Edwin

Re: Binding the Lua library [was: adding a scripting language to an ocaml program]

[Paolo Donadeo]

On Wed, Jul 7, 2010 at 15:50, Sylvain Le Gall <gildor@centi> wrote:
> So far, we have:
> http://bitbucket.org/dpowers/luacaml by David Powers
> a prototype by Paolo Donadeo
> lua-ml by Christian Lindig

I didn't know! I spent a lot of time searching for material on the
topic and I concluded to be the only one OCamler interested in Lua.


> Maybe we can join our forces to create a common binding to Lua from
> OCaml.

I completely agree.


> I propose to host it at http://forge.ocamlcore.org, we can even have an
> hg repository if needed.

Ahem... git or subversion? :-) Not an holy war here, it's only that I
never used hg.


> I would be glad that at least David, Paolo and Christian join ;-)

It's ok for me. First thing, I'll publish my GIT repository with the
prototype ASAP, even with some words and comments in Italian, so to
have something concrete to discuss for.


> but I will have to dedicate a lot of time to real life starting on Friday (or
> before -- depending the time my daughter will choose to born).

Congratulations! :-)


-- 
Paolo ⠠⠵

Re: [Caml-list] Binding the Lua library [was: adding a scripting language to an ocaml program]

[Paolo Donadeo]

> How about translating a program in lua (or lua bytecode) to OCaml
> sourcecode automatically? Would that be possible?

I see two problems with an approach like this:

1) it's complicated: consider that Lua was designed in the first place
to be extremely simple to be embedded in a C program, and also simple
to exchange data to and from the host program. The *only* problem I
found was the interaction with the OCaml GC which of course is not a
problem in C;

2) can the Lua program, statically translated into OCaml, be loaded
and reloaded at run time? I need a scripting language to give a user a
configuration/simple plugin language to be loaded and executed at
runtime, and Lua is ideal because it's trivial to sandbox it.


-- 
Paolo ⠠⠵

Re: Binding the Lua library [was: adding a scripting language to an ocaml program]

[Paolo Donadeo]

> It's ok for me. First thing, I'll publish my GIT repository with the
> prototype ASAP, even with some words and comments in Italian, so to
> have something concrete to discuss for.

Just published on GitHub: http://github.com/pdonadeo/lua_lib

If the joint project will start, I of course advocate hosting it on
forge.ocamlcore.org.


-- 
Paolo ⠠⠵