Native executable symtable

Native executable symtable

[Alex Baretta]

We are writing a library implementing binary client-server protocol 
based on the Marshal module. In order to guarantee some degree of 
type-safety, during the handshaking phase of the communication we need 
the client to send the server the MD5 checksums of all relevant modules. 
The server process must verify that it has all the mentioned modules and 
that the checksums are consistent with its own.

We are currently generating MD5 checksums from our source files with the 
md5sum command. These checksums are then assembled into a new source 
file which is compiled and linked into the both executables. This 
process works coherently for byte and native code, but it has at least 
one important drawback: all relevant modules must be available as source 
files for the md5 source file generation to work. Besides, this makes 
the compilation process exceedingly complex.

We have found that the md5sum can be fetced directly from the executable 
file associated to the process. This technique is documented nowhere, as 
far as I can see, but the source code of Symtable.init_toplevel is very 
informative as to how to do this for bytecode executables. What I would 
like to know is how to implement this technique for native code 
executables. Essentially, how am I supposed to parse the binary 
executable to extract the symtable information. Also, how am I supposed 
to figure out at runtime whether I must use the bytecode method or the 
native code method? Equivalently, how do I know at runtime if the 
process is running a bytecode executable or a native one?

Thank you very much.

Alex


-- 
*********************************************************************
http://www.barettadeit.com/
Baretta DE&IT
A division of Baretta SRL

tel. 02 370 111 55
fax. 02 370 111 54

Our technology:
http://www.asxcaml.org/
http://www.freerp.org/

Re: [Caml-list] Native executable symtable

[Nicolas Cannasse]

> We have found that the md5sum can be fetced directly from the executable
> file associated to the process. This technique is documented nowhere, as
> far as I can see, but the source code of Symtable.init_toplevel is very
> informative as to how to do this for bytecode executables. What I would
> like to know is how to implement this technique for native code
> executables. Essentially, how am I supposed to parse the binary
> executable to extract the symtable information. Also, how am I supposed
> to figure out at runtime whether I must use the bytecode method or the
> native code method? Equivalently, how do I know at runtime if the
> process is running a bytecode executable or a native one?
>
> Thank you very much.
>
> Alex

I would say this is very difficult since it depends on linker used to
produce executable.
For bytecode things are always very easy since the CMO/CMA are just
Marshalled data of some ocaml data structure. As always, it get complicated
when you deal with system specificity issues.

Regards,
Nicolas Cannasse

Re: [Caml-list] Native executable symtable

[Luca Pascali]

Nicolas Cannasse wrote:

> [...]
>
>I would say this is very difficult since it depends on linker used to
>produce executable.
>For bytecode things are always very easy since the CMO/CMA are just
>Marshalled data of some ocaml data structure. As always, it get complicated
>when you deal with system specificity issues.
>
>Regards,
>Nicolas Cannasse
>  
>

I can say that informations about signatures of used modules are present 
in both the bytecode and native executables.
I saw a macroscopical difference between those two results is that in 
the bytecode compiler, at the end of the file, there is a TOC section 
that gives informations about the header sections. One of this sections 
(named CRCS) is the one with the signatures.
In the native compilation, these informations are not lost (at least 
with Ocaml 3.07+2 under Linux. Correct me if I am wrong).
But how can we access those informations?

Module Sys gives a mutable value that says if we are running under a 
toplevel or not (Sys.interactive), too bad there's not something that 
says us if we are using a bytecode compiled application or not.

A method can be looking for the TOC for knowing if the application is 
bytecode or native.
In case of bytecode file, the TOC says us where the CRCS are located 
into the file. But at which offset we can found those CRCS in the native 
code compiled?

Thanks in advance
Luca

-- 
*********************************************************************
Luca Pascali
luca@barettadeit.com
asxcaml-guru@barettadeit.com

http://www.barettadeit.com/
Baretta DE&IT
A division of Baretta SRL

tel. 02 370 111 55
fax. 02 370 111 54

Our technology:
http://www.asxcaml.org/
http://www.freerp.org/

Re: [Caml-list] Native executable symtable

[Keith Wansbrough]

> I can say that informations about signatures of used modules are present 
> in both the bytecode and native executables.
> I saw a macroscopical difference between those two results is that in 
> the bytecode compiler, at the end of the file, there is a TOC section 
> that gives informations about the header sections. One of this sections 
> (named CRCS) is the one with the signatures.
> In the native compilation, these informations are not lost (at least 
> with Ocaml 3.07+2 under Linux. Correct me if I am wrong).
> But how can we access those informations?

The brute-force way is to make sure that you know one of the values in
the table (the CRC of an unchanging module, say), and then just scan
through the executable searching for that value as a bit-pattern.
Once you find it, scan back until you find the CRC header, and you're
done.  Ugly, but it will work on pretty much any platform (unless the
executable is compressed or encrypted).

--KW 8-)

Re: [Caml-list] Native executable symtable

[Luca Pascali]

Keith Wansbrough wrote:

>
>The brute-force way is to make sure that you know one of the values in
>the table (the CRC of an unchanging module, say), and then just scan
>through the executable searching for that value as a bit-pattern.
>Once you find it, scan back until you find the CRC header, and you're
>done.  Ugly, but it will work on pretty much any platform (unless the
>executable is compressed or encrypted).
>
>--KW 8-)
>
>  
>
Ok. Sounds fine.
But for the application we thought is not applicable.

We need those CRCs to for making a signature exchange for a 
client-server communication via socket.
We chosen to use marshalling (instead of pretty printing and parsing) to 
vehicle data on the communication channel, and as you know marshalling 
is not type-safe.

To enforce this type-safety, we thought to exchange those signatures to 
know if structures are taken from the same library or not (the same job 
Dynlink does when you ask to load a file).

So we do not know those values, and we cannot search for them in the file.
It will be more useful for us to know where to find those values in the 
natively compiled file and if this solution is cross platform or not.

LP


-- 
*********************************************************************
Luca Pascali
luca@barettadeit.com
asxcaml-guru@barettadeit.com

http://www.barettadeit.com/
Baretta DE&IT
A division of Baretta SRL

tel. 02 370 111 55
fax. 02 370 111 54

Our technology:
http://www.asxcaml.org/
http://www.freerp.org/

Re: [Caml-list] Native executable symtable

[Keith Wansbrough]

Luca Pascali wrote:

> Keith Wansbrough wrote:
[..]
> >The brute-force way is to make sure that you know one of the values in
> >the table (the CRC of an unchanging module, say), and then just scan
> >through the executable searching for that value as a bit-pattern.
[..]
> Ok. Sounds fine.
> But for the application we thought is not applicable.
[..]
> So we do not know those values, and we cannot search for them in the file.

Make sure there is a value you do know.  For example, write a module 
containing just a big string.  Require that it is always included in 
any executable supported by your system.  Compute its CRC.  Now you 
have a starting-point for searching.

> It will be more useful for us to know where to find those values in the 
> natively compiled file and if this solution is cross platform or not.

No solution will be completely cross-platform; different platforms
have different executable formats.  But if you support say ELF and PE
(COFF), you'll cover a lot of platforms...

--KW 8-)

Re: [Caml-list] Native executable symtable

[Luca Pascali]

Keith Wansbrough wrote:

> [...]
>
>Make sure there is a value you do know.  For example, write a module 
>containing just a big string.  Require that it is always included in 
>any executable supported by your system.  Compute its CRC.  Now you 
>have a starting-point for searching.
>
>  
>
We can try this way.

>>It will be more useful for us to know where to find those values in the 
>>natively compiled file and if this solution is cross platform or not.
>>    
>>
>
>No solution will be completely cross-platform; different platforms
>have different executable formats.  But if you support say ELF and PE
>(COFF), you'll cover a lot of platforms...
>
>--KW 8-)
>
>
>  
>
I knew that.
Since some weeks ago I heard about some Dynlinker that works also from 
natively compiled applications, I thought that someone already hit 
his/her head onto this wall.
I think (and I hope) that the format of the signatures is always the 
same for all the operating systems.

Anyway thanks a lot for all hints.

LP




-- 
*********************************************************************
Luca Pascali
luca@barettadeit.com
asxcaml-guru@barettadeit.com

http://www.barettadeit.com/
Baretta DE&IT
A division of Baretta SRL

tel. 02 370 111 55
fax. 02 370 111 54

Our technology:
http://www.asxcaml.org/
http://www.freerp.org/

Re: [Caml-list] Native executable symtable

[Richard Jones]

[-- Attachment #1: Type: text/plain, Size: 508 bytes --]

On Thu, Nov 11, 2004 at 01:42:24PM +0100, Luca Pascali wrote:
> Since some weeks ago I heard about some Dynlinker that works also from 
> natively compiled applications,

Do you have a link for this?

Rich.

-- 
Richard Jones.  http://www.annexia.org/  http://www.j-london.com/
>>>   http://www.team-notepad.com/ - collaboration tools for teams   <<<
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
Use Perl libs in OCaml - http://www.merjis.com/developers/perl4caml

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Caml-list] Native executable symtable

[Luca Pascali]

Richard Jones wrote:

>On Thu, Nov 11, 2004 at 01:42:24PM +0100, Luca Pascali wrote:
>  
>
>>Since some weeks ago I heard about some Dynlinker that works also from 
>>natively compiled applications,
>>    
>>
>
>Do you have a link for this?
>
>Rich.
>
>  
>
Sorry, I haven't.
I hoped that someone would send me that link too, because looking in the 
past messages I was not able to find it.

Luca


-- 
*********************************************************************
Luca Pascali
luca@barettadeit.com
asxcaml-guru@barettadeit.com

http://www.barettadeit.com/
Baretta DE&IT
A division of Baretta SRL

tel. 02 370 111 55
fax. 02 370 111 54

Our technology:
http://www.asxcaml.org/
http://www.freerp.org/

Re: [Caml-list] Native executable symtable

[malc]

On Sat, 20 Nov 2004, Luca Pascali wrote:

> Richard Jones wrote:
>
>> On Thu, Nov 11, 2004 at 01:42:24PM +0100, Luca Pascali wrote:
>> 
>>> Since some weeks ago I heard about some Dynlinker that works also from 
>>> natively compiled applications,
>>> 
>> 
>> Do you have a link for this?
>> 
>> Rich.
>> 
>> 
> Sorry, I haven't.
> I hoped that someone would send me that link too, because looking in the past 
> messages I was not able to find it.

http://www.boblycat.org/~malc/scaml/
Version of the patch on the page is for 3.07 only (port from 3.06 was done
Vitaly Lugovsky, and in process one flaw was introduced).

-- 
mailto:malc@pulsesoft.com

Re: [Caml-list] Native executable symtable

[Alex Baretta]

malc wrote:

> 
> http://www.boblycat.org/~malc/scaml/
> Version of the patch on the page is for 3.07 only (port from 3.06 was done
> Vitaly Lugovsky, and in process one flaw was introduced).
> 

What does this patch do? I clicked on the link but could not make any 
sense out the page you referenced.

Anyhow, I suppose Luca referred to Asmdynlink, by Fabrice Le Fessant, 
which was distributed with the CDK and probably died together with it. 
An Ocaml-bytecode Ocaml-interpreter was a cool idea, but I suppose 
Xavier et al. are working on a cleaner 100% pure native code solution to 
dynlinking cmx and cmxa files.

Alex


-- 
*********************************************************************
http://www.barettadeit.com/
Baretta DE&IT
A division of Baretta SRL

tel. +39 02 370 111 55
fax. +39 02 370 111 54

Our technology:

The Application System/Xcaml (AS/Xcaml)
<http://www.asxcaml.org/>

The FreerP Project
<http://www.freerp.org/>

Re: [Caml-list] Native executable symtable

[malc]

On Sat, 20 Nov 2004, Alex Baretta wrote:

> malc wrote:
>
>> 
>> http://www.boblycat.org/~malc/scaml/
>> Version of the patch on the page is for 3.07 only (port from 3.06 was done
>> Vitaly Lugovsky, and in process one flaw was introduced).
>> 
>
> What does this patch do? I clicked on the link but could not make any sense 
> out the page you referenced.
>

I don't know how to put it more bluntly than that (already on the page):

<quote>
What is it?
A patch, against OCaml 3.07, making it possible to: create, link against 
and dynamically load shared objects on i386 ELF and Win32 platforms.
</quote>

> Anyhow, I suppose Luca referred to Asmdynlink, by Fabrice Le Fessant, which 
> was distributed with the CDK and probably died together with it. An 
> Ocaml-bytecode Ocaml-interpreter was a cool idea, but I suppose Xavier et al. 
> are working on a cleaner 100% pure native code solution to dynlinking cmx and 
> cmxa files.

Excuse me for being a bit sarcastic, but - hah.

-- 
mailto:malc@pulsesoft.com

Re: [Caml-list] Native executable symtable

[Alex Baretta]

malc wrote:
> On Sat, 20 Nov 2004, Alex Baretta wrote:
> 
>> malc wrote:
>>
>>>
>>> http://www.boblycat.org/~malc/scaml/
>>> Version of the patch on the page is for 3.07 only (port from 3.06 was 
>>> done
>>> Vitaly Lugovsky, and in process one flaw was introduced).
>>>
>>
>> What does this patch do? I clicked on the link but could not make any 
>> sense out the page you referenced.
>>
> 
> I don't know how to put it more bluntly than that (already on the page):
> 
> <quote>
> What is it?
> A patch, against OCaml 3.07, making it possible to: create, link against 
> and dynamically load shared objects on i386 ELF and Win32 platforms.
> </quote>

I read this, but I don't see how this differs from the behavior of 
vanilla Ocaml. I commonly use shared-object libraries in my code, and 
the Ocaml compiler can generate DLLs out of libraries for use within C 
code. The problem addressed by ASMDynlink dynlinking Ocaml compiled code 
within a running native code application. What has Vitaly's patch to do 
with this?

I also don't understand your sarcasm.

Alex

-- 
*********************************************************************
http://www.barettadeit.com/
Baretta DE&IT
A division of Baretta SRL

tel. +39 02 370 111 55
fax. +39 02 370 111 54

Our technology:

The Application System/Xcaml (AS/Xcaml)
<http://www.asxcaml.org/>

The FreerP Project
<http://www.freerp.org/>

Re: [Caml-list] Native executable symtable

[malc]

On Sat, 20 Nov 2004, Alex Baretta wrote:

>>> What does this patch do? I clicked on the link but could not make any 
>>> sense out the page you referenced.
>>> 
>> 
>> I don't know how to put it more bluntly than that (already on the page):
>> 
>> <quote>
>> What is it?
>> A patch, against OCaml 3.07, making it possible to: create, link against 
>> and dynamically load shared objects on i386 ELF and Win32 platforms.
>> </quote>
>
> I read this, but I don't see how this differs from the behavior of vanilla 
> Ocaml. I commonly use shared-object libraries in my code, and the Ocaml 
> compiler can generate DLLs out of libraries for use within C code. The 
> problem addressed by ASMDynlink dynlinking Ocaml compiled code within a 
> running native code application. What has Vitaly's patch to do with this?

Keyword here is "for use within C code", this patch is "for use within
OCaml code", also adressing name mangling, shared GC and runtime. 
Additionally it Natdynlink module is introduced (Natdynlnk mimics the
behaviour of Dynlink, but works with natively compiled OCaml shared
object). A little bit of research on the page would have (like, for
instance, getting the examples provided or reading README) would have
given you just as much information.

>
> I also don't understand your sarcasm.

Sarcasm spurs from the fact that first version of this patch was
introduced in 2001 and developed till 2003. I had hoped that this will
result in some reaction, if not from Inria developers then from OCaml
users, alas this was not the case. Nothing w.r.t. native shared object
and/or dynamic linking of native code has changed since then. I myself
asked and waited for something akin to this for a bit, then decided to
went ahead and did it on my own, big mistake, all signs are there that
people do not need this functionality bad enough to do anything.
So be it.

-- 
mailto:malc@pulsesoft.com

Re: [Caml-list] Native executable symtable

[Ritesh Kumar]

Actually malc, it is a bit dissapointing (not on you but on the general 
Ocaml community) that you said that. Before I converted myself into an 
Ocaml programmer I quite vigorously fought with google to find out the 
answer to the very same question your patch addresses. I eventually 
came across some old newsgroup/mail-list postings which said why shared 
libraries are good/bad ideas in Ocaml and what are the pros and cons. I 
know I am not an expert in this field but id does seem that having 
ocaml shared libraries is a big step towards wider deployment ( and 
what's more, perhaps an entire userspace of its own!). I didn't find 
the reasons given against shared libraries very convincing. Among other 
things I value dynamic module linking (during program runtime) as it is 
with languages like Java/C a lot as they give numerous opportunities to 
the programmer.
The patch that you have developed is a good step forward. I think it 
has a lot of utility and it should be part of the main opt compiler.

Ritesh

--
Impossibility is a relative concept.

On Nov 20, 2004, at 2:17 PM, malc wrote:

> On Sat, 20 Nov 2004, Alex Baretta wrote:
>
>>>> What does this patch do? I clicked on the link but could not make 
>>>> any sense out the page you referenced.
>>> I don't know how to put it more bluntly than that (already on the 
>>> page):
>>> <quote>
>>> What is it?
>>> A patch, against OCaml 3.07, making it possible to: create, link 
>>> against and dynamically load shared objects on i386 ELF and Win32 
>>> platforms.
>>> </quote>
>>
>> I read this, but I don't see how this differs from the behavior of 
>> vanilla Ocaml. I commonly use shared-object libraries in my code, and 
>> the Ocaml compiler can generate DLLs out of libraries for use within 
>> C code. The problem addressed by ASMDynlink dynlinking Ocaml compiled 
>> code within a running native code application. What has Vitaly's 
>> patch to do with this?
>
> Keyword here is "for use within C code", this patch is "for use within
> OCaml code", also adressing name mangling, shared GC and runtime. 
> Additionally it Natdynlink module is introduced (Natdynlnk mimics the
> behaviour of Dynlink, but works with natively compiled OCaml shared
> object). A little bit of research on the page would have (like, for
> instance, getting the examples provided or reading README) would have
> given you just as much information.
>
>>
>> I also don't understand your sarcasm.
>
> Sarcasm spurs from the fact that first version of this patch was
> introduced in 2001 and developed till 2003. I had hoped that this will
> result in some reaction, if not from Inria developers then from OCaml
> users, alas this was not the case. Nothing w.r.t. native shared object
> and/or dynamic linking of native code has changed since then. I myself
> asked and waited for something akin to this for a bit, then decided to
> went ahead and did it on my own, big mistake, all signs are there that
> people do not need this functionality bad enough to do anything.
> So be it.
>
> -- 
> mailto:malc@pulsesoft.com
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

The madness of ignoring people

[Vincenzo Ciancia]

On Saturday 20 November 2004 20:17, malc wrote:
> I had hoped that this will
> result in some reaction, if not from Inria developers then from OCaml
> users, alas this was not the case.

I don't understand why academic environments producing free software 
often tend to deliberately ignore people contributions. Maybe it is 
normal that your feature has no users (yet), and so there is no people 
interested, but it's absolutely NOT normal that people at inria just 
ignore it - just like it's not normal that they sometimes don't answer 
at all nontrivial technical questions about ocaml that _only them_ can 
answer. 

I know, compilers and mailing lists are for free, but there is not a 
wide support network for ocaml, as there is, for example, for many 
software projects such as gcc, or the GNU operating system and so on, 
so the only place where somebody can ask for help about ocaml is here.

And to get back on your issue about the patch, I see it's frustrating to 
be just ignored, and it's a pity to force people, who might have worked 
hard, and for free, in _their_ free time, to stop contributing, because 
they get frustrated by being ignored, without at least explaining the 
reason. I've seen this on other mailing lists in the past, and in case 
someone is going to ask, no, it's not about contributions I made (there 
is no such thing :) ) - but of course I asked unanswered non-trivial 
technical question which I think is the other side of the same coin 
(wondering if this phrase makes sense in english, translated in italian 
it does).

V.

Re: [Caml-list] The madness of ignoring people

[malc]

On Sat, 20 Nov 2004, Vincenzo Ciancia wrote:

> On Saturday 20 November 2004 20:17, malc wrote:
>> I had hoped that this will
>> result in some reaction, if not from Inria developers then from OCaml
>> users, alas this was not the case.
>
> I don't understand why academic environments producing free software
> often tend to deliberately ignore people contributions. Maybe it is
> normal that your feature has no users (yet), and so there is no people
> interested, but it's absolutely NOT normal that people at inria just
> ignore it - just like it's not normal that they sometimes don't answer
> at all nontrivial technical questions about ocaml that _only them_ can
> answer.

Maybe i wasn't clear enough. While working on the patch i (obviously)
asked questions (and was given quite comprehensive answers) from Xavier
Leroy for instance. I also learned his position on the subject. I think
there were maybe 5 people who contacted me personally, 2 of them had some
feature request, one actually reported bugs (thank you Vitaly)

There was/is one/two groups of people really interested in functionality
provided by the patch, one of them offered help, when it became clear i
can not provide enough resources to bring the patch uptodate w.r.t 3.0.8

But in general, most of the people fall into 'wouldn't it be cool to have
native dynamic loading/linking' category. And - yeah it probably would
have...

>
> I know, compilers and mailing lists are for free, but there is not a
> wide support network for ocaml, as there is, for example, for many
> software projects such as gcc, or the GNU operating system and so on,
> so the only place where somebody can ask for help about ocaml is here.
>
> And to get back on your issue about the patch, I see it's frustrating to
> be just ignored, and it's a pity to force people, who might have worked
> hard, and for free, in _their_ free time, to stop contributing, because
> they get frustrated by being ignored, without at least explaining the
> reason. I've seen this on other mailing lists in the past, and in case
> someone is going to ask, no, it's not about contributions I made (there
> is no such thing :) ) - but of course I asked unanswered non-trivial
> technical question which I think is the other side of the same coin
> (wondering if this phrase makes sense in english, translated in italian
> it does).

I wasn't ignored by Inria, i expressed my opinion, asked theirs, got it.
What's a little bit disappointing that nobody (after all those years)
presented any alternative.

Sincerely,
malc

-- 
mailto:malc@pulsesoft.com

Re: [Caml-list] The madness of ignoring people

[Vincenzo Ciancia]

On Sunday 21 November 2004 00:10, malc wrote:
> I wasn't ignored by Inria, i expressed my opinion, asked theirs, got
> it. What's a little bit disappointing that nobody (after all those
> years) presented any alternative.

I misunderstood your "I had hoped that this will result in some 
reaction" :) Ok, my official apologies go to all people I might have 
offended - it was not my intention to troll.

Vincenzo

Re: [Caml-list] The madness of ignoring people

[skaller]

On Sun, 2004-11-21 at 09:43, Vincenzo Ciancia wrote:

> I don't understand why academic environments producing free software 
> often tend to deliberately ignore people contributions. 

(a) their prime interest is theoretical and
(b) they have limited resources

even so, didn't they

(c) integrate O'Labl and camlp4?

They're not ignoring contributions, they're being
selective as they must be.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net

Re: [Caml-list] The madness of ignoring people

[Vincenzo Ciancia]

On Sunday 21 November 2004 13:51, skaller wrote:
> They're not ignoring contributions, they're being
> selective as they must be.

I see that I was wrong, I misunderstood malc's post. I can't bear when I 
see people which are actually "ignored" in the sense that they post 
interesting work and they don't get any reply, but this is not the case 
- obviously nobody is required to integrate any possible patch in the 
mainstream compiler (even if, to speak about this precise case, dynamic 
loading of C code is a very interesting feature).

V.

Re: [Caml-list] The madness of ignoring people

[malc]

On Sun, 21 Nov 2004, Vincenzo Ciancia wrote:

> On Sunday 21 November 2004 13:51, skaller wrote:
>> They're not ignoring contributions, they're being
>> selective as they must be.
>
> I see that I was wrong, I misunderstood malc's post. I can't bear when I
> see people which are actually "ignored" in the sense that they post
> interesting work and they don't get any reply, but this is not the case
> - obviously nobody is required to integrate any possible patch in the
> mainstream compiler (even if, to speak about this precise case, dynamic
> loading of C code is a very interesting feature).

Sorry, but you are wrong again, this particular case is about dynamic
loading of OCaml code, for C code there isn't and never was any problem.

-- 
mailto:malc@pulsesoft.com

Re: [Caml-list] Native executable symtable

[skaller]

On Sun, 2004-11-21 at 06:17, malc wrote:
> all signs are there that
> people do not need this functionality bad enough to do anything.

I certainly desire this and have requested it.
But I can't see more I can do, nor can I use
any patches in redistributable code.

In fact I have done what I can, using Ocaml to build
a compiler for a whole new language, primarily
to make it possible to build an extensible high 
performance system. Had Ocaml been able to do this
I would have concentrated on a C library wrapper 
generator for Ocaml instead.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net

Re: [Caml-list] Native executable symtable

[Richard Jones]

[-- Attachment #1: Type: text/plain, Size: 1261 bytes --]

On Sat, Nov 20, 2004 at 10:17:30PM +0300, malc wrote:
> Sarcasm spurs from the fact that first version of this patch was
> introduced in 2001 and developed till 2003. I had hoped that this will
> result in some reaction, if not from Inria developers then from OCaml
> users, alas this was not the case. Nothing w.r.t. native shared object
> and/or dynamic linking of native code has changed since then. I myself
> asked and waited for something akin to this for a bit, then decided to
> went ahead and did it on my own, big mistake, all signs are there that
> people do not need this functionality bad enough to do anything.
> So be it.

It'd be very useful for mod_caml - mod_caml uses Dynlink to load the
"scripts" and handlers, and hence is limited to bytecode.  Native code
dynamic linking would come in useful.  I'd rather it was part of core
OCaml, or available as a separate library which didn't require OCaml
itself to be recompiled.

Rich.

-- 
Richard Jones.  http://www.annexia.org/  http://www.j-london.com/
>>>   http://www.team-notepad.com/ - collaboration tools for teams   <<<
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
Use Perl libs in OCaml - http://www.merjis.com/developers/perl4caml

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Caml-list] Native executable symtable

[skaller]

On Mon, 2004-11-22 at 02:59, Richard Jones wrote:

> It'd be very useful for mod_caml - mod_caml uses Dynlink to load the
> "scripts" and handlers, and hence is limited to bytecode.  Native code
> dynamic linking would come in useful.  I'd rather it was part of core
> OCaml, or available as a separate library which didn't require OCaml
> itself to be recompiled.

You'll never convince a third party web site supplier to install 
a hacked version of Ocaml. It took ages to get Python server 
side available.

However consider also a second problem with mod_caml: unloading.
That's also essential for a long running process, and can't be done,
even in the bytecode interpreter. There were *also* patches
made for that (mods to the garbage collector to allow code
to be collected).

Finally, compare with Python where a superior implementation
called 'Stackless Python' which used continuation passing
was available for years. It inspired addition of a collector
and generators to be added to CPython, but because it would
require rewriting all the C extensions to obtain stackless
operations, the mods to the core interpreter never made it
into the main distribution.

My point is that if *you* were on the Ocaml team you'd
likely have a different perspective -- if you were going
to add a feature you'd want to be fairly sure the idea
was sound, complete, compatible, worthwhile -- and
engineerable.

I recently asked Hasn Boehm about unloading shared libraries
in C and C++ code and how that would operate with a
garbage collector.. he didn't see much hope of it ever working.
The fact is the changes needed to ISO C/C++ to even make
a *conservative* collector safe are non-trivial.

Felix actually *does* collect shared libraries, but it isn't
safe unless you limit data exchange very carefully to
ensure there can't be any dangling pointers, since the 
collector can't know about C pointers -- to C strings,
C functions, C++ vtables, and perhaps RTTI and exception
handling stuff..

So all of this seems quite difficult .. something
about the American saying "It works in practice" and
the Frenchman saying "Ah yes, but does it work in theory?"


-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net

Re: [Caml-list] Native executable symtable

[malc]

On Sun, 22 Nov 2004, skaller wrote:

> On Mon, 2004-11-22 at 02:59, Richard Jones wrote:
>
>> It'd be very useful for mod_caml - mod_caml uses Dynlink to load the
>> "scripts" and handlers, and hence is limited to bytecode.  Native code
>> dynamic linking would come in useful.  I'd rather it was part of core
>> OCaml, or available as a separate library which didn't require OCaml
>> itself to be recompiled.
>
> You'll never convince a third party web site supplier to install
> a hacked version of Ocaml. It took ages to get Python server
> side available.
>
> However consider also a second problem with mod_caml: unloading.
> That's also essential for a long running process, and can't be done,
> even in the bytecode interpreter. There were *also* patches
> made for that (mods to the garbage collector to allow code
> to be collected).

Excuse me, but, next time, check your facts before posting. Dyanmically
linked modules in shared patch can be unloaded, not being Damien Doligez
i wouldn't go as far as to claim that the unloading process always works
correctly w.r.t GC, but the functionality is certainly there.

Plus, this is just a speculation though, if there are problems with the
way it is done now, i do not feel they are insurmountable.

-- 
mailto:malc@pulsesoft.com

Re: [Caml-list] Native executable symtable

[Richard W.M. Jones]

[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]

On Mon, Nov 22, 2004 at 07:29:51AM +1100, skaller wrote:
> However consider also a second problem with mod_caml: unloading.
> That's also essential for a long running process, and can't be done,
> even in the bytecode interpreter. There were *also* patches
> made for that (mods to the garbage collector to allow code
> to be collected).

This isn't actually much of a problem with mod_caml.  Deployments of
Apache should limit the number of HTTP requests that a single Apache
child can handle (MaxRequestsPerChild), after which the child exits.
This parameter is commonly set low with mod_perl also, since Perl has
several "issues" with running over long periods of time - it was
designed for short command line scripts after all.

Rich.

-- 
Richard Jones.  http://www.annexia.org/  http://www.j-london.com/
>>>   http://www.team-notepad.com/ - collaboration tools for teams   <<<
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
http://winwinsales.co.uk/ - CRM consultancy

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Caml-list] Native executable symtable

[skaller]

On Mon, 2004-11-22 at 10:30, Richard W.M. Jones wrote:
> On Mon, Nov 22, 2004 at 07:29:51AM +1100, skaller wrote:
> > However consider also a second problem with mod_caml: unloading.
> > That's also essential for a long running process, and can't be done,
> > even in the bytecode interpreter. There were *also* patches
> > made for that (mods to the garbage collector to allow code
> > to be collected).
> 
> This isn't actually much of a problem with mod_caml.  Deployments of
> Apache should limit the number of HTTP requests that a single Apache
> child can handle (MaxRequestsPerChild), after which the child exits.
> This parameter is commonly set low with mod_perl also, since Perl has
> several "issues" with running over long periods of time - it was
> designed for short command line scripts after all.

Ah, I see. So Apache is hacked to support broken module
implementations.. :)

There are other applicatins where this isn't possible,
for example Felix was originally designed for a telco
environment running say 20 CPU's, one thread per CPU,
but millions of 'micro' threads (one per phone call).

Restarting one thread in such an environment is possible but messy:
you either need to make the microthreads mobile so you can
move them from one CPU to another, or you have to be
willing to wait 20 minutes whilst some person who likes
long winded speeches (like me) finishes their call.
(Or just disconnect them .. :)

But basically you want to be able to run forever,
even upgrading the call handling logic whilst the
process is running. Total code use here isn't
the issue so much as dynamic upgrading -- the main
reason for unloading isn't to save memory, so much
as to be sure no one is using an old version
which happened to credit your account instead of
debiting it due to a bug .. :)

Anyhow to some extent I sympathise with the Ocaml team,
in that just saying 'dynamic loading of native code'
isn't a full answer. It promises something some programmer
not fully understanding it may try to use only to find
later that it isn't suitable because there are some hard
to understand compromises (eg insecure unloading, different
behaviour on different platforms, etc ..)

BTW: is mod_caml bytecode that much of a 
performance issue for web service, compared
with Python,Php, Perl etc?

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net

Re: [Caml-list] Native executable symtable

[Jean-Marie Gaillourdet]

Hi,

Am Sonntag, 21. November 2004 16:59 schrieb Richard Jones:
> It'd be very useful for mod_caml - mod_caml uses Dynlink to load the
> "scripts" and handlers, and hence is limited to bytecode.  Native code
> dynamic linking would come in useful.  I'd rather it was part of core
> OCaml, or available as a separate library which didn't require OCaml
> itself to be recompiled.

Actually, my impression of the ocaml development is, ocaml is going into the 
same direction as e.g. Java. They are developing a jit compiler to improve 
the performance of the interpreted environment. This is the direction of the 
future as it allows to adapt the native code at runtime, based upon live 
profiling results. We aren't there yet, but this is the direction industry 
and academic world are heading for. Just look at Microsoft's .NET and Sun's 
Java environemt and the huge number of academic papers talking about 
interesting issues with those environemnts. Therefore it might be the right 
time to stop about whining and lamenting the missing native ocaml shared 
library support and to start accepting byte code runtimes as appropriate even 
for performace critical applications. 

Regards,
  Jean-Marie Gaillourdet 

Re: [Caml-list] Native executable symtable

[Ritesh Kumar]

On Nov 21, 2004, at 3:42 PM, Jean-Marie Gaillourdet wrote:

> Hi,
>
> Am Sonntag, 21. November 2004 16:59 schrieb Richard Jones:
>> It'd be very useful for mod_caml - mod_caml uses Dynlink to load the
>> "scripts" and handlers, and hence is limited to bytecode.  Native code
>> dynamic linking would come in useful.  I'd rather it was part of core
>> OCaml, or available as a separate library which didn't require OCaml
>> itself to be recompiled.
>
> Actually, my impression of the ocaml development is, ocaml is going 
> into the
> same direction as e.g. Java. They are developing a jit compiler to 
> improve
> the performance of the interpreted environment. This is the direction 
> of the
> future as it allows to adapt the native code at runtime, based upon 
> live
> profiling results. We aren't there yet, but this is the direction 
> industry
> and academic world are heading for. Just look at Microsoft's .NET and 
> Sun's
> Java environemt and the huge number of academic papers talking about
> interesting issues with those environemnts. Therefore it might be the 
> right
> time to stop about whining and lamenting the missing native ocaml 
> shared
> library support and to start accepting byte code runtimes as 
> appropriate even
> for performace critical applications.
>
> Regards,
>   Jean-Marie Gaillourdet
>


Well, I just know that currently no bytecoded language (Java/.NET/OcaML 
bytecode)  could give me the 7 to 8 times performance boost in run time 
taken by some of the network topology analysis tools I made for my 
research work. Java was especially bad at those algorithms (may be 
because of its huge memory consumption). There has to be a reason why 
C/native OcaML still take all the performance jewels. You are right 
about the fact that dynamic run-time optimizations might be interesting 
from a performance perspective esp. looking at a functional value 
oriented language like ML. However, it is still a "research" problem 
and has research interest. Native shared libraries has importance in 
making OcaML an otherwise important tool for general day to day 
development. Something like C/C++/Java/.NET. This if you see carefully 
is not a "research" problem.


> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs


--
The human being is a bag of chemicals... nothing else.

Re: [Caml-list] Native executable symtable

[Richard Jones]

[-- Attachment #1: Type: text/plain, Size: 1031 bytes --]

On Sun, Nov 21, 2004 at 01:31:34PM -0500, Ritesh Kumar wrote:
> Well, I just know that currently no bytecoded language (Java/.NET/OcaML 
> bytecode)  could give me the 7 to 8 times performance boost in run time 
> taken by some of the network topology analysis tools I made for my 
> research work. [...] You are right 
> about the fact that dynamic run-time optimizations might be interesting 
> from a performance perspective esp. looking at a functional value 
> oriented language like ML. However, it is still a "research" problem 
> and has research interest.

I'd like to say in addition, the HP Dynamo project, where a lot of
this hope for dynamic recompilation came from, recompiled native
executables.

Rich.

-- 
Richard Jones.  http://www.annexia.org/  http://www.j-london.com/
>>>   http://www.team-notepad.com/ - collaboration tools for teams   <<<
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
http://subjectlink.com/ - Lesson plans and source material for teachers

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Caml-list] Native executable symtable

[skaller]

On Mon, 2004-11-22 at 07:42, Jean-Marie Gaillourdet wrote:

> Actually, my impression of the ocaml development is, ocaml is going into the 
> same direction as e.g. Java. 

I sure hope not -- off to Haskell fast if that ever happens :)

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net

Re: [Caml-list] Native executable symtable

[David Brown]

On Thu, Nov 11, 2004 at 11:18:06AM +0100, Alex Baretta wrote:

> We are writing a library implementing binary client-server protocol 
> based on the Marshal module. In order to guarantee some degree of 
> type-safety, during the handshaking phase of the communication we need 
> the client to send the server the MD5 checksums of all relevant modules. 
> The server process must verify that it has all the mentioned modules and 
> that the checksums are consistent with its own.

As long as all of the relevant data structures are exported from some
module, the extract_crc utility (which gets install in the library
directory), can generate all of the CRC.  It needs to be run at compile
time, but can reasonably easily be put into makefile rules.  It just
becomes one additional module.

Getting the order right is a bit tricky, since it becomes something that
depends on many interfaces, some of which have implementations that depend
on the crc module itself.  So, many of the auto-dependency and order
generators won't work.

This method should be any different between native and byte-code, since the
CRC is generated based on the .cmi file which will always be the same.  The
only dependence is that you export the relevant data structures.

Dave

Re: [Caml-list] Native executable symtable

[Xavier Leroy]

> We are writing a library implementing binary client-server protocol 
> based on the Marshal module. In order to guarantee some degree of 
> type-safety, during the handshaking phase of the communication we need 
> the client to send the server the MD5 checksums of all relevant modules. 
> [...]

> We have found that the md5sum can be fetced directly from the executable 
> file associated to the process. This technique is documented nowhere, as 
> far as I can see, but the source code of Symtable.init_toplevel is very 
> informative as to how to do this for bytecode executables. What I would 
> like to know is how to implement this technique for native code 
> executables. Essentially, how am I supposed to parse the binary 
> executable to extract the symtable information.

You cannot, because the checksums you mention (the digests of the
interfaces of the modules) are not included in ocamlopt-generated
executables.  They are included only in ocamlc-generated executables (in
the CRCS section) for use with Dynlink and the toplevel.

At any rate, I think you're on the wrong tracks: the checksums you'll
find in the CRCS section of bytecode executable are those of module
interfaces, not of module implementations.  To establish type
agreement between two processes communicating via
output_value/input_value, you really want the latter, not the former.
(Think of an abstract type implemented differently in the two
processes.  For more details, see e.g. the ICFP'03 paper by Leifer,
Peskine, Sewell and Wansbrough.)

So, you're looking for convenient ways to collect checksums for module
implementations.  An insight that might simplify your build process is
that while these checksums must be computed at compile-time (e.g. by
running md5sum on the source .ml files), they can be collected
together at run-time.  For instance, you could preprocess the .ml
sources of interest so as to insert at the beginning

        let _ = Registry.record_module "Modulename" "checksum"

where "Modulename" is the module name and "checksum" the outcome of
md5sum on the source file.  The Registry.record_module function just
accumulates its arguments in a hashtable or association list, which
can then be consulted during the agreement phase of your protocol.

There are probably many other ways to do it.  But I think your initial
idea (compute checksums of source files at compile-time) is the
correct one, it's just a question of implementing it in a way that
doesn't complicate your build process too much.

- Xavier Leroy

Re: [Caml-list] Native executable symtable

[Alex Baretta]

[I'm sorry for the duplicate: I pressed the <Send> button by mistake.]

Xavier Leroy wrote:

The insight about modules implementing abstract data types differently
is very interesting. I had not thought about this. So, yes, what we
really want is a *hard* checksum, taking into account the 
implementation, too.

>         let _ = Registry.record_module "Modulename" "checksum"

We have considered the Checksum_registry module approach, but it hardly 
scales if you consider that a large portion of any project is composed 
of third-party libraries. What if the user of my network protocol 
intends to marshal PXP trees? I somehow need to fetch the checksums of 
the PXP modules defining the the relevant type. This is the reason for 
looking for a solution which integrates with the virtual machine (* i.e. 
the abstract machine executing the Ocaml program *). However, you are 
telling me that on this point, the bytecode VM and the native code VM 
implement two different abstractions.

Apparently, what we'll need to do is to better integrate the present md5 
digest generation phase with the build process, until our caml breeders 
will give us type safe marshalling.

Alex

-- 
*********************************************************************
http://www.barettadeit.com/
Baretta DE&IT
A division of Baretta SRL

tel. 02 370 111 55
fax. 02 370 111 54

Our technology:
http://www.asxcaml.org/
http://www.freerp.org/