[Caml-list] debugging a JIT compiler (from Ocaml bytecode to machine code [x86,etc...])

[Caml-list] debugging a JIT compiler (from Ocaml bytecode to machine code [x86,etc...])

[Basile Starynkevitch]

Dear All,

As you might have noticed on my home page below, I coded (in C, using
the GNU lightning library) a JIT translator (or compiler) which
interprets Ocaml bytecode by translating it to machine code, using the
GNU lightning library. You'll need the latest CVS version of lightning
from http://savannah.gnu.org/projects/lightning

The intended use should be to replace ocaml's byterun/interp.c with my
jitinterp.c and recompile all the runtime. Details are given in my
homepage below. 

**this program is coded but still buggy** so don't use it *yet*
(except for helping me).

Debugging such a machine code generating program is painful. All
trivial tests (those under CVS in test/testinterp/) passes but a bug
still remain, which causes a segmentation violation (later on... - not
at the faulty JIT codepoint!).

Currently, I debugged most of it using a mixture of following
techniques (enabled only with the -DDEBUG flag).

1. the generated machine code can be disassembled

2. the JIT translator is able to write on a pipe, originally to a Ruby
script (hence the JML_RBYPRINTF name in the C code). (you need a
special startup.c to open this pipe)

3. a specific tiny debugger (using the ptrace system call) has been
coded to st breakpoint appropriately (in the generated machine code).

4. I instrumented also the bytecode interpreter to print its stack and
registers (ie bytecode program counter, stack pointer, accumulator,
...)  and manually compare it with traces from my debugger.

5. the bytecode is expected to stay fixed (this is false for C
callbacks). If it is freed, the generated code should be freed also
(which should be easy to code, since most of the stuff is there).

My problem is that all simple tests run ok, and the few tests that
crash have to run a significant amount, so the trace files are huge.

I suspect that only one or two bug remains, like e.g. a wrong return
from the GC on allocation, which corrupt the (Caml) stack ... The
problem is that I lack of simple programs to exhibit it, and that the
bug don't appear on trivial samples.

I probably won't have time to work on it in the next few weeks, but
any insight or hint is helpful. If you happen to have small test
programs which uses a small fraction of the standard library, it
should help also.

If you would be interested by a JIT ocamlrunj program (with speedup of
at most a factor of 2 w.r.t. to ocamlrun), please tell.

If as a researcher or hacker you happen to write interpreters from
scratch for a new super-duper language, consider using GNU lightning,
it is very interesting and provide good results (which considerably
easier to code with than generating machine code directly).

Regards.


-- 
Basile STARYNKEVITCH -- basile dot starynkevitch at inria dot fr
Project cristal.inria.fr - phone +33 1 3963 5197 - mobile 6 8501 2359
http://cristal.inria.fr/~starynke --- all opinions are only mine 

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

[Caml-list] Re: debugging a JIT compiler (from Ocaml bytecode to machine code [x86,etc...])

[Julian Brown]

On 2004-03-19, Basile Starynkevitch <basile.starynkevitch@inria.fr> wrote:
> Dear All,
>
> As you might have noticed on my home page below, I coded (in C, using
> the GNU lightning library) a JIT translator (or compiler) which
> interprets Ocaml bytecode by translating it to machine code, using the
> GNU lightning library. You'll need the latest CVS version of lightning
> from http://savannah.gnu.org/projects/lightning
>
> The intended use should be to replace ocaml's byterun/interp.c with my
> jitinterp.c and recompile all the runtime. Details are given in my
> homepage below. 
>
> **this program is coded but still buggy** so don't use it *yet*
> (except for helping me).
>
> Debugging such a machine code generating program is painful. All
> trivial tests (those under CVS in test/testinterp/) passes but a bug
> still remain, which causes a segmentation violation (later on... - not
> at the faulty JIT codepoint!).

Wow, sweet. That is an idea I thought of, just out of idle curiosity
really, but I don't have the time (or necessarily the ability!) to pull
it off successfully!

If you haven't already, you might like to try the same method that
Julian Seward used for Valgrind, which is just to stop after N blocks
(and revert to the regular bytecode interpreter, if that's possible). You
can then use a binary search to locate the block which is failing.

Also Valgrind itself might be useful, of course. It does work in the
presence of self-modifying code, IM(H)E.

Cheers,

Julian

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] debugging a JIT compiler (from Ocaml bytecode to machine code [x86,etc...])

[Richard Zidlicky]

On Fri, Mar 19, 2004 at 03:38:38PM +0100, Basile Starynkevitch wrote:

> If as a researcher or hacker you happen to write interpreters from
> scratch for a new super-duper language, consider using GNU lightning,
> it is very interesting and provide good results (which considerably
> easier to code with than generating machine code directly).

Did you comepare it with the code in Kaffe jit? It is a lot more
complicated than lightning but should give results much closer
to a real copmiler. Iirc it is also available for more platforms
than lightning.

Richard

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] debugging a JIT compiler (from Ocaml bytecode to machine code [x86,etc...])

[Sven Luther]

On Fri, Mar 19, 2004 at 07:51:11PM +0100, Richard Zidlicky wrote:
> On Fri, Mar 19, 2004 at 03:38:38PM +0100, Basile Starynkevitch wrote:
> 
> > If as a researcher or hacker you happen to write interpreters from
> > scratch for a new super-duper language, consider using GNU lightning,
> > it is very interesting and provide good results (which considerably
> > easier to code with than generating machine code directly).
> 
> Did you comepare it with the code in Kaffe jit? It is a lot more
> complicated than lightning but should give results much closer
> to a real copmiler. Iirc it is also available for more platforms
> than lightning.

But not on ppc.

Friendly,

Sven Luther

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] debugging a JIT compiler (from Ocaml bytecode to machine code [x86,etc...])

[Walid Taha]

Basile,

This sounds like a very interesting and useful project.

What size examples are you looking for?  If you're looking for small
programs (up to depth 5, say), you can write a small program that
enumerates all them.  After that point, there's just way too many of them.  
Do you want just lambda constructs or more ocaml constructs?

Also, with the small benchmarks that you already have, do you run them
just once or do you put them in a loop?  It could be interesting to see if
putting them in a loop also leads to the eventual seg-fault.

Finally, the MetaOCaml example directory (called "mex") has a test
programs of various sizes.

Walid.

On Fri, 19 Mar 2004, Basile Starynkevitch wrote:

|Dear All,
|
|As you might have noticed on my home page below, I coded (in C, using
|the GNU lightning library) a JIT translator (or compiler) which
|interprets Ocaml bytecode by translating it to machine code, using the
|GNU lightning library. You'll need the latest CVS version of lightning
|from http://savannah.gnu.org/projects/lightning
|
|The intended use should be to replace ocaml's byterun/interp.c with my
|jitinterp.c and recompile all the runtime. Details are given in my
|homepage below. 
|
|**this program is coded but still buggy** so don't use it *yet*
|(except for helping me).
|
|Debugging such a machine code generating program is painful. All
|trivial tests (those under CVS in test/testinterp/) passes but a bug
|still remain, which causes a segmentation violation (later on... - not
|at the faulty JIT codepoint!).
|
|Currently, I debugged most of it using a mixture of following
|techniques (enabled only with the -DDEBUG flag).
|
|1. the generated machine code can be disassembled
|
|2. the JIT translator is able to write on a pipe, originally to a Ruby
|script (hence the JML_RBYPRINTF name in the C code). (you need a
|special startup.c to open this pipe)
|
|3. a specific tiny debugger (using the ptrace system call) has been
|coded to st breakpoint appropriately (in the generated machine code).
|
|4. I instrumented also the bytecode interpreter to print its stack and
|registers (ie bytecode program counter, stack pointer, accumulator,
|..)  and manually compare it with traces from my debugger.
|
|5. the bytecode is expected to stay fixed (this is false for C
|callbacks). If it is freed, the generated code should be freed also
|(which should be easy to code, since most of the stuff is there).
|
|My problem is that all simple tests run ok, and the few tests that
|crash have to run a significant amount, so the trace files are huge.
|
|I suspect that only one or two bug remains, like e.g. a wrong return
|from the GC on allocation, which corrupt the (Caml) stack ... The
|problem is that I lack of simple programs to exhibit it, and that the
|bug don't appear on trivial samples.
|
|I probably won't have time to work on it in the next few weeks, but
|any insight or hint is helpful. If you happen to have small test
|programs which uses a small fraction of the standard library, it
|should help also.
|
|If you would be interested by a JIT ocamlrunj program (with speedup of
|at most a factor of 2 w.r.t. to ocamlrun), please tell.
|
|If as a researcher or hacker you happen to write interpreters from
|scratch for a new super-duper language, consider using GNU lightning,
|it is very interesting and provide good results (which considerably
|easier to code with than generating machine code directly).
|
|Regards.
|
|
|

-- 


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners