Ocaml implementation and low level details

Ocaml implementation and low level details

[Konstantin Tcholokachvili]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

Hello,

I am writing an operating system kernel and I am considering the idea of
rewritting it in Ocaml or make a wrapper.
As I don't  know how Ocaml is implemented in detail so I have the following
question:
If want to code in Ocaml without using the garbage collector, will I be able
to use my own multithreading implementation or will I be limited by Ocaml's
global lock?

Thanks in advance.

Konstantin Tcholokachvili

[-- Attachment #2: Type: text/html, Size: 461 bytes --]

Re: [Caml-list] Ocaml implementation and low level details

[Samuel Thibault]

Konstantin Tcholokachvili, le Thu 28 Jan 2010 13:42:15 +0100, a écrit :
> I am writing an operating system kernel and I am considering the idea of
> rewritting it in Ocaml or make a wrapper.

You may want to have a look at the funk gna project. Although dormant at
the moment, it does boot, has caml threads etc.

http://gna.org/projects/funk/

Samuel

[Caml-list] Ocaml implementation and low level details

[Konstantin Tcholokachvili]

[-- Attachment #1: Type: text/plain, Size: 1033 bytes --]

I browsed the sources of funk OS one year ago, if I remember corectly you
are one og its author.
I assume that if funk manages memory and run threads it's possible to code
an OS from ground up using Ocaml but want to be sure:

- Does I need to disable an option to avoid the garbage collector's use? (I
gues that yes)

- Also need I disable Ocaml theading subsystem? (Obviously yes, but are
there some limitations?)

Are there other important considerations to take?
Do you think that Ocaml is suitable for OS coding (I''m using C now).

Thank you,

Konstantin Tcholokachvili


2010/1/28 Samuel Thibault <samuel.thibault@inria.fr>

Konstantin Tcholokachvili, le Thu 28 Jan 2010 13:42:15 +0100, a écrit :
> > I am writing an operating system kernel and I am considering the idea of
> > rewritting it in Ocaml or make a wrapper.
>
> You may want to have a look at the funk gna project. Although dormant at
> the moment, it does boot, has caml threads etc.
>
> http://gna.org/projects/funk/
>
> Samuel
>

[-- Attachment #2: Type: text/html, Size: 1570 bytes --]

Re: [Caml-list] Ocaml implementation and low level details

[Samuel Thibault]

Konstantin Tcholokachvili, le Thu 28 Jan 2010 14:10:27 +0100, a écrit :
> I browsed the sources of funk OS one year ago, if I remember corectly you are
> one og its author.

Yep :)

> I assume that if funk manages memory and run threads it's possible to code an
> OS from ground up using Ocaml but want to be sure:
> 
> - Does I need to disable an option to avoid the garbage collector's use? (I
> gues that yes)

We didn't need to. All caml-managed memory is in the heap. Of course for
page tables you can not allocate them in the garbage-collected area.

> - Also need I disable Ocaml theading subsystem? (Obviously yes, but are there
> some limitations?)

IIRC we just needed to port it.

> Are there other important considerations to take?

In my memory, mostly the direct access to some kinds of memory, like the
video memory: we faked a string with the -unsafe option to get efficient
direct access.

> Do you think that Ocaml is suitable for OS coding (I''m using C now).

It's much better for all the programmability & safety reasons. Funk
showed that it is possible. Performance should be quite good.  Now the
pragmatic answer would be that Linux already works quite well and has
all the drivers we need, while yet another new kernel would have to
rewrite them all. And about performance, when you see how much Linux
people care about tiny details in their lock implementation etc., a caml
implementation wouldn't suit that.

To sum it up: to get a safe working kernel, caml should be fine. To get
a kernel that works on most hardware and is as efficient as possible,
just take Linux :/

Samuel

Re: [Caml-list] Ocaml implementation and low level details

[Konstantin Tcholokachvili]

[-- Attachment #1: Type: text/plain, Size: 2262 bytes --]

2010/1/28 Samuel Thibault <samuel.thibault@inria.fr>

> Konstantin Tcholokachvili, le Thu 28 Jan 2010 14:10:27 +0100, a écrit :
> > I browsed the sources of funk OS one year ago, if I remember corectly you
> are
> > one og its author.
>
> Yep :)
>
> > I assume that if funk manages memory and run threads it's possible to
> code an
> > OS from ground up using Ocaml but want to be sure:
> >
> > - Does I need to disable an option to avoid the garbage collector's use?
> (I
> > gues that yes)
>
> We didn't need to. All caml-managed memory is in the heap. Of course for
> page tables you can not allocate them in the garbage-collected area.
>
> > - Also need I disable Ocaml theading subsystem? (Obviously yes, but are
> there
> > some limitations?)
>
> IIRC we just needed to port it.
>

OK but as there is a giant lock (as I heard), I'm afraid that the
multithreading subsystem of my kernel will suffer from that.
Am I correct?

>
> > Are there other important considerations to take?
>
> In my memory, mostly the direct access to some kinds of memory, like the
> video memory: we faked a string with the -unsafe option to get efficient
> direct access.
>

So must I also make tricks to have DMA acess?


>
> > Do you think that Ocaml is suitable for OS coding (I''m using C now).
>
> It's much better for all the programmability & safety reasons. Funk
> showed that it is possible. Performance should be quite good.  Now the
> pragmatic answer would be that Linux already works quite well and has
> all the drivers we need, while yet another new kernel would have to
> rewrite them all. And about performance, when you see how much Linux
> people care about tiny details in their lock implementation etc., a caml
> implementation wouldn't suit that.
>

My goal isn't to have a kenel portable across many platforms but only to
some kind of hardware.
It's a hobby project.
Why caml's implementation wouldn't be suitable? Because of the giant lock as
I mentioned before?


>
> To sum it up: to get a safe working kernel, caml should be fine. To get
> a kernel that works on most hardware and is as efficient as possible,
> just take Linux :/
>



>
> Samuel
>


Konstantin Tcholokachvili

[-- Attachment #2: Type: text/html, Size: 3455 bytes --]

Re: [Caml-list] Ocaml implementation and low level details

[Samuel Thibault]

Konstantin Tcholokachvili, le Thu 28 Jan 2010 14:35:50 +0100, a écrit :
>     > - Also need I disable Ocaml theading subsystem? (Obviously yes, but are
>     there
>     > some limitations?)
> 
>     IIRC we just needed to port it.
> 
> 
> OK but as there is a giant lock (as I heard), I'm afraid that the
> multithreading subsystem of my kernel will suffer from that.
> Am I correct?

Ah, the kernel can't be running concurrently, yes. Just like Linux 2.0
was working, actually.

>     > Are there other important considerations to take?
> 
>     In my memory, mostly the direct access to some kinds of memory, like the
>     video memory: we faked a string with the -unsafe option to get efficient
>     direct access.
> 
> So must I also make tricks to have DMA acess?

Yes, unless you get hooks into the caml runtime to be notified of
garbage collection, to update pointers & such.

>     > Do you think that Ocaml is suitable for OS coding (I''m using C now).
> 
>     It's much better for all the programmability & safety reasons. Funk
>     showed that it is possible. Performance should be quite good.  Now the
>     pragmatic answer would be that Linux already works quite well and has
>     all the drivers we need, while yet another new kernel would have to
>     rewrite them all. And about performance, when you see how much Linux
>     people care about tiny details in their lock implementation etc., a caml
>     implementation wouldn't suit that.
> 
> My goal isn't to have a kenel portable across many platforms but only
> to some kind of hardware.  It's a hobby project.

Ok, then you can probably start with the current funk testbed :)

> Why caml's implementation wouldn't be suitable? Because of the giant lock as I
> mentioned before?

Because you do not have as much control over e.g. data alignment & such
as in C. Linux people spend quite some time fine-tuning such small
details and get performance benefits.

Samuel

Re: [Caml-list] Ocaml implementation and low level details

[Konstantin Tcholokachvili]

[-- Attachment #1: Type: text/plain, Size: 2361 bytes --]

Thank you for these infos, I will consider which solution fits the best with
my project: make a trade off, find a more suitable ML implementation or
stay  (sadly) with C.

Konstantin Tcholokachvili

2010/1/28 Samuel Thibault <samuel.thibault@inria.fr>

> Konstantin Tcholokachvili, le Thu 28 Jan 2010 14:35:50 +0100, a écrit :
> >     > - Also need I disable Ocaml theading subsystem? (Obviously yes, but
> are
> >     there
> >     > some limitations?)
> >
> >     IIRC we just needed to port it.
> >
> >
> > OK but as there is a giant lock (as I heard), I'm afraid that the
> > multithreading subsystem of my kernel will suffer from that.
> > Am I correct?
>
> Ah, the kernel can't be running concurrently, yes. Just like Linux 2.0
> was working, actually.
>
> >     > Are there other important considerations to take?
> >
> >     In my memory, mostly the direct access to some kinds of memory, like
> the
> >     video memory: we faked a string with the -unsafe option to get
> efficient
> >     direct access.
> >
> > So must I also make tricks to have DMA acess?
>
> Yes, unless you get hooks into the caml runtime to be notified of
> garbage collection, to update pointers & such.
>
> >     > Do you think that Ocaml is suitable for OS coding (I''m using C
> now).
> >
> >     It's much better for all the programmability & safety reasons. Funk
> >     showed that it is possible. Performance should be quite good.  Now
> the
> >     pragmatic answer would be that Linux already works quite well and has
> >     all the drivers we need, while yet another new kernel would have to
> >     rewrite them all. And about performance, when you see how much Linux
> >     people care about tiny details in their lock implementation etc., a
> caml
> >     implementation wouldn't suit that.
> >
> > My goal isn't to have a kenel portable across many platforms but only
> > to some kind of hardware.  It's a hobby project.
>
> Ok, then you can probably start with the current funk testbed :)
>
> > Why caml's implementation wouldn't be suitable? Because of the giant lock
> as I
> > mentioned before?
>
> Because you do not have as much control over e.g. data alignment & such
> as in C. Linux people spend quite some time fine-tuning such small
> details and get performance benefits.
>
> Samuel
>

[-- Attachment #2: Type: text/html, Size: 3018 bytes --]

Re: [Caml-list] Ocaml implementation and low level details

[Basile STARYNKEVITCH]

Konstantin Tcholokachvili wrote:
> Thank you for these infos, I will consider which solution fits the best 
> with my project: make a trade off, find a more suitable ML 
> implementation or stay  (sadly) with C.


You might also consider some safer system programming languages, like Cyclone or BitC.

You could also consider generating some "safer" C from some other, better, language...

http://en.wikipedia.org/wiki/Cyclone_%28programming_language%29
http://cyclone.thelanguage.org/
http://www.bitc-lang.org/

Regards.

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

Re: [Caml-list] Ocaml implementation and low level details

[Richard Jones]

On Thu, Jan 28, 2010 at 01:42:15PM +0100, Konstantin Tcholokachvili wrote:
> I am writing an operating system kernel and I am considering the idea of
> rewritting it in Ocaml or make a wrapper.
> As I don't  know how Ocaml is implemented in detail so I have the following
> question:
> If want to code in Ocaml without using the garbage collector, will I be able
> to use my own multithreading implementation or will I be limited by Ocaml's
> global lock?

You're probably better off not using the current runtime, but instead
implementing enough of the runtime based on the functions that the
code generator needs.  But reading the rest of the thread it sounds
like you really need to look deeply at the current implementation
first before you are in a position to make any decision (luckily the
runtime of OCaml is not hard to understand, and is mostly written in
C).  Asking if one can code in OCaml "without using the garbage
collector" doesn't really make any sense as a question, since at least
the minor heap is a fundamental concept in the language.  And the
"global lock" just prevents reentrancy in the current implementation
of the GC -- you can easily use one minor heap per thread, although
that is likely to just push the problem elsewhere.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Ocaml implementation and low level details

[Goswin von Brederlow]

Richard Jones <rich@annexia.org> writes:

> On Thu, Jan 28, 2010 at 01:42:15PM +0100, Konstantin Tcholokachvili wrote:
>> I am writing an operating system kernel and I am considering the idea of
>> rewritting it in Ocaml or make a wrapper.
>> As I don't  know how Ocaml is implemented in detail so I have the following
>> question:
>> If want to code in Ocaml without using the garbage collector, will I be able
>> to use my own multithreading implementation or will I be limited by Ocaml's
>> global lock?
>
> You're probably better off not using the current runtime, but instead
> implementing enough of the runtime based on the functions that the
> code generator needs.  But reading the rest of the thread it sounds
> like you really need to look deeply at the current implementation
> first before you are in a position to make any decision (luckily the
> runtime of OCaml is not hard to understand, and is mostly written in
> C).  Asking if one can code in OCaml "without using the garbage
> collector" doesn't really make any sense as a question, since at least
> the minor heap is a fundamental concept in the language.  And the
> "global lock" just prevents reentrancy in the current implementation
> of the GC -- you can easily use one minor heap per thread, although
> that is likely to just push the problem elsewhere.
>
> Rich.

A heap per thread makes inter thread communication more difficult. On
the other hand it allows multi core support and keeps the GC work per
thread small. Still, you can't have interrupts that allocate memory
while the GC runs and such. So you need at least some part that is
outside the GC.

I've played with the idea of using ocaml (or variant thereof) for a
kernel too. My idea was to write a minimal mikrokernel in a restricted
subset of ocaml and then allow ocaml with GC for the drivers and
modules. Each driver/module would be a seperate process with its own
stack and heap and use little memory. So the GC work wouldn't mean long
interruptions even if compacting.

For the core I was thinking that maybe one could cut out heap alocation
and instead use stack allocation with some manual allocation for special
cases for the core. That would mean a number of changes though:

- use in-place algorithms (mutables) for a lot of things
- no deep recursive datatypes as return values
- allocate the maximum size for variant types for returns
- allocate space for the return value on the caller stack

No heap allocation means no GC required. On the other hand a stack value
must never escape the function it is allocated in. Tracking that would
be a huge change in the compiler.

Overall I was afraid the result would be more changed things than
remaining ocaml. So it never wnet further than some thought
epxperiments. But maybe this gives someone some ideas.

MfG
        Goswin

Re: [Caml-list] Ocaml implementation and low level details

[Konstantin Tcholokachvili]

Thank for all your replies, after reading different messages, it seems
that D language fits most of my needs/taste unless I find another ML
based language suitable for kernel coding.

The thing I like about Ocaml is it's ability of type inference and
it's speed, that's why I considered it.

I guess that as JoCaml is based on Ocaml it has the same issues for
kernel coding.

Konstantin Tcholokachvili

2010/1/28, Goswin von Brederlow <goswin-v-b@web.de>:
> Richard Jones <rich@annexia.org> writes:
>
>> On Thu, Jan 28, 2010 at 01:42:15PM +0100, Konstantin Tcholokachvili wrote:
>>> I am writing an operating system kernel and I am considering the idea of
>>> rewritting it in Ocaml or make a wrapper.
>>> As I don't  know how Ocaml is implemented in detail so I have the
>>> following
>>> question:
>>> If want to code in Ocaml without using the garbage collector, will I be
>>> able
>>> to use my own multithreading implementation or will I be limited by
>>> Ocaml's
>>> global lock?
>>
>> You're probably better off not using the current runtime, but instead
>> implementing enough of the runtime based on the functions that the
>> code generator needs.  But reading the rest of the thread it sounds
>> like you really need to look deeply at the current implementation
>> first before you are in a position to make any decision (luckily the
>> runtime of OCaml is not hard to understand, and is mostly written in
>> C).  Asking if one can code in OCaml "without using the garbage
>> collector" doesn't really make any sense as a question, since at least
>> the minor heap is a fundamental concept in the language.  And the
>> "global lock" just prevents reentrancy in the current implementation
>> of the GC -- you can easily use one minor heap per thread, although
>> that is likely to just push the problem elsewhere.
>>
>> Rich.
>
> A heap per thread makes inter thread communication more difficult. On
> the other hand it allows multi core support and keeps the GC work per
> thread small. Still, you can't have interrupts that allocate memory
> while the GC runs and such. So you need at least some part that is
> outside the GC.
>
> I've played with the idea of using ocaml (or variant thereof) for a
> kernel too. My idea was to write a minimal mikrokernel in a restricted
> subset of ocaml and then allow ocaml with GC for the drivers and
> modules. Each driver/module would be a seperate process with its own
> stack and heap and use little memory. So the GC work wouldn't mean long
> interruptions even if compacting.
>
> For the core I was thinking that maybe one could cut out heap alocation
> and instead use stack allocation with some manual allocation for special
> cases for the core. That would mean a number of changes though:
>
> - use in-place algorithms (mutables) for a lot of things
> - no deep recursive datatypes as return values
> - allocate the maximum size for variant types for returns
> - allocate space for the return value on the caller stack
>
> No heap allocation means no GC required. On the other hand a stack value
> must never escape the function it is allocated in. Tracking that would
> be a huge change in the compiler.
>
> Overall I was afraid the result would be more changed things than
> remaining ocaml. So it never wnet further than some thought
> epxperiments. But maybe this gives someone some ideas.
>
> MfG
>         Goswin
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>