[9fans] extending xen to allow driver development in Plan 9

[9fans] extending xen to allow driver development in Plan 9

[ron minnich]

This is mostly for Richard Miller but I don't have his email. But if
you are interested in Xen, read along.

We have an ok xen environment going. Why are we doing this? Per a
certain person at xyz.com, we are looking at giving people a usable
xen-based plan 9 environment, and at the same time letting them do
driver work from Plan 9 by "poking holes" in Xen to let Plan 9 at the
real hardware. Xen supports this, we think, although we have not got
it going yet ...

I already like the situation thus far, as Plan 9 under Xen is a ton
faster than Plan 9 under qemu. You have to see it to believe it; if
anything, the Xen advantage is better than it used to be. I was
surprised.

to get to the point of poking holes in Xen, it turned out I need
pcifront. For pcifront I need xenbus. for xenbus I need xenstore.

There is xenstore support in Plan 9 already, but ...

The xenstore sez: "incomplete". What would it take to complete it?
conservative use of locks in the short term as a hack for really doing
it right in the long term? The comment is this:

 * XXX This is incomplete - needs multiplexing of request/response protocol
 * and locking between driver and kernel-only xenstore_read/write interface.

Should we set up queues for request/response? The locking seems simple
enough, is there something I'm missing?

ron

Re: [9fans] extending xen to allow driver development in Plan 9

[Richard Miller]

> Should we set up queues for request/response?

Not necessarily a queue because I don't think there's a guarantee that
responses come in the same order as requests.  Maybe a hash table of
requests awaiting responses?

> The locking seems simple
> enough, is there something I'm missing?

No, it should be simple.  I just hadn't got around to it yet.

Again: (self)hosted Plan9? Was: [9fans] extending xen to allow driver development in Plan 9

[Georg Lehner]

"ron minnich" <rminnich@gmail.com> writes:
...
> We have an ok xen environment going. Why are we doing this? Per a
> certain person at xyz.com, we are looking at giving people a usable
> xen-based plan 9 environment, and at the same time letting them do
> driver work from Plan 9 by "poking holes" in Xen to let Plan 9 at the
> real hardware. Xen supports this, we think, although we have not got
> it going yet ...
>
> I already like the situation thus far, as Plan 9 under Xen is a ton
> faster than Plan 9 under qemu. You have to see it to believe it; if
> anything, the Xen advantage is better than it used to be. I was
> surprised.
...

I have a similar situation:

- Xen helps me run several Plan9's one the same hardware

- I can give my users a Plan9 environment without taking away the OS
  they are used to work with

- Xen is much faster then Qemu, ok for production use

- as Richard Miller said: ".. the whole point of xen is that physical
  devices become Somebody Else's Problem."

However I think that the same goals could be achieved more natural,
even faster, more stable and more generally aplicable if Plan9 could
be run (self)hosted.

The Hurd can be run as a user space process inside The Hurd.  Made
feasable because of its multi-server nature: the Kernel almost does
not do I/O.  Thus The Hurd allegedly can be debugged and developed
more easily.

I guess the Plan9 Kernel could be separated in two layers, the upper
one just doing "high-level" and 9P-protocol stuff, and a lower one,
providing the #-channel interfaces to the upper layer and doing I/O.

The lower layer could either be comprised of hardware drivers for the
real hardware, or a hosting layer which intermediates between the
block devices and memory managment operations of a certain hosting
operating system and the #-channel interface to the upper layer.

Maybe this approach could also clean up the duplication of code
between 9loader and kernel I have read about in some Plan9 document.

Hardware driver development could also be eased by this approach,
since it is probably easier to pass certain hardware through to a
Linux process (the hosted Plan9 instance), than to go through the
complexities of Xen-Hypervisor - dom0 Linux - domU Plan9 interaction.

And: I know that this approach probably would increase complexity and
reduce performance with respect to the current Plan9 kernel.

Initially I have started to browse the Plan9 kernel source code, Linux
kernel docs, x86 assembler manuals etc., but I realized very fast,
that my spare time will never be sufficient to spot out all required
points to get anywhere with such a project.  However maybe there are
some folks out there who like the idea and have the knowledge to do
it.

Best Regards,

    Jorge-León

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow driver

[Lucio De Re]

> I guess the Plan9 Kernel could be separated in two layers, the upper
> one just doing "high-level" and 9P-protocol stuff, and a lower one,
> providing the #-channel interfaces to the upper layer and doing I/O.

My thoughts are along similar lines.  My approach would be to expose
the device drivers in a hardware dependent "BIOS" as Plan 9 devices,
adding only as much OS glue at this level as makes this possible.
This would be as close as damn what we know as 9load today.

The Plan 9 kernel is the only one that would be able to interface to
this BIOS currently, but over time it ought to be possible, mutatis
mutandis, to boot other OSes on this layer.  My dream is that the BIOS
could then be extended by the hordes of device driver writers and
every compliant OS would be able to use new devices immediately.

But I may be just dreaming.

++L

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow driver

[ron minnich]

On 12/6/06, Lucio De Re <lucio@proxima.alt.za> wrote:
> > I guess the Plan9 Kernel could be separated in two layers, the upper
> > one just doing "high-level" and 9P-protocol stuff, and a lower one,
> > providing the #-channel interfaces to the upper layer and doing I/O.
>
> My thoughts are along similar lines.  My approach would be to expose
> the device drivers in a hardware dependent "BIOS" as Plan 9 devices,
> adding only as much OS glue at this level as makes this possible.
> This would be as close as damn what we know as 9load today.
>
> The Plan 9 kernel is the only one that would be able to interface to
> this BIOS currently, but over time it ought to be possible, mutatis
> mutandis, to boot other OSes on this layer.  My dream is that the BIOS
> could then be extended by the hordes of device driver writers and
> every compliant OS would be able to use new devices immediately.

well, we're trying to sort of do that now. We're using linux as the
driver layer. It's not what you guys want implementation-wise, but it
is something  like the idea.

It's not that drivers are fundamentally hard. It's that the hardware
we work with is undocumented crap. Linux drivers know all the secrets;
we're riding on that knowledge.

ron

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow

[Lucio De Re]

> It's not that drivers are fundamentally hard. It's that the hardware
> we work with is undocumented crap. Linux drivers know all the secrets;
> we're riding on that knowledge.

And so do millions more, perfectly understandably.  The problem is
that "all the world is Linux" is not a good mantra.  Porting drivers
from Linux to, say, NetBSD is a nightmare, multiplied by the number of
useful target OSes.  Whereas XEN largely suffers (only) from
inefficiencies accessing the lower layer, whenever it (and you, if I
understand your recent discussions) tries to punch through the
barrier, the mysteries strike again.

(In passing, I was looking at ISDN adapter drivers with a view to
implementing the functionality under NetBSD.  The Linux driver, in my
opinion, was orders of magnitude better coded than the FreeBSD
version.  Take that any way you like, it has changed my opinions on
Open Source device driver developers.)

To return to the main issue, I think effort applied towards
documenting "undocumented crap" would have a wider scope than adopting
or reverse engineering the knowledge in Linux drivers code.  The
latter is certainly a more immediate objective.  Of course, one then
also needs to deal with binary-only drivers and other such stumbling
blocks, but my hope would be that eventually hardware manufacturers
will get the message or will get deselected :-)

Given the choice between using Linux kernel source as the
documentation, versus Plan 9 kernel source, there are too many good
reasons to pick the Plan 9 option to list them here, where they are in
any case taken for granted.  Hence my preference for a 9load-type BIOS
on which others besides Plan 9 can build.

(The philosophy, probably flawed, is that Open Source principles are
"right" in some transcendent way and that a "good" manufacturer cannot
continue to overlook the benefits of being on the "right" side of the
line.  Communism had the same underlying principle and landed up on
the scrap-heap of history.)

++L

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow

[ron minnich]

> To return to the main issue, I think effort applied towards
> documenting "undocumented crap" would have a wider scope than adopting
> or reverse engineering the knowledge in Linux drivers code.

I agree. We've been doing this kind of documentation for seven years
now. I would say it has gotten harder in the last four years. There is
ever more paranoia and unwillingness to open programming details up.
Just compare the Intel L440 manuals you could get in 1999 vs. what is
at developer.intel.com now. The L440 chipset manual was wonderful,
with full docs and a tutorial on SDRAM programming. The stuff there
now is basically useless for writing a BIOS. We have been told that is
intentional.

>  Of course, one then
> also needs to deal with binary-only drivers and other such stumbling
> blocks, but my hope would be that eventually hardware manufacturers
> will get the message or will get deselected :-)

It's not working so far. Most people want their games fast; they don't
care about whether the chipsets are documented.


>
> (The philosophy, probably flawed, is that Open Source principles are
> "right" in some transcendent way and that a "good" manufacturer cannot
> continue to overlook the benefits of being on the "right" side of the
> line.

It's an ongoing battle. Toss in DRM and things really get ugly. The
preferred vendor approach is to lock things up when we re not
watching. It's our job to keep watching.

ron

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow

[Chad Dougherty]

yep.  right on the money.

Theo, who's been a champion for more open hardware documentation, has a 
really nice set of slides on this topic:

<http://www.openbsd.org/papers/opencon06-docs/index.html>

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow

[Steve Simon]

I am keen enough to try and get some hardware documentation, and maybe
even have a go at codeing a/some drivers for plan9.

I only use hardware which plan9 is compatible with so I don't know
which way to look. At IWP9 modern, inexpensive SATA cards where
mentioned as somwhere we had a gap in our coverage.

Is this all we need? Can anyone suggest a card that fits the bill?

how about modern laptops, we have a ⅞ finished centrino driver which
needs to be finished off, but what graphics chipset is common enough
to make it worthwhile chasing the manufacturer, is nvidia still king?
Do they have weird interrupt controllers, southbridges etc which will
cause problems?

As usual I promise nothing but I will do what I can.

-Steve

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow

[Lucio De Re]

[-- Attachment #1: Type: text/plain, Size: 1460 bytes --]

I seem to have long range objectives: I believe that Plan 9 could
make a better virtualisation host than the current choice for the job
and that we ought to sell to the world that it is worth (a) adopting
the Plan 9 paradigm for hosted device drivers and (b) developing all
future physical device drivers to this paradigm.

Putting it in a different way, consider replacing GRUB with 9LOAD as
first step.  Then extend 9LOAD so that it somehow contains the correct
device drivers for the particular platform and use it to boot the next
stage which accesses device drivers Plan 9-style.  Now, if we can get
the hordes of device driver implementors to focus on this approach,
Plan 9's greatest shortcoming (and all other Open Source OSes's)
suddenly disappears.

Sure, there are efficiency considerations to take into account, as
well as the unlikelihood of persuading Lunix developers to switch
allegiances, but a concerted effort to head that way is certainly not
doomed before even starting, at least not unless I'm subconsciously
ignoring some obvious obstacle.

This idea has been bugging me ever since rminnich started making
noises about Xen and I was tempted to look at it.  It bothers me that
Xen is designed very much with Linux as its model, not just its
primary target.  My claim is that a design founded on a Plan 9
foundation would be almost guaranteed to be more successful.  That's
what I call "faith", I suppose.

++L

[-- Attachment #2: Type: message/rfc822, Size: 3621 bytes --]

From: "Steve Simon" <steve@quintile.net>
To: 9fans@cse.psu.edu
Subject: Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow
Date: Sat, 9 Dec 2006 11:21:55 +0000
Message-ID: <e1e1d3e3266ab5e656eabdae89aa3030@quintile.net>

I am keen enough to try and get some hardware documentation, and maybe
even have a go at codeing a/some drivers for plan9.

I only use hardware which plan9 is compatible with so I don't know
which way to look. At IWP9 modern, inexpensive SATA cards where
mentioned as somwhere we had a gap in our coverage.

Is this all we need? Can anyone suggest a card that fits the bill?

how about modern laptops, we have a ⅞ finished centrino driver which
needs to be finished off, but what graphics chipset is common enough
to make it worthwhile chasing the manufacturer, is nvidia still king?
Do they have weird interrupt controllers, southbridges etc which will
cause problems?

As usual I promise nothing but I will do what I can.

-Steve

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow

[erik quanstrom]

"Steve Simon" <steve@quintile.net> writes

| 
| I am keen enough to try and get some hardware documentation, and maybe
| even have a go at codeing a/some drivers for plan9.
| 
| I only use hardware which plan9 is compatible with so I don't know
| which way to look. At IWP9 modern, inexpensive SATA cards where
| mentioned as somwhere we had a gap in our coverage.

plan 9 does support the marvell chipset.  for example:
	http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm
this is an inexpensive card.

as for on-board sata, plan 9 does support the sata on nforce430 
(and maybe most nforce) boards, but doesn't seem to support 
many modes of ich[5-] sata+pata hackery.  i think fixing that
would yield the most bang-for-the-buck.

personally, these things are also at the top of my list
- usb ohic and ehic.
- gbit ethernet: forcedeth, tg3.

- erik

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow

[geoff]

The SATA controllers I've encountered so far either emulate ordinary
(P)ATA controllers or have a BIOS setting to optionally do so.  I'm
not sure that there's much to be gained by implementing the AHCI or
other oddball interfaces.

As I mentioned at IWP9, I'm integrating Charles Forsyth's OHCI driver
into the devusb framework, and will probably end up having to write
EHCI (USB 2) support.  I've been working on getting Richard Miller's
usbsfs working with a dozen or so USB disk-like devices (MP3 players,
DVD drives, flash disks) and they are now mostly working, after adding
code to probe and use LUNs (logical unit numbers).  Some of the dumber
devices seem to be very sensitive to exactly how you talk to them and
go off into the weeds if they don't approve, so getting a version of
usbsfs that they can all talk to is taking longer than expected.

Gigabit Ethernet seems to be pretty well handled; we've got drivers
for the Intel controllers (though Intel keeps introducing new
not-quite compatible variations) and the Realtek 8169, though it
pauses and thus has low throughput on my machine.  Do the other
gigabit controllers appear on lots of motherboards?

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow

[andrey mirtchovski]

> Do the other gigabit controllers appear on lots of motherboards?

broadcom (tg3) used to be _the_ ethernet for opterons for quite some
time (we saw one board by iWill which had intels only this year for
the first time)...

all our opterons have an 8169 stuck in a pci slot...

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow

[Charles Forsyth]

>>how about modern laptops, we have a ⅞ finished centrino driver which
>>needs to be finished off, but what graphics chipset is common enough

since i've got a modern laptop, i'm hoping to do an i3945 802.11abg (ok, bg) driver
over christmas, in front of the fire, in between mulled wine and hot toddies.
hmm... wireless and legless.  it's not tuneless, because i've got a good set of CDs.

the incomplete linux driver with binary blobs is only just over 16,000 lines, so how
hard can it be?  hmm.  the free/open bsd driver is only 2800 lines.
shome mishtake shurely.

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow

[Francisco J Ballesteros]

16k lines ????
Really?

On 12/10/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> the incomplete linux driver with binary blobs is only just over 16,000 lines, so how
> hard can it be?  hmm.  the free/open bsd driver is only 2800 lines.
> shome mishtake shurely.
>
>

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 92 bytes --]

h% wc -l *.c
  16731 ipw3945.c

and don't forget the binary user-mode supervisory program

[-- Attachment #2: Type: message/rfc822, Size: 3397 bytes --]

From: "Francisco J Ballesteros" <nemo@lsub.org>
To: "Fans of the OS Plan 9 from Bell Labs" <9fans@cse.psu.edu>
Subject: Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow
Date: Sun, 10 Dec 2006 21:56:45 +0100
Message-ID: <8ccc8ba40612101256o718f4ff1ld607351d115b772e@mail.gmail.com>

16k lines ????
Really?

On 12/10/06, Charles Forsyth <forsyth@terzarima.net> wrote:
> the incomplete linux driver with binary blobs is only just over 16,000 lines, so how
> hard can it be?  hmm.  the free/open bsd driver is only 2800 lines.
> shome mishtake shurely.
>
>

Re: Again: (self)hosted Plan9? Was: [9fans] extending xen to allow

[ron minnich]

On 12/9/06, Steve Simon <steve@quintile.net> wrote:
> I am keen enough to try and get some hardware documentation, and maybe
> even have a go at codeing a/some drivers for plan9.

This is great. All these hardware driver ideas are fine.

But lacking a few other bits, i still believe that all the drivers in
the world are not going to be sufficient. I don't even thinks they are
necessary to get people going.

Put it this way. I have a nice web page served out of a Plan 9 system
that shows google maps data etc. I can't view it on Plan 9. What we
have here, is failure to communicate. Drivers won't help this problem.

So what we're trying to do is give people a path from the linux world
to a better place. The idea is that you'll get a linux kernel as a
device driver. You can even do driver development on Plan 9 in this
world if all works out. The linux kernel will simply ignore the
existence of a piece of hardware; Plan 9 will own it. Over time, more
and more bits can be moved to the Plan 9 domain, until at some point,
we realize that on that particular system, we don't need Linux at all
-- and it vanishes in a puff of smoke and mirrors.

This will be way less extensive than xen-knoppix, which is a full
distro more or less. We don't need X11 -- Aki has got drawterm working
on /dev/fb. Systems get pretty peppy once x11 is out of the picture.
We've realized that we need python to run xen, so one thing we might
do is remove all of gnubin and just have a few .py utililties, so as
to minimize the amount of junk we need to have on the linux dom0
partition. If you have python (NOT that I want it; but you can't avoid
Python and XML RPC in Xen, sigh), and you can import os, it's hard to
see why you want all the gnubin junk. How many implementations of cat
do you need, really?

For apps, you can run a linux app in a window under VNC. Aki has
plumbing working between the two worlds. So our browser for now is the
"light" firefox with its slim-waisted 200 MBYTE footprint.

Think of this stuff as analogous to the V5 JCL command, or the GECOS
field in /etc/passwd; a way to get by until you can cut the cords to
the old-style, primitive environment. And you only use that
environment in a batch mode ...

So, for those of you who don't wish to write drivers, there's still a
large number of us who would appreciate a web browser ... and abiword
or similar or ....

thanks

ron
p.s. Aki leaves in a week, and we're all very sad here. He was on a
training assigment, and unfortunately did not train us as much as we
had hoped. But, hope springs ever anew  ... any students wishing to
spend some time here training us are most welcome to! Note that it's
easier for me to set up if you are a .us, but hey, Aki is from Finland
and we still managed it, even though he doesn't speak Linux.