[9fans] ethervirtio

[9fans] ethervirtio

[mischief]

hello,

if anyone is interested in using or reviewing my ethervirtio driver, here it is.

http://9.offblast.org/stuff/ethervirtio.c

this driver was written for 386 and amd64 9front, but according to a short test i did a few months ago, it should work in the labs' kernel. it would be interesting to know if it works in 9atom too.

i tested qemu 1.5, 1.7 and some version around 2.0.

david (0intro) tested the driver on GCE a few months ago, and reported that it did *not* work. i don't recall what exactly happened, but the virtio device in GCE seems to differ from qemu's.

if you run into trouble, you can mail me back or contact me on freenode irc, where my handle is mischief.

Re: [9fans] ethervirtio

[erik quanstrom]

On Mon Dec  1 20:17:56 PST 2014, mischief@9.offblast.org wrote:
> hello,
>
> if anyone is interested in using or reviewing my ethervirtio driver, here it is.
>
> http://9.offblast.org/stuff/ethervirtio.c
>
> this driver was written for 386 and amd64 9front, but according to a short test i did a few months ago, it should work in the labs' kernel. it would be interesting to know if it works in 9atom too.
>
> i tested qemu 1.5, 1.7 and some version around 2.0.
>
> david (0intro) tested the driver on GCE a few months ago, and reported that it did *not* work. i don't recall what exactly happened, but the virtio device in GCE seems to differ from qemu's.
>
> if you run into trouble, you can mail me back or contact me on freenode irc, where my handle is mischief.

neat.  it's easy to drop in.  have you tried in any other virtualized environments?

- erik

Re: [9fans] ethervirtio

[erik quanstrom]

On Mon Dec  1 20:17:56 PST 2014, mischief@9.offblast.org wrote:
> hello,
>
> if anyone is interested in using or reviewing my ethervirtio driver, here it is.
>
> http://9.offblast.org/stuff/ethervirtio.c
>
> this driver was written for 386 and amd64 9front, but according to a short test i did a few months ago, it should work in the labs' kernel. it would be interesting to know if it works in 9atom too.
>
> i tested qemu 1.5, 1.7 and some version around 2.0.
>
> david (0intro) tested the driver on GCE a few months ago, and reported that it did *not* work. i don't recall what exactly happened, but the virtio device in GCE seems to differ from qemu's.
>
> if you run into trouble, you can mail me back or contact me on freenode irc, where my handle is mischief.

what do you mean by PGROUND here, and BY2PG.  do you really mean an explicit 4096 bytes?

- erik

Re: [9fans] ethervirtio

[mischief]

[-- Attachment #1: Type: text/plain, Size: 197 bytes --]

4.1.5.1.4.1 of the spec does say that a page is 4096 for the purposes of legacy virtio, but 2.4.2 just talks about pages. you think i should just use an explicit 4096 instead of PGROUND/BY2PG here?

[-- Attachment #2: Type: message/rfc822, Size: 3385 bytes --]

From: erik quanstrom <quanstro@quanstro.net>
To: 9fans@9fans.net
Subject: Re: [9fans] ethervirtio
Date: Mon, 1 Dec 2014 20:28:13 -0800
Message-ID: <275b11f7b59c509c096d6adc193e6cd1@lilly.quanstro.net>

On Mon Dec  1 20:17:56 PST 2014, mischief@9.offblast.org wrote:
> hello,
>
> if anyone is interested in using or reviewing my ethervirtio driver, here it is.
>
> http://9.offblast.org/stuff/ethervirtio.c
>
> this driver was written for 386 and amd64 9front, but according to a short test i did a few months ago, it should work in the labs' kernel. it would be interesting to know if it works in 9atom too.
>
> i tested qemu 1.5, 1.7 and some version around 2.0.
>
> david (0intro) tested the driver on GCE a few months ago, and reported that it did *not* work. i don't recall what exactly happened, but the virtio device in GCE seems to differ from qemu's.
>
> if you run into trouble, you can mail me back or contact me on freenode irc, where my handle is mischief.

what do you mean by PGROUND here, and BY2PG.  do you really mean an explicit 4096 bytes?

- erik

Re: [9fans] ethervirtio

[erik quanstrom]

also,

MSS really looks out of place.  i would expect MTU.  MSS is a TCP concept.

- erik

Re: [9fans] ethervirtio

[David du Colombier]

> david (0intro) tested the driver on GCE a few months ago, and
> reported that it did *not* work. i don't recall what exactly
> happened, but the virtio device in GCE seems to differ from qemu's.

I haven't tried for a while, since the debugging process is
a bit time consuming.

As far I remember, the virtqueue sizes read from the Virtio
common configuration structure were not correct (as reported
by the "legacy interface").

Rx/Tx/Control were something like 16384/81920/131136, while
the maximum queue size is 32768 according to the specification.

http://docs.oasis-open.org/virtio/virtio/v1.0/virtio-v1.0.html

Setting Tx and Control queue size to more realistic values
seems to work, but then I couldn't get the interruptions to
work properly.

Something seems to be missing during the device initialization.

The original discussion thread was:

https://groups.google.com/forum/#!topic/golang-dev/Vb4fcGwg-dE

It includes links to the code and everything that's needed
to run Plan 9 on GCE (last email).

--
David du Colombier

Re: [9fans] ethervirtio

[mischief]

good news, everyone. ethervirtio seems to work now on google compute engine in both labs (tested by david) and 9front (tested by me). somebody on irc reported that it also works for vultr.com. the driver should work fine in any qemu.

 only the 386 kernel for labs and 9front were tested, but the 9front amd64 kernel should work fine too.


i really appreciate the feedback erik gave me, it helped a lot.

 although, i still don't know if the driver works in 9atom or not! it would be nice if somebody could test it in 9atom and report back.

as for gce, there's one last showstopper: the dhcp server is kind of nuts. so, ipconfig (maybe ip stack?) needs to be fixed. for posterity here is what the gce dhcp server gave us:

ipconfig: got ack from 10.240.0.1
ipconfig: lease=4294967295
ipconfig: ipaddr=10.240.189.103 ipmask=255.255.255.255
ipconfig: ipgw=10.240.0.1
ipconfig: dns=169.254.169.254
ipconfig: dns=10.240.0.1
ipconfig: ntp=169.254.169.254
ipconfig: server=169.254.169.254 sname=

Re: [9fans] ethervirtio

[mischief]

for more information on the dhcp issue with gce, see https://code.google.com/p/google-compute-engine/issues/detail?id=77

Re: [9fans] ethervirtio

[Skip Tavakkolian]

[-- Attachment #1: Type: text/plain, Size: 1232 bytes --]

nice work! thank you all.

is there an image to start from? any instructions?


On Wed, Dec 3, 2014 at 4:05 PM, <mischief@9.offblast.org> wrote:

> good news, everyone. ethervirtio seems to work now on google compute
> engine in both labs (tested by david) and 9front (tested by me). somebody
> on irc reported that it also works for vultr.com. the driver should work
> fine in any qemu.
>
>  only the 386 kernel for labs and 9front were tested, but the 9front amd64
> kernel should work fine too.
>
>
> i really appreciate the feedback erik gave me, it helped a lot.
>
>  although, i still don't know if the driver works in 9atom or not! it
> would be nice if somebody could test it in 9atom and report back.
>
> as for gce, there's one last showstopper: the dhcp server is kind of nuts.
> so, ipconfig (maybe ip stack?) needs to be fixed. for posterity here is
> what the gce dhcp server gave us:
>
> ipconfig: got ack from 10.240.0.1
> ipconfig: lease=4294967295
> ipconfig: ipaddr=10.240.189.103 ipmask=255.255.255.255
> ipconfig: ipgw=10.240.0.1
> ipconfig: dns=169.254.169.254
> ipconfig: dns=10.240.0.1
> ipconfig: ntp=169.254.169.254
> ipconfig: server=169.254.169.254 sname=
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 1651 bytes --]

Re: [9fans] ethervirtio

[mischief]

On Wed Dec  3 20:03:51 EST 2014, skip.tavakkolian@gmail.com wrote:

> nice work! thank you all.
>
> is there an image to start from? any instructions?
>
>

you should look at david's link to the golang-dev thread a few posts ago. he has a labs image and some notes about the setup. my 9front image was very stripped down for testing.

one strange thing about gce is that when the kernel is reading the PCI bar for the sdvirtio device, the port and length are zero, so the driver had to be hacked to set the port and length, which might not work in non-gce hosts. this doesn't happen with the ethervirtio device.

here's the direct link to david's notes. http://www.9legacy.org/9legacy/doc/gce/notes

one particularly important thing to know is that the disk will appear at #S/sd01 with sdvirtio on gce.

Re: [9fans] ethervirtio

[Skip Tavakkolian]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

cool, thanks! i missed the golang-dev thread in sept.


On Wed, Dec 3, 2014 at 5:29 PM, <mischief@9.offblast.org> wrote:

> On Wed Dec  3 20:03:51 EST 2014, skip.tavakkolian@gmail.com wrote:
>
> > nice work! thank you all.
> >
> > is there an image to start from? any instructions?
> >
> >
>
> you should look at david's link to the golang-dev thread a few posts ago.
> he has a labs image and some notes about the setup. my 9front image was
> very stripped down for testing.
>
> one strange thing about gce is that when the kernel is reading the PCI bar
> for the sdvirtio device, the port and length are zero, so the driver had to
> be hacked to set the port and length, which might not work in non-gce
> hosts. this doesn't happen with the ethervirtio device.
>
> here's the direct link to david's notes.
> http://www.9legacy.org/9legacy/doc/gce/notes
>
> one particularly important thing to know is that the disk will appear at
> #S/sd01 with sdvirtio on gce.
>

[-- Attachment #2: Type: text/html, Size: 1490 bytes --]

Re: [9fans] ethervirtio

[David du Colombier]

I've just uploaded a new disk image with the latest
ethervirtio changes.

http://9legacy.org/download/plan9-gce.img.bz2

This image will be able to run on both QEMU
(using virtio net and virtio-scsi disk) and GCE.

Since QEMU (#S/sd00) and GCE (#S/sd01) use a different
controller name, the plan9.ini will let you choose
between QEMU and GCE.

Here is the output of the GCE console:

http://9legacy.org/9legacy/doc/gce/console

You can easily create a Plan 9 disk or cd image
yourself by applying the patches and following
the instructions on:

http://9legacy.org/9legacy/doc/gce/notes

The next step is to fix the DHCP client in ip/ipconfig.

--
David du Colombier

Re: [9fans] ethervirtio

[hiro]

Did anyone compare throughput, pps, etc. with the same on standard rtl
or intel emulated cards?

On 12/4/14, David du Colombier <0intro@gmail.com> wrote:
> I've just uploaded a new disk image with the latest
> ethervirtio changes.
>
> http://9legacy.org/download/plan9-gce.img.bz2
>
> This image will be able to run on both QEMU
> (using virtio net and virtio-scsi disk) and GCE.
>
> Since QEMU (#S/sd00) and GCE (#S/sd01) use a different
> controller name, the plan9.ini will let you choose
> between QEMU and GCE.
>
> Here is the output of the GCE console:
>
> http://9legacy.org/9legacy/doc/gce/console
>
> You can easily create a Plan 9 disk or cd image
> yourself by applying the patches and following
> the instructions on:
>
> http://9legacy.org/9legacy/doc/gce/notes
>
> The next step is to fix the DHCP client in ip/ipconfig.
>
> --
> David du Colombier
>
>

Re: [9fans] ethervirtio

[erik quanstrom]

On Wed Dec  3 16:18:24 PST 2014, mischief@9.offblast.org wrote:
> good news, everyone. ethervirtio seems to work now on google compute engine in both labs (tested by david) and 9front (tested by me). somebody on irc reported that it also works for vultr.com. the driver should work fine in any qemu.

where's the current code?

> as for gce, there's one last showstopper: the dhcp server is kind of nuts. so, ipconfig (maybe ip stack?) needs to be fixed. for posterity here is what the gce dhcp server gave us:

ip/dhcpd -h gce #-h hack

- erik

Re: [9fans] ethervirtio

[David du Colombier]

> where's the current code?

The current code of ethervirtio is available on :

http://9.offblast.org/stuff/ethervirtio.c

My notes contains a list of all the required
patches to run Plan 9 on GCE:

http://9legacy.org/9legacy/doc/gce/notes

--
David du Colombier

Re: [9fans] ethervirtio

[mischief]

[-- Attachment #1: Type: text/plain, Size: 205 bytes --]

Im rewriting parts of the driver after feedback from cinap, namely removing qlock and nfree, removing the transmit callback and rewriting txproc to use qbread. ☺

The current source should work though.

[-- Attachment #2: Type: text/html, Size: 213 bytes --]