[9fans] usbether

[9fans] usbether

[Richard Miller]

I've been struggling to get the raspberry pi's built-in usb ethernet
adapter working.  I can send and receive packets reliably enough to
get a remote file system mounted, but with any kind of heavy use the
usbether input seems to be missing a lot of packets, and everything
gets horribly slow.

After running out of 9pi-specific things to debug, it occurred to me
to try a usb ethernet dongle on an x86 plan 9 machine.  There I
observed the same thing: so many dropped packets that the connection
is unusable.

So, has anyone had success using usbether to connect a plan 9 system
to the outside world?  I am hoping someone can give me an encouraging
report.  I'm a bit worried that it's a fundamental problem with the
plan 9 usb architecture, which is basically synchronous - the host
adapter driver in the kernel will poll a device for input only when
the user-level driver process does a read.  This is ok for things like
usbdisk which have an rpc-like protocol, but seems less well suited to
things like ethernet and serial interfaces, where the equivalent
non-usb kernel drivers use qio to read ahead into a queue of buffers
until the user-level consuming process gets around to reading them.

Would anyone like to share experiences or comments?

Re: [9fans] usbether

[Gorka Guardiola]

On Sat, Sep 29, 2012 at 6:16 PM, Richard Miller <9fans@hamnavoe.com> wrote:
> So, has anyone had success using usbether to connect a plan 9 system
> to the outside world?  I am hoping someone can give me an encouraging
> report.  I'm a bit worried that it's a fundamental problem with the
> plan 9 usb architecture, which is basically synchronous - the host
> adapter driver in the kernel will poll a device for input only when
> the user-level driver process does a read.  This is ok for things like
> usbdisk which have an rpc-like protocol, but seems less well suited to
> things like ethernet and serial interfaces, where the equivalent
> non-usb kernel drivers use qio to read ahead into a queue of buffers
> until the user-level consuming process gets around to reading them.
>
> Would anyone like to share experiences or comments?
>
>

I had a similar problem with serial a while ago. The problem was that
it was taking too long doing too many system calls, doing big reads
and buffering may help you?

G.

Re: [9fans] usbether

[erik quanstrom]

> I had a similar problem with serial a while ago. The problem was that
> it was taking too long doing too many system calls, doing big reads
> and buffering may help you?

why doesn't the device do the buffering?  or is the problem
that usb/ether essentially an event loop of <read packet> <write packet>
and any delays in that pipeline accumulate?

- erik

Re: [9fans] usbether

[Richard Miller]

> why doesn't the device do the buffering?

By "device" do you mean the hardware usb/ether adapter?  It does some
buffering - you can configure it to "burst" as many ether input packets
as will fit in N 512-byte usb packets, in each usb input operation.
I've got N=37 because that's what the linux driver does; of course
there's no documentation to explain what the allowed range is or
what the consequences of varying it would be.  As far as I can see
there's no "overrun" flag for the device to tell me I've missed
reading some packets.

>  or is the problem
> that usb/ether essentially an event loop of <read packet> <write packet>
> and any delays in that pipeline accumulate?

Almost: it's a pipeline of one thread reading a buffer full of packets,
splitting it up, and sending a packet at a time to a second thread, which
writes them to the kernel packet ethernet interface, which stores them
in a Buf queue.

Re: [9fans] usbether

[Gorka Guardiola]

On Sat, Sep 29, 2012 at 8:01 PM, Richard Miller <9fans@hamnavoe.com> wrote:

> Almost: it's a pipeline of one thread reading a buffer full of packets,
> splitting it up, and sending a packet at a time to a second thread, which
> writes them to the kernel packet ethernet interface, which stores them
> in a Buf queue.
>

Taking a cursory glance it looks as if it just reading at most one
full packet at a time
(the size of the buffer is 2000), it may be if they are small you may
get more than one
in one read, but it may not be so, depends on the device and what is
set as maxpkt
which may be changed with usbctl if the device lets you

Making a bigger buffer (after making sure you can actually read more
bytes) and or decoupling
reader and processer by having a buffered channel so that the
reader can spend more time reading may help (it helped for serial).


G.

Re: [9fans] usbether

[Richard Miller]

> Taking a cursory glance it looks as if it just reading at most one
> full packet at a time
> (the size of the buffer is 2000)

That's the asix driver: the raspberry pi adapter has an SMSC LAN95xx,
see /n/sources/contrib/miller/usb/ether/smsc.c for my driver which
reads up to 37*512=18944 bytes at a time.

Re: [9fans] usbether

[erik quanstrom]

On Sat Sep 29 14:02:51 EDT 2012, 9fans@hamnavoe.com wrote:
> > why doesn't the device do the buffering?
>
> By "device" do you mean the hardware usb/ether adapter?  It does some

i ment #u itself.  sorry for being unclear.  actually, echoing
gorka, i think there's going to need to be enough threads/buffering
to decouple the packet pipeline.

it might be that the usb packet shuffler should be in the kernel, and
usbd should poke it from user land.  for a device without more
processing power than a cray 2, this could make a big difference.
i wrote a sd loopback device that turns a file into a sd device;
perhaps the same model would work for ether?  (sdloop is in 9atom.)

- erik

Re: [9fans] usbether

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 255 bytes --]

>the host adapter driver in the kernel will poll a device for input only
when
>the user-level driver process does a read

I'm fairly sure the original driver could post a batch of polling requests,
for just that reason,
but it was a long time ago.

[-- Attachment #2: Type: text/html, Size: 692 bytes --]

Re: [9fans] usbether

[Gorka Guardiola]

[-- Attachment #1: Type: text/plain, Size: 350 bytes --]

On 29/09/2012, at 23:25, Richard Miller <9fans@hamnavoe.com> wrote:


That's the asix driver: the raspberry pi adapter has an SMSC LAN95xx,
see /n/sources/contrib/miller/usb/ether/smsc.c for my driver which
reads up to 37*512=18944 bytes at a time.


And is it really doing it? I mean it asks for many bytes but how many
does it get?

G.

[-- Attachment #2: Type: text/html, Size: 746 bytes --]

Re: [9fans] usbether

[Richard Miller]

> And is it really doing it? I mean it asks for many bytes but how many
> does it get?

It often gets about 8k at a time (a complete 9p message).  I'll have
to add a check to see what the maximum is.

Re: [9fans] usbether

[erik quanstrom]

On Sat Sep 29 17:47:18 EDT 2012, 9fans@hamnavoe.com wrote:
> > And is it really doing it? I mean it asks for many bytes but how many
> > does it get?
>
> It often gets about 8k at a time (a complete 9p message).  I'll have
> to add a check to see what the maximum is.

of course the 64bit question is, which queue is dropping packets.
there are so many to choose from.

- erik

Re: [9fans] usbether

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 383 bytes --]

the qopen sets the queue limit to 8k, which seems a little low, unless it's
reset later.
the later unchecked qpass (with comment) will toss the data away, which is
a little drastic.

On 29 September 2012 22:46, Richard Miller <9fans@hamnavoe.com> wrote:

> It often gets about 8k at a time (a complete 9p message).  I'll have
> to add a check to see what the maximum is.
>

[-- Attachment #2: Type: text/html, Size: 660 bytes --]

Re: [9fans] usbether

[Richard Miller]

> the qopen sets the queue limit to 8k

Which qopen are you referring to?

Re: [9fans] usbether

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 141 bytes --]

the wrong one. never mind.

On 30 September 2012 11:24, Richard Miller <9fans@hamnavoe.com> wrote:

> Which qopen are you referring to?

[-- Attachment #2: Type: text/html, Size: 380 bytes --]

Re: [9fans] usbether

[Tristan]

> After running out of 9pi-specific things to debug, it occurred to me
> to try a usb ethernet dongle on an x86 plan 9 machine.  There I
> observed the same thing: so many dropped packets that the connection
> is unusable.

> So, has anyone had success using usbether to connect a plan 9 system
> to the outside world?  I am hoping someone can give me an encouraging
> report.

Not usb/ether, but my usb/wifi (which is very much based on it) written
for the Libertas (in OLPC) wireless worked a lot better than the any of
my usb ethernet dongles. I never have gotten around to figuring out why.

Good luck!
Tristan

-- 
All original matter is hereby placed immediately under the public domain.

Re: [9fans] usbether

[Richard Miller]

> of course the 64bit question is, which queue is dropping packets.
> there are so many to choose from.

Looks my initial guess was wrong.  I added some readahead to the kernel
usb driver and that didn't help at all.  Then I reduced the device's
buffering ("burst factor"), and increased the intermediate buffering
between input and output processes in the generic part of the usb/ether
driver.  That seems to have done the trick: no more dropped packets.

That's with the raspberry pi built-in usb ether.  My asix-based ethernet
dongle doesn't work any better on an x86 host; but at least it doesn't
work any worse.  Geoff has pushed the new usb/ether code onto sources.
Anybody who is actually using usb/ether - could you try the new version
and let us know if we've broken yours?