[9fans] Plan9 on Pi 3B+

[9fans] Plan9 on Pi 3B+

[Brian L. Stuart]

Has anyone tried Plan 9 on the new Pi 3B+?  I've
run into something that confuses me a bit.  First,
it seems you need the new version of start_cd.elf
to bring up the 3B+.  However, with that, the kernel
throws a lock loop error.  In tracking down the loop,
it happens in startcpus() in archbcm2.c.  The code
here looks like:

    for(i = 0; i < ncpu; i++)
        lock(&startlock[i]);
    cachedwbse(startlock, sizeof startlock);
    for(i = 1; i < ncpu; i++) {
        if(startcpu(i) < 0)
            return i;
        lock(&startlock[i]);
        unlock(&startlock[i]);
    }

So we grab a lock on all the CPUs, then drop into
a second loop where we start CPUs 1 to n.  But
in that loop we grab the lock again and then
immediately unlock it.  This is where the lock
loop happens.  Everything seems to continue
to be happy on both a 2 and a 3B+ if I comment
out the lock in the second loop.  But what is the
rationale for the lock in the second loop?  Was
there a reason for putting it there, or was it an
oversight that wasn't exposed until the new boot
code?

Thanks,
BLS

Re: [9fans] Plan9 on Pi 3B+

[Erik Quanstrom]

[-- Attachment #1: Type: text/html, Size: 155 bytes --]

Re: [9fans] Plan9 on Pi 3B+

[Richard Miller]

> But what is the
> rationale for the lock in the second loop?

Sorry for the confusing code.  The startlocks are locked by
cpu0 before booting the secondary cores.  Each core unlocks
its own startlock when it starts.  Cpu0 blocks on each startlock
waiting for the corresponding core to unlock it, as a way of
serialising the booting of the secondary cores (and detecting
that they have started).

If for some reason the secondary cpus don't start up, the
result is a lock loop.  I really should have used canlock()
instead, to expose a more helpful error message.  But it's
a "should never happen" condition.

It's happening with the new start_cd.elf because the Pi
Foundation made a little change in the firmware which makes
secondary cores wait with a WFE when booting, so the OS
now has to do a SEV to wake them up.

I'm just about to release an update to the bcm port for
the 3B+, including drivers for the new "gigabit" ethernet
(not really, because it's squeezed through a slow usb2 adapter)
and the updated wifi chip.  But in the meantime the following
should let you boot any pi2 or pi3 with the latest start_cd.elf:

--- /sys/src/9/bcm/archbcm2.c	Fri Mar 16 17:08:21 2018
***************
*** 167,172 ****
--- 167,174 ----
  	if(mb->clr[cpu].startcpu)
  		return -1;
  	mb->set[cpu].startcpu = PADDR(cpureset);
+ 	coherence();
+ 	sev();
  	for(i = 0; i < 1000; i++)
  		if(mb->clr[cpu].startcpu == 0)
  			return 0;
***************
*** 196,202 ****
  int
  startcpus(uint ncpu)
  {
! 	int i;

  	for(i = 0; i < ncpu; i++)
  		lock(&startlock[i]);
--- 198,204 ----
  int
  startcpus(uint ncpu)
  {
! 	int i, timeout;

  	for(i = 0; i < ncpu; i++)
  		lock(&startlock[i]);
***************
*** 204,210 ****
  	for(i = 1; i < ncpu; i++){
  		if(startcpu(i) < 0)
  			return i;
! 		lock(&startlock[i]);
  		unlock(&startlock[i]);
  	}
  	return ncpu;
--- 206,215 ----
  	for(i = 1; i < ncpu; i++){
  		if(startcpu(i) < 0)
  			return i;
! 		timeout = 10000000;
! 		while(!canlock(&startlock[i]))
! 			if(--timeout == 0)
! 				return i;
  		unlock(&startlock[i]);
  	}
  	return ncpu;

--- /sys/src/9/bcm/armv7.s	Fri Mar 16 16:47:18 2018
***************
*** 17,22 ****
--- 17,23 ----
  #define WFI	WORD	$0xe320f003	/* wait for interrupt */
  #define WFI_EQ	WORD	$0x0320f003	/* wait for interrupt if eq */
  #define ERET	WORD	$0xe160006e	/* exception return from HYP */
+ #define SEV	WORD	$0xe320f004	/* send event */

  /* tas/cas strex debugging limits; started at 10000 */
  #define MAXSC 1000000
***************
*** 386,391 ****
--- 387,396 ----

  TEXT coherence(SB), $-4
  	BARRIERS
+ 	RET
+
+ TEXT sev(SB), $-4
+ 	SEV
  	RET

  /*

Re: [9fans] Plan9 on Pi 3B+

[Skip Tavakkolian]

[-- Attachment #1: Type: text/plain, Size: 3523 bytes --]

Thanks!   I got one recently and have been meaning to ask about his.

Running Linux, 3B+ seems perceptibly faster than 3B.

On Tue, Apr 3, 2018 at 2:05 PM, Richard Miller <9fans@hamnavoe.com> wrote:

> > But what is the
> > rationale for the lock in the second loop?
>
> Sorry for the confusing code.  The startlocks are locked by
> cpu0 before booting the secondary cores.  Each core unlocks
> its own startlock when it starts.  Cpu0 blocks on each startlock
> waiting for the corresponding core to unlock it, as a way of
> serialising the booting of the secondary cores (and detecting
> that they have started).
>
> If for some reason the secondary cpus don't start up, the
> result is a lock loop.  I really should have used canlock()
> instead, to expose a more helpful error message.  But it's
> a "should never happen" condition.
>
> It's happening with the new start_cd.elf because the Pi
> Foundation made a little change in the firmware which makes
> secondary cores wait with a WFE when booting, so the OS
> now has to do a SEV to wake them up.
>
> I'm just about to release an update to the bcm port for
> the 3B+, including drivers for the new "gigabit" ethernet
> (not really, because it's squeezed through a slow usb2 adapter)
> and the updated wifi chip.  But in the meantime the following
> should let you boot any pi2 or pi3 with the latest start_cd.elf:
>
> --- /sys/src/9/bcm/archbcm2.c   Fri Mar 16 17:08:21 2018
> ***************
> *** 167,172 ****
> --- 167,174 ----
>         if(mb->clr[cpu].startcpu)
>                 return -1;
>         mb->set[cpu].startcpu = PADDR(cpureset);
> +       coherence();
> +       sev();
>         for(i = 0; i < 1000; i++)
>                 if(mb->clr[cpu].startcpu == 0)
>                         return 0;
> ***************
> *** 196,202 ****
>   int
>   startcpus(uint ncpu)
>   {
> !       int i;
>
>         for(i = 0; i < ncpu; i++)
>                 lock(&startlock[i]);
> --- 198,204 ----
>   int
>   startcpus(uint ncpu)
>   {
> !       int i, timeout;
>
>         for(i = 0; i < ncpu; i++)
>                 lock(&startlock[i]);
> ***************
> *** 204,210 ****
>         for(i = 1; i < ncpu; i++){
>                 if(startcpu(i) < 0)
>                         return i;
> !               lock(&startlock[i]);
>                 unlock(&startlock[i]);
>         }
>         return ncpu;
> --- 206,215 ----
>         for(i = 1; i < ncpu; i++){
>                 if(startcpu(i) < 0)
>                         return i;
> !               timeout = 10000000;
> !               while(!canlock(&startlock[i]))
> !                       if(--timeout == 0)
> !                               return i;
>                 unlock(&startlock[i]);
>         }
>         return ncpu;
>
> --- /sys/src/9/bcm/armv7.s      Fri Mar 16 16:47:18 2018
> ***************
> *** 17,22 ****
> --- 17,23 ----
>   #define WFI   WORD    $0xe320f003     /* wait for interrupt */
>   #define WFI_EQ        WORD    $0x0320f003     /* wait for interrupt if
> eq */
>   #define ERET  WORD    $0xe160006e     /* exception return from HYP */
> + #define SEV   WORD    $0xe320f004     /* send event */
>
>   /* tas/cas strex debugging limits; started at 10000 */
>   #define MAXSC 1000000
> ***************
> *** 386,391 ****
> --- 387,396 ----
>
>   TEXT coherence(SB), $-4
>         BARRIERS
> +       RET
> +
> + TEXT sev(SB), $-4
> +       SEV
>         RET
>
>   /*
>
>
>

[-- Attachment #2: Type: text/html, Size: 4540 bytes --]

Re: [9fans] Plan9 on Pi 3B+

[Richard Miller]

Updated kernel for the 3B+ is now on sources.  Also contains
some performance tweaks for ethernet and usb which should
apply to all Pi models.

Kernel source in /n/sources/contrib/miller/9/bcm
USB ether driver in /n/sources/patch/usb-ether-lan78xx
Compiled kernels in /n/sources/contrib/miller/9pi2 and 9pi

Note that the usb-ether-lan78xx patch subsumes and supersedes
usbether-rpi and usb-ether-cdc, so you may need to revert
those before applying the new patch.

Re: [9fans] Plan9 on Pi 3B+

[Richard Miller]

> Running Linux, 3B+ seems perceptibly faster than 3B.

1400Mhz vs 1200Mhz might be perceptible, and the improved
heat management will help if your CPU frequency was being
temperature-throttled.

Ethernet bandwidth for the 3B+ on linux is claimed to be
about 300Mbit/s.  With Plan 9, I'm seeing a limit of
just over 200Mbit/s.  Not gigabit, but still it's a 2x
improvement over earlier models.

Note: you need to use Cat 6 cable to get more than 100MB/s
speed.  I discovered this the hard way.  The orange LED
on the RJ45 socket goes on if the link is in gigabit mode.

Re: [9fans] Plan9 on Pi 3B+

[Steve Simon]

hi,

great work richard, i hope to try this soon. the network performance is the biggest draw for me.

do you any feeling why plan9 sees only 200Mbps ether?

is it the stack design, how plan9 accesses the hardware? or maybe the 300Mbps quoted is only when using jumbo frames?

-Steve


On 5 Apr 2018, at 16:03, Richard Miller <9fans@hamnavoe.com> wrote:

>> Running Linux, 3B+ seems perceptibly faster than 3B.
> 
> 1400Mhz vs 1200Mhz might be perceptible, and the improved
> heat management will help if your CPU frequency was being
> temperature-throttled.
> 
> Ethernet bandwidth for the 3B+ on linux is claimed to be
> about 300Mbit/s.  With Plan 9, I'm seeing a limit of
> just over 200Mbit/s.  Not gigabit, but still it's a 2x
> improvement over earlier models.
> 
> Note: you need to use Cat 6 cable to get more than 100MB/s
> speed.  I discovered this the hard way.  The orange LED
> on the RJ45 socket goes on if the link is in gigabit mode.
> 

Re: [9fans] Plan9 on Pi 3B+

[Bakul Shah]

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

The higher number seems to be from the netperf benchmark. See
https://www.phoronix.com/scan.php?page=article&item=raspberrypi-3-bplus&num=3


> On Apr 5, 2018, at 10:02 AM, Steve Simon <steve@quintile.net> wrote:
> 
> 
> hi,
> 
> great work richard, i hope to try this soon. the network performance is the biggest draw for me.
> 
> do you any feeling why plan9 sees only 200Mbps ether?
> 
> is it the stack design, how plan9 accesses the hardware? or maybe the 300Mbps quoted is only when using jumbo frames?
> 
> -Steve
> 
> 
> On 5 Apr 2018, at 16:03, Richard Miller <9fans@hamnavoe.com> wrote:
> 
>>> Running Linux, 3B+ seems perceptibly faster than 3B.
>> 
>> 1400Mhz vs 1200Mhz might be perceptible, and the improved
>> heat management will help if your CPU frequency was being
>> temperature-throttled.
>> 
>> Ethernet bandwidth for the 3B+ on linux is claimed to be
>> about 300Mbit/s.  With Plan 9, I'm seeing a limit of
>> just over 200Mbit/s.  Not gigabit, but still it's a 2x
>> improvement over earlier models.
>> 
>> Note: you need to use Cat 6 cable to get more than 100MB/s
>> speed.  I discovered this the hard way.  The orange LED
>> on the RJ45 socket goes on if the link is in gigabit mode.
>> 
> 
> 

[-- Attachment #2: Type: text/html, Size: 2676 bytes --]

Re: [9fans] Plan9 on Pi 3B+

[hiro]

"The Python performance with this new Raspberry Pi 3 is now in line
with the Pine A64.
While the PHP performance is between the Pine A64 and ODROID-C2"

from this test it sounds they didn't care much about typical
scientific supercomputing...
and their udp throughput test is slower than the tcp one?!

either they are doing something completely braindamaged or the
hardware is still just garbage.

Re: [9fans] Plan9 on Pi 3B+

[Rui Carmo]

They’re still under the same baseline SoC constraints (plus the USB bus). 

Biggest constraint for me is the lack of RAM, though.

R.

> On 5 Apr 2018, at 23:13, hiro <23hiro@gmail.com> wrote:
> 
> "The Python performance with this new Raspberry Pi 3 is now in line
> with the Pine A64.
> While the PHP performance is between the Pine A64 and ODROID-C2"
> 
> from this test it sounds they didn't care much about typical
> scientific supercomputing...
> and their udp throughput test is slower than the tcp one?!
> 
> either they are doing something completely braindamaged or the
> hardware is still just garbage.
> 

Re: [9fans] Plan9 on Pi 3B+

[Richard Miller]

> do you any feeling why plan9 sees only 200Mbps ether?
>
> is it the stack design, how plan9 accesses the hardware? or maybe the 300Mbps quoted is only when using jumbo frames?

Jumbo frames might help if the bottleneck was in the ip stack,
but I think the limiting factor is the Plan 9 usb infrastructure,
which is written more for simplicity than for maximum speed.