rpi4 shoots to 100% load and is unresponsive

rpi4 shoots to 100% load and is unresponsive

[Romano]

Hello all,

To get my Raspberry Pi 4 up and running, I used the rpi3 iso image from 9front.org to copy to an SD card, and booted from that. That worked fine, but the lack of memory-awareness made some compilations untenable: changeset 7408 only recognized ~500MB of the 4GB I had. So I pulled the latest commits from 9front and installed the kernel both at /arm64/ and /n/pidos/ by going to /sys/src/9/bcm64 and running 'mk install' followed by copying 9pi4 to /n/pidos: that allowed 9front to recognize the 4GB of memory on reboot. I also updated the firmware under /sys/src/boot/bcm/, and copied that over to /n/pidos, thinking that perhaps outdated firmware from the 7408 changeset build could be an issue.

However, both with and without the firmware updates, the load quickly escalates to 100% and the 'Delete' is ineffective after just a little bit of time doing anything. I have to hit the kill switch.  For instance, I was compiling git9, and sometimes it stops during compilation (mk all), other times it'll get a little further and stop during mk install. The mouse can still move around, but clicks don't perform any actions and the stats window just shows the load hitting max and staying there.

Did I miss a recent thread om the mailing list re: this? Can anyone think of a recent commit that might cause that behavior? Anything I can offer from the raspberry pi to help determine the possible cause? Could it be due to not building the compilers, or userland tools, or the kernel in a specific order or a specific way?

Re: [9front] rpi4 shoots to 100% load and is unresponsive

[hiro]

that kinda supports kenji's experiences, too. he was having general
stability issues that this reminds me of.

how's your firmware dated?

when i installed 9front on rpi4 i might have been lucky bec. of the
firmware that i picked - i didn't use the one from the release. that
one was not working at all for me.

but sadly i only ran it for a few minutes, and i didn't stress test anything.

Re: [9front] rpi4 shoots to 100% load and is unresponsive

[Romano]

Thank you, hiro.

The firmware I updated using rpi-eeprom just last week on raspbian, but my understanding of the boot process is that doesn't matter--what's on /n/pidos does. For /n/pidos, I used the files on the iso image on 9front and just an updated rpi4 kernel: that craps out with the 100% load. I also pulled the latest files using mk under /sys/src/boot/bcm, which appears to be tracking the master, not stable, branch in the raspberrypi firmware repo. I copied those files to /n/pidos and still had issues.

It's odd because clearly not everything is going awry, since 'stats' still shows updates.

Did I answer your question, or am I misunderstanding how the firmware works?

On May 7, 2020 7:25:57 PM UTC, hiro <23hiro@gmail.com> wrote:
>that kinda supports kenji's experiences, too. he was having general
>stability issues that this reminds me of.
>
>how's your firmware dated?
>
>when i installed 9front on rpi4 i might have been lucky bec. of the
>firmware that i picked - i didn't use the one from the release. that
>one was not working at all for me.
>
>but sadly i only ran it for a few minutes, and i didn't stress test
>anything.

Re: [9front] rpi4 shoots to 100% load and is unresponsive

[cinap_lenrek]

i could imagine that the following change might have an inmpact?

changeset:   7726:e2d79247a51c
user:        cinap_lenrek@felloff.net
date:        Sun Apr 26 19:54:46 2020 +0200
summary:     kernel: improve page reclaimation strategy and locking

i have not tested this change on raspi, but ran some stress tests
on a thinkpad, filling all memory and memory reclaimation worked
fine there.

in any case, you can try reverting this change and see if it has
any impact?

i do not have time now... maybe this weekend i can look into it
and try on raspberry pi.

--
cinap

Re: [9front] rpi4 shoots to 100% load and is unresponsive

[Romano]

Thanks , cinap.

I noticed that too, but I think my inexperience with hg screwed me, or maybe just too late when I was messing with it. I thought of trying to roll back to an earlier commit to see how things built, and I think I did 'hg update 7726', rather than the changset prior.

If I do 'hg update 7709', then 'mk' in /sys/src/9/bcm64, then copy the kernel over to /n/pidos, would that be heading in the right direction? Or should I revert the problematic commit in my working area, and see how that goes?


On May 7, 2020 7:54:52 PM UTC, cinap_lenrek@felloff.net wrote:
>i could imagine that the following change might have an inmpact?
>
>changeset:   7726:e2d79247a51c
>user:        cinap_lenrek@felloff.net
>date:        Sun Apr 26 19:54:46 2020 +0200
>summary:     kernel: improve page reclaimation strategy and locking
>
>i have not tested this change on raspi, but ran some stress tests
>on a thinkpad, filling all memory and memory reclaimation worked
>fine there.
>
>in any case, you can try reverting this change and see if it has
>any impact?
>
>i do not have time now... maybe this weekend i can look into it
>and try on raspberry pi.
>
>--
>cinap

Re: [9front] rpi4 shoots to 100% load and is unresponsive

[Romano]

I didn't know if image attachments would filter my message, so I've uploaded  some images to pasteboard.co and supllied the links below. When I did 'hg update 7709', rebuilt the kernel, and copied over to /n/pidos, I still get the breakage.. This is using thw latest firmware (mk clean, mk all) under /sys/src/boot/bcm. Note the jumps in load. The first jump is when things start locking up, but sometimes the mouse menu for stats still works for a bit (as shown). If I give it another minute, it'll be at 100%.

https://pasteboard.co/J7jOD5o.jpg
https://pasteboard.co/J7jOYoL.jpg
https://pasteboard.co/J7jPeRz.jpg


On May 7, 2020 8:06:51 PM UTC, Romano <unobe@cpan.org> wrote:
>Thanks , cinap.
>
>I noticed that too, but I think my inexperience with hg screwed me, or
>maybe just too late when I was messing with it. I thought of trying to
>roll back to an earlier commit to see how things built, and I think I
>did 'hg update 7726', rather than the changset prior.
>
>If I do 'hg update 7709', then 'mk' in /sys/src/9/bcm64, then copy the
>kernel over to /n/pidos, would that be heading in the right direction?
>Or should I revert the problematic commit in my working area, and see
>how that goes?
>
>
>On May 7, 2020 7:54:52 PM UTC, cinap_lenrek@felloff.net wrote:
>>i could imagine that the following change might have an inmpact?
>>
>>changeset:   7726:e2d79247a51c
>>user:        cinap_lenrek@felloff.net
>>date:        Sun Apr 26 19:54:46 2020 +0200
>>summary:     kernel: improve page reclaimation strategy and locking
>>
>>i have not tested this change on raspi, but ran some stress tests
>>on a thinkpad, filling all memory and memory reclaimation worked
>>fine there.
>>
>>in any case, you can try reverting this change and see if it has
>>any impact?
>>
>>i do not have time now... maybe this weekend i can look into it
>>and try on raspberry pi.
>>
>>--
>>cinap

Re: [9front] rpi4 shoots to 100% load and is unresponsive

[cinap_lenrek]

ok, could reproduce.

the issue appears to be that cmpswap() fails with negative values.

this broke with the new pid allocator.

void
castest(void)
{
	static long xxx = -1;

print("castest!\n");
	assert(xxx == -1);
	assert(cmpswap(&xxx, -1, 518));	// this fails
	assert(xxx == 518);
}

this could be a bug in the linker or with the assembly. i'll write
once i have a fix.

--
cinap

Re: [9front] rpi4 shoots to 100% load and is unresponsive

[cinap_lenrek]

ok, sign extension bug in cmpswap(). will fix.

--
cinap

Re: [9front] rpi4 shoots to 100% load and is unresponsive

[cinap_lenrek]

ok, fixed.

changeset:   7743:d5c12553e04b
user:        cinap_lenrek@felloff.net
date:        Thu May 07 23:27:27 2020 +0200
summary:     bcm64: fix kernels cmpswap() function

diff -r 07fadec60b37 -r d5c12553e04b sys/src/9/bcm64/l.s
--- a/sys/src/9/bcm64/l.s	Mon May 04 11:57:24 2020 +0200
+++ b/sys/src/9/bcm64/l.s	Thu May 07 23:27:27 2020 +0200
@@ -229,8 +229,8 @@
 
 TEXT cas(SB), 1, $-4
 TEXT cmpswap(SB), 1, $-4
-	MOVW	ov+8(FP), R1
-	MOVW	nv+16(FP), R2
+	MOVWU	ov+8(FP), R1
+	MOVWU	nv+16(FP), R2
 _cas1:
 	LDXRW	(R0), R3
 	CMP	R3, R1

--
cinap

Re: [9front] rpi4 shoots to 100% load and is unresponsive

[Romano]

Confirmed, not locking up anymore: thanks Cinap!

On May 7, 2020 9:29:57 PM UTC, cinap_lenrek@felloff.net wrote:
>ok, fixed.
>
>changeset:   7743:d5c12553e04b
>user:        cinap_lenrek@felloff.net
>date:        Thu May 07 23:27:27 2020 +0200
>summary:     bcm64: fix kernels cmpswap() function
>
>diff -r 07fadec60b37 -r d5c12553e04b sys/src/9/bcm64/l.s
>--- a/sys/src/9/bcm64/l.s	Mon May 04 11:57:24 2020 +0200
>+++ b/sys/src/9/bcm64/l.s	Thu May 07 23:27:27 2020 +0200
>@@ -229,8 +229,8 @@
> 
> TEXT cas(SB), 1, $-4
> TEXT cmpswap(SB), 1, $-4
>-	MOVW	ov+8(FP), R1
>-	MOVW	nv+16(FP), R2
>+	MOVWU	ov+8(FP), R1
>+	MOVWU	nv+16(FP), R2
> _cas1:
> 	LDXRW	(R0), R3
> 	CMP	R3, R1
>
>--
>cinap

Re: [9front] rpi4 shoots to 100% load and is unresponsive

[kokamoto]

It seems that my problem was also solved by this patch.
A copy of large sized file from 9front file server didn't make problem.

I'm now writing this mail from that rpi4 after file copy test.

Kenji

> ok, fixed.
> 
> changeset:   7743:d5c12553e04b
> user:        cinap_lenrek@felloff.net
> date:        Thu May 07 23:27:27 2020 +0200
> summary:     bcm64: fix kernels cmpswap() function
> 
> diff -r 07fadec60b37 -r d5c12553e04b sys/src/9/bcm64/l.s
> --- a/sys/src/9/bcm64/l.s	Mon May 04 11:57:24 2020 +0200
> +++ b/sys/src/9/bcm64/l.s	Thu May 07 23:27:27 2020 +0200
> @@ -229,8 +229,8 @@
>  
>  TEXT cas(SB), 1, $-4
>  TEXT cmpswap(SB), 1, $-4
> -	MOVW	ov+8(FP), R1
> -	MOVW	nv+16(FP), R2
> +	MOVWU	ov+8(FP), R1
> +	MOVWU	nv+16(FP), R2
>  _cas1:
>  	LDXRW	(R0), R3
>  	CMP	R3, R1
> 
> --
> cinap

Re: [9front] rpi4 shoots to 100% load and is unresponsive

[kokamoto]

> It seems that my problem was also solved by this patch.
> A copy of large sized file from 9front file server didn't make problem.

I compiled nsfb for arm64 on this rpi4 machine successfully.
Its compile speed is about twice of r3b...
Even the second generation of ci5 is much quicker.☺

Kenji