[9front] kernel crash in bhyve

[9front] kernel crash in bhyve

[Xiao-Yong Jin]

up to date with hg repo.
under FreeBSD bhyve.

*e820=1 0000000000000000 0000000000008000 1 0000000000008000 000000000000c000 1 000000000000c000 00000000000a0000 1 0000000000100000 0000000000800000 1 0000000000800000 0000000001000000 1 0000000001000000 000000003bfbe000 1 000000003bfbe000 000000003bfde000 1 000000003bfde000 000000003e9d8000 1 000000003e9d8000 000000003ea96000 1 000000003ea96000 000000003ea98000 3 000000003ea98000 000000003ea9a000 1 000000003ea9a000 000000003eaa9000 1 000000003eaa9000 000000003eb3f000 1 000000003eb3f000 000000003f9a9000 1 000000003f9a9000 000000003fb29000 1 000000003fb29000 000000003fb59000 1 000000003fb59000 000000003fb7d000 1 000000003fb7d000 000000003fb82000 3 000000003fb82000 000000003fb89000 4 000000003fb89000 000000003fb8d000 1 000000003fb8d000 000000003ffd0000 1 000000003ffd0000 000000003fff0000 1 000000003fff0000 0000000040000000
*acpi=0x3fb88014
*acpi=1
bootfile=9pc64
bootargs=local!/dev/sdF0/fscache -a tcp!*!564
nobootprompt=local!/dev/sdF0/fscache -a tcp!*!564
mouseport=ps2intellimouse
monitor=vesa
vgasize=1280x800x32
service=cpu
console=0
nora6=
boot

Plan 9
wrmsr to register 0x401(0) on vcpu 0
                                    panic: rampage: out of memory

dumpstack
ktrace /kernel/path 0xffffffff801173bd 0xffffffff8001fbf8 <<EOF
estackx ffffffff80020000
8001fb18=80117117 8001fb90=8024b679 8001fbc8=801173bd 8001fbf0=801173bd 
8001fbf8=8011711f 8001fc08=8016aba3 8001fcc0=80224b7a 8001fcd0=80224ff7 
8001fd08=802137f9 8001fd18=80213885 8001fd38=80247eed 8001fd58=8011339a 
8001fd88=80247f33 8001fd98=80114d6f 8001fdd8=80247f33 8001fde0=80114e53 
8001fe28=80115189 8001fe50=80113812 8001fe58=801134f8 8001fe70=8021378f 
8001fe78=80224b7a 8001fe80=80115b58 8001fea8=80114515 8001fed0=801e483e 
8001fef8=80113980 8001ff20=802136cc 8001ff28=80118d59 8001ff30=801e4aff 
8001ff58=801e638a 8001ffa0=8015c6c1 8001ffa8=801144c4 8001ffe0=80112de9 
8001ffe8=00000000 8001fff0=00000000 8001fff8=801101c9 
EOF
cpu0: exiting
Takes a licking and keeps on ticking...

[9front] Re: kernel crash in bhyve

[Xiao-Yong Jin]

> On Dec 9, 2020, at 12:18 PM, Xiao-Yong Jin <meta.jxy@gmail.com> wrote:
> 
> up to date with hg repo.
> under FreeBSD bhyve.
> 
> *e820=1 0000000000000000 0000000000008000 1 0000000000008000 000000000000c000 1 000000000000c000 00000000000a0000 1 0000000000100000 0000000000800000 1 0000000000800000 0000000001000000 1 0000000001000000 000000003bfbe000 1 000000003bfbe000 000000003bfde000 1 000000003bfde000 000000003e9d8000 1 000000003e9d8000 000000003ea96000 1 000000003ea96000 000000003ea98000 3 000000003ea98000 000000003ea9a000 1 000000003ea9a000 000000003eaa9000 1 000000003eaa9000 000000003eb3f000 1 000000003eb3f000 000000003f9a9000 1 000000003f9a9000 000000003fb29000 1 000000003fb29000 000000003fb59000 1 000000003fb59000 000000003fb7d000 1 000000003fb7d000 000000003fb82000 3 000000003fb82000 000000003fb89000 4 000000003fb89000 000000003fb8d000 1 000000003fb8d000 000000003ffd0000 1 000000003ffd0000 000000003fff0000 1 000000003fff0000 0000000040000000
> *acpi=0x3fb88014
> *acpi=1
> bootfile=9pc64
> bootargs=local!/dev/sdF0/fscache -a tcp!*!564
> nobootprompt=local!/dev/sdF0/fscache -a tcp!*!564
> mouseport=ps2intellimouse
> monitor=vesa
> vgasize=1280x800x32
> service=cpu
> console=0
> nora6=
> boot
> 
> Plan 9
> wrmsr to register 0x401(0) on vcpu 0
>                                    panic: rampage: out of memory
> 
> dumpstack
> ktrace /kernel/path 0xffffffff801173bd 0xffffffff8001fbf8 <<EOF
> estackx ffffffff80020000
> 8001fb18=80117117 8001fb90=8024b679 8001fbc8=801173bd 8001fbf0=801173bd 
> 8001fbf8=8011711f 8001fc08=8016aba3 8001fcc0=80224b7a 8001fcd0=80224ff7 
> 8001fd08=802137f9 8001fd18=80213885 8001fd38=80247eed 8001fd58=8011339a 
> 8001fd88=80247f33 8001fd98=80114d6f 8001fdd8=80247f33 8001fde0=80114e53 
> 8001fe28=80115189 8001fe50=80113812 8001fe58=801134f8 8001fe70=8021378f 
> 8001fe78=80224b7a 8001fe80=80115b58 8001fea8=80114515 8001fed0=801e483e 
> 8001fef8=80113980 8001ff20=802136cc 8001ff28=80118d59 8001ff30=801e4aff 
> 8001ff58=801e638a 8001ffa0=8015c6c1 8001ffa8=801144c4 8001ffe0=80112de9 
> 8001ffe8=00000000 8001fff0=00000000 8001fff8=801101c9 
> EOF
> cpu0: exiting
> Takes a licking and keeps on ticking...
> 

Here's from 9pc.

Plan 9
wrmsr to register 0x401(0) on vcpu 0
                                    panic: rampage: out of memory

dumpstack
ktrace /kernel/path f0107986 f0018d64 <<EOF
estackx f0019000
f0018d04=f010771b f0018d28=f0259c2e f0018d4c=f0107986 f0018d60=f0107986 
f0018d64=f010771f f0018d6c=f0174735 f0018dd4=f0259c2e f0018e18=f0259cc7 
f0018e2c=f0100d56 f0018e38=f02445de f0018e48=f0244aa2 f0018e4c=f0100e91 
f0018e64=f0233450 f0018e68=f0256bdf f0018e6c=f02334a7 f0018e94=f0103495 
f0018ecc=f010572b f0018ed8=f0232da4 f0018edc=f025464e f0018ee4=f02445de 
f0018eec=f0105bc4 f0018f10=f02333de f0018f20=f010589b f0018f34=f01048f7 
f0018f50=f01fd490 f0018f64=f0103a38 f0018f74=f0104100 f0018f84=f01fd77c 
f0018f9c=f01fefaa f0018fcc=f0166e94 f0018fe4=00529000 f0018fe8=f055f99c 
f0018fec=f010287b f0018ff0=f05c25ec f0018ff4=00000000 f0018ff8=f01001d1 
f0018ffc=00000000 
EOF
cpu0: exiting
Takes a licking and keeps on ticking...

Re: [9front] Re: kernel crash in bhyve

[ori]

> wrmsr to register 0x401(0) on vcpu 0
>                                     panic: rampage: out of memory
> 

How much memory did you give it?

Re: [9front] kernel crash in bhyve

[cinap_lenrek]

the wmsr 0x401 is harmless. its trying to initialize
the MCE registers. you can get rid of it by passing
*nomce= to the kernel.

for the panic...

is this a regression or the first time you tried it?

the kernel calls rampage() initially to get memory
for the page tables todo the mappings at KZERO.

if this is a regression, you might try the *nomtrr=
flag. maybe the mtrr's are giving strange cache
attributes which makes the kernel exclude it from
the memory map.

the relevant line is here:

	/* RAM needs to be writeback */
	mtrrexclude(MemRAM, "wb");

maybe you can insert a call of memmapdump() before
and after this line?

--
cinap

Re: [9front] kernel crash in bhyve

[Xiao-Yong Jin]

Here is the boot output from the current 9pc64 with *nomtrr=

Plan 9
wrmsr to register 0x401(0) on vcpu 0
                                    125 holes free
0x00024000 0x0009f000 503808
0x00100000 0x00110000 65536
0x005f7000 0x136ff000 319848448
320417792 bytes free
cpu0: 6785MHz GenuineIntel P6 (AX 000306A9 CX F69A6217 DX 9F8BFBFF)
LAPIC: fee00000 0xffffff00fee00000
ELCR: 0260
cpu0: lapic clock at 268MHz
#l0: virtio: 1000Mbps port 0x2080 irq 6 ea 52540000b008
1024M memory: 336M kernel data, 688M user, 1313M swap

Here is the boot output after adding two memmapdump().

Plan 9
wrmsr to register 0x401(0) on vcpu 0
                                    0000000000000000-0000000000020000 4
0000000000020000-000000000009fc00 2
000000000009fc00-00000000000a0000 4
00000000000a0000-00000000000c0000 80000001
00000000000c0000-00000000000d0000 4
00000000000d0000-00000000000f0000 1
00000000000f0000-0000000000100000 4
0000000000100000-0000000000110000 2
0000000000110000-00000000005f7000 4
00000000005f7000-0000000040000000 2
0000000040000000-0000000100000000 0
0000000000000000-00000000000a0000 4
00000000000a0000-00000000000c0000 80000001
00000000000c0000-00000000000d0000 4
00000000000d0000-00000000000f0000 1
00000000000f0000-0000000040000000 4
0000000040000000-0000000100000000 0
panic: rampage: out of memory

dumpstack
ktrace /kernel/path 0xffffffff801173c7 0xffffffff8001fbf8 <<EOF
estackx ffffffff80020000
8001fb18=80117121 8001fb50=8024ceca 8001fb90=8024b683 8001fbc8=801173c7 
8001fbf0=801173c7 8001fbf8=80117129 8001fc08=8016abad 8001fc68=8024b720 
8001fc78=80110db2 8001fc98=80110e16 8001fcb0=8024d664 8001fcb8=80110f11 
8001fcc0=80224b84 8001fcd0=80225001 8001fcd8=80119cb0 8001fcf0=80224ea8 
8001fcf8=8016a67d 8001fd08=80213803 8001fd10=801a4906 8001fd18=8021388f 
8001fd40=8016a7ef 8001fd58=8011339a 8001fd88=8016a8f7 8001fd98=80114d79 
8001fdb0=8016a956 8001fdd8=80247f3d 8001fde0=80114e5d 8001fe28=80115193 
8001fe50=80113812 8001fe58=801134f8 8001fe70=80213799 8001fe78=80224b84 
8001fe80=80115b62 8001fea8=8011451f 8001fed0=801e4848 8001fef8=80113980 
8001ff20=802136d6 8001ff28=80118d63 8001ff30=801e4b09 8001ff58=801e6394 
8001ffa0=8015c6cb 8001ffa8=801144ce 8001ffe0=80112de9 8001ffe8=00000000 
8001fff0=00000000 8001fff8=801101c9 
EOF
cpu0: exiting
Takes a licking and keeps on ticking...


Here is the boot output from a previous 9pc64.
Everything else is the same.

Plan 9
wrmsr to register 0x401(0) on vcpu 0
                                    125 holes free
0x00024000 0x0009f000 503808
0x00100000 0x00110000 65536
0x005f6000 0x136ff000 319852544
320421888 bytes free
cpu0: 6781MHz GenuineIntel P6 (AX 000306A9 CX F69A6217 DX 9F8BFBFF)
LAPIC: fee00000 0xffffff00fee00000
ELCR: 0260
cpu0: lapic clock at 268MHz
#l0: virtio: 1000Mbps port 0x2080 irq 6 ea 52540000b008
1024M memory: 336M kernel data, 688M user, 1313M swap



> On Dec 9, 2020, at 1:05 PM, cinap_lenrek@felloff.net wrote:
> 
> the wmsr 0x401 is harmless. its trying to initialize
> the MCE registers. you can get rid of it by passing
> *nomce= to the kernel.
> 
> for the panic...
> 
> is this a regression or the first time you tried it?
> 
> the kernel calls rampage() initially to get memory
> for the page tables todo the mappings at KZERO.
> 
> if this is a regression, you might try the *nomtrr=
> flag. maybe the mtrr's are giving strange cache
> attributes which makes the kernel exclude it from
> the memory map.
> 
> the relevant line is here:
> 
> 	/* RAM needs to be writeback */
> 	mtrrexclude(MemRAM, "wb");
> 
> maybe you can insert a call of memmapdump() before
> and after this line?
> 
> --
> cinap

Re: [9front] kernel crash in bhyve

[cinap_lenrek]

yeah, its like i said. after mtrrexclude, theres no
more ram left.

so i guess the mtrr's are probably programmed to
everything uncached.

can you try to boot with *nomtrr= please?

the next step is to see if we can detect this
situation automatically.

--
cinap

Re: [9front] kernel crash in bhyve

[Xiao-Yong Jin]

> On Dec 9, 2020, at 2:52 PM, cinap_lenrek@felloff.net wrote:
> 
> yeah, its like i said. after mtrrexclude, theres no
> more ram left.
> 
> so i guess the mtrr's are probably programmed to
> everything uncached.
> 
> can you try to boot with *nomtrr= please?

The first chunk of the output in my last email was with *nomtrr=
and it booted fine.

Plan 9
wrmsr to register 0x401(0) on vcpu 0
                                   125 holes free
0x00024000 0x0009f000 503808
0x00100000 0x00110000 65536
0x005f7000 0x136ff000 319848448
320417792 bytes free
cpu0: 6785MHz GenuineIntel P6 (AX 000306A9 CX F69A6217 DX 9F8BFBFF)
LAPIC: fee00000 0xffffff00fee00000
ELCR: 0260
cpu0: lapic clock at 268MHz
#l0: virtio: 1000Mbps port 0x2080 irq 6 ea 52540000b008
1024M memory: 336M kernel data, 688M user, 1313M swap

Re: [9front] kernel crash in bhyve

[cinap_lenrek]

ah! yes... i missed it... scrolled too far. thank you!

can you compile and run this program without
any arguments?

http://felloff.net/usr/cinap_lenrek/mtrr.tgz

--
cinap

Re: [9front] kernel crash in bhyve

[Xiao-Yong Jin]

> On Dec 9, 2020, at 4:04 PM, cinap_lenrek@felloff.net wrote:
> 
> ah! yes... i missed it... scrolled too far. thank you!
> 
> can you compile and run this program without
> any arguments?
> 
> http://felloff.net/usr/cinap_lenrek/mtrr.tgz

rdmsr to register 0xc0010010 on vcpu 0
                                      range                0       1000000000 (      1000000000) uc
cache 0x0000000000000000     68719476736 uc

Re: [9front] kernel crash in bhyve

[cinap_lenrek]

The following diff should fix it.

--
cinap

diff -r dcd57f411618 sys/src/9/pc/memory.c
--- a/sys/src/9/pc/memory.c	Wed Dec 09 01:05:14 2020 +0100
+++ b/sys/src/9/pc/memory.c	Thu Dec 10 00:16:49 2020 +0100
@@ -383,8 +383,16 @@
 		}
 	}
 
-	/* RAM needs to be writeback */
-	mtrrexclude(MemRAM, "wb");
+	/*
+	 * Make sure RAM is set to writeback,
+	 * but do a sanity check first checking
+	 * that the kernel text is writeback.
+	 * This is needed as some emulators (bhyve)
+	 * set everything to uncached.
+	 */
+	s = mtrrattr(KTZERO, nil);
+	if(s != nil && strcmp(s, "wb") == 0)
+		mtrrexclude(MemRAM, "wb");
 
 	for(base = memmapnext(-1, MemRAM); base != -1; base = memmapnext(base, MemRAM)){
 		size = memmapsize(base, BY2PG) & ~(BY2PG-1);

Re: [9front] kernel crash in bhyve

[Xiao-Yong Jin]

> On Dec 9, 2020, at 5:18 PM, cinap_lenrek@felloff.net wrote:
> 
> The following diff should fix it.

yes, thanks

Re: [9front] kernel crash in bhyve

[cinap_lenrek]

There was a mistake, it should have been PADDR(KTZERO).
Tho in your case it doesnt matter, as every address
returns uc, so the work around works.

I'll push the fixed version.

Thank you!

--
cinap

Re: [9front] kernel crash in bhyve

[cinap_lenrek]

Btw, would it be possible for you to file a bug
with bhyve?

I do not have a bsd installation nor do i want to
subscribe to their mailinglist right now.

The issue is that the mtrr values they give us
are basically useless.

Qemu, vmware and everyone else gets this right.

The simplest fix might be that they just change
cpuid to tell the guest that mtrr is not supported.

--
cinap

Re: [9front] kernel crash in bhyve

[Xiao-Yong Jin]

> On Dec 10, 2020, at 5:30 AM, cinap_lenrek@felloff.net wrote:
> 
> Btw, would it be possible for you to file a bug
> with bhyve?
> 
> I do not have a bsd installation nor do i want to
> subscribe to their mailinglist right now.
> 
> The issue is that the mtrr values they give us
> are basically useless.
> 
> Qemu, vmware and everyone else gets this right.
> 
> The simplest fix might be that they just change
> cpuid to tell the guest that mtrr is not supported.
> 
> --
> cinap

I'm not sure I understand the issue completely.

I was using a specific debug switch '-w' in bhyve.

                 -w          Ignore accesses to unimplemented Model Specific
                             Registers (MSRs).  This is intended for debug
                             purposes.

Because previously, 9front wouldn't boot without this flag.
Now I guess you added more error checking during the boot,
and 9front boots fine without this '-w'.  However, without
this '-w' to bhyve, the MTRR code you gave me produces an
error,

rdmsr to register 0xc0010010 on vcpu 0
                                      rdmsr c0010010: bad arg in system call
range                0       1000000000 (      1000000000) uc
cache 0x0000000000000000     68719476736 uc

It looks like '#P/msr' just cannot be read.

; cat '#P/msr'
cat: error reading #P/msr: bad arg in system call

Re: [9front] kernel crash in bhyve

[Nick Owens]

you cannot just read #P/msr at offset 0. the offset is significant. io
-M can be used to read an MSR at a specific offset.

On Thu, Dec 10, 2020 at 12:37 PM Xiao-Yong Jin <meta.jxy@gmail.com> wrote:
>
> > On Dec 10, 2020, at 5:30 AM, cinap_lenrek@felloff.net wrote:
> >
> > Btw, would it be possible for you to file a bug
> > with bhyve?
> >
> > I do not have a bsd installation nor do i want to
> > subscribe to their mailinglist right now.
> >
> > The issue is that the mtrr values they give us
> > are basically useless.
> >
> > Qemu, vmware and everyone else gets this right.
> >
> > The simplest fix might be that they just change
> > cpuid to tell the guest that mtrr is not supported.
> >
> > --
> > cinap
>
> I'm not sure I understand the issue completely.
>
> I was using a specific debug switch '-w' in bhyve.
>
>                  -w          Ignore accesses to unimplemented Model Specific
>                              Registers (MSRs).  This is intended for debug
>                              purposes.
>
> Because previously, 9front wouldn't boot without this flag.
> Now I guess you added more error checking during the boot,
> and 9front boots fine without this '-w'.  However, without
> this '-w' to bhyve, the MTRR code you gave me produces an
> error,
>
> rdmsr to register 0xc0010010 on vcpu 0
>                                       rdmsr c0010010: bad arg in system call
> range                0       1000000000 (      1000000000) uc
> cache 0x0000000000000000     68719476736 uc
>
> It looks like '#P/msr' just cannot be read.
>
> ; cat '#P/msr'
> cat: error reading #P/msr: bad arg in system call
>

Re: [9front] kernel crash in bhyve

[Xiao-Yong Jin]

> On Dec 10, 2020, at 5:12 PM, Nick Owens <mischief@offblast.org> wrote:
> 
> you cannot just read #P/msr at offset 0. the offset is significant. io
> -M can be used to read an MSR at a specific offset.

That makes sense.  The registers are 0 with the correct offset.

; for(r in 0x2ff 0x0fe) io -r -M $r
0x0
0x0

In fact, the MTRR registers are all 0:

https://github.com/freebsd/freebsd/blob/e8142e44905fa62147ba10703e7ce2c5464fb2b7/sys/amd64/vmm/intel/vmx_msr.c#L426-L432

> On Dec 10, 2020, at 7:12 PM, ori@eigenstate.org wrote:
> 
> I can try to reproduce and maybe file a bug this
> weekend.

Please do.  Thanks.

Re: [9front] kernel crash in bhyve

[cinap_lenrek]

> That makes sense.  The registers are 0 with the correct offset.

> ; for(r in 0x2ff 0x0fe) io -r -M $r
> 0x0
> 0x0


Yes, that explains it. This also means MTRR's are disabled in MTRRdeftype register.

The processor manual says:

E (MTRRs enabled) flag, bit 11:
MTRRs are enabled when set; all MTRRs are disabled when clear,
and the UC memory type is applied to all of physical memory.

However, the MTRRCap is also 0, basically meaning we have 0
variable length MTRR's and no fixed MTRR support.

I can use that setting so conclude that MTRR's are not supported
instead of relying on CPUID and the sanity check in memory.c.

--
cinap

Re: [9front] kernel crash in bhyve

[ori]

Quoth cinap_lenrek@felloff.net:
> Btw, would it be possible for you to file a bug
> with bhyve?
> 
> I do not have a bsd installation nor do i want to
> subscribe to their mailinglist right now.

I can try to reproduce and maybe file a bug this
weekend.