Re: [developer] Re: illumos/uefi-edk2 update 2023-11

[Stephan Althaus <Stephan.Althaus@Duedinghausen.eu>]

On 10/29/24 16:25, Stephan Althaus wrote:
> Hi All!
>
> I thought i'd try out a GPU passthrough on OI to a Windows vm with 
> bhyve again.
>
> This time it is an AMD RX 6400, and i verified the System settings 
> with FreeBSD 4.1, there the GPU is working, so the BIOS settings and 
> the GPU-ROM are ok.
>
> The VM startscripts and data are within a NVME zpool, so that i can 
> switch to OI easily and have the same with FrreBSD and OI.
>
> When i enable immu-intrmap-enable="true"; in 
> /platform/i86pc/kernel/drv/rootnex.conf
> this leads to error messages when i start the bhyve test script with 
> messages
> APIC ERROR Interrupt on CPU 0. Status 0 = 0 Status 1 = 40.
> on the console and the system is not responsive anymore..
>
> When i leave it toimmu-intrmap-enable=" false", the Windows vm starts 
> and the GPU has error 43 within the device-manager.
>
> What can i do to see where the APIC error comes from ? some dtrace 
> script maybe ?
>
> Any help is appreciated.
>
> Regards,
> Stephan
>
> P.S. The test command:
>
> /usr/sbin/bhyve -U bd60bf59-0ed6-61a1-f0ed-b17e99ada165 -A -H \
> -B 
> 1,manufacturer=OpenIndiana,product=OpenIndianaHVM,version=1.0,serial=69a18a85-c1df-c266-f620-a99a632d3c11,sku=001,family=VirtualMachine 
> \
> -c 4 -m 16G -w \
> -s 31:0,lpc -l bootrom,/home2/vm/BHYVE_UEFI.fd \
> -s 0,hostbridge,model=i440fx  \
> -s 4:0,nvme,/dev/zvol/rdsk/home2/vm/win10vol \
> -s 6:0,virtio-net-viona,vnic0 \
> -s 7:0,ahci-cd,/home2/vm/virtio-win-0.1.240.iso \
> -s 30:0,fbuf,tcp=0.0.0.0:5905,w=1920,h=1080 \
> -s 30:1,xhci,tablet \
> -S \
> -s 8:0,passthru,/dev/ppt0,rom=/home2/vm/Navi24.rom \
> -s 8:1,passthru,/dev/ppt1 \
> win10
>
> $ pptadm
> DEV        VENDOR DEVICE PATH
> /dev/ppt0  1002   743f 
> /pci@7a,0/pci8086,6f08@3/pci1002,1478@0/pci1002,1479@0/display@0
> /dev/ppt1  1002   ab28 
> /pci@7a,0/pci8086,6f08@3/pci1002,1478@0/pci1002,1479@0/pci1002,ab28@0,1
>
>
> On 2/16/24 10:29, Stephan Althaus wrote:
>> Hello!
>>
>> Small update on this..
>>
>> The goal i want to achieve is to have a GPU passthrough on OI to a 
>> Windows vm with bhyve...
>>
>> I now tested linux/proxmox (kvm)  and FreeBSD 14 (bhyve) with 
>> success, so i can assume the Hardware and the BIOS settings are ok.
>>
>> FreeBSD has this version of the edk2 image:
>>
>> $ pkg search edk2
>> edk2-bhyve-g202308_4           EDK2 Firmware for bhyve
>>
>> .. so version 202311 seems not to be needed (?)
>>
>> What i see if i try it on OI, is that
>>
>> immu-intrmap-enable="true";
>>
>> in
>>
>> /platform/i86pc/kernel/drv/rootnex.conf
>>
>> leads to error messages when i start the bhyve test script with messages
>> APIC ERROR Interrupt on CPU 0. Status 0 = 0 Status 1 = 40.
>> on the console and the system is not responsive anymore..
>>
>>
>> Do i _need_ immu-intrmap-enable="true"; ?
>>
>>
>> I currently have these values set:
>>
>> immu-enable="true";
>> #immu-intrmap-enable="true";
>> immu-blacklist=
>>                 "SMBIOS","Supermicro","X10DRI-T",""
>>                 ;
>> immu-dvma-mapping="xlate";
>> immu-dvma-unity-drivers="igb","ahci","ixgbe","ehci","xhci","nvme";
>> immu-dvma-xlate-drivers="mpt";
>>
>> Is anyone around with a success story who has been there and done that ?
>>
>>
>> i want to passtrough this:
>>
>> $ pptadm list -o all
>> DEV        VENDOR DEVICE SUBVENDOR SUBDEVICE REV 
>> PATH                                              LABEL
>> /dev/ppt0  1002   743f   1da2      e458      c7 
>> /pci@7a,0/pci8086,6f08@3/pci1002,1478@0/pci1002,1479@0/display@0 
>> Advanced Micro Devices, Inc. [AMD/ATI] Navi 24 [Radeon RX 6400/6500 
>> XT/6500M]
>> /dev/ppt1  1002   ab28   1002      ab28      0 
>> /pci@7a,0/pci8086,6f08@3/pci1002,1478@0/pci1002,1479@0/pci1002,ab28@0,1 
>> Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio 
>> Controller
>>
>> ... using this test script:
>>
>> $ /usr/sbin/bhyve -U bd60bf59-0ed6-61a1-f0ed-b17e99ada165 -A -H \
>> -B 
>> 1,manufacturer=OpenIndiana,product=OpenIndianaHVM,version=1.0,serial=69a18a85-c1df-c266-f620-a99a632d3c11,sku=001,family=VirtualMachine 
>> \
>> -c 4 -m 16G -w \
>> -s 31:0,lpc -l bootrom,/usr/share/bhyve/firmware/BHYVE.fd,fwcfg=qemu \
>> -s 0,hostbridge,model=i440fx  \
>> -s 4:0,nvme,/dev/zvol/rdsk/home2/vm/win10vol \
>> -s 6:0,virtio-net-viona,vnic0 \
>> -s 30:0,fbuf,vga=off,tcp=0.0.0.0:5905 \
>> -s 30:1,xhci,tablet \
>> -S -s 8:0,passthru,/dev/ppt0,rom=/home2/vm/Navi24.rom \
>> -s 8:1,passthru,/dev/ppt1 \
>> win10
>>
>> The GPU can be seen within the vm (error 43), it seems the card 
>> doesn't get initialized..
>>
>> Any hints are welcome..
>>
>>
>> Regards,
>>
>> Stephan
>>
>>
>>
>> ------------------------------------------
>> illumos: illumos-developer
>> Permalink: 
>> https://illumos.topicbox.com/groups/developer/T0e71dad0bc17329a-M635020990ad99de8499b28ad
>> Delivery options: 
>> https://illumos.topicbox.com/groups/developer/subscription
>
>
>
> ------------------------------------------
> illumos: illumos-developer
> Permalink: 
> https://illumos.topicbox.com/groups/developer/T0e71dad0bc17329a-M54c5f6bf883f52445dbaedfd
> Delivery options: 
> https://illumos.topicbox.com/groups/developer/subscription

Hi!

I tested the move of the vm data to a SATA drive, removed the nvme 
drive, but the APIC Interrupt errors persist.

Which additional drivers of the 168 used
$ modinfo |awk '{print $6}'|sort -u|wc
      168     168    1146
  do i have to blacklist in rootnex.conf ??

Anyone BTDT ?

Here's some messages-output that may be of interest:

user@smhost:~$ grep DMAR /var/adm/messages |grep "8:5
 > ^C
user@smhost:~$ grep DMAR /var/adm/messages |grep "8:5"
Oct 30 08:52:24 smhost acpica: [ID 137418 kern.notice] ACPI: DMAR 
0x0000000079F05458 000138 (v01 ALASKA A M I    00000001 INTL 20091013)
Oct 30 08:52:24 smhost genunix: [ID 958256 kern.info] Processing DMAR 
ACPI table
Oct 30 08:52:24 smhost genunix: [ID 680212 kern.notice] NOTICE: DMAR 
ACPI table: skipping unsupported unit type ATSR
Oct 30 08:52:24 smhost genunix: [ID 533727 kern.notice] NOTICE: DMAR 
ACPI table: skipping unsupported unit type RHSA
user@smhost:~$ grep iomm /var/adm/messages |grep "8:5"
Oct 30 08:52:24 smhost iommulib: [ID 321598 kern.info] NOTICE: 
iommulib_nexus_register: rootnex-1: Succesfully registered NEXUS i86pc 
nexops=fffffffffbd30040
Oct 30 08:52:24 smhost genunix: [ID 970263 kern.info] iommu0: unit setup
Oct 30 08:52:24 smhost genunix: [ID 970279 kern.info] iommu1: unit setup
Oct 30 08:52:24 smhost genunix: [ID 466022 kern.info] iommu0: creating 
domainid arena iommu0_domainid_arena
Oct 30 08:52:24 smhost genunix: [ID 160421 kern.info] iommu1: creating 
domainid arena iommu1_domainid_arena
Oct 30 08:52:24 smhost genunix: [ID 733261 kern.info] isa0: iommu: Found 
LPC device
Oct 30 08:52:24 smhost genunix: [ID 969345 kern.info] isa0: iommu: 
iommu1-domain-2-xlate-DVMA-arena: Creating dvma vmem arena [0x1000 - 
0x9a000]
Oct 30 08:52:24 smhost genunix: [ID 331729 kern.info] isa0: iommu: 
iommu1-domain-2-xlate-DVMA-arena: Adding dvma vmem span [0x100000 - 
0x79034000]
Oct 30 08:52:24 smhost genunix: [ID 325630 kern.info] isa0: iommu: 
iommu1-domain-2-xlate-DVMA-arena: Adding dvma vmem span [0x100000000 - 
0x2080000000]
Oct 30 08:52:24 smhost genunix: [ID 352456 kern.info] vgatext: iommu: 
Found GFX device
Oct 30 08:52:24 smhost genunix: [ID 137511 kern.info] ppt: iommu: Found 
GFX device
Oct 30 08:52:24 smhost genunix: [ID 664311 kern.notice] NOTICE: iommu0 
running
Oct 30 08:52:24 smhost iommulib: [ID 345192 kern.info] NOTICE: 
iommulib_register: iommu0: Detected IOMMU registration from vendor Intel
Oct 30 08:52:24 smhost iommulib: [ID 643661 kern.info] NOTICE: 
iommulib_register: iommu0: Succesfully registered IOMMU unit from 
vendor=Intel, ops=fffffeb1ae8dad80, data=fffffeb1ae136b00, IOMMULIB unitid=1
Oct 30 08:52:24 smhost genunix: [ID 341888 kern.notice] NOTICE: iommu1 
running
Oct 30 08:52:24 smhost iommulib: [ID 345192 kern.info] NOTICE: 
iommulib_register: iommu1: Detected IOMMU registration from vendor Intel
Oct 30 08:52:24 smhost iommulib: [ID 643661 kern.info] NOTICE: 
iommulib_register: iommu1: Succesfully registered IOMMU unit from 
vendor=Intel, ops=fffffeb1ae8dad20, data=fffffeb1ae136880, IOMMULIB unitid=2
Oct 30 08:52:24 smhost amd_iommu: [ID 251261 kern.info] NOTICE: 
amd_iommu: No AMD IOMMU ACPI IVRS table

Regards,

Stephan



------------------------------------------
illumos: illumos-developer
Permalink: https://illumos.topicbox.com/groups/developer/T0e71dad0bc17329a-Me195bb0082c187610dd28f76
Delivery options: https://illumos.topicbox.com/groups/developer/subscription