Recent NVMe changes break testing on a VMware guest?

Recent NVMe changes break testing on a VMware guest?

[Dan McDonald]

I'm hoping this is a quirk of VMware Fusion's virtual NVMe device, but testing this my recent SmartOS release, I found new errors:

kebe(~/smartos-tests/release-20240711)[0]% find . -type f | xargs grep firmware.32
./test_results/20240711T034200/unit/firmware.32/stderr:firmware.32: TEST FAILED: invalid fw load numd 8k gran (3): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
./test_results/20240711T034200/unit/firmware.32/stderr:firmware.32: TEST FAILED: invalid fw load numd 8k gran (4): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
./test_results/20240711T034200/unit/firmware.32/merged:firmware.32: TEST FAILED: invalid fw load numd 8k gran (3): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
./test_results/20240711T034200/unit/firmware.32/merged:firmware.32: TEST FAILED: invalid fw load numd 8k gran (4): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
./test_results/20240711T034200/log:Test: /opt/nvme-tests/tests/unit/firmware.32 (run as root)        [00:00] [FAIL]
./test_results/20240711T034200/log:03:42:00.06 firmware.32: TEST FAILED: invalid fw load numd 8k gran (3): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
./test_results/20240711T034200/log:03:42:00.06 firmware.32: TEST FAILED: invalid fw load numd 8k gran (4): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
kebe(~/smartos-tests/release-20240711)[0]% 


I know I didn't mismerge anything, because there are no diffs between gate and illumos-joyent's nvme code:


smartos-build(~/illumos-merge-tools)[2]% diff -ru illumos-{gate,joyent}/usr/src/uts/intel/nvme
smartos-build(~/illumos-merge-tools)[0]% diff -ru illumos-{gate,joyent}/usr/src/uts/common/io/nvme
smartos-build(~/illumos-merge-tools)[0]% diff -ru illumos-{gate,joyent}/usr/src/uts/common/sys/nvme
smartos-build(~/illumos-merge-tools)[0]% diff -ru illumos-{gate,joyent}/usr/src/common/nvme
smartos-build(~/illumos-merge-tools)[0]% diff -ru illumos-{gate,joyent}/usr/src/cmd/nvmeadm
smartos-build(~/illumos-merge-tools)[0]% diff -ru illumos-{gate,joyent}/usr/src/lib/libnvme
Common subdirectories: illumos-gate/usr/src/lib/libnvme/amd64 and illumos-joyent/usr/src/lib/libnvme/amd64
Common subdirectories: illumos-gate/usr/src/lib/libnvme/common and illumos-joyent/usr/src/lib/libnvme/common
Common subdirectories: illumos-gate/usr/src/lib/libnvme/i386 and illumos-joyent/usr/src/lib/libnvme/i386
smartos-build(~/illumos-merge-tools)[0]% diff -ru illumos-{gate,joyent}/usr/src/test/nvme-tests
Common subdirectories: illumos-gate/usr/src/test/nvme-tests/cmd and illumos-joyent/usr/src/test/nvme-tests/cmd
Common subdirectories: illumos-gate/usr/src/test/nvme-tests/runfiles and illumos-joyent/usr/src/test/nvme-tests/runfiles
Common subdirectories: illumos-gate/usr/src/test/nvme-tests/tests and illumos-joyent/usr/src/test/nvme-tests/tests
Common subdirectories: illumos-gate/usr/src/test/nvme-tests/tests/ioctl and illumos-joyent/usr/src/test/nvme-tests/tests/ioctl
Common subdirectories: illumos-gate/usr/src/test/nvme-tests/tests/libnvme and illumos-joyent/usr/src/test/nvme-tests/tests/libnvme
Common subdirectories: illumos-gate/usr/src/test/nvme-tests/tests/nvmeadm and illumos-joyent/usr/src/test/nvme-tests/tests/nvmeadm
Common subdirectories: illumos-gate/usr/src/test/nvme-tests/tests/unit and illumos-joyent/usr/src/test/nvme-tests/tests/unit
smartos-build(~/illumos-merge-tools)[0]% 


Does anyone with NVMe experience have insight into why the two tests (unit/firmware.*) fail now?

Thanks,
Dan

Re: [developer] Recent NVMe changes break testing on a VMware guest?

[Dan McDonald]

Reproducible in both 32- and 64-bit BTW:

[root@coal /tmp/test]# /opt/nvme-tests/tests/unit/firmware.32 >& /tmp/32
[root@coal /tmp/test]# /opt/nvme-tests/tests/unit/firmware.64 >& /tmp/64
[root@coal /tmp/test]# diff /tmp/{32,64}
56,57c56,57
< TEST PASSED: valid ffirmware.32: TEST FAILED: invalid fw load numd 8k gran (3): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
< firmware.32: TEST FAILED: invalid fw load numd 8k gran (4): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
---
> TEST PASSED: valid ffirmware.64: TEST FAILED: invalid fw load numd 8k gran (3): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
> firmware.64: TEST FAILED: invalid fw load numd 8k gran (4): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
[root@coal /tmp/test]# 

Dan

Re: [developer] Recent NVMe changes break testing on a VMware guest?

[Andy Fiddaman]

On Thu, 11 Jul 2024, Dan McDonald wrote:

> Reproducible in both 32- and 64-bit BTW:
>
> [root@coal /tmp/test]# /opt/nvme-tests/tests/unit/firmware.32 >& /tmp/32
> [root@coal /tmp/test]# /opt/nvme-tests/tests/unit/firmware.64 >& /tmp/64
> [root@coal /tmp/test]# diff /tmp/{32,64}
> 56,57c56,57
> < TEST PASSED: valid ffirmware.32: TEST FAILED: invalid fw load numd 8k gran (3): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
> < firmware.32: TEST FAILED: invalid fw load numd 8k gran (4): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
> ---
> > TEST PASSED: valid ffirmware.64: TEST FAILED: invalid fw load numd 8k gran (3): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
> > firmware.64: TEST FAILED: invalid fw load numd 8k gran (4): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
> [root@coal /tmp/test]#

This is fallout from

	16633 nvme firmware updates cannot enforce update

That change relaxed the restrictions on the allowed firmware chunk size so that
it no-longer needed to confirm to the firmware upgrade granularity, and as a
result the tests are seeing NVME_FIELD_ERR_OK where they expect a bad
value error. I will file a followup and fix it, sorry about that.

Andy

Re: [developer] Recent NVMe changes break testing on a VMware guest?

[Dan McDonald]

On Jul 11, 2024, at 12:40 PM, Andy Fiddaman <andy@omnios.org> wrote:
> 
> 
> On Thu, 11 Jul 2024, Dan McDonald wrote:
> 
>> Reproducible in both 32- and 64-bit BTW:
>> 
>> [root@coal /tmp/test]# /opt/nvme-tests/tests/unit/firmware.32 >& /tmp/32
>> [root@coal /tmp/test]# /opt/nvme-tests/tests/unit/firmware.64 >& /tmp/64
>> [root@coal /tmp/test]# diff /tmp/{32,64}
>> 56,57c56,57
>> < TEST PASSED: valid ffirmware.32: TEST FAILED: invalid fw load numd 8k gran (3): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
>> < firmware.32: TEST FAILED: invalid fw load numd 8k gran (4): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
>> ---
>>> TEST PASSED: valid ffirmware.64: TEST FAILED: invalid fw load numd 8k gran (3): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
>>> firmware.64: TEST FAILED: invalid fw load numd 8k gran (4): found wrong return value NVME_FIELD_ERR_OK (0), expected NVME_FIELD_ERR_BAD_VALUE (3)
>> [root@coal /tmp/test]#
> 
> This is fallout from
> 
>        16633 nvme firmware updates cannot enforce update
> 
> That change relaxed the restrictions on the allowed firmware chunk size so that
> it no-longer needed to confirm to the firmware upgrade granularity, and as a
> result the tests are seeing NVME_FIELD_ERR_OK where they expect a bad
> value error. I will file a followup and fix it, sorry about that.

No apologies needed.  Your explanation is more than enough to allow me to not rebuild this week's SmartOS.

Thanks,
Dan