bhyve guest Solaris 11.4 with e1000g interfcae has a high IP latency

bhyve guest Solaris 11.4 with e1000g interfcae has a high IP latency

[Carsten Grzemba]

[-- Attachment #1: Type: text/plain, Size: 1238 bytes --]

I have a bhyve guest with Solaris 11.4 which uses an e1000g network interface. The bhyve host uses 10G ixgbe interfaces and OmniOS. 
I see there high latency values and packet loss. The packet loss could be CRC cksum errors related.
# ping -s 172.16.1.1 10000
PING 172.16.1.1: 10000 data bytes
10008 bytes from con01.contac.lan (172.16.1.1): icmp_seq=13. time=2.579 ms
10008 bytes from con01.contac.lan (172.16.1.1): icmp_seq=15. time=1.755 ms
10008 bytes from con01.contac.lan (172.16.1.1): icmp_seq=16. time=1.916 ms
10008 bytes from con01.contac.lan (172.16.1.1): icmp_seq=19. time=2.438 ms
10008 bytes from con01.contac.lan (172.16.1.1): icmp_seq=20. time=1.548 ms
10008 bytes from con01.contac.lan (172.16.1.1): icmp_seq=21. time=1.760 ms
10008 bytes from con01.contac.lan (172.16.1.1): icmp_seq=23. time=2.098 ms
10008 bytes from con01.contac.lan (172.16.1.1): icmp_seq=24. time=2.172 ms
10008 bytes from con01.contac.lan (172.16.1.1): icmp_seq=25. time=2.367 ms

Other Bhyve guests (openindiana) with vioif interface work without any trouble on this host.
Is this an Solaris or an e1000g related problem? Is it possible to use any other IP interfaces than e1000g? 
Is there are some tuning possible?

Many thanks!

[-- Attachment #2: Type: text/html, Size: 1601 bytes --]

Re: bhyve guest Solaris 11.4 with e1000g interfcae has a high IP latency

[Carsten Grzemba]

[-- Attachment #1: Type: text/plain, Size: 209 bytes --]

Network analyzer tells me, there is a problem with the header checksum: 0x0000 incorrect, should be ... (may be caused by "IP checksum offload")
Is this a problem with the e1000g driver or is it configurable?

[-- Attachment #2: Type: text/html, Size: 278 bytes --]

Re: [discuss] bhyve guest Solaris 11.4 with e1000g interfcae has a high IP latency

[Volker A. Brandt]

Hi Carsten!


Carsten Grzemba via illumos-discuss writes:
> I have a bhyve guest with Solaris 11.4 which uses an e1000g network interface. The bhyve host uses 10G ixgbe interfaces and OmniOS.
> I see there high latency values and packet loss. The packet loss could be CRC cksum errors related.

[...]

> Other Bhyve guests (openindiana) with vioif interface work without any trouble on this host.
> Is this an Solaris or an e1000g related problem? Is it possible to use any other IP interfaces than e1000g?

Have you tried the "vtionet" driver provided by Solaris?  This has been
on my list of things to test for quite some time...


Cheers -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt        Consulting and Support for Solaris-based Systems
Brandt & Brandt Computer GmbH               |   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim, GERMANY  |         Email: vab@bb-c.de
HR: Amtsgericht Bonn, HRB 10513             | Mastodon: @vab@bonn.social
Geschäftsführer: Rainer J.H. Brandt und Volker A. Brandt

Re: bhyve guest Solaris 11.4 with e1000g interfcae has a high IP latency

[Carsten Grzemba]

[-- Attachment #1: Type: text/plain, Size: 2465 bytes --]

unfortunately no luck with virtio:
WARNING: legacy virtio device detected (pci1af4,1001).
WARNING: vtioblk: MSI-X interrupts not available. This may affect performance
WARNING: legacy virtio device detected (pci1af4,1000).
WARNING: vtionet: MSI-X interrupts not available. This may affect performance


net0: DL_BIND_REQ failed: DL_SYSERR (errno 6)
net0: DL_UNBIND_REQ failed: DL_OUTSTATE

panic[cpu3]/thread=ffffe33000f43ac0: BAD TRAP: type=e (#pf Page fault) rp=ffffe33000f435f0 addr=ffffa1012048800a

sched: #pf Page fault
Bad kernel fault at addr=0xffffa1012048800a
pid=0, pc=0xfffffffffbe40387, sp=0xffffe33000f436e0, eflags=0x10296
cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: 3606b8<smap,smep,osxsav,pcide,xmme,fxsr,pge,pae,pse,de>
cr2: ffffa1012048800a cr3: 7c00000 cr8: c

        rdi: ffffa70019a182c0 rsi: ffffe33000f43734 rdx:                e
        rcx:                0  r8: ffffe33000f43734  r9: ffffa10020488018
        rax: ffffa70019a182c0 rbx:         fffffff2 rbp: ffffe33000f43790
        r10: ffffa1002048800a r11: ffffe33000f4372c r12: ffffa10020488018
        r13: ffffa70019a182c0 r14:                e r15:                0
        fsb:     7fd30f805240 gsb: ffffe08000600000  ds:               4b
         es:               4b  fs:                0  gs:                0
        trp:                e err:                0 rip: fffffffffbe40387
         cs:               30 rfl:            10296 rsp: ffffe33000f436e0
         ss:               38

ffffe33000f43510 unix:die+167 ()
ffffe33000f435e0 unix:trap+1462 ()
ffffe33000f435f0 unix:_cmntrap+ea ()
ffffe33000f43790 mac:mac_fix_cksum_single_common+607 ()
ffffe33000f437b0 mac:mac_fix_cksum_single+88 ()
ffffe33000f43800 mac:mac_fix_cksum+43 ()
ffffe33000f43840 vtionet:vtionet_rx_cksum+109 ()
ffffe33000f438c0 vtionet:vtionet_rx+104 ()
ffffe33000f43950 vtionet:vtionet_do_rx+6d ()
ffffe33000f439b0 vtionet:vtionet_intr_rx+8e ()
ffffe33000f439f0 vtionet:vtionet_intr_fixed+59 ()
ffffe33000f43a40 unix:av_dispatch_autovect+81 ()
ffffe33000f43aa0 unix:dispatch_hardint+72 ()
ffffe33000efb8c0 unix:switch_sp_and_call+13 ()
ffffe33000efb920 unix:do_interrupt+1a2 ()
ffffe33000efb930 unix:_interrupt+c4 ()
ffffe33000efba20 unix:mach_cpu_idle+17 ()
ffffe33000efba60 unix:cpu_idle+2b7 ()
ffffe33000efba70 unix:cpu_idle_adaptive+19 ()
ffffe33000efbaa0 unix:idle+11e ()
ffffe33000efbab0 unix:thread_start+8 ()

syncing file systems... done


[-- Attachment #2: Type: text/html, Size: 2771 bytes --]

Re: [discuss] bhyve guest Solaris 11.4 with e1000g interfcae has a high IP latency

[Dan McDonald]

On Jul 10, 2024, at 12:35 PM, Carsten Grzemba via illumos-discuss <discuss@lists.illumos.org> wrote:
> 
> Network analyzer tells me, there is a problem with the header checksum: 0x0000 incorrect, should be ... (may be caused by "IP checksum offload")
> Is this a problem with the e1000g driver or is it configurable?

Depending on where you place your snoop/tcpdump/etc this is wholly expected.  And packets would drop, not slow down, in the absence of such.

You COULD disable HW checksum in your guest; that would rule out a slow virtualized e1000g part.

Dan

Re: [discuss] bhyve guest Solaris 11.4 with e1000g interfcae has a high IP latency

[Carsten Grzemba]

[-- Attachment #1: Type: text/plain, Size: 375 bytes --]

ping sequence shows the packet drops also. The Solaris guest hosts an IMAP server and the IMAP protocol seems to be sensitive related package lost. At least Thunderbird is stubborn. 
For checksum disable I found only the parameter XsumRX which is releated for Rx. But I see the retransmitted TCP packages and checksum errors on the client outside, hence it is Tx related. 

[-- Attachment #2: Type: text/html, Size: 457 bytes --]

Re: [discuss] bhyve guest Solaris 11.4 with e1000g interfcae has a high IP latency

[bronkoo]

[-- Attachment #1: Type: text/plain, Size: 199 bytes --]

Hi Carsten,
if your 10G ixgbe host interface is configured about jumbo frames (MTU=9000), your bhyve guest e1000g vnic interface on host should set to MTU=1500. May be it inherited jumbo frames...

[-- Attachment #2: Type: text/html, Size: 281 bytes --]

Re: [discuss] bhyve guest Solaris 11.4 with e1000g interfcae has a high IP latency

[Carsten Grzemba]

[-- Attachment #1: Type: text/plain, Size: 503 bytes --]

no, all related has MTU=1500
# dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
ixgbe2      phys      1500   up       --         --
ixgbe0      phys      1500   up       --         --
ixgbe3      phys      9000   up       --         --
ixgbe1      phys      9000   up       --         --
aggr0       aggr      1500   up       --         ixgbe2,ixgbe0
vnic_solaris11 vnic   1500   up       --         aggr0

but finally I have managed  to get it to work with vioif driver.

[-- Attachment #2: Type: text/html, Size: 805 bytes --]