From: Roman Mamedov <rm@romanrm.net>
To: Rumen Telbizov <rumen.telbizov@menlosecurity.com>
Cc: wireguard@lists.zx2c4.com
Subject: Re: WireGuard IRQ distribution
Date: Mon, 26 Jun 2023 19:53:11 +0500 [thread overview]
Message-ID: <20230626195311.03e86a19@nvm> (raw)
In-Reply-To: <CA+FoirDk8ZC09YXC7T6ureaBWG6ZREpfE3cq5MoAkKf-rC=U1g@mail.gmail.com>
On Tue, 9 May 2023 15:17:00 -0700
Rumen Telbizov <rumen.telbizov@menlosecurity.com> wrote:
> Baseline iperf3 performance over plain VLAN:
> * Stable 24Gbit/s and 2Mpps
>
> bmon:
> Gb (RX Bits/second)
> 24.54 .........|.||..|.||.||.||||||..||.||.......................
> 20.45 .........|||||||||||||||||||||||||||||.....................
> 16.36 ........||||||||||||||||||||||||||||||.....................
> 12.27 ........||||||||||||||||||||||||||||||.....................
> 8.18 ........|||||||||||||||||||||||||||||||.....................
> 4.09 ::::::::|||||||||||||||||||||||||||||||:::::::::::::::::::::
> 1 5 10 15 20 25 30 35 40 45 50 55 60
> M (RX Packets/second)
> 2.03 .........|.||..|.||.||.||||||..||.||........................
> 1.69 .........|||||||||||||||||||||||||||||......................
> 1.35 ........||||||||||||||||||||||||||||||......................
> 1.01 ........||||||||||||||||||||||||||||||......................
> 0.68 ........|||||||||||||||||||||||||||||||.....................
> 0.34 ::::::::|||||||||||||||||||||||||||||||:::::::::::::::::::::
> 1 5 10 15 20 25 30 35 40 45 50 55 60
>
> top:
> %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu2 : 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu4 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu6 : 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu7 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu8 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu9 : 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu10 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu11 : 0.0 us, 0.9 sy, 0.0 ni, 16.8 id, 0.0 wa, 0.0 hi, 82.2 si, 0.0 st
> %Cpu12 : 0.0 us, 32.3 sy, 0.0 ni, 65.6 id, 0.0 wa, 0.0 hi, 2.1 si, 0.0 st
> %Cpu13 : 1.0 us, 36.3 sy, 0.0 ni, 59.8 id, 0.0 wa, 0.0 hi, 2.9 si, 0.0 st
> %Cpu14 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu15 : 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>
> The IRQs do pile up behind CPU 11 because iperf3 is single-threaded.
I'm not sure if they pile up because of that, or because of the same reason
you point in WG's case, the 5-tuple being the same for the single TCP
connection of iperf3.
Out of interest, maybe you could try iperf3's UDP mode, and apply the same
port randomization trick as you used for WG, and see if it also makes it see
the better IRQ distribution, and the speed drop?
--
With respect,
Roman
next prev parent reply other threads:[~2023-06-26 14:56 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-09 22:17 Rumen Telbizov
2023-06-26 14:53 ` Roman Mamedov [this message]
2023-06-26 16:18 ` Rumen Telbizov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230626195311.03e86a19@nvm \
--to=rm@romanrm.net \
--cc=rumen.telbizov@menlosecurity.com \
--cc=wireguard@lists.zx2c4.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).