* T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter
@ 2021-04-10 15:27 Lonnie Abelbeck
2021-04-10 15:43 ` Mo Balaa
2021-04-10 15:59 ` Roman Mamedov
0 siblings, 2 replies; 6+ messages in thread
From: Lonnie Abelbeck @ 2021-04-10 15:27 UTC (permalink / raw)
To: WireGuard mailing list
Greetings,
I have been testing the T-Mobile Home Internet (4G/5G fixed wireless) service to a Linode VM via WireGuard.
The TMHI service uses CGNAT plus an additional NAT in their modem/gateway with a MTU of 1420, so WireGuard is configured with a 1340 MTU.
Everything works, but I thought I would share some jitter results that readers here might find interesting.
[gw-lan WGIP:10.4.1.1] -- [TMHI modem/gateway] -- 4G/5G/CGNAT -- [linode WGIP:10.4.1.10]
gw-lan ~ # mtr -wn -c 30 -s 1340 10.4.1.10
...
HOST: gw-lan Loss% Snt Last Avg Best Wrst StDev
1.|-- 10.4.1.10 0.0% 30 88.7 88.9 77.2 99.2 5.4
Looks to be as expected, in the direction of the CGNAT, now the other direction, against the grain of the CGNAT ...
linode ~ # mtr -wn -c 30 -s 1340 10.4.1.1
...
HOST: linode Loss% Snt Last Avg Best Wrst StDev
1.|-- 10.4.1.1 0.0% 30 206.1 243.5 73.8 393.9 97.9
Huge jitter, and is very reproducible. But no packet loss.
Further investigation shows for low traffic rates (linode->gw-lan) the jitter over WireGuard is huge, here are some UDP iperf3 tests showing how the jitter goes down as the traffic rate is increased.
linode ~ # iperf3 -c 10.4.1.1 -u -b 5k -t 30
...
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-30.25 sec 18.9 KBytes 5.11 Kbits/sec 68.428 ms 0/15 (0%) receiver
linode ~ # iperf3 -c 10.4.1.1 -u -b 10k -t 30
...
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-30.30 sec 37.7 KBytes 10.2 Kbits/sec 82.411 ms 0/30 (0%) receiver
linode ~ # iperf3 -c 10.4.1.1 -u -b 50k -t 30
...
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-30.14 sec 184 KBytes 49.9 Kbits/sec 7.532 ms 0/146 (0%) receiver
linode ~ # iperf3 -c 10.4.1.1 -u -b 100k -t 30
...
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-30.10 sec 367 KBytes 100 Kbits/sec 4.182 ms 0/292 (0%) receiver
linode ~ # iperf3 -c 10.4.1.1 -u -b 500k -t 30
...
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-30.11 sec 1.79 MBytes 498 Kbits/sec 1.308 ms 0/1456 (0%) receiver
So using VoIP a higher bitrate CODEC is actually better w.r.t jitter.
Hope others find this interesting.
Lonnie
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter
2021-04-10 15:27 T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter Lonnie Abelbeck
@ 2021-04-10 15:43 ` Mo Balaa
2021-04-10 16:03 ` Lonnie Abelbeck
2021-04-10 15:59 ` Roman Mamedov
1 sibling, 1 reply; 6+ messages in thread
From: Mo Balaa @ 2021-04-10 15:43 UTC (permalink / raw)
To: Lonnie Abelbeck; +Cc: WireGuard mailing list
Thanks for sharing, I have also been running WG tunnels over T-Mobile home internet and haven’t seen any of the jitter you are reporting.
Did you try the same tests (outbound) without running them via WG?
Which modem do you have? How many signal bars are you getting? Also, what does an non-tunneled speed test report?
Cheers
> On Apr 10, 2021, at 10:31, Lonnie Abelbeck <lists@lonnie.abelbeck.com> wrote:
>
> Greetings,
>
> I have been testing the T-Mobile Home Internet (4G/5G fixed wireless) service to a Linode VM via WireGuard.
>
> The TMHI service uses CGNAT plus an additional NAT in their modem/gateway with a MTU of 1420, so WireGuard is configured with a 1340 MTU.
>
> Everything works, but I thought I would share some jitter results that readers here might find interesting.
>
> [gw-lan WGIP:10.4.1.1] -- [TMHI modem/gateway] -- 4G/5G/CGNAT -- [linode WGIP:10.4.1.10]
>
> gw-lan ~ # mtr -wn -c 30 -s 1340 10.4.1.10
> ...
> HOST: gw-lan Loss% Snt Last Avg Best Wrst StDev
> 1.|-- 10.4.1.10 0.0% 30 88.7 88.9 77.2 99.2 5.4
>
> Looks to be as expected, in the direction of the CGNAT, now the other direction, against the grain of the CGNAT ...
>
> linode ~ # mtr -wn -c 30 -s 1340 10.4.1.1
> ...
> HOST: linode Loss% Snt Last Avg Best Wrst StDev
> 1.|-- 10.4.1.1 0.0% 30 206.1 243.5 73.8 393.9 97.9
>
> Huge jitter, and is very reproducible. But no packet loss.
>
> Further investigation shows for low traffic rates (linode->gw-lan) the jitter over WireGuard is huge, here are some UDP iperf3 tests showing how the jitter goes down as the traffic rate is increased.
>
> linode ~ # iperf3 -c 10.4.1.1 -u -b 5k -t 30
> ...
> [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
> [ 5] 0.00-30.25 sec 18.9 KBytes 5.11 Kbits/sec 68.428 ms 0/15 (0%) receiver
>
> linode ~ # iperf3 -c 10.4.1.1 -u -b 10k -t 30
> ...
> [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
> [ 5] 0.00-30.30 sec 37.7 KBytes 10.2 Kbits/sec 82.411 ms 0/30 (0%) receiver
>
> linode ~ # iperf3 -c 10.4.1.1 -u -b 50k -t 30
> ...
> [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
> [ 5] 0.00-30.14 sec 184 KBytes 49.9 Kbits/sec 7.532 ms 0/146 (0%) receiver
>
> linode ~ # iperf3 -c 10.4.1.1 -u -b 100k -t 30
> ...
> [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
> [ 5] 0.00-30.10 sec 367 KBytes 100 Kbits/sec 4.182 ms 0/292 (0%) receiver
>
> linode ~ # iperf3 -c 10.4.1.1 -u -b 500k -t 30
> ...
> [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
> [ 5] 0.00-30.11 sec 1.79 MBytes 498 Kbits/sec 1.308 ms 0/1456 (0%) receiver
>
>
> So using VoIP a higher bitrate CODEC is actually better w.r.t jitter.
>
> Hope others find this interesting.
>
> Lonnie
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter
2021-04-10 15:43 ` Mo Balaa
@ 2021-04-10 16:03 ` Lonnie Abelbeck
0 siblings, 0 replies; 6+ messages in thread
From: Lonnie Abelbeck @ 2021-04-10 16:03 UTC (permalink / raw)
To: Mo Balaa; +Cc: WireGuard mailing list
Hi Mo,
I have the latest hardware (gray cylinder)
--
Typically 3 of 5 bars
4G Band: B2
5G Band: n71
--
Note the huge jitter only occurs inbound, which is difficult to measure without using WireGuard.
The same UDP iperf3 tests outbound (gw-lan->linode) range from 3 to 8 ms jitter, regardless of bitrate.
--
Speed ( less than 2 seconds) 10/15 Mbps (down/up)
slowly ramps up to ...
Speed ( after 20 seconds) up to 200/35 Mbps (down/up)
--
Lonnie
> On Apr 10, 2021, at 10:43 AM, Mo Balaa <buddybalaa@gmail.com> wrote:
>
> Thanks for sharing, I have also been running WG tunnels over T-Mobile home internet and haven’t seen any of the jitter you are reporting.
>
> Did you try the same tests (outbound) without running them via WG?
> Which modem do you have? How many signal bars are you getting? Also, what does an non-tunneled speed test report?
>
> Cheers
>
>
>> On Apr 10, 2021, at 10:31, Lonnie Abelbeck <lists@lonnie.abelbeck.com> wrote:
>>
>> Greetings,
>>
>> I have been testing the T-Mobile Home Internet (4G/5G fixed wireless) service to a Linode VM via WireGuard.
>>
>> The TMHI service uses CGNAT plus an additional NAT in their modem/gateway with a MTU of 1420, so WireGuard is configured with a 1340 MTU.
>>
>> Everything works, but I thought I would share some jitter results that readers here might find interesting.
>>
>> [gw-lan WGIP:10.4.1.1] -- [TMHI modem/gateway] -- 4G/5G/CGNAT -- [linode WGIP:10.4.1.10]
>>
>> gw-lan ~ # mtr -wn -c 30 -s 1340 10.4.1.10
>> ...
>> HOST: gw-lan Loss% Snt Last Avg Best Wrst StDev
>> 1.|-- 10.4.1.10 0.0% 30 88.7 88.9 77.2 99.2 5.4
>>
>> Looks to be as expected, in the direction of the CGNAT, now the other direction, against the grain of the CGNAT ...
>>
>> linode ~ # mtr -wn -c 30 -s 1340 10.4.1.1
>> ...
>> HOST: linode Loss% Snt Last Avg Best Wrst StDev
>> 1.|-- 10.4.1.1 0.0% 30 206.1 243.5 73.8 393.9 97.9
>>
>> Huge jitter, and is very reproducible. But no packet loss.
>>
>> Further investigation shows for low traffic rates (linode->gw-lan) the jitter over WireGuard is huge, here are some UDP iperf3 tests showing how the jitter goes down as the traffic rate is increased.
>>
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 5k -t 30
>> ...
>> [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
>> [ 5] 0.00-30.25 sec 18.9 KBytes 5.11 Kbits/sec 68.428 ms 0/15 (0%) receiver
>>
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 10k -t 30
>> ...
>> [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
>> [ 5] 0.00-30.30 sec 37.7 KBytes 10.2 Kbits/sec 82.411 ms 0/30 (0%) receiver
>>
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 50k -t 30
>> ...
>> [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
>> [ 5] 0.00-30.14 sec 184 KBytes 49.9 Kbits/sec 7.532 ms 0/146 (0%) receiver
>>
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 100k -t 30
>> ...
>> [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
>> [ 5] 0.00-30.10 sec 367 KBytes 100 Kbits/sec 4.182 ms 0/292 (0%) receiver
>>
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 500k -t 30
>> ...
>> [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
>> [ 5] 0.00-30.11 sec 1.79 MBytes 498 Kbits/sec 1.308 ms 0/1456 (0%) receiver
>>
>>
>> So using VoIP a higher bitrate CODEC is actually better w.r.t jitter.
>>
>> Hope others find this interesting.
>>
>> Lonnie
>>
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter
2021-04-10 15:27 T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter Lonnie Abelbeck
2021-04-10 15:43 ` Mo Balaa
@ 2021-04-10 15:59 ` Roman Mamedov
2021-04-10 16:12 ` Lonnie Abelbeck
1 sibling, 1 reply; 6+ messages in thread
From: Roman Mamedov @ 2021-04-10 15:59 UTC (permalink / raw)
To: Lonnie Abelbeck; +Cc: WireGuard mailing list
On Sat, 10 Apr 2021 10:27:23 -0500
Lonnie Abelbeck <lists@lonnie.abelbeck.com> wrote:
> I have been testing the T-Mobile Home Internet (4G/5G fixed wireless) service to a Linode VM via WireGuard.
>
> The TMHI service uses CGNAT plus an additional NAT in their modem/gateway with a MTU of 1420, so WireGuard is configured with a 1340 MTU.
Do they provide IPv6? I see mentions that yes, but with incoming connections
blocked. Might still work for WG.
--
With respect,
Roman
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter
2021-04-10 15:59 ` Roman Mamedov
@ 2021-04-10 16:12 ` Lonnie Abelbeck
2021-04-16 13:56 ` Lonnie Abelbeck
0 siblings, 1 reply; 6+ messages in thread
From: Lonnie Abelbeck @ 2021-04-10 16:12 UTC (permalink / raw)
To: Roman Mamedov; +Cc: WireGuard mailing list
> On Apr 10, 2021, at 10:59 AM, Roman Mamedov <rm@romanrm.net> wrote:
>
> On Sat, 10 Apr 2021 10:27:23 -0500
> Lonnie Abelbeck <lists@lonnie.abelbeck.com> wrote:
>
>> I have been testing the T-Mobile Home Internet (4G/5G fixed wireless) service to a Linode VM via WireGuard.
>>
>> The TMHI service uses CGNAT plus an additional NAT in their modem/gateway with a MTU of 1420, so WireGuard is configured with a 1340 MTU.
>
> Do they provide IPv6? I see mentions that yes, but with incoming connections
> blocked. Might still work for WG.
Hi Roman,
TMHI uses IPv6 for their CGNAT, but AFAIK their current firmware does not support IPv6 on the ethernet ports (I disable their WiFi).
But if (when) they support IPv6, establishing the WireGuard tunnel over IPv6 would be great, and back to a 1420 MTU for WireGuard.
Lonnie
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter
2021-04-10 16:12 ` Lonnie Abelbeck
@ 2021-04-16 13:56 ` Lonnie Abelbeck
0 siblings, 0 replies; 6+ messages in thread
From: Lonnie Abelbeck @ 2021-04-16 13:56 UTC (permalink / raw)
To: Roman Mamedov, Mo Balaa; +Cc: WireGuard mailing list
> On Apr 10, 2021, at 11:12 AM, Lonnie Abelbeck <lists@lonnie.abelbeck.com> wrote:
>
>> On Apr 10, 2021, at 10:59 AM, Roman Mamedov <rm@romanrm.net> wrote:
>>
>> On Sat, 10 Apr 2021 10:27:23 -0500
>> Lonnie Abelbeck <lists@lonnie.abelbeck.com> wrote:
>>
>>> I have been testing the T-Mobile Home Internet (4G/5G fixed wireless) service to a Linode VM via WireGuard.
>>>
>>> The TMHI service uses CGNAT plus an additional NAT in their modem/gateway with a MTU of 1420, so WireGuard is configured with a 1340 MTU.
>>
>> Do they provide IPv6? I see mentions that yes, but with incoming connections
>> blocked. Might still work for WG.
>
> Hi Roman,
>
> TMHI uses IPv6 for their CGNAT, but AFAIK their current firmware does not support IPv6 on the ethernet ports (I disable their WiFi).
>
> But if (when) they support IPv6, establishing the WireGuard tunnel over IPv6 would be great, and back to a 1420 MTU for WireGuard.
A quick update, TMHI does support IPv6 via DHCPv6, but sadly has the same MTU (1420) as IPv4 has, and IPv6 is firewalled inbound (as Roman said).
I now have the WireGuard transport endpoints using IPv6 over TMHI, but the jitter with low bitrate traffic still occurs as with the IPv4 transport endpoint setup. Other than that, it works quite well.
Lonnie
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-04-16 13:56 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-10 15:27 T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter Lonnie Abelbeck
2021-04-10 15:43 ` Mo Balaa
2021-04-10 16:03 ` Lonnie Abelbeck
2021-04-10 15:59 ` Roman Mamedov
2021-04-10 16:12 ` Lonnie Abelbeck
2021-04-16 13:56 ` Lonnie Abelbeck
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).