Development discussion of WireGuard
 help / color / mirror / Atom feed
* T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter
@ 2021-04-10 15:27 Lonnie Abelbeck
  2021-04-10 15:43 ` Mo Balaa
  2021-04-10 15:59 ` Roman Mamedov
  0 siblings, 2 replies; 6+ messages in thread
From: Lonnie Abelbeck @ 2021-04-10 15:27 UTC (permalink / raw)
  To: WireGuard mailing list

Greetings,

I have been testing the T-Mobile Home Internet (4G/5G fixed wireless) service to a Linode VM via WireGuard.

The TMHI service uses CGNAT plus an additional NAT in their modem/gateway with a MTU of 1420, so WireGuard is configured with a 1340 MTU.

Everything works, but I thought I would share some jitter results that readers here might find interesting.

[gw-lan WGIP:10.4.1.1] -- [TMHI modem/gateway] -- 4G/5G/CGNAT -- [linode WGIP:10.4.1.10]

gw-lan ~ # mtr -wn -c 30 -s 1340 10.4.1.10
...
HOST: gw-lan      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.4.1.10  0.0%    30   88.7  88.9  77.2  99.2   5.4

Looks to be as expected, in the direction of the CGNAT, now the other direction, against the grain of the CGNAT ...

linode ~ # mtr -wn -c 30 -s 1340 10.4.1.1
...
HOST: linode     Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.4.1.1  0.0%    30  206.1 243.5  73.8 393.9  97.9

Huge jitter, and is very reproducible.  But no packet loss.

Further investigation shows for low traffic rates (linode->gw-lan) the jitter over WireGuard is huge, here are some UDP iperf3 tests showing how the jitter goes down as the traffic rate is increased.

linode ~ # iperf3 -c 10.4.1.1 -u -b 5k -t 30
...
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-30.25  sec  18.9 KBytes  5.11 Kbits/sec  68.428 ms  0/15 (0%)  receiver

linode ~ # iperf3 -c 10.4.1.1 -u -b 10k -t 30
...
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-30.30  sec  37.7 KBytes  10.2 Kbits/sec  82.411 ms  0/30 (0%)  receiver

linode ~ # iperf3 -c 10.4.1.1 -u -b 50k -t 30
...
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-30.14  sec   184 KBytes  49.9 Kbits/sec  7.532 ms  0/146 (0%)  receiver

linode ~ # iperf3 -c 10.4.1.1 -u -b 100k -t 30
...
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-30.10  sec   367 KBytes   100 Kbits/sec  4.182 ms  0/292 (0%)  receiver

linode ~ # iperf3 -c 10.4.1.1 -u -b 500k -t 30
...
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-30.11  sec  1.79 MBytes   498 Kbits/sec  1.308 ms  0/1456 (0%)  receiver


So using VoIP a higher bitrate CODEC is actually better w.r.t jitter.

Hope others find this interesting.

Lonnie


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter
  2021-04-10 15:27 T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter Lonnie Abelbeck
@ 2021-04-10 15:43 ` Mo Balaa
  2021-04-10 16:03   ` Lonnie Abelbeck
  2021-04-10 15:59 ` Roman Mamedov
  1 sibling, 1 reply; 6+ messages in thread
From: Mo Balaa @ 2021-04-10 15:43 UTC (permalink / raw)
  To: Lonnie Abelbeck; +Cc: WireGuard mailing list

Thanks for sharing, I have also been running WG tunnels over T-Mobile home internet and haven’t seen any of the jitter you are reporting. 

Did you try the same tests (outbound) without running them via WG?
Which modem do you have? How many signal bars are you getting? Also, what does an non-tunneled speed test report?

Cheers 


> On Apr 10, 2021, at 10:31, Lonnie Abelbeck <lists@lonnie.abelbeck.com> wrote:
> 
> Greetings,
> 
> I have been testing the T-Mobile Home Internet (4G/5G fixed wireless) service to a Linode VM via WireGuard.
> 
> The TMHI service uses CGNAT plus an additional NAT in their modem/gateway with a MTU of 1420, so WireGuard is configured with a 1340 MTU.
> 
> Everything works, but I thought I would share some jitter results that readers here might find interesting.
> 
> [gw-lan WGIP:10.4.1.1] -- [TMHI modem/gateway] -- 4G/5G/CGNAT -- [linode WGIP:10.4.1.10]
> 
> gw-lan ~ # mtr -wn -c 30 -s 1340 10.4.1.10
> ...
> HOST: gw-lan      Loss%   Snt   Last   Avg  Best  Wrst StDev
>  1.|-- 10.4.1.10  0.0%    30   88.7  88.9  77.2  99.2   5.4
> 
> Looks to be as expected, in the direction of the CGNAT, now the other direction, against the grain of the CGNAT ...
> 
> linode ~ # mtr -wn -c 30 -s 1340 10.4.1.1
> ...
> HOST: linode     Loss%   Snt   Last   Avg  Best  Wrst StDev
>  1.|-- 10.4.1.1  0.0%    30  206.1 243.5  73.8 393.9  97.9
> 
> Huge jitter, and is very reproducible.  But no packet loss.
> 
> Further investigation shows for low traffic rates (linode->gw-lan) the jitter over WireGuard is huge, here are some UDP iperf3 tests showing how the jitter goes down as the traffic rate is increased.
> 
> linode ~ # iperf3 -c 10.4.1.1 -u -b 5k -t 30
> ...
> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
> [  5]   0.00-30.25  sec  18.9 KBytes  5.11 Kbits/sec  68.428 ms  0/15 (0%)  receiver
> 
> linode ~ # iperf3 -c 10.4.1.1 -u -b 10k -t 30
> ...
> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
> [  5]   0.00-30.30  sec  37.7 KBytes  10.2 Kbits/sec  82.411 ms  0/30 (0%)  receiver
> 
> linode ~ # iperf3 -c 10.4.1.1 -u -b 50k -t 30
> ...
> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
> [  5]   0.00-30.14  sec   184 KBytes  49.9 Kbits/sec  7.532 ms  0/146 (0%)  receiver
> 
> linode ~ # iperf3 -c 10.4.1.1 -u -b 100k -t 30
> ...
> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
> [  5]   0.00-30.10  sec   367 KBytes   100 Kbits/sec  4.182 ms  0/292 (0%)  receiver
> 
> linode ~ # iperf3 -c 10.4.1.1 -u -b 500k -t 30
> ...
> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
> [  5]   0.00-30.11  sec  1.79 MBytes   498 Kbits/sec  1.308 ms  0/1456 (0%)  receiver
> 
> 
> So using VoIP a higher bitrate CODEC is actually better w.r.t jitter.
> 
> Hope others find this interesting.
> 
> Lonnie
> 

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter
  2021-04-10 15:27 T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter Lonnie Abelbeck
  2021-04-10 15:43 ` Mo Balaa
@ 2021-04-10 15:59 ` Roman Mamedov
  2021-04-10 16:12   ` Lonnie Abelbeck
  1 sibling, 1 reply; 6+ messages in thread
From: Roman Mamedov @ 2021-04-10 15:59 UTC (permalink / raw)
  To: Lonnie Abelbeck; +Cc: WireGuard mailing list

On Sat, 10 Apr 2021 10:27:23 -0500
Lonnie Abelbeck <lists@lonnie.abelbeck.com> wrote:

> I have been testing the T-Mobile Home Internet (4G/5G fixed wireless) service to a Linode VM via WireGuard.
> 
> The TMHI service uses CGNAT plus an additional NAT in their modem/gateway with a MTU of 1420, so WireGuard is configured with a 1340 MTU.

Do they provide IPv6? I see mentions that yes, but with incoming connections
blocked. Might still work for WG.

-- 
With respect,
Roman

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter
  2021-04-10 15:43 ` Mo Balaa
@ 2021-04-10 16:03   ` Lonnie Abelbeck
  0 siblings, 0 replies; 6+ messages in thread
From: Lonnie Abelbeck @ 2021-04-10 16:03 UTC (permalink / raw)
  To: Mo Balaa; +Cc: WireGuard mailing list

Hi Mo,

I have the latest hardware (gray cylinder)
--
Typically 3 of 5 bars
4G Band: B2
5G Band: n71
--

Note the huge jitter only occurs inbound, which is difficult to measure without using WireGuard.

The same UDP iperf3 tests outbound (gw-lan->linode) range from 3 to 8 ms jitter, regardless of bitrate.

--
Speed ( less than 2 seconds) 10/15 Mbps (down/up)
slowly ramps up to ...
Speed ( after 20 seconds) up to 200/35 Mbps (down/up)
--

Lonnie


> On Apr 10, 2021, at 10:43 AM, Mo Balaa <buddybalaa@gmail.com> wrote:
> 
> Thanks for sharing, I have also been running WG tunnels over T-Mobile home internet and haven’t seen any of the jitter you are reporting. 
> 
> Did you try the same tests (outbound) without running them via WG?
> Which modem do you have? How many signal bars are you getting? Also, what does an non-tunneled speed test report?
> 
> Cheers 
> 
> 
>> On Apr 10, 2021, at 10:31, Lonnie Abelbeck <lists@lonnie.abelbeck.com> wrote:
>> 
>> Greetings,
>> 
>> I have been testing the T-Mobile Home Internet (4G/5G fixed wireless) service to a Linode VM via WireGuard.
>> 
>> The TMHI service uses CGNAT plus an additional NAT in their modem/gateway with a MTU of 1420, so WireGuard is configured with a 1340 MTU.
>> 
>> Everything works, but I thought I would share some jitter results that readers here might find interesting.
>> 
>> [gw-lan WGIP:10.4.1.1] -- [TMHI modem/gateway] -- 4G/5G/CGNAT -- [linode WGIP:10.4.1.10]
>> 
>> gw-lan ~ # mtr -wn -c 30 -s 1340 10.4.1.10
>> ...
>> HOST: gw-lan      Loss%   Snt   Last   Avg  Best  Wrst StDev
>> 1.|-- 10.4.1.10  0.0%    30   88.7  88.9  77.2  99.2   5.4
>> 
>> Looks to be as expected, in the direction of the CGNAT, now the other direction, against the grain of the CGNAT ...
>> 
>> linode ~ # mtr -wn -c 30 -s 1340 10.4.1.1
>> ...
>> HOST: linode     Loss%   Snt   Last   Avg  Best  Wrst StDev
>> 1.|-- 10.4.1.1  0.0%    30  206.1 243.5  73.8 393.9  97.9
>> 
>> Huge jitter, and is very reproducible.  But no packet loss.
>> 
>> Further investigation shows for low traffic rates (linode->gw-lan) the jitter over WireGuard is huge, here are some UDP iperf3 tests showing how the jitter goes down as the traffic rate is increased.
>> 
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 5k -t 30
>> ...
>> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
>> [  5]   0.00-30.25  sec  18.9 KBytes  5.11 Kbits/sec  68.428 ms  0/15 (0%)  receiver
>> 
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 10k -t 30
>> ...
>> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
>> [  5]   0.00-30.30  sec  37.7 KBytes  10.2 Kbits/sec  82.411 ms  0/30 (0%)  receiver
>> 
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 50k -t 30
>> ...
>> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
>> [  5]   0.00-30.14  sec   184 KBytes  49.9 Kbits/sec  7.532 ms  0/146 (0%)  receiver
>> 
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 100k -t 30
>> ...
>> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
>> [  5]   0.00-30.10  sec   367 KBytes   100 Kbits/sec  4.182 ms  0/292 (0%)  receiver
>> 
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 500k -t 30
>> ...
>> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
>> [  5]   0.00-30.11  sec  1.79 MBytes   498 Kbits/sec  1.308 ms  0/1456 (0%)  receiver
>> 
>> 
>> So using VoIP a higher bitrate CODEC is actually better w.r.t jitter.
>> 
>> Hope others find this interesting.
>> 
>> Lonnie
>> 
> 
> 


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter
  2021-04-10 15:59 ` Roman Mamedov
@ 2021-04-10 16:12   ` Lonnie Abelbeck
  2021-04-16 13:56     ` Lonnie Abelbeck
  0 siblings, 1 reply; 6+ messages in thread
From: Lonnie Abelbeck @ 2021-04-10 16:12 UTC (permalink / raw)
  To: Roman Mamedov; +Cc: WireGuard mailing list



> On Apr 10, 2021, at 10:59 AM, Roman Mamedov <rm@romanrm.net> wrote:
> 
> On Sat, 10 Apr 2021 10:27:23 -0500
> Lonnie Abelbeck <lists@lonnie.abelbeck.com> wrote:
> 
>> I have been testing the T-Mobile Home Internet (4G/5G fixed wireless) service to a Linode VM via WireGuard.
>> 
>> The TMHI service uses CGNAT plus an additional NAT in their modem/gateway with a MTU of 1420, so WireGuard is configured with a 1340 MTU.
> 
> Do they provide IPv6? I see mentions that yes, but with incoming connections
> blocked. Might still work for WG.

Hi Roman,

TMHI uses IPv6 for their CGNAT, but AFAIK their current firmware does not support IPv6 on the ethernet ports (I disable their WiFi).

But if (when) they support IPv6, establishing the WireGuard tunnel over IPv6 would be great, and back to a 1420 MTU for WireGuard.

Lonnie



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter
  2021-04-10 16:12   ` Lonnie Abelbeck
@ 2021-04-16 13:56     ` Lonnie Abelbeck
  0 siblings, 0 replies; 6+ messages in thread
From: Lonnie Abelbeck @ 2021-04-16 13:56 UTC (permalink / raw)
  To: Roman Mamedov, Mo Balaa; +Cc: WireGuard mailing list



> On Apr 10, 2021, at 11:12 AM, Lonnie Abelbeck <lists@lonnie.abelbeck.com> wrote:
> 
>> On Apr 10, 2021, at 10:59 AM, Roman Mamedov <rm@romanrm.net> wrote:
>> 
>> On Sat, 10 Apr 2021 10:27:23 -0500
>> Lonnie Abelbeck <lists@lonnie.abelbeck.com> wrote:
>> 
>>> I have been testing the T-Mobile Home Internet (4G/5G fixed wireless) service to a Linode VM via WireGuard.
>>> 
>>> The TMHI service uses CGNAT plus an additional NAT in their modem/gateway with a MTU of 1420, so WireGuard is configured with a 1340 MTU.
>> 
>> Do they provide IPv6? I see mentions that yes, but with incoming connections
>> blocked. Might still work for WG.
> 
> Hi Roman,
> 
> TMHI uses IPv6 for their CGNAT, but AFAIK their current firmware does not support IPv6 on the ethernet ports (I disable their WiFi).
> 
> But if (when) they support IPv6, establishing the WireGuard tunnel over IPv6 would be great, and back to a 1420 MTU for WireGuard.

A quick update, TMHI does support IPv6 via DHCPv6, but sadly has the same MTU (1420) as IPv4 has, and IPv6 is firewalled inbound (as Roman said).

I now have the WireGuard transport endpoints using IPv6 over TMHI, but the jitter with low bitrate traffic still occurs as with the IPv4 transport endpoint setup.  Other than that, it works quite well.

Lonnie


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2021-04-16 13:56 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-10 15:27 T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter Lonnie Abelbeck
2021-04-10 15:43 ` Mo Balaa
2021-04-10 16:03   ` Lonnie Abelbeck
2021-04-10 15:59 ` Roman Mamedov
2021-04-10 16:12   ` Lonnie Abelbeck
2021-04-16 13:56     ` Lonnie Abelbeck

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).