Outgoing ping required in container environment (even with PersistentKeepalive)

Outgoing ping required in container environment (even with PersistentKeepalive)

[Nico Schottelius]

Good morning,

another day news from the container land. When running wireguard in
kubernetes, deleting the existing pod and replacing it with a new one, I
see the following behaviour:

- The assigned IPv4 address stops being reachable (good so far)
- The assigned IPv4 address is then shortly reachable for about 5 seconds
- The assigned IPv4 address stops being reachable (not good)
- The assigned IPv4 address is again reachable, if I trigger a ping
  through the tunnel inside the container (ok, but why?)

I am using the following configuration:

--------------------------------------------------------------------------------
[Interface]
PrivateKey = ...
ListenPort = 51828
Address = 185.155.29.81/32
PostUp = iptables -t nat -I POSTROUTING -o ipv4 -j MASQUERADE

# upstream
[Peer]
Endpoint = vpn-...:51820
PublicKey = 6BRnQ+dmeFzVCH9RbM1pbJ7u3y3qrl+zUzzYCmC88kE=
AllowedIPs = 0.0.0.0/1, 128.0.0.0/1
PersistentKeepalive = 25
--------------------------------------------------------------------------------

And the following container specification:

--------------------------------------------------------------------------------
  spec:
      containers:
        - name: wireguard
          image: ungleich/ungleich-wireguard:{{ $.Chart.AppVersion }}
          # We only support 1 listener at the moment
          # Outgoing connections are not affected
          ports:
            - containerPort: 51820
          securityContext:
            capabilities:
              # NET_ADMIN for wg
              # NET_RAW for iptables
              add: ["NET_ADMIN", "NET_RAW" ]
          volumeMounts:
            - name: wireguard-config
              mountPath: "/etc/wireguard"
          resources:
            requests:
              memory: {{ $v.memory | default "1Gi" }}
              cpu: {{ $v.cpu | default "1000m" }}
            limits:
              memory: {{ $v.memory | default "1Gi" }}
              cpu: {{ $v.cpu | default "1000m" }}
--------------------------------------------------------------------------------

The strange thing is that after issuing the ping once inside the
container:

--------------------------------------------------------------------------------
[8:41] nb2:~% kubectl -n wireguard exec -ti wireguard-vpn-server-7db664db6f-zl4fz -- ping -c2 -4 google.com
PING google.com (172.217.168.78): 56 data bytes
64 bytes from 172.217.168.78: seq=0 ttl=116 time=9.110 ms
64 bytes from 172.217.168.78: seq=1 ttl=116 time=6.664 ms

--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 6.664/7.887/9.110 ms
--------------------------------------------------------------------------------

The connection stays correctly established.

If anyone has a pointer on what might be going on, any help is
appreciated.

Best regards,

Nico


--
Sustainable and modern Infrastructures by ungleich.ch

Re: Outgoing ping required in container environment (even with PersistentKeepalive)

[Nico Schottelius]

A follow up: we "upgraded" the wireguard container to use the following
entrypoint.sh instead:

--------------------------------------------------------------------------------
set -x

# Ensure everything is clean / show prior state
wg show

# Start all definitions
for conf in /etc/wireguard/*.conf; do
    # Remove leftovers??
    wg-quick down $conf

    # Try to up and if any tunnel fails -> exit
    wg-quick up "$conf" || (sleep 300; exit 1)
done

# Debug output
while true; do
    # Establish tunnels, keepalive alone is not enough for some reason
    ping -c2 185.203.112.1
    ping -c2 2a0a:e5c0::
    wg show
    sleep 300
done
--------------------------------------------------------------------------------

This establishes the connection reliably. I guess that should not be the
case, but effectively, it is.

I was for a moment suspecting that the old container is overlapping
running while the new one is created and whether there is a race
condition, however two points speak against that being the source of the
problem:

- Using the "Recreate strategy" in k8s, the container is first shut
  down. Using this, the behaviour does not change
- Even if my assumption was right, I'd expect a new handshake at some
  point to happen, but even minutes after restarting the container, the
  IPv4 address is not reachable.

Best regards,

Nico

Nico Schottelius <nico.schottelius@ungleich.ch> writes:

> Good morning,
>
> another day news from the container land. When running wireguard in
> kubernetes, deleting the existing pod and replacing it with a new one, I
> see the following behaviour:
>
> - The assigned IPv4 address stops being reachable (good so far)
> - The assigned IPv4 address is then shortly reachable for about 5 seconds
> - The assigned IPv4 address stops being reachable (not good)
> - The assigned IPv4 address is again reachable, if I trigger a ping
>   through the tunnel inside the container (ok, but why?)
>
> I am using the following configuration:
>
> --------------------------------------------------------------------------------
> [Interface]
> PrivateKey = ...
> ListenPort = 51828
> Address = 185.155.29.81/32
> PostUp = iptables -t nat -I POSTROUTING -o ipv4 -j MASQUERADE
>
> # upstream
> [Peer]
> Endpoint = vpn-...:51820
> PublicKey = 6BRnQ+dmeFzVCH9RbM1pbJ7u3y3qrl+zUzzYCmC88kE=
> AllowedIPs = 0.0.0.0/1, 128.0.0.0/1
> PersistentKeepalive = 25
> --------------------------------------------------------------------------------
>
> And the following container specification:
>
> --------------------------------------------------------------------------------
>   spec:
>       containers:
>         - name: wireguard
>           image: ungleich/ungleich-wireguard:{{ $.Chart.AppVersion }}
>           # We only support 1 listener at the moment
>           # Outgoing connections are not affected
>           ports:
>             - containerPort: 51820
>           securityContext:
>             capabilities:
>               # NET_ADMIN for wg
>               # NET_RAW for iptables
>               add: ["NET_ADMIN", "NET_RAW" ]
>           volumeMounts:
>             - name: wireguard-config
>               mountPath: "/etc/wireguard"
>           resources:
>             requests:
>               memory: {{ $v.memory | default "1Gi" }}
>               cpu: {{ $v.cpu | default "1000m" }}
>             limits:
>               memory: {{ $v.memory | default "1Gi" }}
>               cpu: {{ $v.cpu | default "1000m" }}
> --------------------------------------------------------------------------------
>
> The strange thing is that after issuing the ping once inside the
> container:
>
> --------------------------------------------------------------------------------
> [8:41] nb2:~% kubectl -n wireguard exec -ti wireguard-vpn-server-7db664db6f-zl4fz -- ping -c2 -4 google.com
> PING google.com (172.217.168.78): 56 data bytes
> 64 bytes from 172.217.168.78: seq=0 ttl=116 time=9.110 ms
> 64 bytes from 172.217.168.78: seq=1 ttl=116 time=6.664 ms
>
> --- google.com ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max = 6.664/7.887/9.110 ms
> --------------------------------------------------------------------------------
>
> The connection stays correctly established.
>
> If anyone has a pointer on what might be going on, any help is
> appreciated.
>
> Best regards,
>
> Nico


--
Sustainable and modern Infrastructures by ungleich.ch

Re: Outgoing ping required in container environment (even with PersistentKeepalive)

[Roman Mamedov]

On Sun, 08 May 2022 08:34:46 +0200
Nico Schottelius <nico.schottelius@ungleich.ch> wrote:

> The connection stays correctly established.
> 
> If anyone has a pointer on what might be going on, any help is
> appreciated.

Maybe you don't have a corresponding firewall rule, and happen to rely on the
ESTABLISHED,RELATED matching instead? This could explain the "works after a
ping" behaviour. But then the PersistentKeepalive could be expected to help as
well.

-- 
With respect,
Roman