Unexpected experience of site-to-site wireguard tunneling‏‏

["حامد صابر" <hsaber@gmail.com>]

Dear friends,
I have configured 3 wireguard interfaces on 2 servers to act as a
chained VPN for me (to bypass the internet censorship in my country),
with this schema:

client -- wg0 on middle-node -- wg1 on middle node -- wg2 on exit node
(to free internet)

Everything works fine, but after a while, the connection between wg1
and wg2 drops and I can't find the reason. The connection comes back
to action by simply switching the wg1 down and up again using
wg-quick. And the amazing behaviour is that sometimes the connection
comes back to work automatically  after some random time passes,
without any actions from my side (sometimes after a few tens of
minutes, sometimes after a few hours).
When the wg1-wg2 connection is not working, anything else between 2
servers (middle-node and exit-node) works fine. I mean I can ping the
public IP of each server from another part, but the local wireguard ip
of none of them are accessible.

I tried to monitor the situation and read the logs but couldn't find
what is happening here, so please help!

The configuration:
======================

client (my mobile phone):
-------------------------------------------
[Interface]
Address = 10.10.20.2/32
PrivateKey =  <private key of client>
DNS = 10.10.10.1

### Middle Node
[Peer]
PublicKey =  <public key of wg0>
PresharedKey =  <preshared key>
AllowedIPs = 0.0.0.0/0
Endpoint = middle-node:50842
======================

wg0 (in middle-node server):
-------------------------------------------
[Interface]
Address = 10.10.20.1/24
ListenPort = 50842
PrivateKey =  <private key of wg0>

### Client
[Peer]
PublicKey =  <public key of client>
PresharedKey =  <preshared key>
AllowedIPs = 10.10.20.2/32
======================

wg1 (again in middle-node server):
-------------------------------------------
[Interface]
Address = 10.10.10.2/32
PrivateKey =  <private key of wg1>

PostUp = ip route add default dev wg1 table middle
PostUp = ip rule add iif wg0 lookup middle
PostUp = wg set wg1 peer <publickey of wg2 (in exit-node)> allowed-ips 0.0.0.0/0

PreDown = ip route del default dev wg1 table middle
PreDown = ip rule del iif wg0 lookup middle

### Exit Node
[Peer]
PublicKey =  <publickey of wg2 (in exit-node)>
PresharedKey =  <preshared key>
AllowedIPs = 10.10.10.1/32
Endpoint = exit-node:50842
PersistentKeepalive = 25
======================

wg2 (in exit-node server):
-------------------------------------------
[Interface]
Address = 10.10.10.1/24
ListenPort = 50842
PrivateKey =  <private key of wg2>

PostUp   = iptables -A FORWARD -i eth0 -o wg2 -j ACCEPT
PostUp   = iptables -A FORWARD -i wg2 -j ACCEPT
PostUp   = iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

PostDown = iptables -D FORWARD -i eth0 -o wg2 -j ACCEPT
PostDown = iptables -D FORWARD -i wg2 -j ACCEPT
PostDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

### Middle Node
[Peer]
PublicKey =  <publickey of wg1 (in middle-node)>
PresharedKey =  <preshared key>
AllowedIPs = 10.0.0.0/8
======================
======================
======================

Sample log of dmesg when the wg1-wg2 connection is not working:
-------------------------------------------
[Wed Sep  1 11:19:32 2021] wireguard: wg1: Sending keepalive packet to
peer 12 (~exit-node-ip~:50842)
[Wed Sep  1 11:19:44 2021] wireguard: wg0: Sending keepalive packet to
peer 8 (~client-ip~:65323)
[Wed Sep  1 11:19:44 2021] wireguard: wg1: Receiving keepalive packet
from peer 12 (~exit-node-ip~:50842)
[Wed Sep  1 11:20:09 2021] wireguard: wg0: Receiving handshake
initiation from peer 8 (~client-ip~:65323)
[Wed Sep  1 11:20:09 2021] wireguard: wg0: Sending handshake response
to peer 8 (~client-ip~:65323)
[Wed Sep  1 11:20:09 2021] wireguard: wg0: Keypair 2867 destroyed for peer 8
[Wed Sep  1 11:20:09 2021] wireguard: wg0: Keypair 2871 created for peer 8
[Wed Sep  1 11:20:09 2021] wireguard: wg0: Receiving keepalive packet
from peer 8 (~client-ip~:65323)
[Wed Sep  1 11:21:19 2021] wireguard: wg0: Sending keepalive packet to
peer 8 (~client-ip~:65323)
[Wed Sep  1 11:21:24 2021] wireguard: wg1: Retrying handshake with
peer 12 (~exit-node-ip~:50842) because we stopped hearing back after
15 seconds
[Wed Sep  1 11:21:24 2021] wireguard: wg1: Sending handshake
initiation to peer 12 (~exit-node-ip~:50842)
[Wed Sep  1 11:21:30 2021] wireguard: wg1: Handshake for peer 12
(~exit-node-ip~:50842) did not complete after 5 seconds, retrying (try
2)
======================

Sample log of dmesg when the wg1-wg2 connection is coming back using
manual restart:
-------------------------------------------
[Wed Sep  1 11:45:52 2021] wireguard: wg1: Sending handshake
initiation to peer 12 (~exit-node-ip~:50842)
[Wed Sep  1 11:45:52 2021] wireguard: wg0: Sending keepalive packet to
peer 8 (~client-ip~:2335)
[Wed Sep  1 11:45:58 2021] wireguard: wg1: Handshake for peer 12
(~exit-node-ip~:50842) did not complete after 5 seconds, retrying (try
3)
[Wed Sep  1 11:45:58 2021] wireguard: wg1: Sending handshake
initiation to peer 12 (~exit-node-ip~:50842)
[Wed Sep  1 11:45:58 2021] wireguard: wg1: Keypair 2878 destroyed for peer 12
[Wed Sep  1 11:45:58 2021] wireguard: wg1: Peer 12
(~exit-node-ip~:50842) destroyed
[Wed Sep  1 11:45:58 2021] wireguard: wg1: Interface destroyed
[Wed Sep  1 11:45:58 2021] wireguard: wg1: Interface created
[Wed Sep  1 11:45:58 2021] wireguard: wg1: Peer 13 created
[Wed Sep  1 11:45:58 2021] wireguard: wg1: Sending keepalive packet to
peer 13 (~exit-node-ip~:50842)
[Wed Sep  1 11:45:58 2021] wireguard: wg1: Sending handshake
initiation to peer 13 (~exit-node-ip~:50842)
[Wed Sep  1 11:45:58 2021] wireguard: wg1: Receiving handshake
response from peer 13 (~exit-node-ip~:50842)
[Wed Sep  1 11:45:58 2021] wireguard: wg1: Keypair 2881 created for peer 13
[Wed Sep  1 11:46:12 2021] wireguard: wg0: Receiving keepalive packet
from peer 8 (~client-ip~:2335)
[Wed Sep  1 11:46:14 2021] wireguard: wg1: Receiving keepalive packet
from peer 13 (~exit-node-ip~:50842)
[Wed Sep  1 11:46:27 2021] wireguard: wg0: Sending keepalive packet to
peer 8 (~client-ip~:2335)
[Wed Sep  1 11:46:28 2021] wireguard: wg1: Receiving keepalive packet
from peer 13 (~exit-node-ip~:50842)
[Wed Sep  1 11:46:52 2021] wireguard: wg1: Receiving keepalive packet
from peer 13 (~exit-node-ip~:50842)


Thanks in advance for your kind help