From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: rspazzol@redhat.com Received: from krantz.zx2c4.com (localhost [127.0.0.1]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 5b87e6fc for ; Sun, 16 Sep 2018 23:07:28 +0000 (UTC) Received: from mail-lj1-f182.google.com (mail-lj1-f182.google.com [209.85.208.182]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 007c43ea for ; Sun, 16 Sep 2018 23:07:28 +0000 (UTC) Received: by mail-lj1-f182.google.com with SMTP id q127-v6so11568075ljq.11 for ; Sun, 16 Sep 2018 16:09:00 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: References: <20180916165458.GA31165@matrix-dream.net> From: Raffaele Spazzoli Date: Sun, 16 Sep 2018 19:08:58 -0400 Message-ID: Subject: Re: what to do when the peers use different IPs to transmit and receive To: =?UTF-8?B?SXZhbiBMYWLDoXRo?= Content-Type: multipart/alternative; boundary="000000000000489f670576052451" Cc: wireguard@lists.zx2c4.com List-Id: Development discussion of WireGuard List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , --000000000000489f670576052451 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ivan, I tried the SNAT idea, and still have issue. here is an example configuration of one of the nodes: [Interface] ListenPort =3D 5555 PrivateKey =3D --- [Peer] PublicKey =3D H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE=3D AllowedIPs =3D 10.128.2.0/23 Endpoint =3D 192.168.99.12:31112 PersistentKeepalive =3D 25 [Peer] PublicKey =3D 5nC5cyDg9WZ/2R3CPEbM+fSXzsn5Yx1mX48iizdfdHU=3D AllowedIPs =3D 10.128.0.0/23 Endpoint =3D 192.168.99.14:32188 PersistentKeepalive =3D 25 [Peer] PublicKey =3D MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6Dj30E=3D AllowedIPs =3D 10.130.0.0/23 Endpoint =3D 192.168.99.13:31992 PersistentKeepalive =3D 25 [Peer] PublicKey =3D s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/snp3mUk=3D AllowedIPs =3D 10.129.0.0/23 Endpoint =3D 192.168.99.15:30305 PersistentKeepalive =3D 25 [Peer] PublicKey =3D SuO927DbGm2h2I8hcf24LvYWglKp+4wGAuiyisin/yY=3D AllowedIPs =3D 10.131.0.0/23 Endpoint =3D 192.168.99.7:31714 PersistentKeepalive =3D 25 [Peer] PublicKey =3D a+tK21LKdsBkQNqmqdRpvS9HLpz2W8rwDijTPkXEc0Q=3D AllowedIPs =3D 10.129.2.0/23 Endpoint =3D 192.168.99.6:31165 PersistentKeepalive =3D 25 the private IP to VIP map for the peers of this node is: 10.128.2.10-192.168.99.12 10.128.1.94-192.168.99.14 10.130.0.136-192.168.99.13 10.129.1.158-192.168.99.15 10.131.0.199-192.168.99.7 10.129.2.217-192.168.99.6 I create the following iptables rules: sh-4.2# iptables -t nat -n -L Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain INPUT (policy ACCEPT) target prot opt source destination SNAT udp -- 10.128.2.10 0.0.0.0/0 udp dpt:5555 to: 192.168.99.12:5555 SNAT udp -- 10.128.1.94 0.0.0.0/0 udp dpt:5555 to: 192.168.99.14:5555 SNAT udp -- 10.130.0.136 0.0.0.0/0 udp dpt:5555 to: 192.168.99.13:5555 SNAT udp -- 10.129.1.158 0.0.0.0/0 udp dpt:5555 to: 192.168.99.15:5555 SNAT udp -- 10.131.0.199 0.0.0.0/0 udp dpt:5555 to: 192.168.99.7:5555 SNAT udp -- 10.129.2.217 0.0.0.0/0 udp dpt:5555 to: 192.168.99.6:5555 Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination after the handshake the configuration changes to: sh-4.2# wg interface: sdn-tunnel public key: gCFgNjOpObU71Vmjub/R9KIn3MHgzXnKtrh9Tf+W628=3D private key: (hidden) listening port: 5555 peer: s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/snp3mUk=3D endpoint: 10.134.0.1:1033 allowed ips: 10.129.0.0/23 latest handshake: 31 seconds ago transfer: 180 B received, 452 B sent persistent keepalive: every 25 seconds peer: MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6Dj30E=3D endpoint: 10.134.0.1:1032 allowed ips: 10.130.0.0/23 latest handshake: 37 seconds ago transfer: 212 B received, 272 B sent persistent keepalive: every 25 seconds peer: 5nC5cyDg9WZ/2R3CPEbM+fSXzsn5Yx1mX48iizdfdHU=3D endpoint: 10.134.0.1:1031 allowed ips: 10.128.0.0/23 latest handshake: 39 seconds ago transfer: 180 B received, 304 B sent persistent keepalive: every 25 seconds peer: a+tK21LKdsBkQNqmqdRpvS9HLpz2W8rwDijTPkXEc0Q=3D endpoint: 192.168.99.6:31165 allowed ips: 10.129.2.0/23 latest handshake: 41 seconds ago transfer: 156 B received, 180 B sent persistent keepalive: every 25 seconds peer: H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE=3D endpoint: 192.168.99.12:31112 allowed ips: 10.128.2.0/23 latest handshake: 41 seconds ago transfer: 156 B received, 180 B sent persistent keepalive: every 25 seconds peer: SuO927DbGm2h2I8hcf24LvYWglKp+4wGAuiyisin/yY=3D endpoint: 192.168.99.7:31714 allowed ips: 10.131.0.0/23 latest handshake: 41 seconds ago transfer: 156 B received, 180 B sent persistent keepalive: every 25 seconds as you can see some of the endpoint's addresses have changed. the first three are not correct anymore. After the introduction of the iptables rules they change to an IP that makes no sense to me 10.134.0.1 finally here are a few seconds of tcpdump in case it helps: sh-4.2# tcpdump -i eth0 -nn -v tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 23:07:01.045331 IP (tos 0x0, ttl 64, id 27711, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.6.31165: UDP, length 32 23:07:01.045363 IP (tos 0x0, ttl 64, id 53835, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.7.31714: UDP, length 32 23:07:01.045411 IP (tos 0x0, ttl 64, id 27009, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.12.31112: UDP, length 32 23:07:02.758694 IP (tos 0x0, ttl 61, id 19309, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.1.1031 > 10.134.0.72.5555: UDP, length 32 23:07:04.053339 IP (tos 0x0, ttl 64, id 36786, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.72.5555 > 10.134.0.1.1032: UDP, length 32 23:07:07.765375 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.134.0.72 tell 10.134.0.1, length 28 23:07:07.765394 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.134.0.72 is-at 0a:58:0a:86:00:48, length 28 23:07:10.938921 IP (tos 0x0, ttl 61, id 33093, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.1.1033 > 10.134.0.72.5555: UDP, length 32 23:07:26.069271 IP (tos 0x0, ttl 64, id 37778, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.6.31165: UDP, length 32 23:07:26.069271 IP (tos 0x0, ttl 64, id 59175, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.7.31714: UDP, length 32 23:07:26.069303 IP (tos 0x0, ttl 64, id 49067, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.12.31112: UDP, length 32 23:07:27.797284 IP (tos 0x0, ttl 64, id 57007, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.72.5555 > 10.134.0.1.1031: UDP, length 32 23:07:29.079935 IP (tos 0x0, ttl 61, id 18743, offset 0, flags [none], proto UDP (17), length 60) 10.134.0.1.1032 > 10.134.0.72.5555: UDP, length 32 Thanks, Raffaele Raffaele Spazzoli Senior Architect - OpenShift , Containers and PaaS Practice Tel: +1 216-258-7717 On Sun, Sep 16, 2018 at 2:56 PM, Raffaele Spazzoli wrote: > I'll try to make an example > cluster 1 node 1 has private IP1 and VIP1 > cluster 2 node 2 has private IP2 and VIP2 > > each node uses it's private ip for outbound connections. > each node can receive inbound connection on its VIP. > > so the wireguard config file for node1 is going to look like: > > [peer] > endpoint: VIP2:port > > and for node 2: > [peer] > endpoint: VIP1: port > > the problem is that after the handshake, wireguard updates the config to > the following (for example for node2): > [peer] > endpoint: IP1:port > > but IP2 cannot route to IP1... > > I think a well configured SNAT rule may work, although is not elegant > because it forces the cluster to exchange information about their private > IPs. > This should not be needed and in the cloud private IPs are ephemeral.... > > anyway thanks for the advice, I am going to try to use it in my > prototype. > > I still think there is need for a better technical approach for a long > term solution. > > > Thanks, > Raffaele > > Raffaele Spazzoli > Senior Architect - OpenShift , Containers > and PaaS Practice > Tel: +1 216-258-7717 > > > > On Sun, Sep 16, 2018 at 12:54 PM, Ivan Lab=C3=A1th > wrote: > >> Hi, >> >> On Sun, Sep 16, 2018 at 08:21:02AM -0400, Raffaele Spazzoli wrote: >> > ... then the IP that a node uses for its outbound >> > connection is not the same that its peer need to use for its inbound >> > connections. >> >> Who uses what for whose connection? You lost me here. >> Looks like a broken network to me. Does TCP even work? >> >> Anyway, SNAT/DNAT should be able to fix things up, if you want to go >> that route. >> >> Regards, >> Ivan >> > > --000000000000489f670576052451 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Ivan,

I tried the SNAT idea, = and still have issue.=C2=A0
here is an example configuration of o= ne of the nodes:

[Interface]
Listen= Port =3D 5555=C2=A0
PrivateKey =3D ---
=C2=A0
[Peer]
PublicKey =3D H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE= =3D
AllowedIPs =3D 10.128.2.0/23=
PersistentKeepalive =3D 25

[Peer]
PublicKey =3D 5nC5cyDg9WZ/2R3CPEbM+fSXzsn5Yx1mX48iizdfd= HU=3D
AllowedIPs =3D 10.128.0.0/= 23
PersistentKeepalive =3D 25

=
[Peer]
PublicKey =3D MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6D= j30E=3D
AllowedIPs =3D 10.130.0.= 0/23
PersistentKeepalive =3D 25

[Peer]
PublicKey =3D s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/s= np3mUk=3D
AllowedIPs =3D 10.129.= 0.0/23
PersistentKeepalive =3D 25

[Peer]
PublicKey =3D SuO927DbGm2h2I8hcf24LvYWglKp+4wGAui= yisin/yY=3D
AllowedIPs =3D 10.13= 1.0.0/23
PersistentKeepalive =3D 25

[Peer]
PublicKey =3D a+tK21LKdsBkQNqmqdRpvS9HLpz2W8rwDij= TPkXEc0Q=3D
AllowedIPs =3D 10.12= 9.2.0/23
PersistentKeepalive =3D 25
=
the private IP t= o VIP map for the peers of this node is:=C2=A0

<= div>10.128.2.10-192.168.99.12 =20 10.128.1.94-192.168.99.14 =20 10.130.0.136-192.168.99.13 =20 10.129.1.158-192.168.99.15 =20 10.131.0.199-192.168.99.7 =20 10.129.2.217-192.168.99.6

I create the following iptables rules:

sh-4.2# iptables -t= nat -n -L Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain INPUT (policy ACCEPT) target prot opt source destination SNAT udp -- 10.128.2.10 0.0.0= .0/0 udp dpt:5555 to:1= 92.168.99.12:5555 SNAT udp -- 10.128.1.94 0.0.0= .0/0 udp dpt:5555 to:1= 92.168.99.14:5555 SNAT udp -- 10.130.0.136 0.0.0= .0/0 udp dpt:5555 to:1= 92.168.99.13:5555 SNAT udp -- 10.129.1.158 0.0.0= .0/0 udp dpt:5555 to:1= 92.168.99.15:5555 SNAT udp -- 10.131.0.199 0.0.0= .0/0 udp dpt:5555 to:19= 2.168.99.7:5555 SNAT udp -- 10.129.2.217 0.0.0= .0/0 udp dpt:5555 to:19= 2.168.99.6:5555 Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination
<= font color=3D"#363636" face=3D"Menlo, Monaco, Consolas, monospace">
=
after the handshake the configuration changes to:

sh-4.2# wg
interface: sdn-tunnel
=
=C2=A0 public key: gCFgNjOpObU71Vmjub/R9KIn3MHgzXnKtrh9Tf+W6= 28=3D
=C2=A0 private key: (hidden)
= =C2=A0 listening port: 5555

pe= er: s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/snp3mUk=3D
=C2= =A0 endpoint: 10.134.0.1:1033
<= div style=3D"font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;whi= te-space:pre">=C2=A0 allowed ips: 10.129.0= .0/23
=C2=A0 latest handshake: 31 seconds ago
=
=C2=A0 transfer: 180 B received, 452 B sent
=C2=A0 persistent keepalive: every 25 seconds

=
peer: MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6Dj30E=3D
=C2=A0 endpoint: 10.1= 34.0.1:1032
=C2=A0 allowed ips: 10.130.0.0/23
=C2=A0 latest handshake= : 37 seconds ago
=C2=A0 transfer: 212 B received, 272 B= sent
=C2=A0 persistent keepalive: every 25 seconds

peer: 5nC5cyDg9WZ/2R3CPEbM+fSXzsn= 5Yx1mX48iizdfdHU=3D
=C2=A0 endpoint: 10.134.0.1:1031
=C2=A0 allowed ips= : 10.128.0.0/23
= =C2=A0 latest handshake: 39 seconds ago
=C2=A0 transfer= : 180 B received, 304 B sent
=C2=A0 persistent keepaliv= e: every 25 seconds

peer: a+tK= 21LKdsBkQNqmqdRpvS9HLpz2W8rwDijTPkXEc0Q=3D
=C2=A0 endpo= int: 192.168.99.6:31165
=C2=A0 allowed ips: 10.129.2.0/= 23
=C2=A0 latest handshake: 41 seconds ago
=C2=A0 transfer: 156 B received, 180 B sent
=C2=A0 persistent keepalive: every 25 seconds

peer: H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE=3D
=
=C2=A0 endpoint: 192.= 168.99.12:31112
=C2=A0 allowed ips: 10.128.2.0/23
=C2=A0 latest hands= hake: 41 seconds ago
=C2=A0 transfer: 156 B received, 1= 80 B sent
=C2=A0 persistent keepalive: every 25 seconds=

peer: SuO927DbGm2h2I8hcf24LvY= WglKp+4wGAuiyisin/yY=3D
=C2=A0 endpoint: 192.168.99.7:31714
=C2=A0 a= llowed ips: 10.131.0.0/23
=C2=A0 latest handshake: 41 seconds ago
=C2=A0 = transfer: 156 B received, 180 B sent
=C2=A0 persistent = keepalive: every 25 seconds

as= you can see some of the endpoint's addresses have changed.
the first three are not correct anymore.
After the introduction of the iptables rules they change to an IP= that makes no sense to me 10.134.0.1<= /font>

=
finally here are a few seconds of tcpdump in case it helps:

<= /div>
sh-4.2# tcpdump -i eth0 -nn -v tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 26214= 4 bytes 23:07:01.045331 IP (tos 0x0, ttl 64, id 27711, offset 0, flags [none], prot= o UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.6.31165: UDP, length 32 23:07:01.045363 IP (tos 0x0, ttl 64, id 53835, offset 0, flags [none], prot= o UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.7.31714: UDP, length 32 23:07:01.045411 IP (tos 0x0, ttl 64, id 27009, offset 0, flags [none], prot= o UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.12.31112: UDP, length 32 23:07:02.758694 IP (tos 0x0, ttl 61, id 19309, offset 0, flags [none], prot= o UDP (17), length 60) 10.134.0.1.1031 > 10.134.0.72.5555: UDP, length 32 23:07:04.053339 IP (tos 0x0, ttl 64, id 36786, offset 0, flags [none], prot= o UDP (17), length 60) 10.134.0.72.5555 > 10.134.0.1.1032: UDP, length 32 23:07:07.765375 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.134= .0.72 tell 10.134.0.1, length 28 23:07:07.765394 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.134.0.72 is-a= t 0a:58:0a:86:00:48, length 28 23:07:10.938921 IP (tos 0x0, ttl 61, id 33093, offset 0, flags [none], prot= o UDP (17), length 60) 10.134.0.1.1033 > 10.134.0.72.5555: UDP, length 32 23:07:26.069271 IP (tos 0x0, ttl 64, id 37778, offset 0, flags [none], prot= o UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.6.31165: UDP, length 32 23:07:26.069271 IP (tos 0x0, ttl 64, id 59175, offset 0, flags [none], prot= o UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.7.31714: UDP, length 32 23:07:26.069303 IP (tos 0x0, ttl 64, id 49067, offset 0, flags [none], prot= o UDP (17), length 60) 10.134.0.72.5555 > 192.168.99.12.31112: UDP, length 32 23:07:27.797284 IP (tos 0x0, ttl 64, id 57007, offset 0, flags [none], prot= o UDP (17), length 60) 10.134.0.72.5555 > 10.134.0.1.1031: UDP, length 32 23:07:29.079935 IP (tos 0x0, ttl 61, id 18743, offset 0, flags [none], prot= o UDP (17), length 60) 10.134.0.1.1032 > 10.134.0.72.5555: UDP, length 32
=

Thanks,
Raffaele
Raffaele Spazzoli
Tel: +1 216-258-7717

=

=

On Sun, Sep 16, 2018 at 2:56 PM, Raffaele Sp= azzoli <rspazzol@redhat.com> wrote:
I'll try to make an example
cluster = 1 node 1 has private IP1 and VIP1=C2=A0
cluster 2 node 2 has priv= ate IP2 and VIP2

each node uses it's private ip for outbound connections.=C2= =A0
each node can receive inbound connectio= n on its VIP.

so the wireguard config file for node1 is going to look like:
=

[peer]
endpoint: VIP2:port

and for node 2:
[peer]=C2=A0
endpoint: VIP= 1: port

the problem is that after the handshake, wireguard updates the config to = the following (for example for node2):
[pee= r]
endpoint: IP1:port

but IP2 cannot route to IP1= ...

I = think a well configured SNAT rule may work, although is not elegant because= it forces the cluster to exchange information about their private IPs.
This should not be needed and in the cloud pri= vate IPs are ephemeral....

anyway thanks for the advice, I am going to try to use= it=C2=A0 in my prototype.=C2=A0

=
I still think there is need for a better technic= al approach for a long term solution.

<= /div>

=
Thanks,
Raffaele

Raffaele SpazzoliTel: +1 216-258-7717




--000000000000489f670576052451--