From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: uberscubajim@gmail.com Received: from krantz.zx2c4.com (localhost [127.0.0.1]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 00c9b988 for ; Sun, 10 Sep 2017 11:08:39 +0000 (UTC) Received: from mail-wm0-f44.google.com (mail-wm0-f44.google.com [74.125.82.44]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 43e1f8fb for ; Sun, 10 Sep 2017 11:08:39 +0000 (UTC) Received: by mail-wm0-f44.google.com with SMTP id 189so1420708wmh.1 for ; Sun, 10 Sep 2017 04:34:38 -0700 (PDT) Return-Path: Received: from [10.23.162.36] ([141.143.213.13]) by smtp.googlemail.com with ESMTPSA id a33sm3242248edd.67.2017.09.10.04.34.35 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 10 Sep 2017 04:34:35 -0700 (PDT) To: WireGuard mailing list From: Jim Darby Subject: Timing issue (?) with wg-quick up on Raspberry Pi B+ Message-ID: <1e1740c3-f8cf-2ee2-d842-749b687cb737@gmail.com> Date: Sun, 10 Sep 2017 12:34:34 +0100 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="------------371B0ECA8C47CFD0EBEF3BA5" List-Id: Development discussion of WireGuard List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , This is a multi-part message in MIME format. --------------371B0ECA8C47CFD0EBEF3BA5 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit I've been using WireGuard and it's proven great. However, I've noticed a perclirary on a Raspberry Pi B+. When I do a wg-quick up wg0 it echos the commands it's running and they're fine. So it says: [#] ip link add wg0 type wireguard [#] wg setconf wg0 /dev/fd/63 [#] ip address add 192.168.2.3/32 dev wg0 [#] ip link set mtu 1420 dev wg0 [#] ip link set wg0 up [#] ip route add 192.168.2.0/24 dev wg0 However, if I look at wg0 I get: wg0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 UP POINTOPOINT RUNNING NOARP MTU:1420 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) So, it's all worked /apart/ from the ip address add 192.168.2.3/32 dev wg0. It works fine on the faster Raspberry Pi 3 and if run the ip address command manually it performs perfectly. This leads me to suspect some sort of timing problem that rarely shows due to most machines being significantly faster. Any ideas? Regards, Jim. --------------371B0ECA8C47CFD0EBEF3BA5 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit

I've been using WireGuard and it's proven great. However, I've noticed a perclirary on a Raspberry Pi B+.

When I do a wg-quick up wg0 it echos the commands it's running and they're fine. So it says:

[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip address add 192.168.2.3/32 dev wg0
[#] ip link set mtu 1420 dev wg0
[#] ip link set wg0 up
[#] ip route add 192.168.2.0/24 dev wg0

However, if I look at wg0 I get:

wg0       Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          UP POINTOPOINT RUNNING NOARP MTU:1420 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

So, it's all worked apart from the ip address add 192.168.2.3/32 dev wg0.

It works fine on the faster Raspberry Pi 3 and if run the ip address command manually it performs perfectly.

This leads me to suspect some sort of timing problem that rarely shows due to most machines being significantly faster.

Any ideas?

Regards,

Jim.

--------------371B0ECA8C47CFD0EBEF3BA5-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Jason@zx2c4.com Received: from krantz.zx2c4.com (localhost [127.0.0.1]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id a4126b5d for ; Sun, 10 Sep 2017 12:18:33 +0000 (UTC) Received: from frisell.zx2c4.com (frisell.zx2c4.com [192.95.5.64]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 0061d499 for ; Sun, 10 Sep 2017 12:18:33 +0000 (UTC) Received: by frisell.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 8fa6a0c8 for ; Sun, 10 Sep 2017 12:37:36 +0000 (UTC) Received: by frisell.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id 774942a9 (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128:NO) for ; Sun, 10 Sep 2017 12:37:36 +0000 (UTC) Received: by mail-oi0-f48.google.com with SMTP id l74so29993223oih.1 for ; Sun, 10 Sep 2017 05:44:32 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <1e1740c3-f8cf-2ee2-d842-749b687cb737@gmail.com> References: <1e1740c3-f8cf-2ee2-d842-749b687cb737@gmail.com> From: "Jason A. Donenfeld" Date: Sun, 10 Sep 2017 14:44:30 +0200 Message-ID: Subject: Re: Timing issue (?) with wg-quick up on Raspberry Pi B+ To: Jim Darby Content-Type: text/plain; charset="UTF-8" Cc: WireGuard mailing list List-Id: Development discussion of WireGuard List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Hey Jim, That's very odd indeed. Do you have any other network management daemons running on there that could be interfering? Also: what snapshot version, kernel version, distribution, etc are you using? Anything odd in dmesg after running wg-quick? Regards, Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: uberscubajim@gmail.com Received: from krantz.zx2c4.com (localhost [127.0.0.1]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id a1faf6a1 for ; Sun, 10 Sep 2017 12:43:25 +0000 (UTC) Received: from mail-wm0-f50.google.com (mail-wm0-f50.google.com [74.125.82.50]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 6cce9e21 for ; Sun, 10 Sep 2017 12:43:25 +0000 (UTC) Received: by mail-wm0-f50.google.com with SMTP id a137so2198187wma.0 for ; Sun, 10 Sep 2017 06:09:23 -0700 (PDT) Return-Path: Subject: Re: Timing issue (?) with wg-quick up on Raspberry Pi B+ To: "Jason A. Donenfeld" References: <1e1740c3-f8cf-2ee2-d842-749b687cb737@gmail.com> From: Jim Darby Message-ID: <07621915-a53a-03f3-9c75-b7e7d188d109@gmail.com> Date: Sun, 10 Sep 2017 14:09:21 +0100 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------710DCE46A8DFC7BEC3B2F47D" Cc: WireGuard mailing list List-Id: Development discussion of WireGuard List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , This is a multi-part message in MIME format. --------------710DCE46A8DFC7BEC3B2F47D Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Hi Jason, Thanks for the very quick response. It happens with both the August and September snapshots. Kernel 4.9.35+, Raspbian based on Debian 9 and nothing odd in dmesg. However, your comment about network management daemons running is most interesting. Here's an extract from journalctl's output: Sep 09 21:31:28 janus ifplugd(wg0)[6903]: ifplugd 0.28 initializing. Sep 09 21:31:28 janus ifplugd(wg0)[6903]: Using interface wg0/00:00:00:00:00:00 Sep 09 21:31:28 janus ifplugd(wg0)[6903]: Using detection mode: IFF_RUNNING Sep 09 21:31:28 janus ifplugd(wg0)[6903]: Initialization complete, link beat detected. Sep 09 21:31:28 janus ifplugd(wg0)[6903]: Executing '/etc/ifplugd/ifplugd.action wg0 up'. Sep 09 21:31:28 janus ifplugd(wg0)[6903]: client: Ignoring unknown interface wg0=wg0. Sep 09 21:31:29 janus ifplugd(wg0)[6903]: Program executed successfully. So, something is triggering ifplug. I'm unsure what it might be doing and it /seems/ to imply that it's ignoring it but maybe this is the cause of the problem. wg0 is /not/ mentioned in /etc/network/interfaces so maybe ifup is trying to be helpful? There may well be a race condition between ifup and wg-quick… Regards, Jim. On 10/09/17 13:44, Jason A. Donenfeld wrote: > Hey Jim, > > That's very odd indeed. Do you have any other network management > daemons running on there that could be interfering? > > Also: what snapshot version, kernel version, distribution, etc are you > using? Anything odd in dmesg after running wg-quick? > > Regards, > Jason --------------710DCE46A8DFC7BEC3B2F47D Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit

Hi Jason,

Thanks for the very quick response.

It happens with both the August and September snapshots. Kernel 4.9.35+, Raspbian based on Debian 9 and nothing odd in dmesg.

However, your comment about network management daemons running is most interesting. Here's an extract from journalctl's output:

Sep 09 21:31:28 janus ifplugd(wg0)[6903]: ifplugd 0.28 initializing.
Sep 09 21:31:28 janus ifplugd(wg0)[6903]: Using interface wg0/00:00:00:00:00:00
Sep 09 21:31:28 janus ifplugd(wg0)[6903]: Using detection mode: IFF_RUNNING
Sep 09 21:31:28 janus ifplugd(wg0)[6903]: Initialization complete, link beat detected.
Sep 09 21:31:28 janus ifplugd(wg0)[6903]: Executing '/etc/ifplugd/ifplugd.action wg0 up'.
Sep 09 21:31:28 janus ifplugd(wg0)[6903]: client: Ignoring unknown interface wg0=wg0.
Sep 09 21:31:29 janus ifplugd(wg0)[6903]: Program executed successfully.

So, something is triggering ifplug. I'm unsure what it might be doing and it seems to imply that it's ignoring it but maybe this is the cause of the problem. wg0 is not mentioned in /etc/network/interfaces so maybe ifup is trying to be helpful? There may well be a race condition between ifup and wg-quick…

Regards,

Jim.

On 10/09/17 13:44, Jason A. Donenfeld wrote:

Hey Jim,

That's very odd indeed. Do you have any other network management
daemons running on there that could be interfering?

Also: what snapshot version, kernel version, distribution, etc are you
using? Anything odd in dmesg after running wg-quick?

Regards,
Jason

--------------710DCE46A8DFC7BEC3B2F47D-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Jason@zx2c4.com Received: from krantz.zx2c4.com (localhost [127.0.0.1]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id c2702cfc for ; Sun, 10 Sep 2017 14:00:16 +0000 (UTC) Received: from frisell.zx2c4.com (frisell.zx2c4.com [192.95.5.64]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id d5613fbd for ; Sun, 10 Sep 2017 14:00:16 +0000 (UTC) Received: by frisell.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 859b3db1 for ; Sun, 10 Sep 2017 14:19:19 +0000 (UTC) Received: by frisell.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id aee55069 (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128:NO) for ; Sun, 10 Sep 2017 14:19:19 +0000 (UTC) Received: by mail-oi0-f54.google.com with SMTP id r20so34892545oie.0 for ; Sun, 10 Sep 2017 07:26:16 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <07621915-a53a-03f3-9c75-b7e7d188d109@gmail.com> References: <1e1740c3-f8cf-2ee2-d842-749b687cb737@gmail.com> <07621915-a53a-03f3-9c75-b7e7d188d109@gmail.com> From: "Jason A. Donenfeld" Date: Sun, 10 Sep 2017 16:26:14 +0200 Message-ID: Subject: Re: Timing issue (?) with wg-quick up on Raspberry Pi B+ To: Jim Darby Content-Type: text/plain; charset="UTF-8" Cc: WireGuard mailing list List-Id: Development discussion of WireGuard List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On Sun, Sep 10, 2017 at 3:09 PM, Jim Darby wrote: > However, your comment about network management daemons running is most > interesting. Here's an extract from journalctl's output: > Sep 09 21:31:28 janus ifplugd(wg0)[6903]: Executing > '/etc/ifplugd/ifplugd.action wg0 up'. > Sep 09 21:31:28 janus ifplugd(wg0)[6903]: client: Ignoring unknown interface > wg0=wg0. That is interesting. Thanks for that. Indeed it looks like ifplugd is just calling ifup wg0, and I'm not totally sure why that would remove an IP address if there's nothing in /etc/network/interfaces, though I'm not a huge Debian person so there could be a detail I'm overlooking. Another more direct way that might help debug this is `ip monitor all`. On my (working) system, running `ip monitor all` in one window and `wg-quick up martino` in another yields this: [NETCONF]ipv4 dev martino forwarding off rp_filter loose mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off [NETCONF]ipv6 dev martino forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off [LINK]107: martino: mtu 1420 qdisc noop state DOWN group default link/none [ADDR]107: martino inet 10.10.11.100/32 scope global martino valid_lft forever preferred_lft forever [ROUTE]local 10.10.11.100 dev martino table local proto kernel scope host src 10.10.11.100 [ROUTE]ff00::/8 dev martino table local metric 256 linkdown pref medium [ROUTE]2a01:e35:8be7:9122:100::/96 dev martino proto kernel metric 256 linkdown pref medium [ADDR]107: martino inet6 2a01:e35:8be7:9122:100::1/96 scope global valid_lft forever preferred_lft forever [ROUTE]local 2a01:e35:8be7:9122:100::1 dev lo table local proto kernel metric 0 pref medium [LINK]107: martino: mtu 1420 qdisc noqueue state UNKNOWN group default link/none [ROUTE]default dev martino table 51820 metric 1024 pref medium [RULE]32765: not from all fwmark 0xca6c lookup 51820 [RULE]32764: from all lookup main suppress_prefixlength 0 [ROUTE]default dev martino table 51820 scope link [RULE]32765: not from all fwmark 0xca6c lookup 51820 [RULE]32764: from all lookup main suppress_prefixlength 0 If I then type in `ip addr flush dev martino`, I get this: [ADDR]Deleted 107: martino inet 10.10.11.100/32 scope global martino valid_lft forever preferred_lft forever [ROUTE]Deleted local 10.10.11.100 dev martino table local proto kernel scope host src 10.10.11.100 [NEIGH]Deleted 10.10.11.1 dev martino lladdr 08 NOARP [NEIGH]Deleted 66.102.1.127 dev martino lladdr 08 NOARP [NEIGH]Deleted 52.205.56.176 dev martino lladdr 08 NOARP [ADDR]Deleted 107: martino inet6 2a01:e35:8be7:9122:100::1/96 scope global valid_lft forever preferred_lft forever [ROUTE]Deleted local 2a01:e35:8be7:9122:100::1 dev lo table local proto kernel metric 0 pref medium [ROUTE]Deleted 2a01:e35:8be7:9122:100::/96 dev martino proto kernel metric 256 pref medium So, it remains to be seen whether or not something else in userspace is actually interacting with the interface. Once we figure out what, we might be able to monitor all callers of those netlink commands. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: uberscubajim@gmail.com Received: from krantz.zx2c4.com (localhost [127.0.0.1]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id eeed324e for ; Sun, 10 Sep 2017 14:42:51 +0000 (UTC) Received: from mail-wm0-f52.google.com (mail-wm0-f52.google.com [74.125.82.52]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id af79ef83 for ; Sun, 10 Sep 2017 14:42:50 +0000 (UTC) Received: by mail-wm0-f52.google.com with SMTP id i189so25250633wmf.1 for ; Sun, 10 Sep 2017 08:08:50 -0700 (PDT) Return-Path: Subject: Re: Timing issue (?) with wg-quick up on Raspberry Pi B+ To: "Jason A. Donenfeld" References: <1e1740c3-f8cf-2ee2-d842-749b687cb737@gmail.com> <07621915-a53a-03f3-9c75-b7e7d188d109@gmail.com> From: Jim Darby Message-ID: Date: Sun, 10 Sep 2017 16:08:48 +0100 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------2078686C97A9ACA4CCCDA0C5" Cc: WireGuard mailing list List-Id: Development discussion of WireGuard List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , This is a multi-part message in MIME format. --------------2078686C97A9ACA4CCCDA0C5 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit The plot thickens! Here's the output from ip monitor all: [NETCONF]ipv4 dev wg0 forwarding on rp_filter strict mc_forwarding 0 proxy_neigh off [NETCONF]ipv6 dev wg0 forwarding off mc_forwarding 0 proxy_neigh off [LINK]27: wg0: mtu 1420 qdisc noop state DOWN group default link/none [ADDR]27: wg0 inet 192.168.2.3/32 scope global wg0 valid_lft forever preferred_lft forever [ROUTE]local 192.168.2.3 dev wg0 table local proto kernel scope host src 192.168.2.3 [ADDR]Deleted 27: wg0 inet 192.168.2.3/32 scope global wg0 valid_lft forever preferred_lft forever [ROUTE]Deleted local 192.168.2.3 dev wg0 table local proto kernel scope host src 192.168.2.3 [LINK]27: wg0: mtu 1420 qdisc noqueue state UNKNOWN group default link/none [ROUTE]ff00::/8 dev wg0 table local metric 256 [LINK]27: wg0: mtu 1420 state UNKNOWN link/none [LINK]27: wg0: mtu 1420 qdisc noqueue state UNKNOWN group default link/none [ROUTE]192.168.2.0/24 dev wg0 scope link The route delete on the sixth line is most interesting. I wonder where it came from? I'll try it one some Debian 8 machines and see what happens. So far it's /only/ Debain 9 that seems to have the issue. Jim. On 10/09/17 15:26, Jason A. Donenfeld wrote: > On Sun, Sep 10, 2017 at 3:09 PM, Jim Darby wrote: >> However, your comment about network management daemons running is most >> interesting. Here's an extract from journalctl's output: >> Sep 09 21:31:28 janus ifplugd(wg0)[6903]: Executing >> '/etc/ifplugd/ifplugd.action wg0 up'. >> Sep 09 21:31:28 janus ifplugd(wg0)[6903]: client: Ignoring unknown interface >> wg0=wg0. > That is interesting. Thanks for that. Indeed it looks like ifplugd is > just calling ifup wg0, and I'm not totally sure why that would remove > an IP address if there's nothing in /etc/network/interfaces, though > I'm not a huge Debian person so there could be a detail I'm > overlooking. > > Another more direct way that might help debug this is `ip monitor > all`. On my (working) system, running `ip monitor all` in one window > and `wg-quick up martino` in another yields this: > > [NETCONF]ipv4 dev martino forwarding off rp_filter loose mc_forwarding > off proxy_neigh off ignore_routes_with_linkdown off > [NETCONF]ipv6 dev martino forwarding off mc_forwarding off proxy_neigh > off ignore_routes_with_linkdown off > [LINK]107: martino: mtu 1420 qdisc noop > state DOWN group default > link/none > [ADDR]107: martino inet 10.10.11.100/32 scope global martino > valid_lft forever preferred_lft forever > [ROUTE]local 10.10.11.100 dev martino table local proto kernel scope > host src 10.10.11.100 > [ROUTE]ff00::/8 dev martino table local metric 256 linkdown pref medium > [ROUTE]2a01:e35:8be7:9122:100::/96 dev martino proto kernel metric 256 > linkdown pref medium > [ADDR]107: martino inet6 2a01:e35:8be7:9122:100::1/96 scope global > valid_lft forever preferred_lft forever > [ROUTE]local 2a01:e35:8be7:9122:100::1 dev lo table local proto kernel > metric 0 pref medium > [LINK]107: martino: mtu 1420 qdisc > noqueue state UNKNOWN group default > link/none > [ROUTE]default dev martino table 51820 metric 1024 pref medium > [RULE]32765: not from all fwmark 0xca6c lookup 51820 > [RULE]32764: from all lookup main suppress_prefixlength 0 > [ROUTE]default dev martino table 51820 scope link > [RULE]32765: not from all fwmark 0xca6c lookup 51820 > [RULE]32764: from all lookup main suppress_prefixlength 0 > > If I then type in `ip addr flush dev martino`, I get this: > > [ADDR]Deleted 107: martino inet 10.10.11.100/32 scope global martino > valid_lft forever preferred_lft forever > [ROUTE]Deleted local 10.10.11.100 dev martino table local proto kernel > scope host src 10.10.11.100 > [NEIGH]Deleted 10.10.11.1 dev martino lladdr 08 NOARP > [NEIGH]Deleted 66.102.1.127 dev martino lladdr 08 NOARP > [NEIGH]Deleted 52.205.56.176 dev martino lladdr 08 NOARP > [ADDR]Deleted 107: martino inet6 2a01:e35:8be7:9122:100::1/96 scope global > valid_lft forever preferred_lft forever > [ROUTE]Deleted local 2a01:e35:8be7:9122:100::1 dev lo table local > proto kernel metric 0 pref medium > [ROUTE]Deleted 2a01:e35:8be7:9122:100::/96 dev martino proto kernel > metric 256 pref medium > > So, it remains to be seen whether or not something else in userspace > is actually interacting with the interface. Once we figure out what, > we might be able to monitor all callers of those netlink commands. --------------2078686C97A9ACA4CCCDA0C5 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit

The plot thickens! Here's the output from ip monitor all:

[NETCONF]ipv4 dev wg0 forwarding on rp_filter strict mc_forwarding 0 proxy_neigh off
[NETCONF]ipv6 dev wg0 forwarding off mc_forwarding 0 proxy_neigh off
[LINK]27: wg0: <POINTOPOINT,NOARP,200000> mtu 1420 qdisc noop state DOWN group default
    link/none
[ADDR]27: wg0    inet 192.168.2.3/32 scope global wg0
       valid_lft forever preferred_lft forever
[ROUTE]local 192.168.2.3 dev wg0 table local proto kernel scope host src 192.168.2.3
[ADDR]Deleted 27: wg0    inet 192.168.2.3/32 scope global wg0
       valid_lft forever preferred_lft forever
[ROUTE]Deleted local 192.168.2.3 dev wg0 table local proto kernel scope host src 192.168.2.3
[LINK]27: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default
    link/none
[ROUTE]ff00::/8 dev wg0 table local metric 256
[LINK]27: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 state UNKNOWN
    link/none
[LINK]27: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default
    link/none
[ROUTE]192.168.2.0/24 dev wg0 scope link

The route delete on the sixth line is most interesting. I wonder where it came from? I'll try it one some Debian 8 machines and see what happens. So far it's only Debain 9 that seems to have the issue.

Jim.

On 10/09/17 15:26, Jason A. Donenfeld wrote:

On Sun, Sep 10, 2017 at 3:09 PM, Jim Darby <uberscubajim@gmail.com> wrote:

However, your comment about network management daemons running is most
interesting. Here's an extract from journalctl's output:
Sep 09 21:31:28 janus ifplugd(wg0)[6903]: Executing
'/etc/ifplugd/ifplugd.action wg0 up'.
Sep 09 21:31:28 janus ifplugd(wg0)[6903]: client: Ignoring unknown interface
wg0=wg0.

That is interesting. Thanks for that. Indeed it looks like ifplugd is
just calling ifup wg0, and I'm not totally sure why that would remove
an IP address if there's nothing in /etc/network/interfaces, though
I'm not a huge Debian person so there could be a detail I'm
overlooking.

Another more direct way that might help debug this is `ip monitor
all`. On my (working) system, running `ip monitor all` in one window
and `wg-quick up martino` in another yields this:

[NETCONF]ipv4 dev martino forwarding off rp_filter loose mc_forwarding
off proxy_neigh off ignore_routes_with_linkdown off
[NETCONF]ipv6 dev martino forwarding off mc_forwarding off proxy_neigh
off ignore_routes_with_linkdown off
[LINK]107: martino: <POINTOPOINT,NOARP,200000> mtu 1420 qdisc noop
state DOWN group default
   link/none
[ADDR]107: martino    inet 10.10.11.100/32 scope global martino
      valid_lft forever preferred_lft forever
[ROUTE]local 10.10.11.100 dev martino table local proto kernel scope
host src 10.10.11.100
[ROUTE]ff00::/8 dev martino table local metric 256 linkdown pref medium
[ROUTE]2a01:e35:8be7:9122:100::/96 dev martino proto kernel metric 256
linkdown pref medium
[ADDR]107: martino    inet6 2a01:e35:8be7:9122:100::1/96 scope global
      valid_lft forever preferred_lft forever
[ROUTE]local 2a01:e35:8be7:9122:100::1 dev lo table local proto kernel
metric 0 pref medium
[LINK]107: martino: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc
noqueue state UNKNOWN group default
   link/none
[ROUTE]default dev martino table 51820 metric 1024 pref medium
[RULE]32765:    not from all fwmark 0xca6c lookup 51820
[RULE]32764:    from all lookup main suppress_prefixlength 0
[ROUTE]default dev martino table 51820 scope link
[RULE]32765:    not from all fwmark 0xca6c lookup 51820
[RULE]32764:    from all lookup main suppress_prefixlength 0

If I then type in `ip addr flush dev martino`, I get this:

[ADDR]Deleted 107: martino    inet 10.10.11.100/32 scope global martino
      valid_lft forever preferred_lft forever
[ROUTE]Deleted local 10.10.11.100 dev martino table local proto kernel
scope host src 10.10.11.100
[NEIGH]Deleted 10.10.11.1 dev martino lladdr 08 NOARP
[NEIGH]Deleted 66.102.1.127 dev martino lladdr 08 NOARP
[NEIGH]Deleted 52.205.56.176 dev martino lladdr 08 NOARP
[ADDR]Deleted 107: martino    inet6 2a01:e35:8be7:9122:100::1/96 scope global
      valid_lft forever preferred_lft forever
[ROUTE]Deleted local 2a01:e35:8be7:9122:100::1 dev lo table local
proto kernel metric 0 pref medium
[ROUTE]Deleted 2a01:e35:8be7:9122:100::/96 dev martino proto kernel
metric 256 pref medium

So, it remains to be seen whether or not something else in userspace
is actually interacting with the interface. Once we figure out what,
we might be able to monitor all callers of those netlink commands.

--------------2078686C97A9ACA4CCCDA0C5-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Jason@zx2c4.com Received: from krantz.zx2c4.com (localhost [127.0.0.1]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 4212b504 for ; Sun, 10 Sep 2017 21:59:08 +0000 (UTC) Received: from frisell.zx2c4.com (frisell.zx2c4.com [192.95.5.64]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id bee4a7b0 for ; Sun, 10 Sep 2017 21:59:08 +0000 (UTC) Received: by frisell.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 2ce0664b for ; Sun, 10 Sep 2017 22:18:11 +0000 (UTC) Received: by frisell.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id b36aa8dc (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128:NO) for ; Sun, 10 Sep 2017 22:18:11 +0000 (UTC) Received: by mail-io0-f178.google.com with SMTP id v36so3059389ioi.1 for ; Sun, 10 Sep 2017 15:25:09 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: References: <1e1740c3-f8cf-2ee2-d842-749b687cb737@gmail.com> <07621915-a53a-03f3-9c75-b7e7d188d109@gmail.com>

From: "Jason A. Donenfeld" Date: Mon, 11 Sep 2017 00:25:08 +0200 Message-ID: Subject: Re: Timing issue (?) with wg-quick up on Raspberry Pi B+ To: Jim Darby Content-Type: multipart/mixed; boundary="001a113d613c6445370558dd48aa" Cc: WireGuard mailing list List-Id: Development discussion of WireGuard List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , --001a113d613c6445370558dd48aa Content-Type: text/plain; charset="UTF-8" I just wrote the attached script, which will tell you all the processes that have an open netlink socket... There's a bit too much fluff in there on a desktop system, but I imagine the pi might help boil it down to a few candidates. Probably we'll determine it's an ifplugd/ifupdown.action thing, but we'll see. --001a113d613c6445370558dd48aa Content-Type: application/x-sh; name="netlistenwho.sh" Content-Disposition: attachment; filename="netlistenwho.sh" Content-Transfer-Encoding: base64 X-Attachment-Id: f_j7fb33j70 IyEvYmluL2Jhc2gKCltbICRVSUQgPT0gMCBdXSB8fCB7IGVjaG8gIlJ1biBtZSBhcyByb290ISI7 IGV4aXQgMTsgfQoKZGVjbGFyZSAtQSBzb2NrZXRzCndoaWxlIHJlYWQgLXIgZmlsZSBzb2NrOyBk bwoJW1sgJGZpbGUgPX4gL3Byb2MvKFswLTldKykvZmQvWzAtOV0rIF1dIHx8IGNvbnRpbnVlCglw aWQ9IiR7QkFTSF9SRU1BVENIWzFdfSIKCVtbICRzb2NrID1+IHNvY2tldDpcWyhbMC05XSspXF0g XV0gfHwgY29udGludWUKCWlub2RlPSIke0JBU0hfUkVNQVRDSFsxXX0iCglzb2NrZXRzWyRpbm9k ZV09JHBpZApkb25lIDwgPChmaW5kIC1QIC9wcm9jIC1tYXhkZXB0aCAzIC1taW5kZXB0aCAzIC1s bmFtZSAnc29ja2V0OlxbKlxdJyAtcHJpbnRmICclcCAlbFxuJyAyPi9kZXYvbnVsbCkKCmRlY2xh cmUgLWEgcGlkcwp3aGlsZSByZWFkIC1yIF8gXyBfIF8gXyBfIF8gXyBfIGlub2RlOyBkbwoJW1sg JGlub2RlID09IElub2RlIF1dICYmIGNvbnRpbnVlCglwaWRzKz0oICR7c29ja2V0c1skaW5vZGVd fSApCmRvbmUgPCAvcHJvYy9uZXQvbmV0bGluawoKcHMgIiR7cGlkc1tAXX0iCg== --001a113d613c6445370558dd48aa-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Jason@zx2c4.com Received: from krantz.zx2c4.com (localhost [127.0.0.1]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 2abaca75 for ; Sun, 10 Sep 2017 22:04:46 +0000 (UTC) Received: from frisell.zx2c4.com (frisell.zx2c4.com [192.95.5.64]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id bf2b3ed1 for ; Sun, 10 Sep 2017 22:04:46 +0000 (UTC) Received: by frisell.zx2c4.com (ZX2C4 Mail Server) with ESMTP id f30e51d1 for ; Sun, 10 Sep 2017 22:23:49 +0000 (UTC) Received: by frisell.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id 6eadcef8 (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128:NO) for ; Sun, 10 Sep 2017 22:23:48 +0000 (UTC) Received: by mail-io0-f174.google.com with SMTP id v36so3090549ioi.1 for ; Sun, 10 Sep 2017 15:30:47 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: References: <1e1740c3-f8cf-2ee2-d842-749b687cb737@gmail.com> <07621915-a53a-03f3-9c75-b7e7d188d109@gmail.com>

From: "Jason A. Donenfeld" Date: Mon, 11 Sep 2017 00:30:46 +0200 Message-ID: Subject: Re: Timing issue (?) with wg-quick up on Raspberry Pi B+ To: Jim Darby Content-Type: text/plain; charset="UTF-8" Cc: WireGuard mailing list List-Id: Development discussion of WireGuard List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Another "direct" debugging thing you could do is: $ ip link add nlmon0 type nlmon $ ip link set nlmon0 set up And then start looking at the output in Wireshark. When you see the obvious culprit (a rtnetlink message deleting the address), look at the port ID. Then look in /proc/net/netlink to correlate the port id ("Pid") with an inode ("Inode"). Then look at the symbolic link targets of /proc/{pid}/fd/[0-9]+ to see which pid has a fd belonging to that inode. Then we'll know for certain. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: uberscubajim@gmail.com Received: from krantz.zx2c4.com (localhost [127.0.0.1]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 2e2aa285 for ; Sun, 10 Sep 2017 22:57:46 +0000 (UTC) Received: from mail-wr0-f182.google.com (mail-wr0-f182.google.com [209.85.128.182]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id fd140790 for ; Sun, 10 Sep 2017 22:57:46 +0000 (UTC) Received: by mail-wr0-f182.google.com with SMTP id k20so11429359wre.4 for ; Sun, 10 Sep 2017 16:23:48 -0700 (PDT) Return-Path: Message-ID: <59B5C95F.50508@gmail.com> Date: Mon, 11 Sep 2017 00:23:11 +0100 From: Jim Darby MIME-Version: 1.0 To: "Jason A. Donenfeld" Subject: Re: Timing issue (?) with wg-quick up on Raspberry Pi B+ References: <1e1740c3-f8cf-2ee2-d842-749b687cb737@gmail.com> <07621915-a53a-03f3-9c75-b7e7d188d109@gmail.com>

In-Reply-To: Content-Type: multipart/alternative; boundary="------------080505040809050809020201" Cc: WireGuard mailing list List-Id: Development discussion of WireGuard List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , This is a multi-part message in MIME format. --------------080505040809050809020201 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 10/09/17 23:25, Jason A. Donenfeld wrote: > I just wrote the attached script, which will tell you all the > processes that have an open netlink socket... > > There's a bit too much fluff in there on a desktop system, but I > imagine the pi might help boil it down to a few candidates. Probably > we'll determine it's an ifplugd/ifupdown.action thing, but we'll see. Many thanks again for such great work. Here's the output of the netlistenerwho.sh program. It's awash with potential culprits! PID TTY STAT TIME COMMAND 1 ? Ss 2:35 /sbin/init 128 ? Ss 0:01 /lib/systemd/systemd-udevd 770 ? Ss 2:24 /sbin/dhcpcd -q -b 783 ? Ss 0:15 avahi-daemon: running [janus.local] 840 ? Ss 0:32 /lib/systemd/systemd-logind 896 ? Ss 10:19 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 107:112 906 ? S 0:02 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.pid -u dnsmasq -r /run/dnsmasq/resolv.conf -7 /etc/dnsmasq. 1204 ? Ss 0:00 /lib/systemd/systemd --user 18942 ? Ss 0:00 /lib/systemd/systemd --user I also made a great mistake earlier: the problematic system is the version of Raspbian based on begin *8* and /not/ *9*. Debian 9 works fine! I've tweaked the /etc/network/interfaces file to have the line “iface wg0 inet manual” in it so we /shouldn't/ get DHCP running or anything… I tried the nlmon trick but I got the response “RTNETLINK answers: Operation not supported” which isn't too helpful. After some more playing I've found that running the wg-quick script's commands by hand works. But then, I'm /manually/ entering them and that affects timing. I note you perform the ”ip link set wg0 up” /after/ the “ip address add 192.168.2.3/32 dev wg0” and “ip link set mtu 1420 dev wg0”. /However, /the act of creating the interface with the “ip link add wg0 type wireguard” seems to trigger the ip up automatically. The log files give: Sep 10 23:57:51 janus kernel: wireguard: WireGuard 0.0.20170907 loaded. See www.wireguard.com for information. Sep 10 23:57:51 janus kernel: wireguard: Copyright (C) 2015-2017 Jason A. Donenfeld . All Rights Reserved. Sep 10 23:57:51 janus ifplugd(wg0)[14109]: ifplugd 0.28 initializing. Sep 10 23:57:51 janus ifplugd(wg0)[14109]: Using interface wg0/00:00:00:00:00:00 Sep 10 23:57:51 janus ifplugd(wg0)[14109]: Using detection mode: IFF_RUNNING Sep 10 23:57:51 janus ifplugd(wg0)[14109]: Initialization complete, link beat detected. Sep 10 23:57:52 janus ifplugd(wg0)[14109]: Executing '/etc/ifplugd/ifplugd.action wg0 up'. Sep 10 23:57:52 janus ifplugd(wg0)[14109]: client: /sbin/ifup: interface wg0 already configured Sep 10 23:57:52 janus ifplugd(wg0)[14109]: Program executed successfully. Which could well be interesting. I manually ran ifdown then ifup on wg0 and it /didn't/ lose its IP address. Most perplexing! Jim. --------------080505040809050809020201 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

On 10/09/17 23:25, Jason A. Donenfeld wrote:

I just wrote the attached script, which will tell you all the
processes that have an open netlink socket...

There's a bit too much fluff in there on a desktop system, but I
imagine the pi might help boil it down to a few candidates. Probably
we'll determine it's an ifplugd/ifupdown.action thing, but we'll see.

Many thanks again for such great work. Here's the output of the netlistenerwho.sh program. It's awash with potential culprits!

PID TTY STAT TIME COMMAND 1 ? Ss 2:35 /sbin/init 128 ? Ss 0:01 /lib/systemd/systemd-udevd 770 ? Ss 2:24 /sbin/dhcpcd -q -b 783 ? Ss 0:15 avahi-daemon: running [janus.local] 840 ? Ss 0:32 /lib/systemd/systemd-logind 896 ? Ss 10:19 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 107:112 906 ? S 0:02 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.pid -u dnsmasq -r /run/dnsmasq/resolv.conf -7 /etc/dnsmasq. 1204 ? Ss 0:00 /lib/systemd/systemd --user18942 ? Ss 0:00 /lib/systemd/systemd --user

I also made a great mistake earlier: the problematic system is the version of Raspbian based on begin 8 and not 9. Debian 9 works fine!

I've tweaked the /etc/network/interfaces file to have the line “iface wg0 inet manual” in it so we shouldn't get DHCP running or anything…

I tried the nlmon trick but I got the response “RTNETLINK answers: Operation not supported” which isn't too helpful.

After some more playing I've found that running the wg-quick script's commands by hand works. But then, I'm manually entering them and that affects timing.

I note you perform the ”ip link set wg0 up” after the “ip address add 192.168.2.3/32 dev wg0” and “ip link set mtu 1420 dev wg0”. However, the act of creating the interface with the “ip link add wg0 type wireguard” seems to trigger the ip up automatically. The log files give:

Sep 10 23:57:51 janus kernel: wireguard: WireGuard 0.0.20170907 loaded. See www.wireguard.com for information.Sep 10 23:57:51 janus kernel: wireguard: Copyright (C) 2015-2017 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.Sep 10 23:57:51 janus ifplugd(wg0)[14109]: ifplugd 0.28 initializing.Sep 10 23:57:51 janus ifplugd(wg0)[14109]: Using interface wg0/00:00:00:00:00:00Sep 10 23:57:51 janus ifplugd(wg0)[14109]: Using detection mode: IFF_RUNNINGSep 10 23:57:51 janus ifplugd(wg0)[14109]: Initialization complete, link beat detected.Sep 10 23:57:52 janus ifplugd(wg0)[14109]: Executing '/etc/ifplugd/ifplugd.action wg0 up'.Sep 10 23:57:52 janus ifplugd(wg0)[14109]: client: /sbin/ifup: interface wg0 already configuredSep 10 23:57:52 janus ifplugd(wg0)[14109]: Program executed successfully.

Which could well be interesting. I manually ran ifdown then ifup on wg0 and it didn't lose its IP address.

Most perplexing!

Jim.
--------------080505040809050809020201-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Jason@zx2c4.com Received: from krantz.zx2c4.com (localhost [127.0.0.1]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 7b299863 for ; Mon, 11 Sep 2017 00:26:25 +0000 (UTC) Received: from frisell.zx2c4.com (frisell.zx2c4.com [192.95.5.64]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 44ebaeb0 for ; Mon, 11 Sep 2017 00:26:22 +0000 (UTC) Received: by frisell.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 54f4a7dd for ; Mon, 11 Sep 2017 00:45:25 +0000 (UTC) Received: by frisell.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id 1e75ef4d (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128:NO) for ; Mon, 11 Sep 2017 00:45:25 +0000 (UTC) Received: by mail-io0-f176.google.com with SMTP id y123so16933642iod.0 for ; Sun, 10 Sep 2017 17:52:24 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <59B5C95F.50508@gmail.com> References: <1e1740c3-f8cf-2ee2-d842-749b687cb737@gmail.com> <07621915-a53a-03f3-9c75-b7e7d188d109@gmail.com>

<59B5C95F.50508@gmail.com> From: "Jason A. Donenfeld" Date: Mon, 11 Sep 2017 02:52:22 +0200 Message-ID: Subject: Re: Timing issue (?) with wg-quick up on Raspberry Pi B+ To: Jim Darby Content-Type: multipart/alternative; boundary="001a1141a45cfc29ac0558df560b" Cc: WireGuard mailing list List-Id: Development discussion of WireGuard List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , --001a1141a45cfc29ac0558df560b Content-Type: text/plain; charset="UTF-8" I figured it out. ifplugd clears all addresses when bringing up an interface: void interface_up(int fd, char *iface) { ... ((struct sockaddr_in *)(&ifr.ifr_addr))->sin_addr.s_addr = INADDR_ANY; if (ioctl(fd, SIOCSIFADDR, &ifr) < 0) { ... } Since wg-quick sets the address before bringing it up, this running in between is problematic. One workaround in wg-quick would be for me to reverse the order of setting the IP and bringing it up, but this introduces other races I'm not very fond of introducing. So, rather, we should address the larger question: why on earth is ifplugd being started when wg0 is added? Sep 10 23:57:51 janus ifplugd(wg0)[14109]: ifplugd 0.28 initializing. Sep 10 23:57:51 janus ifplugd(wg0)[14109]: Using interface wg0/00:00:00:00:00:00 Sep 10 23:57:51 janus ifplugd(wg0)[14109]: Using detection mode: IFF_RUNNING What causes it to be launched here? Digging a bit deeper, it looks like ifplugd is being launched by a udev rule which calls a Debian file called ifplugd.agent. It is in here that unholy hotplugging occurs. I don't actually have a Debian system running to fish around and see, but my guess is that you have a file /etc/default/ifplugd that has in it HOTPLUG_INTERFACES=all. If you change this to HOTPLUG_INTERFACES="wlan0 eth0" or something more restrictive, things might work better. Just a guess. --001a1141a45cfc29ac0558df560b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

I figured it out. ifplugd clears all addresses when bringi= ng up an interface:

void interface_up(int fd, char = *iface) {

...

=C2=A0 =C2=A0 =C2=A0 =C2=A0 ((struct sock= addr_in *)(&ifr.ifr_addr))->sin_addr.s_addr =3D INADDR_ANY;

=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (ioctl(fd, SIOCSIFADDR, &ifr) < 0) = {

...

}

Since wg-quick s= ets the address before bringing it up, this running in between is problemat= ic. One workaround in wg-quick would be for me to reverse the order of sett= ing the IP and bringing it up, but this introduces other races I'm not = very fond of introducing. So, rather, we should address the larger question= : why on earth is ifplugd being started when wg0 is added?

<= br>

Sep 10 23:57:51 janus ifplugd(wg0)[14109]: ifplugd 0.28 initi= alizing.

Sep 10 23:57:51 janus ifplugd(wg0)[14109]: Using interfa= ce wg0/00:00:00:00:00:00

Sep 10 23:57:51 janus ifplugd(wg0)[14109= ]: Using detection mode: IFF_RUNNING

What ca= uses it to be launched here? Digging a bit deeper, it looks like ifplugd is= being launched by a udev rule which calls a Debian file called ifplugd.age= nt. It is in here that unholy hotplugging occurs.

= I don't actually have a Debian system running to fish around and see, b= ut my guess is that you have a file /etc/default/ifplugd that has in it HOT= PLUG_INTERFACES=3Dall. If you change this to HOTPLUG_INTERFACES=3D"wla= n0 eth0" or something more restrictive, things might work better. Just= a guess.

--001a1141a45cfc29ac0558df560b-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: uberscubajim@gmail.com Received: from krantz.zx2c4.com (localhost [127.0.0.1]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id da588ee9 for ; Mon, 11 Sep 2017 12:09:50 +0000 (UTC) Received: from mail-wr0-f181.google.com (mail-wr0-f181.google.com [209.85.128.181]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id 0d84a1dc for ; Mon, 11 Sep 2017 12:09:50 +0000 (UTC) Received: by mail-wr0-f181.google.com with SMTP id o42so14280194wrb.3 for ; Mon, 11 Sep 2017 05:35:56 -0700 (PDT) Return-Path: Subject: Re: Timing issue (?) with wg-quick up on Raspberry Pi B+ To: "Jason A. Donenfeld" References: <1e1740c3-f8cf-2ee2-d842-749b687cb737@gmail.com> <07621915-a53a-03f3-9c75-b7e7d188d109@gmail.com>

<59B5C95F.50508@gmail.com> From: Jim Darby Message-ID: Date: Mon, 11 Sep 2017 13:35:54 +0100 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------C49051A3A7A4AA368735E5AC" Cc: WireGuard mailing list List-Id: Development discussion of WireGuard List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , This is a multi-part message in MIME format. --------------C49051A3A7A4AA368735E5AC Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Nailed it! Specifically, you were spot on about the /etc/default/ifplugd file and its HOTPLUG_INTERFACES line. Changing that to HOTPLUG_INTERFACES="eth0 eth1" meant that it just worked out of the box. In fact, I suspect that eth0 could be omitted because it's part of the hardware (eth1 is a USB dongle so hotplugging may be useful). I'd prefer a way to specifically /exclude/ wg* interfaces rather then include the known ones but that may be too complex. I hope this discussion and its solution will be a useful part of the WireGuard documentation. It'll hopefully stop others becoming confused in the future. Many thanks for your help in resolving this! I'm very much enjoying playing with WireGuard. I heard about it during your talk at FOSDEM by the way. Regards, Jim. On 11/09/17 01:52, Jason A. Donenfeld wrote: > I figured it out. ifplugd clears all addresses when bringing up an > interface: > > void interface_up(int fd, char *iface) { > ... > ((struct sockaddr_in *)(&ifr.ifr_addr))->sin_addr.s_addr = > INADDR_ANY; > if (ioctl(fd, SIOCSIFADDR, &ifr) < 0) { > ... > } > > Since wg-quick sets the address before bringing it up, this running in > between is problematic. One workaround in wg-quick would be for me to > reverse the order of setting the IP and bringing it up, but this > introduces other races I'm not very fond of introducing. So, rather, > we should address the larger question: why on earth is ifplugd being > started when wg0 is added? > > Sep 10 23:57:51 janus ifplugd(wg0)[14109]: ifplugd 0.28 initializing. > Sep 10 23:57:51 janus ifplugd(wg0)[14109]: Using interface > wg0/00:00:00:00:00:00 > Sep 10 23:57:51 janus ifplugd(wg0)[14109]: Using detection mode: > IFF_RUNNING > > What causes it to be launched here? Digging a bit deeper, it looks > like ifplugd is being launched by a udev rule which calls a Debian > file called ifplugd.agent. It is in here that unholy hotplugging occurs. > > I don't actually have a Debian system running to fish around and see, > but my guess is that you have a file /etc/default/ifplugd that has in > it HOTPLUG_INTERFACES=all. If you change this to > HOTPLUG_INTERFACES="wlan0 eth0" or something more restrictive, things > might work better. Just a guess. --------------C49051A3A7A4AA368735E5AC Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit

Nailed it!

Specifically, you were spot on about the /etc/default/ifplugd file and its HOTPLUG_INTERFACES line. Changing that to HOTPLUG_INTERFACES="eth0 eth1" meant that it just worked out of the box.

In fact, I suspect that eth0 could be omitted because it's part of the hardware (eth1 is a USB dongle so hotplugging may be useful). I'd prefer a way to specifically exclude wg* interfaces rather then include the known ones but that may be too complex.

I hope this discussion and its solution will be a useful part of the WireGuard documentation. It'll hopefully stop others becoming confused in the future.

Many thanks for your help in resolving this! I'm very much enjoying playing with WireGuard. I heard about it during your talk at FOSDEM by the way.

Regards,

Jim.

On 11/09/17 01:52, Jason A. Donenfeld wrote:

I figured it out. ifplugd clears all addresses when bringing up an interface:

void interface_up(int fd, char *iface) {

...

((struct sockaddr_in *)(&ifr.ifr_addr))->sin_addr.s_addr = INADDR_ANY;

if (ioctl(fd, SIOCSIFADDR, &ifr) < 0) {

...

}

Since wg-quick sets the address before bringing it up, this running in between is problematic. One workaround in wg-quick would be for me to reverse the order of setting the IP and bringing it up, but this introduces other races I'm not very fond of introducing. So, rather, we should address the larger question: why on earth is ifplugd being started when wg0 is added?

Sep 10 23:57:51 janus ifplugd(wg0)[14109]: ifplugd 0.28 initializing.

Sep 10 23:57:51 janus ifplugd(wg0)[14109]: Using interface wg0/00:00:00:00:00:00

Sep 10 23:57:51 janus ifplugd(wg0)[14109]: Using detection mode: IFF_RUNNING

What causes it to be launched here? Digging a bit deeper, it looks like ifplugd is being launched by a udev rule which calls a Debian file called ifplugd.agent. It is in here that unholy hotplugging occurs.

I don't actually have a Debian system running to fish around and see, but my guess is that you have a file /etc/default/ifplugd that has in it HOTPLUG_INTERFACES=all. If you change this to HOTPLUG_INTERFACES="wlan0 eth0" or something more restrictive, things might work better. Just a guess.

--------------C49051A3A7A4AA368735E5AC-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Jason@zx2c4.com Received: from krantz.zx2c4.com (localhost [127.0.0.1]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id e4528521 for ; Mon, 11 Sep 2017 12:11:52 +0000 (UTC) Received: from frisell.zx2c4.com (frisell.zx2c4.com [192.95.5.64]) by krantz.zx2c4.com (ZX2C4 Mail Server) with ESMTP id b3ee5e97 for ; Mon, 11 Sep 2017 12:11:52 +0000 (UTC) Received: by frisell.zx2c4.com (ZX2C4 Mail Server) with ESMTP id be564ba6 for ; Mon, 11 Sep 2017 12:30:55 +0000 (UTC) Received: by frisell.zx2c4.com (ZX2C4 Mail Server) with ESMTPSA id f968b7f7 (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128:NO) for ; Mon, 11 Sep 2017 12:30:54 +0000 (UTC) Received: by mail-io0-f174.google.com with SMTP id v36so12135730ioi.1 for ; Mon, 11 Sep 2017 05:37:57 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: References: <1e1740c3-f8cf-2ee2-d842-749b687cb737@gmail.com> <07621915-a53a-03f3-9c75-b7e7d188d109@gmail.com>

<59B5C95F.50508@gmail.com>

From: "Jason A. Donenfeld" Date: Mon, 11 Sep 2017 14:37:56 +0200 Message-ID: Subject: Re: Timing issue (?) with wg-quick up on Raspberry Pi B+ To: Jim Darby Content-Type: text/plain; charset="UTF-8" Cc: WireGuard mailing list List-Id: Development discussion of WireGuard List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Hey Jim, Great to hear this resolved things. > I'm very much enjoying playing with WireGuard. > I heard about it during your talk at FOSDEM by the way. In case you didn't get any stickers there, I'm mailing them out via: https://lists.zx2c4.com/pipermail/wireguard/2017-May/001338.html Jason