Userspace Networking Stack + WireGuard + Go

Userspace Networking Stack + WireGuard + Go

[Jason A. Donenfeld]

Hi,

Sometimes people say things like, "I want to make an application that
connects to the server over WireGuard" or "I want to host this service
application behind WireGuard" or "I want this small utility to initiate
an SSH connection to a specific server over WireGuard." The usual answer
to this is to simply bring up a WireGuard interface on the computer, and
then do networking as usual. And making WireGuard easy enough that
people can pull up ad-hoc interfaces trivially has been a big motivating
factor for the simplicity of the tooling, such as `ip link add wg0 type
wireguard && wg set wg0 ...` and similar commands.

On Linux, adding an interface is easy to do and is a solution that fits
nearly all circumstances. Even if you're unprivileged and want a
WireGuard interface for just a single application that's bound to the
lifetime of that application, you can still use WireGuard's normal
kernel interface inside of a user namespace + a network namespace, and
get a private process-specific WireGuard interface. Pretty cool stuff.
It's not like that's an accident, of course -- WireGuard was originally
developed with Linux's model in mind.

But not all operating systems are as neat as Linux. To that end, we've
tried very hard to make WireGuard integrate as natively as possible into
other operating systems, with native clients for every platform, as well
as APIs for every platform to allow application developers to embed
tunnels: https://www.wireguard.com/embedding/ . This is great and works
well. WireGuard's design centers around the "network interface" as the
central object, and that's what these libraries enable. But still, some
people do not want to think about network interfaces or operating
systems when writing code.

To that end, I've recently written some code that couples a userspace
networking stack (from gVisor) with a userspace WireGuard implementation
(from wireguard-go), along with a little custom DNS client
implementation, so that Go applications can use an in-process WireGuard
interface directly to make TCP and UDP connections and listeners. From
the operating system's point of view, the application just sends and
receives encrypted UDP packets, and never sees the inner TCP or UDP
packets or knows about WireGuard at all.

Here's a quick example of what's possible:

    package main

    import (
        "io"
        "log"
        "net"
        "net/http"

        "golang.zx2c4.com/wireguard/device"
        "golang.zx2c4.com/wireguard/tun"
    )

    func main() {
        tun, tnet, err := tun.CreateNetTUN(
            []net.IP{net.ParseIP("192.168.4.29")},
	    []net.IP{net.ParseIP("8.8.8.8")},
	    1420)
        if err != nil {
            log.Panic(err)
        }
        dev := device.NewDevice(tun, &device.Logger{log.Default(), log.Default(), log.Default()})
        dev.IpcSet(`private_key=a8dac1d8a70a751f0f699fb14ba1cff7b79cf4fbd8f09f44c6e6a90d0369604f
    public_key=25123c5dcd3328ff645e4f2a3fce0d754400d3887a0cb7c56f0267e20fbf3c5b
    endpoint=163.172.161.0:12912
    allowed_ip=0.0.0.0/0
    `)
        dev.Up()

        client := http.Client{
            Transport: &http.Transport{
                DialContext: tnet.DialContext,
            },
        }
        resp, err := client.Get("https://www.zx2c4.com/ip")
        if err != nil {
            log.Panic(err)
        }
        body, err := io.ReadAll(resp.Body)
        if err != nil {
            log.Panic(err)
        }
        log.Println(string(body))
    }

This snippet prints out:

    163.172.161.0
    demo.wireguard.com
    Go-http-client/1.1

, because that HTTPS request is going through the demo box.

Other use cases of this include cloud VM providers to bundle an ssh
client directly into their tooling that connects to a "cloud private
network" over WireGuard. Or hosting utilities that allow people to run
web apps from their own computers that are then accessible on the
Internet by connecting in reverse through WireGuard. Or maybe other use
cases.

The API and the code is new, and potentially incomplete. Please take it
for a spin and let me know your feedback.

Enjoy,
Jason

PS: While the gophers on the list might be happy about this, the
rustaceans may well be left wondering, "what about us?" I hope to have
some positive news about wireguard-rs not before too long.

Re: Userspace Networking Stack + WireGuard + Go

[Julian Orth]

On 13/01/2021 17.04, Jason A. Donenfeld wrote:

 > Even if you're unprivileged and want a WireGuard interface for just a
 > single application that's bound to the lifetime of that application,
 > you can still use WireGuard's normal kernel interface inside of a user
 > namespace + a network namespace, and get a private process-specific
 > WireGuard interface.

That's what my patches from back in 2018 were trying to accomplish.
Unless I've missed something since, I do not see how what you're
describing would work.  Unless you also

- create a TUN device in the network namespace
- add a default route through that TUN device
- manually route all traffic between the init network namespace and your
   network namespace.

Is that what you meant or is there a simpler way?

network namespace wireguard routing [Was: Re: Userspace Networking Stack + WireGuard + Go]

[Jason A. Donenfeld]

In order to prevent this Go thread from being hijacked with Linux
concerns, I've changed the Subject line of the email. Please keep
follow ups in this thread rather than the other.

Response is in line below:

On Wed, Jan 13, 2021 at 5:26 PM Julian Orth <ju.orth@gmail.com> wrote:
>
> On 13/01/2021 17.04, Jason A. Donenfeld wrote:
>
>  > Even if you're unprivileged and want a WireGuard interface for just a
>  > single application that's bound to the lifetime of that application,
>  > you can still use WireGuard's normal kernel interface inside of a user
>  > namespace + a network namespace, and get a private process-specific
>  > WireGuard interface.
>
> That's what my patches from back in 2018 were trying to accomplish.
> Unless I've missed something since, I do not see how what you're
> describing would work.  Unless you also
>
> - create a TUN device in the network namespace
> - add a default route through that TUN device
> - manually route all traffic between the init network namespace and your
>    network namespace.
>
> Is that what you meant or is there a simpler way?

What I meant was:

1. User opens his shell and runs ./blah. That executes in the init
namespace where all the physical interfaces are.
2. blah creates a wireguard interface.
3. blah creates a network namespace.
4. blah moves that wireguard interface into that network namespace.
5. blah calls `setns()` on one of its threads to use that network namespace.

Thinking about this in more detail, I'm guessing you take issue with
step #2? Since that actually might require privileges in the init
namespace?

Jason

Re: network namespace wireguard routing [Was: Re: Userspace Networking Stack + WireGuard + Go]

[Julian Orth]

On 13/01/2021 17.33, Jason A. Donenfeld wrote:
 > In order to prevent this Go thread from being hijacked with Linux
 > concerns, I've changed the Subject line of the email. Please keep
 > follow ups in this thread rather than the other.
 >
 > Response is in line below:
 >
 > On Wed, Jan 13, 2021 at 5:26 PM Julian Orth <ju.orth@gmail.com> wrote:
 >>
 >> On 13/01/2021 17.04, Jason A. Donenfeld wrote:
 >>
 >>   > Even if you're unprivileged and want a WireGuard interface for just a
 >>   > single application that's bound to the lifetime of that application,
 >>   > you can still use WireGuard's normal kernel interface inside of a user
 >>   > namespace + a network namespace, and get a private process-specific
 >>   > WireGuard interface.
 >>
 >> That's what my patches from back in 2018 were trying to accomplish.
 >> Unless I've missed something since, I do not see how what you're
 >> describing would work.  Unless you also
 >>
 >> - create a TUN device in the network namespace
 >> - add a default route through that TUN device
 >> - manually route all traffic between the init network namespace and your
 >>     network namespace.
 >>
 >> Is that what you meant or is there a simpler way?
 >
 > What I meant was:
 >
 > 1. User opens his shell and runs ./blah. That executes in the init
 > namespace where all the physical interfaces are.
 > 2. blah creates a wireguard interface.
 > 3. blah creates a network namespace.
 > 4. blah moves that wireguard interface into that network namespace.
 > 5. blah calls `setns()` on one of its threads to use that network namespace.
 >
 > Thinking about this in more detail, I'm guessing you take issue with
 > step #2? Since that actually might require privileges in the init
 > namespace?

Exactly :). My patches in 2018 were trying to solve this by allowing the
user to change the "transit" network namespace after the device has been
created. The "transit" network namespace being the namespace in which
the Wireguard UDP socket lives. This would not require privileges in the
transit namespace, only some kind of proof that the user can create UDP
sockets in said namespace.

 >
 > Jason
 >

Re: network namespace wireguard routing [Was: Re: Userspace Networking Stack + WireGuard + Go]

[Toke Høiland-Jørgensen]

"Jason A. Donenfeld" <Jason@zx2c4.com> writes:

> In order to prevent this Go thread from being hijacked with Linux
> concerns, I've changed the Subject line of the email. Please keep
> follow ups in this thread rather than the other.
>
> Response is in line below:
>
> On Wed, Jan 13, 2021 at 5:26 PM Julian Orth <ju.orth@gmail.com> wrote:
>>
>> On 13/01/2021 17.04, Jason A. Donenfeld wrote:
>>
>>  > Even if you're unprivileged and want a WireGuard interface for just a
>>  > single application that's bound to the lifetime of that application,
>>  > you can still use WireGuard's normal kernel interface inside of a user
>>  > namespace + a network namespace, and get a private process-specific
>>  > WireGuard interface.
>>
>> That's what my patches from back in 2018 were trying to accomplish.
>> Unless I've missed something since, I do not see how what you're
>> describing would work.  Unless you also
>>
>> - create a TUN device in the network namespace
>> - add a default route through that TUN device
>> - manually route all traffic between the init network namespace and your
>>    network namespace.
>>
>> Is that what you meant or is there a simpler way?
>
> What I meant was:
>
> 1. User opens his shell and runs ./blah. That executes in the init
> namespace where all the physical interfaces are.
> 2. blah creates a wireguard interface.
> 3. blah creates a network namespace.
> 4. blah moves that wireguard interface into that network namespace.
> 5. blah calls `setns()` on one of its threads to use that network namespace.
>
> Thinking about this in more detail, I'm guessing you take issue with
> step #2? Since that actually might require privileges in the init
> namespace?

5. also requires CAP_SYS_ADMIN (and I think by extension, so does 3.,
and 4.). From 'man setns':

       Network, IPC, time, and UTS namespaces
              In order to reassociate itself with a new network, IPC,
              time, or UTS namespace, the caller must have the
              CAP_SYS_ADMIN capability both in its own user namespace
              and in the user namespace that owns the target namespace.

-Toke

Re: network namespace wireguard routing [Was: Re: Userspace Networking Stack + WireGuard + Go]

[Jason A. Donenfeld]

On Wed, Jan 13, 2021 at 5:46 PM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> 5. also requires CAP_SYS_ADMIN (and I think by extension, so does 3.,
> and 4.). From 'man setns':
>
>        Network, IPC, time, and UTS namespaces
>               In order to reassociate itself with a new network, IPC,
>               time, or UTS namespace, the caller must have the
>               CAP_SYS_ADMIN capability both in its own user namespace
>               and in the user namespace that owns the target namespace.

For this, you just create a new user namespace first. You can try it
yourself from the command line:

zx2c4@thinkpad ~ $ unshare -n
unshare: unshare failed: Operation not permitted
zx2c4@thinkpad ~ $ unshare -Un
nobody@thinkpad ~ $ ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

Re: network namespace wireguard routing [Was: Re: Userspace Networking Stack + WireGuard + Go]

[Toke Høiland-Jørgensen]

"Jason A. Donenfeld" <Jason@zx2c4.com> writes:

> On Wed, Jan 13, 2021 at 5:46 PM Toke Høiland-Jørgensen <toke@toke.dk> wrote:
>> 5. also requires CAP_SYS_ADMIN (and I think by extension, so does 3.,
>> and 4.). From 'man setns':
>>
>>        Network, IPC, time, and UTS namespaces
>>               In order to reassociate itself with a new network, IPC,
>>               time, or UTS namespace, the caller must have the
>>               CAP_SYS_ADMIN capability both in its own user namespace
>>               and in the user namespace that owns the target namespace.
>
> For this, you just create a new user namespace first. You can try it
> yourself from the command line:
>
> zx2c4@thinkpad ~ $ unshare -n
> unshare: unshare failed: Operation not permitted
> zx2c4@thinkpad ~ $ unshare -Un
> nobody@thinkpad ~ $ ip a
> 1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

Ah, right, of course, silly me :)

-Toke

Re: Userspace Networking Stack + WireGuard + Go

[Jason A. Donenfeld]

Another example, for the curious. This one hosts a web server entirely in
userspace. All the kernel sees are incoming and outgoing encrypted
WireGuard UDP packets.

    package main
    import (
        "io"
        "log"
        "net"
        "net/http"
        "golang.zx2c4.com/wireguard/device"
        "golang.zx2c4.com/wireguard/tun"
    )
    func main() {
        tun, tnet, err := tun.CreateNetTUN(
            []net.IP{net.ParseIP("192.168.4.29")},
            []net.IP{net.ParseIP("8.8.8.8"), net.ParseIP("8.8.4.4")},
            1420,
        )
        if err != nil {
            log.Panic(err)
        }
        dev := device.NewDevice(tun, &device.Logger{log.Default(), log.Default(), log.Default()})
        dev.IpcSet(`private_key=a8dac1d8a70a751f0f699fb14ba1cff7b79cf4fbd8f09f44c6e6a90d0369604f
    public_key=25123c5dcd3328ff645e4f2a3fce0d754400d3887a0cb7c56f0267e20fbf3c5b
    endpoint=163.172.161.0:12912
    allowed_ip=0.0.0.0/0
    persistent_keepalive_interval=25
    `)
        dev.Up()
        listener, err := tnet.ListenTCP(&net.TCPAddr{Port: 80})
        if err != nil {
            log.Panicln(err)
        }
        http.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
            log.Printf("> %s - %s - %s", request.RemoteAddr, request.URL.String(), request.UserAgent())
            io.WriteString(writer, "Hello from userspace TCP!")
        })
        err = http.Serve(listener, nil)
        if err != nil {
            log.Panicln(err)
        }
    }

Here's a gif: https://data.zx2c4.com/wireguard-go-userspace-networking-to-host-a-server.gif

Re: Userspace Networking Stack + WireGuard + Go

[Marc-André Lureau]

Hi Julian

On Wed, Jan 13, 2021 at 8:28 PM Julian Orth <ju.orth@gmail.com> wrote:
>
> On 13/01/2021 17.04, Jason A. Donenfeld wrote:
>
>  > Even if you're unprivileged and want a WireGuard interface for just a
>  > single application that's bound to the lifetime of that application,
>  > you can still use WireGuard's normal kernel interface inside of a user
>  > namespace + a network namespace, and get a private process-specific
>  > WireGuard interface.
>
> That's what my patches from back in 2018 were trying to accomplish.
> Unless I've missed something since, I do not see how what you're
> describing would work.  Unless you also
>
> - create a TUN device in the network namespace
> - add a default route through that TUN device
> - manually route all traffic between the init network namespace and your
>    network namespace.
>
> Is that what you meant or is there a simpler way?


I am not a network admin, but I agree. Setting up this kind of user
network namespace isn't trivial and requires some privileges. It would
be nice if the kernel or some services provided a simpler way. (fwiw,
some time ago I did some experimental/research work for VM &
containers at https://gitlab.freedesktop.org/elmarco/vnet)

-- 
Marc-André Lureau