Development discussion of WireGuard
 help / color / mirror / Atom feed
* [ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here
@ 2021-03-15 14:46 Jason A. Donenfeld
  2021-03-15 19:49 ` Reto
                   ` (2 more replies)
  0 siblings, 3 replies; 5+ messages in thread
From: Jason A. Donenfeld @ 2021-03-15 14:46 UTC (permalink / raw)
  To: wireguard

Hi everybody,

I’m pleased to announce that WireGuard now runs inside the FreeBSD
kernel, with a driver called if_wg. It has full support of wg(8) and
wg-quick(8) [5], as well as general integration into FreeBSD userland.
Performance should be decent. The implementation in FreeBSD’s main
branch should pretty much work, though it’s something of a so-so work in
progress. To learn what I mean there, read on…

Sometime ago, a popular firewall vendor tasked a developer with writing
a WireGuard implementation for FreeBSD. They didn’t bother reaching out
to the project. That’s okay, I figured, I’ll reach out and see if I can
help and coordinate. What followed over the next year was a series of
poor communications – messages unanswered, code reviews ignored, that
kind of thing. Usually collaborations I’ve had with others have been
full of excitement, but it just didn’t work out here. In the few
discussions we were able to have, I did get across some key points,
like, “you’ll save a bunch of time if you use the OpenBSD code as a
starting point.” But mostly it seemed like a stop-and-go effort that the
WireGuard project didn’t have much to do with. Then, at some point,
whatever code laying around got merged into the FreeBSD tree and the
developer tasked with writing it moved on.

Fortunately, two weeks before FreeBSD 13.0 was due to be released,
FreeBSD core developer Kyle Evans emailed the list about integrating
wireguard-tools (wg(8) and such). In the ensuing discussion I mentioned
that we really need to get the actual if_wg kernel implementation up to
snuff. We took the conversation to IRC, and agreed that we should work
on figuring out what to do before the release date. At the same time,
Matt Dunwoodie, who worked on the OpenBSD implementation, also took a
look at what had become of that implementation in FreeBSD. Over the next
week, the three of us dug in and completely reworked the implementation
from top to bottom, each one of us pushing commits and taking passes
through the code to ensure correctness. The result was [6]. It was an
incredible effort. The collaboration was very fast paced and exciting.
Matt and Kyle are terrific programmers and fun to work with too.

The first step was assessing the current state of the code the previous
developer had dumped into the tree. It was not pretty. I imagined
strange Internet voices jeering, “this is what gives C a bad name!”
There were random sleeps added to “fix” race conditions, validation
functions that just returned true, catastrophic cryptographic
vulnerabilities, whole parts of the protocol unimplemented, kernel
panics, security bypasses, overflows, random printf statements deep in
crypto code, the most spectacular buffer overflows, and the whole litany
of awful things that go wrong when people aren’t careful when they write
C. Or, more simply, it seems typical of what happens when code ships
that wasn’t meant to. It was essentially an incomplete half-baked
implementation – nothing close to something anybody would want on a
production machine. Matt had to talk me out of just insisting they pull
the code entirely, and rework it more slowly and carefully for the next
release cycle. And he was right: nobody would have agreed to do that,
and it would only have fostered frustration from folks genuinely
enthusiastic about if_wg. So our one and only option was to iteratively
improve it as fast as we could during the two weeks before release, and
try to make it as simple and close as possible to OpenBSD so that we
could benefit from the previous analysis done there. With that as our
mission, we set out auditing and rewriting code.

One curious thing of note is that there were 40,000 lines of optimized
crypto implementations pulled out of the Linux kernel compat module but
not really wired up correctly, and mangled beyond repair with mazes of
Linux→FreeBSD ifdefs. I wound up replacing this with an 1,800 line file,
crypto.c [1], containing all of the cryptographic primitives needed to
implement WireGuard. Aside from its place in the FreeBSD story, this is
kind of neat in its own right: these are simple, but fast enough,
reference implementations. It’s not deliberately tiny or obfuscated like
TweetNaCl is, yet is still just a single file, and the Curve25519 field
arithmetic in it is formally verified. Maybe other projects will find
use for it. Future releases will hopefully get rid of crypto.c and hook
into FreeBSD’s already existing optimized implementations [4], which
should give a nice performance boost, but given the time crunch, having
something boring, safe, and simple seemed like the way to go.

We reduced the project structure down to four C files – the
aforementioned crypto.c, two files copied verbatim from OpenBSD –
wg_noise.c and wg_cookie.c – and if_wg.c, the actual interface device
driver implementation and protocol logic. The IPC interface was reworked
as well, and wg(8) in the wireguard-tools package grew support for it
(also rewritten from the original attempt). The three of us spent
countless hours across three time zones auditing state machine logic,
running trials,  and generally trying to get this working and workable.
There are now even a few automated tests!

I think we’ve mostly succeeded in producing something that behaves like
WireGuard. The net result certainly isn’t perfect, though – the Linux
and OpenBSD implementations were long, careful, slow projects by
comparison – but it is at least a base on which to build and improve
over time. Going forward, I think there’ll be additional systems coding
issues to work out – locking, lifetimes, races, and that sort of thing.
But now that there’s at least a stable base, developers can work out
remaining issues incrementally.

But perhaps this is a good moment to step back and ask how we got here,
and what WireGuard itself really is.

Traditionally, network protocols are specified in a document of protocol
behaviors. Then different organizations implement that specification.
Then everybody interoperates and all goes well. In practice, it often
doesn’t go well (see IPsec woes), but this at least has been the
traditional way of doing this on the Internet, and in some ways it
works.

But that is not the approach taken by the WireGuard project. In
contrast, WireGuard is both a protocol and a set of implementations,
implemented with a particular set of security and safety techniques.
That’s a radical departure from the traditional model, and one surely to
raise some grumbles amongst graybeards. But I believe this is a
necessary and beneficial quality for having the types of high assurance
software that is needed for core Internet security infrastructure. When
you use WireGuard, you’re not just using some protocol that is capable
of producing packets that are legible by others. You’re also using an
implementation that’s been designed to avoid security pitfalls, and that
provides interfaces for using it that mitigate footguns. In that way,
the WireGuard project is more expansive than a mere protocol project or
a mere software project or a mere cryptography project or a mere
specification project or a mere interface project. It combines all of
those things into a single unified approach. (For this same reason, the
original WireGuard paper [2] has been difficult for folks to categorize.
Is this a systems paper? A networking paper? A crypto paper?)

Because of that, I think this was an understandable predicament. After
all, why shouldn’t a company be able to task a developer with writing
some ring-0 WireGuard code in C? And why does it matter to me whether
the code is garbage if it can at least produce protocol packets? The
reason is that the WireGuard project’s mission is wider than that. We
deeply care about code quality and implementation particulars.

While we now have the FreeBSD code in a maintainable state, there are
other projects too that could use some attention from us. Looking
forward, for example, we hope to be able to lend a hand similarly to the
NetBSD developers soon to help them finish their implementation; this is
long overdue on my part, and I owe them some time and energy there. And
I hope that others don’t hesitate to email the list asking for
collaboration. This kind of thing is, of course, one of the reasons that
the project as an organization exists.

To return to the primary announcement, I had originally hoped to say
that this would be shipping for 13.0, and have some instructions for
setup there, but unfortunately, and contrary to our plans, it looks
exceedingly likely that given the grave issues we found in the existing
code, they’ll in the end just disable the module from the release, and
revisit for 13.1, rather than merging our fixes a few short days before
the release. That’s a bit of a bummer, given how hard we worked to get
things done in the time crunch, but it’s also probably a very wise
decision that takes some courage to make, and this will give us more
time to really get this rock solid for 13.1.

As well, hopefully we’ll have backport modules for the 13.0 and 12.y
release, making it as available as possible. Kyle or I will update the
list when  we’ve got a standalone backport module ready, with
instructions, as well as updating [3] per usual. There’s also ongoing
work to integrate WireGuard interface setup into rc, and hopefully that
will land during the the next release window, as well as the
aforementioned improvements to optimized crypto and systems issues.

Enjoy,
Jason

[1] https://cgit.freebsd.org/src/tree/sys/dev/if_wg/crypto.c?id=74ae3f3e33b810248da19004c58b3581cd367843
[2] https://www.wireguard.com/papers/wireguard.pdf
[3] https://www.wireguard.com/install/
[4] https://lists.freebsd.org/pipermail/freebsd-hackers/2021-March/057076.html
[5] https://lists.zx2c4.com/pipermail/wireguard/2021-March/006493.html	
[6] https://cgit.freebsd.org/src/commit/?id=74ae3f3e33b810248da19004c58b3581cd367843

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here
  2021-03-15 14:46 [ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here Jason A. Donenfeld
@ 2021-03-15 19:49 ` Reto
  2021-03-16  1:18 ` Jason A. Donenfeld
  2021-03-19 13:03 ` Kyle Evans
  2 siblings, 0 replies; 5+ messages in thread
From: Reto @ 2021-03-15 19:49 UTC (permalink / raw)
  To: wireguard

That's awesome!

Thanks a lot for your hard work and dedication on wireguard.
I really appreciate what you do including your willingness to work with non
Linux OSes.

Cheers,
Reto

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here
  2021-03-15 14:46 [ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here Jason A. Donenfeld
  2021-03-15 19:49 ` Reto
@ 2021-03-16  1:18 ` Jason A. Donenfeld
  2021-03-19 13:03 ` Kyle Evans
  2 siblings, 0 replies; 5+ messages in thread
From: Jason A. Donenfeld @ 2021-03-16  1:18 UTC (permalink / raw)
  To: wireguard, Scott Long; +Cc: secteam, cperciva

Hi Scott,

It's certainly disappointing to receive your email like that. I see that
you're really upset, and I think we have a shared interest in
deescalating this. You've mentioned the desire to talk about this in
public, so I'm responding in the announcement thread to you, as this
seems like the appropriate venue for discussion.  I've also CC'd the
FreeBSD Security Officer. I've responded to your email and its threats
in line mailing-list style below:

On Mon, Mar 15, 2021 at 6:08 PM Scott Long <scottl@netgate.com> wrote:
> What you and Kyle did was tell the world that there are a number of
> zero-day exploits in the code.  You gave us no details until after the
> fact, gave us no time to mitigate, correct, and publish before your
> announcement and Kyle's code drop, and used the opportunity to
> bash the code, and by extension us, for your own self-gain.

I have no idea how you possibly have arrived at this here. FreeBSD 13 is
currently at RC2 or RC3, if I recall correctly. Our whole motivation for
working on this so tirelessly and with so much energy and devotion over
the week was to get things fixed up before 13 was actually released. I
dropped everything else I was working on to focus on this. I'm actually
with my folks in Colorado and should be spending more time with them but
the idea of FreeBSD 13 shipping with all those bugs seemed like a really
important thing to address.

Here's what we did, step by step:

1. We made a clone of the freebsd git repo where we could all push
   changes to master, to git.zx2c4.com like everything else.

2. We took turns taking big passes at the codebase, reading it, auditing
   it, fixing things. We did a lot of syncing and comparison with the
   OpenBSD code base. For example, in one step, we reordered the functions
   to match the flow of OpenBSD so we could see it more easily side by
   side.

3. We kept pushing commits like this in that manner.

4. We didn't know where we'd end up, but after a few days of this, the
   number of changes we had accumulated and fishy business we had observed
   was massive.

5. When we thought we reached a good resting point -- and when it became
   clear that our rapid fire changes were becoming too numerous -- we
   squashed that down to a summary commit, and Kyle committed it.

6. Seeing that the crypto needed to be re-accelerated, I published a
   call for help here --
   https://lists.freebsd.org/pipermail/freebsd-hackers/2021-March/057076.html

7. I published wireguard-tools with the improved wg(8) command here --
   https://lists.zx2c4.com/pipermail/wireguard/2021-March/006493.html

8. I sent a summary of everything to the mailing list here --
   https://lists.zx2c4.com/pipermail/wireguard/2021-March/006494.html

I also made sure to email a FreeBSD Security Officer, Colin, to keep
him posted about the state of this, so he was kept apprised.

Again, the whole idea was: two weeks until release! If I suggest the
code is just disabled, people will be really upset, so instead we'll try
to fix as much as fast as possible before the release. And actually,
really focused sprints like that are fun and stimulating. We were all
really enjoying writing code until the torrents of anger started coming
from you about our efforts. (And I also learned that this isn't the
first time you guys tried to intimidate an open source project -- see
this summary of the opnSense defamatory website Netgate created --
https://opnsense.org/opnsense-com/ .)

And in the process, too, I've tried to be in contact with you and Jim
and let you know what our intentions are and to diffuse tensions. I
spent time on a video call trying to describe to you some of the
security things we found, in case it wasn't possible for you to use the
new code right away. I've also made it abundantly clear to you how much
I want to work WITH Netgate. When that Reddit thread cropped up, I
offered to you multiple times to send a message to it telling people
that we've spoken and it looks like you have a good plan and every
things going to be okay, but you didn't respond to my offer.

On Mon, Mar 15, 2021 at 6:08 PM Scott Long <scottl@netgate.com> wrote:
> None
> of these actions reflect good-faith collaboration, and your statements
> that you think that our work is "really cool" ring hollow.  Maybe you
> and Kyle are really naive and thought that this was how people
> normally collaborate in the security and business worlds, and that
> everyone would rush to praise you and shower you with contracts
> and funding.  And hey, maybe that’ll happen, you certainly have
> made a splash.  That of course, comes at our expense, and it’s
> likely to be a pretty damaging one.  Now with the Ars article, I’m
> starting to wonder if this was a coordinated smear campaign.
> Normally we don’t get this kind of attention.  It’s going to be painful
> to navigate our way out.

I'm not sure how to address these types of... theories. I certainly AM
acting in good faith. I can't possibly see how it benefits anybody to
have WireGuard and Netgate at odds with one another. You guys make a
neat product based on open source software, and I think it's great
you're planning to add WireGuard to it. My interest is in making sure
the underlying operating system -- FreeBSD 13 -- does that well.  (Or, I
think you mentioned to me this morning that you're actually planning to
base it on FreeBSD 12 for some time, but same idea for the backport
you're working on, I guess.)

And there's no "smear campaign" -- that's outrageous. I have no idea how
that would help anybody at all. I should point out, again to you, how
grateful I am that Netgate got the initial work on this started. Had you
guys not gotten if_wg rolling, it's not clear to me that it'd exist in
the first place.

From your end, it's very disappointing that you're (below) planning to
attack security researchers and kernel programmers who took the time to
rewrite code to make it better, hustling to get it in before FreeBSD 13
is released.

On Mon, Mar 15, 2021 at 6:08 PM Scott Long <scottl@netgate.com> wrote:
> I’ll be writing an article tonight outlining the mistakes that we made
> and what we will be doing to correct it.  I’ll also be highlighting that
> this incident has been a textbook example of the wrong way for
> people to collaborate in the security community, and that extreme
> caution should be taken in any future dealings with you.  Your
> actions are self serving and in bad faith, and your words are
> hollow and untrustworthy.  I don’t care if you disagree or don’t see
> it this way, this is the effect that you’ve had.

It makes sense to communicate with your customers about things with your
upcoming products if you feel it's necessary. But threatening that
you're going to highlight "that extreme caution should be taken in any
future dealings with you" sounds to me like a threat of some intense
slander. And again, attacking security researchers and kernel
programmers who took time to rewrite code to make it better before a
release deadline...  That's ...wow. I wish you would not go on the
attack like that.

Instead, it seems like we can both be glad that we're on the path to
having better code! And that there's a team of people volunteering to do
this and enjoying writing that code. We're all working together to make
things better. We worked really super hard on fixing up this code base.
It was a LOT of work. And I would like to keep working on it and feeling
good about it.

So I'm just disappointed that somehow this has devolved for you into
making threats like that.

On Mon, Mar 15, 2021 at 6:08 PM Scott Long <scottl@netgate.com> wrote:
> I’ve also spoken with the FreeBSD Security Officer, and we’ve
> agreed that wireguard will be removed from all branches of FreeBSD
> until further notice.  I’ve also informed Kyle of this.  I do not support
> its reintroduction into FreeBSD, whether in the src tree or in the
> ports tree, at this time.  As for pfSense, we are conducting an
> audit and will decide on the best course of action for our customers
> and our company.

I actually think that's a great idea. I was under the impression
initially that removing it would be a total non-starter and would just
make people upset, hence this hurried mad dash to turn things around.
But having an opportunity to take it out of tree and develop it slowly
and carefully, like we did for Linux and OpenBSD, sounds a lot better.
This also meshes well with what we had planned to do, which I mentioned
in the announcement email this morning -- have this available as an out
of tree module that people can try out. Then, after we've got it really
solid, reintroduce it in the normal FreeBSD ways of doing that. (I'm not
so familiar with that latter process, but I trust that Kyle & folks know
that part well.)

On Mon, Mar 15, 2021 at 6:08 PM Scott Long <scottl@netgate.com> wrote:
>
> Scott

So, that's where we are. You're upset and making threats to slander
security researchers and kernel programmers. That's really
disappointing. All I can do is ask you: please don't? We have much more
to gain by supporting each other than being in conflict. I just want to
make sure FreeBSD 13 (or 14) gets a great WireGuard implementation. And
you want to make sure your customers get great products. I see no reason
why these are conflicting goals.

I hope this can deescalate, and we can work together on this. I'm
confident that our goals are ultimately aligned very well.

Regards,
Jason

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here
  2021-03-15 14:46 [ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here Jason A. Donenfeld
  2021-03-15 19:49 ` Reto
  2021-03-16  1:18 ` Jason A. Donenfeld
@ 2021-03-19 13:03 ` Kyle Evans
  2021-03-19 13:38   ` Kyle Evans
  2 siblings, 1 reply; 5+ messages in thread
From: Kyle Evans @ 2021-03-19 13:03 UTC (permalink / raw)
  To: Jason A. Donenfeld; +Cc: WireGuard mailing list

On Mon, Mar 15, 2021 at 9:47 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
>
> Hi everybody,
>
> [... snip ...]
>
> The first step was assessing the current state of the code the previous
> developer had dumped into the tree. It was not pretty. I imagined
> strange Internet voices jeering, “this is what gives C a bad name!”

This was a highly unnecessary jab.

> There were random sleeps added to “fix” race conditions

This is true.

> validation functions that just returned true

I looked back at the history here just now, and I count one,
wg_allowedip_valid, that pretty much got ripped out anyways.

> catastrophic cryptographic vulnerabilities
> whole parts of the protocol unimplemented

I'm not qualified to speak about these two, which are perhaps the
worst from the public's perspective.

> kernel panics

These are true.

> security bypasses
> overflows

I only recall one of each.

> random printf statements deep in crypto code

This is true.

> the most spectacular buffer overflows

English is a crappy language, but this sounds like you're advertising
more than one.  There was one, and for the record to anyone watching to
dispel "the word on the street" that it's potentially an RCE: based on what's
happening I cannot imagine how you could usefully turn it into an RCE.
Local privileged execution, 100%, but looking at it further, you've got to be
pretty skilled to make it before killing the kernel elsewhere.

> and the whole litany of awful things that go wrong when people aren’t
> careful when they write C.

This is an exceedingly broad statement, and it would have been good to provide
some pointers to these.

> [...]

You know that I don't appreciate how you handled this initial communication,
as I've told you a number of times. Now that I look at the history again, I'm
even more disappointed in how you handled this because there are some
pretty broad statements in here and language that could go either way.

I've additionally recommended, as a developer and not in any kind of official
capacity, that we can't include if_wg in any future version of base.  We cannot
have our users being put at the risk of this kind of publicity if we
can't get security
advisories issued in time. Ports is a fine place, where security issues can be
addressed expeditiously and more in line with how the WireGuard project chooses
to handle them.

Thanks,

Kyle Evans

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here
  2021-03-19 13:03 ` Kyle Evans
@ 2021-03-19 13:38   ` Kyle Evans
  0 siblings, 0 replies; 5+ messages in thread
From: Kyle Evans @ 2021-03-19 13:38 UTC (permalink / raw)
  To: Jason A. Donenfeld; +Cc: WireGuard mailing list

On Fri, Mar 19, 2021 at 8:03 AM Kyle Evans <kevans@freebsd.org> wrote:
>
> On Mon, Mar 15, 2021 at 9:47 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> >
> > Hi everybody,
> >
> > [... snip ...]
> >
> > The first step was assessing the current state of the code the previous
> > developer had dumped into the tree. It was not pretty. I imagined
> > strange Internet voices jeering, “this is what gives C a bad name!”
>
> This was a highly unnecessary jab.
>
> > There were random sleeps added to “fix” race conditions
>
> This is true.
>
> > validation functions that just returned true
>
> I looked back at the history here just now, and I count one,
> wg_allowedip_valid, that pretty much got ripped out anyways.
>
> > catastrophic cryptographic vulnerabilities
> > whole parts of the protocol unimplemented
>
> I'm not qualified to speak about these two, which are perhaps the
> worst from the public's perspective.
>
> > kernel panics
>
> These are true.
>
> > security bypasses
> > overflows
>
> I only recall one of each.
>
> > random printf statements deep in crypto code
>
> This is true.
>
> > the most spectacular buffer overflows
>
> English is a crappy language, but this sounds like you're advertising
> more than one.  There was one, and for the record to anyone watching to
> dispel "the word on the street" that it's potentially an RCE: based on what's
> happening I cannot imagine how you could usefully turn it into an RCE.
> Local privileged execution, 100%, but looking at it further, you've got to be
> pretty skilled to make it before killing the kernel elsewhere.
>

Ah, I have to clarify this one; I suspect if you're acting as a wg
gateway, it's possible. It
still doesn't look like an easy one to exploit.

> > and the whole litany of awful things that go wrong when people aren’t
> > careful when they write C.
>
> This is an exceedingly broad statement, and it would have been good to provide
> some pointers to these.
>
> > [...]
>
> You know that I don't appreciate how you handled this initial communication,
> as I've told you a number of times. Now that I look at the history again, I'm
> even more disappointed in how you handled this because there are some
> pretty broad statements in here and language that could go either way.
>
> I've additionally recommended, as a developer and not in any kind of official
> capacity, that we can't include if_wg in any future version of base.  We cannot
> have our users being put at the risk of this kind of publicity if we
> can't get security
> advisories issued in time. Ports is a fine place, where security issues can be
> addressed expeditiously and more in line with how the WireGuard project chooses
> to handle them.
>
> Thanks,
>
> Kyle Evans

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-03-19 13:38 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-15 14:46 [ANNOUNCE] WireGuard for FreeBSD in development for 13.y – and a note of how we got here Jason A. Donenfeld
2021-03-15 19:49 ` Reto
2021-03-16  1:18 ` Jason A. Donenfeld
2021-03-19 13:03 ` Kyle Evans
2021-03-19 13:38   ` Kyle Evans

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).