[9fans] upas/smtp changes for STARTTLS, AUTH PLAIN.

[9fans] upas/smtp changes for STARTTLS, AUTH PLAIN.

[Dan Cross]

Early last week, I sent a note asking how to make upas/smtp speak
TLS and do SMTP authentication, because Columbia now requires me
to do so if I want to send mail through their servers (which I
do).  Here are patches that implement both, though perhaps in an
imperfect way.  In particular, I haven't figured out all of the
failure cases yet (though I am pretty careful not to send the
password if encryption isn't on).

Some notes on this: upas/smtp sends the `HELO' message when
connecting to a remote server.  That's fine, but doesn't let you
know whether the remote side will do TLS via the STARTTLS verb,
so I changed the hello() routine in /sys/src/cmd/upas/smtp/smtp.c
to send EHLO instead.  Since I was leery of breaking anything, I
decided when making this change that it would probably be best to
just copy smtp.c into esmtp.c and do everything there.  Hence, I
created /sys/src/cmd/upas/smtp/esmtp.c and changed the mkfile to
reflect that.  I also fixed a bug in the mkfile where the clean:
target didn't properly remove all build targets.  The mkfile diff
is:

term% diff /sys/src/cmd/upas/smtp/mkfile mkfile
4a5
> 	esmtp
26c27
< $O.smtpd:	smtpd.tab.$O rmtdns.$O spam.$O rfc822.tab.$O
---
> $O.smtpd:		smtpd.tab.$O rmtdns.$O spam.$O rfc822.tab.$O
28c29
< $O.smtp:	rfc822.tab.$O mxdial.$O
---
> $O.smtp $O.esmtp:	rfc822.tab.$O mxdial.$O
30c31
< smtpd.$O: 	smtpd.h
---
> smtpd.$O: 		smtpd.h
32c33
< smtp.$O to.$O: 	smtp.h
---
> smtp.$O esmtp.$O to.$O:	smtp.h
33a35
>
43c45
< 	rm -f *.[$OS] [$OS].$TARG smtpd.tab.c rfc822.tab.c y.tab.? y.debug $TARG
---
> 	rm -f *.[$OS] [$OS].^($TARG) smtpd.tab.c rfc822.tab.c y.tab.? y.debug $TARG
term%

Note that only the last stanza is the bug fix, and the rest is
adding support for esmtp.

Since the changes to smtp.c required to turn it into esmtp.c are
fairly localized, and the file is pretty long, I'm just including
the diff's here.  I've tested this reasonably well.  I haven't
yet added support for checking the remote server's certificate,
though the code is there.

term% diff /sys/src/cmd/upas/smtp/smtp.c esmtp.c
3a4,6
> #include <mp.h>
> #include <libsec.h>
> #include <auth.h>
6c9,11
< char*	hello(char*);
---
> static	char*	dotls(char*);
> static	char*	doauth(void);
> char*	hello(char*, int);
32a38,39
> int	trysecure;	/* Try to use TLS if the other side supports it */
> int	tryauth;	/* Try to authenticate, if supported */
38a46
> char	*user;		/* user we are authenticating as, if authenticating */
91a100,102
> 	case 'a':
> 		tryauth = 1;
> 		break;
103a115,120
> 	case 's':
> 		trysecure = 1;
> 		break;
> 	case 'u':
> 		user = ARGF();
> 		break;
167c184
< 	if((rv = hello(hellodomain)) != 0)
---
> 	if((rv = hello(hellodomain, 0)) != 0)
253c270,271
<  *  exchange names with remote host
---
>  *  exchange names with remote host, possibly
>  *  enable encryption and do authentication.
254a273,338
> static char *
> dotls(char *me)
> {
> 	TLSconn *c;
> 	Thumbprint *goodcerts;
> 	int fd;
> 	uchar hash[SHA1dlen];
>
> 	c = mallocz(sizeof(*c), 1);
> 	if (c == nil) {
> 		return Giveup;
> 	}
> 	dBprint("STARTTLS\r\n");
> 	getreply();
> 	fd = tlsClient(Bfildes(&bout), c);
> 	if (fd < 0)
> 		return Giveup;
> 	goodcerts = initThumbprints("/sys/lib/tls/smtp",
> 	    "/sys/lib/tls/smtp.exclude");
> 	if (goodcerts != nil) {
> 		sha1(c->cert, c->certlen, hash, nil);
> 		if (!okThumbprint(hash, goodcerts)) {
> 			//Return Giveup;
> 		}
> 		freeThumbprints(goodcerts);
> 	}
> 	Bterm(&bin);
> 	Bterm(&bout);
> 	Binit(&bin, fd, OREAD);
> 	fd = dup(fd, -1);
> 	Binit(&bout, fd, OWRITE);
> 	return(hello(me, 1));
> }
>
> static char *
> doauth(void)
> {
> 	char *buf, *base64;
> 	UserPasswd *p;
> 	int n;
>
> 	p = auth_getuserpasswd(nil,
> 	    "proto=pass service=smtp server=%q user=%q",
> 	    ddomain, user);
> 	if (p == nil)
> 		return Giveup;
> 	n = strlen(p->user) + strlen(p->passwd) + 3;
> 	buf = malloc(n);
> 	if (buf == nil)
> 		return Retry;	/* Out of memory */
> 	base64 = malloc(2 * n);
> 	if (base64 == nil) {
> 		free(buf);
> 		return Retry;	/* Out of memory */
> 	}
> 	snprint(buf, n, "%c%s%c%s", 0, p->user, 0, p->passwd);
> 	enc64(base64, 2 * n, (uchar *)buf, n - 1);
> 	free(buf);
> 	dBprint("AUTH PLAIN %s\r\n", base64);
> 	free(base64);
> 	if (getreply() != 2) {
> 		return Retry;
> 	}
> 	return(0);
> }
>
256c340
< hello(char *me)
---
> hello(char *me, int encrypted)
257a342,345
> 	String *r;
> 	char *p, *s, *t;
>
> 	if (!encrypted)
266,267c354,355
< 	dBprint("HELO %s\r\n", me);
< 	switch(getreply()){
---
> 	dBprint("EHLO %s\r\n", me);
> 	switch (getreply()) {
274a363,388
> 	r = s_clone(reply);
> 	if (r == nil) {
> 		return Retry;	/* Out of memory */
> 	}
> 	for (s = s_to_c(r), t = strchr(s, '\n'); s != nil && *s != '\0'; s = t, t = strchr(t, '\n')) {
> 		if (t != nil)
> 			*t++ = '\0';
> 		for (p = s; *p != '\0'; p++)
> 			*p = toupper(*p);
> 		if (!encrypted && trysecure &&
> 		    (strcmp(s, "250-STARTTLS") == 0 ||
> 		    strcmp(s, "250 STARTTLS") == 0))
> 		{
> 			s_free(r);
> 			return(dotls(me));
> 		}
> 		if (tryauth && encrypted &&
> 		    (strncmp(s, "250 AUTH", strlen("250 AUTH")) == 0 ||
> 		    strncmp(s, "250-AUTH", strlen("250 AUTH")) == 0) &&
> 		    strstr(s, "PLAIN") != nil)
> 		{
> 			s_free(r);
> 			return(doauth());
> 		}
> 	}
> 	s_free(r);
term%

Note that for authentication to work correctly, a factotum loaded
with the appropriate key must be in esmtp's namespace.  If not,
esmtp regards this as a failure and returns Giveup.  Use of both
TLS and SMTP AUTH is controlled by new command line options: -a
(for AUTH), -s (stands for Secure, for TLS), and -u (if the user
one wishes to authenticate as is different from what getuser(2)
returns).

I invoke it out of /lib/mail/remotemail as:

exec /bin/upas/esmtp -asu mycuid -h $fd $addr $sender $*

(Where mycuid is my Columbia login name.  Actually, it's not, but
*you* know what I mean).

Finally, since I was at it, I added support for the PLAIN
authentication mechanism to smtpd.  Note, however, that I haven't
tested this (I don't have a mail server running upas to do that,
unfortunately).  That said, it's fairly simple, so I probably
messed it up somewhere.  But the changes compile, and the only
other supported plaintext authentication method (LOGIN) is
non-standard and supposedly deprecated (not to mention slow), and
it bugged me that outgoing smtp and incoming smtp should be
assymetric with respect to what they support (though I didn't
bother implementing STARTTLS support, which I think is kind of
dumb, frankly).

Also, RFC2554 says that a client can send the first part of the
authentication sequence in the same message as the AUTH verb and
mechanism, to cut out a round trip.  I added support for this in
smtpd.c for the PLAIN and LOGIN auth methods.  Again, these last
sets of changes are untested, but do compile.  Anyway, here are
the patches:

term% diff /sys/src/cmd/upas/smtp/smtpd.y smtpd.y
64a65,66
> 		| 'a' 'u' 't' 'h' spaces name spaces string CRLF
> 			{ auth($6.s, $8.s); }
66c68
< 			{ auth($6.s); }
---
> 			{ auth($6.s, nil); }
term%

term% diff /sys/src/cmd/upas/smtp/smtpd.h smtpd.h
40c40
< void	auth(String *);
---
> void	auth(String *, String *);
term%

term% diff /sys/src/cmd/upas/smtp/smtpd.c smtpd.c
254c254
< 			reply("250 AUTH CRAM-MD5 LOGIN\r\n");
---
> 			reply("250 AUTH CRAM-MD5 PLAIN LOGIN\r\n");
916c916
< auth(String *mech)
---
> auth(String *mech, String *resp)
924a925
> 	char *user, *pass;
935c936
< 	if (cistrcmp(s_to_c(mech), "login") == 0) {
---
> 	if (cistrcmp(s_to_c(mech), "plain") == 0) {
942,945c943,977
< 		reply("334 VXNlcm5hbWU6\r\n");
< 		s_resp1_64 = s_new();
< 		if (getcrnl(s_resp1_64, &bin) <= 0)
< 			goto bad_sequence;
---
> 		s_resp1_64 = resp;
> 		if (s_resp1_64 == nil) {
> 			reply("334 \r\n");
> 			s_resp1_64 = s_new();
> 			if (getcrnl(s_resp1_64, &bin) <= 0) {
> 				goto bad_sequence;
> 			}
> 		}
> 		s_resp1 = s_dec64(s_resp1_64);
> 		if (s_resp1 == nil) {
> 			rejectcount++;
> 			reply("501 Cannot decode base64\r\n");
> 			goto bomb_out;
> 		}
> 		memset(s_to_c(s_resp1_64), 'X', s_len(s_resp1_64));
> 		user = (s_to_c(s_resp1) + strlen(s_to_c(s_resp1)) + 1);
> 		pass = user + (strlen(user) + 1);
> 		ai = auth_userpasswd(user, pass);
> 		authenticated = ai != nil;
> 		memset(pass, 'X', strlen(pass));
> 		goto windup;
> 	}
> 	else if (cistrcmp(s_to_c(mech), "login") == 0) {
>
> 		if (!passwordinclear) {
> 			rejectcount++;
> 			reply("538 Encryption required for requested authentication mechanism\r\n");
> 			goto bomb_out;
> 		}
> 		if (resp == nil) {
> 			reply("334 VXNlcm5hbWU6\r\n");
> 			s_resp1_64 = s_new();
> 			if (getcrnl(s_resp1_64, &bin) <= 0)
> 				goto bad_sequence;
> 		}
term%

I'd appreciate feedback on this stuff, particularly on how
various errors are handled.  Thanks!

	- Dan C.

Re: [9fans] upas/smtp changes for STARTTLS, AUTH PLAIN.

[nigel]

>> I believe that means 99.9% of the world.
>
> Really?  I couldn't find a single document that described it, which is
> why I did PLAIN in the client.  The documentation at sendmail.org said
> it was something one had to ask Mark Crispin about.
>

There's no documentation for it, no. It's just that it's the only form of
authentication Microsoft Outlook [Express] supports. I think the Netscape
communicator email client is also equally 'challenged'. Thus it might not
be documented, but it certainly is used a lot.

After checking around I concluded that any other email client that
supported authentication would do something sensible with a
challenge response, and so the only 'in the clear' mechanism required
was LOGIN. Not that there is any problem with having PLAIN as well,
I just didn't know of a client with which I could test it.

Re: [9fans] upas/smtp changes for STARTTLS, AUTH PLAIN.

[Dan Cross]

> > messed it up somewhere.  But the changes compile, and the only
> > other supported plaintext authentication method (LOGIN) is
> > non-standard and supposedly deprecated (not to mention slow), and
>
> If you have a client which only supports LOGIN authentication,
> deprecated or not, then you are very grateful.

I didn't remove support for LOGIN, just added support for PLAIN.

> I believe that means 99.9% of the world.

Really?  I couldn't find a single document that described it, which is
why I did PLAIN in the client.  The documentation at sendmail.org said
it was something one had to ask Mark Crispin about.

	- Dan C.

Re: [9fans] upas/smtp changes for STARTTLS, AUTH PLAIN.

[nigel]

> messed it up somewhere.  But the changes compile, and the only
> other supported plaintext authentication method (LOGIN) is
> non-standard and supposedly deprecated (not to mention slow), and

If you have a client which only supports LOGIN authentication,
deprecated or not, then you are very grateful. I believe that means
99.9% of the world.