[9fans] Thoughts on squashing the spam bug.

[9fans] Thoughts on squashing the spam bug.

[Dan Cross]

Everyone's been talking a good game, but so far, no one's really
thought up a comprenehsive *plan* to fix the problem.  There are
several issues at work here:

	1) A lot of people seem to believe that the problem is
	   social in nature, and can be solved legally.
	2) Others think a purely technical solution will work.
	3) Some feel we should ditch SMTP entirely and come
	   up with something new.
	4) Some feel strong authentication will fix the problem,
	5) Others feel probabilistic analysis of mail will catch
	   everything, or most everything.

1) Isn't realistic.  You can't solve a problem with a law unless you
   have some way to enforce the law.  Currently, we don't.

2) Is only part of the solution.  Yes, one needs technical
   infrastructure before anything else will work.  However,
   technical instrastructure by itself won't solve the problem.

3) That's all well and good, but one needs to be cogniscent of the
   existing installed base and inertia behind those protocols.  They
   aren't going to be displaced over night, and what's more, unless the
   alternative solution is *really* *really* good, with all the
   migration issues worked out, and with clear advantages over the
   existing status quo, it won't happen at all.  A lot of people are
   fond of saying that a better technical solution will solve
   everything, that people will be compelled to switch once they see
   how great the new thing is, etc, etc, etc.  However, I'm afraid that
   just won't pan out.  Experience has shown that all too often a
   technical superior alternative doesn't displace the reigning leader
   for other, non-technical issues: installed base, lack of a clear
   migration path, intellectual and emotional investment in the
   existing infrastructure, etc.

4) See point 2, above.

5) This will never work.  The spammers will just find there way around
   them.  How/Why?  Because the probabilities involved are simply too
   big.  It's too easy to just flood the user with spams that are
   better and better to that user's approximations of what isn't spam.
   Eventually, something will get through.

In the process, we've managed to generate probably on the order of 500
or more email messages about it, but without much real progress.  While
this is fun, kinda, it's not very productive.

I suggest doing the following:

a) Come up with a meta-protocol for delivery of email in a
   special-purpose hierarchy.  What I mean by this, is design a
   filesystem that can be imported for delivering mail.  Just importing
   /mail isn't going to fly; it's too big of a hammer.

b) Make sure that protocol is relatively straight-forward, and then
   mimic it in a `standard' protocol that could be approved by the IETF
   as a replacement for ESMTP/SMTP.  This would have to include a
   meta-protocol for strong authentication, and the protocol would have
   to have some major advantage over SMTP, such as much better
   performance, or lower latency, or what have you.  I'd say start with
   something like RSMTP, but that's patented.  :-(

   Note: authentication has to be by some meta-protocol, because people
   are going to have to transition to it.  It's not going to happen
   over night.

c) Contact the authors/maintainers of the most common MTA's and
   convince them to implement this protocol, and the scaffolding around
   it.  If they don't, you're screwed.  But if sendmail and postfix
   implement it, exim will come along.  If you can somehow convince Dan
   Bernstein to implement it, you've covered a pretty significant
   portion of Internet email.  Once that happens, Microsoft might even
   come on board.  You could even keep SMTP over TLS using SASL
   authentication for message submission from your
   (yet-to-be-converted) MUA's, but make MTA<->MTA communication use
   the new protocol.

d) Once you have something in place, go to the IETF and get it
   standardized, and then get them to ratify it as the replacement for
   SMTP.

Anyway, that's my plan to start on building a new email infrastructure.
I suppose the first step is to design a filesystem for message submission
over 9p.  Anyone got any ideas?

	- Dan C.

Re: [9fans] Thoughts on squashing the spam bug.

[boyd]

i got some

Re: [9fans] Thoughts on squashing the spam bug.

[Geoff Collyer]

Actually, as far as I know, RSMTP isn't patented yet; the application
is still being processed/ignored by the patent office.  (The original
hope was that this would go *much* faster and that we could then
persuade Lucent to dedicate the patent to the public, per set-id, and
thus not charge licence fees.)

Authentication is the hard part: how do you prove, in court if
necessary, that a given message came from a particular real person?
Short of legal liability, I think PGP provides the most obvious way to
provide a degree of authentication that would be useful for filtering
by sender.

Note again that I only import /mail/box, not /mail.  Point (c) can be
simpler: make the transmitter and listener programs drop into
sendmail, postfix, upas, etc. and (x)inetd, respectively, so anybody
can choose to use the new protocol without hacking (just tweaking
configuration files).  One can maintain a list of other domains known
to listen for the new protocol; one can also just try to connect using
the new protocol for a moment before falling back to smtp.

Points (b) and (d) seem dauntingly hard to me: the IETF has become a
political organisation, with legendarily fearsome politics.  I don't
know what it takes to get a protocol through the IETF relatively
undamaged.  Do you have to know the right people and have them like
you?  Does the protocol have to avoid use of the most-significant bit
(gag)?  Does it have to use CRLF line termination (in text)?  Just how
dumbed-down and compatible with all past mistakes does it have to be?

I'd be inclined instead to just build something that works and let the
IETF catch up, as done with UTF-8.  (Also with Unix and C; POSIX and
ANSI eventually caught up, and I'm glad that Unix and C weren't
designed with acceptability to standards bodies in mind.)  So my plan
would be: build something good that doesn't require a flash cut-over,
document it, make drop-in components for sendmail, etc. and inetd,
etc., distribute the works freely, and let the IETF write it up when
they get around to it.  (This actually is how the IETF theoretically
works: they supposedly bless existing practice, rather than inventing
untested protocols and decreeing that the serfs shall use them.)

Re: [9fans] Thoughts on squashing the spam bug.

[boyd]

    If you can somehow convince Dan Bernstein to implement it

that's a big fucking _if_.

Re: [9fans] Thoughts on squashing the spam bug.

[boyd]

    I'd be inclined instead to just build something that works and let the
    IETF catch up, as done with UTF-8.

i'm with you, captain.

i know dan wants to do it right, but it's just too hard.

Re: [9fans] Thoughts on squashing the spam bug.

[boyd]

    i know dan wants to do it right, but it's just too hard.

then again, he may be crazy enough and smart enough to do it anyway.

hoo-ahh!!

Re: [9fans] Thoughts on squashing the spam bug.

[Derek Fawcus]

On Sun, Sep 28, 2003 at 06:49:15PM -0400, boyd@sdgm.net wrote:
>     If you can somehow convince Dan Bernstein to implement it
>
> that's a big fucking _if_.

Well he seems to have already come to the conclusion that some form of
postage is needed for email...

Mind there are a lot of addon patches distributed for qmail,  and it's
quite easy to plug things into it.  So one could produce an alternate
transport mechanism as a program to run instead of the qmail-smtpd and
qmail-remote (?  can't remember which is SMTP client) programs.

DF

Re: [9fans] Thoughts on squashing the spam bug.

[Dan Cross]

> then again, he may be crazy enough and smart enough to do it anyway.

Thanks Boyd, that's flattering.  :-)

> hoo-ahh!!

But one thing I have to say.  You keep using Marine Corps imagery, but
in the Corps, we say, ``Ooo-Rah.''  ``Hoo-ahh'' is something the US
Army says.  Ooorah has an interesting history; apparantly it originated
in the 1950's as a mutation of ``aarugha'', which was the sound made
by the dive klaxon on the old diesel submarines of the era. Recon
platoons used to be transported on those, and they adopted the sound
as a running chant.  When belted out from the stomach, as when running,
it sounds like ``ooorah.''

	- Dan C.

Re: [9fans] Thoughts on squashing the spam bug.

[boyd]

i call it as see it & i got the the Hoo-ahh from my mate in the PAARNG.

another good man.  from ny.

Re: [9fans] Thoughts on squashing the spam bug.

[Dan Cross]

> i call it as see it & i got the the Hoo-ahh from my mate in the PAARNG.
>
> another good man.  from ny.

That is no excuse, maggot!  You will not laugh, you will not cry!  You
will learn by the numbers.  I will teach you.  Now get up!  Get on your
feet!

	- Dan C.

Re: [9fans] Thoughts on squashing the spam bug.

[boyd]

:)

this guy is an lt and drives an M-1 abrambs and spent 8 months in bosnia (un peackeeping).

Re: [9fans] Thoughts on squashing the spam bug.

[boyd]

i see :-)

Re: [9fans] Thoughts on squashing the spam bug.

[boyd]

bernstein is a lot like choate, except bernstein is smart.

i have had the odd engagement with him ...