[9fans] re: spam filtering fs

[9fans] re: spam filtering fs

[matt]

in light of recent forged To: & From:
the challenge/response method is going to generate twice as much traffic for such a mail.

As far as I can tell, without trying it, the example pipeto sends a copy of the message, virus payload and all to whoever the From: headers suggest, obviating the need to infect the plan9 host to spread itself.

You also run the risk of ending up sending to an infected host and throwing your email address into another steaming pot.

The next generation will snarf text from existing mails with the intention of defeating Bayes type filters.

Re: [9fans] re: spam filtering fs

[boyd, rounin]

> If only we could get MUAs configured to handle encrypted mail by
> default, users would get used to it without even noticing.  And only
> S/MIME (or PEM) would be feasible, ...

not sure PEM is such a good choice, 'cos many of the UA's don't
do it and it means you still have to deliver the message.  ron doesn't
want that, nor do i.  given that, a random token in the sig would
avoid the 'chainsaw meets butter' syndrome.

next i'd vote for s/mime.

Re: [9fans] re: spam filtering fs

[Lucio De Re]

On Sun, Sep 07, 2003 at 07:22:04PM -0700, Geoff Collyer wrote:
>
> The only advantage of PGP is that it's what people use.  Phil
> Zimmermann spoke at Apple last year and said that at that time, if you
> measured how people encrypted their mail on the wire (when they did,
> which wasn't terribly often), it was virtually all using PGP's formats
> (they could be using PGP or gpg or something else).  Everything else
> combined was epsilon (negligible) and PGP was the rest.  His comment a
> year ago was that ``PGP *is* encrypted mail''.

If only we could get MUAs configured to handle encrypted mail by
default, users would get used to it without even noticing.  And only
S/MIME (or PEM) would be feasible, as a self-generated certificate can
then be included in the message.  Not a lot of security, but we're
dealing with lusers here.

Once everyone has built up a collection of certificates for their
correspondents, I believe that some peer pressure could trigger a
quest for third-party certification.  But the tools must be in place
first.

++L

Re: [9fans] re: spam filtering fs

[Geoff Collyer]

The only advantage of PGP is that it's what people use.  Phil
Zimmermann spoke at Apple last year and said that at that time, if you
measured how people encrypted their mail on the wire (when they did,
which wasn't terribly often), it was virtually all using PGP's formats
(they could be using PGP or gpg or something else).  Everything else
combined was epsilon (negligible) and PGP was the rest.  His comment a
year ago was that ``PGP *is* encrypted mail''.

Re: [9fans] re: spam filtering fs

[boyd, rounin]

outlook does s/mime, but it doesn't let you play with headers :(

the trouble is with the MTAs and i/we have no (or not much) control over it.

i don't want the tcp connection in the first place, but if i have to
accept (which i more or less have too) i wanna shut the spammers
down most ricky tick.  so it's SMTP, but then how to i get my UA
to convince the MTA to let me tell 'em who i am.

i see no easy solution to this.

a token in the sig would be simple.  s/mime would be much better,
but then you have to teach all the world about s/mime.

pgp is a train wreck.

Re: [9fans] re: spam filtering fs

[ron minnich]

On Sun, 7 Sep 2003, boyd, rounin wrote:

>
> btw: i'm with ron all the way, but it's not feasable.

yeah, but we're still going to set it up on the 9grid :-)

It's a private party, but all of you will get engraved invites sooner or
later.

:-)

ron

Re: [9fans] re: spam filtering fs

[Dan Cross]

David Presotto <presotto@closedmind.org> writes:
> You're going to have to derive a canonical form of the message
> so that you have something to sha1 that won't change as it
> traverses the network.  Not hard, just need a definition.
> Something that includes the important headers (From:,
> Sender:, Reply-to:) and the body would be enough.  You
> might want to worry about making cr-lf == lf.

That's easy enough. Btw- it thinking about it highlights a hole
in my idea of the nonce/hmac of nonce method: it's vulnerable to
replay attacks, unless you keep a log of nonce's.

> Then you need a way to add the signature.  I'm happy with an
> S/MIME attachment but some people here hate S/MIME.

An RFC822 header would be both easier and, I think, more efficient.

> Then you need a database of shared keys.  If it were public
> key encryption, you could put the public half on a shared
> server but since these are secrets, we have to keep them to
> ourselves somewhere.

Here it comes down to only expending effort approriate to the value
of the information being defended.  This isn't meant to be a mechanism
for ensuring the absolute integrity of the message; it's meant to cut
down and try to eliminate spam.  Appending an HMAC to a message
someplace is a low cost way to do that.  How you choose to store the
keys you use for that hmac is up to you.  Putting them in a file in
your home directory should be enough; it should be understood up
front that this isn't a mechanism for fool-proof identification of
the sender.

> Finally, you need a way to introduce yourself to someone and
> give them a token.  This is always the hard part.  PGP sort
> of solves it with trusted places to leave the public key
> and by passing notes that are fingerprints of the public keys.
> Of course, the easier you make this, the easier it is for the
> spammers to insert themselves.

Again, it doesn't have to be foolproof.  What it *does* have to do
is raise the bar enough to defeat the automated address harvesters.
A simple way to transfer tokens is to send someone an image (or a
pointer to one) that depicts a string (maybe a sentence of diceware
words or something) and use that as the secret.  A human being has
to look at the image to figure out what the string is, which is going
to *drastically* cut down on the ability of machines to harvest
addresses for spamming (at least until image recognition software
gets more sophisticated).  I don't care if the guy sitting next to
me sees it, as long as he's not a spammer.

> Anyways, by the time you're done, you've defined PGP.  Why not
> use PGP?  If you're trying to be simple, you don't need their
> encryption (which hardly anyone uses anyways) or their complicated
> rules for trust relationships.  Then you have exactly what
> you're asking for and you stay compatible with some part of
> the world.

PGP solves the problem at the wrong level.  Here, one can attempt to
solve it at the SMTP transaction level.  With PGP, that's possible, but
more cumbersome; it's really set up for users to do authenticate each
other.  What's more, I still contend that even PGP has a much higher
`startup cost' than a simple HMAC using a shared secret, where sharing
the secret isn't really that important, and it's unnecessary to keep it
absolutely secure.

Besides, using PGP for this is like cutting butter with a chainsaw.  I
want something simple and lightweight.  Since this *can* be really
lightweight, getting vendors to incorporate it might be easier; PGP has
been around for 20 years, and it's still not a part of most MUA's.  Why
is that?  Even TLS has found it's way into outlook; why not PGP?  Are
there commercial restrictions?  Is it just too complex?  Given that
it's not universal, what part of the world am I becoming incompatible
with?  S/MIME is similar: comparatively speaking, nobody uses it.  I
shouldn't need to write an ASN.1 compiler to put the base64 encoding of
an HMAC in an RFC822 style header, and I shouldn't need that much
scaffolding to solve this problem.

Finally, despite recent changes in legislation, PGP is still crypto,
and there may be export problems.  An HMAC isn't; it's just hashing,
and so doesn't have those problems.

	- Dan C.

Re: [9fans] re: spam filtering fs

[David Presotto]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

You're going to have to derive a canonical form of the message
so that you have something to sha1 that won't change as it
traverses the network.  Not hard, just need a definition.
Something that includes the important headers (From:,
Sender:, Reply-to:) and the body would be enough.  You
might want to worry about making cr-lf == lf.

Then you need a way to add the signature.  I'm happy with an
S/MIME attachment but some people here hate S/MIME.

Then you need a database of shared keys.  If it were public
key encryption, you could put the public half on a shared
server but since these are secrets, we have to keep them to
ourselves somewhere.

Finally, you need a way to introduce yourself to someone and
give them a token.  This is always the hard part.  PGP sort
of solves it with trusted places to leave the public key
and by passing notes that are fingerprints of the public keys.
Of course, the easier you make this, the easier it is for the
spammers to insert themselves.

Anyways, by the time you're done, you've defined PGP.  Why not
use PGP?  If you're trying to be simple, you don't need their
encryption (which hardly anyone uses anyways) or their complicated
rules for trust relationships.  Then you have exactly what
you're asking for and you stay compatible with some part of
the world.

[-- Attachment #2: Type: message/rfc822, Size: 5080 bytes --]

From: Dan Cross <cross@math.psu.edu>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] re: spam filtering fs
Date: Sat, 06 Sep 2003 21:56:38 -0400
Message-ID: <200309070156.h871ucj14201@augusta.math.psu.edu>

Dave wrote:
> I'ld rather not have to keep a secret and a counter for everyone I want to
> exchange mail with.  Messages get lost and reordered so at the very least
> I need to accept some range of possible sha1ings.  I also want to accept
> mail from people I haven't talked to before but have proved to someone else
> that they aren't spammers.  I'm happier with the /mail idea than this one.

Then I reiterate my second suggestion:

[...]  Or, and even simpler, take the token and sha it with the
contents of the message.  The token itself doesn't show up in any
archives anywhere, and the scheme is immune to problems with bounces
getting sequence numbers out of whack, and you get some modicum of
integrity checking on the message itself.  A way around the client
problem is to build it into the MTA (but the MTA's on both sides have
to support it).

An alternative to this is to provide a nonce in the SMTP transaction or
in header, and the result of the HMAC of the nonce keyedwith the secret
shared between client and sender.  We've already started doing things
like this with the ESMTP AUTH stuff; MAL FROM: now has an ``AUTH''
parameter that can be hung off of it.  We could add a NONCE field,
too.  RCPT TO: can take an AUTHSIG or something, which is some sort of
signature on the nonce value.  We could add an RFC822 style header
called `Integrity: ' that contained the base64 encoding of both the
nonce and the signature in the form, <algo>:<data> (where <algo> is
the hashing algorithm used for the HMAC construction).

Ron had written:
> yeah but ... I don't even want the data coming into my machine. Is that
> covered too? I really want to get these spammers rejected instantly,
> which is why i liked the file system idea.

[Note: I really enjoyed Geoff's colorful description of the spam
problem subsequent to my reply to this....]

I've been thinking about this, and come to some conclusions.  First,
that one has to do whatever it is one decides to do within the context
of SMTP or ESMTP.  While importing a filesystem would be a nice,
elegant solution, it's just not realistic.  And the reason is that it's
not us that's the problem, but everyone else, and everyone else is
firmly mired in the religion of the Internet, which says that SMTP is
the one true way to do mail.  So, sucky though the protocol is, if you
want to do something that has real impact in the next one to two years,
you have to do it within the context of the pre-existing theology.  I
don't like it, but there it is.

Anyway, given that, I really think the simplest way is to do is to
append some sort of signature to a email in the SMTP transaction, but
it doesn't have to be nearly as complex as PGP or S/MIME; something
simpler is going to raise the bar sufficiently to thwart a lot of
the spammers forever.  Just doing an HMAC of a nonce, keyed with a
shared token, is going to stop a lot of the garbage that currently
filters through.

	- Dan C.

Re: [9fans] re: spam filtering fs

[boyd, rounin]

> Hopefully, eventually, it *will* change.  When I say ``it's just not
> realistic,'' I should qualify that a bit more.  ``It's just not
> realistic on a large scale any time soon, where soon is defined within
> the next five years.''

yup

> Who's with me?

i'm with you captain, i can already smell the napalm burning.

smells like ... victory ...

can we try the 'token in the sig'?  it's gotta beat ASN.1 ...

btw: i'm with ron all the way, but it's not feasable.

Re: [9fans] re: spam filtering fs

[Dan Cross]

> > I've been thinking about this, and come to some conclusions.  First,
> > that one has to do whatever it is one decides to do within the context
> > of SMTP or ESMTP.  While importing a filesystem would be a nice,
> > elegant solution, it's just not realistic.
>
> I'm not willing to accept this. You are right, I have no argument with
> what you are saying, but I started out with 5-level baudot paper tapes and
> we're not using them any more, so the compatibility argument, while
> strong, can be trumped at some point by superior technology.

Hopefully, eventually, it *will* change.  When I say ``it's just not
realistic,'' I should qualify that a bit more.  ``It's just not
realistic on a large scale any time soon, where soon is defined within
the next five years.''  The thing that worries me is that ``at some
point'' statement; it took the world a while to get over its
fascination with paper tape.  What's more, I'm sure we've both seen
good technology get glossed over, never fledge, and ultimately fail.
Look at Betamax, for instance.

> I don't want these SPAM bastards putting one byte of data on my machine.
> That rules out any approach based on *SMTP.

Well, gosh, they're going to do that anyway even if you go the
filesystem route.  Tauth transactions, attaches, and reads and writes
from/to the auth file all transfer data.  Ultimately, this is probably
less than putting stuff in an SMTP transaction, but if you

> That said, you are correct that only *SMTP-based approach will have big
> impact in the 1-2 year time frame.

Yeah.  Sad but true.  Okay, now it's time to get working.  After
all, ``did we stand still when the German's bombed Pearl Harbor?''[*]
Who's with me?

	- Dan C.


--
[*] For our non-US readers; that's a line from the movie, `Animal House,'
about college fraternities.  If you ever get a chance to see it, I
recommend it.

Re: [9fans] re: spam filtering fs

[ron minnich]

On Sat, 6 Sep 2003, Dan Cross wrote:

> I've been thinking about this, and come to some conclusions.  First,
> that one has to do whatever it is one decides to do within the context
> of SMTP or ESMTP.  While importing a filesystem would be a nice,
> elegant solution, it's just not realistic.

I'm not willing to accept this. You are right, I have no argument with
what you are saying, but I started out with 5-level baudot paper tapes and
we're not using them any more, so the compatibility argument, while
strong, can be trumped at some point by superior technology.

I don't want these SPAM bastards putting one byte of data on my machine.
That rules out any approach based on *SMTP.

That said, you are correct that only *SMTP-based approach will have big
impact in the 1-2 year time frame.

ron

Re: [9fans] re: spam filtering fs

[Dan Cross]

Dave wrote:
> I'ld rather not have to keep a secret and a counter for everyone I want to
> exchange mail with.  Messages get lost and reordered so at the very least
> I need to accept some range of possible sha1ings.  I also want to accept
> mail from people I haven't talked to before but have proved to someone else
> that they aren't spammers.  I'm happier with the /mail idea than this one.

Then I reiterate my second suggestion:

[...]  Or, and even simpler, take the token and sha it with the
contents of the message.  The token itself doesn't show up in any
archives anywhere, and the scheme is immune to problems with bounces
getting sequence numbers out of whack, and you get some modicum of
integrity checking on the message itself.  A way around the client
problem is to build it into the MTA (but the MTA's on both sides have
to support it).

An alternative to this is to provide a nonce in the SMTP transaction or
in header, and the result of the HMAC of the nonce keyedwith the secret
shared between client and sender.  We've already started doing things
like this with the ESMTP AUTH stuff; MAL FROM: now has an ``AUTH''
parameter that can be hung off of it.  We could add a NONCE field,
too.  RCPT TO: can take an AUTHSIG or something, which is some sort of
signature on the nonce value.  We could add an RFC822 style header
called `Integrity: ' that contained the base64 encoding of both the
nonce and the signature in the form, <algo>:<data> (where <algo> is
the hashing algorithm used for the HMAC construction).

Ron had written:
> yeah but ... I don't even want the data coming into my machine. Is that
> covered too? I really want to get these spammers rejected instantly,
> which is why i liked the file system idea.

[Note: I really enjoyed Geoff's colorful description of the spam
problem subsequent to my reply to this....]

I've been thinking about this, and come to some conclusions.  First,
that one has to do whatever it is one decides to do within the context
of SMTP or ESMTP.  While importing a filesystem would be a nice,
elegant solution, it's just not realistic.  And the reason is that it's
not us that's the problem, but everyone else, and everyone else is
firmly mired in the religion of the Internet, which says that SMTP is
the one true way to do mail.  So, sucky though the protocol is, if you
want to do something that has real impact in the next one to two years,
you have to do it within the context of the pre-existing theology.  I
don't like it, but there it is.

Anyway, given that, I really think the simplest way is to do is to
append some sort of signature to a email in the SMTP transaction, but
it doesn't have to be nearly as complex as PGP or S/MIME; something
simpler is going to raise the bar sufficiently to thwart a lot of
the spammers forever.  Just doing an HMAC of a nonce, keyed with a
shared token, is going to stop a lot of the garbage that currently
filters through.

	- Dan C.

Re: [9fans] re: spam filtering fs

[boyd, rounin]

> % cat > /mail/box/boyd/pipeto
> #!/bin/rc
> cat > /dev/null
> exit 0

LOL

but, simple is good.

Re: [9fans] re: spam filtering fs

[David Presotto]

% cat > /mail/box/boyd/pipeto
#!/bin/rc
cat > /dev/null
exit 0

Re: [9fans] re: spam filtering fs

[boyd, rounin]

as i've said before (thanks to Pere Ubu and Living Colour):

    But I don't need a cure
    D-don't need a cure
    D-don't need a cure
    I need a final solution

and i want a _simple_, easily implemented 'final solution'.

Re: [9fans] re: spam filtering fs

[ron minnich]

On Thu, 4 Sep 2003, Lucio De Re wrote:

> Hm.  I thought ISODE was semi-proprietary?  The NetBSD group had
> some comments on it, but it was a long time ago.  I'll have to do
> some digging.

no, I used to cut magtapes for that project (a nice cash cow for UDEL
while it lasted). I also dealt with the code, ack, ack, arg.

Leaves a lingering distaste in me to this day.

But my memory was northrop worked out a kind of open source license but
they didn't want to get sued, so it had some odd clauses.

ron

Re: [9fans] re: spam filtering fs

[George Michaelson]

On Thu, 4 Sep 2003 08:10:39 +0200 Lucio De Re <lucio@proxima.alt.za> wrote:

> On Thu, Sep 04, 2003 at 04:15:52PM +1000, George Michaelson wrote:
> >
> > ISODE. bloatware, but does have YACC based core behaviour. Used to work on
> > it.
> >
> Hm.  I thought ISODE was semi-proprietary?  The NetBSD group had
> some comments on it, but it was a long time ago.  I'll have to do
> some digging.
>
> ++L

Well post '86/7 sometime yes, a form of ISODE went semi-proprietary, but all
code up to that time was fully in the open eye, and it had ASN.1 for sure. the
newer'packed encoding' stuff, I don't know how well it does that. but the basic
B.E.R. should be fine.

I think there would be ISODE code out there post 1990 which was safe too in
fact.

(it had your basic hold-harmless clause on it, from memory)

-George


--
George Michaelson       |  APNIC
Email: ggm@apnic.net    |  PO Box 2131 Milton QLD 4064
Phone: +61 7 3367 0490  |  Australia
  Fax: +61 7 3367 0482  |  http://www.apnic.net

Re: [9fans] re: spam filtering fs

[George Michaelson]

ISODE. bloatware, but does have YACC based core behaviour. Used to work on it.

Marshall Rose code. (lots of lazy eval embedded side-effects in if/then/else
logic) but the core of the code is pretty small. his ASN.1 stuff does encode to
string, or file descriptor, or parse-struct.

I doubt if you'd need the recent code, for a core ASN.1 parser.

there are also heaps of stripped-down codebases for SNMP.

INRIA did a free parser which was pretty small. Christian Huitema (now with
Microsoft research) would know about it.

-george

Re: [9fans] re: spam filtering fs

[Lucio De Re]

On Thu, Sep 04, 2003 at 04:15:52PM +1000, George Michaelson wrote:
>
> ISODE. bloatware, but does have YACC based core behaviour. Used to work on it.
>
Hm.  I thought ISODE was semi-proprietary?  The NetBSD group had
some comments on it, but it was a long time ago.  I'll have to do
some digging.

++L

Re: [9fans] re: spam filtering fs

[Lucio De Re]

On Wed, Sep 03, 2003 at 03:54:03PM -0400, David Presotto wrote:
>
> We have a whole bunch of people here who each started writing an ASN.1
> compiler (or stub generator or whatever you call it), gathered up
> the 4 feet worth of documentation, started writing YACC scripts,
> and then said ``they can't possibly mean that, this is too much work,
> I think I'll so something easy instead.''

Any leftovers that I may be able to use?  ASN.1 (well, BER, anyway)
looks like a one-off solution to a large number of problems in my
unsophisticated view.  It is overspecified, but that's ITU-T for
you:  specifications with no (easily identified) legal loopholes.

All ASN.1 compilers I have ever heard mentioned are proprietary
tools, having a Plan 9 "open source" version would be real asset.

Not that I'm an authority on this, I seem to be working very much
in a vacuum every time I consider any ITU-T offering.

++L

PS: To be perfectly honest, I think Inferno/Limbo would be even more
useful to develop mail processing tools and I wonder if forsyth and
friends shouldn't be closely involved.  Then we have stuff that will
run on Windows, where it is much more necessary.

Re: [9fans] re: spam filtering fs

[Lucio De Re]

On Wed, Sep 03, 2003 at 02:25:27PM +0200, boyd, rounin wrote:
>
> the PKI is shot full of holes.
>
References, please.  I'm sure you're right, but I can't just quote
you :-)

> hacking [E]SMTP is probably a bad idea and only solves
> the problem for the hacked implementations.
>
Might be a good springboard for factotum to get some prime time.

> i still go for token in the sig.  it's simple and raises the bar -- that's
> what security is about.

PGP, then?  Accessing the public key servers is still a requirement,
although that's not hard to implement.

Russ should have a partial port of PGP that we'll need.

++L

Re: [9fans] re: spam filtering fs

[boyd, rounin]

> and then said ``they can't possibly mean that, this is too much work,
> I think I'll so something easy instead.''

oh yes they did.  i saw the culprit give an X.[45]00 course and ASN.1
was part of it.  i should have retired to the bar.

Re: [9fans] re: spam filtering fs

[David Presotto]

[-- Attachment #1: Type: text/plain, Size: 317 bytes --]

We have a whole bunch of people here who each started writing an ASN.1
compiler (or stub generator or whatever you call it), gathered up
the 4 feet worth of documentation, started writing YACC scripts,
and then said ``they can't possibly mean that, this is too much work,
I think I'll so something easy instead.''

[-- Attachment #2: Type: message/rfc822, Size: 2422 bytes --]

From: "boyd, rounin" <boyd@insultant.net>
To: <9fans@cse.psu.edu>
Subject: Re: [9fans] re: spam filtering fs
Date: Wed, 3 Sep 2003 14:03:53 +0200
Message-ID: <019201c37213$72df7ca0$b9844051@insultant.net>

> No guarantee, but I'm game.  I'm neither afraid of ASN.1 nor a fan of
> reinventing the wheel.  Anyone else volunteering?

last time i was at the labs ['98] presotto was playing with it.

S/MIME is probably a better choice if we are determined on overkill.

Re: [9fans] re: spam filtering fs

[Dan Cross]

> > If it is our network bandwidth that we consider to be valuable ...
>
> ... then we wouldn't be reading 9fans.

9fans is low bandwidth compared to some of the ridiculously over-used
Unix lists.

	- Dan C.

Re: [9fans] re: spam filtering fs

[ron minnich]

On Wed, 3 Sep 2003, Lucio De Re wrote:

> On Tue, Sep 02, 2003 at 09:56:05AM -0400, Eric Grosse wrote:
> >
> > Any volunteers to implement S/MIME for Plan 9?   A couple of us here at
> > Bell Labs have worked on it off and on, but there aren't enough free
> > hands here to get it done promptly.  Step one is to implement CMS (also
> > known as PKCS#7 or rfc2315) starting from the ASN.1 goo in
> > /sys/src/libsec/port/x509.c or, if you prefer, by porting an ASN.1
> > compiler.
> >
> No guarantee, but I'm game.  I'm neither afraid of ASN.1 nor a fan of
> reinventing the wheel.  Anyone else volunteering?

the file system still looks easier to me than all this stuff.

ron

Re: [9fans] re: spam filtering fs

[matt]

If it is our network bandwidth that we consider to be valuable then perhaps a strategy of assigning variable bandwidth to remote connections dependent on certain tokens could be an option.

Bandwidth assigned by a weighting from combinations of remote machine / envelope sender / E?HELO field could mean welcome email is accepted quickly whereas unusual mail is given less resources.

If you refused connections from any IP in .ru that didn't have a known envelope sender during peak hours then you could be smoothing your demand.

I am thinking that in such a scheme all mail will be delivered eventually.
I haven't studied the subject but I suppose that monitoring the connections to your mailserver should be revealing with regard to the weighting system. A sudden rush of email from a remote IP can be reacted to progressively throttling the bandwidth available to that IP.

This has little bearing on your mbox management I suppose, except such information can be passed through in extra headers.

Re: [9fans] re: spam filtering fs

[Russ Cox]

> If it is our network bandwidth that we consider to be valuable ...

... then we wouldn't be reading 9fans.

Re: [9fans] re: spam filtering fs

[boyd, rounin]

> There are probably big problems with this, ... But
> it struck me as I was reading Geoff's message.

yes, there are.

Re: [9fans] re: spam filtering fs

[boyd, rounin]

> This could almost be cobbled together today using SMTP and POP3 or
> (god help us) IMAP4.

in a previous life i argued (bitterly) for a while about the futility of
adding an email interface to a system.  i have spent far too many
hours/days/weeks/years? with 822 and SMTP and the problem
is less than trivial.

Re: [9fans] re: spam filtering fs

[boyd, rounin]

> yeah but ... I don't even want the data coming into my machine. Is that
> covered too? I really want to get these spammers rejected instantly, which
> is why i liked the file system idea.

vix had already done it years ago with EBGP, but the problem is that
spam sites pop up without end.  target 'em and then blow 'em away
i say, which means you have to accept the message and any solution
must be 'fail safe'; no message should be chucked because some 3rd
party thing is bust and you can't get to/at it.

Re: [9fans] re: spam filtering fs

[boyd, rounin]

the PKI is shot full of holes.

hacking [E]SMTP is probably a bad idea and only solves
the problem for the hacked implementations.

i still go for token in the sig.  it's simple and raises the bar -- that's
what security is about.

Re: [9fans] re: spam filtering fs

[boyd, rounin]

> No guarantee, but I'm game.  I'm neither afraid of ASN.1 nor a fan of
> reinventing the wheel.  Anyone else volunteering?

last time i was at the labs ['98] presotto was playing with it.

S/MIME is probably a better choice if we are determined on overkill.

Re: [9fans] re: spam filtering fs

[Lyndon Nerenberg]

> My mail exchanger accepts mail that is "certified" and for which it
> has the certificate public key.  Certified mail contains either a
> signature in the body as with PGP or a header of some description,
> encrypted with the sender's private key so it can be decrypted and
> validated.  A preferable form of encryption would be at the SMTP
> protocol level, but this is a different model.

SMTP AUTH cannot (reasonably) solve this problem. What SMTP AUTH is
intended to address is the problem where a road warrior's laptop needs to
inject mail via a home-agent MTA. It can only authenticate the laptop to
the home MTA. It cannot authenticate the originator of the mail coming
from the laptop. (PGP and S/MIME try to solve that problem.)

Open SMTP relays fall into two categories:

1) those operated by people who haven't a clue, or

2) those operated by people who need to allow remote relay but are too
   stupid (or cheap) to acquire MUA software that supports SMTP AUTH
   for just this purpose.

If people would use SMTP AUTH to solve problem #2, problem #0 (the need
for PGP or S/MIME signatures to bypass filters) would mostly just go away.

--lyndon

Re: [9fans] re: spam filtering fs

[lucio]

On Wed, Sep 03, 2003 at 10:24:54AM +0200, Fco.J.Ballesteros wrote:
>
> > not frightened off getting a certificate.  And some form of recourse
> > in the event of someone stealing the e-mail address, and that's the
> > hard part, sadly.
>
> Not just the sad, but also the common part. Most of the spam I get
> seems to use addresses from someone else.
>
> I'm afraid that certifying the from address would not work.
> I hope bayes is right.

This is the scenario I think would work:

My mail exchanger accepts mail that is "certified" and for which it
has the certificate public key.  Certified mail contains either a
signature in the body as with PGP or a header of some description,
encrypted with the sender's private key so it can be decrypted and
validated.  A preferable form of encryption would be at the SMTP
protocol level, but this is a different model.

The message may convey the public key in the headers as suggested by
the Privacy Enhanced Mail (PEM) RFCs, but then there has to be a CA in
the certificate hierarchy that validates the trust.

If trust cannot be validated, I suggest that a group of public
certificate servers, probably including the existing PGP public key
servers, should be queried for the certificate/public key.  If the
certification cannot be established in this fashion, then the
difficult procedure comes into action.

Here we expect the exchanger to submit a request to a preferred public
certificate server that causes the sender to be polled.  If the sender
replies with a valid certificate (or public key), it is stored in the
public server and forwarded to the exchanger, if not, then within some
time limit the exchanger is notified.

I hope I didn't abbreviate the above beyond usefulness, I'll be happy
to expand if I haven't been clear in any way.  And I will of course be
interested in flaws as well as improvements.

++L

PS: At the SMTP level, I would suggest an exchange between servers
that has contractual value.  In other words, the sending exchanger
ought to accept legal liability for mail it insists in forwarding.
Legislation to this effect would have to be enacted, naturally.

Re: [9fans] re: spam filtering fs

[Fco.J.Ballesteros]

> not frightened off getting a certificate.  And some form of recourse
> in the event of someone stealing the e-mail address, and that's the
> hard part, sadly.

Not just the sad, but also the common part. Most of the spam I get
seems to use addresses from someone else.

I'm afraid that certifying the from address would not work.
I hope bayes is right.

Re: [9fans] re: spam filtering fs

[Lucio De Re]

On Tue, Sep 02, 2003 at 09:56:05AM -0400, Eric Grosse wrote:
>
> Any volunteers to implement S/MIME for Plan 9?   A couple of us here at
> Bell Labs have worked on it off and on, but there aren't enough free
> hands here to get it done promptly.  Step one is to implement CMS (also
> known as PKCS#7 or rfc2315) starting from the ASN.1 goo in
> /sys/src/libsec/port/x509.c or, if you prefer, by porting an ASN.1
> compiler.
>
No guarantee, but I'm game.  I'm neither afraid of ASN.1 nor a fan of
reinventing the wheel.  Anyone else volunteering?

> By the way, I've happily used PGP for many years but decided that S/MIME
> was more likely to catch on because it is already moderately well
> supported by default in Outlook and Netscape/Mozilla.
>
PGP and S/MIME address different audiences.  What is needed is a
lightweight CA that merely certifies an e-mail address (like the
PGP public key servers if they were restricted to e-mail addresses
on a first-come, first-served basis) so that users are encouraged,
not frightened off getting a certificate.  And some form of recourse
in the event of someone stealing the e-mail address, and that's the
hard part, sadly.

++L

Re: [9fans] re: spam filtering fs

[Fco.J.Ballesteros]

> A way to exchange tokens: instead of doing it via email, generate an
> image for an unknown user, put it on a public web server somewhere, and
> send them a URL.  Once they get there, have them send back a
> description of the image and then send them a token.  This defeats

But I would send/reply then almost no mail; unless I fell it's worth
the burden of running your procedure, which is not too often in my
case ☺.

Re: [9fans] re: spam filtering fs

[david presotto]

I'ld rather not have to keep a secret and a counter for everyone I want to
exchange mail with.
Messages get lost and reordered so at the very least I need to accept some
range of possible
sha1ings.  I also want to accept mail from people I haven't talked to before
but have proved
to someone else that they aren't spammers.  I'm happier with the /mail idea
than this one.

----- Original Message -----
From: "Dan Cross" <cross@math.psu.edu>
To: <9fans@cse.psu.edu>
Sent: Tuesday, September 02, 2003 8:59 PM
Subject: Re: [9fans] re: spam filtering fs


> Dave writes:
> > What smime (and pgp) can achieve is digital signing so that spammers
> > can't masquerade with From:'s of people in your white list.
>
> So does having an X-header that has a token in it.  One easy way around
> the harvesting-from-a-mailing-list-archive thing is doing something
> S/Key-ish:  The first time you send an email to someone, send the token
> sha'ed 100,000 times.  The next time, send it sha'ed 99,999 times,
> etc.  Both sides keep track of the token and the current sequence
> number.  Or, and even simpler, take the token and sha it with the
> contents of the message.  The token itself doesn't show up in any
> archives anywhere, and the scheme is immune to problems with bounces
> getting sequence numbers out of whack, and you get some modicum of
> integrity checking on the message itself.  A way around the client
> problem is to build it into the MTA (but the MTA's on both sides have
> to support it).
>
> Ron writes:
> > yeah but ... I don't even want the data coming into my machine. Is that
> > covered too? I really want to get these spammers rejected instantly,
which
> > is why i liked the file system idea.
>
> I think we've lost that battle.  Some knocking at the castle gates
> is always going to happen now days.  :-(
>
> - Dan C.
>
>

Re: [9fans] re: spam filtering fs

[Micah Stetson]

On Tue, Sep 02, 2003 at 06:50:34PM -0700, Geoff Collyer wrote:
> Knocking at the gate is one thing, crashing through with a bus full of
> aggressive salesmen in loud checks and plaids is another.  A scheme
> that seems to be closer to what Ron wants would be to have the sending
> system contact the receiving system and announce that user so-and-so

I wonder if a combined approach would work.  Once the
receiving system gets MAIL FROM and RCPT TO from the
sending system, it checks the 'to' user's white list and,
if the sender address isn't on it, it refuses the message
(maybe delaying somewhat in doing so, i.e. tarpitting)
with a 'wait till later' error code and sticks the from
address in a grey list.  It then constructs a challenge
message which is sent to the sender and asks a question
that no current computer program can answer.  When that
question is answered correctly (either by e-mail to a
secondary address, i.e. user+auth-<unique-id>@domain.com,
or by filling out an HTML form or whatever), the address
is put in the user's white-list automatically and further
mail is let through.

Some difficulty comes when a user whose mail server
implements this policy tries to contact another user with
a similar anti-spam system.  The best way to handle this
is to whitelist every address the user sends mail to,
however, that may be troublesome to implement in certain
situations.  Perhaps it would be acceptable to give a
unique identifier to every challenge that goes out and
make it be from user+auth-<id>@domain.com.  Then mail
that comes to that kind of address is checked to see if
the id matches that of a challenge sent to the from
address.  Then we know that this message is either an
answer to the challenge, or another challenge.  These
could be differentiated by placing an X-Challenge header
on every challenge message.  Right and wrong answers
would be handled directly, and messages claiming to be
a challenge could be placed in a holding area that the
user would check periodically.

The point of this is that no modification is needed either
to the protocol or the user agent.  Everything is done
on the recipient's mail server.  Even if spam did get
through (i.e. the challenges aren't good enough, or the
spam is masquerading as a challenge), you know that the
sending address is a valid, reachable address.  This is
certainly a win over the current system.

There are probably big problems with this, as I haven't
applied more than a few minutes of thought to it.  But
it struck me as I was reading Geoff's message.

Micah

Re: [9fans] re: spam filtering fs

[Geoff Collyer]

Knocking at the gate is one thing, crashing through with a bus full of
aggressive salesmen in loud checks and plaids is another.  A scheme
that seems to be closer to what Ron wants would be to have the sending
system contact the receiving system and announce that user so-and-so
has a message of this size with this message-id that he wants to send
to user thus-and-such on the receiving system; possibly there's some
form of authentication that it really is so-and-so sending it.  If
thus-and-such's white-list permits it (presumably based on the sending
user), the message is let through; if his black-list denies it, the
message is refused, and if neither list mentions the sender, the
sending system is told `we'll get back to you', the message is not
accepted, and the receiving system appends a brief notification to
thus-and-such's gray-list.  Periodically, maybe once a day,
thus-and-such scans his gray-list and if he sees a message that he
wants to look at it, he fetches it by message-id from the sending
system, incurring the band-width expenditure then.

This could almost be cobbled together today using SMTP and POP3 or
(god help us) IMAP4.  The major change would be to have the sending
system hang on to unsent (gray-listed) messages for some period so
that they may be retrieved.  I suppose that could almost be
accomplished by having smtpd return the cabbalistic SMTP return code
meaning `try again later'.

Re: [9fans] re: spam filtering fs

[Dan Cross]

Dave writes:
> What smime (and pgp) can achieve is digital signing so that spammers
> can't masquerade with From:'s of people in your white list.

So does having an X-header that has a token in it.  One easy way around
the harvesting-from-a-mailing-list-archive thing is doing something
S/Key-ish:  The first time you send an email to someone, send the token
sha'ed 100,000 times.  The next time, send it sha'ed 99,999 times,
etc.  Both sides keep track of the token and the current sequence
number.  Or, and even simpler, take the token and sha it with the
contents of the message.  The token itself doesn't show up in any
archives anywhere, and the scheme is immune to problems with bounces
getting sequence numbers out of whack, and you get some modicum of
integrity checking on the message itself.  A way around the client
problem is to build it into the MTA (but the MTA's on both sides have
to support it).

Ron writes:
> yeah but ... I don't even want the data coming into my machine. Is that
> covered too? I really want to get these spammers rejected instantly, which
> is why i liked the file system idea.

I think we've lost that battle.  Some knocking at the castle gates
is always going to happen now days.  :-(

	- Dan C.

Re: [9fans] re: spam filtering fs

[ron minnich]

On Tue, 2 Sep 2003, david presotto wrote:

> What smime (and pgp) can achieve is digital signing so that spammers
> can't masquerade with From:'s of people in your white list.


yeah but ... I don't even want the data coming into my machine. Is that
covered too? I really want to get these spammers rejected instantly, which
is why i liked the file system idea.

ron

Re: [9fans] re: spam filtering fs

[david presotto]

What smime (and pgp) can achieve is digital signing so that spammers can't
masquerade
with From:'s of people in your white list.
----- Original Message -----
From: "Dan Cross" <cross@math.psu.edu>
To: <9fans@cse.psu.edu>
Sent: Tuesday, September 02, 2003 12:08 PM
Subject: Re: [9fans] re: spam filtering fs


> > Another way of achieving authentication for email is to implement and
> > use S/MIME or PGP.  I'm not sure either that or "import ... /mail"
solves
> > the computational cost of spam if the bad guys create invalid
signatures,
> > but it does make a white-list filter more effective.
>
> I see the two as complimentary.  Just because you're securing the contents
> of the wagon by wrapping them in a patrol of the King's men-at-arms
doesn't
> mean you shouldn't also endeavor to clear out the highway robbers.
>
> > Any volunteers to implement S/MIME for Plan 9?   A couple of us here at
> > Bell Labs have worked on it off and on, but there aren't enough free
> > hands here to get it done promptly.  Step one is to implement CMS (also
> > known as PKCS#7 or rfc2315) starting from the ASN.1 goo in
> > /sys/src/libsec/port/x509.c or, if you prefer, by porting an ASN.1
> > compiler.
>
> Help!  I'm melting!
>
> > By the way, I've happily used PGP for many years but decided that S/MIME
> > was more likely to catch on because it is already moderately well
> > supported by default in Outlook and Netscape/Mozilla.
>
> I thought there was an effort to merge OpenPGP and S/MIME in some way?
> S/MIME requires a lot of scaffolding to use effectively; PGP has a much
> lower startup cost.  That said, I'm not a big fan of either.  Most
> people don't need that level of privacy (despite what they may think,
> no one's out to get them and the FBI could care less about their D&D
> campaign plans).  For cutting down on spam, this seems like cutting
> butter with a chainsaw.  A much simpler method would be to just put an
> X- header with some sort of agreed upon token into one's email.  Is it
> secure?  Not really, no, but it'll defeat 99% of the wannabes, and
> that's a lot of bang for the buck.  Of course, either would be nice to
> have for other reasons (everyone knows the government *really is* out
> to get Boyd, for instance...).
>
> A way to exchange tokens: instead of doing it via email, generate an
> image for an unknown user, put it on a public web server somewhere, and
> send them a URL.  Once they get there, have them send back a
> description of the image and then send them a token.  This defeats
> auto-harvesters that are smart enough to send you back a reply to our
> ``send this string back if you're not a spammer'' token.  This will
> work for a while until the spammers start to implement image
> recognition software.
>
> - Dan C.
>
>

Re: [9fans] re: spam filtering fs

[boyd, rounin]

> A way to exchange tokens: instead of doing it via email, generate an
> image for an unknown user, put it on a public web server somewhere, and
> send them a URL.  Once they get there, have them send back a
> description of the image and then send them a token.

i like this idea but if you stick the token as an X-* (with every message)
field it's next to impossible (or i don't know how to do it) to get Outlook
and maybe other UAs to a) insert the token and b) insert it.

other problem is whether the token is on a per recipient basis.  it just
can't be done with Outlook.  however, if you give out a unique
token to people you like Outlook can then apply a rule if it's in the
body (say signature) of the message.

i know it's susceptible to 'man in the middle' attacks, but it sure would
kill a lotta spam.  oh shit, it's also harvestable from mailing lists that
are archived.

in fact you can just mail people 'make sure this token is in your sig
or mail from you will be spam killed'.  the 'image' in this case is
the content of the message sent to you.

you wanna try this out, dan?

Re: [9fans] re: spam filtering fs

[Dan Cross]

> Another way of achieving authentication for email is to implement and
> use S/MIME or PGP.  I'm not sure either that or "import ... /mail" solves
> the computational cost of spam if the bad guys create invalid signatures,
> but it does make a white-list filter more effective.

I see the two as complimentary.  Just because you're securing the contents
of the wagon by wrapping them in a patrol of the King's men-at-arms doesn't
mean you shouldn't also endeavor to clear out the highway robbers.

> Any volunteers to implement S/MIME for Plan 9?   A couple of us here at
> Bell Labs have worked on it off and on, but there aren't enough free
> hands here to get it done promptly.  Step one is to implement CMS (also
> known as PKCS#7 or rfc2315) starting from the ASN.1 goo in
> /sys/src/libsec/port/x509.c or, if you prefer, by porting an ASN.1
> compiler.

Help!  I'm melting!

> By the way, I've happily used PGP for many years but decided that S/MIME
> was more likely to catch on because it is already moderately well
> supported by default in Outlook and Netscape/Mozilla.

I thought there was an effort to merge OpenPGP and S/MIME in some way?
S/MIME requires a lot of scaffolding to use effectively; PGP has a much
lower startup cost.  That said, I'm not a big fan of either.  Most
people don't need that level of privacy (despite what they may think,
no one's out to get them and the FBI could care less about their D&D
campaign plans).  For cutting down on spam, this seems like cutting
butter with a chainsaw.  A much simpler method would be to just put an
X- header with some sort of agreed upon token into one's email.  Is it
secure?  Not really, no, but it'll defeat 99% of the wannabes, and
that's a lot of bang for the buck.  Of course, either would be nice to
have for other reasons (everyone knows the government *really is* out
to get Boyd, for instance...).

A way to exchange tokens: instead of doing it via email, generate an
image for an unknown user, put it on a public web server somewhere, and
send them a URL.  Once they get there, have them send back a
description of the image and then send them a token.  This defeats
auto-harvesters that are smart enough to send you back a reply to our
``send this string back if you're not a spammer'' token.  This will
work for a while until the spammers start to implement image
recognition software.

	- Dan C.

Re: [9fans] re: spam filtering fs

[Dan Cross]

> what if we chuck smtp altogether?
>
> no matter how you cut it, the problem originates with smtp and its
> dependence on the kindness of strangers. That was all fine back in The
> Good Old Days, but it really sucks now.

I mentioned all of this to a well-known Internet apologist once, and
was laughed down.  The response was, ``you're an idiot if you think
good technology will solve this problem.''

I see a problem with perception: people perceive SMTP and other
protocols as being good, so they look past their failings.  To really
make a change, you have to demonstrate on a large scale why doing it
the right way (I mean, come on; not authenticating mail connections is
just stupid) is better.

	- Dan C.

Re: [9fans] re: spam filtering fs

[Eric Grosse]

Another way of achieving authentication for email is to implement and
use S/MIME or PGP.  I'm not sure either that or "import ... /mail" solves
the computational cost of spam if the bad guys create invalid signatures,
but it does make a white-list filter more effective.

Any volunteers to implement S/MIME for Plan 9?   A couple of us here at
Bell Labs have worked on it off and on, but there aren't enough free
hands here to get it done promptly.  Step one is to implement CMS (also
known as PKCS#7 or rfc2315) starting from the ASN.1 goo in
/sys/src/libsec/port/x509.c or, if you prefer, by porting an ASN.1
compiler.

By the way, I've happily used PGP for many years but decided that S/MIME
was more likely to catch on because it is already moderately well
supported by default in Outlook and Netscape/Mozilla.

Eric

Re: [9fans] re: spam filtering fs

[boyd, rounin]

> And Iraq.

well i'd dump my M16-A2 and my M4 and get me an MP5-A[35].

and wear level III armor all the time.

Re: [9fans] re: spam filtering fs

[boyd, rounin]

> I was hoping somebody would mention that option, it's probably still legal
> here in NM.

3 months, after they've done the rounds of the psych hospitals i'll be able
to buy another M9 [beretta 92FS]:

    http://www.insultant.net/tools

Re: [9fans] re: spam filtering fs

[Skip Tavakkolian]

> I was hoping somebody would mention that option, it's probably still legal
> here in NM.

And Iraq.

Re: [9fans] re: spam filtering fs

[boyd, rounin]

> what if we chuck smtp altogether?

i like SMTP.  it's 822 that makes it a nightmare.

i'd chuck it, but ESMTP with challenge/response would be an easy hack.

but the real problem is that nobody wants to do security right.

been to a US airport lately?

Re: [9fans] re: spam filtering fs

[ron minnich]

On Tue, 2 Sep 2003, boyd, rounin wrote:

> > But this business of spam filtering sounds like negotiations with burglars
> > after they are inside your living room and packing up your DVD collection.
>
> yup, shoot the fuckers ;)  9x19 124 grain FMJ.

I was hoping somebody would mention that option, it's probably still legal
here in NM.

ron

Re: [9fans] re: spam filtering fs

[boyd, rounin]

> But this business of spam filtering sounds like negotiations with burglars
> after they are inside your living room and packing up your DVD collection.

yup, shoot the fuckers ;)  9x19 124 grain FMJ.

Re: [9fans] re: spam filtering fs

[ron minnich]

what if we chuck smtp altogether?

no matter how you cut it, the problem originates with smtp and its
dependence on the kindness of strangers. That was all fine back in The
Good Old Days, but it really sucks now.

Could we set up a mail protocol that requires something like this:

import somewhere /mail /mail

then open files in there and write to them. Maybe just one file, called
'mailto' or some such.

The fact that you can import means that you have gotten inside a trust
boundary. You'll have a verifiable trace of who wrote what when.

If you're filtering messages after you get them, it's already too late --
they've wasted your disk space and your net bandwidth. The numbers for
ISPs are depressing, in that most of their net bandwidth is SPAM-width.

Obviously you've got to have hosts that span multiple trust boundaries.
Maybe that's too easy to get around. Also, this removes the ability of
someone you don't know to send you mail, but I'm about ready to give that
up, as most of the people I don't know who send me mail now want my bank
account numbers or want to send me a ruler.

But this business of spam filtering sounds like negotiations with burglars
after they are inside your living room and packing up your DVD collection.

Just wondering.
ron

Re: [9fans] re: spam filtering fs

[steve.simon]

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]

Markov chains are the next generation of spam filtering tools
(some of the GPLed tools already do this).

It involves keeping frequencies for token sequences rather than just
individual tokens. It makes the spammers job a lot harder, though
not impossible. The downside is the increase in filters database size
as the number of tokens increases exponentially with chain length.

It all comes down to making the filters model of written emails have more
parameters and being more accurate than the spammers until in the limit
the spam email looks like a valid one (IE. it contains usefull information :-)

-Steve

[-- Attachment #2: Type: message/rfc822, Size: 1092 bytes --]

From: 9fans@cse.psu.edu
To: 9fans@cse.psu.edu
Subject: [9fans] re: spam filtering fs
Date: Mon, 01 Sep 2003 21:31:16 +0100
Message-ID: <1270037699@snellwilcox.com>

in light of recent forged To: & From:
the challenge/response method is going to generate twice as much traffic for such a mail.

As far as I can tell, without trying it, the example pipeto sends a copy of the message, virus payload and all to whoever the From: headers suggest, obviating the need to infect the plan9 host to spread itself.

You also run the risk of ending up sending to an infected host and throwing your email address into another steaming pot.

The next generation will snarf text from existing mails with the intention of defeating Bayes type filters.