[9fans] spam filtering (again)

[9fans] spam filtering (again)

[Steve Simon]

Sadly the spamers seem to be raising the bar,
retrying emails and bypassing the (until now)
very effective plan9 greylist process.

If others are seeing this (and I assume they are)
what is the next step, ratfs(4)? Please assume I
don't have a Lunix machine so SpamCop et ál are
out of the question.

-Steve

Re: [9fans] spam filtering (again)

[geoff]

Greylisting isn't nearly as a effective as the smtp callback that Russ
added (I believe).  smtpd invokes /mail/lib/validatesender if it
exists, and it calls back to validate the nominal sender address
(which must be a valid address according to the RFCs).  Some badly-run
organisations that you want to get mail from insist on sending from
invalid addresses, so validatesender also lists exceptions that don't
require a callback.

The next step?  I've got a new SMTP replacement protocol almost
designed that should cure spam completely and not even let it across
the wire.

A shortened version of my /mail/lib/validatesender is:

#!/bin/rc
# validatesender - call back to validate the nominal sender, which
#	is required by the RFCs to be a valid address
rfork en
fn usage {
	echo 'usage: validatesender [-n /net] plan9.bell-labs.com glenda' >[1=2]
	exit usage
}

echo '$' validatesender $sysname $pid $* >>/sys/log/smtpd.mx

netroot=/net
if(~ $1 -n){
	shift
	netroot=$1
	shift
}
if(! ~ $#* 2)
	usage

dom=$1
addr=$2

# # Lucent only - use external network when mail from external domains
# # is delivered to us internally
netroot=/net
# if(! ~ $dom *.lucent.com lucent.com *.bell-labs.com bell-labs.com){
# 	if(! test -d /net.alt/tcp)
# 		import outside /net.alt
# 	if(test -d /net.alt/tcp)
# 		netroot=/net.alt
# }

# exceptions; mainly for speed by excluding known heavy mailers
if (~ $dom collyer.net collyer.ca && ~ $addr claudia claudia2 geoff)
	exit ''
if (~ $dom cse.psu.edu && ~ $addr '9fans-bounces+geoff.9fans=collyer.net')
	exit ''
if (~ $dom plan9.bell-labs.com && ~ addr jmk rsc brucee)
	exit ''
if (~ $dom paytrust.com && ~ $addr Support)
	exit ''
if (~ $dom boldfish.ieee.org && ~ $addr owner-institute-news)
	exit ''
# ...
# grr, they say `temporary failure' on call-back (could be just greylisting)
if (~ $dom openbsd.org && ~ $addr austin)
	exit ''
# buggered mail systems
if (~ $dom bounces.amazon.com && ~ $addr emailSenderApp+* RealTimeEmail+*)
	exit ''
# aggressive greylisters (and not too smart)
if (~ $dom *.kagi.com kagi.com)
	exit ''

#
# IDIOTS!
#

# cpu Oct 28 15:57:53 [mcafee.com/216.49.92.104] unity4.mcafee.com!msk_us_ex_multipart_list.UM.A.420.48 -> collyer.net!claudia sendercheck: deferred: smtp ping: cpu 822 452 4.2.2 Mailbox full
# cpu Oct 28 16:37:10 [mcafee.com/216.49.92.104] unity4.mcafee.com!msk_us_ex_multipart_list.UM.A.420.48 -> collyer.net!claudia sendercheck: deferred: smtp ping: cpu 7934 smtp: dns: dns failure (/net/tcp!unity4.mcafee.com)
if (~ $dom *.mcafee.com mcafee.com)
	exit ''

# cpu Sep 13 04:20:15 [sgtulmg02-out.sabre.com/151.193.220.19] p75.travelocity.com!tcy -> collyer.net!CLAUDIA sendercheck: deferred: smtp ping: cpu 608687 smtp timeout: connection to /net/tcp!p75.travelocity.com timed out
if (~ $dom *.travelocity.com && ~ $addr tcy)
	exit ''

# cpu Jul 24 17:53:05 [svtii.com/64.175.247.43] svtii.com!seminars -> collyer.net!geoff.ieee sendercheck: rejected: smtp ping: cpu 281560 550 seminars@svtii.com...User unknown
if (~ $dom svtii.com && ~ $addr seminars)
	exit ''

# cpu Jul  2 15:35:44 [VMTA07.S2U2.COM/63.210.43.137] vonage.emsg.net!vonage-return.1846960754 -> collyer.net!geoff.vonage sendercheck: deferred: smtp ping: cpu 215594 smtp timeout: connection to /net/tcp!vonage.emsg.net timed out
if (~ $dom vonage.emsg.net && ~ $addr vonage-return.*)
	exit ''

# cpu Apr  8 01:52:43 [maint4.acm.org/199.222.69.155] acm.org!renewal_receipt -> collyer.net!geoff.acm sendercheck: deferred: smtp ping: cpu 67519 smtp timeout: connection to /net/tcp!acm.org timed out
if (~ $dom maint?.acm.org acm.org)
	exit ''

# $ validatesender cpuf 11876 -n /net surf1.idzap.com wwwzap
# cpuf 11876 smtp: connection refused (/net/tcp!surf1.idzap.com)
# cpu Mar 21 15:35:13 [mx11.sac.fedex.com/199.81.193.118] www51.sac.fedex.com!swww -> collyer.net!geoff sendercheck: deferred: smtp ping: cpu 68992 smtp timeout: connection to /net/tcp!www51.sac.fedex.com timed out
# cpu Mar  6 02:51:48 [masquerade.micron.com/137.201.242.130] micron.com!crucial-do-not-reply -> collyer.net!geoff.crucial sendercheck: rejected: smtp ping: cpu 14579 550 5.1.1 <crucial-do-not-reply@micron.com>... <crucial-do-not-reply@micro
if (~ $dom *.idzap.com idzap.com *.fedex.com *.micron.com micron.com)
	exit ''
# ...

#
# unwelcome spamming vermin
#
if (~ $dom gothnation.com && ~ $addr mosraytai)
	exit 'rejected: smtp ping: bugger off'
if (~ $dom medallionrealators.com)
	exit 'rejected: smtp ping: bugger off'
if (~ $dom cs.toronto.edu && ~ $addr funny-owner)
	exit 'rejected: smtp ping: go away. i asked to be removed from funny@cs.toronto.edu months ago, and now you are just forwarding spam'
if (~ $dom *.sales.overstock.com *.Sales.Overstock.com)
	exit 'rejected: you are violating the RFCs by not sending from a replyable return address'

#
# the real machinery
#

# changed /dev/null to postmaster so that some of dmr's mail could get
# in.  He was getting mail from people who refuse mail from <>. - rsc
if (x=`{upas/smtp -p $netroot/tcp!$dom postmaster $addr >[2=1] |
    sed 's/^/'$sysname' '$pid' /' | tee -a /sys/log/smtpd.mx | tail -1})
	exit ''

smtpstatus=$status
if (~ $#x 0)
	x=$smtpstatus
if (~ $smtpstatus *'Permanent Failure'*)
	exit 'rejected: smtp ping: '^$"x
exit 'deferred: smtp ping: '^$"x

Re: [9fans] spam filtering (again)

[uriel]

> Greylisting isn't nearly as a effective as the smtp callback that Russ
> added (I believe).  smtpd invokes /mail/lib/validatesender if it
> exists, and it calls back to validate the nominal sender address
> (which must be a valid address according to the RFCs).  Some badly-run
> organisations that you want to get mail from insist on sending from
> invalid addresses, so validatesender also lists exceptions that don't
> require a callback.
What do you think of:
http://kmself.home.netcom.com/Rants/challenge-response.html

> The next step?  I've got a new SMTP replacement protocol almost
> designed that should cure spam completely and not even let it across
> the wire.
I have been yearning for that protocol to be made public since boyd
told me about it years ago.

I hope it is released before SMTP gets replaced by some XML-over-HTTP
abomination.

The jabber monkeys have been talking about replacing SMTP with their
byzantine XML proto for years, fortunately I don't think there is much
danger of that, in 7 years of work on their main target area of
instant messaging they have barely reached the functionality level of
irc...  but surely they have produced over one hundred specifications!
that is progress.

But I hear now that google is backing jabber, oh, I love google so
much...  thanks once more for helping plunge the software industry
into the deepest pits of sorrow, pain and complexity.

uriel


> 
> A shortened version of my /mail/lib/validatesender is:
> 
> #!/bin/rc
> # validatesender - call back to validate the nominal sender, which
> #	is required by the RFCs to be a valid address
> rfork en
> fn usage {
> 	echo 'usage: validatesender [-n /net] plan9.bell-labs.com glenda' >[1=2]
> 	exit usage
> }
> 
> echo '$' validatesender $sysname $pid $* >>/sys/log/smtpd.mx
> 
> netroot=/net
> if(~ $1 -n){
> 	shift
> 	netroot=$1
> 	shift
> }
> if(! ~ $#* 2)
> 	usage
> 
> dom=$1
> addr=$2
> 
> # # Lucent only - use external network when mail from external domains
> # # is delivered to us internally
> netroot=/net
> # if(! ~ $dom *.lucent.com lucent.com *.bell-labs.com bell-labs.com){
> # 	if(! test -d /net.alt/tcp)
> # 		import outside /net.alt
> # 	if(test -d /net.alt/tcp)
> # 		netroot=/net.alt
> # }
> 
> # exceptions; mainly for speed by excluding known heavy mailers
> if (~ $dom collyer.net collyer.ca && ~ $addr claudia claudia2 geoff)
> 	exit ''
> if (~ $dom cse.psu.edu && ~ $addr '9fans-bounces+geoff.9fans=collyer.net')
> 	exit ''
> if (~ $dom plan9.bell-labs.com && ~ addr jmk rsc brucee)
> 	exit ''
> if (~ $dom paytrust.com && ~ $addr Support)
> 	exit ''
> if (~ $dom boldfish.ieee.org && ~ $addr owner-institute-news)
> 	exit ''
> # ...
> # grr, they say `temporary failure' on call-back (could be just greylisting)
> if (~ $dom openbsd.org && ~ $addr austin)
> 	exit ''
> # buggered mail systems
> if (~ $dom bounces.amazon.com && ~ $addr emailSenderApp+* RealTimeEmail+*)
> 	exit ''
> # aggressive greylisters (and not too smart)
> if (~ $dom *.kagi.com kagi.com)
> 	exit ''
> 
> #
> # IDIOTS!
> #
> 
> # cpu Oct 28 15:57:53 [mcafee.com/216.49.92.104] unity4.mcafee.com!msk_us_ex_multipart_list.UM.A.420.48 -> collyer.net!claudia sendercheck: deferred: smtp ping: cpu 822 452 4.2.2 Mailbox full
> # cpu Oct 28 16:37:10 [mcafee.com/216.49.92.104] unity4.mcafee.com!msk_us_ex_multipart_list.UM.A.420.48 -> collyer.net!claudia sendercheck: deferred: smtp ping: cpu 7934 smtp: dns: dns failure (/net/tcp!unity4.mcafee.com)
> if (~ $dom *.mcafee.com mcafee.com)
> 	exit ''
> 
> # cpu Sep 13 04:20:15 [sgtulmg02-out.sabre.com/151.193.220.19] p75.travelocity.com!tcy -> collyer.net!CLAUDIA sendercheck: deferred: smtp ping: cpu 608687 smtp timeout: connection to /net/tcp!p75.travelocity.com timed out
> if (~ $dom *.travelocity.com && ~ $addr tcy)
> 	exit ''
> 
> # cpu Jul 24 17:53:05 [svtii.com/64.175.247.43] svtii.com!seminars -> collyer.net!geoff.ieee sendercheck: rejected: smtp ping: cpu 281560 550 seminars@svtii.com...User unknown
> if (~ $dom svtii.com && ~ $addr seminars)
> 	exit ''
> 
> # cpu Jul  2 15:35:44 [VMTA07.S2U2.COM/63.210.43.137] vonage.emsg.net!vonage-return.1846960754 -> collyer.net!geoff.vonage sendercheck: deferred: smtp ping: cpu 215594 smtp timeout: connection to /net/tcp!vonage.emsg.net timed out
> if (~ $dom vonage.emsg.net && ~ $addr vonage-return.*)
> 	exit ''
> 
> # cpu Apr  8 01:52:43 [maint4.acm.org/199.222.69.155] acm.org!renewal_receipt -> collyer.net!geoff.acm sendercheck: deferred: smtp ping: cpu 67519 smtp timeout: connection to /net/tcp!acm.org timed out
> if (~ $dom maint?.acm.org acm.org)
> 	exit ''
> 
> # $ validatesender cpuf 11876 -n /net surf1.idzap.com wwwzap
> # cpuf 11876 smtp: connection refused (/net/tcp!surf1.idzap.com)
> # cpu Mar 21 15:35:13 [mx11.sac.fedex.com/199.81.193.118] www51.sac.fedex.com!swww -> collyer.net!geoff sendercheck: deferred: smtp ping: cpu 68992 smtp timeout: connection to /net/tcp!www51.sac.fedex.com timed out
> # cpu Mar  6 02:51:48 [masquerade.micron.com/137.201.242.130] micron.com!crucial-do-not-reply -> collyer.net!geoff.crucial sendercheck: rejected: smtp ping: cpu 14579 550 5.1.1 <crucial-do-not-reply@micron.com>... <crucial-do-not-reply@micro
> if (~ $dom *.idzap.com idzap.com *.fedex.com *.micron.com micron.com)
> 	exit ''
> # ...
> 
> #
> # unwelcome spamming vermin
> #
> if (~ $dom gothnation.com && ~ $addr mosraytai)
> 	exit 'rejected: smtp ping: bugger off'
> if (~ $dom medallionrealators.com)
> 	exit 'rejected: smtp ping: bugger off'
> if (~ $dom cs.toronto.edu && ~ $addr funny-owner)
> 	exit 'rejected: smtp ping: go away. i asked to be removed from funny@cs.toronto.edu months ago, and now you are just forwarding spam'
> if (~ $dom *.sales.overstock.com *.Sales.Overstock.com)
> 	exit 'rejected: you are violating the RFCs by not sending from a replyable return address'
> 
> #
> # the real machinery
> #
> 
> # changed /dev/null to postmaster so that some of dmr's mail could get
> # in.  He was getting mail from people who refuse mail from <>. - rsc
> if (x=`{upas/smtp -p $netroot/tcp!$dom postmaster $addr >[2=1] |
>     sed 's/^/'$sysname' '$pid' /' | tee -a /sys/log/smtpd.mx | tail -1})
> 	exit ''
> 
> smtpstatus=$status
> if (~ $#x 0)
> 	x=$smtpstatus
> if (~ $smtpstatus *'Permanent Failure'*)
> 	exit 'rejected: smtp ping: '^$"x
> exit 'deferred: smtp ping: '^$"x

Re: [9fans] spam filtering (again)

[geoff]

validatesender isn't doing challenge/response, though I use that too.
It's just insisting that the nominal MAIL FROM:<> address be valid,
unless you've exempted it explicitly.  I guess my reponse to the web
page is that it's my inbox, so I'm justified in controlling what lands
in it.  I'm also paying for the bandwidth into my house, so I'm
justified in preventing spam from crossing my DSL line.
validatesender works really well (I had to watch the smtpd logs at
first, but I tend to do that anyway), and challenge/response as
implemented by presotto in pipeto (what I call pipeto.token) works
well too.

Actually the `next step' protocol I'm talking about is not RSMTP; I
think that one is stuck in the wheels of the patent office and they've
stopped turning.  I call this new one NSTP (non-spam transfer
protocol) and it includes some properties of RSMTP but is a new
protocol.  It's also not finished and I haven't found anyone who wants
to pay me to finish it and implement it yet.

Re: [9fans] spam filtering (again)

[ems]

On Wed, 2006-02-08 at 01:09 -0800, geoff@collyer.net wrote:
> The next step?  I've got a new SMTP replacement protocol almost
> designed that should cure spam completely and not even let it across
> the wire.

djb came up with a very simple solution, http://cr.yp.to/im2000.html

I have somewhere on my to do list, to make a mail replacement, using the
concept found in im2000 (but over 9P, not http), but I am sure someone
else might be able to get there first if he wishes.

Maybe we can convince the genius, Rob Pike, to convince Google to do it,
that way such a replacement would get off the ground, very quickly, and
hyped up to off the chart numbers?

-- ems

Re: [9fans] spam filtering (again)

[uriel]

> On Wed, 2006-02-08 at 01:09 -0800, geoff@collyer.net wrote:
> Maybe we can convince the genius, Rob Pike, to convince Google to do it,

Google might be interested if it uses an XML format, over WebDAV,
allows embedded JavaScript and AJAX inside emails and the server is
written in a combination of Java and C++ components talking to each
other over SOAP.  The client implementation can either be in Python,
XUL or Flash.  And don't forget some DNS extensions to tie it all
together.

Oh, sorry, that would be very much like Jabber, which is what Google
is using already... we are too late, the Web 5.0 is upon us.

uriel