Re: [9fans] spam filtering (again)

[geoff@collyer.net]

Greylisting isn't nearly as a effective as the smtp callback that Russ
added (I believe).  smtpd invokes /mail/lib/validatesender if it
exists, and it calls back to validate the nominal sender address
(which must be a valid address according to the RFCs).  Some badly-run
organisations that you want to get mail from insist on sending from
invalid addresses, so validatesender also lists exceptions that don't
require a callback.

The next step?  I've got a new SMTP replacement protocol almost
designed that should cure spam completely and not even let it across
the wire.

A shortened version of my /mail/lib/validatesender is:

#!/bin/rc
# validatesender - call back to validate the nominal sender, which
#	is required by the RFCs to be a valid address
rfork en
fn usage {
	echo 'usage: validatesender [-n /net] plan9.bell-labs.com glenda' >[1=2]
	exit usage
}

echo '$' validatesender $sysname $pid $* >>/sys/log/smtpd.mx

netroot=/net
if(~ $1 -n){
	shift
	netroot=$1
	shift
}
if(! ~ $#* 2)
	usage

dom=$1
addr=$2

# # Lucent only - use external network when mail from external domains
# # is delivered to us internally
netroot=/net
# if(! ~ $dom *.lucent.com lucent.com *.bell-labs.com bell-labs.com){
# 	if(! test -d /net.alt/tcp)
# 		import outside /net.alt
# 	if(test -d /net.alt/tcp)
# 		netroot=/net.alt
# }

# exceptions; mainly for speed by excluding known heavy mailers
if (~ $dom collyer.net collyer.ca && ~ $addr claudia claudia2 geoff)
	exit ''
if (~ $dom cse.psu.edu && ~ $addr '9fans-bounces+geoff.9fans=collyer.net')
	exit ''
if (~ $dom plan9.bell-labs.com && ~ addr jmk rsc brucee)
	exit ''
if (~ $dom paytrust.com && ~ $addr Support)
	exit ''
if (~ $dom boldfish.ieee.org && ~ $addr owner-institute-news)
	exit ''
# ...
# grr, they say `temporary failure' on call-back (could be just greylisting)
if (~ $dom openbsd.org && ~ $addr austin)
	exit ''
# buggered mail systems
if (~ $dom bounces.amazon.com && ~ $addr emailSenderApp+* RealTimeEmail+*)
	exit ''
# aggressive greylisters (and not too smart)
if (~ $dom *.kagi.com kagi.com)
	exit ''

#
# IDIOTS!
#

# cpu Oct 28 15:57:53 [mcafee.com/216.49.92.104] unity4.mcafee.com!msk_us_ex_multipart_list.UM.A.420.48 -> collyer.net!claudia sendercheck: deferred: smtp ping: cpu 822 452 4.2.2 Mailbox full
# cpu Oct 28 16:37:10 [mcafee.com/216.49.92.104] unity4.mcafee.com!msk_us_ex_multipart_list.UM.A.420.48 -> collyer.net!claudia sendercheck: deferred: smtp ping: cpu 7934 smtp: dns: dns failure (/net/tcp!unity4.mcafee.com)
if (~ $dom *.mcafee.com mcafee.com)
	exit ''

# cpu Sep 13 04:20:15 [sgtulmg02-out.sabre.com/151.193.220.19] p75.travelocity.com!tcy -> collyer.net!CLAUDIA sendercheck: deferred: smtp ping: cpu 608687 smtp timeout: connection to /net/tcp!p75.travelocity.com timed out
if (~ $dom *.travelocity.com && ~ $addr tcy)
	exit ''

# cpu Jul 24 17:53:05 [svtii.com/64.175.247.43] svtii.com!seminars -> collyer.net!geoff.ieee sendercheck: rejected: smtp ping: cpu 281560 550 seminars@svtii.com...User unknown
if (~ $dom svtii.com && ~ $addr seminars)
	exit ''

# cpu Jul  2 15:35:44 [VMTA07.S2U2.COM/63.210.43.137] vonage.emsg.net!vonage-return.1846960754 -> collyer.net!geoff.vonage sendercheck: deferred: smtp ping: cpu 215594 smtp timeout: connection to /net/tcp!vonage.emsg.net timed out
if (~ $dom vonage.emsg.net && ~ $addr vonage-return.*)
	exit ''

# cpu Apr  8 01:52:43 [maint4.acm.org/199.222.69.155] acm.org!renewal_receipt -> collyer.net!geoff.acm sendercheck: deferred: smtp ping: cpu 67519 smtp timeout: connection to /net/tcp!acm.org timed out
if (~ $dom maint?.acm.org acm.org)
	exit ''

# $ validatesender cpuf 11876 -n /net surf1.idzap.com wwwzap
# cpuf 11876 smtp: connection refused (/net/tcp!surf1.idzap.com)
# cpu Mar 21 15:35:13 [mx11.sac.fedex.com/199.81.193.118] www51.sac.fedex.com!swww -> collyer.net!geoff sendercheck: deferred: smtp ping: cpu 68992 smtp timeout: connection to /net/tcp!www51.sac.fedex.com timed out
# cpu Mar  6 02:51:48 [masquerade.micron.com/137.201.242.130] micron.com!crucial-do-not-reply -> collyer.net!geoff.crucial sendercheck: rejected: smtp ping: cpu 14579 550 5.1.1 <crucial-do-not-reply@micron.com>... <crucial-do-not-reply@micro
if (~ $dom *.idzap.com idzap.com *.fedex.com *.micron.com micron.com)
	exit ''
# ...

#
# unwelcome spamming vermin
#
if (~ $dom gothnation.com && ~ $addr mosraytai)
	exit 'rejected: smtp ping: bugger off'
if (~ $dom medallionrealators.com)
	exit 'rejected: smtp ping: bugger off'
if (~ $dom cs.toronto.edu && ~ $addr funny-owner)
	exit 'rejected: smtp ping: go away. i asked to be removed from funny@cs.toronto.edu months ago, and now you are just forwarding spam'
if (~ $dom *.sales.overstock.com *.Sales.Overstock.com)
	exit 'rejected: you are violating the RFCs by not sending from a replyable return address'

#
# the real machinery
#

# changed /dev/null to postmaster so that some of dmr's mail could get
# in.  He was getting mail from people who refuse mail from <>. - rsc
if (x=`{upas/smtp -p $netroot/tcp!$dom postmaster $addr >[2=1] |
    sed 's/^/'$sysname' '$pid' /' | tee -a /sys/log/smtpd.mx | tail -1})
	exit ''

smtpstatus=$status
if (~ $#x 0)
	x=$smtpstatus
if (~ $smtpstatus *'Permanent Failure'*)
	exit 'rejected: smtp ping: '^$"x
exit 'deferred: smtp ping: '^$"x