Re: [9fans] /n/sources/patch/spamhaus

Re: [9fans] /n/sources/patch/spamhaus

[erik quanstrom]

> I don't have a false positive mailbox to skim.
> I run Mail Avenger, which lets me run shell scripts
[...]
> I run Spam Assassin.  If SA thinks the mail is spam,
> SMTP rejects it rather than saving it or deciding to
> reject it later and having to send a bounce.  That

for me, there's one problem with this alternative to
the "stupid" spamhaus solution — it requires i run
a linux server.

i never claimed that spamhaus is a panecea.  but it
does solve a large portion of the problem for me
in a tiny shell script that runs on plan 9.

whitelists can handle some unfortunate conflicts. and
at this point, i think managing a spamhaus exception
list is going to be easier than managing content
based filtering.

- erik

Re: [9fans] /n/sources/patch/spamhaus

[Russ Cox]

> for me, there's one problem with this alternative to
> the "stupid" spamhaus solution — it requires i run
> a linux server.

It does not.

There are content-based filters for Plan 9.
Or I bet you could port many of the standard
Unix ones over without much trouble.

Mail Avenger is a little harder to replace, but
still not very difficult.  I said at the outset that
you'd need to make a few changes to upas
(specifically, the introduction of a validatemessage
script).

Finally, the mail server for swtch.com does run
the software I mentioned, but it does not run Linux.

> i never claimed that spamhaus is a panecea.  but it
> does solve a large portion of the problem for me
> in a tiny shell script that runs on plan 9.
>
> whitelists can handle some unfortunate conflicts. and
> at this point, i think managing a spamhaus exception
> list is going to be easier than managing content
> based filtering.

Good luck.

The vigilante lists are too easy to end up on incorrectly
and too hard to get removed from, as Forsyth noted
already.  I've been on the wrong end of them too
many times.  If someone is using an RBL or a Barracuda
box that thinks I'm a spammer, I don't spend the
effort anymore to get my mail through.  I just assume
the person I'm emailing doesn't want my mail.
(I do the same for people running the "please reply
to this message with a magic token" scripts.)

Russ

Re: [9fans] /n/sources/patch/spamhaus

[Nate S]

On Tue, May 13, 2008 at 4:07 PM, erik quanstrom <quanstro@quanstro.net> wrote:
>
>  at work we have a barracuda box which seems to
>  be completely content based.  it's false positive
>  rate is significant.  so you actually need to skim
>  up to a hundred questionable messages per week.
>

more trouble than it's worth, blech.

Re: [9fans] /n/sources/patch/spamhaus

[Russ Cox]

My mail server does not run Plan 9, so my own
setup would require some implementation work,
as I mentioned before.

>  how do you maintain content-based filtering without
>  spending time on it on a regular basis?

I typically see one or two spams a day that make
it through, and I save those into a mailbox named "spam"
that a cron job uses to retrain the filter (Spam Assassin).

I also see a lot of spam going to bogus addresses
at swtch.com (they are valid at swtch.com.au),
and I feed those in as spam samples.  I don't know
how much that actually helps.

>  at work we have a barracuda box which seems to
>  be completely content based.  it's false positive
>  rate is significant.  so you actually need to skim
>  up to a hundred questionable messages per week.
>
>  i find that skimming through lists like this is very
>  error prone.

I don't have a false positive mailbox to skim.
I run Mail Avenger, which lets me run shell scripts
during the SMTP session to decide whether to let
it continue.  (It was the inspiration for validateaddress
and validatesender.)  In addition to checking the
sender and the recipient, I can run a program over
the body before accepting the mail, so that's where
I run Spam Assassin.  If SA thinks the mail is spam,
SMTP rejects it rather than saving it or deciding to
reject it later and having to send a bounce.  That
means my mail server doesn't contribute to someone
else's backscatter problems, and if someone does
send something that looks like spam, they get
immediate feedback about it not going through, rather
than hoping I will see it in a spam box.  The reject
response in SMTP explains that the mail looks like
spam and gives a magic word to put in the subject
if it is not spam.

I've been using this setup for a few years now.
I have had exactly one real message that was falsely
rejected that I remember, and it was a big chain
forward that arguably was spam, although it was
a real person sending it.  I have also had two
legitimate commercial emails (receipts) rejected
as spam, but I knew they were on the way so I looked
for them.  (I do spool the rejected messages to a file,
both for my own peace of mind and to handle cases
like these.)

It's not perfect, but it is far better than having to watch
a spam folder.

Russ

Re: [9fans] /n/sources/patch/spamhaus

[erik quanstrom]

>>  your solution for backscatter is a good one.  but
>>  how does it do against non backscatter?  this
>>  is also a significant problem.  generally >100
>>  messages per day for me.
>
> content-based filtering works fine for me.

how do you maintain content-based filtering without
spending time on it on a regular basis?

at work we have a barracuda box which seems to
be completely content based.  it's false positive
rate is significant.  so you actually need to skim
up to a hundred questionable messages per week.

i find that skimming through lists like this is very
error prone.

- erik

Re: [9fans] /n/sources/patch/spamhaus

[Russ Cox]

>  your solution for backscatter is a good one.  but
>  how does it do against non backscatter?  this
>  is also a significant problem.  generally >100
>  messages per day for me.

content-based filtering works fine for me.

>  am i an idiot for objecting to this?

i never said you were an idiot.
i said that RBLs are idiocy, and they are.

russ

Re: [9fans] /n/sources/patch/spamhaus

[erik quanstrom]

> I use Greylisting [1], and it's been really effective. No false
> positives (so far), and 0 to 2 spam messages a day. All this for a
> mild ~15 minute delay on genuine emails (but only for the first time).

sites like plan9.bell-labs.com tend not resend email with prec.  bulk
even when given a 45x error.

smtpd 451'd a couple messages within the last week due to dns errors.
the sender did not retry.   i assume that they would have not been resent
regardless of the text of the 451 message.

i did run greylisting for several days.  i found it cut down on spam
only about 20%.  those bots are getting smarter.

- erik

Re: [9fans] /n/sources/patch/spamhaus

[Anant Narayanan]

On 13-May-08, at 4:17 AM, erik quanstrom wrote:
> what's a better idea.  having an extra 6400 spam emails
> is the problem.  how to i solve this without using spamhaus?

I use Greylisting [1], and it's been really effective. No false
positives (so far), and 0 to 2 spam messages a day. All this for a
mild ~15 minute delay on genuine emails (but only for the first time).

--
Anant

[1] http://en.wikipedia.org/wiki/Greylisting

Re: [9fans] /n/sources/patch/spamhaus

[erik quanstrom]

> So you're blocking mail from forsyth in order
> to block spam bounces from <>?
>
> I already told you how I solved this when it
> happened to me, and it has been 100% effective

your solution for backscatter is a good one.  but
how does it do against non backscatter?  this
is also a significant problem.  generally >100
messages per day for me.

am i an idiot for objecting to this?

- erik

Re: [9fans] /n/sources/patch/spamhaus

[Russ Cox]

> the problem is that spf only validates that the sender is an
> allowed sender.  this is ineffective against backscatter
> attacks.  i've gotten as many as 500 backscatter spam in 4 hrs.
> so this is a significant issue for me.

So you're blocking mail from forsyth in order
to block spam bounces from <>?

I already told you how I solved this when it
happened to me, and it has been 100% effective
without the false positives you get from idiocy like RBLs.
I've arranged that all mail I send has an SMTP
return address of rsc+zzz@swtch.com, for some
value of zzz (right now zzz=bounces), and then
I reject mail from <> to plain rsc@swtch.com
with a comment explaining the backscatter issue.
It's 99% of the benefit of SRS with 1% of the work.

You would have to change smtpd to pass the sender
as $2 to validateaddress to implement this on Plan 9,
but it is not hard.

Russ

Re: [9fans] /n/sources/patch/spamhaus

[Wes Kussmaul]

Charles Forsyth wrote:


> this is all reminiscent of the nonsense of RFC1413


I think that people are finally ready to accept the fact that packets on the outdoor highway do not disclose the
intentions of their senders and that they contain no meaningful information about the identity of their senders.

The solution to the problem that is the source of spam and malware will be presented next Thursday at about 5:00 gmt+1:

http://www.itu.int/osg/csd/cybersecurity/WSIS/agenda-3_new.html

You can see a preview here (wip, requires flash):

http://quietenjoyment.net/slides/

Re: [9fans] /n/sources/patch/spamhaus

[a]

// rfc 2317 allows arbitrary cidrs to be delegated.  so far,
// i've always been able to get reverse mappings set up
// for static addresses.

I think you've been lucky, or have been dealing with better ISPs.
Apart from my home ADSL line, I share a commercial SDSL with
some folks. We've got a /123 or /124 (I forget right now) which
they won't delegate. Which seems reasonable, from a network
management point of view, I guess. When we asked, they gave
us an email address to mail updates to; the human on the other
end was always responsive and the updates got in place quickly.
Then one day the email address stopped working, and further
inquiries returned the same email address (including in mail
where we're complaining that it didn't work). Reality has
slowly diverged from our published reverse mappings.

Thankfully (?), I've seldom found this to be a problem for mail,
in practical terms. I use one of these mismatched hosts as my
mail server (after my ISP's went flaky again), and get less than a
dozen rejects a year (although it's an admitedly low-traffic site).
Anthony

Re: [9fans] /n/sources/patch/spamhaus

[erik quanstrom]

>> The solution for people on dynamic addresses (typically with some
>> generic and non-matching PTR record, though I haven't checked yours) is
>> likely to relay out through your ISP's mail server.
>
> because of the way the DNS is put together, PTR records cannot be relied upon.
> ownership of the DNS entries for a name are unrelated to the DNS entries for the in-addr
> entries for a set of IP addresses to which they map.

while true, this doesn't change many large site's email practices.
many do check reverse mappings.  (i don't recall particular sites.)
barracuda boxes check reverse ip mappings.

rfc 2317 allows arbitrary cidrs to be delegated.  so far, i've always
been able to get reverse mappings set up for static addresses.

- erik

Re: [9fans] /n/sources/patch/spamhaus

[Charles Forsyth]

> The solution for people on dynamic addresses (typically with some
> generic and non-matching PTR record, though I haven't checked yours) is
> likely to relay out through your ISP's mail server.

because of the way the DNS is put together, PTR records cannot be relied upon.
ownership of the DNS entries for a name are unrelated to the DNS entries for the in-addr
entries for a set of IP addresses to which they map.

this is all reminiscent of the nonsense of RFC1413

Re: [9fans] /n/sources/patch/spamhaus

[Jason Gurtz]

The botnets have ruined the sandbox forever.

On 5/12/2008 18:34, Charles Forsyth wrote:
> well, i'm now on the list for the simple reason that i got a different
> cable modem, which prompted a new IP address.

The solution for people on dynamic addresses (typically with some
generic and non-matching PTR record, though I haven't checked yours) is
likely to relay out through your ISP's mail server.

~JasonG, who even while possessing a static IP, suffers from having it
in the middle of a "dynamic range", also has a non-matching PTR, and
yes, does experience deliverability issues from time to time.

--

Re: [9fans] /n/sources/patch/spamhaus

[erik quanstrom]

i agree that spamhaus is a big hammer.  i'm open to suggestions.
preferably ones that do not require daily maintence.

> Things like SPF don't catch as much spam (yet; it'll improve as the
> acceptance improves), but have a very attractive false hit rate.

the rate just isn't good enough for me.  here's a tabulation of date,
total rejections (other than invalid helo -- that would add another
zero to the first column) and spf rejections over the past 4 days for me:

May  9	   335	0
May 10	   237	2
May 11	   137	0
May 12	   140	2

unfortunately, 2 of the spf rejections were actually 451's due to dns timeout.

(one thing i forgot to mention before is that spf does not protect against
bots sending through legit (according to spf) mail servers.)

spf can be a very good tool.  around apr 15th, spf was pretty good at
bouncing email proporting to be from irs.gov that was not caught by
spamhaus.

> This is what I'm doing now, since many of these folks assume that
> everyone on the end of a DSL or cable line are spammers, and many
> provide no way for me to tell them I'm not. It sucks; my ISPs mail
> server is okay, but certainly not 100% reliable, and adds another
> hop I'd rather not worry about.

i've successfully gotten static cidrs off spamhaus' black lists.
i believe they have a link right off their home page.

- erik

Re: [9fans] /n/sources/patch/spamhaus

[a]

// Althrought I'd like it to be different, blacklists are quite effective
// blocking spam. It's the best solution as long as we continue using SMTP.

This entirely depends how you prioritize things. If "best" and "effective"
are measured on what percentage of spam emails get blocked, yes,
services like spamhaus can be very effective, possibly the most effective
(short of drastic things like turning off smtp).

The problem in the real world is that "best" and "effective" also have to
incorporate a measure of legitimate emails blocked; in those metrics,
spamhaus does fairly poorly. It's the same problem with all the net's
vigilante groups: as Charles said, there's no good way to contest or
correct the data (nor, in many cases, to find out what got you listed).

Things like SPF don't catch as much spam (yet; it'll improve as the
acceptance improves), but have a very attractive false hit rate.

// In the end I ended up using my ISP's SMTP server as 'smarthost'
// to send mail.

This is what I'm doing now, since many of these folks assume that
everyone on the end of a DSL or cable line are spammers, and many
provide no way for me to tell them I'm not. It sucks; my ISPs mail
server is okay, but certainly not 100% reliable, and adds another
hop I'd rather not worry about.

Anthony

Re: [9fans] /n/sources/patch/spamhaus

[erik quanstrom]

> please don't, or at least check spf before spamhaus.
> the quality of their data is at best questionable,
> and there is no (usable) way to correct it.

the problem is that spf only validates that the sender is an
allowed sender.  this is ineffective against backscatter
attacks.  i've gotten as many as 500 backscatter spam in 4 hrs.
so this is a significant issue for me.

- erik

Re: [9fans] /n/sources/patch/spamhaus

[Armando Camarero]

Can you send mail to, for example, gmail.com addresses?

I found myself in the same situation months ago (I host my email at
home): my mail was bloked by almost every server.

Althrought I'd like it to be different, blacklists are quite effective
blocking spam. It's the best solution as long as we continue using SMTP.

In the end I ended up using my ISP's SMTP server as 'smarthost' to send
mail.

Charles Forsyth escribi�:
> as i was saying ...
>
> Your request ``mail net!quanstro.net quanstro '' failed (code smtp 2838130: Permanent Failure).
> The symptom was:
>
> Mon May 12 21:57:03 BST 2008 connect to net!quanstro.net:
> 554 5.7.1 rejected: spamhaus: sh policy
>
> ===> 2/ (message/rfc822) [inline]
> To: quanstro@quanstro.net
> Subject: re: [9fans] /n/sources/patch/spamhaus
> From: Charles Forsyth <forsyth@terzarima.net>
> Date: Mon, 12 May 2008 21:56:46 +0100
>
>
>> what leads you to say spamhaus's data is questionable?
>>
>
> well, i'm now on the list for the simple reason that i got a different
> cable modem, which prompted a new IP address.
>
>
>

Re: [9fans] /n/sources/patch/spamhaus

[erik quanstrom]

what's a better idea.  having an extra 6400 spam emails
is the problem.  how to i solve this without using spamhaus?

- erik

On Mon May 12 18:32:04 EDT 2008, forsyth@terzarima.net wrote:
> as i was saying ...
>
> Your request ``mail net!quanstro.net quanstro '' failed (code smtp 2838130: Permanent Failure).
> The symptom was:
>
> Mon May 12 21:57:03 BST 2008 connect to net!quanstro.net:
> 554 5.7.1 rejected: spamhaus: sh policy
>
> ===> 2/ (message/rfc822) [inline]
> To: quanstro@quanstro.net
> Subject: re: [9fans] /n/sources/patch/spamhaus
> From: Charles Forsyth <forsyth@terzarima.net>
> Date: Mon, 12 May 2008 21:56:46 +0100
>
> > what leads you to say spamhaus's data is questionable?
>
> well, i'm now on the list for the simple reason that i got a different
> cable modem, which prompted a new IP address.

Re: [9fans] /n/sources/patch/spamhaus

[Charles Forsyth]

as i was saying ...

Your request ``mail net!quanstro.net quanstro '' failed (code smtp 2838130: Permanent Failure).
The symptom was:

Mon May 12 21:57:03 BST 2008 connect to net!quanstro.net:
554 5.7.1 rejected: spamhaus: sh policy

===> 2/ (message/rfc822) [inline]
To: quanstro@quanstro.net
Subject: re: [9fans] /n/sources/patch/spamhaus
From: Charles Forsyth <forsyth@terzarima.net>
Date: Mon, 12 May 2008 21:56:46 +0100

> what leads you to say spamhaus's data is questionable?

well, i'm now on the list for the simple reason that i got a different
cable modem, which prompted a new IP address.

Re: [9fans] /n/sources/patch/spamhaus

[Charles Forsyth]

> the quality of their data is at best questionable,

as is their rationale

[9fans] /n/sources/patch/spamhaus

[Charles Forsyth]

please don't, or at least check spf before spamhaus.
the quality of their data is at best questionable,
and there is no (usable) way to correct it.