[9fans] need help with improving my spam filtering

[9fans] need help with improving my spam filtering

[Robert Raschke]

Hi,

I have set up my Plan 9 box to receive email for myself.  It's a very
simple setup with no relaying, just for my own little maildrop.

This year the spam started to arrive in slightly more annoying
quantities, and a few months ago I managed to set up the bayes
filtering for my email.  This worked for a while, but now I am getting
more and more spam getting through (mostly because of the anti-bayes
mechanism of hiding the spam in a picture and sending it together with
random but apparently OK text).

So, I'd like to expand my spam stopping capabilities.  I have read
smtpd(6), ratfs(4), and scanmail(8) but find myself slightly at sea on
how to combine these to allow me to filter out some of the more
obvious spam.

If I understand correctly, I can start ratfs(4) and since it'll read
the default /mail/lib/blocked file my smtpd(6) will then block some
emails.  I can also replace qer(8) with scanmail(8) in my
/mail/lib/qmail and therefore block out even more emails.

The thing I don't yet fully grasp is how I manage the resulting
holding queues and log files and how I properly administer the
/mail/lib/blocked and /mail/lib/patterns files.

Can anyone point me in the right direction for learning more on how to
use ratfs(4) and scanmail(8) for effective spam filtering?

Robby

PS I guess an alternative approach would be for me to use fresh email
   addresses for mailing lists every so often.  But that feels
   shortsighted somehow.

Re: [9fans] need help with improving my spam filtering

[andrey mirtchovski]

i'll use your email to raise another question on a related note: it
would help slightly if the 9fans.net archives did not include full
sender email in their message display.

Re: [9fans] need help with improving my spam filtering

[Russ Cox]

> i'll use your email to raise another question on a related note: it
> would help slightly if the 9fans.net archives did not include full
> sender email in their message display.

Done.
Russ

Re: Re: [9fans] need help with improving my spam filtering

[andrey mirtchovski]

thank you!

On 10/15/06, Russ Cox <rsc@swtch.com> wrote:
> > i'll use your email to raise another question on a related note: it
> > would help slightly if the 9fans.net archives did not include full
> > sender email in their message display.
>
> Done.
> Russ
>

Re: [9fans] need help with improving my spam filtering

[Heiko Dudzus]

> This worked for a while, but now I am getting
> more and more spam getting through (mostly because of the anti-bayes
> mechanism of hiding the spam in a picture and sending it together with
> random but apparently OK text).

Same here.  I made a hold rule for this in /mail/lib/patterns, allowing
only some people and lists to send GIFs. Works for the moment but
could be too restrictive.

> So, I'd like to expand my spam stopping capabilities.  I have read
> smtpd(6), ratfs(4), and scanmail(8) but find myself slightly at sea on
> how to combine these to allow me to filter out some of the more
> obvious spam.
> If I understand correctly, I can start ratfs(4) and since it'll read
> the default /mail/lib/blocked file my smtpd(6) will then block some
> emails.  

Based on the connecting IP address and originating account, yes.

> I can also replace qer(8) with scanmail(8) in my
> /mail/lib/qmail and therefore block out even more emails.

Based on the content of the mail.  Because you said, it's just for
your own little maildrop, be aware that incoming mail can only be
filtered with scanmail(8) when it gets resent (and queued) to your own
system with the little trick discussed in this thread:

http://9fans.net/archive/2002/03/257

> PS I guess an alternative approach would be for me to use fresh email
>    addresses for mailing lists every so often.  But that feels
>    shortsighted somehow.

Greylisting is another option.  (When I saw in the source, that
greylist.c can deal with entire whitelisted subnets, it was an option
for me again)

Heiko

Re: [9fans] need help with improving my spam filtering

[Steve Simon]

I have had quite a bit of success with greylisting, however in the
last 3 or 4 months it has become less effective.

The vast majority of soam I get these days is from spam virus infected
machines on DSL lines in the USA (sorry but it true).

I was coinsidering adding features to smtpd and ratfs to allow regexes
so I could add rules for DSL lines. Most ISPs seem to use a fixed format for
the reverse IP addresses for their DSL accounts - though each ISP has its
own unique format.

There is also some work being done on an SPF validator at present but I
will let them announce it as they see fit.

-Steve

PS: I have resurected Boyds script dws (Die Worthless Spammer) to send a
complaint thought most ISPs seem to ignore them, but I feel I should try.

Re: [9fans] need help with improving my spam filtering

[Lyndon Nerenberg]

> The vast majority of soam I get these days is from spam virus infected
> machines on DSL lines in the USA (sorry but it true).

Why apologize for their crud???

> I was considering adding features to smtpd and ratfs to allow regexes
> so I could add rules for DSL lines. Most ISPs seem to use a fixed format for
> the reverse IP addresses for their DSL accounts - though each ISP has its
> own unique format.

I really don't think this will work.  The physical source of spam is way 
too mobile for anyone (or thing) to track.  But the content is still 
amenable to being smacked by bayes-like tools.  Although the bastards are 
getting better -- the last couple of months have seen a wee bit more crud 
get past the filters.  But we still win overall.

Anyway ... simple ingress filtering doesn't work.  Its better to 
concentrate on writing better algorithms that will smack this crud down 
while it still lives in the swamps.

--lyndon

Re: [9fans] need help with improving my spam filtering

[Steve Simon]

> > machines on DSL lines in the USA (sorry but it true).
> 
> Why apologize for their crud???

I was appologising for my apparent generalisation that the USA is responsible
for the spam I receive ☺.

> I really don't think this will work.  The physical source of spam is way 
> too mobile for anyone (or thing) to track.  

I guess it depends upon who is spamming you and how they get your address.
My address is exposed through my whois records and via my posing to this
and a few other newsgroup.

I have found that the vast majority of the spam I get is from dsl lines
and I believe this is from spam relay virusus. I DO thing being able to
block reverse IP addresses by regex will help, in my case. 

Perhaps the source of your spam is different to mine?

I guess I should just write the code and measure the results.

-Steve

Re: [9fans] need help with improving my spam filtering

[Heiko Dudzus]

Robert Raschke wrote:
> So, I'd like to expand my spam stopping capabilities.  
[..]
> If I understand correctly, I can start ratfs(4) and since it'll read
> the default /mail/lib/blocked file my smtpd(6) will then block some
> emails.

BTW: The smtpd binary on sources dates from Nov 20 2005.  There were
changes to the source in the meantime, causing smtpd to hang up on
some more bogus helo strings than before.  If you didn't already
rebuild smtpd, you can reduce the amount of delivered spam that way.

HTH a little bit, Heiko