[9fans] spam filtering fs

[9fans] spam filtering fs

[steve.simon]

Hi,

I'am starting to think about a spam filtering again.

I plan to use Paul Grahams ideas plus the changes
sugested by Gary Robinson; Basicially a Naive Baiesian
classifier.

I think its best to implement it as a filesystem which overlays
upas/fs which is transparent to valid email but opaque to spam,
this meanst the token frequency database remains in RAM, not
having to be reloaded to test each new email.

In order for the filter to learn the user must be able to classify
the few spams that slip through, so I propose to wstat() emails
to zero length before deletion if they are spam and simply delete
them if they are valid.

In the longer term I have some ideas about fingerprinting attached
images, adding tokens for "is html email", "has MS screen-saver attached"
etc.

Anyone any opinions on this approach?

-Steve

Re: [9fans] spam filtering fs

[Fco.J.Ballesteros]

[-- Attachment #1: Type: text/plain, Size: 247 bytes --]

I'd prefer a plain filter that could be called form pipeto, and
regarding training, I'd prefer to be able to use a different program,
say, upas/spam -y <spammail, so I could run it from within my mail
reader. Since you asked for opinions... :-)

[-- Attachment #2: Type: message/rfc822, Size: 2395 bytes --]

From: steve.simon@snellwilcox.com
To: 9fans@cse.psu.edu
Subject: [9fans] spam filtering fs
Date: Mon, 1 Sep 2003 13:48:39 +0100
Message-ID: <b89f5f87b787fdcd291b8f80d2d87dd8@yourdomain.dom>

Hi,

I'am starting to think about a spam filtering again.

I plan to use Paul Grahams ideas plus the changes
sugested by Gary Robinson; Basicially a Naive Baiesian
classifier.

I think its best to implement it as a filesystem which overlays
upas/fs which is transparent to valid email but opaque to spam,
this meanst the token frequency database remains in RAM, not
having to be reloaded to test each new email.

In order for the filter to learn the user must be able to classify
the few spams that slip through, so I propose to wstat() emails
to zero length before deletion if they are spam and simply delete
them if they are valid.

In the longer term I have some ideas about fingerprinting attached
images, adding tokens for "is html email", "has MS screen-saver attached"
etc.

Anyone any opinions on this approach?

-Steve

Re: [9fans] spam filtering fs

[David Presotto]

rsc already wrote and is using a bayesian filter via pipeto.

Re: [9fans] spam filtering fs

[Dan Cross]

> rsc already wrote and is using a bayesian filter via pipeto.

Is it in the distribution?

	- Dan C.

Re: [9fans] spam filtering fs

[steve.simon]

The problem with the "upas/spam -y <spammail" approach is that the
word frequency tables need to be kept up to date for the best results,
and making the user type this in is unlikely to do this.

If you forget to add a new strain of spam or valid emails to the tables they
will not continue to learn and thus the threshold of spamminess used will
drift from the optimum - you will start to lose valid emails or get more spam.

An important feature of a user interface for baieien spam classifiers is
the equal ease with which the user can submit spam and valid emails to
the training code, otherwise the whole system becomes biased in favour
of least effort for the user :-)

-Steve

Re: [9fans] spam filtering fs

[Fco.J.Ballesteros]

[-- Attachment #1: Type: text/plain, Size: 250 bytes --]

You could assume that all mails that have not classified as spam, are not
spam. I could then simply define a `S' script and run it from acme to
let you know which ones are spam. But if rsc has one already done, we'd
better use his interface as is.

[-- Attachment #2: Type: message/rfc822, Size: 2293 bytes --]

From: steve.simon@snellwilcox.com
To: 9fans@cse.psu.edu
Subject: Re: [9fans] spam filtering fs
Date: Mon, 1 Sep 2003 15:36:56 +0100
Message-ID: <88fd25c4b29b0d3dcf8b2629d0259786@yourdomain.dom>

The problem with the "upas/spam -y <spammail" approach is that the
word frequency tables need to be kept up to date for the best results,
and making the user type this in is unlikely to do this.

If you forget to add a new strain of spam or valid emails to the tables they
will not continue to learn and thus the threshold of spamminess used will
drift from the optimum - you will start to lose valid emails or get more spam.

An important feature of a user interface for baieien spam classifiers is
the equal ease with which the user can submit spam and valid emails to
the training code, otherwise the whole system becomes biased in favour
of least effort for the user :-)

-Steve

Re: [9fans] spam filtering fs

[boyd, rounin]

> You could assume that all mails that have not classified as spam, are not
> spam. I could then simply define a `S' script and run it from acme to
> let you know which ones are spam.

i used to do something like that on lunix:

    http://www.insultant.net/repo/dws.html

i _cannot_ use acme.  it's just not for me.