spam filtering by bayesian classifier and imap - ideas

spam filtering by bayesian classifier and imap - ideas

[Andreas Fuchs]

Hi,

I am about to move my mails from an nnml spool to an imap-based
setup, and, naturally, I wonder how to keep spam filtering as speedy
and useful as it currently is.

My plan (which requires a shell account and a spam classifier on the
imap server) is this one:

1. Have a spam filter which runs on the server pre-classify mails. Spam
   mails should get a X-Spam: yes header or similar.

2. Use gnus and nnimap-split-fancy to split mail into the correct
   groups.

3. When exiting a group, do this:
   1| when in a spam group, move all non-spam-marked articles to a group
      `INBOX.ham.reclassify'.
   1| when not in a spam group, move all spam-marked articles to a group
      `INBOX.spam.reclassify'.
   2 Start, via ssh (or let a cron job run), an imap client on the
     server which reclassifies the articles and moves spam to the spam
     inbox and ham to the regular inbox, so that it can be re-split.

Does that make sense? Would this be usable on a multi-user imap server?
Comments? Who would be interesed in this?

-- 
Andreas Fuchs, <asf@acm.org>, asf@jabber.at, antifuchs

Re: spam filtering by bayesian classifier and imap - ideas

[Kai Großjohann]

Andreas Fuchs <asf@void.at> writes:

> Does that make sense? Would this be usable on a multi-user imap server?
> Comments? Who would be interesed in this?

Sounds like a plan.
-- 
Ambibibentists unite!

Re: spam filtering by bayesian classifier and imap - ideas

[Xavier MAILLARD]

[-- Attachment #1: Type: text/plain, Size: 1679 bytes --]

On Sun, 19 Jan 2003, Andreas Fuchs said:

>  Hi,

Hi,

>  I am about to move my mails from an nnml spool to an imap-based

Same here :) I've just switched to use Cyrus Imapd here instead of
qpopper.

>  setup, and, naturally, I wonder how to keep spam filtering as speedy
>  and useful as it currently is.

Sounds interesting.

>  My plan (which requires a shell account and a spam classifier on the
>  imap server) is this one:
>
>  1. Have a spam filter which runs on the server pre-classify
>     mails. Spam mails should get a X-Spam: yes header or similar.

Same here. I am doing this using a weird scheme composed with
bogofilter, spamassassin/razor and UCE native rules of my MTA.

>  2. Use gnus and nnimap-split-fancy to split mail into the correct
>     groups.

Yup

>  3. When exiting a group, do this:
>     1| when in a spam group, move all non-spam-marked articles to a
>        group `INBOX.ham.reclassify'.  1| when not in a spam group,
>        move all spam-marked articles to a group
>        `INBOX.spam.reclassify'.  2 Start, via ssh (or let a cron job
>        run), an imap client on the server which reclassifies the
>        articles and moves spam to the spam inbox and ham to the
>        regular inbox, so that it can be re-split.

Sounds good to me.

>  Does that make sense? Would this be usable on a multi-user imap
>  server?  Comments? Who would be interesed in this?

I like this approach and I am interested in implementing such a thing
with a config based on yours :)

Please tell me when you got something functionnal. I will on my side
try to do something closed to you.

Cheers,

zeDek

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]

Re: spam filtering by bayesian classifier and imap - ideas

[Mats Lidell]

>>>>> asf wrote:

asf> 2. Use gnus and nnimap-split-fancy to split mail into the correct
asf>    groups.

I use procmail so that the splitting also is done on the server. YMMW
but it is nice and fast to have the splitting in just one place. No
need to update the splitting rules on different locations and makes it
possible to on occasions use other MUAs without to many consequences.

asf> 3. When exiting a group, do this:
asf>    1| when in a spam group, move all non-spam-marked articles to a group
asf>       `INBOX.ham.reclassify'.

I have started to use this scheme by setting the
"spam-process-destination" to a reclassify-as-ham folder. This might
seem strange but I have found that when the server already have moved
all potential spam to a spam folder the best way is to fool spam.el
that the marked articles are spam and should be moved away. With this
setup I just mark the ham-articles with M-d and they are moved out on
exit to the reclassify-as-ham folder.

This is however not ideal. It would be more natural to move the
articles from the spam folder to the right folder, where they belong,
at once. On the other hand it is hard to see how to support this for
any MUA without just relying on a simple move to a specific
folder. With gnus it is probably very easy to add that when you move
an article from the spam folder it is also copied to the
reclassify-as-ham folder.

asf> Does that make sense? Would this be usable on a multi-user imap
asf> server?  Comments? Who would be interesed in this?

I have left to design a script, executed by crontab or ssh or maybe by
procmail, so that the reclassified messages are reclassified and after
that reinserted back into the users mail handling (INBOX!?). The goal
I have is to make it possible to use statistic spam filtering with any
MUA (using imap) in our environment.

Yours
-- 
%% Mats

Re: spam filtering by bayesian classifier and imap - ideas

[Xavier MAILLARD]

[-- Attachment #1: Type: text/plain, Size: 535 bytes --]

On Mon, 20 Jan 2003, Mats Lidell stated:

>>>>> asf wrote:
>  
>  asf> 2. Use gnus and nnimap-split-fancy to split mail into the
>  asf>    correct groups.
>  
>  I use procmail so that the splitting also is done on the server. YMMW
>  but it is nice and fast to have the splitting in just one place. No
>  need to update the splitting rules on different locations and makes
>  it possible to on occasions use other MUAs without to many
>  consequences.

IMHO there is absolutely no need to use another MUA :)
  
zeDek

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]