Re: Bogofilter

[Ian Soboroff <org@acm.isoboroff>]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Ian Soboroff <org@acm.isoboroff> writes:
>
>> It's nice to see such a craze over naive-Bayes filtering techniques,
>> but they can get overtrained pretty easily.
>
> Yeah.  I don't know much about automatic classification, but I seem
> to recall that naive-Bayes is not the most effective method.
>
> So are there better algorithms around and is there an implementation
> that can be integrated into Gnus, similar to ifile?

There are boatloads of text classification algorithms.  Naive Bayes is
the canonical second best solution to any problem, and has the
advantage of being fast.  Support Vector Machines are better but NB
can get quite close in some data.  SVMs are hard to update, but to be
honest an email classifier could probably be just fine retraining
overnight.

My favorite classifier tool is Andrew McCallum's BOW toolkit.  It does
NB, SVM, kNN, EM, and probably three other things I forgot about, and
has nice support for doing measurements and experiments.  I was _this_
close to writing a scoring module for Gnus based on it, when I ran
across ifile.

The _right_ thing to do is something like nnir, that is, a classifier
framework that you can plug anything into underneath.  ifile-gnus.el
is probably most of what's needed (plus a couple more functions to
easily move mail without triggering a reclassification).

Ian