Training for ham and training for spam

Training for ham and training for spam

[Jake Colman]

Is it truly necessary to train for ham or can I just train for spam.  I have
a number of mailing lists that filter into their own folders and are mostly
spam-free.  Is there any benefit to classifying them as ham folders and
specifying an exit processor?  Or will it just add processing time?

On the other hand, I have a group that mostly gets spam so I have classified
it as a spam group and I do pass it through the exit processor.

Does this make sense?

-- 
Jake Colman                     

Principia Partners LLC                    Phone: (201) 209-2467
Harborside Financial Center                 Fax: (201) 946-0320
902 Plaza Two                          E-mail: colman@ppllc.com
Jersey City, NJ 07311                 www.principiapartners.com

Re: Training for ham and training for spam

[Ted Zlatanov]

On Wed, 29 Oct 2003, colman@ppllc.com wrote:

> Is it truly necessary to train for ham or can I just train for spam.
> I have a number of mailing lists that filter into their own folders
> and are mostly spam-free.  Is there any benefit to classifying them
> as ham folders and specifying an exit processor?  Or will it just
> add processing time?

Training for ham is useful, and pretty much every statistical filter
recommends it.  The decision of whether it's worth the CPU time and
memory is up to you and depends on the particular filter.

> On the other hand, I have a group that mostly gets spam so I have
> classified it as a spam group and I do pass it through the exit
> processor.
> 
> Does this make sense?

Sure.

Ted

Re: Training for ham and training for spam

[Michael Shields]

In message <76u15ru93j.fsf@newjersey.ppllc.com>,
Jake Colman <colman@ppllc.com> wrote:
> Is it truly necessary to train for ham or can I just train for spam.

You need to train for both; Bayesian filters work not only by
recognizing what features are correlated with spam, but also what
features indicate that mail is likely to be ham.
-- 
Shields.

Re: Training for ham and training for spam

[Ian Dobbie]

Michael Shields <shields@msrl.com> writes:

> In message <76u15ru93j.fsf@newjersey.ppllc.com>,
> Jake Colman <colman@ppllc.com> wrote:
>> Is it truly necessary to train for ham or can I just train for spam.
>
> You need to train for both; Bayesian filters work not only by
> recognizing what features are correlated with spam, but also what
> features indicate that mail is likely to be ham.

...And if you dont all your mail will end up in the spam group. I just
setup the spam and didnt quite get the ham config right so it wasnt
working. All my mail went into the spam folder, until I worked out
what I had done wrong.

I would recommend against having large volume mailing lists in the ham
filter all the time. Maybe train on a few hundred messages and then
dont bother. However I am using an old, slow machine so CPU is a major
factor.

Ian

Re: Training for ham and training for spam

[Michael Shields]

In message <m38yn4t90x.fsf@biostaff03.nuigalway.ie>,
Ian Dobbie <ian.dobbie@nuigalway.ie> wrote:
> I would recommend against having large volume mailing lists in the ham
> filter all the time. Maybe train on a few hundred messages and then
> dont bother. However I am using an old, slow machine so CPU is a major
> factor.

If learning is slow, then maybe you should do it asynchronously by
using the settings that copy ham and spam into folders for training,
and then use a cronjob to learn from those in the background.  I use:

    (setq gnus-spam-process-newsgroups
          '(("^INBOX" (gnus-group-ham-exit-processor-copy))))
    (setq gnus-spam-process-destinations
          '(("^INBOX" "INBOX.SA-spam")))
    (setq gnus-ham-process-destinations
          '(("^INBOX" "INBOX.SA-ham")))
    (setq spam-move-spam-nonspam-groups-only nil)

The major motivation for that feature was to have filtering done on a
different machine, the IMAP server.  But it also means that I don't
have to wait for either learning or filtering, since they happen in
the background from cron and procmail respectively.

Another idea would be to have a knob that allowed you to train on only
every n-th message; it would make processing n times faster, and since
the Bayesian filters are statistical they would still work ok.  You
would only set this knob after building up a database of a few hundred
messages.
-- 
Shields.