spam-split with bogofilter

spam-split with bogofilter

[Karl Eichwalder]

My next project is to make use of spam.el and bogofilter (version
0.10.3.1); in *Messages* I can already see the good will of involved
tools:

Reading active file via nnml...  
nnml: Reading incoming mail from file...  
spam-split: calling the spam-check-bogofilter function 
Wrote /home/ke/Mail/mail/rest/24782 
nnml: Reading incoming mail (1 new)...done 

Exiting summary buffer and applying spam rules
Registering spam with bogofilter
Marking spam as expired without moving it
Registering ham with Bogofilter
Expiring articles...done

And I can mark spam with M-d and when I leave a mail group with spam
marked mails, the mails will we move to the spam group.  Now I want
spam.el/bogofilter to jump in earlier -- how can I make them storing
incoming spam mail straight away in a special spam group?  I'm using
these settings:

(require 'spam)
(setq spam-use-bogofilter t)
(setq gnus-spam-process-newsgroups
      '(("nnml:mail\\..*" (gnus-group-spam-exit-processor-bogofilter
			   gnus-group-ham-exit-processor-bogofilter)))
      gnus-spam-process-destinations '(("nnml:mail\\..*" "nnml:spambox"))
      spam-junk-mailgroups '("nnml:spambox")
      spam-split-group "nnml:spambox")

(setq nnmail-split-fancy
      '(|
	(: spam-split)
	("reply-to" "dssslist" "mail.dssslist")
        ;; and many more rules
        ))

Must I call bogofilter via .procmailrc?

-- 
ke@suse.de (work) / keichwa@gmx.net (home):              |
http://www.gnu.franken.de/ke/                            |      ,__o
Free Translation Project:                                |    _-\_<,
http://www.iro.umontreal.ca/contrib/po/HTML/             |   (*)/'(*)

Re: spam-split with bogofilter

[Ted Zlatanov]

On Thu, 20 Mar 2003, keichwa@gmx.net wrote:
> My next project is to make use of spam.el and bogofilter (version
> 0.10.3.1); in *Messages* I can already see the good will of involved
> tools:
> 
> And I can mark spam with M-d and when I leave a mail group with spam
> marked mails, the mails will we move to the spam group.  Now I want
> spam.el/bogofilter to jump in earlier -- how can I make them storing
> incoming spam mail straight away in a special spam group?  I'm using
> these settings:
[...]
> Must I call bogofilter via .procmailrc?

No, spam-use-bogofilter calls bogofilter for you (although for local
mail, spam-use-bogofilter-headers is also OK if you pre-process with
Bogofilter in the .procmailrc).

Your rules are OK.  The spam should be going to "nnml:spambox".  Does
a message get misclassified? Go to the message's buffer and do t M-:
(spam-split) (note that the 't' command is to show the full message in
the buffer).  Does that return the value of spam-split-group?  Also
try saving the message in a file and run bogofilter on it, to see if
it thinks the message is spam.

Also, this is obvious, but make sure that spam-bogofilter-path is
correct.  It should be automatically set to (find-executable "bogofilter").

Finally, note that (unfortunately) ham will be registered with
Bogofilter multiple times, I'll fix that when I get the registry
working.

Let me know if you need more help
Ted

Re: spam-split with bogofilter

[Karl Eichwalder]

Ted Zlatanov <tzz@lifelogs.com> writes:

Thanks for your help!  This morning, with a sudden, it starts working.
Strange.  But now it's to overzealous ;)  All mail did end in the spam
box.

There is/was also an additional problem: "~/Mail/nnml:spambox" (Gnus
name: "nnml:nnml:spambox") was used instead of "~/Mail/spambox".

> Your rules are OK.  The spam should be going to "nnml:spambox".  Does
> a message get misclassified? Go to the message's buffer and do t M-:
> (spam-split) (note that the 't' command is to show the full message in
> the buffer).  Does that return the value of spam-split-group?  Also
> try saving the message in a file and run bogofilter on it, to see if
> it thinks the message is spam.

Yes, now it does; for example:

$ bogofilter -v </tmp/30495 
X-Bogosity: Yes, tests=bogofilter, spamicity=0.666327, version=0.10.3.1

But this was a good message -- obviously, bogofilter needs more
training...

I must reread the docs.

-- 
ke@suse.de (work) / keichwa@gmx.net (home):              |
http://www.gnu.franken.de/ke/                            |      ,__o
Free Translation Project:                                |    _-\_<,
http://www.iro.umontreal.ca/contrib/po/HTML/             |   (*)/'(*)

Re: spam-split with bogofilter

[Stefan Reichör]

On Fri, 21 Mar 2003, Karl Eichwalder stated:

>  Ted Zlatanov <tzz@lifelogs.com> writes:
>  
>  Thanks for your help! This morning, with a sudden, it starts
>  working. Strange. But now it's to overzealous ;) All mail did end
>  in the spam box.
>  
>  There is/was also an additional problem: "~/Mail/nnml:spambox"
>  (Gnus name: "nnml:nnml:spambox") was used instead of
>  "~/Mail/spambox".
>  
>  But this was a good message -- obviously, bogofilter needs more
>  training...
>  
>  I must reread the docs.

I had the same problem when I started using spam-stat.
I solved it the following way:
I entered the spam group
I displayed the whole ham article with: C-u g
I switched to the article buffer and typed: M-: (spam-stat-buffer-is-non-spam)
At last I respooled the article (B r)

Is there already an automated way for doing this?

Stefan.

Re: spam-split with bogofilter

[Karl Eichwalder]

Stefan Reichör <xsteve@riic.at> writes:

> I entered the spam group
> I displayed the whole ham article with: C-u g
> I switched to the article buffer and typed: M-: (spam-stat-buffer-is-non-spam)
> At last I respooled the article (B r)

Thanks for advise -- I tried the same but the article ends up in the
"spambox" again ;-(

> Is there already an automated way for doing this?

Theoretically it should be enough to unmark all false positives; leaving
the spam group then should respool the articles as ham -- unfortunately,
something prevent it from happening.

-- 
ke@suse.de (work) / keichwa@gmx.net (home):              |
http://www.gnu.franken.de/ke/                            |      ,__o
Free Translation Project:                                |    _-\_<,
http://www.iro.umontreal.ca/contrib/po/HTML/             |   (*)/'(*)

Re: spam-split with bogofilter

[Ted Zlatanov]

On Fri, 21 Mar 2003, keichwa@gmx.net wrote:
> Theoretically it should be enough to unmark all false positives;
> leaving the spam group then should respool the articles as ham --
> unfortunately, something prevent it from happening.

I think I see what's wrong:

(setq gnus-spam-process-newsgroups
      '(("nnml:mail\\..*" (gnus-group-spam-exit-processor-bogofilter
			   gnus-group-ham-exit-processor-bogofilter)))
      gnus-spam-process-destinations '(("nnml:mail\\..*" "nnml:spambox"))
      spam-junk-mailgroups '("nnml:spambox")
      spam-split-group "nnml:spambox")

"nnml:spambox" does not match the spam-exit-processor or the
ham-exit-processor you have selected.  What you have will process spam
and ham only when you're in a "nnml:mail.*" group.  Which is OK, since
you have spam-process-destination set to go to "nnml:spambox" so the
processed spam will go there.  It's just that in "nnml:spambox"
nothing will happen.

Ted

Re: spam-split with bogofilter

[Karl Eichwalder]

Ted Zlatanov <tzz@lifelogs.com> writes:

> (setq gnus-spam-process-newsgroups
>       '(("nnml:mail\\..*" (gnus-group-spam-exit-processor-bogofilter
> 			   gnus-group-ham-exit-processor-bogofilter)))
>       gnus-spam-process-destinations '(("nnml:mail\\..*" "nnml:spambox"))
>       spam-junk-mailgroups '("nnml:spambox")
>       spam-split-group "nnml:spambox")
>
> "nnml:spambox" does not match the spam-exit-processor or the
> ham-exit-processor you have selected.  What you have will process spam
> and ham only when you're in a "nnml:mail.*" group.  Which is OK, since
> you have spam-process-destination set to go to "nnml:spambox" so the
> processed spam will go there.  It's just that in "nnml:spambox"
> nothing will happen.

Is it okay to add spambox this way:

(setq gnus-spam-process-newsgroups
      '(("nnml:\\(mail\\..*\\|spambox\\)"
                          ;; ^^^^^^^^ add spambox here?
                          (gnus-group-spam-exit-processor-bogofilter
			   gnus-group-ham-exit-processor-bogofilter)))
      gnus-spam-process-destinations '(("nnml:mail\\..*" "nnml:spambox"))
      spam-junk-mailgroups '("spambox")  ;; no "nnml:" prefix here
      spam-split-group "spambox")        ;; likewise

Is it necessary to provide a gnus-ham-process-destinations setting,
too?  How?  I want false positives to go thru the usual split procedure.


-- 
ke@suse.de (work) / keichwa@gmx.net (home):              |
http://www.gnu.franken.de/ke/                            |      ,__o
Free Translation Project:                                |    _-\_<,
http://www.iro.umontreal.ca/contrib/po/HTML/             |   (*)/'(*)

Re: spam-split with bogofilter

[Ted Zlatanov]

On Fri, 21 Mar 2003, keichwa@gmx.net wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> (setq gnus-spam-process-newsgroups
>>       '(("nnml:mail\\..*"
>>       (gnus-group-spam-exit-processor-bogofilter
>> 			   gnus-group-ham-exit-processor-bogofilter)))
>>       gnus-spam-process-destinations '(("nnml:mail\\..*"
>>       "nnml:spambox")) spam-junk-mailgroups '("nnml:spambox")
>>       spam-split-group "nnml:spambox")
>>
>> "nnml:spambox" does not match the spam-exit-processor or the
>> ham-exit-processor you have selected.  What you have will process
>> spam and ham only when you're in a "nnml:mail.*" group.  Which is
>> OK, since you have spam-process-destination set to go to
>> "nnml:spambox" so the processed spam will go there.  It's just that
>> in "nnml:spambox" nothing will happen.
> 
> Is it okay to add spambox this way:
> 
> (setq gnus-spam-process-newsgroups
>       '(("nnml:\\(mail\\..*\\|spambox\\)"
>                           ;; ^^^^^^^^ add spambox here?
>                           (gnus-group-spam-exit-processor-bogofilter
> 			   gnus-group-ham-exit-processor-bogofilter)))

That works, you can also put a separate entry in the list variable -
customize it and you'll see what I mean.  I think that's cleaner
(IMHO) than compound regular expressions.

> Is it necessary to provide a gnus-ham-process-destinations setting,
> too?

It's necessary iff you want ham, when processed, to go somewhere else
when you are in a spam group.  Since you said you want that, it is
necessary to set the ham-process-destination parameter.

> How?

Do 'G c' on "spambox" to customize the group parameters, or customize
the variable gnus-ham-process-destinations.

> I want false positives to go thru the usual split procedure.

There's no "respooling" action yet, you can only set
ham-process-destination to a group name.  If you *really* want to
respool, I guess we can add a 'respool option to the
spam/ham-process-destination parameters, in addition to a group name.

Ted

Re: spam-split with bogofilter

[Karl Eichwalder]

Ted Zlatanov <tzz@lifelogs.com> writes:

> That works, you can also put a separate entry in the list variable -
> customize it and you'll see what I mean.  I think that's cleaner
> (IMHO) than compound regular expressions.

[...]

> It's necessary iff you want ham, when processed, to go somewhere else
> when you are in a spam group.  Since you said you want that, it is
> necessary to set the ham-process-destination parameter.

Now these are my settings:

(setq spam-use-bogofilter t)
(setq gnus-spam-process-newsgroups
      '(("nnml:mail\\..*"
	 (gnus-group-spam-exit-processor-bogofilter
	  gnus-group-ham-exit-processor-bogofilter))
	("nnml:spambox"
	 (gnus-group-spam-exit-processor-bogofilter
	  gnus-group-ham-exit-processor-bogofilter)))
      gnus-spam-process-destinations '(("nnml:mail\\..*" "nnml:spambox"))
      gnus-ham-process-destinations '(("nnml:spambox" "nnml:mail.rest"))
      spam-junk-mailgroups '("spambox")
      spam-split-group "spambox")

(setq gnus-spam-newsgroup-contents
      '(("nnml:spambox"  gnus-group-spam-classification-spam)
	("nnml:mail\\.spambox"  gnus-group-spam-classification-spam)
	("nnml:.*" gnus-group-spam-classification-ham)))

> Do 'G c' on "spambox" to customize the group parameters, or customize
> the variable gnus-ham-process-destinations.

Done.  But unmarking a message in "spambox" und quitting this group
is not enough to register the message as ham and move it back to the
mail.rest group.

Currently this workaround does the trick for me.

. Unmark message
. Widen the message (C-u g)
. Go to the article buffer (C-x o) and mark the message (C-x h)
. Call bogofilter on the message to register it as ham
  (M-| bogofilter -n RET)
. C-x o
. Respool the message (B r)

The more false positives are registered the less I'll have to do this
manual intervention ;-)  Thus far I'm quite happy with the spam
detection now.  Thanks for all your help!

(BTW, I don't use customize because it looks too confusing to me or --
for simple variables it's overkill.)

-- 
ke@suse.de (work) / keichwa@gmx.net (home):              |
http://www.gnu.franken.de/ke/                            |      ,__o
Free Translation Project:                                |    _-\_<,
http://www.iro.umontreal.ca/contrib/po/HTML/             |   (*)/'(*)

Re: spam-split with bogofilter

[Ted Zlatanov]

On Sat, 22 Mar 2003, keichwa@gmx.net wrote:
> Now these are my settings:
> 
> (setq spam-use-bogofilter t)
> (setq gnus-spam-process-newsgroups
>       '(("nnml:mail\\..*"
> 	 (gnus-group-spam-exit-processor-bogofilter
> 	  gnus-group-ham-exit-processor-bogofilter))
> 	("nnml:spambox"
> 	 (gnus-group-spam-exit-processor-bogofilter
> 	  gnus-group-ham-exit-processor-bogofilter)))
>       gnus-spam-process-destinations '(("nnml:mail\\..*"
>       "nnml:spambox")) gnus-ham-process-destinations
>       '(("nnml:spambox" "nnml:mail.rest")) spam-junk-mailgroups
>       '("spambox") spam-split-group "spambox")
> 
> (setq gnus-spam-newsgroup-contents
>       '(("nnml:spambox"  gnus-group-spam-classification-spam)
> 	("nnml:mail\\.spambox"  gnus-group-spam-classification-spam)
> 	("nnml:.*" gnus-group-spam-classification-ham)))
> 
> Done.  But unmarking a message in "spambox" und quitting this group
> is not enough to register the message as ham and move it back to the
> mail.rest group.

Can you trace through spam-summary-prepare-exit?  If you don't know
Elisp, try putting (debug "got here") statements in the body to see if
a certain point is reached; do C-x e of that function to load the new
definition.  You should be able to do that in a one window, and try to
exit a summary buffer in another.

Let me know if any of the above made sense :)

> Currently this workaround does the trick for me.
> 
> . Unmark message
> . Widen the message (C-u g)
> . Go to the article buffer (C-x o) and mark the message (C-x h)
> . Call bogofilter on the message to register it as ham
>   (M-| bogofilter -n RET)
> . C-x o
> . Respool the message (B r)

Yeah, except for respooling, this is exactly what the bogofilter exit
processor is supposed to do.

> (BTW, I don't use customize because it looks too confusing to me or
> -- for simple variables it's overkill.)

I don't mind it so much, and it ensures that simple and complex
variables alike are correctly set up.  But it's your choice, of course
:)

Ted

Re: spam-split with bogofilter

[Ted Zlatanov]

On Fri, 21 Mar 2003, xsteve@riic.at wrote:
> On Fri, 21 Mar 2003, Karl Eichwalder stated:
> 
>>  Ted Zlatanov <tzz@lifelogs.com> writes:
>>  
>>  Thanks for your help! This morning, with a sudden, it starts
>>  working. Strange. But now it's to overzealous ;) All mail did end
>>  in the spam box.
>>  
>>  There is/was also an additional problem: "~/Mail/nnml:spambox"
>>  (Gnus name: "nnml:nnml:spambox") was used instead of
>>  "~/Mail/spambox".
>>  
>>  But this was a good message -- obviously, bogofilter needs more
>>  training...
>>  
>>  I must reread the docs.
> 
> I had the same problem when I started using spam-stat.  I solved it
> the following way: I entered the spam group I displayed the whole
> ham article with: C-u g I switched to the article buffer and typed:
> M-: (spam-stat-buffer-is-non-spam) At last I respooled the article
> (B r)
> 
> Is there already an automated way for doing this?

It should be automatic when you quit a group summary with
gnus-group-ham-exit-processor-bogofilter enabled for that group.  That
processes the ham with bogofilter.  You can also set the
ham-process-destination parameter if you want the processed ham to go
to another group after registration.

Ted