How do you make spam automatically move to spam group on exit?

How do you make spam automatically move to spam group on exit?

[Emerick Rogul]

I've configured spam.el to use bogofilter for its spam processing.  To
keep things simple, let's say I have two groups, nnml:inbox and
nnml:spam.

On nnml:inbox, I have the following set via group parameters (BTW, am
I setting these correctly?):

((spam-contents 'gnus-group-spam-classification-ham)
 (spam-process '(gnus-group-ham-exit-processor-bogofilter
                 gnus-group-spam-exit-processor-bogofilter)))

On nnml:spam, I have the following set via group parameters:

((spam-contents 'gnus-group-spam-classification-spam)
 (spam-process 'gnus-group-spam-exit-processor-bogofilter))

When I accidentally receive a piece of spam in nnml:inbox, I want to
mark it as such and have it automatically moved to nnml:spam on group
exit (which I believe is how spam.el is supposed to function, no?)
However, after exiting nnml:inbox, my spam is entered into
bogofilter's database and then expired, but not moved to nnml:spam
(the *Messages* buffer says "Marking spam as expired without moving
it").

What do I need to do in order to move the spam into nnml:spam
automagically?  I think I must be missing something, but I'm not sure
what.

-Emerick
-- 
-------------------------------------------------------------------------
Emerick Rogul             /\/     "how young are you, how old am i?
emerick@cs.bu.edu         /\/      let's count the rings around my eyes."
------------------------------------------------- 'i will dare', the mats

Re: How do you make spam automatically move to spam group on exit?

[Ted Zlatanov]

On Fri, 07 Mar 2003, emerick@cs.bu.edu wrote:
> I've configured spam.el to use bogofilter for its spam processing.
> To keep things simple, let's say I have two groups, nnml:inbox and
> nnml:spam.
> 
> On nnml:inbox, I have the following set via group parameters (BTW,
> am I setting these correctly?):
> 
> ((spam-contents 'gnus-group-spam-classification-ham)
>  (spam-process '(gnus-group-ham-exit-processor-bogofilter
>                  gnus-group-spam-exit-processor-bogofilter)))

They should look reasonable with "G c" - I believe what you have is
correct.  Note that your ham will be processed repeatedly on every
exit, because spam.el does not currently remember what articles have
been processed as ham/spam.  The spam won't be processed repeatedly,
because it will be expired and moved when the group is exited.  I'm
working on this with the registry, but for now the stats will be
somewhat inaccurately biased.

> On nnml:spam, I have the following set via group parameters:
> 
> ((spam-contents 'gnus-group-spam-classification-spam)
>  (spam-process 'gnus-group-spam-exit-processor-bogofilter))
> 
> When I accidentally receive a piece of spam in nnml:inbox, I want to
> mark it as such and have it automatically moved to nnml:spam on
> group exit (which I believe is how spam.el is supposed to function,
> no?)  

Yes.  The spam is already marked as expired in nnml:inbox, so when
it's moved to nnml:spam, it won't be processed again.

> However, after exiting nnml:inbox, my spam is entered into
> bogofilter's database and then expired, but not moved to nnml:spam
> (the *Messages* buffer says "Marking spam as expired without moving
> it").

> What do I need to do in order to move the spam into nnml:spam
> automagically?  I think I must be missing something, but I'm not
> sure what.

Set the spam-process-destination parameter of "nnml:inbox" to
"nnml:spam" (you need the server prefix, IIRC).  That should do it.

Ted

Re: How do you make spam automatically move to spam group on exit?

[Emerick Rogul]

Ted Zlatanov writes:

> Set the spam-process-destination parameter of "nnml:inbox" to
> "nnml:spam" (you need the server prefix, IIRC).  That should do it.

That fixed it, thanks!

The only remaining problem I have is that my ham doesn't appear to be
getting processed by bogofilter (spam is being added to spamlist.db,
but goodlist.db is never being updated at all).  Do I need to do
anything special in order to have bogofilter process a message as ham?
I have `gnus-group-ham-exit-processor-bogofilter' defined in the group
parameters for my ham group (nnml:inbox).

Here are my current group parameters for nnml:inbox:

((spam-contents gnus-group-spam-classification-ham)
 (spam-process
  (gnus-group-spam-exit-processor-bogofilter gnus-group-ham-exit-processor-bogofilter))
 (spam-process-destination . "nnml:spam")
 (visible . t))

-Emerick
-- 
-------------------------------------------------------------------------
Emerick Rogul             /\/     "how young are you, how old am i?
emerick@cs.bu.edu         /\/      let's count the rings around my eyes."
------------------------------------------------- 'i will dare', the mats

Re: How do you make spam automatically move to spam group on exit?

[Ted Zlatanov]

On Fri, 07 Mar 2003, emerick@cs.bu.edu wrote:
> The only remaining problem I have is that my ham doesn't appear to
> be getting processed by bogofilter (spam is being added to
> spamlist.db, but goodlist.db is never being updated at all).  Do I
> need to do anything special in order to have bogofilter process a
> message as ham?  I have `gnus-group-ham-exit-processor-bogofilter'
> defined in the group parameters for my ham group (nnml:inbox).

Look at your ham marks - only those articles marked as ham (Unread
messages do not count!) will be processed as ham.  Otherwise your
settings look OK.

Ted

Re: How do you make spam automatically move to spam group on exit?

[Andreas Fuchs]

Today, Ted Zlatanov <tzz@lifelogs.com> wrote:
> Look at your ham marks - only those articles marked as ham (Unread
> messages do not count!) will be processed as ham.  Otherwise your
> settings look OK.

BTW, why do unread messages not count? My usage pattern is to move
wrongly filed messages to the group in which they belong, because I'm
using adaptive scoring; so, I would like to do starting from my spambox:

1. mark ham message as unread
2. exit group, have it moved to an intermediary group
4. move (ideally by B r) message to correct group.

adding the unread mark to spam-ham-marks (in the spam group only) with:

(add-hook 'gnus-summary-prepared-hook
	  (lambda ()
	    (when (string-equal gnus-newsgroup-name
	    "nnml+private:spambox") ; that's my spam-split-group name.
	      (add-to-list (make-local-variable 'spam-spam-marks)
			   'gnus-expirable-mark)
	      (add-to-list (make-local-variable 'spam-ham-marks)
			   'gnus-unread-mark)
	      (setq spam-ham-marks (remove 'gnus-expirable-mark
	      spam-ham-marks)))))
              
works pretty well; although spam.el marks unread messages as read,
somehow. Could that have something to do with registering them as ham
when I exit the spam group?

Thanks,
-- 
Andreas Fuchs, <asf@acm.org>, asf@jabber.at, antifuchs
irc.freenode.net's #emacs - online emacs advice from IRC addicts

Re: How do you make spam automatically move to spam group on exit?

[Ted Zlatanov]

On Sat, 8 Mar 2003, asf@void.at wrote:
> Today, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> Look at your ham marks - only those articles marked as ham (Unread
>> messages do not count!) will be processed as ham.  Otherwise your
>> settings look OK.
> 
> BTW, why do unread messages not count? 

I know it seems counter-intuitive, but unread messages are neither ham
nor spam.  They are just mail.  Spam groups are the only place where
unread messages get processed automatically (given the spam mark).

> My usage pattern is to move wrongly filed messages to the group in
> which they belong, because I'm using adaptive scoring; so, I would
> like to do starting from my spambox:
> 
> 1. mark ham message as unread
> 2. exit group, have it moved to an intermediary group
> 4. move (ideally by B r) message to correct group.
> adding the unread mark to spam-ham-marks (in the spam group only)
> with:
> 
> (add-hook 'gnus-summary-prepared-hook
> 	  (lambda ()
> 	    (when (string-equal gnus-newsgroup-name
> 	    "nnml+private:spambox") ; that's my spam-split-group name.
> 	      (add-to-list (make-local-variable 'spam-spam-marks)
> 			   'gnus-expirable-mark)
> 	      (add-to-list (make-local-variable 'spam-ham-marks)
> 			   'gnus-unread-mark)
> 	      (setq spam-ham-marks (remove 'gnus-expirable-mark
> 	      spam-ham-marks)))))
>               
> works pretty well;

I like that solution.  Maybe there should be a group parameter for the
spam-spam-marks and spam-ham-marks.

> although spam.el marks unread messages as read, somehow. Could that
> have something to do with registering them as ham when I exit the
> spam group?

Unfortunately, spam groups *always* mark unread articles as
spam. That's the way they work.  You should use the
ham-process-destination parameter to move out ham, once you've marked
it as ham, and you should not use the unread mark to be a ham mark.
Use the gnus-read-mark (which already is a ham mark by default, and
will also positively affect the adaptive score).  That way you just
read the message if it's ham, quit the group, and the message will be
moved automatically to the ham-process-destination.

spam-ham-move-routine will be invoked after all the other
spam/ham processors to move the ham out of a spam group to the
spam-process-destination.

I realize this is quite confusing, but spam.el is very accomodating to
various ways of looking at spam and processing it.  It would be very,
very nice if in addition to the group and variable customizations,
users could have a graphical interface where they could see the visual
flow of spam and ham based on their variable selections.  If not that,
at least some way of mechanically translating all the spam.el settings
into English would be nice.  Something like:

"Globally, you use the Bogofilter and BBDB incoming mail splitters.
BBDB looks for people in the BBDB database and flags mail from them as
ham, so it will never go to a spam group (instead, it will go to the
next place in the nnmail-split-fancy list).  Bogofilter looks for spam
mail and moves spam, when found, to group A.

"group A is a spam group.  On exit, spam articles (those marked with
the spam-mark, the killed-mark, ...) will be processed with the
Bogofilter spam processor.  On exit, ham articles (those marked with
the read-mark, ...) will be moved to group B."

It's not difficult, but it's a lot of work to implement that.

Ted

Re: How do you make spam automatically move to spam group on exit?

[Andreas Fuchs]

Today, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> BTW, why do unread messages not count? 
> 
> I know it seems counter-intuitive, but unread messages are neither ham
> nor spam.  They are just mail.  Spam groups are the only place where
> unread messages get processed automatically (given the spam mark).

Mm-hm, I see.

> I like that solution.  Maybe there should be a group parameter for the
> spam-spam-marks and spam-ham-marks.

That's what my hack (titled a solution) is trying to emulate. (-;

> Unfortunately, spam groups *always* mark unread articles as
> spam. That's the way they work.  You should use the
> ham-process-destination parameter to move out ham, once you've marked
> it as ham, and you should not use the unread mark to be a ham mark.
> Use the gnus-read-mark (which already is a ham mark by default, and
> will also positively affect the adaptive score).  That way you just
> read the message if it's ham, quit the group, and the message will be
> moved automatically to the ham-process-destination.
> 
> spam-ham-move-routine will be invoked after all the other
> spam/ham processors to move the ham out of a spam group to the
> spam-process-destination.

Could spam-ham-move-routine mark the messages as unread after moving
them? I do really would like to read the message in the "right" group,
and not in my spambox or my "false positive" box.

Thanks for the explanation,
-- 
Andreas Fuchs, <asf@acm.org>, asf@jabber.at, antifuchs
irc.freenode.net's #emacs - online emacs advice from IRC addicts

Re: How do you make spam automatically move to spam group on exit?

[Ted Zlatanov]

On Sat, 8 Mar 2003, asf@void.at wrote:
> Today, Ted Zlatanov <tzz@lifelogs.com> wrote:
>> Unfortunately, spam groups *always* mark unread articles as
>> spam. That's the way they work.  You should use the
>> ham-process-destination parameter to move out ham, once you've
>> marked it as ham, and you should not use the unread mark to be a
>> ham mark.  Use the gnus-read-mark (which already is a ham mark by
>> default, and will also positively affect the adaptive score).  That
>> way you just read the message if it's ham, quit the group, and the
>> message will be moved automatically to the ham-process-destination.
>> 
>> spam-ham-move-routine will be invoked after all the other
>> spam/ham processors to move the ham out of a spam group to the
>> spam-process-destination.
> 
> Could spam-ham-move-routine mark the messages as unread after moving
> them? I do really would like to read the message in the "right"
> group, and not in my spambox or my "false positive" box.

I added the spam-mark-ham-unread-before-move-from-spam-group variable,
enjoy.

Ted

Re: How do you make spam automatically move to spam group on exit?

[Emerick Rogul]

Ted Zlatanov writes:

> Look at your ham marks - only those articles marked as ham (Unread
> messages do not count!) will be processed as ham.

My ham group is an auto-expirable group.  It seems that it gets the
'E' mark before it's sent to the ham processor; when I exit the group,
none of my ham is processed by the ham processor.  If I explicitly
mark the ham with 'd' before exiting the group, it's correctly
processed as ham.  Is that really the correct behavior?  I'd expect
any new non-spam mail in a ham group to be processed by the ham
processor on group exit (regardless of whether or not the message was
auto-expired).

-Emerick
-- 
-------------------------------------------------------------------------
Emerick Rogul             /\/     "how young are you, how old am i?
emerick@cs.bu.edu         /\/      let's count the rings around my eyes."
------------------------------------------------- 'i will dare', the mats

Re: How do you make spam automatically move to spam group on exit?

[Jorge Godoy]

Emerick Rogul <emerick@cs.bu.edu> writes:

> My ham group is an auto-expirable group.  It seems that it gets the
> 'E' mark before it's sent to the ham processor; when I exit the group,
> none of my ham is processed by the ham processor.  If I explicitly
> mark the ham with 'd' before exiting the group, it's correctly
> processed as ham.  Is that really the correct behavior?  I'd expect
> any new non-spam mail in a ham group to be processed by the ham
> processor on group exit (regardless of whether or not the message was
> auto-expired).

I never remember right, but would it be impossible for you to change
this specific group from auto expire to total expire? The result will
be the same without the 'E' marks.


See you,
-- 
Godoy.    <godoy@metalab.unc.edu>

Re: How do you make spam automatically move to spam group on exit?

[Andreas Fuchs]

On 2003-03-08, Ted Zlatanov <tzz@lifelogs.com> wrote:
> I added the spam-mark-ham-unread-before-move-from-spam-group variable,
> enjoy.

Great, thank you!

-- 
Andreas Fuchs, <asf@acm.org>, asf@jabber.at, antifuchs
irc.freenode.net's #emacs - online emacs advice from IRC addicts

Re: How do you make spam automatically move to spam group on exit?

[Emerick Rogul]

Jorge Godoy writes:

> I never remember right, but would it be impossible for you to change
> this specific group from auto expire to total expire? The result
> will be the same without the 'E' marks.

Yes, that actually works, so I'm content with that (thanks!)

I think that I've finally puzzled-out the rest of the configuration
necessary to use bogofilter with spam.el to suit my own needs.  I've
done all of my configuration via gnus variables rather than group
parameters, because I'd rather have it all gathered together in one
place and I have a lot of groups.  The way I have things setup,
misclassified ham is moved to nnml:inbox and misclassified spam is
moved to nnml:spam on group exit.  All nnml mailgroups are configured
to perform both spam and ham processing on group exit.  All nnml
groups are classified as ham, but nnml:spam is classified as spam (I'm
hoping that this does the right thing, ie. the spam classification of
nnml:spam overrides its ham classification - I believe I have the
ordering of the groups correct, and that seems like the behavior that
most people would want: "everything is ham, except this group").

I haven't done extensive testing of this yet, but it seems to do the
right thing so far (the positive path seems to work, at least).  How's
this look to you, Ted?  Does what I've configured below match what
I've said above?  If so, maybe other people will find this config
useful.

Anyway, here it is:

(require 'spam)

(setq spam-use-bogofilter t)

(setq gnus-ham-process-destinations
      '(("^nnml:.*" "nnml:inbox")))

(setq gnus-spam-process-destinations
      '(("^nnml:.*" "nnml:spam")))

(setq gnus-spam-process-newsgroups
      '(("^nnml:.*" (gnus-group-ham-exit-processor-bogofilter
                     gnus-group-spam-exit-processor-bogofilter))))

(setq gnus-spam-newsgroup-contents
      '(("^nnml:spam" gnus-group-spam-classification-spam)
        ("^nnml:.*" gnus-group-spam-classification-ham)))

-Emerick
-- 
-------------------------------------------------------------------------
Emerick Rogul             /\/     "how young are you, how old am i?
emerick@cs.bu.edu         /\/      let's count the rings around my eyes."
------------------------------------------------- 'i will dare', the mats

Re: How do you make spam automatically move to spam group on exit?

[Ted Zlatanov]

On Sun, 09 Mar 2003, emerick@cs.bu.edu wrote:
> Ted Zlatanov writes:
> 
>> Look at your ham marks - only those articles marked as ham (Unread
>> messages do not count!) will be processed as ham.
> 
> My ham group is an auto-expirable group.  It seems that it gets the
> 'E' mark before it's sent to the ham processor; when I exit the
> group, none of my ham is processed by the ham processor.  If I
> explicitly mark the ham with 'd' before exiting the group, it's
> correctly processed as ham.  Is that really the correct behavior?

The intent is to have messages that you read marked "R" (read-mark),
and that's a ham mark.  With auto-expiry that doesn't work, of
course.

> I'd expect any new non-spam mail in a ham group to be processed by
> the ham processor on group exit (regardless of whether or not the
> message was auto-expired).

I suppose it could be added as a feature,
spam-treat-non-spam-as-ham-in-ham-group or something like that.  Is
Jorge's solution of using total-expire inadequate?

Thanks
Ted

Re: How do you make spam automatically move to spam group on exit?

[Ted Zlatanov]

On Sun, 09 Mar 2003, emerick@cs.bu.edu wrote:
> I think that I've finally puzzled-out the rest of the configuration
> necessary to use bogofilter with spam.el to suit my own needs.  I've
> done all of my configuration via gnus variables rather than group
> parameters, because I'd rather have it all gathered together in one
> place and I have a lot of groups.  

I like to use "G c" on topics, personally, but thanks to the nice Gnus
parameters it's all one thing :)

> How's this look to you, Ted?  Does what I've configured below match
> what I've said above?  If so, maybe other people will find this
> config useful.
> 
> Anyway, here it is:
> 
> (require 'spam)
> 
> (setq spam-use-bogofilter t)
> 
> (setq gnus-ham-process-destinations
>       '(("^nnml:.*" "nnml:inbox")))
> 
> (setq gnus-spam-process-destinations
>       '(("^nnml:.*" "nnml:spam")))
> 
> (setq gnus-spam-process-newsgroups
>       '(("^nnml:.*" (gnus-group-ham-exit-processor-bogofilter
>                      gnus-group-spam-exit-processor-bogofilter))))
> 
> (setq gnus-spam-newsgroup-contents
>       '(("^nnml:spam" gnus-group-spam-classification-spam)
>         ("^nnml:.*" gnus-group-spam-classification-ham)))

Well, let me translate it back to English for you, and see if you
agree with the process.

Bogofilter will split your incoming mail into spam-split-group ("spam"
by default), or pass it through to the next item in your
nnmail-split-fancy.

Whenever ham is found in a spam group (as determined by your action of
marking it with a ham-mark) it will be moved to "nnml:inbox".

Whenever spam is found in any group, it will be moved to "nnml:spam"
after being processed and expired (spam will not be processed twice,
since it's no longer considered spam when it is expired).  You can use
spam-move-spam-nonspam-groups-only if you want that to happen only in
non-spam groups.

All groups have Bogofilter as their ham and spam exit processor.  (But
Bogofilter gets invoked multiple times for the same ham!  That's the
case for now, unfortunately, since we lack the registry.)

Any group beginning with "nnml:spam" is a spam group.

Any other group is a ham group.

Looks reasonable to me.  I like to keep most of my groups
unclassified, but that's not a big deal, and I think you're right,
most people prefer a binary spam/ham classification for all mail to
the ternary setup I use.

Ted

Re: How do you make spam automatically move to spam group on exit?

[Emerick Rogul]

Ted Zlatanov writes:

> I suppose it could be added as a feature,
> spam-treat-non-spam-as-ham-in-ham-group or something like that.  Is
> Jorge's solution of using total-expire inadequate?

Nope, Jorge's solution of making the group toal-expirable is
completely adequate.  It might make sense to update the manual to
mention that auto-expire is incompatible with the way that spam.el
works with hame, though; that behavior definitely had me confused for
a while.

-Emerick
-- 
-------------------------------------------------------------------------
Emerick Rogul             /\/     "how young are you, how old am i?
emerick@cs.bu.edu         /\/      let's count the rings around my eyes."
------------------------------------------------- 'i will dare', the mats

Re: How do you make spam automatically move to spam group on exit?

[Emerick Rogul]

Ted Zlatanov writes:

> All groups have Bogofilter as their ham and spam exit processor.
> (But Bogofilter gets invoked multiple times for the same ham!
> That's the case for now, unfortunately, since we lack the registry.)

Is it possible to somehow set the expired mark on processed ham so
that it's not processed again (kind of similar to how spam is handled
now)?  It seems like that would allow you to only process the ham
once, since you wouldn't process already expired ham.  But maybe I'm
missing something, I haven't been keeping up with all of the spam
conversations on here, unfortunately.

Anyway, it seems that I now have this setup pretty well, and
BogoFilter's already correctly processing a lot of my incoming spam.
Thanks for the excellent work on this!

-Emerick
-- 
-------------------------------------------------------------------------
Emerick Rogul             /\/     "how young are you, how old am i?
emerick@cs.bu.edu         /\/      let's count the rings around my eyes."
------------------------------------------------- 'i will dare', the mats

Re: How do you make spam automatically move to spam group on exit?

[Ted Zlatanov]

On Mon, 10 Mar 2003, emerick@cs.bu.edu wrote:
> Ted Zlatanov writes:
> 
>> All groups have Bogofilter as their ham and spam exit processor.
>> (But Bogofilter gets invoked multiple times for the same ham!
>> That's the case for now, unfortunately, since we lack the
>> registry.)
> 
> Is it possible to somehow set the expired mark on processed ham so
> that it's not processed again (kind of similar to how spam is
> handled now)?  

I suspect most people don't want me to expire their mail...

Perhaps I could introduce a processed-ham-mark, similar to the
forwarded/replied-to marks that are added to an article.  But that
does not answer the question of which processor was applied to the
ham/spam.  The registry will have to be implemented for this to work
properly, I'm afraid.

> Anyway, it seems that I now have this setup pretty well, and
> BogoFilter's already correctly processing a lot of my incoming spam.
> Thanks for the excellent work on this!

I'm glad it's working for you.  I've got to finish the registry one of
these days.  I'm working on per-group spam/ham marks, and of course
there's Real Work to be done, but I'll get to it soon.  If anyone's
interested in helping, of course, I won't object :)

Ted

Re: How do you make spam automatically move to spam group on exit?

[Ted Zlatanov]

On Mon, 10 Mar 2003, tzz@lifelogs.com wrote:
> On Mon, 10 Mar 2003, emerick@cs.bu.edu wrote:
>> Is it possible to somehow set the expired mark on processed ham so
>> that it's not processed again (kind of similar to how spam is
>> handled now)?  
> 
> I suspect most people don't want me to expire their mail...
> 
> Perhaps I could introduce a processed-ham-mark, similar to the
> forwarded/replied-to marks that are added to an article.  But that
> does not answer the question of which processor was applied to the
> ham/spam.  The registry will have to be implemented for this to work
> properly, I'm afraid.

I just thought of a hack to get around this while I'm holding things
up.  Set the bogofilter -s/-n switches to -S/-N (there are variables
for that in the spam-bogofilter customization group) when you enter
any group other than nnml:inbox and nnml:spam, and restore them when
you exit the group.  That way, any reclassification will be harmless
(at worst, you'll declassify and then reclassify the same text), and
you'll only get real classification in nnml:inbox and nnml:spam.

Ted