spam.el and ham in a spam group

spam.el and ham in a spam group

[Jody Klymak]

Hello,

I have been using spam.el for a while with bogofilter.  It seems to do
a pretty good job.  However, getting messages to be run through the
*ham* filter seems cumbersome.  This has come up recently with a few
emails; they are being sorted into my spam-folder (which is designated
as a spam-folder).  I catch them there and mark them as read and they
are moved to my Inbox.  However, they are never sent to bogofilter as
ham examples, and therefore continue to pile up in my spam-folder.

I didn't want to set my Inbox as a ham group because I didn't want all
the mails in the group processed *every* time I exit the group.  So,
it seems that for these cases, I need to send to a special "ham"
group, which I need to enter, and leave before they can be processed
and sorted into my Inbox.

That seems a little clumsy, so either I am missing something, *or*,
I'd suggest that items in my spam folder that I mark as non-spam
should be ham processed.  Is that possible?

On the whole, however, spam.el is great.  Thanks a lot to Ted for
setting this up!

Cheers,   Jody

-- 
Jody Klymak      http://mixing.coas.oregonstate.edu/people/jklymak/
mailto:jklymak@coas.oregonstate.edu   

Re: spam.el and ham in a spam group

[Ted Zlatanov]

On Mon, 21 Jul 2003, jklymak@coas.oregonstate.edu wrote:
> I didn't want to set my Inbox as a ham group because I didn't want
> all the mails in the group processed *every* time I exit the group.

Ah yes, I've been meaning to use the gnus-registry for that, so
messages are only processed once.  It's not hard, I'm just unable to
do it right now.  The tracking data needs to be stored in the "extra"
field of the registry entry, the rest is already handled by the
registry.

> That seems a little clumsy, so either I am missing something, *or*,
> I'd suggest that items in my spam folder that I mark as non-spam
> should be ham processed.  Is that possible?

Currently it's not, because we only process ham in ham groups, but I
can add a setting spam-process-ham-in-nonham-groups if you wish.

Ted

Re: spam.el and ham in a spam group

[Jody Klymak]

Hello Ted,

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Mon, 21 Jul 2003, jklymak@coas.oregonstate.edu wrote:
>
>> That seems a little clumsy, so either I am missing something, *or*,
>> I'd suggest that items in my spam folder that I mark as non-spam
>> should be ham processed.  Is that possible?
>
> Currently it's not, because we only process ham in ham groups, but I
> can add a setting spam-process-ham-in-nonham-groups if you wish.

I guess that I would like that.  Well, unless leaving my Inbox as a
ham group and rehamming the messages over and over is not bad for
bogofilter.  Its certainly quick enough, so that is not a problem.

Here is another problem, and perhaps more important: I have my Inbox
set so that I do not see "old" (O) or "expired" (E) messages, only
"important" (!) ones.  Moving "read" (d) items from the spam folder
into the Inbox made them "old".  So, they do not show up in the
buffer.  I have to C-u into the group, mark the message as "important"
so that I can deal with it.

Can the "important" mark be made a ham mark?  Then I could mark
important items in my spam group, and have them moved to by Inbox.

Thanks,  Jody

-- 
Jody Klymak      http://mixing.coas.oregonstate.edu/people/jklymak/
mailto:jklymak@coas.oregonstate.edu   

Re: spam.el and ham in a spam group

[Danny Siu]

Ted Zlatanov writes:

  Ted> On Mon, 21 Jul 2003, jklymak@coas.oregonstate.edu wrote:
  >> I didn't want to set my Inbox as a ham group because I didn't want all
  >> the mails in the group processed *every* time I exit the group.

i have the same problem too.  some unfiltered spam may end up in my Inbox,
which is not set as ham folder, and it is my least interest to
mis-classified them.

is this a general problem with current gnus spam filtering?  what can the
end user do when a mail is mis-classified?  i use spam-stat and have some
time to time i still get mis-classified spam in Inbox or ham in Spam-Box.

other than that, gnus-spam, especially spam-stat is simply wonderful. thanks
Ted!

-- 
Danny Siu

Re: spam.el and ham in a spam group

[David Z Maze]

Danny Siu <dsiu@adobe.com> writes:

> i have the same problem too.  some unfiltered spam may end up in my Inbox,
> which is not set as ham folder, and it is my least interest to
> mis-classified them.
>
> is this a general problem with current gnus spam filtering?  what can the
> end user do when a mail is mis-classified?

Press 'S x' on the misclassified (false-negative) mail; when you exit
the group, it will be refiled to your spam group (and not processed as
ham).  If something is misclassified as a false positive, the best
thing I've found to do is to mark the article as unread ('M SPC') and
then move it to the real group.  (It'd be nice if it called
gnus-summary-respool-article for ham articles in spam groups in a way
that spam-split did nothing and returned nil.)

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: spam.el and ham in a spam group

[Ted Zlatanov]

On Tue, 22 Jul 2003, dsiu@adobe.com wrote:
> Ted Zlatanov writes:
> 
>   Ted> On Mon, 21 Jul 2003, jklymak@coas.oregonstate.edu wrote:
>   >> I didn't want to set my Inbox as a ham group because I didn't
>   >> want all the mails in the group processed *every* time I exit
>   >> the group.
> 
> i have the same problem too.  some unfiltered spam may end up in my
> Inbox, which is not set as ham folder, and it is my least interest
> to mis-classified them.

> is this a general problem with current gnus spam filtering?  what
> can the end user do when a mail is mis-classified?  i use spam-stat
> and have some time to time i still get mis-classified spam in Inbox
> or ham in Spam-Box.

You mark the spam as such (M-d or `S x').  You exit the Inbox or any
other group.  Any group with spam will have its spam processed
(INCLUDING spam groups, UNLESS you set
spam-move-spam-nonspam-groups-only, in which case the spam-processor
is still invoked but the spam-destination is not used).

The spam will be processed by the group's spam-processor.

Then, the spam will marked as expired and moved to the
spam-destination parameter of the group.

I hope this clarifies the process.

Ted

Re: spam.el and ham in a spam group

[Chris Green]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I hope this clarifies the process.

How should ham in a spam group be handled?  AFAICT, the
ham-exit stuff only runs in a ham group.

The common tasks that I have to do are:

 - unregister a message as spam if I didn't catch it was a false
   positive on the first exit from a spam group

  ( doing this manually via )

    C-x h M-| bogofilter -S  C-x h M-| bogofilter -n

 - register a message as non-spam if I catch it the first time

    C-x h M-| bogofilter -n

   instead of not registering it at all

I may very well be missing something obvious. M-d handles the spam in
the ham group very well although I'm not positive it does the
unregister step.
-- 
Chris Green <cmg@sourcefire.com>
 "Not everyone holds these truths to be self-evident, so we've worked
                  up a proof of them as Appendix A." --  Paul Prescod

Re: spam.el and ham in a spam group

[Ted Zlatanov]

On Wed, 23 Jul 2003, cmg@sourcefire.com wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> I hope this clarifies the process.
> 
> How should ham in a spam group be handled?  AFAICT, the
> ham-exit stuff only runs in a ham group.

The original message that started this thread asked this question
(hence the subject), and I answered that I will add a flag to process
ham from a spam group.

> The common tasks that I have to do are:
> 
>  - unregister a message as spam if I didn't catch it was a false
>    positive on the first exit from a spam group
> 
>   ( doing this manually via )
> 
>     C-x h M-| bogofilter -S  C-x h M-| bogofilter -n
> 
>  - register a message as non-spam if I catch it the first time
> 
>     C-x h M-| bogofilter -n
> 
>    instead of not registering it at all
> 
> I may very well be missing something obvious. M-d handles the spam
> in the ham group very well although I'm not positive it does the
> unregister step.

Unregistering and generally tracking spam/ham registrations won't be
done until the registry is hooked up with spam.el.  It's a planned
feature but won't be done right away due to lack of time.  Sorry.

Ted

Re: spam.el and ham in a spam group

[Ted Zlatanov]

On Tue, 22 Jul 2003, jklymak@coas.oregonstate.edu wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:

>> Currently it's not, because we only process ham in ham groups, but
>> I can add a setting spam-process-ham-in-nonham-groups if you wish.
> 
> I guess that I would like that.  Well, unless leaving my Inbox as a
> ham group and rehamming the messages over and over is not bad for
> bogofilter.  Its certainly quick enough, so that is not a problem.

I think registering ham many times will skew the bogofilter
statistics, but I don't know for sure since I don't use it personally.

spam-process-ham-in-nonham-groups is added to the CVS spam.el - try it
out and let me know how it goes.

> Here is another problem, and perhaps more important: I have my Inbox
> set so that I do not see "old" (O) or "expired" (E) messages, only
> "important" (!) ones.  Moving "read" (d) items from the spam folder
> into the Inbox made them "old".  So, they do not show up in the
> buffer.  I have to C-u into the group, mark the message as
> "important" so that I can deal with it.
> 
> Can the "important" mark be made a ham mark?  Then I could mark
> important items in my spam group, and have them moved to by Inbox.

Sure, you can customize the gnus-parameter-ham-marks-alist variable or
the ham-marks parameter of the topic or group you want.  Just add
gnus-ticked-mark to the list (it's on the list of possible choices in
the Customize menu now as well, if you use that).

Ted

Re: spam.el and ham in a spam group

[Jody Klymak]

Hello Ted, 

Ted Zlatanov <tzz@lifelogs.com> writes:

> spam-process-ham-in-nonham-groups is added to the CVS spam.el - try it
> out and let me know how it goes.
>

Hmmmm, I guess this is not *quite* what I wanted.  I only want to
spam-process-ham-in-spam-groups!  i.e. if I have a false positive,
tick it, and ham-process it.  As it is now, setting this variable to
true processes all ham in all my groups.  I'm sorry for not catching
this in your previous email.  

>> Can the "important" mark be made a ham mark?  Then I could mark
>> important items in my spam group, and have them moved to by Inbox.
>
> Sure, you can customize the gnus-parameter-ham-marks-alist variable or
> the ham-marks parameter of the topic or group you want.  Just add
> gnus-ticked-mark to the list (it's on the list of possible choices in
> the Customize menu now as well, if you use that).

This is great.  Thank you!

Thanks again - spam.el is really great.

Cheers,  Jody

-- 
Jody Klymak      http://mixing.coas.oregonstate.edu/people/jklymak/
mailto:jklymak@coas.oregonstate.edu   

Re: spam.el and ham in a spam group

[Jody Klymak]

Hi again,

This does what I want; sorry that its not in diff-form; I'm not secure
enough in my lisp programming to think it should be included
verbatim.  

    (when (or (spam-group-ham-contents-p gnus-newsgroup-name)
	      (and (spam-group-spam-contents-p gnus-newsgroup-name) 
	        spam-process-ham-in-spam-groups))

Thanks again,  Jody

-- 
Jody Klymak      http://mixing.coas.oregonstate.edu/people/jklymak/
mailto:jklymak@coas.oregonstate.edu   

Re: spam.el and ham in a spam group

[Ted Zlatanov]

On Wed, 23 Jul 2003, jklymak@coas.oregonstate.edu wrote:
> This does what I want; sorry that its not in diff-form; I'm not
> secure enough in my lisp programming to think it should be included
> verbatim.
> 
>     (when (or (spam-group-ham-contents-p gnus-newsgroup-name)
> 	      (and (spam-group-spam-contents-p gnus-newsgroup-name) 
> 	        spam-process-ham-in-spam-groups))

Thanks for the suggestion, I added spam-process-ham-in-spam-groups as
well in CVS.

You can generate patches very easily.  Just run 

cvs diff -u FILENAME

after you save your changes to FILENAME, assuming you use CVS to check
the latest Gnus out.

Ted

Re: spam.el and ham in a spam group

[Jody Klymak]

Hello Ted,

Ted Zlatanov <tzz@lifelogs.com> writes:

> Thanks for the suggestion, I added spam-process-ham-in-spam-groups as
> well in CVS.

Thanks, I think that this does essentially what I want now.  And I
don't think I'll be over-training so long as I just send
false-positives to bogofilter.  The fun thing about these Bayesian
schemes is that only time will tell if it starts wandering!

> You can generate patches very easily.  Just run 
>
> cvs diff -u FILENAME

Ah, cool.

Thanks again - I think you've done a very nice job on this addition to
gnus.

Cheers,  Jody

-- 
Jody Klymak      http://mixing.coas.oregonstate.edu/people/jklymak/
mailto:jklymak@coas.oregonstate.edu   

Re: spam.el and ham in a spam group

[Stefan Reichör]

On Thu, 24 Jul 2003, Ted Zlatanov uttered the following:

On Wed, 23 Jul 2003, jklymak@coas.oregonstate.edu wrote:
> > This does what I want; sorry that its not in diff-form; I'm not
> > secure enough in my lisp programming to think it should be
> > included verbatim.
> > 
> >     (when (or (spam-group-ham-contents-p gnus-newsgroup-name)
> > 	      (and (spam-group-spam-contents-p gnus-newsgroup-name) 
> > 	        spam-process-ham-in-spam-groups))
>  
>  Thanks for the suggestion, I added spam-process-ham-in-spam-groups
>  as well in CVS.
>  

I am not sure, how the spam-process-ham-in-spam-groups should work.

What I really want is a possibility to mark some messages as ham and
to respool them. How can I achieve this?

Now I use spam-stat as spam filter and I use the following function
to register the message as non spam. After that I set the process
mark. I apply that for every ham message in the spam group.

(defun xsteve-summary-is-non-spam ()
  (interactive)
  (gnus-summary-show-article t)
  (other-window 1)
  (goto-char (point-min))
  (spam-stat-buffer-is-non-spam)
  (other-window -1)
  (gnus-summary-clear-mark-forward 1)
  (next-line -1)
  (gnus-summary-mark-as-processable 1))


Before I exit the group I respool the ham messages via B r

This way the messages are delivered to the group they belong.


I think it would be nice to add something like this to spam.el
Or can spam.el handle this already?

--
  Stefan.

Re: spam.el and ham in a spam group

[Ted Zlatanov]

On Fri, 25 Jul 2003, xsteve@riic.at wrote:
> I am not sure, how the spam-process-ham-in-spam-groups should work.

It allows a ham-processor to be set for a spam group.

> What I really want is a possibility to mark some messages as ham and
> to respool them. How can I achieve this?

With the ham-process-destination group/topic parameter, as described
in the manual.

> Now I use spam-stat as spam filter and I use the following function
> to register the message as non spam. After that I set the process
> mark. I apply that for every ham message in the spam group.
> 
> (defun xsteve-summary-is-non-spam ()
>   (interactive)
>   (gnus-summary-show-article t)
>   (other-window 1)
>   (goto-char (point-min))
>   (spam-stat-buffer-is-non-spam)
>   (other-window -1)
>   (gnus-summary-clear-mark-forward 1)
>   (next-line -1)
>   (gnus-summary-mark-as-processable 1))
> 
> 
> Before I exit the group I respool the ham messages via B r
> 
> This way the messages are delivered to the group they belong.
> 
> 
> I think it would be nice to add something like this to spam.el
> Or can spam.el handle this already?

Yes, you just need to add the spam-stat ham processor to the spam
group's ham-processor group/topic parameter, and set a
ham-process-destination group/topic parameter for the same spam group.
It will do what you have written, I think.

Ted