spam.el reporting and moving ham out of spam groups.

spam.el reporting and moving ham out of spam groups.

[Daniel Pittman]

I use the 'train on errors' method for teaching SpamAssassin, which
means that I don't want to be passing all the detected spam or ham
through my reporting system.

I also use the `spam-split' method to filter mail detected by
SpamAssassin headers into a spam group.


What I would like to achieve is that anything marked as unread is
reported as ham (by email :) and then respooled on exit from the spam
group.

Everything else will be killed or marked read, then total-expired.
I would rather *not* give everything the spam mark on entry either,
since I use score files to really obvious stuff as read.


Unfortunately, it doesn't seem possible to set spam.el to process
messages as ham without setting the group classification as 'spam'.

Setting the group as 'spam' means that everything gets the spam mark on
entry...

Is there a way around this, or am I asking the impossible?

   Daniel
-- 
There is no reality except the one contained within us. That is why so many
people live such an unreal life. They take the images outside them for reality
and never allow the world within to assert itself.
        -- Hermann Hesse

Re: spam.el reporting and moving ham out of spam groups.

[Kai Grossjohann]

Daniel Pittman <daniel@rimspace.net> writes:

> Everything else will be killed or marked read, then total-expired.
> I would rather *not* give everything the spam mark on entry either,
> since I use score files to really obvious stuff as read.

I don't understand the last sentence.  What is the problem about
having everything get the spam mark?

Hm.  Perhaps you can use splitting to move the really really obvious
spam into one group, and the just-normal suspected spam into another
group?

Kai

Re: spam.el reporting and moving ham out of spam groups.

[Daniel Pittman]

On 26 May 2004, Kai Grossjohann wrote:
> Daniel Pittman <daniel@rimspace.net> writes:
>
>> Everything else will be killed or marked read, then total-expired.
>> I would rather *not* give everything the spam mark on entry either,
>> since I use score files to really obvious stuff as read.
>
> I don't understand the last sentence.  What is the problem about
> having everything get the spam mark?

Anything with the spam mark is fed to the spam processor on group exit,
which isn't desirable but can be worked around.

Using the spam mark is very visually distinctive, overriding the reduced
score, etc, visual display. While desirable elsewhere, I don't want it
here.

Adding the spam mark to articles overrides any other mark, such as Y,
then was assigned by the scoring process, etc.

> Hm.  Perhaps you can use splitting to move the really really obvious
> spam into one group, and the just-normal suspected spam into another
> group?

The maintenance cost of that would be very high. I would need to
manually edit split rules to add a new "this is spam" indication, rather
than L s s t -ing it.

The SpamAssassin score is not reliable enough an indicator - this is the
content that was ambiguous enough that I need to check for false
positives.

     Daniel

Sure, call me lazy, but it has worked to date. :)
-- 
The two greatest obstacles to democracy in the United States are, first, the
widespread delusion among the poor that we have a democracy, and second, the
chronic terror among the rich, lest we get it.
        -- Edward Dowling

Re: spam.el reporting and moving ham out of spam groups.

[Ted Zlatanov]

On Wed, 26 May 2004, daniel@rimspace.net wrote:

> Using the spam mark is very visually distinctive, overriding the reduced
> score, etc, visual display. While desirable elsewhere, I don't want it
> here.
> 
> Adding the spam mark to articles overrides any other mark, such as Y,
> then was assigned by the scoring process, etc.

Don't forget you can override the default spam-marks, so Y or others
can be spam-marks too.

I don't know if that will help you, but it may be of interest.

Ted

Re: spam.el reporting and moving ham out of spam groups.

[Adam Sjøgren]

On Tue, 25 May 2004 23:17:34 +1000, Daniel wrote:

> Unfortunately, it doesn't seem possible to set spam.el to process
> messages as ham without setting the group classification as 'spam'.

That's right, I think.

I think spam.el would be doing "what-I-mean" more if it ham-processed
messages marked as ham in unclassified groups in addition to spam
groups (that is, all non-ham groups).

(If I mark something as ham, I want it ham-processed (unless it is in
a ham-group, then I've probably marked it by mistake O:-))

I'm using this tiny patch:

@@ -776,8 +838,8 @@
       (spam-ham-copy-routine
        (gnus-parameter-ham-process-destination gnus-newsgroup-name)))
 
-    ;; now move all ham articles out of spam groups
-    (when (spam-group-spam-contents-p gnus-newsgroup-name)
+    ;; now move all ham articles out of non-ham groups
+    (when (not (spam-group-ham-contents-p gnus-newsgroup-name))
       (gnus-message 5 "Moving ham messages from spam group")
       (spam-ham-move-routine
        (gnus-parameter-ham-process-destination gnus-newsgroup-name))))

As discussed in the thread that starts with:
 <news:4ny8nn0wpu.fsf@lifelogs.com>

I think the jury is still out on whether the change makes sense
(except to me :-))


  Best regards,

-- 
 "Stop blowing holes in my ship!"                             Adam Sjøgren
                                                         asjo@koldfront.dk

Re: spam.el reporting and moving ham out of spam groups.

[Daniel Pittman]

On 26 May 2004, Adam Sjøgren wrote:
> On Tue, 25 May 2004 23:17:34 +1000, Daniel wrote:
>
>> Unfortunately, it doesn't seem possible to set spam.el to process
>> messages as ham without setting the group classification as 'spam'.
>
> That's right, I think.
>
> I think spam.el would be doing "what-I-mean" more if it ham-processed
> messages marked as ham in unclassified groups in addition to spam
> groups (that is, all non-ham groups).

That is also undesirable to me, since I want to 'train on errors' for
non-spam as well. :)

What I would really like to be able to do is set, on a per-group basis:

1. which (if any) spam learner is run on spam-marked articles.
2. which (if any) ham learner is run on ham-marked articles.
3. just delete the damn spam marked messages.
4. mark some-or-all messages with the spam mark by default

At the moment I can't have a different configuration for 1 and 2, since
only the `spam-process' group parameter is used, and that defines both
the ham and spam learning side of things.

I only want option 4 to turn it off since it isn't useful for a "train
on errors" paradigm. :)

> (If I mark something as ham, I want it ham-processed (unless it is in
> a ham-group, then I've probably marked it by mistake O:-))

...unless there is some explicit ham mark I have missed, "marked as ham"
means "read, killed or scored down", at least by default...

[...]

> I think the jury is still out on whether the change makes sense
> (except to me :-))

It doesn't make sense to me, because this "process all ham" settings
seems to be the *only* difference between an "unclassified" and a "ham"
group.

Unless I have missed something.
       Daniel
-- 
Beware of all enterprises that require new clothes.
        -- Henry David Thoreau

Re: spam.el reporting and moving ham out of spam groups.

[Adam Sjøgren]

On Wed, 26 May 2004 14:46:27 +1000, Daniel wrote:

>> I think spam.el would be doing "what-I-mean" more if it
>> ham-processed messages marked as ham in unclassified groups in
>> addition to spam groups (that is, all non-ham groups).

> That is also undesirable to me, since I want to 'train on errors'
> for non-spam as well. :)

I don't understand that. I train on error for both spam and ham.

With what I described, messages marked as ham get ham-processed
whenever they are not in a ham-group.

If a message is in a ham-group it is not an error when the message is
ham?

(That is, you can't "train on error" on a ham-message when it's
correctly classified and put in a ham-group, right?)

>> (If I mark something as ham, I want it ham-processed (unless it is
>> in a ham-group, then I've probably marked it by mistake O:-))

> ...unless there is some explicit ham mark I have missed, "marked as
> ham" means "read, killed or scored down", at least by default...

Oh, I've set:

 (ham-marks
  (gnus-ticked-mark))

so I just use that and nothing else. Maybe that's skewing/obscuring my
comments.


Basically my setup is:

      Incoming email

      Does spam.el think it's spam?

 Yes:                      No:

 Put in nnml:spam          Do normal splitting

a) If a message in nnml:spam is ham, I mark it as ham, CRM114 is
   trained on it, and the message is run through normal splitting.

b) If a message not in nnml:spam is spam, I mark it as spam, CRM114 is
   trained on it, and the message is moved to nnml:spam.

I'm probably missing your points because this is simpler/more naïve
than your setup.


  Best regards,

-- 
 "Just the words?" What a terrible to thing to say            Adam Sjøgren
  about writing!                                         asjo@koldfront.dk

Re: spam.el reporting and moving ham out of spam groups.

[Ted Zlatanov]

On Tue, 25 May 2004, asjo@koldfront.dk wrote:

On Tue, 25 May 2004 23:17:34 +1000, Daniel wrote:
> 
>> Unfortunately, it doesn't seem possible to set spam.el to process
>> messages as ham without setting the group classification as 'spam'.
> 
> That's right, I think.
> 
> I think spam.el would be doing "what-I-mean" more if it ham-processed
> messages marked as ham in unclassified groups in addition to spam
> groups (that is, all non-ham groups).
> 
> (If I mark something as ham, I want it ham-processed (unless it is in
> a ham-group, then I've probably marked it by mistake O:-))
> 
> I'm using this tiny patch:

But shouldn't spam-process-ham-in-nonham-groups do this?  If it
doesn't, there's a bug in my code.

Ted

Re: spam.el reporting and moving ham out of spam groups.

[Adam Sjøgren]

On 26 May 2004 12:16:07 -0400, Ted wrote:

> But shouldn't spam-process-ham-in-nonham-groups do this? If it
> doesn't, there's a bug in my code.

I just tested with latest cvs; yes it does work.

I don't think it did when I originally made the modification to
spam.el (I'm quite sure I didn't just make the change for fun, and in
my configuration the variable was set to t, so I did try it before
modifying spam.el...)

Please forget all about that one, then :-)


  Best regards,

-- 
 "Just the words?" What a terrible to thing to say            Adam Sjøgren
  about writing!                                         asjo@koldfront.dk

Re: spam.el reporting and moving ham out of spam groups.

[Adam Sjøgren]

On Wed, 26 May 2004 22:29:15 +0200, Adam wrote:

> On 26 May 2004 12:16:07 -0400, Ted wrote:
>> But shouldn't spam-process-ham-in-nonham-groups do this? If it
>> doesn't, there's a bug in my code.

> I just tested with latest cvs; yes it does work.

Uh, it gets processed, but not respooled.

I enter nnml:spam; these are the group parameters:

((spam-process
  '(ham spam-use-crm114))
 (ham-process-destination
  'respool)
 (spam-process-destination . "nnml:spam")
 (ham-marks
  (gnus-ticked-mark))

I mark (!) some articles that are ham and exit the group (q) - the
articles are registred as ham:

 Registering 6  articles as ham using backend spam-use-crm114

But they are not respooled.

If I enter nnml:spam again, the articles are still ticked, and B q
returns "cvs" (the group they should have been respooled to).

Shouldn't they be moved (respooled)?


I'm setting this in .gnus:

 spam-log-to-registry t
 spam-junk-mailgroups nil ; Everything must be unclassified.
 spam-move-spam-nonspam-groups-only t ; Only process spam in nonspam groups.
 spam-process-ham-in-spam-groups t
 spam-process-ham-in-nonham-groups t
 spam-mark-ham-unread-before-move-from-spam-group t
 spam-use-crm114 t


Maybe I need to set "spam-move-ham-in-nonham-groups" to t?


  Best regards,

-- 
 "Our voodoo-dolls are full of hopes"                         Adam Sjøgren
                                                         asjo@koldfront.dk

Re: spam.el reporting and moving ham out of spam groups.

[Ted Zlatanov]

On Fri, 28 May 2004, asjo@koldfront.dk wrote:

On Wed, 26 May 2004 22:29:15 +0200, Adam wrote: 
> On 26 May 2004 12:16:07 -0400, Ted wrote:
>>> But shouldn't spam-process-ham-in-nonham-groups do this? If it
>>> doesn't, there's a bug in my code.
> 
>> I just tested with latest cvs; yes it does work.
> 
> Uh, it gets processed, but not respooled.

> Maybe I need to set "spam-move-ham-in-nonham-groups" to t?

There is no such parameter now.  The logic for moving ham out of spam
groups is:

    ;; now move all ham articles out of spam groups
    (when (spam-group-spam-contents-p gnus-newsgroup-name)
      (let ((num
	     (spam-ham-move-routine
	      (gnus-parameter-ham-process-destination gnus-newsgroup-name))))
	(when (> num 0)
	  (gnus-message 6 "%d ham messages were moved from spam group" num)))))

so it only works in spam groups.  The belief is that ham is already
in the right place outside of spam groups - why should you respool
it?  It only makes sense to process ham outside of spam groups.

Anyhow, I guess this logic can be changed, but I don't think it makes
sense.  I'm not sure anyone but you would find it useful, and adding
an extra option similar to spam-move-spam-nonspam-groups-only would
complicate the code and documentation without gain for the other
users.

If you feel strongly about this, and/or anyone else wants to comment,
feel free.  I just don't want to make spam.el too esoteric.

Thanks
Ted

Re: spam.el reporting and moving ham out of spam groups.

[Adam Sjøgren]

On 7 Jun 2004 13:12:27 -0400, Ted wrote:

>> On 26 May 2004 12:16:07 -0400, Ted wrote:

>>>> But shouldn't spam-process-ham-in-nonham-groups do this? If it
>>>> doesn't, there's a bug in my code.

>>> I just tested with latest cvs; yes it does work.

>> Uh, it gets processed, but not respooled.

>> Maybe I need to set "spam-move-ham-in-nonham-groups" to t?

> There is no such parameter now.

I know.

> The logic for moving ham out of spam groups is:

[...]

Yes - that is what the (third part of the) patch we are discussing
(<news:874qq4bjr2.fsf@koldfront.dk>) changes.

> The belief is that ham is already in the right place outside of spam
> groups - why should you respool it? It only makes sense to process
> ham outside of spam groups.

That I want to change this, is an artefact stemming from the fact that
I do not want spam in my spam-group automatically marked expirable. 
Thus I keep my spam-group not classified as spam, but non-classified.

> Anyhow, I guess this logic can be changed, but I don't think it
> makes sense.

If you mark something as ham in a non-classified group, it makes sense
to process it, but not to move/respool it?

Okay, I see the point - to me both choices make sense (so "my version"
doesn't make _more_ sense than yours :-))

> If you feel strongly about this, and/or anyone else wants to
> comment, feel free. I just don't want to make spam.el too esoteric.

Don't make is more esoteric on my account. I can easily keep patching.


The root of my configurations mess is that articles in groups
classified as spam automatically get marked expirable (which I don't
want), so if it's easier to make that configurable, that could perhaps
be a compromise. It would solve the whole problem for me.

Again, if you think that is making spam.el too cluttered, don't change
it for just one person :-)


  Best regards,

-- 
 "Who ees thees Kählveen?"                                    Adam Sjøgren
                                                         asjo@koldfront.dk

Re: spam.el reporting and moving ham out of spam groups.

[Ted Zlatanov]

On Mon, 07 Jun 2004, asjo@koldfront.dk wrote:

> The root of my configurations mess is that articles in groups
> classified as spam automatically get marked expirable (which I don't
> want), so if it's easier to make that configurable, that could perhaps
> be a compromise. It would solve the whole problem for me.
> 
> Again, if you think that is making spam.el too cluttered, don't change
> it for just one person :-)

I think I'll leave spam.el alone for now.  During the redesign which
is coming up, I'll think about the flow of spam/ham and may end up
doing what you describe anyhow.

Ted

Re: spam.el reporting and moving ham out of spam groups.

[Ted Zlatanov]

On Tue, 25 May 2004, daniel@rimspace.net wrote:

> I use the 'train on errors' method for teaching SpamAssassin, which
> means that I don't want to be passing all the detected spam or ham
> through my reporting system.
> 
> I also use the `spam-split' method to filter mail detected by
> SpamAssassin headers into a spam group.
> 
> 
> What I would like to achieve is that anything marked as unread is
> reported as ham (by email :) and then respooled on exit from the spam
> group.
> 
> Everything else will be killed or marked read, then total-expired.
> I would rather *not* give everything the spam mark on entry either,
> since I use score files to really obvious stuff as read.

> Unfortunately, it doesn't seem possible to set spam.el to process
> messages as ham without setting the group classification as 'spam'.

Can you try spam-process-ham-in-nonham-groups?

> Setting the group as 'spam' means that everything gets the spam mark on
> entry...
> 
> Is there a way around this, or am I asking the impossible?

Nothing is impossible, except skiing through a revolving door :)

I just added a new variable,
spam-mark-new-messages-in-spam-group-as-spam which is t by default.
If you set it to nil, you'll get the effect you want.

Ted

Re: spam.el reporting and moving ham out of spam groups.

[Miles Bader]

"Ted Zlatanov" <tzz@lifelogs.com> writes:
> Nothing is impossible, except skiing through a revolving door :)

I think you could probably wing that one too, with some of these
enormous motorized revolving doors you see at hotels & the like... :-O

-Miles
-- 
My spirit felt washed.  With blood.  [Eli Shin, on "The Passion of the Christ"]

Re: spam.el reporting and moving ham out of spam groups.

[Daniel Pittman]

On 27 May 2004, Ted Zlatanov wrote:
> On Tue, 25 May 2004, daniel@rimspace.net wrote:

[...]

>> What I would like to achieve is that anything marked as unread is
>> reported as ham (by email :) and then respooled on exit from the spam
>> group.
>>
>> Everything else will be killed or marked read, then total-expired.
>> I would rather *not* give everything the spam mark on entry either,
>> since I use score files to really obvious stuff as read.
>
>> Unfortunately, it doesn't seem possible to set spam.el to process
>> messages as ham without setting the group classification as 'spam'.
>
> Can you try spam-process-ham-in-nonham-groups?

Well, I suppose I could for the spirit of scientific inquiry, if you
wish, but...

>> Setting the group as 'spam' means that everything gets the spam mark on
>> entry...
>>
>> Is there a way around this, or am I asking the impossible?
>
> Nothing is impossible, except skiing through a revolving door :)
>
> I just added a new variable,
> spam-mark-new-messages-in-spam-group-as-spam which is t by default.
> If you set it to nil, you'll get the effect you want.

...with this set to `t', spam.el deals with mail the way I want with my
spam catch-all classified as "spam", and everything else unclassified.

I did need to make a couple of changes to get `spam-report-by-resend' to
do the right thing with the new group parameter you added, though.

Your code would bind the result of searching for the group parameter as
the value of `spam-report-resend-to' unconditionally, which means that
where the group parameter did not exist, the value was `nil'.

This then triggered the "no set resend address" logic which, of course,
didn't actually set the variable as expected, resulting in things not
working as expected.

If there *was* a group parameter, the result was a list containing a
string with the destination address, not the address itself. That needed
to be unpacked before resending...

The following patch does these things, and it all works nicely for me now.

   Daniel

Index: spam.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/spam.el,v
retrieving revision 7.44
diff -u -u -r7.44 spam.el
--- spam.el	26 May 2004 16:38:07 -0000	7.44
+++ spam.el	27 May 2004 06:08:20 -0000
@@ -779,9 +779,10 @@
 Respects the process/prefix convention.  Also see
 `spam-report-resend-to'."
   (interactive "P")
-  (let ((spam-report-resend-to 
-	 (gnus-parameter-spam-resend-to gnus-newsgroup-name))
-	(articles (gnus-summary-work-articles n)))
+  (let* ((spam-report-resend-to-group-param (gnus-parameter-spam-resend-to gnus-newsgroup-name))
+         (spam-report-resend-to (or (nth 0 spam-report-resend-to-group-param)
+                                    spam-report-resend-to))
+         (articles (gnus-summary-work-articles n)))
     (spam-report-resend articles)
     (dolist (article articles)
       (gnus-summary-remove-process-mark article))))
@@ -2114,8 +2115,11 @@
     (apply 'spam-report-gmane articles)))
 
 (defun spam-report-resend-register-routine (articles)
-  (let ((spam-report-resend-to (gnus-parameter-spam-resend-to gnus-newsgroup-name)))
-	(spam-report-resend articles)))
+  (let* ((resend-to-gp (gnus-parameter-spam-resend-to gnus-newsgroup-name))
+         (spam-report-resend-to (or (and (listp resend-to-gp)
+                                         (nth 0 resend-to-gp))
+                                    spam-report-resend-to)))
+    (spam-report-resend articles)))
 
 \f
 ;;;; Bogofilter


-- 
You must not quote to me what I once said. I am wiser now.
        -- Romy Schneider (1966)

Re: spam.el reporting and moving ham out of spam groups.

[Ted Zlatanov]

On Thu, 27 May 2004, daniel@rimspace.net wrote:

> I did need to make a couple of changes to get `spam-report-by-resend' to
> do the right thing with the new group parameter you added, though.
> 
> Your code would bind the result of searching for the group parameter as
> the value of `spam-report-resend-to' unconditionally, which means that
> where the group parameter did not exist, the value was `nil'.
> 
> This then triggered the "no set resend address" logic which, of course,
> didn't actually set the variable as expected, resulting in things not
> working as expected.
> 
> If there *was* a group parameter, the result was a list containing a
> string with the destination address, not the address itself. That needed
> to be unpacked before resending...

Oops, sorry!  Comitted to CVS now (note the use of car-safe, and
lines have to be less than 80 chars).

Thanks
Ted