Trouble with spam.el and ifile

Trouble with spam.el and ifile

[David Z Maze]

[-- Attachment #1: Type: text/plain, Size: 338 bytes --]

I've been using ifile-gnus.el for spam sorting, and figured I'd try
spam.el's newfangled ifile support.  Really fresh CVS...

Problem 1: Gnus tries to feed all of my mail into a "non-spam" group.
It looks like the sense of spam-ifile-all-categories is backwards in
the code from what the documentation says; inverting the test fixes
it.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 561 bytes --]

--- spam.el.orig        Tue Jan  7 11:41:17 2003
+++ spam.el     Tue Jan  7 11:41:22 2003
@@ -522,7 +522,7 @@
        (if (not (eobp))
            (setq category (buffer-substring (point) (spam-point-at-eol))))
        (when (not (zerop (length category))) ; we need a category here
-         (if spam-ifile-all-categories
+         (if (not spam-ifile-all-categories)
              (when (string-equal spam-ifile-spam-category category)
                (setq return spam-split-group))
            (setq return category)))))  ; always accept the ifile category

[-- Attachment #3: Type: text/plain, Size: 551 bytes --]


Problem 2: Spam doesn't actually seem to get noticed by ifile, and if
I visit my spam group the articles are marked "Y" rather than "H".  My
.gnus file has, inside a setq block:

      spam-junk-mailgroups '("mail.misc.spam")
      spam-split-group "mail.misc.spam"
      spam-use-ifile t

My nnmail-split-fancy begins with (: spam-split) as pretty much the
first thing.  Any hints?

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Trouble with spam.el and ifile

[Ted Zlatanov]

On Tue, 07 Jan 2003, dmaze@MIT.EDU wrote:
> I've been using ifile-gnus.el for spam sorting, and figured I'd try
> spam.el's newfangled ifile support.  Really fresh CVS...

Excellent, my own guinea pig!  I mean, tester... :)

> Problem 1: Gnus tries to feed all of my mail into a "non-spam"
> group.  It looks like the sense of spam-ifile-all-categories is
> backwards in the code from what the documentation says; inverting
> the test fixes it.

Oops.  It's an "unless" now.  Thanks.

> Problem 2: Spam doesn't actually seem to get noticed by ifile, and
> if I visit my spam group the articles are marked "Y" rather than
> "H".  My .gnus file has, inside a setq block:
> 
>       spam-junk-mailgroups '("mail.misc.spam")
>       spam-split-group "mail.misc.spam"
>       spam-use-ifile t
> 
> My nnmail-split-fancy begins with (: spam-split) as pretty much the
> first thing.  Any hints?

Make sure you use the (customize-group "spam") interface.  It's much
easier to get the format right that way.

Your spam group(s) need to have the ifile spam-processor added, and to
have the spam-contents set to spam.  Customize the group or topic, or
you can do it through customizing the gnus-spam-process-newsgroups
variable.  See my earlier message for a quick primer on spam/ham
processors and group contents.

The 'Y' mark stands for a lowered score, right?  spam.el only marks
unread articles as spam on summary entry in a spam group, and only
processes spam-marked articles with the group's spam processors on
summary exit (for any group, not just spam groups).

Ted

Re: Trouble with spam.el and ifile

[David Z Maze]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 07 Jan 2003, dmaze@MIT.EDU wrote:
>> I've been using ifile-gnus.el for spam sorting, and figured I'd try
>> spam.el's newfangled ifile support.  Really fresh CVS...
>
> Excellent, my own guinea pig!  I mean, tester... :)

Oops, knew I shouldn't have spoken up.  :-)

>> Problem 2: Spam doesn't actually seem to get noticed by ifile, and
>> if I visit my spam group the articles are marked "Y" rather than
>> "H".  My .gnus file has, inside a setq block:
>> 
>>       spam-junk-mailgroups '("mail.misc.spam")
>>       spam-split-group "mail.misc.spam"
>>       spam-use-ifile t
>> 
>> My nnmail-split-fancy begins with (: spam-split) as pretty much the
>> first thing.  Any hints?
>
> Make sure you use the (customize-group "spam") interface.  It's much
> easier to get the format right that way.

Eew.  Um, I mean, okay.  The custom interface still doesn't make
things really clear.  (And I missed things doing C-h v spam-<TAB>,
because some of the variables are gnus-spam-*.)  But now I have

      gnus-spam-newsgroup-contents '(("mail.misc.spam" gnus-group-spam-classification-spam))
      gnus-spam-process-newsgroups '((".*" (gnus-group-spam-exit-processor-ifile)))
      gnus-spam-process-destinations '((".*" "nnml:mail.misc.spam"))
      spam-junk-mailgroups '("mail.misc.spam")
      spam-split-group "mail.misc.spam"
      spam-use-ifile t

Is this right?  It should be clearer in the documentation/Customize
descriptions where you need the backend prefix and where you don't.
It looks like all of this is set up with gnus-define-group-parameter,
and examples with e.g. expiry suggest you don't want it; it still
doesn't seem to work for me either way.  Now at least incoming
articles get filed properly, but if I find a misclassified spam
article and run 'S x' on it, it gets the 'H' mark but nothing happens
when I exit the group.  This happens both with and without the nnml:
prefix in the regexps above.

> Your spam group(s) need to have the ifile spam-processor added, and to
> have the spam-contents set to spam.  Customize the group or topic, or
> you can do it through customizing the gnus-spam-process-newsgroups
> variable.

...I *think* I've got this...

> The 'Y' mark stands for a lowered score, right?  spam.el only marks
> unread articles as spam on summary entry in a spam group, and only
> processes spam-marked articles with the group's spam processors on
> summary exit (for any group, not just spam groups).

Aha.  This makes it a little trickier for me to test, since most of my
spam happens to have a low score in my spam group.  If I mark an
article not-spam in the spam group, does it get refiled to the next
best group on exit?

Also, part of Jeremy Brown's ifile-gnus distribution is a Bourne shell
script that can scan an nnml directory tree and prime ifile with your
extant messages.  It might be good to incorporate this or something
like it into Gnus.  (This is especially true for spam-stat; doing this
classification entirely in elisp looks like it might be a big
performance win over ifile, but having to manually initialize
spam-stat from several hundred not-spam nnml groups would suck.)

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Trouble with spam.el and ifile

[Ted Zlatanov]

On Tue, 07 Jan 2003, dmaze@MIT.EDU wrote:
> The custom interface still doesn't make things really clear.  (And I
> missed things doing C-h v spam-<TAB>, because some of the variables
> are gnus-spam-*.)  

That's because they are in gnus.el, and are Gnus-global variables
rather than spam.el variables.  If Lars or other Gnus people think I
should rename them, no problem (but it will break existing setups).

> But now I have
> 
>       gnus-spam-newsgroup-contents '(("mail.misc.spam" gnus-group-spam-classification-spam))
>       gnus-spam-process-newsgroups '((".*" (gnus-group-spam-exit-processor-ifile)))
>       gnus-spam-process-destinations '((".*" "nnml:mail.misc.spam"))
>       spam-junk-mailgroups '("mail.misc.spam") 
>       spam-split-group "mail.misc.spam"
>       spam-use-ifile t
> 
> Is this right?  

Yes.  It says "mail.misc.spam is a spam group; all my groups should be
spam-processed with ifile; all spam articles should be moved to
mail.misc.spam after spam proecsssing; all my incoming spam should be
filtered, through ifile, into mail.misc.spam."

> It should be clearer in the documentation/Customize descriptions
> where you need the backend prefix and where you don't.

Yeah, the docs need work for sure.  I've been trying to get the
package functionality in place first.

> It looks like all of this is set up with
> gnus-define-group-parameter, and examples with e.g. expiry suggest
> you don't want it

I'm not sure what you mean.

> it still doesn't seem to work for me either way.  Now at least
> incoming articles get filed properly, but if I find a misclassified
> spam article and run 'S x' on it, it gets the 'H' mark but nothing
> happens when I exit the group.  This happens both with and without
> the nnml: prefix in the regexps above.

You don't need the prefix.  Let's start with just the "spam" groups
for spam-contents and spam-process:

      gnus-spam-newsgroup-contents '(("spam" gnus-group-spam-classification-spam))
      gnus-spam-process-newsgroups '(("spam" (gnus-group-spam-exit-processor-ifile)))

See if that triggers the ifile spam processing on summary exit.  You
must have some spam-marked articles.  You can test the processing
manually with M-: (spam-summary-prepare-exit) - that's the function
that gets invoked at summary exit.

>> The 'Y' mark stands for a lowered score, right?  spam.el only marks
>> unread articles as spam on summary entry in a spam group, and only
>> processes spam-marked articles with the group's spam processors on
>> summary exit (for any group, not just spam groups).
> 
> Aha.  This makes it a little trickier for me to test, since most of
> my spam happens to have a low score in my spam group.  

You can customize spam-spam-marks to include the low score mark, not
just the spam-mark.

> If I mark an article not-spam in the spam group, does it get refiled
> to the next best group on exit?

No.  You have to move it manually (but that functionality could be
added).  I assume you mean "mark unread," since there is no not-spam
mark.  An unread article is not ham or spam, it's unclassified.  The
spam-ham-marks only matter in a ham group, only in such a group are
they considered by the ham processor.

> Also, part of Jeremy Brown's ifile-gnus distribution is a Bourne
> shell script that can scan an nnml directory tree and prime ifile
> with your extant messages.  It might be good to incorporate this or
> something like it into Gnus.  

I can do that.

> (This is especially true for spam-stat; doing this classification
> entirely in elisp looks like it might be a big performance win over
> ifile, 

I haven't tested the performance, but ifile definitely has better
lexing than spam-stat according to the documentation.

Ted

Re: Trouble with spam.el and ifile

[David Z Maze]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 07 Jan 2003, dmaze@MIT.EDU wrote:
>> Is this right?  
>
> Yes.  It says "mail.misc.spam is a spam group; all my groups should be
> spam-processed with ifile; all spam articles should be moved to
> mail.misc.spam after spam proecsssing; all my incoming spam should be
> filtered, through ifile, into mail.misc.spam."

Okay, good, that's what I want.

>> It should be clearer in the documentation/Customize descriptions
>> where you need the backend prefix and where you don't.
>
> Yeah, the docs need work for sure.  I've been trying to get the
> package functionality in place first.

Everything seems to be matched against gnus-newsgroup-name, FWIW,
which does include the backend prefix; since regular expression
matching generally matches substrings, though, it's probably okay to
either have it or not.

> See if that triggers the ifile spam processing on summary exit.  You
> must have some spam-marked articles.  You can test the processing
> manually with M-: (spam-summary-prepare-exit) - that's the function
> that gets invoked at summary exit.

spam-summary-prepare-exit, eh?  That has

  ;; Only for spam groups, we expire and maybe move articles
  (when (spam-group-spam-contents-p gnus-newsgroup-name)
    (spam-mark-spam-as-expired-and-move-routine
     (gnus-parameter-spam-process-destination gnus-newsgroup-name)))

...but that seems to be the opposite behavior from what I expect, I
want spam to be shuffled into a spam group only if the current group
isn't one.  But if I understand the function: spam-marked articles are
noted by the spam backend, then spam-marked articles are refiled, then
in ham groups, the remaining articles are noted as ham by the spam
backend.

>> If I mark an article not-spam in the spam group, does it get refiled
>> to the next best group on exit?
>
> No.  You have to move it manually (but that functionality could be
> added).  I assume you mean "mark unread," since there is no not-spam
> mark.

Mark unread would work; "any ham mark" or "any not-spam mark" might
make sense too.  It sounds like this is falling into "feature request"
land, though.

>> (This is especially true for spam-stat; doing this classification
>> entirely in elisp looks like it might be a big performance win over
>> ifile, 
>
> I haven't tested the performance, but ifile definitely has better
> lexing than spam-stat according to the documentation.

It looks like ifile reads its database, processes a single message,
and writes its database; lather, rinse, repeat.  If you're dealing
with a networked filesystem, reading and writing the database once per
message is a big lose.  I've heard rumors that moving ~/.idata to
local disk improvies this.  Doing filtering inside Emacs means that
the database can live in memory until I do a save in Gnus.

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Trouble with spam.el and ifile

[David Z Maze]

Two more things:

-- 'unless' isn't the opposite of 'if' (it's the opposite of 'when',
   though).  `spam-check-ifile' now unconditionally sets the group
   name to what gets returned from ifile, regardless of the setting of
   spam-ifile-all-categories, which is wrong.  You really want the
   (if (not spam-ifile-all-categories) ...) form my original patch
   had.

-- Is there any way to specify a group as a ham group besides group
   parameters?  I want every nnml group except my spam group to be
   ham, but not groups in other backends, and my topic hierarchy mixes
   news, mail, and other things.  Setting ham on a per-group basis is
   prohibitive with several hundred nnml groups and automatic group
   generation from split methods.

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Trouble with spam.el and ifile

[Ted Zlatanov]

On Tue, 07 Jan 2003, dmaze@MIT.EDU wrote:
> -- 'unless' isn't the opposite of 'if' (it's the opposite of 'when',
> though).  `spam-check-ifile' now unconditionally sets the group name
> to what gets returned from ifile, regardless of the setting of
> spam-ifile-all-categories, which is wrong.  You really want the (if
> (not spam-ifile-all-categories) ...) form my original patch had.

I see what I did, I missed the else part of the if - I thought it was
just the 'if'.  Oops.

> -- Is there any way to specify a group as a ham group besides group
>    parameters?  I want every nnml group except my spam group to be
>    ham, but not groups in other backends, and my topic hierarchy
>    mixes news, mail, and other things.  Setting ham on a per-group
>    basis is prohibitive with several hundred nnml groups and
>    automatic group generation from split methods.

I'm not sure what other way to provide.  You can already do it  by
regex (which should be able to do an exception).  How would you
suggest handling your case?  A function hook?

By the way, I think that if you set the spam group's contents to spam,
it will override a global regex of ham for "nnml.*" - have you tried
that?

Ted

Re: Trouble with spam.el and ifile

[Ted Zlatanov]

On Tue, 07 Jan 2003, dmaze@MIT.EDU wrote:
> Everything seems to be matched against gnus-newsgroup-name, FWIW,
> which does include the backend prefix; since regular expression
> matching generally matches substrings, though, it's probably okay to
> either have it or not.

...except for spam-split-group (which, when set to "mail" for
instance, may mean "nnimap:mail" or "nnml:mail" because of the way
splits work).  Also spam-junk-mailgroups is a list of names, not
regexes.  I didn't convert it to regexes because there's already a
mechanism, through the spam-contents parameter, to specify a group is
spam through a regex.


> spam-summary-prepare-exit, eh?  That has
> 
>   ;; Only for spam groups, we expire and maybe move articles
>   (when (spam-group-spam-contents-p gnus-newsgroup-name)
>     (spam-mark-spam-as-expired-and-move-routine
>      (gnus-parameter-spam-process-destination gnus-newsgroup-name)))
> 
> ...but that seems to be the opposite behavior from what I expect, I
> want spam to be shuffled into a spam group only if the current group
> isn't one.  But if I understand the function: spam-marked articles
> are noted by the spam backend, then spam-marked articles are
> refiled, then in ham groups, the remaining articles are noted as ham
> by the spam backend.

The behavior made sense to me at the time, but I see what you mean and
my original thought was wrong.  Perhaps it should be either reversed
to apply to groups that are not spam (ham + unclassified), or *all*
groups should have their spam-marked articles processed by
spam-mark-spam-as-expired-and-move-routine.  Should I pick one or the
other, or make it yet another user option?

>>> If I mark an article not-spam in the spam group, does it get
>>> refiled to the next best group on exit?
>>
>> No.  You have to move it manually (but that functionality could be
>> added).  I assume you mean "mark unread," since there is no
>> not-spam mark.
> 
> Mark unread would work; "any ham mark" or "any not-spam mark" might
> make sense too.  It sounds like this is falling into "feature
> request" land, though.

Well, it can be added, and it makes sense as well - you don't want ham
articles in a spam group.  Maybe Yet Another User Option.

> It looks like ifile reads its database, processes a single message,
> and writes its database; lather, rinse, repeat.  If you're dealing
> with a networked filesystem, reading and writing the database once
> per message is a big lose.  

The code is much simpler when you deal with one message at a time.
That can be fixed, but take a look at the bogofilter register routine
which does multiple messages to see why it may be a little complex.
Hmm, maybe I can have the registration routine take a process as an
optional parameter.

> I've heard rumors that moving ~/.idata to local disk improvies this.

Maybe Yet Another User Option to customize the .idata file name for
spam.el use of ifile...  That's an easy one to add, and I can imagine
people might want to keep spam.el's ifile .idata and their regular
.idata separate as well.

> Doing filtering inside Emacs means that the database can live in
> memory until I do a save in Gnus.

You mean spam-stat.el?  Yes, but ifile's lexer is supposed to be
better.  I don't have performance numbers in terms of speed or spam
detection percentages, though.

Ted

Re: Trouble with spam.el and ifile

[David Z Maze]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 07 Jan 2003, dmaze@MIT.EDU wrote:
>> spam-summary-prepare-exit, eh?  That has
>> 
>>   ;; Only for spam groups, we expire and maybe move articles
>>   (when (spam-group-spam-contents-p gnus-newsgroup-name)
>>     (spam-mark-spam-as-expired-and-move-routine
>>      (gnus-parameter-spam-process-destination gnus-newsgroup-name)))
>> 
>> ...but that seems to be the opposite behavior from what I expect
>
> The behavior made sense to me at the time, but I see what you mean and
> my original thought was wrong.  Perhaps it should be either reversed
> to apply to groups that are not spam (ham + unclassified), or *all*
> groups should have their spam-marked articles processed by
> spam-mark-spam-as-expired-and-move-routine.  Should I pick one or the
> other, or make it yet another user option?

Picking one or the other is probably fine.  Reversing the sense of the
test makes sense to me.

>> -- Is there any way to specify a group as a ham group besides group
>>    parameters?  I want every nnml group except my spam group to be
>>    ham, but not groups in other backends, and my topic hierarchy
>>    mixes news, mail, and other things.  Setting ham on a per-group
>>    basis is prohibitive with several hundred nnml groups and
>>    automatic group generation from split methods.
>
> I'm not sure what other way to provide.  You can already do it  by
> regex (which should be able to do an exception).  How would you
> suggest handling your case?  A function hook?

By regex would be great, but I don't know which variable to set.  (By
regex is even better if spamness overrides hamness, since then
"nnml:.*" can be ham.)  In particular,

(defun spam-group-ham-contents-p (group)
  (if (stringp group)
      (memq 'gnus-group-spam-classification-ham
            (gnus-parameter-spam-contents group))
    nil))

doesn't do any obvious testing of group against a regex;
gnus-group-spam-contents-p at least tests if the group is a member of
spam-junk-mailgroups.  *pause*  *enlightenment*  I want to set

(setq gnus-spam-newsgroup-contents
  '(("nnml:.*"             gnus-group-spam-classification-ham)
    ("nnml:mail.misc.spam" gnus-group-spam-classification-spam)))

since it looks like the magic in gnus-define-group-parameter picks the
last matching regexp from the alist.  Probably (gnus)Filtering Spam
Using spam.el should talk about this; now that I think I have some
understanding, would you like me to write up some text?

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Trouble with spam.el and ifile

[David Z Maze]

David Z Maze <dmaze@MIT.EDU> writes:

> *enlightenment*  I want to set
>
> (setq gnus-spam-newsgroup-contents
>   '(("nnml:.*"             gnus-group-spam-classification-ham)
>     ("nnml:mail.misc.spam" gnus-group-spam-classification-spam)))
>
> since it looks like the magic in gnus-define-group-parameter picks the
> last matching regexp from the alist.

Replacing experimentation with enlightenment, this is backwards; it
takes the *first* regexp.  So I now have in my .gnus:

(setq gnus-spam-newsgroup-contents
      '(("nnml:mail.misc.spam" gnus-group-spam-classification-spam)
	("nnml:.*" gnus-group-spam-classification-ham))
      gnus-spam-process-newsgroups
      '(("nnml:.*" (gnus-group-spam-exit-processor-ifile)))
      gnus-spam-process-destinations
      '(("nnml:.*" "nnml:mail.misc.spam"))
      spam-junk-mailgroups '("mail.misc.spam")
      spam-split-group "mail.misc.spam"
      spam-use-ifile t)

And it appears to do almost everything I want, except for sorting away
spam or ham appearing in the other sort of group.

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Trouble with spam.el and ifile

[Ted Zlatanov]

On Wed, 08 Jan 2003, tzz@lifelogs.com wrote:
> The behavior made sense to me at the time, but I see what you mean
> and my original thought was wrong.  Perhaps it should be either
> reversed to apply to groups that are not spam (ham + unclassified),
> or *all* groups should have their spam-marked articles processed by
> spam-mark-spam-as-expired-and-move-routine.  Should I pick one or
> the other, or make it yet another user option?

OK, done.  The logic is fixed and it respects
spam-move-spam-nonspam-groups-only now to determine if all groups or
nonspam groups only get their spam moved to the
spam-process-destination parameter.

>>>> If I mark an article not-spam in the spam group, does it get
>>>> refiled to the next best group on exit?
>>>
>>> No.  You have to move it manually (but that functionality could be
>>> added).  I assume you mean "mark unread," since there is no
>>> not-spam mark.
>> 
>> Mark unread would work; "any ham mark" or "any not-spam mark" might
>> make sense too.  It sounds like this is falling into "feature
>> request" land, though.

I added a spam-ham-move-routine that does this, based on the
ham-process-destination group parameter.

> Maybe Yet Another User Option to customize the .idata file name for
> spam.el use of ifile...  That's an easy one to add, and I can
> imagine people might want to keep spam.el's ifile .idata and their
> regular .idata separate as well.

Done, the variable spam-ifile-database-path is respected if set.

Ted

Re: Trouble with spam.el and ifile

[Ted Zlatanov]

On Wed, 08 Jan 2003, dmaze@MIT.EDU wrote:
> By regex would be great, but I don't know which variable to set.
> (By regex is even better if spamness overrides hamness, since then
> "nnml:.*" can be ham.)  In particular,
> 
> (defun spam-group-ham-contents-p (group)
>   (if (stringp group)
>       (memq 'gnus-group-spam-classification-ham
>             (gnus-parameter-spam-contents group))
>     nil))
> 
> doesn't do any obvious testing of group against a regex;
> gnus-group-spam-contents-p at least tests if the group is a member
> of spam-junk-mailgroups.  *pause* *enlightenment* I want to set
> 
> (setq gnus-spam-newsgroup-contents
>   '(("nnml:.*"             gnus-group-spam-classification-ham)
>     ("nnml:mail.misc.spam" gnus-group-spam-classification-spam)))
> 
> since it looks like the magic in gnus-define-group-parameter picks
> the last matching regexp from the alist.  

I just use the gnus-define-group-parameter magic, it seems to do the
right thing.  But did you try setting the group spam-contents
parameter with `G c' instead of gnus-spam-newsgroup-contents?  That
should override the global setting.

> Probably (gnus)Filtering Spam Using spam.el should talk about this;
> now that I think I have some understanding, would you like me to
> write up some text?

Yes, if you could look it over for inaccuracies, especially
considering the new ifile modifications and the new behavior with
spam- and ham-moving, I would greatly appreciate it.

Ted