CRM114 Mailfilter and spam.el

CRM114 Mailfilter and spam.el

[Adam Sjøgren]

  Hi.


So, I've begun to take a stab at switcing from spam-stat.el to CRM114
Mailfilter and spam.el.

I followed the instructions in the documentation of spam.el about how
to add another back end ("Extending the Spam ELisp package"), basing
my additions on the bogofilter stuff, as I guess bogofilter has a
similar interface to CRM114 Mailfilter.

I think I've got the spam.el <-> CRM114 Mailfilter part working. I can
do spam-crm114-score and get the number that mailfilter.crm adds to
the header X-CRM114-Status, and splitting puts email in my spam-group
- learning spam seems to work.

Now I just need to figure out how to configure spam.el to call
mailfilter.crm --learnspam/--learnnonspam at the *appropriate* moments
only.

Boxing with adding crm114 to spam.el has made me a little, uhm, less
motivated to experiment further (I'm bad a elisp and it takes a while
to figure out _how_ to debug that for instance '--fileprefix=
blahblah' is very much different from '--fileprefix=blahblah' to
mailfilter.crm...), so here goes my question:

The documentation for CRM114 underlines heavily and repeatedly that
one should *only* train on failures. So I want to only do
mailfilter.crm --learnspam on spam that was classified as ham by
CRM114, and I want to only do mailfilter.crm --learnnonspam on ham
that was classified by CRM114. No learning in any other case.

Also, I don't really want special "training"-groups.

Ideally I'd like to just have a workflow like this:

 * Email comes in, and is split with spam-split. Spam goes into
   "nnml:spam", ham goes through my usual, other splitrules (bbdb and
   stuff).

 * If a spam is misclassified as ham, I would like to mark that
   message in some way, so that it is moved to nnml:spam and
   mailfilter.crm is called with --learnspam on it.

 * If a ham is misclassified as spam, and therefore is split into
   nnml:spam, I would like to move it to the correct group and have
   mailfilter.crm called on it with --learnnonspam.

 * I would like spam.el *never* to mark *any* message as expirable.

That's it. But how?


  Best regards,

-- 
 "Mr. Cotton's... parrot. Same question."                     Adam Sjøgren
                                                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Kai Grossjohann]

asjo@koldfront.dk (Adam Sjøgren) writes:

>  * If a spam is misclassified as ham, I would like to mark that
>    message in some way, so that it is moved to nnml:spam and
>    mailfilter.crm is called with --learnspam on it.

I think you want to set spam-process-ham-in-spam-groups for this one.

>  * If a ham is misclassified as spam, and therefore is split into
>    nnml:spam, I would like to move it to the correct group and have
>    mailfilter.crm called on it with --learnnonspam.

But I can't find spam-process-spam-in-ham-groups.  Hm.

Kai

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On Fri, 02 Apr 2004 08:22:25 +0200, Kai wrote:

> asjo@koldfront.dk (Adam Sjøgren) writes:
>> * If a spam is misclassified as ham, I would like to mark that
>> message in some way, so that it is moved to nnml:spam and
>> mailfilter.crm is called with --learnspam on it.

> I think you want to set spam-process-ham-in-spam-groups for this
> one.

>> * If a ham is misclassified as spam, and therefore is split into
>> nnml:spam, I would like to move it to the correct group and have
>> mailfilter.crm called on it with --learnnonspam.

> But I can't find spam-process-spam-in-ham-groups. Hm.

Okay, I got it almost working (by reading the documentation one more
time).

My understanding so far is this:

 * I want all my groups to be unclassified - because I only want to
   train on errors (marking them by hand). I never want to train on
   properly classified ham nor spam.

 * On the topic that contains all my normal email-groups, I've set
   these parameters:

    ((spam-process
      '(spam spam-use-crm114))
     (spam-process-destination . "nnml:spam"))

   This means that if spam ends up in one of my normal email-groups, I
   mark it with M-d ($), and it crm114 is called to learn it as spam,
   and it is moved to nnml:spam.

   Success! This is what I want, and it works.

 * On my nnml:spam-group (where spam is split to, and moved to as per
   above) I've set these parameters:

    ((spam-process
      '(ham spam-use-crm114))
     (ham-process-destination . "nnml:normal")
     (spam-process-destination . "nnml:spam")
     (ham-marks
      (gnus-ticked-mark))

   What I want to achieve with this is: When a ham ends up in nnml:spam,
   I tick it (!). When I leave the group, crm114 should learn it as
   ham, and the article should be moved to nnml:normal[1].

   This only works partially: The ticked articles are processed by
   crm114 to be learned as ham (fine), but the articles are not moved
   to nnml:normal - they stay in nnml:spam.

   (I guess the spam-process-destination should be removed, as it
    isn't used).

 * My global variables are set like this:

    (setq
     ; Only process each article once:
     spam-log-to-registry t
     ; Only process spam in nonspam groups (i.e. Train On Error only):
     spam-move-spam-nonspam-groups-only t
     ; All my groups are unclassified, I should remove this one:
     spam-process-ham-in-spam-groups t
     ; Process ham in ham and unclassified groups:
     spam-process-ham-in-nonham-groups t
     ; I want ham to be unread when moved out (it never is, but that's
     ; what I would like if it did move):
     spam-mark-ham-unread-before-move-from-spam-group t
     ; CRM114 Mailfilter (I will submit a patch):
     spam-use-crm114 t
     spam-crm114-path "/home/asjo/crm114.pl" ; (for debugging)
     spam-crm114-database-directory "--fileprefix=/home/asjo/.crm114/")

   And I use (: spam-split) in my fancy split-rules.

This is very cool - it's beginning to make sense to me _and_ it almost
works :-)

I've only got the "ham-marked articles in nnml:spam should be moved as
well as processed"-problem left.


  Best regards,

    Adam

[1]: If the articles I mark as ham in nnml:spam could be 1) learned as
     ham by crm114 and then 2) respooled, that would be even better
     than 2) moved to nnml:normal.

-- 
 "Mr. Cotton's... parrot. Same question."                     Adam Sjøgren
                                                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On Fri, 02 Apr 2004 08:22:25 +0200, Kai wrote:

> asjo@koldfront.dk (Adam Sjøgren) writes:
>> * If a spam is misclassified as ham, I would like to mark that
>> message in some way, so that it is moved to nnml:spam and
>> mailfilter.crm is called with --learnspam on it.

> I think you want to set spam-process-ham-in-spam-groups for this
> one.

>> * If a ham is misclassified as spam, and therefore is split into
>> nnml:spam, I would like to move it to the correct group and have
>> mailfilter.crm called on it with --learnnonspam.

> But I can't find spam-process-spam-in-ham-groups. Hm.

Okay, I got it almost working (by reading the documentation one more
time).

My understanding so far is this:

 * I want all my groups to be unclassified - because I only want to
   train on errors (marking them by hand). I never want to train on
   properly classified ham nor spam.

 * On the topic that contains all my normal email-groups, I've set
   these parameters:

    ((spam-process
      '(spam spam-use-crm114))
     (spam-process-destination . "nnml:spam"))

   This means that if spam ends up in one of my normal email-groups, I
   mark it with M-d ($), and it crm114 is called to learn it as spam,
   and it is moved to nnml:spam.

   Success! This is what I want, and it works.

 * On my nnml:spam-group (where spam is split to, and moved to as per
   above) I've set these parameters:

    ((spam-process
      '(ham spam-use-crm114))
     (ham-process-destination . "nnml:normal")
     (spam-process-destination . "nnml:spam")
     (ham-marks
      (gnus-ticked-mark))

   What I want to achieve with this is: When a ham ends up in nnml:spam,
   I tick it (!). When I leave the group, crm114 should learn it as
   ham, and the article should be moved to nnml:normal[1].

   This only works partially: The ticked articles are processed by
   crm114 to be learned as ham (fine), but the articles are not moved
   to nnml:normal - they stay in nnml:spam.

   (I guess the spam-process-destination should be removed, as it
    isn't used).

 * My global variables are set like this:

    (setq
     ; Only process each article once:
     spam-log-to-registry t
     ; Only process spam in nonspam groups (i.e. Train On Error only):
     spam-move-spam-nonspam-groups-only t
     ; All my groups are unclassified, I should remove this one:
     spam-process-ham-in-spam-groups t
     ; Process ham in ham and unclassified groups:
     spam-process-ham-in-nonham-groups t
     ; I want ham to be unread when moved out (it never is, but that's
     ; what I would like if it did move):
     spam-mark-ham-unread-before-move-from-spam-group t
     ; CRM114 Mailfilter (I will submit a patch):
     spam-use-crm114 t
     spam-crm114-path "/home/asjo/crm114.pl" ; (for debugging)
     spam-crm114-database-directory "--fileprefix=/home/asjo/.crm114/")

   And I use (: spam-split) in my fancy split-rules.

This is very cool - it's beginning to make sense to me _and_ it almost
works :-)

I've only got the "ham-marked articles in nnml:spam should be moved as
well as processed"-problem left.


  Best regards,

    Adam

[1]: If the articles I mark as ham in nnml:spam could be 1) learned as
     ham by crm114 and then 2) respooled, that would be even better
     than 2) moved to nnml:normal.

-- 
 "Mr. Cotton's... parrot. Same question."                     Adam Sjøgren
                                                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Jody Klymak]

Hello,

>>  * If a ham is misclassified as spam, and therefore is split into
>>    nnml:spam, I would like to move it to the correct group and have
>>    mailfilter.crm called on it with --learnnonspam.
>
> But I can't find spam-process-spam-in-ham-groups.  Hm.

I think the equivalent functionality is in:

--8<---------------cut here---------------start------------->8---
   If a spam is found in any group (this can be changed to only non-spam
groups with `spam-move-spam-nonspam-groups-only'), it is processed by
the active `spam-processors'
--8<---------------cut here---------------end--------------->8---

Cheers,  Jody
-- 
Jody Klymak      http://opg1.ucsd.edu/~jklymak/
mailto:jklymak@ucsd.edu   

Re: CRM114 Mailfilter and spam.el

[Ted Zlatanov]

On Thu, 01 Apr 2004, asjo@koldfront.dk wrote:

> So, I've begun to take a stab at switcing from spam-stat.el to
> CRM114 Mailfilter and spam.el.

Great!  Just so you know, I plan to add a better way to add backends
to spam.el; something like (spam-add-statistical-backend) for
instance.  It's not going to be ready for a while, but it's been on
my mind.

> Now I just need to figure out how to configure spam.el to call
> mailfilter.crm --learnspam/--learnnonspam at the *appropriate*
> moments only.

Yes, this is very important and I've been trying to get the
gnus-registry to cooperate with spam.el on this.

>  * Email comes in, and is split with spam-split. Spam goes into
>    "nnml:spam", ham goes through my usual, other splitrules (bbdb
>    and stuff).

OK.

>  * If a spam is misclassified as ham, I would like to mark that
>    message in some way, so that it is moved to nnml:spam and
>    mailfilter.crm is called with --learnspam on it.

Set the spam-mark on the message.  When you exit the group, the spam
message will be moved to the group named by spam-process-destination
(multiples are OK).  Also, the exit spam processor will be invoked on
the spam message - so set your exit spam processor to crm114.

>  * If a ham is misclassified as spam, and therefore is split into
>    nnml:spam, I would like to move it to the correct group and have
>    mailfilter.crm called on it with --learnnonspam.

Set a ham-mark (usually the tick mark) on the message.  The ham
message, when you quit the group, will be moved to the
ham-process-destination (which can be multiple groups) and processed
with the ham exit processor.

If you set spam-mark-ham-unread-before-move-from-spam-group, the ham
will also be marked unread before it's moved out of the spam group.

>  * I would like spam.el *never* to mark *any* message as expirable.

This is hardcoded right now in
spam-mark-spam-as-expired-and-move-routine:

	  (gnus-summary-mark-article article gnus-expirable-mark)

but it could be changed.  Why would you NOT want to expire spam?

Ted

Re: CRM114 Mailfilter and spam.el

[Ted Zlatanov]

On Fri, 02 Apr 2004, kai@emptyhost.emptydomain.de wrote:

> asjo@koldfront.dk (Adam Sjøgren) writes:
> 
>>  * If a spam is misclassified as ham, I would like to mark that
>>    message in some way, so that it is moved to nnml:spam and
>>    mailfilter.crm is called with --learnspam on it.
> 
> I think you want to set spam-process-ham-in-spam-groups for this
> one.

Yes, sorry I forgot to mention that!

>>  * If a ham is misclassified as spam, and therefore is split into
>>    nnml:spam, I would like to move it to the correct group and have
>>    mailfilter.crm called on it with --learnnonspam.
> 
> But I can't find spam-process-spam-in-ham-groups.  Hm.

I assumed when writing that part of spam.el that no one likes spam in
their ham, whereas some people like ham in the spam.

Ted (I am :)

Re: CRM114 Mailfilter and spam.el

[Ted Zlatanov]

On Fri, 02 Apr 2004, asjo@koldfront.dk wrote:

> This is very cool - it's beginning to make sense to me _and_ it
> almost works :-)

I'm very glad.  CRM114 support is sure to be beneficial to other Gnus
users as well!

> I've only got the "ham-marked articles in nnml:spam should be moved
> as well as processed"-problem left.

See spam-ham-copy-or-move-routine for the exact mechanics of this.
I'm pretty sure something is misconfigured, because I move ham out of
my spam groups all the time.  But I did not see something obviously
wrong in your setup.

> [1]: If the articles I mark as ham in nnml:spam could be 1) learned
>      as ham by crm114 and then 2) respooled, that would be even
>      better than 2) moved to nnml:normal.

Try putting 'respool in the ham-process-destination and see if it
works for you.  spam-ham-copy-or-move-routine has the code used - it's
not a true respooling, so you may not be happy with it, but it tries
to simulate the respooling.

Ted

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On Fri, 02 Apr 2004 12:06:04 -0500, Ted wrote:

> On Thu, 01 Apr 2004, asjo@koldfront.dk wrote:
>> So, I've begun to take a stab at switcing from spam-stat.el to
>> CRM114 Mailfilter and spam.el.

> Great! Just so you know, I plan to add a better way to add backends
> to spam.el; something like (spam-add-statistical-backend) for
> instance. It's not going to be ready for a while, but it's been on
> my mind.

Good idea - I've basically copied/search/replaced the bogofilter-parts
for now.

>> * If a ham is misclassified as spam, and therefore is split into
>> nnml:spam, I would like to move it to the correct group and have
>> mailfilter.crm called on it with --learnnonspam.

> Set a ham-mark (usually the tick mark) on the message. The ham
> message, when you quit the group, will be moved to the
> ham-process-destination (which can be multiple groups) and processed
> with the ham exit processor.

This I can't get working (see my previous email with my complete setup
described).

>> * I would like spam.el *never* to mark *any* message as expirable.

> This is hardcoded right now in
> spam-mark-spam-as-expired-and-move-routine:

> 	  (gnus-summary-mark-article article gnus-expirable-mark)

> but it could be changed. Why would you NOT want to expire spam?

(I worked around it by having my nnml:spam be unclassified :-))

I want to keep all email I get - if nothing else for statistical
purposes. (And if some filter turns out to be really, really good when
trained with a billion spam-messages, I'll be ready!)


  Thanks for the reply, and spam.el :-),

    Adam

-- 
 "Mr. Cotton's... parrot. Same question."                     Adam Sjøgren
                                                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Ted Zlatanov]

On Fri, 02 Apr 2004, asjo@koldfront.dk wrote:

On Fri, 02 Apr 2004 12:06:04 -0500, Ted wrote:

>> Set a ham-mark (usually the tick mark) on the message. The ham
>> message, when you quit the group, will be moved to the
>> ham-process-destination (which can be multiple groups) and
>> processed with the ham exit processor.
> 
> This I can't get working (see my previous email with my complete
> setup described).

Like I said, it looks correct to me, can you trace the
spam-ham-copy-or-move-routine function?

>> Why would you NOT want to expire spam?
> 
> I want to keep all email I get - if nothing else for statistical
> purposes. (And if some filter turns out to be really, really good
> when trained with a billion spam-messages, I'll be ready!)

Then you can just move all spam to a spam group and set it to never
run the expiry process, no?  If you insist on keeping the spam in the
original group I guess I can make that mark configurable...  It just
seems like very few people would find it useful, and spam.el has too
many options already.

I find little value in keeping old spam, personally.  Old ham is very
valuable, but old spam tends to lose value with time.

Ted

Re: CRM114 Mailfilter and spam.el

[Kai Grossjohann]

asjo@koldfront.dk (Adam Sjøgren) writes:

>      ; All my groups are unclassified, I should remove this one:

It seems nnml:spam is classificied as spam?

Kai

Re: CRM114 Mailfilter and spam.el

[Kai Grossjohann]

asjo@koldfront.dk (Adam Sjøgren) writes:

>  * I want all my groups to be unclassified - because I only want to
>    train on errors (marking them by hand). I never want to train on
>    properly classified ham nor spam.

I wonder what does the classification of a group have to do with
training?

Kai

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On Sat, 03 Apr 2004 15:29:03 +0200, Kai wrote:

> asjo@koldfront.dk (Adam Sjøgren) writes:
>> * I want all my groups to be unclassified - because I only want to
>> train on errors (marking them by hand). I never want to train on
>> properly classified ham nor spam.

> I wonder what does the classification of a group have to do with
> training?

A group classified as spam automatically gets unseen or unread
articles marked as spam:

  Spam groups have the additional characteristic that, when entered,
  any unseen or unread articles (depending on the
  `spam-mark-only-unseen-as-spam' variable) will be marked as spam. 
  Thus, mail split into a spam group gets automatically marked as spam
  when you enter the group.

   - from "Spam ELisp Package Sequence of Events".

I only want to Train On Error - so I do not want spam that has been
recognized correctly as spam to be processed as spam (passed to crm114
for training).

So, if my spam-group as classified as spam, I have to remove all those
automatically set marks by hand.


  Best regards,

   Adam

-- 
 "Mr. Cotton's... parrot. Same question."                     Adam Sjøgren
                                                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On Sat, 03 Apr 2004 15:28:02 +0200, Kai wrote:

> asjo@koldfront.dk (Adam Sjøgren) writes:
>> ; All my groups are unclassified, I should remove this one:

> It seems nnml:spam is classificied as spam?

No, I haven't set a 'spam-content' parameter on the group, and I
haven't set gnus-spam-newsgroup-contents nor spam-junk-mailgroups
either.

Hmm: It seems that spam-junk-mailgroups defaults to: ("spam"
"mail.junk" "poste.pourriel"), yet my nnml:spam group is unclassified
(which is what I want, but not what I expect from reading the docs on
classification:

     Gnus learns from the spam you get. You have to collect your spam
  in one or more spam groups, and set or customize the variable
  `spam-junk-mailgroups' as appropriate. You can also declare groups
  to contain spam by setting their group parameter `spam-contents' to
  `gnus-group-spam-classification-spam', or by customizing the
  corresponding variable `gnus-spam-newsgroup-contents'. The
  `spam-contents' group parameter and the
  `gnus-spam-newsgroup-contents' variable can also be used to declare
  groups as _ham_ groups if you set their classification to
  `gnus-group-spam-classification-ham'. If groups are not classified
  by means of `spam-junk-mailgroups', `spam-contents', or
  `gnus-spam-newsgroup-contents', they are considered _unclassified_. 
  All groups are unclassified by default.

   - from "Spam ELisp Package Global Variables".)

My nnml:spam group really is unclassified:

 (spam-group-spam-contents-p "nnml:spam") => nil

but:

 (spam-group-spam-contents-p "spam") => ("spam" "mail.junk" "poste.pourriel")

So maybe that explains why my nnml:spam group is unclassied even
though I haven't set spam-junk-mailgroups to nil?


  Best regards,

-- 
 "Mr. Cotton's... parrot. Same question."                     Adam Sjøgren
                                                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On Fri, 02 Apr 2004 12:44:17 -0500, Ted wrote:

> On Fri, 02 Apr 2004, asjo@koldfront.dk wrote:

>> I want to keep all email I get - if nothing else for statistical
>> purposes. (And if some filter turns out to be really, really good
>> when trained with a billion spam-messages, I'll be ready!)

> Then you can just move all spam to a spam group and set it to never
> run the expiry process, no?

I could, but I'm weary of marking things for deletion and then
overriding the deletion with one setting.

(When I mess up that one setting later, I lose stuff - I've done
similar things by accidentally expiring newsspools in INN2 too many
times).

> If you insist on keeping the spam in the original group I guess I
> can make that mark configurable... It just seems like very few
> people would find it useful, and spam.el has too many options
> already.

I agree. Don't make it configurable on my account.


  Best regards,

-- 
 "Mr. Cotton's... parrot. Same question."                     Adam Sjøgren
                                                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On Sat, 03 Apr 2004 16:22:11 +0200, Adam wrote:

> I could, but I'm weary of marking things for deletion and then
> overriding the deletion with one setting.

No, I'm not. I'm _wary_.


 *blush*,

-- 
 "Mr. Cotton's... parrot. Same question."                     Adam Sjøgren
                                                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On Fri, 02 Apr 2004 12:14:44 -0500, Ted wrote:

>> I've only got the "ham-marked articles in nnml:spam should be moved
>> as well as processed"-problem left.

> See spam-ham-copy-or-move-routine for the exact mechanics of this. 

I just tried doing "debug-on-entry" on that fun; ticked a ham in
nnml:spam and exited. The function wasn't called.

I will try to investigate a little more...


  Best regards,

-- 
 "Mr. Cotton's... parrot. Same question."                     Adam Sjøgren
                                                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On Sat, 03 Apr 2004 23:10:46 +0200, Adam wrote:

> On Fri, 02 Apr 2004 12:14:44 -0500, Ted wrote:

>>> I've only got the "ham-marked articles in nnml:spam should be
>>> moved as well as processed"-problem left.

>> See spam-ham-copy-or-move-routine for the exact mechanics of this.

Okay, I found out why. In spam-summary-prepare-exit, almost at the
bottom, spam-ham-move-routine is called if and only if the current
group is classified as a spam-group.

This seems to contradict the documentation, where it says:

     So, when you exit a group, the `spam-processors' are applied, if any
  are set, and the processed mail is moved to the
  `ham-process-destination' or the `spam-process-destination' depending
  on the article's classification.  If the `ham-process-destination' or
  the `spam-process-destination', whichever is appropriate, are `nil',
  the article is left in the current group.

   - Spam ELisp Package Sequence of Events

The documentation doesn't mention the codes requirement that the group
be classified as spam for ham to be moved.

I think the code should change, what do you think?


  Best regards,

-- 
 "Mr. Cotton's... parrot. Same question."                     Adam Sjøgren
                                                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On Sat, 03 Apr 2004 23:19:22 +0200, Adam wrote:

> On Sat, 03 Apr 2004 23:10:46 +0200, Adam wrote:
>> On Fri, 02 Apr 2004 12:14:44 -0500, Ted wrote:

>>>> I've only got the "ham-marked articles in nnml:spam should be
>>>> moved as well as processed"-problem left.

>>> See spam-ham-copy-or-move-routine for the exact mechanics of this.

I'm stepping through it with edebug-defun, and I don't understand
quite how it all fits together. It looks like "todo" is first set to a
list of the article numbers to process, but then set to nil:

 (defun spam-ham-copy-or-move-routine (copy groups)
   (gnus-summary-kill-process-mark)
   (let ((todo (spam-list-articles gnus-newsgroup-articles 'ham))
         (backend-supports-deletions
          (gnus-check-backend-function
           'request-move-article gnus-newsgroup-name))
         (respool-method (gnus-find-method-for-group gnus-newsgroup-name))
         article mark todo deletep respool)

When I step through it "todo" is (43567) at first, but if I look at
todo later on, it's nil.

I'm wondering about the last quoted line above "article mark todo
deletep respool" - maybe todo shouldn't be there? Please advise.

(My lack of elisp grasp shows again :-))


  Best regards,

-- 
 "Mr. Cotton's... parrot. Same question."                     Adam Sjøgren
                                                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

On Sat, 03 Apr 2004 23:48:08 +0200, Adam wrote:

> I'm wondering about the last quoted line above "article mark todo
> deletep respool" - maybe todo shouldn't be there? Please advise.

> (My lack of elisp grasp shows again :-))

It works for me with the attached minimal patch.


  Best regards,

     Adam

-- 
 "Mr. Cotton's... parrot. Same question."                     Adam Sjøgren
                                                         asjo@koldfront.dk

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: move-ham-from-nonham-groups.patch --]
[-- Type: text/x-patch, Size: 981 bytes --]

--- spam.el     2 Apr 2004 19:09:16 -0000       7.27
+++ spam.el     3 Apr 2004 21:54:27 -0000
@@ -776,8 +776,8 @@
       (spam-ham-copy-routine
        (gnus-parameter-ham-process-destination gnus-newsgroup-name)))
 
-    ;; now move all ham articles out of spam groups
-    (when (spam-group-spam-contents-p gnus-newsgroup-name)
+    ;; now move all ham articles out of non-ham groups
+    (when (not (spam-group-ham-contents-p gnus-newsgroup-name))
       (gnus-message 5 "Moving ham messages from spam group")
       (spam-ham-move-routine
        (gnus-parameter-ham-process-destination gnus-newsgroup-name))))
@@ -859,7 +859,7 @@
         (gnus-check-backend-function
          'request-move-article gnus-newsgroup-name))
        (respool-method (gnus-find-method-for-group gnus-newsgroup-name))
-       article mark todo deletep respool)
+       article mark deletep respool)
 
     (when (member 'respool groups)
       (setq respool t)                 ; boolean for later

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

[-- Attachment #1: Type: text/plain, Size: 1048 bytes --]

On Thu, 01 Apr 2004 22:54:06 +0200, Adam wrote:

> So, I've begun to take a stab at switcing from spam-stat.el to
> CRM114 Mailfilter and spam.el.

Attached is a patch for spam.el that adds support for CRM114
Mailfilter (until the general external statistical processor interface
is designed/implemented).

It is basically the bogofilter stuff copied, search/replaced and
modified slightly.

(Included is also the two small changes in the patch in
 <87oeq8g1up.fsf@virgil.koldfront.dk>).

To use it, I set this in my .gnus:

 spam-use-crm114 t
 spam-crm114-path "/usr/share/crm114/mailfilter.crm"
 spam-crm114-database-directory "--fileprefix=/home/asjo/.crm114/"

As you can see from the comments, I couldn't figure out how to
automatically add '--fileprefix=' to the database directory when set,
so I took the easy, ugly way out.


  Best regards,

    Adam

-- 
 "Mr. Cotton's... parrot. Same question."                     Adam Sjøgren
                                                         asjo@koldfront.dk

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: crm114-mailfilter.patch --]
[-- Type: text/x-patch, Size: 14514 bytes --]

*** spam.el	2 Apr 2004 19:09:16 -0000	7.27
--- spam.el	3 Apr 2004 23:55:02 -0000
***************
*** 263,268 ****
--- 263,273 ----
    :type 'boolean
    :group 'spam)
  
+ (defcustom spam-use-crm114 nil
+   "Whether CRM114 Mailfilter should be used by `spam-split'."
+   :type 'boolean
+   :group 'spam)
+ 
  (defcustom spam-install-hooks (or
  			       spam-use-dig
  			       spam-use-gmane-xref
***************
*** 281,287 ****
  			       spam-use-BBDB-exclusive
  			       spam-use-ifile
  			       spam-use-stat
! 			       spam-use-spamoracle)
    "Whether the spam hooks should be installed.
  Default to t if one of the spam-use-* variables is set."
    :group 'spam
--- 286,293 ----
  			       spam-use-BBDB-exclusive
  			       spam-use-ifile
  			       spam-use-stat
! 			       spam-use-spamoracle
!                                spam-use-crm114)
    "Whether the spam hooks should be installed.
  Default to t if one of the spam-use-* variables is set."
    :group 'spam
***************
*** 515,520 ****
--- 521,573 ----
    :type 'string
    :group 'spam-spamassassin)
  
+ (defgroup spam-crm114 nil
+   "Spam CRM114 Mailfilter configuration."
+   :group 'spam)
+ 
+ (defcustom spam-crm114-path (executable-find "mailfilter.crm")
+   "File path of the CRM114 Mailfilter executable program."
+   :type '(choice (file :tag "Location of CRM114 Mailfilter")
+ 		 (const :tag "CRM114 Mailfilter is not installed"))
+   :group 'spam-crm114)
+ 
+ (defcustom spam-crm114-header "X-CRM114-Status"
+   "The header that CRM114 Mailfilter inserts in messages."
+   :type 'string
+   :group 'spam-crm114)
+ 
+ (defcustom spam-crm114-spam-switch "--learnspam"
+   "The switch that CRM114 Mailfilter uses to register spam messages."
+   :type 'string
+   :group 'spam-crm114)
+ 
+ (defcustom spam-crm114-ham-switch "--learnnonspam"
+   "The switch that CRM114 Mailfilter uses to register ham messages."
+   :type 'string
+   :group 'spam-crm114)
+ 
+ (defcustom spam-crm114-spam-strong-switch "--UNKNOWN"
+   "The switch that CRM114 Mailfilter uses to unregister ham messages."
+   :type 'string
+   :group 'spam-crm114)
+ 
+ (defcustom spam-crm114-ham-strong-switch "--UNKNOWN"
+   "The switch that CRM114 Mailfilter uses to unregister spam messages."
+   :type 'string
+   :group 'spam-crm114)
+ 
+ (defcustom spam-crm114-positive-spam-header "^SPAM"
+   "The regex on `spam-crm114-header' for positive spam identification."
+   :type 'regexp
+   :group 'spam-crm114)
+ 
+ (defcustom spam-crm114-database-directory nil
+   "Directory path of the CRM114 Mailfilter databases."
+   :type '(choice (directory
+ 		  :tag "Location of the CRM114 Mailfilter database directory")
+ 		 (const :tag "Use the default"))
+   :group 'spam-crm114)
+ 
  ;;; Key bindings for spam control.
  
  (gnus-define-keys gnus-summary-mode-map
***************
*** 605,610 ****
--- 658,664 ----
      (gnus-group-spam-exit-processor-stat         spam spam-use-stat)
      (gnus-group-spam-exit-processor-spamoracle   spam spam-use-spamoracle)
      (gnus-group-spam-exit-processor-spamassassin spam spam-use-spamassassin)
+     (gnus-group-spam-exit-processor-crm114       spam spam-use-crm114)
      (gnus-group-ham-exit-processor-ifile         ham spam-use-ifile)
      (gnus-group-ham-exit-processor-bogofilter    ham spam-use-bogofilter)
      (gnus-group-ham-exit-processor-stat          ham spam-use-stat)
***************
*** 612,618 ****
      (gnus-group-ham-exit-processor-BBDB          ham spam-use-BBDB)
      (gnus-group-ham-exit-processor-copy          ham spam-use-ham-copy)
      (gnus-group-ham-exit-processor-spamassassin  ham spam-use-spamassassin)
!     (gnus-group-ham-exit-processor-spamoracle    ham spam-use-spamoracle))
    "The `spam-list-of-processors' list.
  This list contains pairs associating a ham/spam exit processor
  variable with a classification and a spam-use-* variable.")
--- 666,673 ----
      (gnus-group-ham-exit-processor-BBDB          ham spam-use-BBDB)
      (gnus-group-ham-exit-processor-copy          ham spam-use-ham-copy)
      (gnus-group-ham-exit-processor-spamassassin  ham spam-use-spamassassin)
!     (gnus-group-ham-exit-processor-spamoracle    ham spam-use-spamoracle)
!     (gnus-group-ham-exit-processor-crm114        ham spam-use-crm114))
    "The `spam-list-of-processors' list.
  This list contains pairs associating a ham/spam exit processor
  variable with a classification and a spam-use-* variable.")
***************
*** 657,662 ****
--- 712,720 ----
  (defun spam-group-spam-processor-spamoracle-p (group)
    (spam-group-processor-p group 'gnus-group-spam-exit-processor-spamoracle))
  
+ (defun spam-group-spam-processor-crm114-p (group)
+   (spam-group-processor-p group 'gnus-group-spam-exit-processor-crm114))
+ 
  (defun spam-group-ham-processor-bogofilter-p (group)
    (spam-group-processor-p group 'gnus-group-ham-exit-processor-bogofilter))
  
***************
*** 678,683 ****
--- 736,744 ----
  (defun spam-group-ham-processor-spamoracle-p (group)
    (spam-group-processor-p group 'gnus-group-ham-exit-processor-spamoracle))
  
+ (defun spam-group-ham-processor-crm114-p (group)
+   (spam-group-processor-p group 'gnus-group-ham-exit-processor-crm114))
+ 
  (defun spam-report-articles-gmane (n)
    "Report the current message as spam.
  Respects the process/prefix convention."
***************
*** 686,691 ****
--- 747,753 ----
      (gnus-summary-remove-process-mark article)
      (spam-report-gmane article)))
  
+ ; asjo: Perhaps add spam-crm114-score here? How?
  (defun spam-generic-score ()
    (interactive)
    "Invoke whatever scoring method we can."
***************
*** 718,724 ****
  	     (new-articles (spam-list-articles
  			    gnus-newsgroup-articles
  			    classification))
! 	     (changed-articles (spam-set-difference new-articles old-articles)))
  	;; now that we have the changed articles, we go through the processors
  	(dolist (processor-param spam-list-of-processors)
  	  (let ((processor (nth 0 processor-param))
--- 780,786 ----
  	     (new-articles (spam-list-articles
  			    gnus-newsgroup-articles
  			    classification))
! 	     (changed-articles (gnus-set-difference new-articles old-articles)))
  	;; now that we have the changed articles, we go through the processors
  	(dolist (processor-param spam-list-of-processors)
  	  (let ((processor (nth 0 processor-param))
***************
*** 776,783 ****
        (spam-ham-copy-routine
         (gnus-parameter-ham-process-destination gnus-newsgroup-name)))
  
!     ;; now move all ham articles out of spam groups
!     (when (spam-group-spam-contents-p gnus-newsgroup-name)
        (gnus-message 5 "Moving ham messages from spam group")
        (spam-ham-move-routine
         (gnus-parameter-ham-process-destination gnus-newsgroup-name))))
--- 838,845 ----
        (spam-ham-copy-routine
         (gnus-parameter-ham-process-destination gnus-newsgroup-name)))
  
!     ;; now move all ham articles out of non-ham groups
!     (when (not (spam-group-ham-contents-p gnus-newsgroup-name))
        (gnus-message 5 "Moving ham messages from spam group")
        (spam-ham-move-routine
         (gnus-parameter-ham-process-destination gnus-newsgroup-name))))
***************
*** 785,804 ****
    (setq spam-old-ham-articles nil)
    (setq spam-old-spam-articles nil))
  
- (defun spam-set-difference (list1 list2)
-   "Return a set difference of LIST1 and LIST2.  
- When either list is nil, the other is returned."
-   (if (and list1 list2)
-       ;; we have two non-nil lists
-       (progn
- 	(dolist (item (append list1 list2))
- 	  (when (and (memq item list1) (memq item list2))
- 	    (setq list1 (delq item list1))
- 	    (setq list2 (delq item list2))))
- 	(append list1 list2))
-     ;; if either of the lists was nil, return the other one
-     (if list1 list1 list2)))
- 
  (defun spam-mark-junk-as-spam-routine ()
    ;; check the global list of group names spam-junk-mailgroups and the
    ;; group parameters
--- 847,852 ----
***************
*** 859,865 ****
  	 (gnus-check-backend-function
  	  'request-move-article gnus-newsgroup-name))
  	(respool-method (gnus-find-method-for-group gnus-newsgroup-name))
! 	article mark todo deletep respool)
  
      (when (member 'respool groups)
        (setq respool t)			; boolean for later
--- 907,913 ----
  	 (gnus-check-backend-function
  	  'request-move-article gnus-newsgroup-name))
  	(respool-method (gnus-find-method-for-group gnus-newsgroup-name))
! 	article mark deletep respool)
  
      (when (member 'respool groups)
        (setq respool t)			; boolean for later
***************
*** 1006,1012 ****
      (spam-use-spamassassin-headers 	. 	spam-check-spamassassin-headers)
      (spam-use-spamassassin 		. 	spam-check-spamassassin)
      (spam-use-bogofilter-headers 	. 	spam-check-bogofilter-headers)
!     (spam-use-bogofilter 	 	. 	spam-check-bogofilter))
    "The spam-list-of-checks list contains pairs associating a
  parameter variable with a spam checking function.  If the
  parameter variable is true, then the checking function is called,
--- 1054,1061 ----
      (spam-use-spamassassin-headers 	. 	spam-check-spamassassin-headers)
      (spam-use-spamassassin 		. 	spam-check-spamassassin)
      (spam-use-bogofilter-headers 	. 	spam-check-bogofilter-headers)
!     (spam-use-bogofilter 	 	. 	spam-check-bogofilter)
!     (spam-use-crm114 	 	        . 	spam-check-crm114))
    "The spam-list-of-checks list contains pairs associating a
  parameter variable with a spam checking function.  If the
  parameter variable is true, then the checking function is called,
***************
*** 1028,1034 ****
      spam-use-bogofilter
      spam-use-blackholes
      spam-use-spamassassin
!     spam-use-spamoracle)
    "The spam-list-of-statistical-checks list contains all the mail
  splitters that need to have the full message body available.
  Note that you should fetch extra headers if you don't like this,
--- 1077,1084 ----
      spam-use-bogofilter
      spam-use-blackholes
      spam-use-spamassassin
!     spam-use-spamoracle
!     spam-use-crm114)
    "The spam-list-of-statistical-checks list contains all the mail
  splitters that need to have the full message body available.
  Note that you should fetch extra headers if you don't like this,
***************
*** 1209,1215 ****
      (spam-use-bogofilter spam-bogofilter-register-ham-routine
  			 spam-bogofilter-register-spam-routine
  			 spam-bogofilter-unregister-ham-routine
! 			 spam-bogofilter-unregister-spam-routine))
    "The spam-registration-functions list contains pairs
  associating a parameter variable with the ham and spam
  registration functions, and the ham and spam unregistration
--- 1259,1270 ----
      (spam-use-bogofilter spam-bogofilter-register-ham-routine
  			 spam-bogofilter-register-spam-routine
  			 spam-bogofilter-unregister-ham-routine
! 			 spam-bogofilter-unregister-spam-routine)
!     (spam-use-crm114 spam-crm114-register-ham-routine
! 		     spam-crm114-register-spam-routine
!                      ;; does CRM114 Mailfilter support unregistration?
! 		     nil
! 		     nil))
    "The spam-registration-functions list contains pairs
  associating a parameter variable with the ham and spam
  registration functions, and the ham and spam unregistration
***************
*** 2162,2167 ****
--- 2217,2307 ----
  
  (defun spam-spamassassin-unregister-ham-routine (articles)
    (spam-spamassassin-register-with-sa-learn articles nil t))
+ \f
+ ;;;; CRM114 Mailfilter
+ (defun spam-check-crm114-headers (&optional score)
+   (let ((header (message-fetch-field spam-crm114-header))
+ 	(spam-split-group (if spam-split-symbolic-return
+ 			      'spam
+ 			    spam-split-group)))
+     (when header			; return nil when no header
+       (if score				; scoring mode
+ 	  (if (string-match "( pR: \\([0-9.-]+\\)" header)
+ 	      (match-string 1 header)
+ 	    "0")
+ 	;; spam detection mode
+ 	(when (string-match spam-crm114-positive-spam-header
+ 			    header)
+ 	  spam-split-group)))))
+ 
+ ;; return something sensible if the score can't be determined
+ (defun spam-crm114-score ()
+   "Get the CRM114 Mailfilter pR"
+   (interactive)
+   (save-window-excursion
+     (gnus-summary-show-article t)
+     (set-buffer gnus-article-buffer)
+     (let ((score (or (spam-check-crm114-headers t)
+ 		     (spam-check-crm114 t))))
+       (gnus-summary-show-article)
+       (message "pR: %s" score)
+       (or score "0"))))
+ 
+ (defun spam-check-crm114 (&optional score)
+   "Check the CRM114 Mailfilter backend for the classification of this message"
+   (let ((article-buffer-name (buffer-name))
+ 	(db spam-crm114-database-directory)
+ 	return)
+     (with-temp-buffer
+       (let ((temp-buffer-name (buffer-name)))
+ 	(save-excursion
+ 	  (set-buffer article-buffer-name)
+ 	  (apply 'call-process-region
+ 		 (point-min) (point-max)
+ 		 spam-crm114-path
+ 		 nil temp-buffer-name nil
+ 		 (if db `("", db) `("")))) ; asjo: I don't know how to do this right.
+                                            ;       I'd like to add --fileprefix=db if
+                                            ;       db is set.
+ 	(setq return (spam-check-crm114-headers score))))
+     return))
+ 
+ (defun spam-crm114-register-with-crm114 (articles
+ 						 spam
+ 						 &optional unregister)
+   "Register an article, given as a string, as spam or non-spam."
+   (dolist (article articles)
+     (let ((article-string (spam-get-article-as-string article))
+ 	  (db spam-crm114-database-directory)
+ 	  (switch (if unregister
+ 		      (if spam
+ 			  spam-crm114-spam-strong-switch
+ 			spam-crm114-ham-strong-switch)
+ 		    (if spam
+ 			spam-crm114-spam-switch
+ 		      spam-crm114-ham-switch))))
+       (when (stringp article-string)
+ 	(with-temp-buffer
+ 	  (insert article-string)
+ 
+ 	  (apply 'call-process-region
+ 		 (point-min) (point-max)
+ 		 spam-crm114-path
+ 		 nil nil nil switch
+ 		 (if db `("", db) `("")))))))) ; asjo: See above.
+ 
+ (defun spam-crm114-register-spam-routine (articles &optional unregister)
+   (spam-crm114-register-with-crm114 articles t unregister))
+ 
+ (defun spam-crm114-unregister-spam-routine (articles)
+   (spam-crm114-register-spam-routine articles t))
+ 
+ (defun spam-crm114-register-ham-routine (articles &optional unregister)
+   (spam-crm114-register-with-crm114 articles nil unregister))
+ 
+ (defun spam-crm114-unregister-ham-routine (articles)
+   (spam-crm114-register-ham-routine articles t))
+ 
  \f
  ;;;; Hooks
  

Re: CRM114 Mailfilter and spam.el

[Jesper Harder]

asjo@koldfront.dk (Adam Sjøgren) writes:

> + 		 (if db `("", db) `("")))) ; asjo: I don't know how to do this right.
> +                                            ;       I'd like to add --fileprefix=db if
> +                                            ;       db is set.

Do you mean:

(defun spam-check-crm114 (&optional score)
  "Check the CRM114 Mailfilter back end for the classification of this message."
   (let ((article-buffer-name (buffer-name))
 	(db spam-crm114-database-directory))
     (with-temp-buffer
       (let ((temp-buffer-name (buffer-name)))
	 (with-current-buffer article-buffer-name
	   (apply 'call-process-region
		  (point-min) (point-max)
		  spam-crm114-path
		  nil temp-buffer-name nil
		  (when db (list (concat "--fileprefix=" db)))))
	 (spam-check-crm114-headers score)))))

[untested!]

> + (defcustom spam-crm114-path (executable-find "mailfilter.crm")
> +   "File path of the CRM114 Mailfilter executable program."
> +   :type '(choice (file :tag "Location of CRM114 Mailfilter")
> + 		 (const :tag "CRM114 Mailfilter is not installed"))
> +   :group 'spam-crm114)

This isn't a very good name[1].  I think `spam-crm114-program' would
be clearer.


[1] From the GNU coding standards:

  Please do not use the term ``pathname'' that is used in Unix
  documentation; use ``file name'' (two words) instead.  We use the
  term ``path'' only for search paths, which are lists of directory
  names.
-- 
Jesper Harder                                <http://purl.org/harder/>

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On Sun, 04 Apr 2004 03:59:53 +0200, Jesper wrote:

> asjo@koldfront.dk (Adam Sjøgren) writes:

>> + (if db `("", db) `("")))) ; asjo: I don't know how to do this

[...]

> Do you mean:

[...]

>  (when db (list (concat "--fileprefix=" db)))))

Yes, I think so. I didn't encounter the `-syntax before.

> [untested!]

I will test, thanks!

>> + (defcustom spam-crm114-path (executable-find "mailfilter.crm") +
>> "File path of the CRM114 Mailfilter executable program."

[...]

> This isn't a very good name[1]. I think `spam-crm114-program' would
> be clearer.

Agreed. It was copied from spam-bogofilter-path. For what it's worth,
there are also spam-ifile-path, spam-spamassasin-path and
spam-sa-learn-path there - so it is consistent with the naming in
spam.el.


  Best regards,

-- 
 "Some people like cupcakes better.                           Adam Sjøgren
  I for one care less for them!"                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Ted Zlatanov]

On Sat, 03 Apr 2004, asjo@koldfront.dk wrote:

> Hmm: It seems that spam-junk-mailgroups defaults to: ("spam"
> "mail.junk" "poste.pourriel"), yet my nnml:spam group is
> unclassified (which is what I want, but not what I expect from
> reading the docs on classification:

The docs are not clear on when a regex is expected, and when an exact
match is expected.  spam-junk-mailgroups expects an exact match.
Also, it's a variable I would like to deprecate, since its job is
done by other mechanisms already.

Ted

Re: CRM114 Mailfilter and spam.el

[Ted Zlatanov]

On Sun, 04 Apr 2004, asjo@koldfront.dk wrote:

On Sun, 04 Apr 2004 03:59:53 +0200, Jesper wrote:
> 
>> This isn't a very good name[1]. I think `spam-crm114-program' would
>> be clearer.
> 
> Agreed. It was copied from spam-bogofilter-path. For what it's
> worth, there are also spam-ifile-path, spam-spamassasin-path and
> spam-sa-learn-path there - so it is consistent with the naming in
> spam.el.

That's my fault.  I can change the variable names but it will break
backwards compatibility.  Any opinions?

(wishing for a :deprecated option in defcustom)

Thanks
Ted

Re: CRM114 Mailfilter and spam.el

[Jesper Harder]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Sun, 04 Apr 2004, asjo@koldfront.dk wrote:
>
>> Agreed. It was copied from spam-bogofilter-path. For what it's
>> worth, there are also spam-ifile-path, spam-spamassasin-path and
>> spam-sa-learn-path there - so it is consistent with the naming in
>> spam.el.
>
> That's my fault.  I can change the variable names but it will break
> backwards compatibility.  Any opinions?

I suppose you could do something like this to keep compatibility:

  (defvar spam-ifile-path nil)
  (make-obsolete-variable 'spam-ifile-path 'spam-ifile-program)

  (defcustom spam-ifile-program (or spam-ifile-path 
                                    (executable-find "ifile"))

`defvaralias' would be nicer, but it's not in Emacs 21.3.

-- 
Jesper Harder                                <http://purl.org/harder/>

Re: CRM114 Mailfilter and spam.el

[Kai Grossjohann]

asjo@koldfront.dk (Adam Sjøgren) writes:

> I only want to Train On Error - so I do not want spam that has been
> recognized correctly as spam to be processed as spam (passed to crm114
> for training).
>
> So, if my spam-group as classified as spam, I have to remove all those
> automatically set marks by hand.

You could also tell the nnml:spam group not to send spam to crm114,
but to just delete it.  I think that works by changing the spam exit
processor group parameter.

I have these parameters on nnml:spam:

((spam-contents gnus-group-spam-classification-spam)
 (spam-process
  ((spam spam-use-bogofilter)
   (ham spam-use-bogofilter)))
 (ham-process-destination . respool)
 (ham-marks
  (gnus-read-mark gnus-ticked-mark)))

which means that I want to run bogofilter on the stuff from that
group, which is the opposite of what you want.

Kai

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]

On Sun, 04 Apr 2004 15:07:34 +0200, Adam wrote:

>> [untested!]

> I will test, thanks!

Works great, thanks again.

Attached is a patch with the changes you suggested incorporated.


  Best regards,

-- 
 "Do not feed the oysters under the clouds"                   Adam Sjøgren
                                                         asjo@koldfront.dk

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: crm114-mailfilter-2.patch --]
[-- Type: text/x-patch, Size: 10232 bytes --]

Index: spam.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/spam.el,v
retrieving revision 7.27
diff -u -r7.27 spam.el
--- spam.el	2 Apr 2004 19:09:16 -0000	7.27
+++ spam.el	6 Apr 2004 21:34:42 -0000
@@ -263,6 +263,11 @@
   :type 'boolean
   :group 'spam)
 
+(defcustom spam-use-crm114 nil
+  "Whether CRM114 Mailfilter should be used by `spam-split'."
+  :type 'boolean
+  :group 'spam)
+
 (defcustom spam-install-hooks (or
 			       spam-use-dig
 			       spam-use-gmane-xref
@@ -281,7 +286,8 @@
 			       spam-use-BBDB-exclusive
 			       spam-use-ifile
 			       spam-use-stat
-			       spam-use-spamoracle)
+			       spam-use-spamoracle
+                               spam-use-crm114)
   "Whether the spam hooks should be installed.
 Default to t if one of the spam-use-* variables is set."
   :group 'spam
@@ -515,6 +521,53 @@
   :type 'string
   :group 'spam-spamassassin)
 
+(defgroup spam-crm114 nil
+  "Spam CRM114 Mailfilter configuration."
+  :group 'spam)
+
+(defcustom spam-crm114-program (executable-find "mailfilter.crm")
+  "File path of the CRM114 Mailfilter executable program."
+  :type '(choice (file :tag "Location of CRM114 Mailfilter")
+		 (const :tag "CRM114 Mailfilter is not installed"))
+  :group 'spam-crm114)
+
+(defcustom spam-crm114-header "X-CRM114-Status"
+  "The header that CRM114 Mailfilter inserts in messages."
+  :type 'string
+  :group 'spam-crm114)
+
+(defcustom spam-crm114-spam-switch "--learnspam"
+  "The switch that CRM114 Mailfilter uses to register spam messages."
+  :type 'string
+  :group 'spam-crm114)
+
+(defcustom spam-crm114-ham-switch "--learnnonspam"
+  "The switch that CRM114 Mailfilter uses to register ham messages."
+  :type 'string
+  :group 'spam-crm114)
+
+(defcustom spam-crm114-spam-strong-switch "--UNKNOWN"
+  "The switch that CRM114 Mailfilter uses to unregister ham messages."
+  :type 'string
+  :group 'spam-crm114)
+
+(defcustom spam-crm114-ham-strong-switch "--UNKNOWN"
+  "The switch that CRM114 Mailfilter uses to unregister spam messages."
+  :type 'string
+  :group 'spam-crm114)
+
+(defcustom spam-crm114-positive-spam-header "^SPAM"
+  "The regex on `spam-crm114-header' for positive spam identification."
+  :type 'regexp
+  :group 'spam-crm114)
+
+(defcustom spam-crm114-database-directory nil
+  "Directory path of the CRM114 Mailfilter databases."
+  :type '(choice (directory
+		  :tag "Location of the CRM114 Mailfilter database directory")
+		 (const :tag "Use the default"))
+  :group 'spam-crm114)
+
 ;;; Key bindings for spam control.
 
 (gnus-define-keys gnus-summary-mode-map
@@ -605,6 +658,7 @@
     (gnus-group-spam-exit-processor-stat         spam spam-use-stat)
     (gnus-group-spam-exit-processor-spamoracle   spam spam-use-spamoracle)
     (gnus-group-spam-exit-processor-spamassassin spam spam-use-spamassassin)
+    (gnus-group-spam-exit-processor-crm114       spam spam-use-crm114)
     (gnus-group-ham-exit-processor-ifile         ham spam-use-ifile)
     (gnus-group-ham-exit-processor-bogofilter    ham spam-use-bogofilter)
     (gnus-group-ham-exit-processor-stat          ham spam-use-stat)
@@ -612,7 +666,8 @@
     (gnus-group-ham-exit-processor-BBDB          ham spam-use-BBDB)
     (gnus-group-ham-exit-processor-copy          ham spam-use-ham-copy)
     (gnus-group-ham-exit-processor-spamassassin  ham spam-use-spamassassin)
-    (gnus-group-ham-exit-processor-spamoracle    ham spam-use-spamoracle))
+    (gnus-group-ham-exit-processor-spamoracle    ham spam-use-spamoracle)
+    (gnus-group-ham-exit-processor-crm114        ham spam-use-crm114))
   "The `spam-list-of-processors' list.
 This list contains pairs associating a ham/spam exit processor
 variable with a classification and a spam-use-* variable.")
@@ -657,6 +712,9 @@
 (defun spam-group-spam-processor-spamoracle-p (group)
   (spam-group-processor-p group 'gnus-group-spam-exit-processor-spamoracle))
 
+(defun spam-group-spam-processor-crm114-p (group)
+  (spam-group-processor-p group 'gnus-group-spam-exit-processor-crm114))
+
 (defun spam-group-ham-processor-bogofilter-p (group)
   (spam-group-processor-p group 'gnus-group-ham-exit-processor-bogofilter))
 
@@ -678,6 +736,9 @@
 (defun spam-group-ham-processor-spamoracle-p (group)
   (spam-group-processor-p group 'gnus-group-ham-exit-processor-spamoracle))
 
+(defun spam-group-ham-processor-crm114-p (group)
+  (spam-group-processor-p group 'gnus-group-ham-exit-processor-crm114))
+
 (defun spam-report-articles-gmane (n)
   "Report the current message as spam.
 Respects the process/prefix convention."
@@ -686,6 +747,7 @@
     (gnus-summary-remove-process-mark article)
     (spam-report-gmane article)))
 
+; asjo: Perhaps add spam-crm114-score here? How?
 (defun spam-generic-score ()
   (interactive)
   "Invoke whatever scoring method we can."
@@ -776,8 +838,8 @@
       (spam-ham-copy-routine
        (gnus-parameter-ham-process-destination gnus-newsgroup-name)))
 
-    ;; now move all ham articles out of spam groups
-    (when (spam-group-spam-contents-p gnus-newsgroup-name)
+    ;; now move all ham articles out of non-ham groups
+    (when (not (spam-group-ham-contents-p gnus-newsgroup-name))
       (gnus-message 5 "Moving ham messages from spam group")
       (spam-ham-move-routine
        (gnus-parameter-ham-process-destination gnus-newsgroup-name))))
@@ -859,7 +921,7 @@
 	 (gnus-check-backend-function
 	  'request-move-article gnus-newsgroup-name))
 	(respool-method (gnus-find-method-for-group gnus-newsgroup-name))
-	article mark todo deletep respool)
+	article mark deletep respool)
 
     (when (member 'respool groups)
       (setq respool t)			; boolean for later
@@ -1006,7 +1068,8 @@
     (spam-use-spamassassin-headers 	. 	spam-check-spamassassin-headers)
     (spam-use-spamassassin 		. 	spam-check-spamassassin)
     (spam-use-bogofilter-headers 	. 	spam-check-bogofilter-headers)
-    (spam-use-bogofilter 	 	. 	spam-check-bogofilter))
+    (spam-use-bogofilter 	 	. 	spam-check-bogofilter)
+    (spam-use-crm114 	 	        . 	spam-check-crm114))
   "The spam-list-of-checks list contains pairs associating a
 parameter variable with a spam checking function.  If the
 parameter variable is true, then the checking function is called,
@@ -1028,7 +1091,8 @@
     spam-use-bogofilter
     spam-use-blackholes
     spam-use-spamassassin
-    spam-use-spamoracle)
+    spam-use-spamoracle
+    spam-use-crm114)
   "The spam-list-of-statistical-checks list contains all the mail
 splitters that need to have the full message body available.
 Note that you should fetch extra headers if you don't like this,
@@ -1209,7 +1273,12 @@
     (spam-use-bogofilter spam-bogofilter-register-ham-routine
 			 spam-bogofilter-register-spam-routine
 			 spam-bogofilter-unregister-ham-routine
-			 spam-bogofilter-unregister-spam-routine))
+			 spam-bogofilter-unregister-spam-routine)
+    (spam-use-crm114 spam-crm114-register-ham-routine
+		     spam-crm114-register-spam-routine
+                     ;; does CRM114 Mailfilter support unregistration?
+		     nil
+		     nil))
   "The spam-registration-functions list contains pairs
 associating a parameter variable with the ham and spam
 registration functions, and the ham and spam unregistration
@@ -2162,6 +2231,89 @@
 
 (defun spam-spamassassin-unregister-ham-routine (articles)
   (spam-spamassassin-register-with-sa-learn articles nil t))
+\f
+;;;; CRM114 Mailfilter
+(defun spam-check-crm114-headers (&optional score)
+  (let ((header (message-fetch-field spam-crm114-header))
+	(spam-split-group (if spam-split-symbolic-return
+			      'spam
+			    spam-split-group)))
+    (when header			; return nil when no header
+      (if score				; scoring mode
+	  (if (string-match "( pR: \\([0-9.-]+\\)" header)
+	      (match-string 1 header)
+	    "0")
+	;; spam detection mode
+	(when (string-match spam-crm114-positive-spam-header
+			    header)
+	  spam-split-group)))))
+
+;; return something sensible if the score can't be determined
+(defun spam-crm114-score ()
+  "Get the CRM114 Mailfilter pR"
+  (interactive)
+  (save-window-excursion
+    (gnus-summary-show-article t)
+    (set-buffer gnus-article-buffer)
+    (let ((score (or (spam-check-crm114-headers t)
+		     (spam-check-crm114 t))))
+      (gnus-summary-show-article)
+      (message "pR: %s" score)
+      (or score "0"))))
+
+(defun spam-check-crm114 (&optional score)
+  "Check the CRM114 Mailfilter backend for the classification of this message"
+  (let ((article-buffer-name (buffer-name))
+	(db spam-crm114-database-directory)
+	return)
+    (with-temp-buffer
+      (let ((temp-buffer-name (buffer-name)))
+	(save-excursion
+	  (set-buffer article-buffer-name)
+	  (apply 'call-process-region
+		 (point-min) (point-max)
+		 spam-crm114-program
+		 nil temp-buffer-name nil
+                 (when db (list (concat "--fileprefix=" db)))))
+	(setq return (spam-check-crm114-headers score))))
+    return))
+
+(defun spam-crm114-register-with-crm114 (articles
+						 spam
+						 &optional unregister)
+  "Register an article, given as a string, as spam or non-spam."
+  (dolist (article articles)
+    (let ((article-string (spam-get-article-as-string article))
+	  (db spam-crm114-database-directory)
+	  (switch (if unregister
+		      (if spam
+			  spam-crm114-spam-strong-switch
+			spam-crm114-ham-strong-switch)
+		    (if spam
+			spam-crm114-spam-switch
+		      spam-crm114-ham-switch))))
+      (when (stringp article-string)
+	(with-temp-buffer
+	  (insert article-string)
+
+	  (apply 'call-process-region
+		 (point-min) (point-max)
+		 spam-crm114-program
+		 nil nil nil
+                 (when db (list switch (concat "--fileprefix=" db)))))))))
+
+(defun spam-crm114-register-spam-routine (articles &optional unregister)
+  (spam-crm114-register-with-crm114 articles t unregister))
+
+(defun spam-crm114-unregister-spam-routine (articles)
+  (spam-crm114-register-spam-routine articles t))
+
+(defun spam-crm114-register-ham-routine (articles &optional unregister)
+  (spam-crm114-register-with-crm114 articles nil unregister))
+
+(defun spam-crm114-unregister-ham-routine (articles)
+  (spam-crm114-register-ham-routine articles t))
+
 \f
 ;;;; Hooks
 

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On Tue, 06 Apr 2004 10:12:57 +0200, Kai wrote:

> asjo@koldfront.dk (Adam Sjøgren) writes:
>> I only want to Train On Error - so I do not want spam that has been
>> recognized correctly as spam to be processed as spam (passed to
>> crm114 for training).

>> So, if my spam-group as classified as spam, I have to remove all
>> those automatically set marks by hand.

> You could also tell the nnml:spam group not to send spam to crm114,
> but to just delete it. I think that works by changing the spam exit
> processor group parameter.

Ah, yes.

Except that I'm weird in that I don't want my spam marked as
expirable/deleted.


  Best regards,

    Adam

-- 
 "Uden Torleif var det blevet en calzone!"                    Adam Sjøgren
                                                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Ted Zlatanov]

Adam,

I comitted your patch, assuming your papers are on file (it's been a
while since we discussed this).  The two spam.el hunks below didn't
belong in the patch, I think, and I didn't consider them carefully so
let me know if I should.  Also, I added a scoring function for CRM114
to spam-generic-score and fixed a few small problems.

Thanks
Ted

@@ -776,8 +838,8 @@
       (spam-ham-copy-routine
        (gnus-parameter-ham-process-destination gnus-newsgroup-name)))
 
-    ;; now move all ham articles out of spam groups
-    (when (spam-group-spam-contents-p gnus-newsgroup-name)
+    ;; now move all ham articles out of non-ham groups
+    (when (not (spam-group-ham-contents-p gnus-newsgroup-name))
       (gnus-message 5 "Moving ham messages from spam group")
       (spam-ham-move-routine
        (gnus-parameter-ham-process-destination gnus-newsgroup-name))))
@@ -859,7 +921,7 @@
 	 (gnus-check-backend-function
 	  'request-move-article gnus-newsgroup-name))
 	(respool-method (gnus-find-method-for-group gnus-newsgroup-name))
-	article mark todo deletep respool)
+	article mark deletep respool)
 
     (when (member 'respool groups)
       (setq respool t)			; boolean for later

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On 20 May 2004 14:39:25 -0400, Ted wrote:

> Adam, I comitted your patch, assuming your papers are on file (it's
> been a while since we discussed this).

Alas, they're not, yet. I've done almost nothing in elisp before, and
only sent an email to FSF a couple of days ago when Lars posted the
procedure publicly.

(I think the FSF is hiding how to do this quite aptly; I did go and
look for it at their website a while back without finding any
instructions. Maybe I was just being stupid.)

> The two spam.el hunks below didn't belong in the patch, I think, and
> I didn't consider them carefully so let me know if I should.

Please do consider them - or just say why you think they shouldn't go
in.

The first patch changes the behaviour as the comment change says -
instead of only moving ham articles out of spam-groups, it moves ham
articles out of non-ham-groups.

This makes sense in my setup, because I do not want to declare my
nnml:spam-group as spam (I do not want spam marked as expired
automatically), so I leave it as unclassified (so it's non-ham, but
not spam).

(Oh, the following call to gnus-message should be modified as well, to
reflect the change, if you deem it acceptable).

Would it break anyone elses setup?

When I mark an article as ham in an unclassified group, what would I
be trying to tell Gnus/spam.el if not "please move it to a/the
ham-group"?

(This is not a retorical question; I know people have very different
ideas of how they want to handle their mix of ham'n'spam.)

@@ -776,8 +838,8 @@
       (spam-ham-copy-routine
        (gnus-parameter-ham-process-destination gnus-newsgroup-name)))
 
-    ;; now move all ham articles out of spam groups
-    (when (spam-group-spam-contents-p gnus-newsgroup-name)
+    ;; now move all ham articles out of non-ham groups
+    (when (not (spam-group-ham-contents-p gnus-newsgroup-name))
       (gnus-message 5 "Moving ham messages from spam group")
       (spam-ham-move-routine
        (gnus-parameter-ham-process-destination gnus-newsgroup-name))))

spam.el doesn't work for me without the second, following, patch,
which I basically thought was a typo-correction; alas I'm only
starting to learn elisp, so I may very well be wrong.

@@ -859,7 +921,7 @@
 	 (gnus-check-backend-function
 	  'request-move-article gnus-newsgroup-name))
 	(respool-method (gnus-find-method-for-group gnus-newsgroup-name))
-	article mark todo deletep respool)
+	article mark deletep respool)
 
     (when (member 'respool groups)
       (setq respool t)			; boolean for later

> Also, I added a scoring function for CRM114 to spam-generic-score
> and fixed a few small problems.

Cool!


  Best regards,

-- 
 "Some people like cupcakes better.                           Adam Sjøgren
  I for one care less for them!"                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Lars Magne Ingebrigtsen]

asjo@koldfront.dk (Adam Sjøgren) writes:

> (I think the FSF is hiding how to do this quite aptly; I did go and
> look for it at their website a while back without finding any
> instructions. Maybe I was just being stupid.)

I think it's policy -- the copyright assignment request is supposed
to be sent out by maintainers to contributors when someone
contributes, and not the other way around.  For some reason or
other.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On Sun, 23 May 2004 01:44:03 +0200, Lars wrote:

> I think it's policy -- the copyright assignment request is supposed
> to be sent out by maintainers to contributors when someone
> contributes, and not the other way around. For some reason or other. 
> :-)

Ah, I see. I guess it makes sense, although it is a little
counter-intuitive for people of a slightly more timid disposition.


  Best regards,

-- 
 "Stop blowing holes in my ship!"                             Adam Sjøgren
                                                         asjo@koldfront.dk

Re: CRM114 Mailfilter and spam.el

[Ted Zlatanov]

On Thu, 20 May 2004, asjo@koldfront.dk wrote:

> The first patch changes the behaviour as the comment change says -
> instead of only moving ham articles out of spam-groups, it moves ham
> articles out of non-ham-groups.
> 
> This makes sense in my setup, because I do not want to declare my
> nnml:spam-group as spam (I do not want spam marked as expired
> automatically), so I leave it as unclassified (so it's non-ham, but
> not spam).

I think, as I mentioned, that spam-process-ham-in-nonham-groups
is the solution to your request.

> spam.el doesn't work for me without the second, following, patch,
> which I basically thought was a typo-correction; alas I'm only
> starting to learn elisp, so I may very well be wrong.

You're right, and I didn't catch it earlier.  Looks like Emacs
Lisp allows you to declare a variable twice in the same let
binding.  Weird.  Fixed in CVS, thanks!

Ted

Re: CRM114 Mailfilter and spam.el

[Adam Sjøgren]

On 26 May 2004 12:23:18 -0400, Ted wrote:

> I think, as I mentioned, that spam-process-ham-in-nonham-groups is
> the solution to your request.

It is, thanks!

>> spam.el doesn't work for me without the second, following, patch,
>> which I basically thought was a typo-correction; alas I'm only
>> starting to learn elisp, so I may very well be wrong.

> You're right, and I didn't catch it earlier. Looks like Emacs Lisp
> allows you to declare a variable twice in the same let binding. 
> Weird. Fixed in CVS, thanks!

Cool.


  Best regards,

-- 
 "I've used about 20 operating systems in my career and       Adam Sjøgren
  written two. I've programmed in over 40 languages.     asjo@koldfront.dk
  I've tasted about 75 flavors of ice cream. Limiting me
  to one of each would be a pity."