Gnus and SpamAssassin

Gnus and SpamAssassin

[Tim McNamara]

I'm trying to integrate Gnus and SpamAssassin for spam management and
have been looking around in Info as well as my.gnus.org, where there
are two Lisp recipes for combining fancy splitting and SpamAssassin.
My knowledge of Lisp isn't quite good enough to compare the two
meaningfully; it also seems like one could integrate a BBDB based
whitelist into this as well; ISTR there is something in the Gnus
and/or BBDB docs.

After looking at the two approaches, as well as the Lisp code for
reporting spam to sa-learn, here's what's in my .gnus thanks to the
generosity of the authors and a bit of cut-and-paste.  I'm not sure
whether to place the Lisp for using SpamAssassin before or after the
usual splits I use to sort out my personal mail, a mailing list I
subscribe to, and the rest of it.  What I want to do is to split out
the mailing list e-mails before processing them through SpamAssassin
to save time, since that is never spam; then run the rest of the
e-mail through SpamAssassin and sort it into "Personal," "mail.misc"
and "spamkill" (where I can examine it before expiring it, and mark
it as spam or ham for sa-learn).

I'd be grateful if anyone could point out the inevitable egregious
errors:



(setq nnmail-split-methods 'nnmail-split-fancy)

(defun hc:fancy-split-spamassassin ()
   (save-excursion
     (set-buffer " *nnmail incoming*")
     (call-process-region (point-min) (point-max) "spamc" t t nil "-f")
     (goto-char (point-min))
     (when (re-search-forward "^x-spam-flag: yes$" nil t)
           ;; edit the following to the group where spam is to be dropped
           "spamkill")))

   (setq nnmail-split-fancy
     '(| ("to" "timmcn@bitstream\\.net" "Personal")
         ("to" "internet-bob@bikelist\\.org" "iBOB")
         "mail.misc"))

(defun report-spam () 
  "Submit Spam." 
  (interactive) 
  (dolist (art gnus-newsgroup-processable) 
    (gnus-summary-goto-article art) 
    (gnus-summary-show-raw-article) 
    (gnus-summary-save-in-pipe "sa-learn --spam") 
    (gnus-summary-delete-article)))

(defun report-ham () 
  "Submit Ham." 
  (interactive) 
  (dolist (art gnus-newsgroup-processable) 
    (gnus-summary-goto-article art) 
    (gnus-summary-show-raw-article) 
    (gnus-summary-save-in-pipe "sa-learn --ham")))

Re: Gnus and SpamAssassin

[Tim McNamara]

Tim McNamara <timmcn@bitstream.net> writes:

> Hubert Chan <hubert@uhoreg.ca> writes:
>
>> spam.el from CVS gnus has SpamAssassin integration, including
>> sa-learn registration.  (Unfortunately, no documentation yet for
>> the SpamAssassin part, but reading the rest of the info file
>> regarding spam.el should give you a good idea on how to get it
>> working, and all the variables can be changed via customize.)  You
>> should be able to just use CVS spam.el along with your current
>> version of gnus, though you may find a definition from gnus.el
>> helpful, namely the (very long) "(gnus-define-group-parameter
>> spam-process .... )" expression.  You can just stick that anywhere
>> in your .gnus file.
>>
>> For BBDB whitelisting, you can check out
>> http://my.gnus.org/node/view/35.
>
> Thanks, I'll check that out this evening.  I had wondered about
> spam.el in this context, but hadn't seen that SpamAssassin might be
> possible even if undocumented.  Probably a matter of being unskilled
> in Lisp.

My current setup just stuck two e-mails in nnml:bogus, which is not a
group created in my .gnus.  I assume that means something is screwed up?

Re: Gnus and SpamAssassin

[Hubert Chan]

[-- Attachment #1: Type: text/plain, Size: 635 bytes --]

>>>>> "Tim" == Tim McNamara <timmcn@bitstream.net> writes:

[...]

Tim> Thanks, I'll check that out this evening.  I had wondered about
Tim> spam.el in this context, but hadn't seen that SpamAssassin might be
Tim> possible even if undocumented.  Probably a matter of being
Tim> unskilled in Lisp.

SpamAssassin was just added to CVS last week.  Hopefully documentation
will be there by the next real release of gnus.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]

Re: Gnus and SpamAssassin

[Hubert Chan]

[-- Attachment #1: Type: text/plain, Size: 626 bytes --]

>>>>> "Tim" == Tim McNamara <timmcn@bitstream.net> writes:

[...]

Tim> Ah.  I though using setq allowed the use of an apostophe rather
Tim> than double quotes- at least that was the impression I got from the
Tim> "Intro to Emacs Lisp" in Info.  I'll change it and see how it
Tim> works.

'foo and "foo" are different in Lisp.  'foo is a symbol, and "foo" is a
string.  call-process-region expects a string.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]

Re: Gnus and SpamAssassin

[Tim McNamara]

This was written to my .gnus on starting Gnus the first time after
changing the spam-filter code:

X-Mail-Format-Warning: Bad RFC2822 header formatting in (setq
gnus-select-method '(nntp "news.bitstream.net"))
X-Spam-Level: **
X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on Stella-Blue.local
X-Spam-Status: No, hits=2.9 required=7.5 tests=DATE_MISSING,FROM_NO_LOWER 
	autolearn=no version=2.63

Any idea what that was about?  And why did it write to .gnus?
http://www.faqs.org/rfcs/rfc2822.html wasn't all that helpful.

I've never seen anything like this before.  The only changes to .gnus
have been the spam code we've been talking about.  The current code
is:

  ;;Use spamc to process articles for spam filtering
(setq spam-filter-script "/usr/bin/spamc")

(setq nnmail-split-methods 'nnmail-split-fancy)

(defun sa-on-message ()
  ;;Use SpamAssassin to split spam from ham
 
     (widen)
     (call-process-region (point-min) (point-max)
			  spam-filter-script t t nil)
     (goto-char (point-min))
     (when (re-search-forward "^X-Spam-Status: yes$" nil t)
       "spam"))

   (setq nnmail-split-fancy
     '(| ("to" "internet-bob@bikelist\\.org" "iBOB")
         (: sa-on-message)
         ("to" "timmcn@bitstream\\.net" "Inbox")
         "mail.misc"))

Re: Gnus and SpamAssassin

[Hubert Chan]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

>>>>> "Tim" == Tim McNamara <timmcn@bitstream.net> writes:

Tim> This was written to my .gnus on starting Gnus the first time after
Tim> changing the spam-filter code:

[... snip ...]

My guess is  that you somehow managed to run sa-on-message on your
.gnus buffer.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]

Re: Gnus and SpamAssassin

[Ted Zlatanov]

On Tue, 27 Jan 2004, timmcn@bitstream.net wrote:

[regarding the spam.el documentation]

> Well, for example, whether one would use spam-split with
> SpamAssassin (or another process such as Bogofilter, which has some
> documentation); also how to use a bbdb whitelist to break out e-mail
> from those senders before processing the rest (thinking this would
> speed things up by running fewer e-mails through spamc).  

All of the above can be done.  SA was just added by Hubert Chan so the
documentation is not ready yet. To use the BBDB whitelist you just set
spam-use-BBDB to t, and spam-split should automatically skip the spam
checks on messages from people in the BBDB.

> The documentation is fairly dense and seems to assume an
> acquaintence with both the mechanics of Gnus and Lisp that I don't
> have yet; fortunately this is starting to make more sense!  I'll
> look through it again and see if I can be more specific.

I'm afraid this is my fault.  It's easier to write code than to
document it.  It's hard for me to see the problems because I'm so
familiar with the code; if you have particular suggestions on the
spam.el section I would greatly appreciate them.

Thanks
Ted

Re: Gnus and SpamAssassin

[Tim McNamara]

Tim McNamara <timmcn@bitstream.net> writes:

To follow up on my follow-up, the following results in "iBOB" mail
being split accurately but all other mail ending up in nnml:bogus.
Obviously I did something wrong...

> (setq nnmail-split-methods 'nnmail-split-fancy)
>
> (require 'spam)
> (defun sa-on-message ()
>    ;;Use SpamAssassin to catch spam into "spam"
>   
>       (widen)
>       (let ((spam-split-symbolic-return t)
>             (spam-spamassassin-path "/usr/bin/spamc"))
>        (when (eq 'spam (spam-check-spamassassin))
>         "spam")))
>
>    (setq nnmail-split-fancy
>      '(| ("to" "internet-bob@bikelist\\.org" "iBOB")
>          (: sa-on-message)
>          ("to" "timmcn@bitstream\\.net" "Inbox")
>          "mail.misc"))

Re: Gnus and SpamAssassin

[Tim McNamara]

Hmm, this setup also seems to be scanning newsgroups for spam and
echoes something about "expiring spam without moving" every time I
leave a newsgroup.  There is no activity associated with SpamAssassin
in this, per the logs.

Re: Gnus and SpamAssassin

[Ted Zlatanov]

On Thu, 29 Jan 2004, hubert@uhoreg.ca wrote:

>>>>>> "Tim" == Tim McNamara <timmcn@bitstream.net> writes:
> 
> Tim> Tim McNamara <timmcn@bitstream.net> writes: To follow up on my
> Tim> follow-up, the following results in "iBOB" mail being split
> Tim> accurately but all other mail ending up in nnml:bogus.
> Tim> Obviously I did something wrong...
> 
> Hmm.  I don't see anything wrong right away.

Ditto.

> If nobody else figures anything out, can you try the following:
> - in the nnml:bogus group, select any message
> - hit C-u g (this will open up a raw view of the message)
> - switch to the buffer containing the message
> - type M-x (sa-on-message)
>   (i.e. hit M-x, and type (sa-...))
> - hopefully, this will pop up a debug buffer.  (If not, the error is
>   somewhere else)
> - send the contents of the debug buffer

I think you meant M-: instead of M-x :)

Also M-x toggle-debug-on-error would probably be helpful to Tim
before he follows your steps.

Tim: you can call sa-on-message in any buffer, not just a message
buffer.  It shouldn't fail with an error, no matter what.  The spam
checker may not return sensible results, but it won't die.

Ted

modifying incoming messages (was: Gnus and SpamAssassin)

[Ted Zlatanov]

On Wed, 28 Jan 2004, timmcn@bitstream.net wrote:

> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> Remember, you're replacing text when you use that
>> call-process-region.
> 
> Well, don't I want it to do that to an extent?  I want it to add
> "****SPAM****" to the subject or "X-Spam-Flag = YES" or whatever the
> precise syntax is.  Or is there a different way of doing this?

I am not sure what the side effects of modifying an incoming message
are.  At least in spam.el I avoid it, and the Gnus manual doesn't say
it's OK in any examples.  I would be very cautious, this is a great
way to lose mail or cause errors in Gnus.  I hope someone better
acquainted with the Gnus backends will speak up on this.

Your goal of modifying the message is probably best served by a
filter that precedes Gnus in the order of mail delivery.  The goal of
detecting spam, however, is achievable without modifying the message.

Ted

Re: Gnus and SpamAssassin

[Kai Grossjohann]

Tim McNamara <timmcn@bitstream.net> writes:

> (defun hc:fancy-split-spamassassin ()
>    (save-excursion
>      (set-buffer " *nnmail incoming*")
>      (call-process-region (point-min) (point-max) "spamc" t t nil "-f")
>      (goto-char (point-min))
>      (when (re-search-forward "^x-spam-flag: yes$" nil t)
>            ;; edit the following to the group where spam is to be dropped
>            "spamkill")))

It seems this function can be achieved by normal fancy splitting, with
an entry like

        ("x-spam-flag" "yes" "spamkill")

Compare this entry with

        ("to" "internet-bob@bikelist\\.org" "iBOB")

>    (setq nnmail-split-fancy
>      '(| ("to" "timmcn@bitstream\\.net" "Personal")
>          ("to" "internet-bob@bikelist\\.org" "iBOB")
>          "mail.misc"))

But maybe I'm missing something crucial.

Kai