Gnus SPAM support, and email based reporting.

Gnus SPAM support, and email based reporting.

[Daniel Pittman]

Hey there.  I would like to move to using the Gnus SPAM management
support, to deal with reporting missed SPAM and false positives to my
spam filtering tool.

I run dspam on my server, before pulling mail down to my laptop.

So, to report an incorrectly classified message, I need to resend the
message as-is to a specific address on the server, then delete it
locally.

It doesn't look like the SPAM support has any in-built code to let me do
that.

So, have anyone looked at this before, or am I venturing into uncharted
territory here?

          Daniel

-- 
Unfortunately, the 'Tall Poppy' syndrome is rife in Australia, it is based on
pure and simple envy. This syndrome demands that anyone that is successful and
rises above others should be cut down immediately.
        -- Ron Lawrence, _The Tall Poppy Syndrome_, 1998

Re: Gnus SPAM support, and email based reporting.

[Derrell.Lipman]

Daniel Pittman <daniel@rimspace.net> writes:

> So, to report an incorrectly classified message, I need to resend the
> message as-is to a specific address on the server, then delete it
> locally.

I would begin using the gnus spam support as well, if that feature were
available!  (I haven't seen anything indicating that it is.)

Derrell

Re: Gnus SPAM support, and email based reporting.

[Ted Zlatanov]

On Sat, 17 Apr 2004, Derrell.Lipman@unwireduniverse.com wrote:

> Daniel Pittman <daniel@rimspace.net> writes:
> 
>> So, to report an incorrectly classified message, I need to resend
>> the message as-is to a specific address on the server, then delete
>> it locally.
> 
> I would begin using the gnus spam support as well, if that feature
> were available!  (I haven't seen anything indicating that it is.)

We can add something like Jochen Kupper's code to spam-report.el if
everyone is OK with it.  Patches are preferred, I won't be able to
get to it right away.

Ted

Re: Gnus SPAM support, and email based reporting.

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> We can add something like Jochen Kupper's code to spam-report.el if
> everyone is OK with it.

Note that there are two different ways of reporting the spam: 
 - as resends 
 - MIME attachments

Jochen's code seems to support the MIME attachment approach.  There's
also some use of digests in there that I don't quite understand?
Doesn't the digesting in Gnus strip headers from the digested
messages?

In any case, I would like it to be possible to set the reporting
method (ie. resend or MIME attachment), as well as the reporting
address, in the Group parameters (they may be different for different
IMAP servers, and even IMAP folders on the same server, in my case).

Re: Gnus SPAM support, and email based reporting.

[Jochen Küpper]

On Thu, 22 Apr 2004 11:10:57 +0200 Steinar Bang wrote:

>>>>>> Ted Zlatanov <tzz-mIZUurteI1BWk0Htik3J/w@public.gmane.org>:

>> We can add something like Jochen Kupper's code to spam-report.el if
>> everyone is OK with it.

Should probably be written a little nicer...

Steinar> Note that there are two different ways of reporting the spam: 
Steinar> - as resends 
Steinar> - MIME attachments

Agreed.

Steinar> Jochen's code seems to support the MIME attachment approach.  

Yep, that's what we need here.

Steinar> There's also some use of digests in there that I don't quite
Steinar> understand? 

Me neither;) The way I use it is that I process mark all
ill-classified articles and then send them with a single command
(jk/spam-report, bound to "$").

Steinar> Doesn't the digesting in Gnus strip headers from the digested
Steinar> messages?

I have the following (should probably be set temporarily inside the
function then).
,----[ C-h v gnus-uu-digest-headers RET ]
| gnus-uu-digest-headers's value is nil
| 
| Documentation:
| *List of regexps to match headers included in digested messages.
| The headers will be included in the sequence they are matched.  If nil
| include all headers.
| 
| You can customize this variable.
| 
| Defined in `gnus-uu'.
`----

Steinar> In any case, I would like it to be possible to set the reporting
Steinar> method (ie. resend or MIME attachment), as well as the reporting
Steinar> address, in the Group parameters

Seems totally reasonable... 


I'd like to also move the messages to certain Groups when reported,
maybe I get to do that sometime soon.

Greetings,
Jochen
-- 
Einigkeit und Recht und Freiheit                http://www.Jochen-Kuepper.de
    Liberté, Égalité, Fraternité                GnuPG key: CC1B0B4D
        (Part 3 you find in my messages before fall 2003.)

Re: Gnus SPAM support, and email based reporting.

[Ted Zlatanov]

On Thu, 22 Apr 2004, jochen@fhi-berlin.mpg.de wrote:

> I'd like to also move the messages to certain Groups when reported,
> maybe I get to do that sometime soon.

If you don't want to use the spam.el spam-process-destination
parameter, just look at spam.el for how it's done automatically and
you can reuse that code.

Ted

Re: Gnus SPAM support, and email based reporting.

[Jochen Küpper]

On Thu, 22 Apr 2004 16:11:05 -0400 Ted Zlatanov wrote:

Ted> On Thu, 22 Apr 2004, jochen-+It19tn3Rl9sbm7dSapR3bNAH6kLmebB@public.gmane.org wrote:

>> I'd like to also move the messages to certain Groups when reported,
>> maybe I get to do that sometime soon.

Ted> If you don't want to use the spam.el spam-process-destination
Ted> parameter, just look at spam.el for how it's done automatically
Ted> and you can reuse that code.

I must confess I never found the time to look at the spam.el docs (nor
code) so far.

Well, I simply put a gnus-summary-save-process-mark /
gnus-summary-yank-process-mark pair around the spam reporting and use
gnus-summary-move-article to get the articles moved away... Seems to
work nicely (for a few hours;)

Greetings,
Jochen
-- 
Einigkeit und Recht und Freiheit                http://www.Jochen-Kuepper.de
    Liberté, Égalité, Fraternité                GnuPG key: CC1B0B4D
        (Part 3 you find in my messages before fall 2003.)

Re: Gnus SPAM support, and email based reporting.

[Ted Zlatanov]

On Thu, 22 Apr 2004, jochen@fhi-berlin.mpg.de wrote:

On Thu, 22 Apr 2004 16:11:05 -0400 Ted Zlatanov wrote: Ted> On Thu, 22 Apr 2004, jochen@fhi-berlin.mpg.de wrote:
> 
>>> I'd like to also move the messages to certain Groups when
>>> reported, maybe I get to do that sometime soon.
> 
> Ted> If you don't want to use the spam.el spam-process-destination
> Ted> parameter, just look at spam.el for how it's done automatically
> Ted> and you can reuse that code.
> 
> I must confess I never found the time to look at the spam.el docs
> (nor code) so far.
> 
> Well, I simply put a gnus-summary-save-process-mark /
> gnus-summary-yank-process-mark pair around the spam reporting and
> use gnus-summary-move-article to get the articles moved
> away... Seems to work nicely (for a few hours;)

I think that's what spam.el does as well.

Ted

Re: Gnus SPAM support, and email based reporting.

[Ted Zlatanov]

On Thu, 22 Apr 2004, sb@dod.no wrote:

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
> 
>> We can add something like Jochen Kupper's code to spam-report.el if
>> everyone is OK with it.
> 
> Note that there are two different ways of reporting the spam: 
>  - as resends 
>  - MIME attachments

We'll just add a digest and attachment version of the reporter:

(defun spam-report-resend-attachment (&rest articles)
...)
(defun spam-report-resend-digest (&rest articles)
...)

and then the user can use the one they want; we can even have an
alias spam-report-resend that can be bound to the appropriate
function.

I'll be glad to review it as a patch, like I said I can't do the
whole thing right now :(

> In any case, I would like it to be possible to set the reporting
> method (ie. resend or MIME attachment), as well as the reporting
> address, in the Group parameters (they may be different for
> different IMAP servers, and even IMAP folders on the same server, in
> my case).

The reporting method is just a spam exit processor.

The reporting address can be a group parameter, sure.

Ted

Re: Gnus SPAM support, and email based reporting.

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> On Thu, 22 Apr 2004, sb@dod.no wrote:

>> Note that there are two different ways of reporting the spam: 
>> - as resends 
>> - MIME attachments

> We'll just add a digest and attachment version of the reporter:

> (defun spam-report-resend-attachment (&rest articles)
> ...)
> (defun spam-report-resend-digest (&rest articles)
> ...)

That would make it three different ways.  By "resend" I meant what
happens when you do `S D r'.  It's what I need for one of my
addresses. 

The difference between resend attachment and resend digest is that
with the latter, the message/rfc822 parts are wrapped inside a
multipart/digest, and the first part should be skipped when unpacking.
I don't know how significant this difference would be for the
unpacking filter...?

Re: Gnus SPAM support, and email based reporting.

[Daniel Pittman]

On Fri, 23 Apr 2004, Steinar Bang wrote:
>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
> 
>> On Thu, 22 Apr 2004, sb@dod.no wrote:
> 
>>> Note that there are two different ways of reporting the spam: 
>>> - as resends 
>>> - MIME attachments
> 
>> We'll just add a digest and attachment version of the reporter:
> 
>> (defun spam-report-resend-attachment (&rest articles)
>> ...)
>> (defun spam-report-resend-digest (&rest articles)
>> ...)
> 
> That would make it three different ways.  By "resend" I meant what
> happens when you do `S D r'.  It's what I need for one of my
> addresses.

That, also, is what I need to do, because...

> The difference between resend attachment and resend digest is that
> with the latter, the message/rfc822 parts are wrapped inside a
> multipart/digest, and the first part should be skipped when unpacking.
> I don't know how significant this difference would be for the
> unpacking filter...?

...at the moment I don't have an 'unpack' filter, I just deliver to an
mbox on the server, and that is then fed whole into the Baysean learning
tool.

Having to write a filter to extract the message content from a digest
before sending it into the learning package would not make me happy. :)

       Daniel
-- 
The lives of the best of us are spent in choosing between evils.
        -- Junius

Re: Gnus SPAM support, and email based reporting.

[Ted Zlatanov]

On Fri, 23 Apr 2004, sb@dod.no wrote:

>>>>>> Ted Zlatanov <tzz@lifelogs.com>:
> 
>> On Thu, 22 Apr 2004, sb@dod.no wrote:
> 
>>> Note that there are two different ways of reporting the spam: 
>>> - as resends 
>>> - MIME attachments
> 
>> We'll just add a digest and attachment version of the reporter:
> 
>> (defun spam-report-resend-attachment (&rest articles)
>> ...)
>> (defun spam-report-resend-digest (&rest articles)
>> ...)
> 
> That would make it three different ways.  By "resend" I meant what
> happens when you do `S D r'.  It's what I need for one of my
> addresses. 

The idea is that the spam exit processor will do it automatically
when you exit the group, so you don't need S D r or anything like
that - just to mark spam articles.

> The difference between resend attachment and resend digest is that
> with the latter, the message/rfc822 parts are wrapped inside a
> multipart/digest, and the first part should be skipped when
> unpacking.  I don't know how significant this difference would be
> for the unpacking filter...?

Let's list the possibilities and we'll see if they should all be
separate spam exit processors or we can generalize to 2 or 3
processors and do the rest with group/topic parameters.

Ted

Re: Gnus SPAM support, and email based reporting.

[Daniel Pittman]

On 22 Apr 2004, Ted Zlatanov wrote:
> On Sat, 17 Apr 2004, Derrell.Lipman@unwireduniverse.com wrote:
>
>> Daniel Pittman <daniel@rimspace.net> writes:
>>
>>> So, to report an incorrectly classified message, I need to resend
>>> the message as-is to a specific address on the server, then delete
>>> it locally.
>>
>> I would begin using the gnus spam support as well, if that feature
>> were available!  (I haven't seen anything indicating that it is.)
>
> We can add something like Jochen Kupper's code to spam-report.el if
> everyone is OK with it.  Patches are preferred, I won't be able to
> get to it right away.

Well, having a quick poke around turned up some bits and pieces of code
that I got to do the right thing pretty easily in terms of doing the
resend...

I had a look at spam.el to see if I could work out how to hook it up to
the actual spam mark stuff and, boy, was that a shock.

I got lost somewhere around the second or third abstraction around what
really did look like very much the same code.

Ah, well.  Hopefully someone who knows the code better can point the way
to getting this registered to process spam messages on exit, and post a
short howto guide that tells me:

1. how to get the spam mark code activated
2. how to get the 'report by resend' code used
3. how to *not* use any of the other features

Thanks,
        Daniel

Index: spam-report.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/spam-report.el,v
retrieving revision 7.7
diff -u -u -r7.7 spam-report.el
--- spam-report.el	25 Feb 2004 23:53:49 -0000	7.7
+++ spam-report.el	15 May 2004 13:22:15 -0000
@@ -79,6 +79,13 @@
   :type 'file
   :group 'spam-report)
 
+
+(defcustom spam-report-resend-to nil
+  "Email address that spam articles are resent to when reporting."
+  :type 'string
+  :group 'spam-report)
+
+
 (defvar spam-report-url-ping-temp-agent-function nil
   "Internal variable for `spam-report-agentize' and `spam-report-deagentize'.
 This variable will store the value of `spam-report-url-ping-function' from
@@ -108,6 +115,23 @@
 		(spam-report-url-ping host report))
 	    (gnus-message 3 "Could not find X-Report-Spam in article %d..."
 			  article)))))))
+
+
+(defun spam-report-resend (&rest articles)
+  "Report an article as spam by resending via email."
+  (dolist (article articles)
+    (gnus-message 6 "Reporting spam article %d to <%s>..." article spam-report-resend-to)
+    ;; This is ganked from the `gnus-summary-resend-message' function.
+    ;; It involves rendering the SPAM, which is undesirable, but there does
+    ;; not seem to be a nicer way to achieve this to my limited knowledge.
+    ;; -- <daniel@rimspace.net>
+    ;; select this particular article
+    (gnus-summary-select-article nil nil nil article)
+    ;; resend it to the destination address
+    (save-excursion
+      (set-buffer gnus-original-article-buffer)
+      (message-resend spam-report-resend-to))))
+    
 
 (defun spam-report-url-ping (host report)
   "Ping a host through HTTP, addressing a specific GET resource using

-- 
It is preoccupation with possessions, more than anything else,
that prevents us from living freely and nobly.
        -- Bertrand Russell

Re: Gnus SPAM support, and email based reporting.

[Ted Zlatanov]

[-- Attachment #1: Type: text/plain, Size: 1933 bytes --]

On Sat, 15 May 2004, daniel@rimspace.net wrote:

> Well, having a quick poke around turned up some bits and pieces of code
> that I got to do the right thing pretty easily in terms of doing the
> resend...

Are your FSF papers on file?  I don't know how to check.

> I had a look at spam.el to see if I could work out how to hook it up to
> the actual spam mark stuff and, boy, was that a shock.
> 
> I got lost somewhere around the second or third abstraction around what
> really did look like very much the same code.

I'm planning to do a major overhaul (version 3 of spam.el, so to
speak) where a lot of the unnecessary complexity is at least only
done once.  Probably by the end of the year.

> Ah, well.  Hopefully someone who knows the code better can point the way
> to getting this registered to process spam messages on exit, and post a
> short howto guide that tells me:
> 
> 1. how to get the spam mark code activated

spam.el sets up group exit actions that do things with spam
articles.  You can do your own actions on those articles, there's
nothing about spam.el that makes it mandatory for handling the
spam-marked articles.

> 2. how to get the 'report by resend' code used

With spam.el, you just make (spam spam-use-resend) the exit
processor of the group.  All spam articles will be processed
through it.

> 3. how to *not* use any of the other features

Don't enable any of the spam-use-* variables, INCLUDING
spam-use-resend, because spam-use-resend is not a variable, it's
only a symbol name.  Don't call spam-split in your split rules.
Call spam-initialize to load spam.el.  That should do it.

I'm attaching the patch that sets up a new spam exit processor for
this purpose.  Please test and get back to me if you are OK with
it.

The Gmane code, by the way (not the spam-use-gmane-xref spam
check, but the (spam spam-use-gmane) spam exit processor) is what
spam-report-resend follows as an example.

Ted


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: resend.patch --]
[-- Type: text/x-patch, Size: 6385 bytes --]

? resend.patch
Index: gnus.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus.el,v
retrieving revision 7.18
diff -u -r7.18 gnus.el
--- gnus.el	12 May 2004 21:00:54 -0000	7.18
+++ gnus.el	18 May 2004 20:13:20 -0000
@@ -1896,6 +1896,7 @@
 	    (const :tag "Spam: Blacklist"     (spam spam-use-blacklist))
 	    (const :tag "Spam: Bsfilter"      (spam spam-use-bsfilter))
 	    (const :tag "Spam: Gmane Report"  (spam spam-use-gmane))
+	    (const :tag "Spam: Resend Message"(spam spam-use-resend))
 	    (const :tag "Spam: ifile"	      (spam spam-use-ifile))
 	    (const :tag "Spam: Spam Oracle"   (spam spam-use-spamoracle))
 	    (const :tag "Spam: Spam-stat"     (spam spam-use-stat))
Index: spam-report.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/spam-report.el,v
retrieving revision 7.7
diff -u -r7.7 spam-report.el
--- spam-report.el	25 Feb 2004 23:53:49 -0000	7.7
+++ spam-report.el	18 May 2004 20:13:20 -0000
@@ -79,12 +79,36 @@
   :type 'file
   :group 'spam-report)
 
+(defcustom spam-report-resend-to (or 
+				  (when (length user-mail-address) 
+				    user-mail-address)
+				  "nonexistent-user-please-fix@invalid.domain")
+  "Email address that spam articles are resent to when reporting."
+  :type 'string
+  :group 'spam-report)
+
 (defvar spam-report-url-ping-temp-agent-function nil
   "Internal variable for `spam-report-agentize' and `spam-report-deagentize'.
 This variable will store the value of `spam-report-url-ping-function' from
 before `spam-report-agentize' was run, so that `spam-report-deagentize' can
 undo that change.")
 
+(defun spam-report-resend (&rest articles)
+  "Report an article as spam by resending via email."
+  (dolist (article articles)
+    (gnus-message 6 
+		  "Reporting spam article %d to <%s>..." 
+		  article spam-report-resend-to)
+    ;; This is ganked from the `gnus-summary-resend-message' function.
+    ;; It involves rendering the SPAM, which is undesirable, but there does
+    ;; not seem to be a nicer way to achieve this.
+    ;; select this particular article
+    (gnus-summary-select-article nil nil nil article)
+    ;; resend it to the destination address
+    (save-excursion
+      (set-buffer gnus-original-article-buffer)
+      (message-resend spam-report-resend-to))))
+
 (defun spam-report-gmane (&rest articles)
   "Report an article as spam through Gmane"
   (dolist (article articles)
@@ -93,10 +117,11 @@
 		   (string-match spam-report-gmane-regex gnus-newsgroup-name)))
       (gnus-message 6 "Reporting spam article %d to spam.gmane.org..." article)
       (if spam-report-gmane-use-article-number
-	  (spam-report-url-ping "spam.gmane.org"
-				(format "/%s:%d"
-					(gnus-group-real-name gnus-newsgroup-name)
-					article))
+	  (spam-report-url-ping 
+	   "spam.gmane.org"
+	   (format "/%s:%d"
+		   (gnus-group-real-name gnus-newsgroup-name)
+		   article))
 	(with-current-buffer nntp-server-buffer
 	  (gnus-request-head article gnus-newsgroup-name)
 	  (goto-char (point-min))
Index: spam.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/spam.el,v
retrieving revision 7.37
diff -u -r7.37 spam.el
--- spam.el	17 May 2004 14:28:21 -0000	7.37
+++ spam.el	18 May 2004 20:13:20 -0000
@@ -55,7 +55,8 @@
 
 ;; autoload spam-report
 (eval-and-compile
-  (autoload 'spam-report-gmane "spam-report"))
+  (autoload 'spam-report-gmane "spam-report")
+  (autoload 'spam-report-resend "spam-report"))
 
 ;; autoload gnus-registry
 (eval-and-compile
@@ -659,7 +660,9 @@
     nil))
 
 (defvar spam-list-of-processors
+  ;; note the resend and gmane processors are not defined in gnus.el
   '((gnus-group-spam-exit-processor-report-gmane spam spam-use-gmane)
+    (gnus-group-spam-exit-processor-report-resend spam spam-use-resend)
     (gnus-group-spam-exit-processor-bogofilter   spam spam-use-bogofilter)
     (gnus-group-spam-exit-processor-bsfilter	 spam spam-use-bsfilter)
     (gnus-group-spam-exit-processor-blacklist    spam spam-use-blacklist)
@@ -705,6 +708,9 @@
 (defun spam-group-spam-processor-report-gmane-p (group)
   (spam-group-processor-p group 'gnus-group-spam-exit-processor-report-gmane))
 
+(defun spam-group-spam-processor-report-resend-p (group)
+  (spam-group-processor-p group 'gnus-group-spam-exit-processor-report-resend))
+
 (defun spam-group-spam-processor-bogofilter-p (group)
   (spam-group-processor-p group 'gnus-group-spam-exit-processor-bogofilter))
 
@@ -742,13 +748,23 @@
   (spam-group-processor-p group 'gnus-group-ham-exit-processor-spamoracle))
 
 (defun spam-report-articles-gmane (n)
-  "Report the current message as spam.
+  "Report the current message as spam via Gmane.
 Respects the process/prefix convention."
   (interactive "P")
   (dolist (article (gnus-summary-work-articles n))
     (gnus-summary-remove-process-mark article)
     (spam-report-gmane article)))
 
+(defun spam-report-articles-resend (n)
+  "Report the current message as spam by resending it.
+Respects the process/prefix convention.  Also see
+`spam-report-resend-to'."
+  (interactive "P")
+  (let ((articles (gnus-summary-work-articles n)))
+    (spam-report-resend articles)
+    (dolist (article articles)
+      (gnus-summary-remove-process-mark article))))
+
 (defun spam-necessary-extra-headers ()
   "Return the extra headers spam.el thinks are necessary."
   (let (list)
@@ -1323,12 +1339,16 @@
 			 spam-stat-register-spam-routine
 			 spam-stat-unregister-ham-routine
 			 spam-stat-unregister-spam-routine)
-    ;; note that spam-use-gmane is not a legitimate check
+    ;; note that spam-use-gmane and spam-use-resend are not legitimate checks
     (spam-use-gmane      nil
 			 spam-report-gmane-register-routine
 			 ;; does Gmane support unregistration?
 			 nil
 			 nil)
+    (spam-use-resend     nil
+			 spam-report-resend-register-routine
+			 nil
+			 nil)
     (spam-use-spamassassin spam-spamassassin-register-ham-routine
 			   spam-spamassassin-register-spam-routine
 			   spam-spamassassin-unregister-ham-routine
@@ -2055,6 +2075,9 @@
 (defun spam-report-gmane-register-routine (articles)
   (when articles
     (apply 'spam-report-gmane articles)))
+
+(defun spam-report-resend-register-routine (articles)
+  (spam-report-resend articles))
 
 \f
 ;;;; Bogofilter

Re: Gnus SPAM support, and email based reporting.

[Daniel Pittman]

On 19 May 2004, Ted Zlatanov wrote:
> On Sat, 15 May 2004, daniel@rimspace.net wrote:

Sorry for taking so long to get back to you on this; real life got in
the way.

>> Well, having a quick poke around turned up some bits and pieces of
>> code that I got to do the right thing pretty easily in terms of doing
>> the resend...
>
> Are your FSF papers on file?  I don't know how to check.

Yes, I have papers for Gnus on file. IIRC, the only way to check is to
mail the people at fencepost and ask, or to remember, or to ask the
submitter. :)

>> I had a look at spam.el to see if I could work out how to hook it up
>> to the actual spam mark stuff and, boy, was that a shock.
>>
>> I got lost somewhere around the second or third abstraction around
>> what really did look like very much the same code.
>
> I'm planning to do a major overhaul (version 3 of spam.el, so to
> speak) where a lot of the unnecessary complexity is at least only
> done once.  Probably by the end of the year.

It struck me that there was an awful lot of writing of wrapper functions
around what is, at heart, pipe to command A to learn as spam, or command
B to learn as ham.

OTOH, I don't really understand your code so well and I am probably
missing the reason why a bunch of this complexity exists.

[...]

>> 2. how to get the 'report by resend' code used
>
> With spam.el, you just make (spam spam-use-resend) the exit
> processor of the group.  All spam articles will be processed
> through it.

That didn't work.  I needed to add to the group parameters:

    (spam-process (gnus-group-spam-exit-processor-report-resend))

The other one didn't trigger, for some reason.

[...]

> I'm attaching the patch that sets up a new spam exit processor for
> this purpose.  Please test and get back to me if you are OK with
> it.

It worked, and let me debug my code.  I had to make the following change
to current CVS, since `spam-report-resend' is passed a list of articles,
not a variable number of arguments.

Er, and finally, I think that it would be better to set `defcustom
spam-report-resend-to' to `nil' by default.

Then, in the report function we could test that address and if it was
`nil', display an error to the user and instruct spam.el to mark the
article as "not processed"

That seems more user-friendly to me.  Alternately, we could actually
prompt for the destination and save that with custom, but I don't know
how to do that - custom is still a mysterious (and probably explosive)
black box to me. :)

        Daniel


] cvs diff -u lisp/spam.el
Index: lisp/spam-report.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/spam-report.el,v
retrieving revision 7.9
diff -u -r7.9 spam-report.el
--- lisp/spam-report.el 20 May 2004 18:32:35 -0000      7.9
+++ lisp/spam-report.el 24 May 2004 11:06:07 -0000
@@ -93,7 +93,7 @@
 before `spam-report-agentize' was run, so that `spam-report-deagentize' can
 undo that change.")
 
-(defun spam-report-resend (&rest articles)
+(defun spam-report-resend (articles)
   "Report an article as spam by resending via email."
   (dolist (article articles)
     (gnus-message 6 
@@ -133,6 +133,7 @@
                (spam-report-url-ping host report))
            (gnus-message 3 "Could not find X-Report-Spam in article %d..."
                          article)))))))
+
 
 (defun spam-report-url-ping (host report)
   "Ping a host through HTTP, addressing a specific GET resource using


-- 
Champagne has the taste of an apple peeled with a steel knife.
        -- Aldous Huxley

Re: Gnus SPAM support, and email based reporting.

[Kai Grossjohann]

Daniel Pittman <daniel@rimspace.net> writes:

> That seems more user-friendly to me.  Alternately, we could actually
> prompt for the destination and save that with custom, but I don't know
> how to do that - custom is still a mysterious (and probably explosive)
> black box to me. :)

C-h f customize-set-variable RET

Kai

Re: Gnus SPAM support, and email based reporting.

[Daniel Pittman]

On 24 May 2004, Kai Grossjohann wrote:
> Daniel Pittman <daniel@rimspace.net> writes:
>
>> That seems more user-friendly to me.  Alternately, we could actually
>> prompt for the destination and save that with custom, but I don't know
>> how to do that - custom is still a mysterious (and probably explosive)
>> black box to me. :)
>
> C-h f customize-set-variable RET

So, that is all that is needed to set the value on user input?

Will custom handle prompting the user about saving the changes and
everything else?  If so, that is much less frightening than I expected.
:)

        Daniel
-- 
Keep a diary and one day it'll keep you.
        -- Mae West

Re: Gnus SPAM support, and email based reporting.

[Kai Grossjohann]

Daniel Pittman <daniel@rimspace.net> writes:

> On 24 May 2004, Kai Grossjohann wrote:
>> Daniel Pittman <daniel@rimspace.net> writes:
>>
>>> That seems more user-friendly to me.  Alternately, we could actually
>>> prompt for the destination and save that with custom, but I don't know
>>> how to do that - custom is still a mysterious (and probably explosive)
>>> black box to me. :)
>>
>> C-h f customize-set-variable RET
>
> So, that is all that is needed to set the value on user input?
>
> Will custom handle prompting the user about saving the changes and
> everything else?  If so, that is much less frightening than I expected.
> :)

I think it doesn't save.  For this, customize-save-customized might
be useful.

But I don't know how to save a single option.

Kai

Re: Gnus SPAM support, and email based reporting.

[Michael Schierl]

Kai Grossjohann <kai@emptydomain.de> writes:

> I think it doesn't save.  For this, customize-save-customized might
> be useful.
>
> But I don't know how to save a single option.

(customize-save-variable 'foo 'bar) does the trick for me...

Michael

Re: Gnus SPAM support, and email based reporting.

[Ted Zlatanov]

On Mon, 24 May 2004, daniel@rimspace.net wrote:

On 19 May 2004, Ted Zlatanov wrote:

>> I'm planning to do a major overhaul (version 3 of spam.el, so to
>> speak) where a lot of the unnecessary complexity is at least only
>> done once.  Probably by the end of the year.
> 
> It struck me that there was an awful lot of writing of wrapper functions
> around what is, at heart, pipe to command A to learn as spam, or command
> B to learn as ham.
> 
> OTOH, I don't really understand your code so well and I am probably
> missing the reason why a bunch of this complexity exists.

The complexity is necessary to accomodate all the spam detection and
training backends.  BBDB, for instance, is VERY different from
Bogofilter.  Some backends don't detect incoming spam (the reporters),
others don't learn (spam-use-blackholes).  Like I said, I'll try to
simplify it, but the code will have to remain very abstract to handle
all the possible backends.  There will probably be
spam-register-backend which will handle all the various types of
backends.

>> With spam.el, you just make (spam spam-use-resend) the exit
>> processor of the group.  All spam articles will be processed
>> through it.
> 
> That didn't work.  I needed to add to the group parameters:
> 
>     (spam-process (gnus-group-spam-exit-processor-report-resend))
> 
> The other one didn't trigger, for some reason.

Looks like a bug in my code, because
gnus-group-spam-exit-processor-report-resend is supposed to be no
longer necessary.  I'll look today.

> Er, and finally, I think that it would be better to set `defcustom
> spam-report-resend-to' to `nil' by default.
> 
> Then, in the report function we could test that address and if it was
> `nil', display an error to the user and instruct spam.el to mark the
> article as "not processed"
> 
> That seems more user-friendly to me.  Alternately, we could actually
> prompt for the destination and save that with custom, but I don't know
> how to do that - custom is still a mysterious (and probably explosive)
> black box to me. :)

The resend-to address should be a Gnus group/topic variable that
overrides the default spam-report-resend-to, I think.  What do you think?

I applied your bug fix, thanks!

Ted

Re: Gnus SPAM support, and email based reporting.

[Daniel Pittman]

On 24 May 2004, Ted Zlatanov wrote:
> On Mon, 24 May 2004, daniel@rimspace.net wrote:
> On 19 May 2004, Ted Zlatanov wrote:
>
>>> I'm planning to do a major overhaul (version 3 of spam.el, so to
>>> speak) where a lot of the unnecessary complexity is at least only
>>> done once.  Probably by the end of the year.
>>
>> It struck me that there was an awful lot of writing of wrapper functions
>> around what is, at heart, pipe to command A to learn as spam, or command
>> B to learn as ham.
>>
>> OTOH, I don't really understand your code so well and I am probably
>> missing the reason why a bunch of this complexity exists.
>
> The complexity is necessary to accomodate all the spam detection and
> training backends.  BBDB, for instance, is VERY different from
> Bogofilter.  Some backends don't detect incoming spam (the reporters),
> others don't learn (spam-use-blackholes).  Like I said, I'll try to
> simplify it, but the code will have to remain very abstract to handle
> all the possible backends.  There will probably be
> spam-register-backend which will handle all the various types of
> backends.

*nod*  I should say that I don't mean to criticize your work. Getting
this all hanging together seems like a large workload.  Thanks for doing
that.

[...]

>> Er, and finally, I think that it would be better to set `defcustom
>> spam-report-resend-to' to `nil' by default.
>>
>> Then, in the report function we could test that address and if it was
>> `nil', display an error to the user and instruct spam.el to mark the
>> article as "not processed"
>>
>> That seems more user-friendly to me.  Alternately, we could actually
>> prompt for the destination and save that with custom, but I don't know
>> how to do that - custom is still a mysterious (and probably explosive)
>> black box to me. :)
>
> The resend-to address should be a Gnus group/topic variable that
> overrides the default spam-report-resend-to, I think. What do you
> think?

I agree with that; it makes sense to be able to treat it just like the
other spam settings.  The default prompting thing would also be good,
though, for the better "out of the box" experience, I think.

Give me a minute...

...and this should implement precisely that.  No handling of the group
stuff yet, because I am not sure where that is best hooked in.

I *think* it would be best done by fetching it in the spam.el code, then
using `let' to bind `spam-report-resend-to' for the duration of the
call...

Feels good to be writing some code again. Mmmm, code.

        Daniel

Index: spam-report.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/spam-report.el,v
retrieving revision 7.9
diff -u -u -r7.9 spam-report.el
--- spam-report.el	20 May 2004 18:32:35 -0000	7.9
+++ spam-report.el	24 May 2004 14:16:53 -0000
@@ -79,11 +79,10 @@
   :type 'file
   :group 'spam-report)
 
-(defcustom spam-report-resend-to (or 
-				  (when (length user-mail-address) 
-				    user-mail-address)
-				  "nonexistent-user-please-fix@invalid.domain")
-  "Email address that spam articles are resent to when reporting."
+(defcustom spam-report-resend-to nil
+  "Email address that spam articles are resent to when reporting.
+If not set, the user will be prompted to enter a value which will be
+saved for future use."
   :type 'string
   :group 'spam-report)
 
@@ -93,8 +92,11 @@
 before `spam-report-agentize' was run, so that `spam-report-deagentize' can
 undo that change.")
 
-(defun spam-report-resend (&rest articles)
+(defun spam-report-resend (articles)
   "Report an article as spam by resending via email."
+  (unless spam-report-resend-to
+    (customize-set-variable spam-report-resend-to
+                            (read-from-minibuffer "email address to resend SPAM to? ")))
   (dolist (article articles)
     (gnus-message 6 
 		  "Reporting spam article %d to <%s>..." 
@@ -133,6 +135,7 @@
 		(spam-report-url-ping host report))
 	    (gnus-message 3 "Could not find X-Report-Spam in article %d..."
 			  article)))))))
+
 
 (defun spam-report-url-ping (host report)
   "Ping a host through HTTP, addressing a specific GET resource using



-- 
The true believer is eternally incomplete, eternally insecure.
        -- Eric Hoffer

Re: Gnus SPAM support, and email based reporting.

[Ted Zlatanov]

On Tue, 25 May 2004, daniel@rimspace.net wrote:

> I agree with that; it makes sense to be able to treat it just like the
> other spam settings.  The default prompting thing would also be good,
> though, for the better "out of the box" experience, I think.
> 
> Give me a minute...
> 
> ...and this should implement precisely that.  

I'm not sure I like this auto-magic setting on first use.  Does anyone
else like/dislike it?  Seems to me like we should choose a valid
default or nil, but not ask the user unless they want to be asked.
Of course, I could be wrong, but nowhere else in Gnus is this used
AFAIK.

> No handling of the group stuff yet, because I am not sure where that
> is best hooked in.

I looked at gnus.el, and the attached patch adds the parameter.  Let
me know if the *parameter* looks OK and I can add the rest of the glue
to make spam.el do a lexical let on spam-report-resend-to.

Ted

Index: gnus.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus.el,v
retrieving revision 7.21
diff -u -r7.21 gnus.el
--- gnus.el     20 May 2004 18:53:28 -0000      7.21
+++ gnus.el     24 May 2004 17:55:59 -0000
@@ -1823,6 +1823,24 @@
    "The spam classification (spam, ham, or neither) of this group.
 When a spam group is entered, all unread articles are marked as spam.")
 
+  (gnus-define-group-parameter
+   spam-resend-to
+   :type list
+   :function-document
+   "The address to get spam resent (through spam-report-resend)."
+   :variable gnus-spam-resend-to
+   :variable-default nil
+   :variable-document
+   "The address to get spam resent (through spam-report-resend)."
+   :variable-group spam
+   :variable-type '(repeat
+                   (list :tag "Group address for resending spam"
+                         (regexp :tag "Group")
+                         (string :tag "E-mail address for resending spam (requires the spam-use-resend exit processor)")))
+   :parameter-type 'string :tag "E-mail address for resending spam (requires the spam-use-resend exit processor)"
+   :parameter-document
+   "The address to get spam resent (through spam-report-resend).")
+
   (defvar gnus-group-spam-exit-processor-ifile "ifile"
     "OBSOLETE: The ifile summary exit spam processor.")

Re: Gnus SPAM support, and email based reporting.

[Daniel Pittman]

On 25 May 2004, Ted Zlatanov wrote:
> On Tue, 25 May 2004, daniel@rimspace.net wrote:
>
>> I agree with that; it makes sense to be able to treat it just like the
>> other spam settings.  The default prompting thing would also be good,
>> though, for the better "out of the box" experience, I think.
>>
>> Give me a minute...
>>
>> ...and this should implement precisely that.  
>
> I'm not sure I like this auto-magic setting on first use.  Does anyone
> else like/dislike it?  Seems to me like we should choose a valid
> default or nil, but not ask the user unless they want to be asked.
> Of course, I could be wrong, but nowhere else in Gnus is this used
> AFAIK.

Well, I don't believe that we *can* find a valid default value.
I certainly know of nothing that would work for ever a small proportion
of people, let alone a majority.

This isn't unprecedented within Emacs either; PSGML already does this
for `user-email-address'.

I don't think that this should be the recommended way of setting the
value, either, just that it is the nicest option when we don't have a
value already - and it isn't likely to trigger in any automated setup
that I can see...

>> No handling of the group stuff yet, because I am not sure where that
>> is best hooked in.
>
> I looked at gnus.el, and the attached patch adds the parameter.  Let
> me know if the *parameter* looks OK and I can add the rest of the glue
> to make spam.el do a lexical let on spam-report-resend-to.

At the moment the code only deals with a string `spam-report-resend-to';
I wonder if it is best to support more than one address there...

Anyway, that looks reasonable to me, with the caveat that I don't know
that code well.

     Daniel
-- 
The secret of happiness is this: Let your interests be as wide as
possible, and let your reactions to the things and persons that
interest you be as far as possible friendly rather than hostile. 
        -- Bertrand Russell

Re: Gnus SPAM support, and email based reporting.

[Ted Zlatanov]

Daniel Pittman wrote:
> Well, I don't believe that we *can* find a valid default value.
> I certainly know of nothing that would work for ever a small proportion
> of people, let alone a majority.

> This isn't unprecedented within Emacs either; PSGML already does this
> for `user-email-address'.

OK, that's fine then.  Comitted.

> I don't think that this should be the recommended way of setting the
> value, either, just that it is the nicest option when we don't have a
> value already - and it isn't likely to trigger in any automated setup
> that I can see...

> >> No handling of the group stuff yet, because I am not sure where that
> >> is best hooked in.
> >
> > I looked at gnus.el, and the attached patch adds the parameter.  Let
> > me know if the *parameter* looks OK and I can add the rest of the glue
> > to make spam.el do a lexical let on spam-report-resend-to.

> At the moment the code only deals with a string `spam-report-resend-to';
> I wonder if it is best to support more than one address there...

No, I don't think that's necessary.  It would add extra complexity and
I really doubt it's all that useful to have multiple resend-to
addresses.

Try out the code I just comitted and let me know how it works for you.

Ted

spam.el processor code refactored (was: Gnus SPAM support, and email based reporting.)

[Ted Zlatanov]

Ted wrote:
> Looks like a bug in my code, because
> gnus-group-spam-exit-processor-report-resend is supposed to be no
> longer necessary.  I'll look today.

I changed all the processor-p functionality to be simpler and
better.  I also removed the requirement for the obsolete processor
symbols in spam-list-of-processors.  If Gnus acts strange (spam
doesn't get processed, you get messages from Maxwell's demon, etc.)
let me know.

Ted

Re: Gnus SPAM support, and email based reporting.

[Jochen Küpper]

On Sat, 17 Apr 2004 22:40:45 +1000 Daniel Pittman wrote:

Daniel> So, to report an incorrectly classified message, I need to
Daniel> resend the message as-is to a specific address on the server,
Daniel> then delete it locally.

Well, I have to send it as an MIME attachment, but his is what I use;
might be a starting point:


(defun jk/ham-spam-report (to-address)
  "Digest and forward all articles in this series to the specified address."
  (let
      ((gnus-uu-save-in-digest t)
       (file (mm-make-temp-file (nnheader-concat gnus-uu-tmp-dir "spam-report")))
       (message-forward-as-mime t)
       (mail-parse-charset gnus-newsgroup-charset)
       (mail-parse-ignored-charsets gnus-newsgroup-ignored-charsets)
       gnus-uu-digest-buffer subject from)
    (let ((gnus-article-reply (gnus-summary-work-articles nil)))
      (gnus-setup-message 'forward
	(setq gnus-uu-digest-from-subject nil)
	(setq gnus-uu-digest-buffer
	      (gnus-get-buffer-create " *gnus spam-report*"))
	(gnus-uu-decode-save nil file)
	(switch-to-buffer gnus-uu-digest-buffer)
	(let ((message-forward-decoded-p t))
	  (message-forward nil t))))
    (setq gnus-uu-digest-from-subject nil)
    ;; Ok, finish mail
    ;; Subject:
    (goto-char (point-min))
    (when (re-search-forward "^Subject: ")
      (delete-region (point) (point-at-eol))
      (insert "ham/spam report"))
    ;; To:
    (goto-char (point-min))
    (when (re-search-forward "^To:")
      (delete-region (point) (point-at-eol))
      (insert " " to-address))
    ;; Gcc
    (goto-char (point-min))
    (when (re-search-forward "^Gcc:")
      (delete-region (point-at-bol) (point-at-eol))
      (insert " nnimap+FHI:sent/" (format-time-string "%Y/%m")
              " nnimap+FHI:spam/reported"))
    ;; remove signature
    (when (message-goto-signature)
      (delete-region (- (point) 4) (point-max)))
    ;; and send
    (message-send-and-exit)
    ;; finally, mark articles expire-able
    ))

(defun jk/ham-report ()
  "Digest and forward all articles in this series to nospam-+It19tn3Rl8lNQzPPfjNRg==@public.gmane.orgde."
  (interactive)
  (jk/ham-spam-report "nospam-o3gYboyLymOzQB+pC5nmwQ==@public.gmane.org"))

(defun jk/spam-report ()
  "Digest and forward all articles in this series to spam-+It19tn3Rl9sbm7dSapR3bNAH6kLmebB@public.gmane.org."
  (interactive)
  (jk/ham-spam-report "spam-o3gYboyLymOzQB+pC5nmwQ==@public.gmane.org"))


Greetings,
Jochen
-- 
Einigkeit und Recht und Freiheit                http://www.Jochen-Kuepper.de
    Liberté, Égalité, Fraternité                GnuPG key: CC1B0B4D
        (Part 3 you find in my messages before fall 2003.)