[spam-report.el] Offline Gmane spam reporting

[spam-report.el] Offline Gmane spam reporting

[Reiner Steib]

Hi,

when I'm reading Gmane groups offline, I cannot use the spam-report
functions.  Wouldn't it be nice to collect the requests (the URLs) in
a file and use them later when online?

The new function `spam-report-url-to-file' collects the request in a
file, see the attached patch[1].  I also added a custom choice in
`spam-report-url-ping-function' to insert a user defined function and
tried to improve the doc-string of this variable.

I'm not sure if we should use `spam-directory' or `gnus-directory' in
the default value of `spam-report-requests-file'.  Is `spam-report.el'
supposed to be independent from `spam.el'?  At least it doesn't
require it.  Somehow related to this: I think `spam.el' should _not_
hard-code "~/News/spam/" as `spam-directory' but use
`gnus-directory'[2] instead, see its doc-string.

I didn't care about code for using the requests file when being online
(yet).  Personally, I would just use...

  file=~/News/spam-report-requests.url
  GET `cat file` && : > file

... from the shell (or some ip-up script).  But we may also add an
interactive lisp function for that (loop over all lines in the file,
use `spam-report-url-ping-plain' and erase the file when done).

Unless there are objections, I will commit both patches along with an
update for the manual.

Bye, Reiner.

[1]
--8<---------------cut here---------------start------------->8---
--- spam-report.el.~6.7.~	Mon Jan  5 11:58:59 2004
+++ spam-report.el	Sun Feb  1 15:08:22 2004
@@ -59,12 +59,24 @@
 
 (defcustom spam-report-url-ping-function
   'spam-report-url-ping-plain
-  "Function to use for url ping spam reporting."
+  "Function to use for url ping spam reporting.
+The function must accept the arguments `host' and `report'."
   :type '(choice
 	  (const :tag "Connect directly"
 		 spam-report-url-ping-plain)
 	  (const :tag "Use the external program specified in `mm-url-program'"
-		 spam-report-url-ping-mm-url))
+		 spam-report-url-ping-mm-url)
+	  (const :tag "Store request URLs in `spam-report-requests-file'"
+		 spam-report-url-to-file)
+	  (function :tag "User defined function" nil))
+  :group 'spam-report)
+
+(defcustom spam-report-requests-file
+  (nnheader-concat gnus-directory "spam-report-requests.url")
+  ;; Is there a convention for the extension of such a file?
+  ;; Should we use `spam-directory'?
+  "File where spam report request are stored."
+  :type 'file
   :group 'spam-report)
 
 (defun spam-report-gmane (&rest articles)
@@ -121,6 +133,17 @@
     (let ((url (concat "http://" host "/" report)))
       (mm-url-insert url t))))
 
+(defun spam-report-url-to-file (host report)
+  "Collect spam report requests in `spam-report-requests-file'.
+Customize `spam-report-url-ping-function' to use this function."
+  (let ((url (concat "http://" host "/" report))
+	(file spam-report-requests-file))
+    (gnus-message 9 "Writing URL `%s' to file `%s'" url file)
+    (with-temp-buffer
+      (insert url)
+      (newline)
+      (append-to-file (point-min) (point-max) file))))
+
 (provide 'spam-report)
 
 ;;; spam-report.el ends here.
--8<---------------cut here---------------end--------------->8---

[2]
--8<---------------cut here---------------start------------->8---
--- spam.el.~6.142.~	Mon Jan 26 21:06:50 2004
+++ spam.el	Sun Feb  1 15:02:00 2004
@@ -73,7 +73,7 @@
 (defgroup spam nil
   "Spam configuration.")
 
-(defcustom spam-directory "~/News/spam/"
+(defcustom spam-directory (nnheader-concat gnus-directory "spam/")
   "Directory for spam whitelists and blacklists."
   :type 'directory
   :group 'spam)
--8<---------------cut here---------------end--------------->8---
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: [spam-report.el] Offline Gmane spam reporting

[Kai Grossjohann]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:

> I didn't care about code for using the requests file when being online
> (yet).  Personally, I would just use...
>
>   file=~/News/spam-report-requests.url
>   GET `cat file` && : > file
>
> ... from the shell (or some ip-up script).  But we may also add an
> interactive lisp function for that (loop over all lines in the file,
> use `spam-report-url-ping-plain' and erase the file when done).

All of what you have suggested is Good Stuff.

But the real killer would be integration with the agent: arrange it so
that reports are processed right away when online, and that they are
processed when sending the queue or something when offline.

WDYT?

Kai

Re: [spam-report.el] Offline Gmane spam reporting

[Reiner Steib]

On Sun, Feb 01 2004, Kai Grossjohann wrote:

> Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:
>> But we may also add an interactive lisp function for that (loop
>> over all lines in the file, use `spam-report-url-ping-plain' and
>> erase the file when done).

Let's call this function `spam-report-process-queue' for now.

> All of what you have suggested is Good Stuff.
>
> But the real killer would be integration with the agent: arrange it so
> that reports are processed right away when online, and that they are
> processed when sending the queue or something when offline.

I'm not familiar with the agent (I don't use it), but I guess this
would be quite easy.  Maybe like this:

(add-to-list 'gnus-agent-plugged-hook
	     (lambda ()
	       (spam-report-process-queue);; not implemented yet.
	       (setq spam-report-url-ping-function
		     'spam-report-url-ping-plain)))
(add-to-list 'gnus-agent-unplugged-hook
	     (lambda ()
	       (setq spam-report-url-ping-function
		     'spam-report-url-to-file)))

Or am I missing something?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: [spam-report.el] Offline Gmane spam reporting

[Kai Grossjohann]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:

> On Sun, Feb 01 2004, Kai Grossjohann wrote:
>
>> But the real killer would be integration with the agent: arrange it so
>> that reports are processed right away when online, and that they are
>> processed when sending the queue or something when offline.
>
> I'm not familiar with the agent (I don't use it), but I guess this
> would be quite easy.  Maybe like this:
>
> (add-to-list 'gnus-agent-plugged-hook
> 	     (lambda ()
> 	       (spam-report-process-queue);; not implemented yet.

Maybe integrate this into sending the queue?

> 	       (setq spam-report-url-ping-function
> 		     'spam-report-url-ping-plain)))
> (add-to-list 'gnus-agent-unplugged-hook
> 	     (lambda ()
> 	       (setq spam-report-url-ping-function
> 		     'spam-report-url-to-file)))

It might be useful to see what the agent does for sending mail.  I
think it works like this: on unplugging, set a variable to the current
value of message-send-mail-function, set message-send-mail-function to
the agent-queueing function.  On plugging, set
message-send-mail-function to the old value stashed away in the
special variable.

WDYT?

Kai

Re: [spam-report.el] Offline Gmane spam reporting

[Ted Zlatanov]

On Sun, 01 Feb 2004, 4.uce.03.r.s@nurfuerspam.de wrote:

> when I'm reading Gmane groups offline, I cannot use the spam-report
> functions.  Wouldn't it be nice to collect the requests (the URLs)
> in a file and use them later when online?
> 
> The new function `spam-report-url-to-file' collects the request in a
> file, see the attached patch[1].  I also added a custom choice in
> `spam-report-url-ping-function' to insert a user defined function
> and tried to improve the doc-string of this variable.

Very nice, thank you.  Go ahead and apply, but make sure you also
update gnus.el so the new spam-report function is an available spam
processor.  Let me know if you need help with any of it.

> I'm not sure if we should use `spam-directory' or `gnus-directory'
> in the default value of `spam-report-requests-file'.  Is
> `spam-report.el' supposed to be independent from `spam.el'?  At
> least it doesn't require it.  Somehow related to this: I think
> `spam.el' should _not_ hard-code "~/News/spam/" as `spam-directory'
> but use `gnus-directory'[2] instead, see its doc-string.

I would use spam-directory.  I definitely think spam-directory should
be different from gnus-directory, because of the high volume of data
that can accumulate in the blacklists for instance.  The default
should be (format "%s/spam" gnus-directory) so users can override it
iff they want to.  Do you agree?

> I didn't care about code for using the requests file when being
> online (yet).  Personally, I would just use...
> 
>   file=~/News/spam-report-requests.url
>   GET `cat file` && : > file
> 
> ... from the shell (or some ip-up script).  But we may also add an
> interactive lisp function for that (loop over all lines in the file,
> use `spam-report-url-ping-plain' and erase the file when done).

I agree with Kai, this should be doable when Gnus becomes plugged.
GET and such are not always available, and there's no need to
complicate the user's life.  The manual should say "if you want to
queue reports and you use the Agent, do (add-hook...".

Let me know which parts of the above you want me to do.  Thanks!

Ted

Re: [spam-report.el] Offline Gmane spam reporting

[Reiner Steib]

On Mon, Feb 02 2004, Ted Zlatanov wrote:

> On Sun, 01 Feb 2004, 4.uce.03.r.s@nurfuerspam.de wrote:
>
>> when I'm reading Gmane groups offline, I cannot use the spam-report
>> functions.  Wouldn't it be nice to collect the requests (the URLs)
>> in a file and use them later when online?
[...]
> Very nice, thank you.  Go ahead and apply, but make sure you also
> update gnus.el so the new spam-report function is an available spam
> processor.  Let me know if you need help with any of it.

Sorry, I couldn't anything referring to
`spam-report-url-ping-function' in `gnus.el'.  If it's needed, please
install it.

>> I'm not sure if we should use `spam-directory' or `gnus-directory'
>> in the default value of `spam-report-requests-file'.  Is
>> `spam-report.el' supposed to be independent from `spam.el'?  At
>> least it doesn't require it.

Any comment on this?  I have used the following now:

(defcustom spam-report-requests-file
  (nnheader-concat gnus-directory "spam/" "spam-report-requests.url")

>> Somehow related to this: I think `spam.el' should _not_ hard-code
>> "~/News/spam/" as `spam-directory' but use `gnus-directory'[2]
>> instead, see its doc-string.
>
> I would use spam-directory.  I definitely think spam-directory should
> be different from gnus-directory, because of the high volume of data
> that can accumulate in the blacklists for instance.  The default
> should be (format "%s/spam" gnus-directory) so users can override it
> iff they want to.  Do you agree?

Yes, I agree.  I didn't want to change the default value of
`spam-directory', only its initialization, which should be derived
from `gnus-directory' in some way (see my path in the previous
posting):

[...]
-(defcustom spam-directory "~/News/spam/"
+(defcustom spam-directory (nnheader-concat gnus-directory "spam/")
[...]

I don't know the reason for using `nnheader-concat', but I guess there
is a good reason, because it's used elsewhere in most of the Gnus
files.

I've committed my previously posted patch plus some additions:

	* spam.el (spam-directory): Derive from `gnus-directory'.

	* spam-report.el (spam-report-url-to-file)
	(spam-report-requests-file): New function and variable for offline
	reporting.
	(spam-report-url-ping-function): Add `spam-report-url-to-file'
	and user defined function.
	(spam-report-url-ping-mm-url): Remove doubled slash.

> I agree with Kai, this should be doable when Gnus becomes plugged.
> GET and such are not always available, and there's no need to
> complicate the user's life.  

Okay, here's my suggestion for `spam-report-process-queue' (not
committed yet, please test) and the agent hooks:

--8<---------------cut here---------------start------------->8---
(defun spam-report-process-queue (&optional file keep)
  "Report all queued requests from `spam-report-requests-file'.

If KEEP is t, leave old requests in the file.  If KEEP is
the symbol `ask', query before flushing the queue file."
  (interactive
   (list (read-file-name
	  "File: "
	  (file-name-directory spam-report-requests-file)
	  spam-report-requests-file
	  nil
	  (file-name-nondirectory spam-report-requests-file)
	  spam-report-requests-file)
	 current-prefix-arg))
  (or file (setq file spam-report-requests-file))
  (save-excursion
    (set-buffer (find-file-noselect file))
    (goto-char (point-min))
    (while (and (not (eobp))
		(re-search-forward
		 "http://\\([^/]+\\)\\(/.*\\) *$" (point-at-eol) t))
      (spam-report-url-ping-plain (match-string 1) (match-string 2))
      (forward-line 1))
    (if (or (eq keep nil)
	    (and (eq keep 'ask)
		 (y-or-n-p
		  (format
		   "Flush requests from `%s'? " (current-buffer)))))
	(progn
	  (gnus-message 7 "Flushing request file `%s'"
			spam-report-requests-file)
	  (erase-buffer)
	  (save-buffer)
	  (kill-buffer (current-buffer)))
      (gnus-message 7 "Keeping requests in `%s'" spam-report-requests-file))))

(add-to-list 'gnus-agent-plugged-hook
	     (lambda ()
	       (setq spam-report-url-ping-function
		     'spam-report-url-ping-plain))
	       (spam-report-process-queue));; see below
(add-to-list 'gnus-agent-unplugged-hook
	     (lambda ()
	       (setq spam-report-url-ping-function
		     'spam-report-url-to-file)))
--8<---------------cut here---------------end--------------->8---

> The manual should say "if you want to queue reports and you use the
> Agent, do (add-hook...".
>
> Let me know which parts of the above you want me to do.  Thanks!

Please check if something should be added to `gnus.el' (see above).
Could someone check if the suggested functions and hooks work when
using the agent (I don't use the agent)?

AFAICS, the `spam-report-url-ping*' stuff isn't documented in the
manual.  This should be added together with the new
`spam-report-url-to-file' function, probably in (info "(gnus)Gmane
Spam Reporting").

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: [spam-report.el] Offline Gmane spam reporting

[Ted Zlatanov]

On Tue, 03 Feb 2004, 4.uce.03.r.s@nurfuerspam.de wrote:

On Mon, Feb 02 2004, Ted Zlatanov wrote: > On Sun, 01 Feb 2004,
4.uce.03.r.s@nurfuerspam.de wrote: >
>>> when I'm reading Gmane groups offline, I cannot use the
>>> spam-report functions.  Wouldn't it be nice to collect the
>>> requests (the URLs) in a file and use them later when online?
> [...]
>> Very nice, thank you.  Go ahead and apply, but make sure you also
>> update gnus.el so the new spam-report function is an available spam
>> processor.  Let me know if you need help with any of it.
> 
> Sorry, I couldn't anything referring to
> `spam-report-url-ping-function' in `gnus.el'.  If it's needed,
> please install it.

Oh I see, I thought this was a whole new function when in fact it's a
choice of spam-report-url-ping-function.  gnus.el has
gnus-group-spam-exit-processor-report-gmane which can be associated
with exiting from a group to report all spam in it to Gmane, and
eventually calls spam-report-url-ping-function.

>>> I'm not sure if we should use `spam-directory' or `gnus-directory'
>>> in the default value of `spam-report-requests-file'.  Is
>>> `spam-report.el' supposed to be independent from `spam.el'?  At
>>> least it doesn't require it.
> 
> Any comment on this?  I have used the following now:
> 
> (defcustom spam-report-requests-file
>   (nnheader-concat gnus-directory "spam/"
>   "spam-report-requests.url")

That's fine.

> Yes, I agree.  I didn't want to change the default value of
> `spam-directory', only its initialization, which should be derived
> from `gnus-directory' in some way (see my path in the previous
> posting):
> 
> [...]
> -(defcustom spam-directory "~/News/spam/"
> +(defcustom spam-directory (nnheader-concat gnus-directory "spam/")
> [...]

Oh, I see.  Excellent!  I like it a lot.

> I don't know the reason for using `nnheader-concat', but I guess
> there is a good reason, because it's used elsewhere in most of the
> Gnus files.

Heh, cargo cult at its best...  I have no idea either, but I'll keep
this in mind.

> I've committed my previously posted patch plus some additions:
> 
> 	* spam.el (spam-directory): Derive from `gnus-directory'.
> 
> 	* spam-report.el (spam-report-url-to-file)
> 	(spam-report-requests-file): New function and variable for
> 	offline reporting.  (spam-report-url-ping-function): Add
> 	`spam-report-url-to-file' and user defined function.
> 	(spam-report-url-ping-mm-url): Remove doubled slash.

Great, thanks!

> (defun spam-report-process-queue (&optional file keep)
>   "Report all queued requests from `spam-report-requests-file'.
> 
> If KEEP is t, leave old requests in the file.  If KEEP is
> the symbol `ask', query before flushing the queue file."
>   (interactive
>    (list (read-file-name
> 	  "File: "
> 	  (file-name-directory spam-report-requests-file)
> 	  spam-report-requests-file
> 	  nil
> 	  (file-name-nondirectory spam-report-requests-file)
> 	  spam-report-requests-file)
> 	 current-prefix-arg))
>   (or file (setq file spam-report-requests-file))
>   (save-excursion
>     (set-buffer (find-file-noselect file))
>     (goto-char (point-min))
>     (while (and (not (eobp))
> 		(re-search-forward
> 		 "http://\\([^/]+\\)\\(/.*\\) *$" (point-at-eol) t))
>       (spam-report-url-ping-plain (match-string 1) (match-string 2))
>       (forward-line 1))
>     (if (or (eq keep nil)
> 	    (and (eq keep 'ask)
> 		 (y-or-n-p
> 		  (format
> 		   "Flush requests from `%s'? " (current-buffer)))))
> 	(progn
> 	  (gnus-message 7 "Flushing request file `%s'"
> 			spam-report-requests-file)
> 	  (erase-buffer)
> 	  (save-buffer)
> 	  (kill-buffer (current-buffer)))
>       (gnus-message 7 "Keeping requests in `%s'"
>       spam-report-requests-file))))

I would use spam-report-url-ping-function instead of s-r-u-p-plain
there, unless s-r-u-p-function is equal to s-r-u-p-to-file (then we
fall back to s-r-u-p-plain or maybe alert the user he's doing
something strange).

> (add-to-list 'gnus-agent-plugged-hook
> 	     (lambda ()
> 	       (setq spam-report-url-ping-function
> 		     'spam-report-url-ping-plain))
> 	       (spam-report-process-queue));; see below
> (add-to-list 'gnus-agent-unplugged-hook
> 	     (lambda ()
> 	       (setq spam-report-url-ping-function
> 		     'spam-report-url-to-file)))

I would save the old spam-report-url-ping-function in the unplugged
hook, and restore it in the plugged hook, in case the user wants
something other than 's-r-u-p-plain (they may already have
's-r-u-p-to-file for instance!)

The code looks great otherwise, thanks so much for doing it!

> AFAICS, the `spam-report-url-ping*' stuff isn't documented in the
> manual.  This should be added together with the new
> `spam-report-url-to-file' function, probably in (info "(gnus)Gmane
> Spam Reporting").

OK, I'll put it on my TODO list (unless someone else wants to do it).

Ted

Re: [spam-report.el] Offline Gmane spam reporting

[Reiner Steib]

On Thu, Feb 05 2004, Ted Zlatanov wrote:

> On Tue, 03 Feb 2004, 4.uce.03.r.s@nurfuerspam.de wrote:
>> (defun spam-report-process-queue (&optional file keep)
>>   "Report all queued requests from `spam-report-requests-file'.
[...]
>>       (spam-report-url-ping-plain (match-string 1) (match-string 2))
[...]
> I would use spam-report-url-ping-function instead of s-r-u-p-plain
> there, 

Changed to...

  (funcall spam-report-url-ping-function (match-string 1) (match-string 2))

> unless s-r-u-p-function is equal to s-r-u-p-to-file (then we fall
> back to s-r-u-p-plain or maybe alert the user he's doing something
> strange).

Good idea...

  (if (eq spam-report-url-ping-function 'spam-report-url-to-file)
      (error (concat "Cannot process requests when "
		     "`spam-report-url-ping-function' is "
		     "`spam-report-url-to-file'."))
    (gnus-message 7 "Processing requests using `%s'."
		  spam-report-url-ping-function))

I have committed `spam-report-process-queue' now.

BTW: `spam-report-process-queue' calls `spam-report-url-ping-function'
for each report, i.e. when using `spam-report-url-ping-plain', we do
(open-network-stream ...) for each report.  If we always (or often)
have the same host, we could spare some overhead by sending multiple
"GET"s in one connection.  But I don't know if the performance gain is
worth the effort.

>> (add-to-list 'gnus-agent-plugged-hook
>> 	     (lambda ()
>> 	       (setq spam-report-url-ping-function
>> 		     'spam-report-url-ping-plain))
>> 	       (spam-report-process-queue));; see below
>> (add-to-list 'gnus-agent-unplugged-hook
>> 	     (lambda ()
>> 	       (setq spam-report-url-ping-function
>> 		     'spam-report-url-to-file)))
>
> I would save the old spam-report-url-ping-function in the unplugged
> hook, and restore it in the plugged hook, in case the user wants
> something other than 's-r-u-p-plain (they may already have
> 's-r-u-p-to-file for instance!)

If it gets more complicated, I would suggest to define named
functions, because then users may benefit of improvements without
having to edit the lambdas.  Please install what you have in mind (I'm
not sure how you intend to preserve the values and I'm not familiar
with the agent).

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: [spam-report.el] Offline Gmane spam reporting

[Ted Zlatanov]

On Thu, 05 Feb 2004, 4.uce.03.r.s@nurfuerspam.de wrote:

> BTW: `spam-report-process-queue' calls
> `spam-report-url-ping-function' for each report, i.e. when using
> `spam-report-url-ping-plain', we do (open-network-stream ...) for
> each report.  If we always (or often) have the same host, we could
> spare some overhead by sending multiple "GET"s in one connection.
> But I don't know if the performance gain is worth the effort.

I don't think that makes much of a difference, the big overhead is on
the Gmane server where the request is parsed.  Maybe if Lars gives us
an interface for submitting a lot of spam reports at once...

I can actually see why Lars *wouldn't* make it easy to submit lots of
spam reports at once.  It would make it easier to maliciously submit
lots of false spam reports.  I don't know if that's a concern, but I
thought of it :)

> If it gets more complicated, I would suggest to define named
> functions, because then users may benefit of improvements without
> having to edit the lambdas.  Please install what you have in mind
> (I'm not sure how you intend to preserve the values and I'm not
> familiar with the agent).

I added the code to spam-report.el, see what you think.  I make
special exceptions for users that always report to a file anyway.

Thanks
Ted

Re: [spam-report.el] Offline Gmane spam reporting

[Reiner Steib]

On Mon, Feb 09 2004, Ted Zlatanov wrote:

> On Thu, 05 Feb 2004, 4.uce.03.r.s@nurfuerspam.de wrote:
[...]
>> If it gets more complicated, I would suggest to define named
>> functions, because then users may benefit of improvements without
>> having to edit the lambdas.  Please install what you have in mind
>> (I'm not sure how you intend to preserve the values and I'm not
>> familiar with the agent).
>
> I added the code to spam-report.el, see what you think.  I make
> special exceptions for users that always report to a file anyway.

I changed (or added) the doc-strings a little.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/