How many people mark spam in gmane groups?

How many people mark spam in gmane groups?

[Ted Stern]

Since I started reading the ding-gnus and bbdb-info mailing lists through
gmane, I've been marking a fair number of messages as spam.

At the moment, this entails me exposing the X-Report-Spam header using "C-u g"
in the summary buffer, then using browse-url-at-point to send the link to my
browser.

Is there some way to automate this a bit more?  It would be nice if there were
a single keystroke (say "$ R") to report spam.

Ted
-- 
 Ted Stern                           Engineering Applications
 Cray Inc.                               office: 206-701-2182
 411 First Avenue South, Suite 600         cell: 206-383-1049
 Seattle, WA 98104-2860                     FAX: 206-701-2500
 Debuggers' motto:  Frango ut patefaciam -- I break in order to reveal

Re: How many people mark spam in gmane groups?

[Reiner Steib]

On Wed, Apr 02 2003, Ted Stern wrote:

> At the moment, this entails me exposing the X-Report-Spam header
> using "C-u g" in the summary buffer, then using browse-url-at-point
> to send the link to my browser.
> Is there some way to automate this a bit more?  

Sure, it's Gnus. :-)

;; See <873crvd7nb.fsf@doohan.bang.priv.no> <m3fzwk3hcq.fsf@quimbies.gnus.org>
;; This functions also work when we don't read from news.gname.org.
;;;###autoload
(defun rs-gnus-summary-fetch-field (field &optional no-err)
  "Return the value of the header FIELD of current article.
Unlike `gnus-fetch-field', `rs-gnus-summary-fetch-field' should be called from
a summary buffer and works also if no article buffer was present before."
  (let ((url) (close-art-buf))
    (unless (buffer-live-p gnus-article-buffer)
      (gnus-summary-show-article 'raw)
      (setq close-art-buf t))
    (set-buffer gnus-article-buffer)
    (setq url (gnus-fetch-field field))
    (when close-art-buf
      ;; (kill-buffer gnus-article-buffer) ;; FIXME
      (gnus-summary-expand-window))
    (if no-err
	url
      (if url
	  url
	(progn
	  (message "Field `%s' not found in article number %d."
		   field (gnus-summary-article-number))
	  nil)))))

;;;###autoload
(defun rs-gnus-summary-gmane-report-spam ()
  "Report current Gmane article as spam using `browse-url'."
  (interactive)
  (let* ((field "X-Report-Spam")
	 (url (rs-gnus-summary-fetch-field field)))
    (if url (browse-url url))))

;;;###autoload
(defun rs-gnus-summary-gmane-report-unspam ()
  "Report current Gmane article as _not_ spam using `browse-url'."
  (interactive)
  (let* ((field "X-Report-Unspam")
	 (url (rs-gnus-summary-fetch-field field)))
    (if url (browse-url url))))

> It would be nice if there were a single keystroke (say "$ R") to
> report spam.

I didn't use the above code very often, so I'm not sure if it works
reliable.

Should we add something like this to Gnus?  Opinions?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: How many people mark spam in gmane groups?

[Jon Ericson]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:

> On Wed, Apr 02 2003, Ted Stern wrote:
>
>> It would be nice if there were a single keystroke (say "$ R") to
>> report spam.
>
> I didn't use the above code very often, so I'm not sure if it works
> reliable.
>
> Should we add something like this to Gnus?  Opinions?

I would certainly welcome it.  

Jon
-- 
  My frame was not hidden from you
    when I was made in the secret place.
  When I was woven together in the depths of the earth,
    your eyes saw my unformed body.  -- Psalm 139:15-16a (NIV)

Re: How many people mark spam in gmane groups?

[Ted Zlatanov]

On Wed, 02 Apr 2003, 4.uce.03.r.s@nurfuerspam.de wrote:
> On Wed, Apr 02 2003, Ted Stern wrote:
> 
>> At the moment, this entails me exposing the X-Report-Spam header
>> using "C-u g" in the summary buffer, then using browse-url-at-point
>> to send the link to my browser.
>> Is there some way to automate this a bit more?  
> 
> Sure, it's Gnus. :-)
> 
[ code omitted ]
> 
>> It would be nice if there were a single keystroke (say "$ R") to
>> report spam.
> 
> I didn't use the above code very often, so I'm not sure if it works
> reliable.
> 
> Should we add something like this to Gnus?  Opinions?

Although spam.el might be too heavyweight for something like this, a
spam/ham summary exit processor ("reporter"?) that can use the
X-Report-Spam header would work.  Then the spam-marked messages could
be batch-processed as described above, then copied (not moved!) if
spam-process-destination is set for the newsgroup.

This would require some changes to spam.el, since it has been used
only with mail groups, so it's not a trivial addition.

Ted

Re: How many people mark spam in gmane groups?

[Reiner Steib]

On Sat, Apr 05 2003, Ted Zlatanov wrote:

> On Wed, 02 Apr 2003, 4.uce.03.r.s@nurfuerspam.de wrote:
>> On Wed, Apr 02 2003, Ted Stern wrote:

[ Gmane / X-Report-Spam / X-Report-Unspam ]
> [ code omitted ]
>> 
>>> It would be nice if there were a single keystroke (say "$ R") to
>>> report spam.
>> 
>> I didn't use the above code very often, so I'm not sure if it works
>> reliable.
>> 
>> Should we add something like this to Gnus?  Opinions?
>
> Although spam.el might be too heavyweight for something like this, a
> spam/ham summary exit processor ("reporter"?) that can use the
> X-Report-Spam header would work.  Then the spam-marked messages could
> be batch-processed as described above, then copied (not moved!) if
> spam-process-destination is set for the newsgroup.
>
> This would require some changes to spam.el, since it has been used
> only with mail groups, so it's not a trivial addition.

I don't use spam.el yet (but I'm sure I will in the future, because
it's certainly a very nice and necessary feature), so I don't know how
the `*gmane-report-*spam' would fit in there.  As you say that it's
not trivial, it should be easier to put it in `gnus-sum.el'[1].
Another Gmane related command that I use often is "Browse current
Gmane article using `browse-url'.".  As I use leafnode, I don't have
the same article numbers as news.gmane.org has, but I can (ab)use the
X-Report-{Unspam,Spam} header for this:

(defun rs-gnus-summary-gmane-browse-article ()
  "Browse current Gmane article using `browse-url'."
  (interactive)
  (let* ((url (or (rs-gnus-summary-fetch-field "X-Report-Spam" t)
		  (rs-gnus-summary-fetch-field "X-Report-Unspam"))))
    (if url
	(browse-url
	 (gnus-replace-in-string
	  (gnus-replace-in-string
	   url "http://\\(un\\)?spam\\." "http://article.")
	  ":\\([0-9]+\\)\\'" "/\\1")))))

For the `*gmane-report-*spam' commands, we could use similar key
bindings as for the spam/ham reporting.

Bye, Reiner.

[1] If we add more Gmane related commands, it might be better to
    create a `gnus-gmane.el' file for this.  But maybe such a file
    already exists on the disks of the Gmane author. ;-)
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: How many people mark spam in gmane groups?

[Ted Zlatanov]

On Sun, 06 Apr 2003, 4.uce.03.r.s@nurfuerspam.de wrote:
> I don't use spam.el yet (but I'm sure I will in the future, because
> it's certainly a very nice and necessary feature), so I don't know
> how the `*gmane-report-*spam' would fit in there.  As you say that
> it's not trivial, it should be easier to put it in `gnus-sum.el'[1].

Well, the support for read-only backends is not done for spam.el, this
is not a spam reporting issue as much as using spam.el with nntp for
instance.

The way it would work is very simple:

- add a group parameter (global through a regex or specific for that
  group) for the spam-processor.  The spam-processor takes whatever
  messages are marked as spam at summary exit and does something to
  them (for instance, train Bogofilter).

- mark messages as spam in that group

- exit the group - the spam-processor will be automatically invoked

The first one would be done just once, of course.

The problem is that nntp does not split articles like nnmail
backends.  So detecting new articles will be done differently, and
instead of moving them to a spam group as spam.el does now, it would
only mark them as spam.  It would work similarly to adaptive scoring.

As I said, it's a pretty big rewrite for spam.el.  I read Usenet a
lot, though, so it will bother me enough eventually that I'll
implement it.  I have to finish the registry and other new spam.el
features first, though :)

Ted

Re: How many people mark spam in gmane groups?

[Andreas Fuchs]

On 2003-04-07, Ted Zlatanov <tzz@lifelogs.com> wrote:
> The way it would work is very simple:
> 
> - add a group parameter (global through a regex or specific for that
>   group) for the spam-processor.  The spam-processor takes whatever
>   messages are marked as spam at summary exit and does something to
>   them (for instance, train Bogofilter).
> 
> - mark messages as spam in that group
> 
> - exit the group - the spam-processor will be automatically invoked
> 
> The first one would be done just once, of course.

Nice concept. In search for a more immediate solution, I have produced
the code in <URL:http://article.gmane.org/gmane.discuss/3079> (and
forgot to cross-post, of course), which allows spam.el-like usage of the
gmane spam (not unspam, unfortunately, that would be too slow here)
reporting mechanism.

Larsi suggested adding it to spam.el, but IMHO it would also fit into a
gmane.el (along with a few more gmane hacks other gnusers have produced
- any calls?).

Have fun with it (and go forth and tag spam on gmane.org appropriately
now!),
-- 
Andreas Fuchs, <asf@acm.org>, asf@jabber.at, antifuchs
irc.freenode.net's #emacs - online emacs advice from IRC addicts

Re: How many people mark spam in gmane groups?

[Lars Magne Ingebrigtsen]

Andreas Fuchs <asf@void.at> writes:

> Larsi suggested adding it to spam.el, but IMHO it would also fit into a
> gmane.el (along with a few more gmane hacks other gnusers have produced
> - any calls?).

Or perhaps a gnus-report.el.

Darn.  Too long name.  gnus-rprt.el, then.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: How many people mark spam in gmane groups?

[Andreas Fuchs]

On 2003-04-12, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> Or perhaps a gnus-report.el.

Hm, a spam report interface for gnus would really be nice. I guess we
all have some kind of spamcop-reporting setup (used more or less
regularly), no?

Another thing: I noticed that my code breaks when tramp is
active. Better use this mutation of it:

(defun gnus-gmane-spam-prepare-exit ()
  "Process articles marked as spam in gmane groups."
  (let ((article-no 0))
    (when (string-match "^nntp+.*:gmane." gnus-newsgroup-name)
      (spam-generic-register-routine
      (lambda (article)
	(with-current-buffer nntp-server-buffer
	  (gnus-message 6 "Reporting spam nr. %s to spam.gmane.org..." (incf article-no))
	  (gnus-request-head article gnus-newsgroup-name)
	  (goto-char (point-min))
	  (when (re-search-forward "^X-Report-Spam: \\(http://.*\\)$" nil t)
	    (let (file-name-handler-alist)
	      (url-retrieve (match-string 1) (lambda () nil))))))
      'identity))))

Have fun,
-- 
Andreas Fuchs, <asf@acm.org>, asf@jabber.at, antifuchs
irc.freenode.net's #emacs - online emacs advice from IRC addicts

Re: How many people mark spam in gmane groups?

[Lars Magne Ingebrigtsen]

Andreas Fuchs <asf@void.at> writes:

> Hm, a spam report interface for gnus would really be nice. I guess we
> all have some kind of spamcop-reporting setup (used more or less
> regularly), no?

Didn't somebody write such a thing?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: How many people mark spam in gmane groups?

[Ted Zlatanov]

I would suggest spam-report.el (considering that gnus-report.el is too
generic, this is really a spam reporting package).  Other spam
reporting functions can go in it as needed.

Also, we can add code to spam-report.el to make it usable by itself,
so users without spam.el can still use it.

How does the following (untested) code look?

(defun spam-report-gmane (article)
  (when (string-match "^nntp+.*:gmane." gnus-newsgroup-name)
    (with-current-buffer nntp-server-buffer
      (gnus-message 6 "Reporting spam article %d to spam.gmane.org..." article)
      (gnus-request-head article gnus-newsgroup-name)
      (goto-char (point-min))
      (when (re-search-forward "^X-Report-Spam: \\(http://.*\\)$" nil t)
	(let (file-name-handler-alist)
	  (url-retrieve (match-string 1) (lambda () nil)))))))

(defun spam-report-gmane-register-spam-routine ()
  (spam-generic-register-routine 
   spam-report-gmane
   nil))

I took out the article-no nicety because I changed things around, and
print the article number instead - feel free to fix it.  If you like
this, I'll have spam.el load spam-report.el and provide a spam exit
processor that will do the right thing - to be consistent with the
rest of spam.el.  I'm pretty sure we'll need to fix assumptions
spam.el makes about the backend (spam-process-destination needs to be
disabled, for instance), but overall I see no problem with this and I
think it's a good idea.

For those who aren't following the discussion, the upshot will be that
you add a gmane spam exit processor to your gmane groups, and the
processor will send the spam report automatically for the messages you
have marked as spam in the summary buffer.

Ted

Re: How many people mark spam in gmane groups?

[Andreas Fuchs]

Today, Ted Zlatanov <tzz@lifelogs.com> wrote:
> I would suggest spam-report.el (considering that gnus-report.el is too
> generic, this is really a spam reporting package).  Other spam
> reporting functions can go in it as needed.

Good.

> How does the following (untested) code look?
> 
> (defun spam-report-gmane (article)
>   (when (string-match "^nntp+.*:gmane." gnus-newsgroup-name)
[rest snipped]

Looks good to me. Making a spam exit processor only for gmane groups
would make the initial string-match in spam-report-gmane unnecessary, of
course.
 
> I took out the article-no nicety because I changed things around, and
> print the article number instead - feel free to fix it.  

It's more useful the way you do it.

> If you like this, I'll have spam.el load spam-report.el and provide a
> spam exit processor that will do the right thing - to be consistent
> with the rest of spam.el.  I'm pretty sure we'll need to fix
> assumptions spam.el makes about the backend (spam-process-destination
> needs to be disabled, for instance), but overall I see no problem with
> this and I think it's a good idea.

Great.

-- 
Andreas Fuchs, <asf@acm.org>, asf@jabber.at, antifuchs
irc.freenode.net's #emacs - online emacs advice from IRC addicts

Re: How many people mark spam in gmane groups?

[Reiner Steib]

On Mon, Apr 14 2003, Ted Zlatanov wrote:

> (defun spam-report-gmane (article)
>   (when (string-match "^nntp+.*:gmane." gnus-newsgroup-name)
>     (with-current-buffer nntp-server-buffer
>       (gnus-message 6 "Reporting spam article %d to spam.gmane.org..." article)
>       (gnus-request-head article gnus-newsgroup-name)
>       (goto-char (point-min))
>       (when (re-search-forward "^X-Report-Spam: \\(http://.*\\)$" nil t)
> 	(let (file-name-handler-alist)
> 	  (url-retrieve (match-string 1) (lambda () nil)))))))

I let leafnode fetch from news.gmane.org, so my Gmane groups don't
match "^nntp+.*:gmane." and the article numbers aren't of any use.

See my first post: <news:v91y0kx2jn.fsf@marauder.physik.uni-ulm.de>.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: How many people mark spam in gmane groups?

[Ted Zlatanov]

On Mon, 14 Apr 2003, 4.uce.03.r.s@nurfuerspam.de wrote:
> I let leafnode fetch from news.gmane.org, so my Gmane groups don't
> match "^nntp+.*:gmane." and the article numbers aren't of any use.

For the regex match, no problem.  It will be a group/topic parameter.

> See my first post: <news:v91y0kx2jn.fsf@marauder.physik.uni-ulm.de>.

OK, several things I hope you can clarify - remember I'm not that good
with the Gnus internals :)

- why do your own gnus-fetch-field?

- with spam.el, the code gets invoked at summary exit, so probably
  there's no need to do all that article viewing magic in
  rs-gnus-summary-fetch-field.

Thanks
Ted

Re: How many people mark spam in gmane groups?

[Reiner Steib]

On Tue, Apr 15 2003, Ted Zlatanov wrote:

> - why do your own gnus-fetch-field?

Assume you enter the group with RET.  Then you don't have an article
buffer (`gnus-fetch-field' need to be called from an article buffer).

> - with spam.el, the code gets invoked at summary exit, so probably
>   there's no need to do all that article viewing magic in
>   rs-gnus-summary-fetch-field.

I don't think that you can avoid it completely, since fetching the
X-Report-Spam header is the only reliable way (AFAIK) to get the
article number on Gmane.  (But there might be much better
implementations than mine. ;-) )

Two other possibilities might work:

- If the X-Report-Spam headers are included in the overview of the
  newsserver, it could be easier.  But to my knowledge, this isn't
  possible with leafnode.  Or is it sufficient to modify
  `gnus-extra-headers'?

- Gmane might allow spam reporting by Message-Id, say
  http://spam.gmane.org/gmane.<group>/mid/<message-id> or
  http://spam.gmane.org/mid/<message-id>.

  I don't know if this is feasible.

If supporting my (leafnode-) setup it too much work for you, please
just go ahead don't waste too much time on it.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: How many people mark spam in gmane groups?

[Steinar Bang]

>>>>> Reiner Steib <4.uce.03.r.s@nurfuerspam.de>:


> I let leafnode fetch from news.gmane.org, so my Gmane groups don't
> match "^nntp+.*:gmane." and the article numbers aren't of any use.

I, on the other hand, read directly from news.gmane.org, and prefer
not to open and display the spam articles, so I use this to report the
spam (sligthly hacked from something Larsi hacked up):

(defun gmane-gnus-summary-report-spam ()
  "Report gmane spam."
  (interactive)
  (browse-url
   (format "http://spam.gmane.org/%s:%d"
	   (gnus-group-real-name gnus-newsgroup-name)
	   (gnus-summary-article-number))))

Re: How many people mark spam in gmane groups?

[Reiner Steib]

On Tue, Apr 15 2003, Steinar Bang wrote:

> I, on the other hand, read directly from news.gmane.org, and prefer
> not to open and display the spam articles, 

This is the reason, why I used "(gnus-summary-show-article 'raw)".

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: How many people mark spam in gmane groups?

[Ted Zlatanov]

On Tue, 15 Apr 2003, sb@dod.no wrote:
> (defun gmane-gnus-summary-report-spam ()
>   "Report gmane spam."
>   (interactive)
>   (browse-url
>    (format "http://spam.gmane.org/%s:%d"
> 	   (gnus-group-real-name gnus-newsgroup-name)
> 	   (gnus-summary-article-number))))

Since browse-url is fairly slow and tends to require interactivity,
and we need a fast write-only URL ping, I put together the following
code out of net-utils.el:

(progn
  (with-temp-buffer
    (let ((host "spam.gmane.org")) 
      (or (setq tcp-connection
		(open-network-stream 
		 "URL ping"
		 (buffer-name)
		 host
		 80))
	(error "Could not open connection to %s" host))
      (set-marker (process-mark tcp-connection) (point-min))
      (set-process-filter tcp-connection 'net-utils-remove-ctrl-m-filter)
      (process-send-string tcp-connection 
			   (format "GET /gmane.emacs.help:8479 HTTP/1.1\nHost: spam.gmane.org\n\n")))))

(the article in the code above was real spam, btw)

It seems correct to me.  Since the temp-buffer gets killed, that will
kill the network connection as well, right?

If people like this, I will use it instead of the
browse-url/url-retrieve functions in spam-report.el.

spam-report.el is almost ready except for this detail.

Ted

Re: How many people mark spam in gmane groups?

[Andreas Fuchs]

Today, Ted Zlatanov <tzz@lifelogs.com> wrote:
> (progn
>   (with-temp-buffer
>     (let ((host "spam.gmane.org")) 
>       (or (setq tcp-connection
> 		(open-network-stream 
> 		 "URL ping"
> 		 (buffer-name)
> 		 host
> 		 80))
> 	(error "Could not open connection to %s" host))
>       (set-marker (process-mark tcp-connection) (point-min))
>       (set-process-filter tcp-connection
>       'net-utils-remove-ctrl-m-filter) (process-send-string
>       tcp-connection
> 			   (format "GET /gmane.emacs.help:8479
> 			   HTTP/1.1\nHost: spam.gmane.org\n\n")))))


> It seems correct to me.  Since the temp-buffer gets killed, that will
> kill the network connection as well, right?

The important question is, will this work if the network connection
gets killed before gmane handles the request? This was what made me use
(asynchronous) url-retrieve, which works pretty much like your
solution above, only (seemingly) more asynchronously. (-:

Maybe somebody with knowledge of php can tell us how this kind of error
gets handled or if it is an error at all?

-- 
Andreas Fuchs, <asf@acm.org>, asf@jabber.at, antifuchs
irc.freenode.net's #emacs - online emacs advice from IRC addicts

Re: How many people mark spam in gmane groups?

[Lars Magne Ingebrigtsen]

Andreas Fuchs <asf@void.at> writes:

> Maybe somebody with knowledge of php can tell us how this kind of error
> gets handled or if it is an error at all?

Hm...  it depends on whether Apache stops executing the PHP code
before it starts outputting any text.

No, wait.  spam.php outputs a bit of text, then does the accounting,
and then outputs more text.  Apache will discover that the connection
is down after the first bit, thereby killing off the process.

I've now changed spam.php to do all processing first, then output all
text afterwards.  So it should now be safe to just say "GET blabla" at
spam.php and then close the connection.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: How many people mark spam in gmane groups?

[Andreas Fuchs]

Today, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> I've now changed spam.php to do all processing first, then output all
> text afterwards.  So it should now be safe to just say "GET blabla" at
> spam.php and then close the connection.

Yay! In that case, I'm all for Ted's solution (obviously (-:).

-- 
Andreas Fuchs, <asf@acm.org>, asf@jabber.at, antifuchs
irc.freenode.net's #emacs - online emacs advice from IRC addicts

Re: How many people mark spam in gmane groups?

[Ted Zlatanov]

On Mon, 07 Apr 2003, tzz@lifelogs.com wrote:
> - add a group parameter (global through a regex or specific for that
>   group) for the spam-processor.  The spam-processor takes whatever
>   messages are marked as spam at summary exit and does something to
>   them (for instance, train Bogofilter).
> 
> - mark messages as spam in that group
> 
> - exit the group - the spam-processor will be automatically invoked
> 
> The first one would be done just once, of course.

This is now set up through spam-report.el and spam.el; you can use the
spam-report-gmane function in spam-report.el on its own if you wish or
you can add the gnus-group-spam-exit-processor-report-gmane processor
to the exit spam processors of your Gmane groups.  The Gmane groups
have to be NNTP groups, I think, in order for the code to work (it
uses nntp-server-buffer).  It seemed to work OK for me; I looked at
the network traffic and saw the acknowledgment from gmane.org arriving
for each reported article, so I'm pretty sure it reports the spam
properly.

There are variables you can customize in spam-report.el, for instance
reporting can be done through the fast inference of the X-Report-Spam
URL from the article number (default), or through the slower parsing
of the article itself for those whose article numbers don't correspond
to the Gmane article numbers.

It should be very easy to add more spam reporters to spam-report.el,
and I can glue them into spam.el as we go.  I don't know much about
spam reporters, but there seem to be many of them and a lot of
incidental code for using them from Gnus.

Ted

Re: How many people mark spam in gmane groups?

[Jon Ericson]

Ted Zlatanov <tzz@lifelogs.com> writes:

> This is now set up through spam-report.el and spam.el; you can use
> the spam-report-gmane function in spam-report.el on its own if you
> wish or you can add the gnus-group-spam-exit-processor-report-gmane
> processor to the exit spam processors of your Gmane groups.  The
> Gmane groups have to be NNTP groups, I think, in order for the code
> to work (it uses nntp-server-buffer).  It seemed to work OK for me;
> I looked at the network traffic and saw the acknowledgment from
> gmane.org arriving for each reported article, so I'm pretty sure it
> reports the spam properly.

I finally got around to trying this out, and yes, it does report spam
properly.  (I'm one of the people who does spam reporting approval for
Gmane, so I can be certain about that. :)

Is there an easy way to set up the group properties for all of my
Gmane groups?  I couldn't get spam-report-gmane-regex to work for some
reason.[1] It seems like this should be a property of the server, not
the individual groups.  Or perhaps we are really looking at a new sort
of protocol built on top of nntp.  At any rate, I would prefer not to
have to twiddle with the properties of each new Gmane group I add.

But enough of my griping; this stuff is great.  I might even see about
ditching my current nnmail-split-fancy system and try the other bits
of spam.el.

Jon

Footnotes: 
[1]  Most likely I screwed up the regex.  I'd rather set properties in
     any case, 'cause it seems less of a hack.

-- 
  Everyone who does evil hates the light, and will not come into the
  light for fear that his deeds will be exposed.  But whoever lives by
  the truth comes into the light, so that it may be seen plainly that
  what he has done has been done through God."  -- John 3:20-21 (NIV)

Re: How many people mark spam in gmane groups?

[Ted Zlatanov]

On Fri, 25 Apr 2003, Jon.Ericson@jpl.nasa.gov wrote:
> Is there an easy way to set up the group properties for all of my
> Gmane groups?  

Put all of them under a topic and then customize the topic properties
like you would customize a group.  It works very nicely.

> I couldn't get spam-report-gmane-regex to work for some reason.[1]
> It seems like this should be a property of the server, not the
> individual groups.  Or perhaps we are really looking at a new sort
> of protocol built on top of nntp.  At any rate, I would prefer not
> to have to twiddle with the properties of each new Gmane group I
> add.

> But enough of my griping; this stuff is great.  I might even see
> about ditching my current nnmail-split-fancy system and try the
> other bits of spam.el.
> 
> Jon
> 
> Footnotes: [1] Most likely I screwed up the regex.  I'd rather set
> properties in any case, 'cause it seems less of a hack.

spam-report.el doesn't really do anything automatic, you have to load
spam.el for that.  When you load spam.el, you can use the Gmane spam
processor for the Gmane groups (there is a list of spam processors per
group/topic/regex you can customize).  Then, when you exit the group,
spam-report-gmane will be called for every article you marked as spam
in the group.

spam.el is harmless if you don't use any spam/ham processors, and
don't use spam-split in your mail splitting.

On its own, spam-report.el could be used if you manually iterate over
the list of spam articles in the gnus-newsgroup-spam-marked list at
summary exit.  That's what spam.el does, it just automates things.

Thanks
Ted

Re: How many people mark spam in gmane groups?

[Jon Ericson]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Fri, 25 Apr 2003, Jon.Ericson@jpl.nasa.gov wrote:
>> Is there an easy way to set up the group properties for all of my
>> Gmane groups?  
>
> Put all of them under a topic and then customize the topic properties
> like you would customize a group.  It works very nicely.

Assuming you don't use topics for, um, topically related groups. :)

I am using spam.el, but spam-report-gmane-regex doesn't work for me.
If I understand the code, I don't see how it could be useful.  The
only place it's used is in spam-report-gmane, but that's only called
by spam-summary-prepare-exit if the gnus-group-spam-exit-processor-
report-gmane group parameter is set.  It seems like the check against
the regexp should be in spam.el and not in spam-report.el.  Or am I
missing something?

(I hope I don't come off as sarcastic; I really do appreciate the bits
of spam.el that I'm now using.  The only thing preventing me from
using more features is inertia.)

Jon
-- 
  Everyone who does evil hates the light, and will not come into the
  light for fear that his deeds will be exposed.  But whoever lives by
  the truth comes into the light, so that it may be seen plainly that
  what he has done has been done through God."  -- John 3:20-21 (NIV)

Re: How many people mark spam in gmane groups?

[Ted Zlatanov]

On Tue, 29 Apr 2003, Jon.Ericson@jpl.nasa.gov wrote:
> I am using spam.el, but spam-report-gmane-regex doesn't work for me.
> If I understand the code, I don't see how it could be useful.  The
> only place it's used is in spam-report-gmane, but that's only called
> by spam-summary-prepare-exit if the gnus-group-spam-exit-processor-
> report-gmane group parameter is set.  It seems like the check
> against the regexp should be in spam.el and not in spam-report.el.
> Or am I missing something?

spam-report-gmane-regex is consulted only when spam-report-gmane is
invoked.  To invoke it, you have to activate the
gnus-group-spam-exit-processor-report-gmane spam exit processor.
Customize the variable gnus-spam-process-newsgroups (you could also do
it with group/topic parameters but you already said you can't).

> (I hope I don't come off as sarcastic; I really do appreciate the
> bits of spam.el that I'm now using.  The only thing preventing me
> from using more features is inertia.)

Not at all, I appreciate feedback and suggestions.  I know spam.el is
a bit intimidating to new users, and will try to put some useful
novice info in the manual in time for Gnus 5.10.

Thanks
Ted

Re: How many people mark spam in gmane groups?

[Jon Ericson]

Ted Zlatanov <tzz@lifelogs.com> writes:

> spam-report-gmane-regex is consulted only when spam-report-gmane is
> invoked.  To invoke it, you have to activate the
> gnus-group-spam-exit-processor-report-gmane spam exit processor.
> Customize the variable gnus-spam-process-newsgroups (you could also
> do it with group/topic parameters but you already said you can't).

Eureka!  g-s-p-n was the variable I was looking for[1].  But now I'm even
more confused about s-r-g-r.  It seems there are several ways to have
s-r-g invoked:

* Set the group/topic parameter.  In this case s-r-g-r is irrelevant
  because you've already indicated precisely what groups are Gmane
  groups.

* Set g-s-p-n to indicate a regexp matching Gmane groups and applying
  the g-g-s-e-p-r-g exit processor to those groups.  If the regexp in
  g-s-p-n is accurate, s-r-g-r is irrelevant.

* Call s-r-g independent of spam.el.  In this case, s-r-g-r might be
  useful, but the documentation doesn't make any sense:

    String matching Gmane newsgroups if wanted, e.g. "^nntp+.*:gmane."[2]
    This is probably handled better with group/topic parameters.

  It'd make more sense to say something like:

    Regexp matching Gmane newsgroups.  If you are using spam.el,
    consider setting gnus-spam-process-newsgroups or
    the gnus-group-spam-exit-processor-report-gmane group/topic
    parameter instead.

> I know spam.el is a bit intimidating to new users, and will try to
> put some useful novice info in the manual in time for Gnus 5.10.

I must admit that I only skimmed the manual.  I'm sure a more detailed
reading of the spam sections would have helped.

Jon

P.S. Why is the Mail-Followup-To set to "Jon Ericson
     <Jon.Ericson@jpl.nasa.gov>, ding@gnus.org"?  When I hit 'F' to
     reply, the mail was to be sent to me and CCed to the list.  Is
     this what is intended?

Footnotes: 
[1]  The comment "This only makes sense for mail groups." is now
     outdated, by the way.

[2]  I think this should be "^nntp\+.*:gmane\." since + and . are
     special characters.

-- 
  Everyone who does evil hates the light, and will not come into the
  light for fear that his deeds will be exposed.  But whoever lives by
  the truth comes into the light, so that it may be seen plainly that
  what he has done has been done through God."  -- John 3:20-21 (NIV)

Re: How many people mark spam in gmane groups?

[Ted Zlatanov]

On Tue, 29 Apr 2003, Jon.Ericson@jpl.nasa.gov wrote:
> Eureka!  g-s-p-n was the variable I was looking for[1].  But now I'm
> even more confused about s-r-g-r.  It seems there are several ways
> to have s-r-g invoked:
> 
> * Set the group/topic parameter.  In this case s-r-g-r is irrelevant
>   because you've already indicated precisely what groups are Gmane
>   groups.

Yup.  s-r-g-r was in place before the spam.el integration, that's why
I left it in place.  With the doc changes you mention it should be
useful enough to keep.

I'll try to stick a reference to spam-report.el in the manual,
meanwhile.

> * Set g-s-p-n to indicate a regexp matching Gmane groups and
>   applying the g-g-s-e-p-r-g exit processor to those groups.  If the
>   regexp in g-s-p-n is accurate, s-r-g-r is irrelevant.

Yup.

> * Call s-r-g independent of spam.el.  In this case, s-r-g-r might be
>   useful, but the documentation doesn't make any sense:
> 
>     String matching Gmane newsgroups if wanted,
>     e.g. "^nntp+.*:gmane."[2] This is probably handled better with
>     group/topic parameters.
> 
>   It'd make more sense to say something like:
> 
>     Regexp matching Gmane newsgroups.  If you are using spam.el,
>     consider setting gnus-spam-process-newsgroups or
>     the gnus-group-spam-exit-processor-report-gmane group/topic
>     parameter instead.

Go ahead and make a patch of the docs you mention here and elsewhere
in this message and anything else you think needs work in
spam-report.el, I'll commit the patch.

> P.S. Why is the Mail-Followup-To set to "Jon Ericson
>      <Jon.Ericson@jpl.nasa.gov>, ding@gnus.org"?  When I hit 'F' to
>      reply, the mail was to be sent to me and CCed to the list.  Is
>      this what is intended?

No idea, I just hit 'f' on your message.

> Footnotes: 
> [1]  The comment "This only makes sense for mail groups." is now
>      outdated, by the way.

Yup.

> [2]  I think this should be "^nntp\+.*:gmane\." since + and . are
>      special characters.

Agreed.  I work with Perl regular expressions too much, it tends to
confuse me quite a bit when I write Lisp.

Thanks
Ted

Re: How many people mark spam in gmane groups?

[Jon Ericson]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 29 Apr 2003, Jon.Ericson@jpl.nasa.gov wrote:
>>   It'd make more sense to say something like:
>> 
>>     Regexp matching Gmane newsgroups.  If you are using spam.el,
>>     consider setting gnus-spam-process-newsgroups or
>>     the gnus-group-spam-exit-processor-report-gmane group/topic
>>     parameter instead.
>
> Go ahead and make a patch of the docs you mention here and elsewhere
> in this message and anything else you think needs work in
> spam-report.el, I'll commit the patch.

Ok:

diff -u spam-report.el.old spam-report.el
--- spam-report.el.old  2003-04-30 12:04:51.000000000 -0700
+++ spam-report.el      2003-04-30 12:08:38.000000000 -0700
@@ -35,8 +35,10 @@
   "Spam reporting configuration.")
 
 (defcustom spam-report-gmane-regex nil
-  "String matching Gmane newsgroups if wanted, e.g. \"^nntp+.*:gmane.\"
-This is probably handled better with group/topic parameters."
+  "Regexp matching Gmane newsgroups, e.g. \"^nntp\\+.*:gmane\\.\"
+If you are using spam.el, consider setting gnus-spam-process-newsgroups 
+or the gnus-group-spam-exit-processor-report-gmane group/topic parameter 
+instead."
   :type 'regexp
   :group 'spam-report)
 
diff -u gnus.el.old gnus.el
--- gnus.el.old 2003-04-30 12:24:35.000000000 -0700
+++ gnus.el     2003-04-30 12:22:48.000000000 -0700
@@ -1875,8 +1875,7 @@
    "*Groups in which to automatically process spam or ham articles with
 a backend on summary exit.  If non-nil, this should be a list of group
 name regexps that should match all groups in which to do automatic
-spam processing, associated with the appropriate processor.  This only makes sense
-for mail groups."
+spam processing, associated with the appropriate processor."
    :variable-group spam
    :variable-type '(repeat :tag "Spam/Ham Processors" 
                           (list :tag "Spam Summary Exit Processor Choices"


>> P.S. Why is the Mail-Followup-To set to "Jon Ericson
>>      <Jon.Ericson@jpl.nasa.gov>, ding@gnus.org"?  When I hit 'F' to
>>      reply, the mail was to be sent to me and CCed to the list.  Is
>>      this what is intended?
>
> No idea, I just hit 'f' on your message.

Ok.  For this message, I did a `C-c C-f C-a' to see what would happen.
Looks like it did the right thing.  I wonder where the problem is?

Jon
-- 
  Everyone who does evil hates the light, and will not come into the
  light for fear that his deeds will be exposed.  But whoever lives by
  the truth comes into the light, so that it may be seen plainly that
  what he has done has been done through God."  -- John 3:20-21 (NIV)

Re: How many people mark spam in gmane groups?

[Ted Zlatanov]

On Wed, 30 Apr 2003, Jon.Ericson@jpl.nasa.gov wrote:
> Ted Zlatanov <tzz@lifelogs.com> writes:
> 
>> Go ahead and make a patch of the docs you mention here and
>> elsewhere in this message and anything else you think needs work in
>> spam-report.el, I'll commit the patch.
> 
> Ok:
> 
> diff -u spam-report.el.old spam-report.el
...
> diff -u gnus.el.old gnus.el
...

I comitted this, thanks.

Ted

Re: How many people mark spam in gmane groups?

[Jon Ericson]

Ted Zlatanov <tzz@lifelogs.com> writes:

> It seemed to work OK for me; I looked at the network traffic and saw
> the acknowledgment from gmane.org arriving for each reported
> article, so I'm pretty sure it reports the spam properly.

I just sent a message that reports that spam-report.el works, but I
might have spoken too soon.  I used gmane.test to mark three messages
as spam.  When I quit the summary buffer the messages from spam-report
showed all three message numbers were reported to Gmane.  But the
Gmane approval page showed only the last article reported three times.

I've cross-posted to gmane.discuss as this appears to be an interface
issue.  

Jon
-- 
  Everyone who does evil hates the light, and will not come into the
  light for fear that his deeds will be exposed.  But whoever lives by
  the truth comes into the light, so that it may be seen plainly that
  what he has done has been done through God."  -- John 3:20-21 (NIV)

Re: How many people mark spam in gmane groups?

[Lars Magne Ingebrigtsen]

Jon Ericson <Jon.Ericson@jpl.nasa.gov> writes:

> I just sent a message that reports that spam-report.el works, but I
> might have spoken too soon.  I used gmane.test to mark three messages
> as spam.  When I quit the summary buffer the messages from spam-report
> showed all three message numbers were reported to Gmane.  But the
> Gmane approval page showed only the last article reported three times.

Hm...  perhaps the changes I made weren't foolproof, anyway.

Do you have a way to reproduce this by, er, more reproducible means?
For instance, three wgets fired off in sequence?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: How many people mark spam in gmane groups?

[Jon Ericson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Jon Ericson <Jon.Ericson@jpl.nasa.gov> writes:
>
>> I just sent a message that reports that spam-report.el works, but I
>> might have spoken too soon.  I used gmane.test to mark three messages
>> as spam.  When I quit the summary buffer the messages from spam-report
>> showed all three message numbers were reported to Gmane.  But the
>> Gmane approval page showed only the last article reported three times.
>
> Hm...  perhaps the changes I made weren't foolproof, anyway.
>
> Do you have a way to reproduce this by, er, more reproducible means?
> For instance, three wgets fired off in sequence?

Ok.  Testing from Gnus shows that just two articles are enough to
cause the bug.  It seems something like this:

  wget http://spam.gmane.org/gmane.test:615 http://spam.gmane.org/gmane.test:625

does not trigger the bug.  Looks like Gmane is off the hook.

Then I went back to Gnus and used spam-report to mark those articles
as spam, saw a message about reporting 615 to Gmane, and when I looked
at http://gmane.org/approve.php I found that article 609 in that group
was reported as spam twice by me.

After setting spam-report-gmane-use-article-number to nil, the bug
disappears.  Here's the bit of code from spam-report.el that this
variable toggles:

	  (spam-report-url-ping "spam.gmane.org" 
		    (format "/%s:%d"
			    (gnus-group-real-name gnus-newsgroup-name)
			    (gnus-summary-article-number)))

I think that (gnus-summary-article-number) is the problem.  From help:

  `gnus-summary-article-number' is a compiled Lisp macro
    -- loaded from "gnus-sum"
  (gnus-summary-article-number)

  Documentation:
  The article number of the article on the current line.
  If there isn't an article number here, then we return the current
  article number.

This isn't really what we want in this case, since it will return the
same article number each time it's called.

Should that branch of code be eliminated?  The only reason given for
it is speed, which really doesn't seem to be a problem in the first
place.

Looks like I'm learning lisp in spite of myself. :)

Jon
-- 
  Everyone who does evil hates the light, and will not come into the
  light for fear that his deeds will be exposed.  But whoever lives by
  the truth comes into the light, so that it may be seen plainly that
  what he has done has been done through God."  -- John 3:20-21 (NIV)

Re: How many people mark spam in gmane groups?

[Ted Zlatanov]

On Mon, 28 Apr 2003, Jon.Ericson@jpl.nasa.gov wrote:
> After setting spam-report-gmane-use-article-number to nil, the bug
> disappears.  Here's the bit of code from spam-report.el that this
> variable toggles:
> 
> 	  (spam-report-url-ping "spam.gmane.org" 
> 		    (format "/%s:%d"
> 			    (gnus-group-real-name gnus-newsgroup-name)
> 			    (gnus-summary-article-number)))
> 
> I think that (gnus-summary-article-number) is the problem.

Yes, it was.  I fixed this in CVS, please take a look and try with
spam-report-gmane-use-article-number.  I have no idea why I used
(gnus-summary-article-number) instead of the variable article which
already holds the article number, as passed to spam-report-gmane.

I want to keep the optimization in place.  NNTP retrieval can be slow,
I've been on a dialup line before.

Thanks
Ted

Re: How many people mark spam in gmane groups?

[Jon Ericson]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Mon, 28 Apr 2003, Jon.Ericson@jpl.nasa.gov wrote:
>> After setting spam-report-gmane-use-article-number to nil, the bug
>> disappears.  Here's the bit of code from spam-report.el that this
>> variable toggles:
>> 
>> 	  (spam-report-url-ping "spam.gmane.org" 
>> 		    (format "/%s:%d"
>> 			    (gnus-group-real-name gnus-newsgroup-name)
>> 			    (gnus-summary-article-number)))
>> 
>> I think that (gnus-summary-article-number) is the problem.
>
> Yes, it was.  I fixed this in CVS, please take a look and try with
> spam-report-gmane-use-article-number.  I have no idea why I used
> (gnus-summary-article-number) instead of the variable article which
> already holds the article number, as passed to spam-report-gmane.
>
> I want to keep the optimization in place.  NNTP retrieval can be slow,
> I've been on a dialup line before.

I just checked the snapshot of this morning, and it looks like this
fix solved the problem.  

Thanks,
Jon
-- 
  Everyone who does evil hates the light, and will not come into the
  light for fear that his deeds will be exposed.  But whoever lives by
  the truth comes into the light, so that it may be seen plainly that
  what he has done has been done through God."  -- John 3:20-21 (NIV)

Re: How many people mark spam in gmane groups?

[Raja R Harinath]

Hi,

Ted Stern <stern+gnus@cray.com> writes:

> Since I started reading the ding-gnus and bbdb-info mailing lists
> through gmane, I've been marking a fair number of messages as spam.
>
> At the moment, this entails me exposing the X-Report-Spam header
> using "C-u g" in the summary buffer, then using browse-url-at-point
> to send the link to my browser.

You can use 't' to show all headers.  Scroll down to X-Report-Spam,
and you have a nice clickable button for the spam-reporting URL.

Of course, it would be nice to do it from the summary buffer.

- Hari
-- 
Raja R Harinath ------------------------------ harinath@cs.umn.edu

Re: How many people mark spam in gmane groups?

[Daniel Néri]

Raja R Harinath <harinath@cs.umn.edu> writes:

> You can use 't' to show all headers.  Scroll down to X-Report-Spam,
> and you have a nice clickable button for the spam-reporting URL.

Or make it visible by default:

 (add-to-list 'gnus-visible-headers "^x-report-\\(un\\)?spam:" t)


Regards,

-- 
Daniel Neri
dne@mayonnaise.net