BUG: NoCeM and foreign groups: does it work?

BUG: NoCeM and foreign groups: does it work?

[Carl D. Roth]

[-- Attachment #1: Type: text/plain, Size: 1707 bytes --]

I've been `using' the NoCeM support for a little while now, but it's
only recently that I got it to *work* -- that is, to save a cache
file.  It brought up some inconsistencies in the NoCeM code that I'd
like to bring up for discussion:

(This discussion relates to gnus 5.4.64, but it's known to occur in
the earlier 5.4 releases)

1.  gnus-nocem-scan-groups seems incapable of scanning for NoCeM in
    foreign groups.  That is, virtual group names in the NoCeM/active
    file have the undesired effect of scanning the local version of
    the group instead.

    This sounds like a stupid point, but my local news server doesn't
    carry all of the NoCeM bulletin groups.

2.  Are all of the NoCeM issuers spelled right?  It appears that
    "clewis@ferret.ocunix.on.ca" and "jem@xpat.com" may be spelled
    incorrectly in the gnus-nocem.el file.

3.  The real deficiency is in how gnus-nocem-enter-article decides
    when a message-id should enter the nocem cache.  Basically, unless
    one of the groups mentioned in the nocem article matches a
    locally-subscribed group, the scan will miss.

    I noticed this problem because I orthogonally set up my newsrc so
    that *all* groups are foreign...  I watched nocem run for about a
    week without matching a single article!

The way I've worked around points 1. and 3. above is to slightly
modify gnus-nocem-enter-article to match groups out of a *different*
hashtb than gnus-newsrc-hashtb.  This new hash table consists of the
gnus-group-real-name mappings of all of the groups in the newsrc.
This way, the nocem groups match up, and I can start accumulating
message-ids.

Please let me know if I've been approaching this in the wrong way!

[-- Attachment #2: .signature --]
[-- Type: text/plain, Size: 64 bytes --]

Carl Roth  mailto:roth@cse.ucsc.edu  (finger for PGP public key)

Re: BUG: NoCeM and foreign groups: does it work?

[Raja R Harinath]

roth@cse.ucsc.edu (Carl D. Roth) writes:
> I've been `using' the NoCeM support for a little while now, but it's
> only recently that I got it to *work* -- that is, to save a cache
> file.  It brought up some inconsistencies in the NoCeM code that I'd
> like to bring up for discussion:
[snip]
> 2.  Are all of the NoCeM issuers spelled right?  It appears that
>     "clewis@ferret.ocunix.on.ca" and "jem@xpat.com" may be spelled
>     incorrectly in the gnus-nocem.el file.

Yup.

The entries in gnus-nocem-issuers are:

   "clewis@ferret.ocunix.on.ca;" ; Chris Lewis
   "jem@xpat.com;"               ; Despammer from Korea

They had the `;' when `gnus-nocem.el' was originally written, but
apparently they've dropped it in the newer NoCeM reports.

- Hari
-- 
Raja R Harinath ------------------------------ harinath@cs.umn.edu
"When all else fails, read the instructions."      -- Cahn's Axiom
"Our policy is, when in doubt, do the right thing."   -- Roy L Ash

Re: BUG: NoCeM and foreign groups: does it work?

[David Moore]

roth@cse.ucsc.edu (Carl D. Roth) writes:

> 1.  gnus-nocem-scan-groups seems incapable of scanning for NoCeM in
>     foreign groups.  That is, virtual group names in the NoCeM/active
>     file have the undesired effect of scanning the local version of
>     the group instead.
> 
>     This sounds like a stupid point, but my local news server doesn't
>     carry all of the NoCeM bulletin groups.
> 

I use the following to scan some foreign groups at a slightly less
frequent rate than normal local groups.  Since it works, it suggests
that foreign groups work also.  Actually, i have news.zippo.com as a
secondary server, if that makes a difference.

(gnus-demon-add-nocem)
(gnus-demon-add-handler 'dmoore::gnus-demon-scan-nocem-zippo 120 t)

(defun dmoore::gnus-demon-scan-nocem-zippo ()
  (let ((gnus-nocem-groups
	 (mapcar (lambda (x) (concat "nntp+news.zippo.com:" x))
		 gnus-nocem-groups))
	(nntp-nov-is-evil nil))
    (save-window-excursion
      (gnus-nocem-scan-groups)
      ;; FIX FIX FIX why is this nntp:... ?
      (gnus-close-server "nntp:news.zippo.com"))))


> 3.  The real deficiency is in how gnus-nocem-enter-article decides
>     when a message-id should enter the nocem cache.  Basically, unless
>     one of the groups mentioned in the nocem article matches a
>     locally-subscribed group, the scan will miss.
> 
>     I noticed this problem because I orthogonally set up my newsrc so
>     that *all* groups are foreign...  I watched nocem run for about a
>     week without matching a single article!
> 
> The way I've worked around points 1. and 3. above is to slightly
> modify gnus-nocem-enter-article to match groups out of a *different*
> hashtb than gnus-newsrc-hashtb.  This new hash table consists of the
> gnus-group-real-name mappings of all of the groups in the newsrc.
> This way, the nocem groups match up, and I can start accumulating
> message-ids.

yeah, something like this is a good idea.


-- 
David Moore <dmoore@ucsd.edu>       | Computer Systems Lab      __o
UCSD Dept. Computer Science - 0114  | Work: (619) 534-8604    _ \<,_
La Jolla, CA 92093-0114             | Fax:  (619) 534-1445   (_)/ (_)
<URL:http://oj.egbt.org/dmoore/>    | In a cloud bones of steel.

Re: BUG: NoCeM and foreign groups: does it work?

[Carl D. Roth]

[-- Attachment #1: Type: text/plain, Size: 689 bytes --]

David Moore <dmoore@UCSD.EDU> writes:

> I use the following to scan some foreign groups at a slightly less
> frequent rate than normal local groups.  Since it works, it suggests
> that foreign groups work also.  Actually, i have news.zippo.com as a
> secondary server, if that makes a difference.

The gnus-secondary-servers list shouldn't make a difference.  Are you
*sure* it's scanning for nocem in the foreign groups?  My experience
was that the group names were foreign, but gnus would scan only on the
local server (using the real-names).  That is, if I listed a group
twice, on two different servers, gnus would scan the same *local*
server twice for the different article ranges.

[-- Attachment #2: .signature --]
[-- Type: text/plain, Size: 64 bytes --]

Carl Roth  mailto:roth@cse.ucsc.edu  (finger for PGP public key)

Re: BUG: NoCeM and foreign groups: does it work?

[Lars Magne Ingebrigtsen]

roth@cse.ucsc.edu (Carl D. Roth) writes:

> The way I've worked around points 1. and 3. above is to slightly
> modify gnus-nocem-enter-article to match groups out of a *different*
> hashtb than gnus-newsrc-hashtb.  This new hash table consists of the
> gnus-group-real-name mappings of all of the groups in the newsrc.

Yes, that sounds like the right way to do it.  Could you mail me
patches? 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: BUG: NoCeM and foreign groups: does it work?

[Carl D. Roth]

[-- Attachment #1: Type: text/plain, Size: 648 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> roth@cse.ucsc.edu (Carl D. Roth) writes:

> > The way I've worked around points 1. and 3. above is to slightly
> > modify gnus-nocem-enter-article to match groups out of a *different*
> > hashtb than gnus-newsrc-hashtb.  This new hash table consists of the
> > gnus-group-real-name mappings of all of the groups in the newsrc.

> Yes, that sounds like the right way to do it.  Could you mail me
> patches? 

I hope posting is OK...

The patch itself also adds a debugging statement to
nocem-check-article.  I run nocem in batch mode, and I want to know
about all of the `invalid' nocem issuers.


[-- Attachment #2: .signature --]
[-- Type: text/plain, Size: 65 bytes --]

Carl Roth  mailto:roth@cse.ucsc.edu  (finger for PGP public key)

[-- Attachment #3: gnus-nocem.diff --]
[-- Type: application/octet-stream, Size: 1784 bytes --]

--- gnus-nocem.el.orig	Fri Jul 25 12:25:51 1997
+++ gnus-nocem.el	Fri Jul 25 12:29:06 1997
@@ -96,6 +96,24 @@
 (defun gnus-nocem-cache-file ()
   (concat (file-name-as-directory gnus-nocem-directory) "cache"))
 
+;;
+;; faster lookups for group names:
+;;
+
+(defvar gnus-real-group-hashtb nil
+  "Real-name mappings of subscribed groups.")
+
+(defun gnus-fill-real-hashtb ()
+  "Fill up a hash table with the real-name mappings from the user's
+active file."
+  (setq gnus-real-group-hashtb (gnus-make-hashtable 123))
+  (mapcar '(lambda (group)
+	     (setq group (gnus-group-real-name (car group)))
+	     (gnus-sethash group
+			   t
+			   gnus-real-group-hashtb))
+	  gnus-newsrc-alist))
+
 (defun gnus-nocem-scan-groups ()
   "Scan all NoCeM groups for new NoCeM messages."
   (interactive)
@@ -105,6 +123,8 @@
     (gnus-make-directory gnus-nocem-directory)
     ;; Load any previous NoCeM headers.
     (gnus-nocem-load-cache)
+    ;; Get the group name mappings:
+    (gnus-fill-real-hashtb)
     ;; Read the active file if it hasn't been read yet.
     (and (file-exists-p (gnus-nocem-active-file))
 	 (not gnus-nocem-active)
@@ -185,6 +205,8 @@
 	(narrow-to-region b e)
 	(setq issuer (mail-fetch-field "issuer"))
 	(widen)
+	(or (member issuer gnus-nocem-issuers)
+	    (message "invalid NoCeM issuer: %s" issuer))
 	(and (member issuer gnus-nocem-issuers) ; We like her....
 	     (gnus-nocem-verify-issuer issuer) ; She is who she says she is...
 	     (gnus-nocem-enter-article)	; We gobble the message..
@@ -223,7 +245,7 @@
 	  (set group nil))
 	 (t
 	  (when (gnus-gethash (gnus-group-real-name (symbol-name group))
-			      gnus-newsrc-hashtb)
+			      gnus-real-group-hashtb)
 	    ;; Valid group.
 	    (beginning-of-line)
 	    (while (= (following-char) ?\t)