NoCeM flakiness in 0.74 and higher

NoCeM flakiness in 0.74 and higher

[Sudish Joseph]

It's trying to fetch the entire spool each time (well, I kill it
at the start).  

My News/NoCeM/active has:
(setq gnus-nocem-active (quote (
	("news.admin.net-abuse.announce" (10845 . 11363))
	("alt.nocem.misc" (25378 . 26700)))))	

news.lists.filters isn't in there coz I just upgraded to 0.76 in the
hopes of getting proper treatment for alt.nocem.misc (which, BTW,
shouldn't be checked anymore, right? Or maybe not, I haven't been
following this stuff.)  No dice, 0.76 just tried to get everything
again.

[atreides]% telnet news nntp
[...]
group news.lists.filters
211 273 5 306 news.lists.filters
group alt.nocem.misc
211 1122 25533 26811 alt.nocem.misc
[...]

Here's where I stopped gnus:
Checking article 25608 in alt.nocem.misc for NoCeM...
Good signature from user "AutoMoose1 (Automated Cancelmoose)".
Verifying...

It'd been running for a longish while before I killed it.  I wathced
it start on news.lists.filters, said "yay, no more alt.nocem.misc",
went for a smoke to give it time to scan the 250+ articles in n.l.f
and came back to find it'd finished that and gone back to the a.n.m
flakiness. 

This is pretty ugly.  I have to set g-u-nocem to nil to see new news.
It'd be neat if there was an explicit -batch interface to NoCeM,
expecially since the gnus-daemon stuff has never worked for me in
XEmacs.  I've been meaning to see why, never had the time.

While I'm whining about stuff I'm not fixing/coding myself, I might as
well mention that having the nocem stuff work off the scoring
mechanism would be tres cool.  

In particular, it'd be neat if one could have an alist that let you
specify the type of score entry you want generated for articles
that're being eliminated by nocem.  A followup entry (I.e., low score
on references) would be the most useful.  (Subject scores are less
useful, given that a spammer could well choose a subject line that
matches useful articles.)

An alist might be useful if we could come up with a reasonably concise
description of a nocem-issuer that combined newsgroup context.
Something that allowed statements like "issue followup scoredowns for
nocems issued for articles in groups that match this regexp and issued
by this person".  Maybe not very useful.  A single option allowing
score type selection would be very useful, though.

Bleagh, apologies for the rambling.

-Sudish

Re: NoCeM flakiness in 0.74 and higher

[Lars Magne Ingebrigtsen]

Sudish Joseph <sudish@mindspring.com> writes:

> It'd been running for a longish while before I killed it.  I wathced
> it start on news.lists.filters, said "yay, no more alt.nocem.misc",
> went for a smoke to give it time to scan the 250+ articles in n.l.f
> and came back to find it'd finished that and gone back to the a.n.m
> flakiness. 

Can't you just remove a.n.m from `gnus-nocem-groups'?

> In particular, it'd be neat if one could have an alist that let you
> specify the type of score entry you want generated for articles
> that're being eliminated by nocem.  A followup entry (I.e., low score
> on references) would be the most useful.

Yes, that would be useful.  It might be tricky to implement in an
efficient fashion, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: NoCeM flakiness in 0.74 and higher

[David Moore]

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:

> Sudish Joseph <sudish@mindspring.com> writes:
> Can't you just remove a.n.m from `gnus-nocem-groups'?
> 
> > In particular, it'd be neat if one could have an alist that let you
> > specify the type of score entry you want generated for articles
> > that're being eliminated by nocem.  A followup entry (I.e., low score
> > on references) would be the most useful.
> 
> Yes, that would be useful.  It might be tricky to implement in an
> efficient fashion, though.

	If you just want to _hide_ followups rather than score them, I
mailed a semi-patch to do this to the ding list last week.

-- 
David Moore <dmoore@ucsd.edu>       | Computer Systems Lab      __o
UCSD Dept. Computer Science - 0114  | Work: (619) 534-8604    _ \<,_
La Jolla, CA 92093-0114             | Fax:  (619) 534-1445   (_)/ (_)
<URL:http://oj.egbt.org/dmoore/>    | In a cloud bones of steel.

Re: NoCeM flakiness in 0.74 and higher

[Sudish Joseph]

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:
> Can't you just remove a.n.m from `gnus-nocem-groups'?

I did.  But from what I posted, isn't it indicative of a (pretty
scary) bug?  Namely that gnus has lost track of what articles it has
already seen.

>> In particular, it'd be neat if one could have an alist that let you
>> specify the type of score entry you want generated for articles
>> that're being eliminated by nocem.  A followup entry (I.e., low score
>> on references) would be the most useful.

> Yes, that would be useful.  It might be tricky to implement in an
> efficient fashion, though.

It could as simple as providing a variable, gnus-nocem-scorefile.  If
non-nil, nocem handling in done via score files.  Instead of inserting
msg-id's in News/NoCeM/cache, insert a "followup" entry in this
scorefile.  

The user is free to have that scorefile used in whatever groups they
see fit, using any of the many options gnus provides.  I, for example,
would add a 'files entry for it in all.SCORE.

The nice thing of doing it this way is that I can run a -batch nocem
process w/o ever having to worry about coming up with lightweight
locking for NoCeM/active.  Do a score file cache flush and you have
the latest scorefile.

Of course, if the user ever decides to use that scorefile in another
context, all bets are off.  

Hmm, to address David's worry about the size of the resulting
scorefile: if this method remains as simple as only allowing score
entries of one type (it might be followup or subject, but fixed), we
can even do it w/o changing the format of NoCeM/active.

We'd need to add some form of score-file handler hook that lets us
treat that file as a normal scorefile and massage it's data into
normal score format at the time it gets read into the score cache --
not hard or time intensive, each entry is of the same format.  Note
that not treating it as a scorefile would mean all types of kluges to
allow concurrent -batch access and would be a lose, IMO.

Enough fantasizing for today.
Later,
-Sudish

Re: NoCeM flakiness in 0.74 and higher

[David Moore]

Sudish Joseph <sudish@mindspring.com> writes:

> It's trying to fetch the entire spool each time (well, I kill it
> at the start).  
> 
> My News/NoCeM/active has:
> (setq gnus-nocem-active (quote (
> 	("news.admin.net-abuse.announce" (10845 . 11363))
> 	("alt.nocem.misc" (25378 . 26700)))))	
> 
> news.lists.filters isn't in there coz I just upgraded to 0.76 in the
> hopes of getting proper treatment for alt.nocem.misc (which, BTW,
> shouldn't be checked anymore, right? Or maybe not, I haven't been
> following this stuff.)  No dice, 0.76 just tried to get everything
> again.

	news.lists.filters replaces alt.nocem.misc and
news.admin.net-abuse.bulletins replaces news.admin.net-abuse.announce.
I left the old ones in there for the time being, since there are still
some messages in them to allow a smoother transition.  You can certainly
remove the two old names from the list if the new ones have arrived at
your site.

> Here's where I stopped gnus:
> Checking article 25608 in alt.nocem.misc for NoCeM...
> Good signature from user "AutoMoose1 (Automated Cancelmoose)".
> Verifying...

	If it was stuck verifying, you could always (setq
gnus-nocem-verifier nil).  This is also substantially faster, at the
cost that someone might forge a nocem posting.

> This is pretty ugly.  I have to set g-u-nocem to nil to see new news.
> It'd be neat if there was an explicit -batch interface to NoCeM,
> expecially since the gnus-daemon stuff has never worked for me in
> XEmacs.  I've been meaning to see why, never had the time.

	I've been thinking about a batch interface also, and might do it
since I'm thinking of rewriting some of the nocem code to support some
things in the soon to be official nocem BNF and prototype rfc.

> While I'm whining about stuff I'm not fixing/coding myself, I might as
> well mention that having the nocem stuff work off the scoring
> mechanism would be tres cool.  

	Yeah, that might work. :)  But I still haven't tried using
scoring at all, hmm, maybe some time when xemacs stabilizes a bit.

> In particular, it'd be neat if one could have an alist that let you
> specify the type of score entry you want generated for articles
> that're being eliminated by nocem.  A followup entry (I.e., low score
> on references) would be the most useful.  (Subject scores are less
> useful, given that a spammer could well choose a subject line that
> matches useful articles.)

	Hmm, that's a thought.  I've been leaning the other direction of
having automatic hiding of followups to nocem messages.  My NoCeM cache
varies from about 200k to 800k.  If the information in there was spread
out over several separate group score files, and also expanded to do
followups via current scoring mechanism, I could see it taking up some
pretty significant amounts of disk.  5-10M easily.  (or maybe not. :)

	There is no subject information available in the nocem control
messages, only message id.  So you'd have to score on that.

> An alist might be useful if we could come up with a reasonably concise
> description of a nocem-issuer that combined newsgroup context.
> Something that allowed statements like "issue followup scoredowns for
> nocems issued for articles in groups that match this regexp and issued
> by this person".  Maybe not very useful.  A single option allowing
> score type selection would be very useful, though.

	Yeah, this is might be a good idea. You'd also want to include
some other nocem issues such as the content type specifier.

-- 
David Moore <dmoore@ucsd.edu>       | Computer Systems Lab      __o
UCSD Dept. Computer Science - 0114  | Work: (619) 534-8604    _ \<,_
La Jolla, CA 92093-0114             | Fax:  (619) 534-1445   (_)/ (_)
<URL:http://oj.egbt.org/dmoore/>    | In a cloud bones of steel.

Re: NoCeM flakiness in 0.74 and higher

[Sudish Joseph]

In article <rv20cpt12l.fsf@sdnp5.ucsd.edu>,
David Moore <dmoore@UCSD.EDU> writes:
> some messages in them to allow a smoother transition.  You can certainly
> remove the two old names from the list if the new ones have arrived at
> your site.

Cool, I'll zap n.a.n.a as well.  Tx.

>> Here's where I stopped gnus:
>> Checking article 25608 in alt.nocem.misc for NoCeM...
>> Good signature from user "AutoMoose1 (Automated Cancelmoose)".
>> Verifying...

> 	If it was stuck verifying, you could always (setq
> gnus-nocem-verifier nil).  This is also substantially faster, at the
> cost that someone might forge a nocem posting.

Oops, I should've been clearer.  It isn't hanging--it's reading
articles that it's marked as read in NoCeM/active.  The entry for
alt.nocem.misc in my orig post is ("alt.nocem.misc" (25378 . 26700)).
But it's verifying article 25608 above.

> 	I've been thinking about a batch interface also, and might do it
> since I'm thinking of rewriting some of the nocem code to support some
> things in the soon to be official nocem BNF and prototype rfc.

Yay!  Much power to you.  :-)

> 	Yeah, that might work. :)  But I still haven't tried using
> scoring at all, hmm, maybe some time when xemacs stabilizes a bit.

You might want to look at it before you do anything to gnus-nocem.el.
The best part of the whole scoring interface/implementation is that
it's so damned flexible (bows deeply in the direction of Per and
Lars).  

In particular, using it for any data whose sole purpose is article
purging/selection is a win in many ways: easier/better to reuse
(great) code, users may use the existing score file interface to
manipulate this data, less formats used for data the better, score
processing is extremely fast, etc.

> 	There is no subject information available in the nocem control
> messages, only message id.  So you'd have to score on that.

I was thinking in terms of generating a subject score entry at the
time the actual article was encountered during scoring.  Since we have
the whole xref record, getting the subject is easy.  The (big)
drawback is that we'd need a new score field type (exactly like
'followup 'cept that it lowers the subject score).  A new score type
would be perfect for references as well: 'nocem might mean kill this
article and generate a "lower on references" entry.

Not a big deal.  A flat "lower all nocem'ed articles on msg-id and
references" is plentiful.  I wouldn't use Subject: anyway, and I doubt
many would.

Argh, I'm running speculatively at the mouth again, 
-Sudish

Re: NoCeM flakiness in 0.74 and higher

[David Moore]

Sudish Joseph <sudish@mindspring.com> writes:

> >> Here's where I stopped gnus:
> >> Checking article 25608 in alt.nocem.misc for NoCeM...
> >> Good signature from user "AutoMoose1 (Automated Cancelmoose)".
> >> Verifying...
> 
> > 	If it was stuck verifying, you could always (setq
> > gnus-nocem-verifier nil).  This is also substantially faster, at the
> > cost that someone might forge a nocem posting.
> 
> Oops, I should've been clearer.  It isn't hanging--it's reading
> articles that it's marked as read in NoCeM/active.  The entry for
> alt.nocem.misc in my orig post is ("alt.nocem.misc" (25378 . 26700)).
> But it's verifying article 25608 above.

	Oh, interesting.  I've not seen this happen and looking in the
code in 0.74 can't really see where it might be happening.
gnus-nocem-scan-groups should call gnus-retrieve-headers with 1+26700 to
26811 (the max active from nntp in this case).  Maybe it's not loading
your cache properly?  Anyone else seeing something like this?


-- 
David Moore <dmoore@ucsd.edu>       | Computer Systems Lab      __o
UCSD Dept. Computer Science - 0114  | Work: (619) 534-8604    _ \<,_
La Jolla, CA 92093-0114             | Fax:  (619) 534-1445   (_)/ (_)
<URL:http://oj.egbt.org/dmoore/>    | In a cloud bones of steel.