Red GNUS suggestion: pseudo Xref: handling w/o NOV support.

Red GNUS suggestion: pseudo Xref: handling w/o NOV support.

[Sudish Joseph]

My news admin pointed out a good reason for not having Xref: full in
their overview.fmt.  The vast majority of their clients use PC-based
readers and none of those readers support xref handling.  In the
course of the discussion, I had what I think is a good idea, but's I'm
too sleepy to know for sure or to rewrite it.  I'm just forwarding the
post and hitting bed, you decide. :>

Um, the last few paras are the main thing, the stuff about
nntp-nov-is-evil and nntp-server-opened-hook would be nice, too.

-Sudish 

------- Start of forwarded message -------
From: Sudish Joseph <sudish@mindspring.com>
Newsgroups: mindspring.discussion
Subject: Re: Add XREF to mindspring's overview.fmt...pretty please?
Date: 19 Jun 1996 01:42:26 -0400
Organization: MindSpring Enterprises
Message-ID: <m2rarcqri5.fsf@atreides.erehwon.org>
References: <m2688oslv1.fsf@atreides.erehwon.org>
	<4q7rhm$2np@kaamos.mindspring.com>
	<m2vigoqw7b.fsf@atreides.erehwon.org>
	<4q810d$39b@kaamos.mindspring.com>
	<knyblkgyu3.fsf@xena.mindspring.com>

Robert Sanders <rsanders@mindspring.net> writes:
> I've thought about enhancing GNUS 5.2 to keep a message ID database
> for already seen crossposted articles.  

This isn't necessary, you could just grok the Xref: header and kill on
article-id.  GNUS does this if nntp-nov-is-evil is t.  Or, it did,
last time I fixed it. :-)

What seems needed here is to do the above even when nntp-nov-is-evil
is nil.  Adding a small function to nntp-server-opened-hook that did a
``LIST overview.fmt'', checked for ``Xref: full'', and set a variable
that was checked in addition to nntp-nov-is-evil for the above stuff
is all that's needed.

> The drawbacks are a) actually having to modify the GNUS code without
> the benefit of intensive psychiatric care,

Oh, rubbish.  Lars's code is clean and fun to hack.  

> b) being somewhat otherwise occupied, 

No cure for that, unfortunately. :(

> and c) emacs' lack of an efficient random access on-disk database
> system.  Saving, loading, and walking huge alists isn't my idea of
> fast or fun.

There's a patch that adds gdbm support to Emacs floating around.  If
you're really interested, I could dig up a reference.  I believe Bill
Perry's added this to XEmacs already--not sure if it's going to be in
19.14, though.

There was some talk of adding an nngdbm backend to GNUS--it'd give all
the benefits of nnml w/o the hit of large directory lookups.

> If you're a patient man, setting gnus-nov-is-evil to 't' will solve
> the problem, albeit in a painfully slow manner.  All I can say is that
> it's not terribly bad behind the office T1.

Ugh.  No thanks. :)

Hmm, it just struck me.  We already have all the message-id's for
articles that we haven't read.  All we have to do is generate a
transient scorefile that lowered on those msg-ids.  This wouldn't be
slow for even large groups, scoring is very fast in GNUS.

Heck, we wouldn't even have to write it to disk, the way GNUS already
caches score entries (they're only read once per session, irrespective
of your access pattern).  This would be akin to how other xref-capable
readers do this when xref info is available.  Writing that file to
disk is useless coz all concerned articles are already on your server,
by definition.

This would be a significant win, coz no other reader does this:
crosspost handling w/o xrefs for unread articles.

-Sudish 
------- End of forwarded message -------

Re: Red GNUS suggestion: pseudo Xref: handling w/o NOV support.

[Lars Magne Ingebrigtsen]

Sudish Joseph <sudish@mindspring.com> writes:

> My news admin pointed out a good reason for not having Xref: full in
> their overview.fmt.  The vast majority of their clients use PC-based
> readers and none of those readers support xref handling. 

That might be a good reason if not for a slight niggle:  It's
bullshit.  Most modern newsreaders that run under MicroSloth
"operating systems" use the Xref header.  (At least that's what I've
been told.)

Even if it were true -- so what?  Adding Xref to the NOV databases
doesn't hurt newsreaders that don't make use of the information.  So
not adding Xref to the databases is rather, uhm.  Let me stop there
before I get nailed by CDA² (coming soon to an Internet near you).

> Hmm, it just struck me.  We already have all the message-id's for
> articles that we haven't read.  All we have to do is generate a
> transient scorefile that lowered on those msg-ids.  This wouldn't be
> slow for even large groups, scoring is very fast in GNUS.

This is a very good idea.  Actually, we don't even have to use
scoring.  We just add the Message-IDs to the NoCeM hashtable, which
will efficiently censor all articles that we have already read.

The drawback here is (of course) that such an approach wouldn't work
if you read one article in one session and the other copy of the
article in another session.  We could write the list of Message-IDs
already seen to a file, but that file would grow rather big...

-- 
  "Yes.  The journey through the human heart 
     would have to wait until some other time."

Re: Red GNUS suggestion: pseudo Xref: handling w/o NOV support.

[Per Abrahamsen]

>>>>> "SJ" == Sudish Joseph <sudish@mindspring.com> writes:

SJ> The vast majority of their clients use PC-based
SJ> readers and none of those readers support xref handling.  

Mozilla 3.0 supports xref handling.  Last I heard, the last (beta?)
version of Free Agent supports xrefs too.

Re: Red GNUS suggestion: pseudo Xref: handling w/o NOV support.

[Steven L Baur]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:

Lars> Sudish Joseph <sudish@mindspring.com> writes:
>> My news admin pointed out a good reason for not having Xref: full in
>> their overview.fmt.  The vast majority of their clients use PC-based
>> readers and none of those readers support xref handling. 

 ...
Lars> The drawback here is (of course) that such an approach wouldn't work
Lars> if you read one article in one session and the other copy of the
Lars> article in another session.  We could write the list of Message-IDs
Lars> already seen to a file, but that file would grow rather big...

Another approach would appear to be making use of the XHDR XREF
command if available.  This would require issuing two commands instead
of one when getting the overview, and merging the information
returned, but should still be faster than the alternatives.

-- 
steve@miranova.com baur
Unsolicited commercial e-mail will be proofread for $250/hour.
Andrea Seastrand: For your vote on the Telecom bill, I will vote for anyone
except you in November.

Re: Red GNUS suggestion: pseudo Xref: handling w/o NOV support.

[Lars Magne Ingebrigtsen]

Steven L Baur <steve@miranova.com> writes:

> Another approach would appear to be making use of the XHDR XREF
> command if available.  This would require issuing two commands instead
> of one when getting the overview, and merging the information
> returned, but should still be faster than the alternatives.

But the XHDR command isn't nice when dealing with headers that are not
stored in the NOV database.  In fact, it'll be as slow as just
requesting the article heads.  So `gnus-nov-is-evil' is just as good,
I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: Red GNUS suggestion: pseudo Xref: handling w/o NOV support.

[Sudish Joseph]

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:
> Sudish Joseph <sudish@mindspring.com> writes:
> > My news admin pointed out a good reason for not having Xref: full in
> > their overview.fmt.  The vast majority of their clients use PC-based
> > readers and none of those readers support xref handling.=20
> 
> That might be a good reason if not for a slight niggle:  It's
> bullshit.  Most modern newsreaders that run under MicroSloth
> "operating systems" use the Xref header.  (At least that's what I've
> been told.)

I did a dejanews search on 'xref add' (using "performance" as a
keyword was a lose) over news.software.* before I accepted that
statement (remember, I was the one wanting xref :-).  Most hits fit one
of 3 classes:  a) INN/NOV faq :-)  b) Suggestions that xref be added
w/o any performance analysis  c) Statements to the effect that no PC
readers support xref, so why should J. Random Sysop add it.  

> Even if it were true -- so what?  Adding Xref to the NOV databases
> doesn't hurt newsreaders that don't make use of the information.  So
> not adding Xref to the databases is rather, uhm.  Let me stop there
> before I get nailed by CDA=B2 (coming soon to an Internet near you).

This was more or less my argument.  The newsdmin pointed out that
there will be a hit in download time due to the extra size of the
xover data.  The comment about "most PC readers" was in response to me
pointing out that an xref-capable reader will download less headers
and thus save space, which blew away my argument.  He put it rather
well: from the pov of an ISP, one of the more important selling points
is "headers download faster".

Hey, I fought the good fight, stop making me feel guilty. :)

> > Hmm, it just struck me.  We already have all the message-id's for
> > articles that we haven't read.  All we have to do is generate a
> > transient scorefile that lowered on those msg-ids.  This wouldn't be
> > slow for even large groups, scoring is very fast in GNUS.
> 
> This is a very good idea.  Actually, we don't even have to use
> scoring.  We just add the Message-IDs to the NoCeM hashtable, which
> will efficiently censor all articles that we have already read.

Maybe move the hashtb funcs into nnoo or something?  A new user might
well not enable NoCeM support (or those suckers who, like me, cannot
use a free US-released PGP in the US w/o residency, and cannot import
a non-US version into the US--has this changed, anyone?)  Not a big
deal, though.

Also, what do you think about adding a adding a test of overview.fmt
to nntp-server-opened-hook?  I think there's a good idea there.  Why
not have a probe function that sent XOVER, etc., to the server and
determined the optimal config for nntp reading?  Running this every
time seems bogus...maybe a nntp-suggest-kewl-config command?  There
aren't very many critical variables to setup here and it might reduce
the size of the oh-so-neat-FAQ that Steve Baur has blessed us with.
This would let people who have no wish to ever know the meaning of the
word XOVER have a decent shot of acheiving their goal. :-)

> The drawback here is (of course) that such an approach wouldn't work
> if you read one article in one session and the other copy of the
> article in another session.  We could write the list of Message-IDs
> already seen to a file, but that file would grow rather big...

Yeah, writing that out to a file seems yucky.  How about running that
scorefile/hashtb over all groups when quitting GNUS?

-Sudish

Re: Red GNUS suggestion: pseudo Xref: handling w/o NOV support.

[Lars Magne Ingebrigtsen]

Sudish Joseph <sudish@mindspring.com> writes:

> I did a dejanews search on 'xref add' (using "performance" as a
> keyword was a lose) over news.software.* before I accepted that
> statement (remember, I was the one wanting xref :-).  Most hits fit one
> of 3 classes:  a) INN/NOV faq :-)  b) Suggestions that xref be added
> w/o any performance analysis  c) Statements to the effect that no PC
> readers support xref, so why should J. Random Sysop add it.  

Well, you should take what you read on news.software.readers with a
few shovels of salt.  There's quite a lot of people there who don't
know *anything* about *anything*.   (And that's just speaking of
newsreader authors.  The users are *really* clueless.  :-)  

> This was more or less my argument.  The newsdmin pointed out that
> there will be a hit in download time due to the extra size of the
> xover data.

The size increase will be in the 10% range.  However, since
modern newsreaders are in fact able to make use of the data, it will
mean that less data will be needed to be transferred from the ISP to
the users since one doesn't have to read any article twice.

> Maybe move the hashtb funcs into nnoo or something?  A new user might
> well not enable NoCeM support (or those suckers who, like me, cannot
> use a free US-released PGP in the US w/o residency, and cannot import
> a non-US version into the US--has this changed, anyone?)  Not a big
> deal, though.

Well, I sorta meant that one could use the same mechanism without
using the actual NoCeM code.

> Also, what do you think about adding a adding a test of overview.fmt
> to nntp-server-opened-hook?  I think there's a good idea there.  Why
> not have a probe function that sent XOVER, etc., to the server and
> determined the optimal config for nntp reading?  Running this every
> time seems bogus...maybe a nntp-suggest-kewl-config command? 

That's a good idea.  I've added this to the Red Gnus todo list.

> Yeah, writing that out to a file seems yucky.  How about running that
> scorefile/hashtb over all groups when quitting GNUS?

Requesting headers for all groups when exiting Gnus?  I don't think
that's a good idea.

-- 
  "Yes.  The journey through the human heart 
     would have to wait until some other time."

Re: Red GNUS suggestion: pseudo Xref: handling w/o NOV support.

[Sudish Joseph]

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:
> Well, you should take what you read on news.software.readers with a
> few shovels of salt.  There's quite a lot of people there who don't
> know *anything* about *anything*.   (And that's just speaking of
> newsreader authors.  The users are *really* clueless.  :-)  

Very true.  I passed on the data Per provided about Netscape 3.0 +
Agent and they're adding Xref. :-)  This place has the most reasonable
sysadmin I've seen, yet.  The man was actually discussing why he
thought xref might be a lose.  On being provided a sound reason, he
Did The Right Thing.  Amazing, IMO.  A pearl among admins.

> > Yeah, writing that out to a file seems yucky.  How about running that
> > scorefile/hashtb over all groups when quitting GNUS?
> 
> Requesting headers for all groups when exiting Gnus?  I don't think
> that's a good idea.

*blush*  I didn't write that, really.  You have to remove some of the
AI stuff from message-mode...

-Sudish