Re: Nntp inefficiencies

Re: Nntp inefficiencies

[Lars Magne Ingebrigtsen]

Kurt Swanson <ksw@dna.lth.se> writes:

>     Is Gnus actually trying to get headers for 118989 non-existing
>     articles?

Yes. 

>     Why doesn't it do a LISTGROUP to obtain the list of
>     existing articles?

Because I didn't know about that command?

I've now added it to nntp.el, but how should it be used?  I see two
possible uses -- when entering a group, to get the real list of
articles first, and when `g'-ing, to get the real number of unread
articles in the group buffer.  The latter would probably take far too
long, though, to be of any use.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Nntp inefficiencies

[Paul Franklin]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2919 bytes --]

>>>>> Kurt Swanson writes:

 > Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

 >> and when `g'-ing, to get the real number of unread
 >> articles in the group buffer.  The latter would probably take far too
 >> long, though, to be of any use.

 > No, this wouldn't be good, but we could get the desired information
 > much easier - enter the group (in nntp parlance).  It seems that the
 > entire universe in the gnus-plane is based upon the response to the
 > nntp command, "LIST ACTIVE «group-name»".  The active files for mail
 > groups and for the cache have the same format as the output from this
 > command:
 >   "«group-name» «highest-article-number» «lowest-article-number» y"
 > I have no idea what purpose the "y" is for, well maybe it's a
 > moderation flag? If one instead entered the group via:
 >   "GROUP «group-name»",
 > the response one gets is
 >   "211 «number-available» «lowest» «highest» «group-name»"

 > This is of course an O(1) operation, instead of the O(n) LISTGROUP,
 > when one is only interested in the number of available articles.

 > It would be nice if the entire gnus-plane used the GROUPS sort of
 > format - including the number of articles in the range.  This is
 > especially useful in mail groups and caches.  For instance, I keep a
 > special nnml group for all email which needs my attention to which I
 > have not responded.  Many times my attention can't be given for
 > months, and despite my usually never having more than 20-30 such
 > messages, the difference between max & min article number is often >
 > 200.  Thus I am often queried if I really want to bring up 231 (or
 > whatever) messages, when there are in reality only 17...

Yup; a lot of people have requested something like this.  And Lars
keeps saying that it's too hard to compute, in a variety of
circumstances.  But I think there's another way...

By caching a tiny bit of extra information, Gnus can make a much
better guess at an article count.  Suppose that whenever you exit a
group, it records the actual article count, and the «highest»
article number according to the active file.  Call these cached
values CCount and CHi.

CCount now tells us how many articles are actually below CHi.

Then we do a "g".  We now have new values Lo and Hi.  We can now
calculate an article count:
	CHi = max(CHi, Lo-1)
	count = CCount + (Hi - CHi)
This article count is still not exact, but it's a lot closer.

For nntp groups, this avoids entering the group.  (Is this always
fast enough that it's a non-issue?)  But the real win is other
groups.  Lars hasn't wanted Gnus to go through the unread article
ranges on the fly and compute a real article count.  But since
articles should only disappear from mail groups while in a group or
while in a group or while running expiry, it should be pretty easy
to keep this data totally up to date.

I might have time the week after next to hack on this...

--Paul

Re: Nntp inefficiencies

[Lars Magne Ingebrigtsen]

Paul Franklin <paul@cs.washington.edu> writes:

> This article count is still not exact, but it's a lot closer.

The article count is irrelevant.  It can't be used for anything.  What
we need to know is which articles exist; not how many articles exist.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Nntp inefficiencies

[Lars Magne Ingebrigtsen]

Kurt Swanson <ksw@dna.lth.se> writes:

> It would be nice if the entire gnus-plane used the GROUPS sort of
> format - including the number of articles in the range.  

Use it for what, if I may ask?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Nntp inefficiencies

[David Hedbor]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Paul Franklin <paul@cs.washington.edu> writes:
> 
> > This article count is still not exact, but it's a lot closer.
> 
> The article count is irrelevant.  It can't be used for anything.  What
> we need to know is which articles exist; not how many articles exist.

It might be irrelevant for Gnus, but I think it's relevant for
myself. I want a correct count (mainly on mail groups) because:

1) It looks better when it says "59" instead of "19186" or
   whatever. Purely visual.
2) When I enter a group I don't want a:
   "How many articles from foo (default 2817): "
   when I really only have a 50 or so. This is somewhat annoying.


-- 
[ Below is a random fortune, which is unrelated to the above message. ]
Machines that have broken down will work perfectly when the repairman arrives.

Re: Nntp inefficiencies

[Lars Magne Ingebrigtsen]

Kurt Swanson <ksw@dna.lth.se> writes:

> > The article count is irrelevant.  It can't be used for anything.  What
> > we need to know is which articles exist; not how many articles exist.
> 
> It can be used for displaying the article count.

How?

Gnus has the following read ranges for a group:

((1 . 45))

The active info for the group is

(40 . 50)

Gnus displays "5 unread".

Now, the count of actual articles in the group is 7.

How many unread articles are there really in this group?

1) 5
2) 4
3) 3
4) 2
5) 1
6) all of the above
7) none of the above
8) green
9) to be decided at a later date

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Nntp inefficiencies

[Lars Magne Ingebrigtsen]

Kurt Swanson <ksw@dna.lth.se> writes:

> The example does not illustrate the need.  Consider the following:
> 
> The user has already read 1-5000

Making the numbers bigger does not alter the principle.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Nntp inefficiencies

[Lars Magne Ingebrigtsen]

Kurt Swanson <ksw@dna.lth.se> writes:

> It's not the size but the relation between them.  Take a look again.
> The simple idea is that Gnus should never report more than the total
> number of server-available articles as unread.

And in the case I outlined, what should Gnus do?

This is not a rhetorical question.

In the special case you described, getting the number right is
trivial, but it's a fairly unusual situation.  

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Nntp inefficiencies

[Alan Shutko]

>>>>> "L" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

L> Kurt Swanson <ksw@dna.lth.se> writes:
>> The user has already read 1-5000

L> Making the numbers bigger does not alter the principle.

Yes, it does.  I've had gnus take down my emacs by trying to read a
group where although there were, say, a couple hundred articles, it
said there were a couple million.  After Gnus exhausted my sytem's
memory, it went down....  I think at the time, I traced it down to
Gnus using a vector for the articles, and naively trying to make a
vector that was many millions long.

-- 
Alan Shutko <ats@acm.org> - By consent of the corrupted
While having never invented a sin, I'm trying to perfect several.

Re: Nntp inefficiencies

[Lars Magne Ingebrigtsen]

Alan Shutko <ats@acm.org> writes:

> I think at the time, I traced it down to Gnus using a vector for the
> articles, and naively trying to make a vector that was many millions
> long.

That's not the same problem.  Using LISTGROUP can be used to deal with
this situation.  What we're discussing is the number that's displayed
in the group buffer.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Nntp inefficiencies

[Yair Friedman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> In the special case you described, getting the number right is
> trivial, but it's a fairly unusual situation.  
> 

So can you plase describe the usual situation?

When our server had NOV problems (Strange Line etc.) I edebuged some of
the code to understand what commands are issued and what is the strange
response.  I found out that gnus issus XOVER command for a a very long
and as seems to me unneeded range (Usually 1-MAX).

It happend to me before that there was only 1 unread article in a group
(This number was correctly displayed in the article count though), gnus
spent a lot of time reading 3M of something (nov headers?) to show me a
3 lined article.

There is some confusion regarding the sequences and ranges of nntp.
Maybe it just me confused, but I suspect that the wrong ranges are used
sometimes.

Lars, can you please put some more information regarding this?

--
Yair.

Re: Nntp inefficiencies

[Paul Franklin]

>>>>> Lars Magne Ingebrigtsen writes:

 > Kurt Swanson <ksw@dna.lth.se> writes:

 >> It's not the size but the relation between them.  Take a look again.
 >> The simple idea is that Gnus should never report more than the total
 >> number of server-available articles as unread.

 > And in the case I outlined, what should Gnus do?

 > This is not a rhetorical question.

 > In the special case you described, getting the number right is
 > trivial, but it's a fairly unusual situation.  

Just so everyone's clear, Kurt and I are looking at slightly
different situations.

I'm looking at the mail backends where Gnus has complete control
over what's going on.  For these backends, it should be trivial to
get the number exactly right in all cases, and I think it is.  But
Gnus overestimates in ways which result in large errors with certain
usage habits, as has been noted.

As an extra bonus, it'll help Kurt's scenario somewhat.  :)

Lars, if I get this working, do you want patches?  (I think this
involves adding a group parameter, since I need to remember two
extra per-group integers.  Is there a better way?)

--Paul

Re: Nntp inefficiencies

[Lars Magne Ingebrigtsen]

Yair Friedman <yfriedma@JohnBryce.Co.Il> writes:

> > In the special case you described, getting the number right is
> > trivial, but it's a fairly unusual situation.  
> 
> So can you plase describe the usual situation?

The usual situation is that some articles (usually the FAQs) have a
longer expiry than the rest of the articles.  This leads to huge
active ranges, where the user usually has most of that range marked as 
read.  In this situation, having the actual article count doesn't help 
at all.

> When our server had NOV problems (Strange Line etc.) I edebuged some of
> the code to understand what commands are issued and what is the strange
> response.  I found out that gnus issus XOVER command for a a very long
> and as seems to me unneeded range (Usually 1-MAX).

That sounds like you have `gnus-fetch-old-headers' switched on.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Nntp inefficiencies

[Lars Magne Ingebrigtsen]

Kurt Swanson <ksw@dna.lth.se> writes:

> It's a fairly _usual_ situation for me.

It's a usual situation for you to read groups where the entire active
range is outside of the range of the read articles?  In what usual
situations does this happen?

> Even in the situation I gave, Gnus cannot return an absolutely correct
> unread count unless a listgroup is done.

Yes, it can.  Your example had non-overlapping read ranges and active
range, which means that the number of unread articles is identical to
the number of actual articles in the group.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Nntp inefficiencies

[Lars Magne Ingebrigtsen]

Paul Franklin <paul@cs.washington.edu> writes:

> Just so everyone's clear, Kurt and I are looking at slightly
> different situations.
> 
> I'm looking at the mail backends where Gnus has complete control
> over what's going on.  For these backends, it should be trivial to
> get the number exactly right in all cases, and I think it is.

Well, the situation is really the same for, say, nntp and nnmh.  You
can do a LISTGROUP to get an overview of which articles exist in an
nntp group, and you can do a `directory-files' to get a list of which
articles exist in an nnmh group.  These actions (by my guesstimate) is 
in the same order of magnitude of work and time.

> Lars, if I get this working, do you want patches?  (I think this
> involves adding a group parameter, since I need to remember two
> extra per-group integers.  Is there a better way?)

Could you outline what algorithm you plan on using?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Nntp inefficiencies

[Kurt Swanson]

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> skriver:
> Kurt Swanson <kurt@dna.lth.se> writes:

>>> It's a usual situation for you to read groups where the entire active
>>> range is outside of the range of the read articles?  In what usual
>>> situations does this happen?

>> Every single time I check out a new newsgroup.

> I'd call that a kinda unusual situation.  I don't think it's worth
> doing anything about this just for that.

It's not at all unusual for me to do this.  Also, this happens to me
when I don't read a newsgroup for a few days, from servers with short
storage times, etc.  It is a recurring problem I often encounter, and
to which a fix is easily implemented.

Also, none of this addresses the extremely common, absurd problem in
which gnus can't even keep track of its own messages count.
-- 
Kurt Swanson, Department of Computer Science, Lund University.
kurt@dna.lth.se (© 1998 Kurt Swanson AB)
.