building a smarter gnus

building a smarter gnus

[Wes Hardaker]

half of the speed problems associated with gnus derive from the fact
that gnus uses entire lists of articles when requesting stuff from the
lower ranges.

From what I've gathered (I've started implementing a nnsourceforge
component, which is functionally working but slow), article lists
passed to nnXX-request-articles are always a fully expanded list
(which is causes memory and speed problems).  Wouldn't it be more
efficient to maybe pass it a compressed list instead and say "retrieve
anything from this range"?  nnimap, for instance, I think checks every
article in the list passed to it which causes problems if there are
articles which don't exist anymore.  my nnsourceforge backend is full
of holes, since I'm hoping to use the bug tracking number as the
article id:

  4        273006: blah                           020328
           ^^^^^^
           max - min

So, I don't think it would be hard to create a new function called
"nnXX-request-ranged-articles" which could accept a range instead.  If
the function didn't exist, then fall back to the one that only accepts
an expanded list.

Of course, I really don't know what I'm talking about and I'm just
about nnoo'ed in the head at this point.

Functionally, I want to write a nnXX-request-articles function that is
not passed a list of 273006 articles.  Rather I want to be passed
(226228 . 536386) or something.  I can fill in the holes much more
quickly and using less memory space.  It looks like this isn't
possible right now?

(in part because nnxx-request-group must only return num, max and min
without a valid range list?)

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."

Re: building a smarter gnus

[Paul Jarc]

Wes Hardaker <wes@hardakers.net> wrote:
> From what I've gathered (I've started implementing a nnsourceforge
> component, which is functionally working but slow), article lists
> passed to nnXX-request-articles are always a fully expanded list
> (which is causes memory and speed problems).

Do you mean nnXX-retrieve-headers?

> Wouldn't it be more efficient to maybe pass it a compressed list
> instead and say "retrieve anything from this range"?

According to the manual, backends are already *encouraged* to support
that in nnchoke-retrieve-headers, but Gnus doesn't take advantage of
it because not all backends *do* support it yet.

> So, I don't think it would be hard to create a new function called
> "nnXX-request-ranged-articles" which could accept a range instead.

It should indeed be a separate function, I think.  That's the most
convenient way for the backend to express that it's capable of
handling ranges.

> It looks like this isn't possible right now?
>
> (in part because nnxx-request-group must only return num, max and min
> without a valid range list?)

That isn't a problem.  It's already ok for backends to fail to provide
headers for requested articles that don't exist, AFAICT, so it's ok
for Gnus to request headers for too many articles.


paul

Re: building a smarter gnus

[Wes Hardaker]

>>>>> On Thu, 28 Mar 2002 19:54:33 -0500, prj@po.cwru.edu (Paul Jarc) said:

Paul> Wes Hardaker <wes@hardakers.net> wrote:
>> From what I've gathered (I've started implementing a nnsourceforge
>> component, which is functionally working but slow), article lists
>> passed to nnXX-request-articles are always a fully expanded list
>> (which is causes memory and speed problems).

Paul> Do you mean nnXX-retrieve-headers?

Uh, yeah.

>> Wouldn't it be more efficient to maybe pass it a compressed list
>> instead and say "retrieve anything from this range"?

Paul> According to the manual, backends are already *encouraged* to
Paul> support that in nnchoke-retrieve-headers, but Gnus doesn't take
Paul> advantage of it because not all backends *do* support it yet.

Grr...  You know, I searched for "backend" in the manual and didn't
see anything so just copied an example.  I now see I should have
searched for "back end".  Sigh.

>> So, I don't think it would be hard to create a new function called
>> "nnXX-request-ranged-articles" which could accept a range instead.

Paul> It should indeed be a separate function, I think.  That's the most
Paul> convenient way for the backend to express that it's capable of
Paul> handling ranges.

Yeah, that seems sensible to me.  Now, who's going to implement the
support for doing this :-)

>> It looks like this isn't possible right now?
>> 
>> (in part because nnxx-request-group must only return num, max and min
>> without a valid range list?)

Paul> That isn't a problem.  It's already ok for backends to fail to
Paul> provide headers for requested articles that don't exist, AFAICT,
Paul> so it's ok for Gnus to request headers for too many articles.

Right.  But it's inefficient, in my opinion, to request an extremely
large number of articles when only a few exist.

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."

Re: building a smarter gnus

[Paul Jarc]

Wes Hardaker <wes@hardakers.net> wrote:
> You know, I searched for "backend" in the manual and didn't see
> anything so just copied an example.  I now see I should have
> searched for "back end".  Sigh.

The manual used to say "backend", but it was changed at RMS's request.
Feel free to let him know how much you appreciate it. :)

>> It should indeed be a separate function, I think.  That's the most
>> convenient way for the backend to express that it's capable of
>> handling ranges.

And while we're at it, we can make this new function return the
headers as a Lisp object instead of in the " *nntpd*" buffer.  nntp is
the only backend that *really* needs to parse all that text.  Backends
that determine their own NOV data storage format can choose a format
designed to work well with whatever Gnus<->backend interface we come
up with, so we might as well choose an interface that makes less work
for Gnus.

> Yeah, that seems sensible to me.  Now, who's going to implement the
> support for doing this :-)

I'll take a look at it after I finish some work on nnmaildir.

> But it's inefficient, in my opinion, to request an extremely large
> number of articles when only a few exist.

Efficiency is not a matter of opinion.  We can measure the speed of
each method.


paul

Re: building a smarter gnus

[Wes Hardaker]

>>>>> On Tue, 02 Apr 2002 14:42:36 -0500, prj@po.cwru.edu (Paul Jarc) said:

Paul> The manual used to say "backend", but it was changed at RMS's request.
Paul> Feel free to let him know how much you appreciate it. :)

I hate to ask, but why?  (I'm sure it was on this list and I missed it).

Paul> And while we're at it, we can make this new function return the
Paul> headers as a Lisp object instead of in the " *nntpd*" buffer.  nntp is
Paul> the only backend that *really* needs to parse all that text.  Backends
Paul> that determine their own NOV data storage format can choose a format
Paul> designed to work well with whatever Gnus<->backend interface we come
Paul> up with, so we might as well choose an interface that makes less work
Paul> for Gnus.

Good idea.

>> Yeah, that seems sensible to me.  Now, who's going to implement the
>> support for doing this :-)

Paul> I'll take a look at it after I finish some work on nnmaildir.

Excellent.

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."

Re: building a smarter gnus

[Paul Jarc]

Wes Hardaker <wes@hardakers.net> wrote:
>>>>>> On Tue, 02 Apr 2002 14:42:36 -0500, prj@po.cwru.edu (Paul Jarc) said:
>
> Paul> The manual used to say "backend", but it was changed at RMS's request.
> Paul> Feel free to let him know how much you appreciate it. :)
>
> I hate to ask, but why?  (I'm sure it was on this list and I missed it).

He didn't say, AFAICT.

>>> Yeah, that seems sensible to me.  Now, who's going to implement the
>>> support for doing this :-)
>
> Paul> I'll take a look at it after I finish some work on nnmaildir.

Hmm... maybe it would be better to skip this and work on using strings
instead of numbers for article identifiers.  If we're heading there
eventually (I hope), then ranges will have to be thrown out or
reinvented anyway.


paul

Re: building a smarter gnus

[John H Palmieri]

prj@po.cwru.edu (Paul Jarc) writes:

> Wes Hardaker <wes@hardakers.net> wrote:
>>>>>>> On Tue, 02 Apr 2002 14:42:36 -0500, prj@po.cwru.edu (Paul Jarc) said:
>>
>> Paul> The manual used to say "backend", but it was changed at RMS's request.
>> Paul> Feel free to let him know how much you appreciate it. :)
>>
>> I hate to ask, but why?  (I'm sure it was on this list and I missed it).
>
> He didn't say, AFAICT.

I think the issue was that "backend" is not a word, whereas "back" and
"end" are.

-- 
J. H. Palmieri
Dept of Mathematics, Box 354350    mailto:palmieri@math.washington.edu
University of Washington           http://www.math.washington.edu/~palmieri/
Seattle, WA 98195-4350

Re: building a smarter gnus

[Paul Jarc]

John H Palmieri <palmieri@math.washington.edu> wrote:
> prj@po.cwru.edu (Paul Jarc) writes:
>> Wes Hardaker <wes@hardakers.net> wrote:
>>>>>>>> On Tue, 02 Apr 2002 14:42:36 -0500, prj@po.cwru.edu (Paul Jarc) said:
>>>> The manual used to say "backend", but it was changed at RMS's request.
>>>
>>> I hate to ask, but why?  (I'm sure it was on this list and I missed it).
>>
>> He didn't say, AFAICT.
>
> I think the issue was that "backend" is not a word, whereas "back" and
> "end" are.

That's how ShengHuo interpreted the request, but RMS didn't actually
say that, at least not in the message ShengHuo quoted.
<URL:http://www.gnus.org/list-archives/ding/200109/msg00324.html>


paul

Re: building a smarter gnus

[Wes Hardaker]

>>>>> On Tue, 02 Apr 2002 17:18:42 -0500, prj@po.cwru.edu (Paul Jarc) said:

Paul> That's how ShengHuo interpreted the request, but RMS didn't actually
Paul> say that, at least not in the message ShengHuo quoted.
Paul> <URL:http://www.gnus.org/list-archives/ding/200109/msg00324.html>

Ahh.  I do remember that discussion now.  Silly then, silly now.

Terminology is defined for particular applications so that a
consistent method of conversing can be established.  Using "backend"
means we've defined a new term to mean the underpinnings of gnus and
how it interacts with something else.  If Lars says "please look at my
new backend", I'm much more likely to go look for his code in the CVS
tree.  On the other hand, if he had said "please look at my back end"
I'm much more likely to go and decode his xface ;-)  Can someone send
RMS a note and tell him "emacs" is not a word, could he please use
"editor" instead?

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."

Re: building a smarter gnus

[Wes Hardaker]

>>>>> On Tue, 02 Apr 2002 15:43:57 -0500, prj@po.cwru.edu (Paul Jarc) said:

Paul> Hmm... maybe it would be better to skip this and work on using
Paul> strings instead of numbers for article identifiers.  If we're
Paul> heading there eventually (I hope), then ranges will have to be
Paul> thrown out or reinvented anyway.

Well, ranges are supported now by some backends (though, uh never
tested?) so why not add a quick hack to pass those ranges to the "back
end"s that can handle them until you get strings implemented (which
will probably be a lot farther down the road)?

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."

Re: building a smarter gnus

[Paul Jarc]

Wes Hardaker <wes@hardakers.net> wrote:
> Can someone send RMS a note and tell him "emacs" is not a word,
> could he please use "editor" instead?

You can.  But he didn't actually say that "backend" is not a word, so
I'd suggest starting by asking him why he doesn't like "backend".


paul

Re: building a smarter gnus

[Harry Putnam]

Wes Hardaker <wes@hardakers.net> writes:

> Grr...  You know, I searched for "backend" in the manual and didn't
> see anything so just copied an example.  I now see I should have
> searched for "back end".  Sigh.

There is lots of stuff like that in emacs manual too.
Yesterday I went looking for the key strokes to change the size of a
window.

M-x apropos increase.*size
M-x apropos increase
M-x apropos size

Finally tried something really general
M-x apropos window  And after enough steping thru, found:
  C-x ^		enlarge-window

I never did find the opposite command
But it turns out you can do it with -numbers like
    C-u -2 C-x ^  or C-- 2 C-x ^
 And there is:
     C-x =  balance-windows

I had invented a few new curse words by the time I finally found it.
And just guessed the rest.

Re: building a smarter gnus

[Wes Hardaker]

>>>>> On Tue, 02 Apr 2002 17:39:45 -0500, prj@po.cwru.edu (Paul Jarc) said:

Paul> Wes Hardaker <wes@hardakers.net> wrote:
>> Can someone send RMS a note and tell him "emacs" is not a word,
>> could he please use "editor" instead?

Paul> You can.

(I don't feel up to starting another pointless flame war today.  I
mean, the change was made so....)

Paul> But he didn't actually say that "backend" is not a word, so I'd
Paul> suggest starting by asking him why he doesn't like "backend".

Probably because adding a word to his personal dictionary is a bad thing?
-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."

Re: building a smarter gnus

[Toby Speight]

0> In article <m3it79zy2l.fsf@multivac.cwru.edu>,
0> Paul R. Jarc <URL:mailto:prj@po.cwru.edu> ("Paul") wrote:

Paul> Wes Hardaker <wes@hardakers.net> wrote:
>> You know, I searched for "backend" in the manual and didn't see
>> anything so just copied an example.  I now see I should have
>> searched for "back end".  Sigh.

Paul> The manual used to say "backend", but it was changed at RMS's
Paul> request.  Feel free to let him know how much you appreciate
Paul> it. :)

Perhaps it should have one instance of "backend" - in the index,
pointing at the entry for "back end".  That would redirect searchers
to the right place.

Re: building a smarter gnus

[Paul Jarc]

Toby Speight <streapadair@gmx.net> wrote:
> In article <m3it79zy2l.fsf@multivac.cwru.edu>,
> Paul R. Jarc <URL:mailto:prj@po.cwru.edu> ("Paul") wrote:
>> The manual used to say "backend", but it was changed
>
> Perhaps it should have one instance of "backend" - in the index,
> pointing at the entry for "back end".  That would redirect searchers
> to the right place.

I don't think it would have helped in this case.  The index entry for
"back end" is just a short description of what a backend is, not the
backend interface documentation.  Is there any way to make something
findable by a *search*, rather than an index lookup, without adding
visible text?


paul

Re: building a smarter gnus

[Lars Magne Ingebrigtsen]

Wes Hardaker <wes@hardakers.net> writes:

> From what I've gathered (I've started implementing a nnsourceforge
> component, which is functionally working but slow), article lists
> passed to nnXX-request-articles are always a fully expanded list
> (which is causes memory and speed problems). 

As previously noted, that would be nice, and was something I thought
should be in the back end interface design.

However, it's never really been followed up, and just putting ranges
in the back end interface wouldn't really help all that much.  For
instance, when entering a group, Gnus uses a list of article numbers
in the summary buffer for doing various things, so we already have
the list, and it's probably the same list that's passed to the back
ends.

And there would probably have to be written range iterators --

(dorange (article ranges)
  ...)

And a memq on a list of integers is much, much faster than
`gnus-member-of-range', I think.  So I think it's entirely possible
that any gains one get from not having to uncompress the ranges would
be lost by having the other operations be slower.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen