overview -> .db files?

overview -> .db files?

[Wes Hardaker]

Here's a thought:  XEmacs at least (I don't know about Emacs) offers
the ability to make use of true database files for things.  How
difficult would it be to (optionally of course) change the .overview
files into db files for speed purposes?

Just thinking...

-- 
"The trouble with having an open mind, of course, is that people will
 insist on coming along and trying to put things in it."   -- Terry Pratchett

Re: overview -> .db files?

[Kai Großjohann]

Wes Hardaker <wes@hardakers.net> writes:

> Here's a thought:  XEmacs at least (I don't know about Emacs) offers
> the ability to make use of true database files for things.  How
> difficult would it be to (optionally of course) change the .overview
> files into db files for speed purposes?

I think it might not be so easy.  I think the backend interface works
like this: Gnus tells the backend to retrieve some headers, the
backend puts the headers into a buffer in NOV format, then Gnus reads
that buffer.

But I think there are already provisions for using headers instead of
NOV format.  So maybe that's a spot where you can hook in and return
the headers to Gnus in a Lisp data structure without going via a
buffer.

It would be a major undertaking.  Please work on it :-)

Another possibility which might be easier is to keep the backend
interface and to just use the db files as a faster way to extract the
right headers.  Currently, I think nnml just inserts the whole
.overview file into a buffer and then uses search-forward to find
things.  (Or maybe it has a binary search algorithm?  Then it would
be difficult to make it a lot faster.)

But if you set gnus-verbose and gnus-verbose-backends to 10 I think
you will see that fetching headers is not the most time-consuming
part when using Gnus.  There are some cases where header fetching is
slow, but now we have the agent cache which speeds things up a lot.

At least for me, the slow part about using Gnus is the generation of
the summary buffer.  I wonder if that can be made faster somehow.
Is there a way to create a buffer with a lot of text properties and
stuff really quickly?  What happens if we princ (buffer-string) to a
file and then read that representation into memory and use a single
insert call to create a buffer from it?

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: overview -> .db files?

[Wes Hardaker]

>>>>> On Thu, 31 Oct 2002 19:21:44 +0100, kai.grossjohann@uni-duisburg.de (Kai Großjohann) said:

Kai> But I think there are already provisions for using headers
Kai> instead of NOV format.  So maybe that's a spot where you can hook
Kai> in and return the headers to Gnus in a Lisp data structure
Kai> without going via a buffer.

Yeah, I realized it would likely affect both the place where Gnus and
the backends were involved.  Essentially functions would have to wrap
access to NOV calls, and what the NOV calls did behind the scenes
would either look stuff up in a list (as is now) or in a DB.

Kai> It would be a major undertaking.  Please work on it :-)

Doubtful.  I suspect it'd be a bit over my head with respect to gnus
code.  I'm sure I could do it, but the undertaking would be even
larger for me ;-)

Kai> Another possibility which might be easier is to keep the backend
Kai> interface and to just use the db files as a faster way to extract
Kai> the right headers.

Yeah, I thought of that as well.  Actually, just making the overview
files be .el files would probably be a big win.

Kai> At least for me, the slow part about using Gnus is the generation of
Kai> the summary buffer.  I wonder if that can be made faster somehow.

Yeah.  The patch I wrote a while ago to display the summary buffer as
it's building it is a good thing, because it lets you view stuff to at
least keep you occupied while it's building.

(I've been bugging my boss about his half of the assignment rights for
it as well as the nnsf code, but he's a busy guy)

Kai> Is there a way to create a buffer with a lot of text properties and
Kai> stuff really quickly?  What happens if we princ (buffer-string) to a
Kai> file and then read that representation into memory and use a single
Kai> insert call to create a buffer from it?

That's worth trying.

I've often thought that on exiting a summary buffer, you have all the
display information since you've already created it, so couldn't we
save the current buffer to a file, reload it and reapply all the
regions and insert new contents?  That *may* be faster than rebuilding
it from scratch every time.

-- 
"The trouble with having an open mind, of course, is that people will
 insist on coming along and trying to put things in it."   -- Terry Pratchett

Re: overview -> .db files?

[Kai Großjohann]

Wes Hardaker <wes@hardakers.net> writes:

>>>>>> On Thu, 31 Oct 2002 19:21:44 +0100, kai.grossjohann@uni-duisburg.de (Kai Großjohann) said:
>
> Kai> But I think there are already provisions for using headers
> Kai> instead of NOV format.  So maybe that's a spot where you can hook
> Kai> in and return the headers to Gnus in a Lisp data structure
> Kai> without going via a buffer.
>
> Yeah, I realized it would likely affect both the place where Gnus and
> the backends were involved.  Essentially functions would have to wrap
> access to NOV calls, and what the NOV calls did behind the scenes
> would either look stuff up in a list (as is now) or in a DB.

Ah, you had in mind the *right* solution :-)

> Kai> It would be a major undertaking.  Please work on it :-)
>
> Doubtful.  I suspect it'd be a bit over my head with respect to gnus
> code.  I'm sure I could do it, but the undertaking would be even
> larger for me ;-)

Not all of us can be expected to be gurus like Lars and ShengHuo :-)
I admire those who grok the Gnus code.

> Kai> Another possibility which might be easier is to keep the backend
> Kai> interface and to just use the db files as a faster way to extract
> Kai> the right headers.
>
> Yeah, I thought of that as well.  Actually, just making the overview
> files be .el files would probably be a big win.

Indeed.  Ah, that could be a first step.

I don't know, however, if Gnus sometimes looks at parts of the
overview file only.

> Kai> At least for me, the slow part about using Gnus is the generation of
> Kai> the summary buffer.  I wonder if that can be made faster somehow.
>
> Yeah.  The patch I wrote a while ago to display the summary buffer as
> it's building it is a good thing, because it lets you view stuff to at
> least keep you occupied while it's building.
>
> (I've been bugging my boss about his half of the assignment rights for
> it as well as the nnsf code, but he's a busy guy)

Oh yes, I think I wanted to install that.

> Kai> Is there a way to create a buffer with a lot of text properties and
> Kai> stuff really quickly?  What happens if we princ (buffer-string) to a
> Kai> file and then read that representation into memory and use a single
> Kai> insert call to create a buffer from it?
>
> That's worth trying.
>
> I've often thought that on exiting a summary buffer, you have all the
> display information since you've already created it, so couldn't we
> save the current buffer to a file, reload it and reapply all the
> regions and insert new contents?  That *may* be faster than rebuilding
> it from scratch every time.

Yes, inserting the new information might be possible if you look at
how `/ N' and `/ O' do it.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)

Re: overview -> .db files?

[Paul Jarc]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) wrote:
> Wes Hardaker <wes@hardakers.net> writes:
>> Actually, just making the overview files be .el files would
>> probably be a big win.

nnmaildir does essentially that, with one file per article.
$ cat mail/ding/.nnmaildir/nov/1036172673.12726.multivac.cwru.edu ; echo
[1 1118 "<84fzulnq3s.fsf@crybaby.uni-duisburg.de>" ["Re: overview -> .db files?kai.grossjohann@uni-duisburg.de (Kai =?iso-8859-1?q?Gro=DFjohann?=)      Fri, 01 Nov 2002 18:42:47 +0100" "<sdu1j2pp60.fsf@wanderer.hardakers.net> <844rb2o4ef.fsf@crybaby.uni-duisburg.de> <sdela6iamq.fsf@wanderer.hardakers.net>      4841   68" "" (15810 48513) nil]]

> I don't know, however, if Gnus sometimes looks at parts of the
> overview file only.

Sure; nnchoke-retrieve-headers is called with just the range of
articles that Gnus wants to show in the *Summary* buffer.

>> I've often thought that on exiting a summary buffer, you have all the
>> display information since you've already created it, so couldn't we
>> save the current buffer to a file, reload it and reapply all the
>> regions and insert new contents?  That *may* be faster than rebuilding
>> it from scratch every time.

I think lots of things could invalidate the stored version, though.
Changes in scoring, gnus-summary-line-format, article marks...


paul

Re: overview -> .db files?

[Kai Großjohann]

prj@po.cwru.edu (Paul Jarc) writes:

> I think lots of things could invalidate the stored version, though.
> Changes in scoring, gnus-summary-line-format, article marks...

Yes.  But most of the time, these things aren't changed, so reusing
the old data would be a win.

It seems the code that implements `/ N' already knows how to deal
with sorting and, probably, scoring.  It's also quite fast.  I wonder
if its ideas could be reused.  I haven't looked at the code, so I
have no idea what's involved.

kai
-- 
~/.signature is: umop ap!sdn    (Frank Nobis)