* overview -> .db files? @ 2002-10-31 16:07 Wes Hardaker 2002-10-31 18:21 ` Kai Großjohann 0 siblings, 1 reply; 6+ messages in thread From: Wes Hardaker @ 2002-10-31 16:07 UTC (permalink / raw) Here's a thought: XEmacs at least (I don't know about Emacs) offers the ability to make use of true database files for things. How difficult would it be to (optionally of course) change the .overview files into db files for speed purposes? Just thinking... -- "The trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it." -- Terry Pratchett ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: overview -> .db files? 2002-10-31 16:07 overview -> .db files? Wes Hardaker @ 2002-10-31 18:21 ` Kai Großjohann 2002-10-31 21:03 ` Wes Hardaker 0 siblings, 1 reply; 6+ messages in thread From: Kai Großjohann @ 2002-10-31 18:21 UTC (permalink / raw) Wes Hardaker <wes@hardakers.net> writes: > Here's a thought: XEmacs at least (I don't know about Emacs) offers > the ability to make use of true database files for things. How > difficult would it be to (optionally of course) change the .overview > files into db files for speed purposes? I think it might not be so easy. I think the backend interface works like this: Gnus tells the backend to retrieve some headers, the backend puts the headers into a buffer in NOV format, then Gnus reads that buffer. But I think there are already provisions for using headers instead of NOV format. So maybe that's a spot where you can hook in and return the headers to Gnus in a Lisp data structure without going via a buffer. It would be a major undertaking. Please work on it :-) Another possibility which might be easier is to keep the backend interface and to just use the db files as a faster way to extract the right headers. Currently, I think nnml just inserts the whole .overview file into a buffer and then uses search-forward to find things. (Or maybe it has a binary search algorithm? Then it would be difficult to make it a lot faster.) But if you set gnus-verbose and gnus-verbose-backends to 10 I think you will see that fetching headers is not the most time-consuming part when using Gnus. There are some cases where header fetching is slow, but now we have the agent cache which speeds things up a lot. At least for me, the slow part about using Gnus is the generation of the summary buffer. I wonder if that can be made faster somehow. Is there a way to create a buffer with a lot of text properties and stuff really quickly? What happens if we princ (buffer-string) to a file and then read that representation into memory and use a single insert call to create a buffer from it? kai -- ~/.signature is: umop ap!sdn (Frank Nobis) ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: overview -> .db files? 2002-10-31 18:21 ` Kai Großjohann @ 2002-10-31 21:03 ` Wes Hardaker 2002-11-01 17:42 ` Kai Großjohann 0 siblings, 1 reply; 6+ messages in thread From: Wes Hardaker @ 2002-10-31 21:03 UTC (permalink / raw) Cc: ding >>>>> On Thu, 31 Oct 2002 19:21:44 +0100, kai.grossjohann@uni-duisburg.de (Kai Großjohann) said: Kai> But I think there are already provisions for using headers Kai> instead of NOV format. So maybe that's a spot where you can hook Kai> in and return the headers to Gnus in a Lisp data structure Kai> without going via a buffer. Yeah, I realized it would likely affect both the place where Gnus and the backends were involved. Essentially functions would have to wrap access to NOV calls, and what the NOV calls did behind the scenes would either look stuff up in a list (as is now) or in a DB. Kai> It would be a major undertaking. Please work on it :-) Doubtful. I suspect it'd be a bit over my head with respect to gnus code. I'm sure I could do it, but the undertaking would be even larger for me ;-) Kai> Another possibility which might be easier is to keep the backend Kai> interface and to just use the db files as a faster way to extract Kai> the right headers. Yeah, I thought of that as well. Actually, just making the overview files be .el files would probably be a big win. Kai> At least for me, the slow part about using Gnus is the generation of Kai> the summary buffer. I wonder if that can be made faster somehow. Yeah. The patch I wrote a while ago to display the summary buffer as it's building it is a good thing, because it lets you view stuff to at least keep you occupied while it's building. (I've been bugging my boss about his half of the assignment rights for it as well as the nnsf code, but he's a busy guy) Kai> Is there a way to create a buffer with a lot of text properties and Kai> stuff really quickly? What happens if we princ (buffer-string) to a Kai> file and then read that representation into memory and use a single Kai> insert call to create a buffer from it? That's worth trying. I've often thought that on exiting a summary buffer, you have all the display information since you've already created it, so couldn't we save the current buffer to a file, reload it and reapply all the regions and insert new contents? That *may* be faster than rebuilding it from scratch every time. -- "The trouble with having an open mind, of course, is that people will insist on coming along and trying to put things in it." -- Terry Pratchett ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: overview -> .db files? 2002-10-31 21:03 ` Wes Hardaker @ 2002-11-01 17:42 ` Kai Großjohann 2002-11-06 17:44 ` Paul Jarc 0 siblings, 1 reply; 6+ messages in thread From: Kai Großjohann @ 2002-11-01 17:42 UTC (permalink / raw) Wes Hardaker <wes@hardakers.net> writes: >>>>>> On Thu, 31 Oct 2002 19:21:44 +0100, kai.grossjohann@uni-duisburg.de (Kai Großjohann) said: > > Kai> But I think there are already provisions for using headers > Kai> instead of NOV format. So maybe that's a spot where you can hook > Kai> in and return the headers to Gnus in a Lisp data structure > Kai> without going via a buffer. > > Yeah, I realized it would likely affect both the place where Gnus and > the backends were involved. Essentially functions would have to wrap > access to NOV calls, and what the NOV calls did behind the scenes > would either look stuff up in a list (as is now) or in a DB. Ah, you had in mind the *right* solution :-) > Kai> It would be a major undertaking. Please work on it :-) > > Doubtful. I suspect it'd be a bit over my head with respect to gnus > code. I'm sure I could do it, but the undertaking would be even > larger for me ;-) Not all of us can be expected to be gurus like Lars and ShengHuo :-) I admire those who grok the Gnus code. > Kai> Another possibility which might be easier is to keep the backend > Kai> interface and to just use the db files as a faster way to extract > Kai> the right headers. > > Yeah, I thought of that as well. Actually, just making the overview > files be .el files would probably be a big win. Indeed. Ah, that could be a first step. I don't know, however, if Gnus sometimes looks at parts of the overview file only. > Kai> At least for me, the slow part about using Gnus is the generation of > Kai> the summary buffer. I wonder if that can be made faster somehow. > > Yeah. The patch I wrote a while ago to display the summary buffer as > it's building it is a good thing, because it lets you view stuff to at > least keep you occupied while it's building. > > (I've been bugging my boss about his half of the assignment rights for > it as well as the nnsf code, but he's a busy guy) Oh yes, I think I wanted to install that. > Kai> Is there a way to create a buffer with a lot of text properties and > Kai> stuff really quickly? What happens if we princ (buffer-string) to a > Kai> file and then read that representation into memory and use a single > Kai> insert call to create a buffer from it? > > That's worth trying. > > I've often thought that on exiting a summary buffer, you have all the > display information since you've already created it, so couldn't we > save the current buffer to a file, reload it and reapply all the > regions and insert new contents? That *may* be faster than rebuilding > it from scratch every time. Yes, inserting the new information might be possible if you look at how `/ N' and `/ O' do it. kai -- ~/.signature is: umop ap!sdn (Frank Nobis) ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: overview -> .db files? 2002-11-01 17:42 ` Kai Großjohann @ 2002-11-06 17:44 ` Paul Jarc 2002-11-07 7:37 ` Kai Großjohann 0 siblings, 1 reply; 6+ messages in thread From: Paul Jarc @ 2002-11-06 17:44 UTC (permalink / raw) kai.grossjohann@uni-duisburg.de (Kai Großjohann) wrote: > Wes Hardaker <wes@hardakers.net> writes: >> Actually, just making the overview files be .el files would >> probably be a big win. nnmaildir does essentially that, with one file per article. $ cat mail/ding/.nnmaildir/nov/1036172673.12726.multivac.cwru.edu ; echo [1 1118 "<84fzulnq3s.fsf@crybaby.uni-duisburg.de>" ["Re: overview -> .db files?kai.grossjohann@uni-duisburg.de (Kai =?iso-8859-1?q?Gro=DFjohann?=) Fri, 01 Nov 2002 18:42:47 +0100" "<sdu1j2pp60.fsf@wanderer.hardakers.net> <844rb2o4ef.fsf@crybaby.uni-duisburg.de> <sdela6iamq.fsf@wanderer.hardakers.net> 4841 68" "" (15810 48513) nil]] > I don't know, however, if Gnus sometimes looks at parts of the > overview file only. Sure; nnchoke-retrieve-headers is called with just the range of articles that Gnus wants to show in the *Summary* buffer. >> I've often thought that on exiting a summary buffer, you have all the >> display information since you've already created it, so couldn't we >> save the current buffer to a file, reload it and reapply all the >> regions and insert new contents? That *may* be faster than rebuilding >> it from scratch every time. I think lots of things could invalidate the stored version, though. Changes in scoring, gnus-summary-line-format, article marks... paul ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: overview -> .db files? 2002-11-06 17:44 ` Paul Jarc @ 2002-11-07 7:37 ` Kai Großjohann 0 siblings, 0 replies; 6+ messages in thread From: Kai Großjohann @ 2002-11-07 7:37 UTC (permalink / raw) prj@po.cwru.edu (Paul Jarc) writes: > I think lots of things could invalidate the stored version, though. > Changes in scoring, gnus-summary-line-format, article marks... Yes. But most of the time, these things aren't changed, so reusing the old data would be a win. It seems the code that implements `/ N' already knows how to deal with sorting and, probably, scoring. It's also quite fast. I wonder if its ideas could be reused. I haven't looked at the code, so I have no idea what's involved. kai -- ~/.signature is: umop ap!sdn (Frank Nobis) ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2002-11-07 7:37 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2002-10-31 16:07 overview -> .db files? Wes Hardaker 2002-10-31 18:21 ` Kai Großjohann 2002-10-31 21:03 ` Wes Hardaker 2002-11-01 17:42 ` Kai Großjohann 2002-11-06 17:44 ` Paul Jarc 2002-11-07 7:37 ` Kai Großjohann
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).