[Patch] Faster nnfolder-retrieve-headers

[Patch] Faster nnfolder-retrieve-headers

[Jonas Kvarnström]

[-- Attachment #1: Type: text/plain, Size: 1474 bytes --]

Here's a small patch (against 5.8.7) that makes
nnfolder-retrieve-headers considerably faster when retrieving headers
in folders with a large number of "gaps" (non-existing article
numbers) in the range being retrieved -- for example, when you've
moved a lot of articles to other folders, or when you enter a group
with C-u RET.

The problem with the old code was that every single time it attempted
to find a non-existing article it had to search the entire folder
file, possibly a number of megabytes.  (Since the search started from
where the previous article was found, and since articles are generally
placed in numerical order, there was generally no performance problem
when you tried to find an *existing* article.)

The new code uses (nnfolder-existing-articles) to scan the entire
buffer a single time, and then quickly skips any article number that
doesn't exist.

Possible drawbacks: Marginally slower if all articles actually exist
(but should be faster as soon as two or three articles are missing).
Temporarily uses a little bit more memory for the list of article
numbers.

Benchmarks: The relevant part of nnfolder-retrieve-headers used to
take 390-400 seconds when I entered my mail.ding folder with C-u RET
(3.5 MB, about 6000 articles existing, greatest article number around
22000).  With the patch, it takes around 3 seconds.  This doesn't
include scoring or actually generating the summary buffer, so the
perceived speedup isn't quite that dramatic.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnfolder.el.diff --]
[-- Type: text/x-patch, Size: 1310 bytes --]

--- nnfolder.el	Sat Jul  1 13:10:51 2000
+++ nnfolder-new.el	Sat Dec 30 13:42:16 2000
@@ -118,18 +118,20 @@
 	(goto-char (point-min))
 	(if (stringp (car articles))
 	    'headers
-	  (while (setq article (pop articles))
-	    (set-buffer nnfolder-current-buffer)
-	    (when (nnfolder-goto-article article)
-	      (setq start (point))
-	      (setq stop (if (search-forward "\n\n" nil t)
-			     (1- (point))
-			   (point-max)))
-	      (set-buffer nntp-server-buffer)
-	      (insert (format "221 %d Article retrieved.\n" article))
-	      (insert-buffer-substring nnfolder-current-buffer start stop)
-	      (goto-char (point-max))
-	      (insert ".\n")))
+	  (let ((existing-articles (nnfolder-existing-articles)))
+	    (while (setq article (pop articles))
+	      (when (member article existing-articles)
+		(set-buffer nnfolder-current-buffer)
+		(when (nnfolder-goto-article article)
+		  (setq start (point))
+		  (setq stop (if (search-forward "\n\n" nil t)
+				 (1- (point))
+			       (point-max)))
+		  (set-buffer nntp-server-buffer)
+		  (insert (format "221 %d Article retrieved.\n" article))
+		  (insert-buffer-substring nnfolder-current-buffer start stop)
+		  (goto-char (point-max))
+		  (insert ".\n")))))
 
 	  (set-buffer nntp-server-buffer)
 	  (nnheader-fold-continuation-lines)

Re: [Patch] Faster nnfolder-retrieve-headers

[ShengHuo ZHU]

"Jonas Kvarnström" <jonkv@ida.liu.se> writes:

[...]

> The new code uses (nnfolder-existing-articles) to scan the entire
> buffer a single time, and then quickly skips any article number that
> doesn't exist.

I have added this idea into the Gnus, but use
gnus-sorted-intersection.  Actually, nnfolder (Oort Gnus v0.01) uses
NOV, which makes nnfolder-retrieve-headers even faster.

ShengHuo