Re: Holes in article sequence

[jonkv@ida.liu.se]

[-- Attachment #1: Type: text/plain, Size: 1938 bytes --]

(Apparently, I should write more followups to mailing list articles.
Maybe then I'd finally learn to use f/F instead of r/R...  I did NOT
intend to send the first reply privately.)

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Aha!  nnml already does it the other way around -- it looks at the
> actual articles in the group, and then works on the intersection
> between the articles it's asked to expire and the ones it knows
> exists.  nnfolder, on the other hand, just iterates over the entire
> list of expirable articles, which results in a call to
> `nnfolder-goto-article' for each article in the list.  That must be
> slooow...

I looked at this about a year ago, since I was using nnfolder for some 
large groups (for example, for the linux-kernel mailing list) and it
was taking forever to expire those groups.  I came to the same
conclusion you just did.  

AFAICT, nnfolder-request-expire-articles is often called with a huge
list of articles, because of holes or whatever.  Then, it iterates
through that entire list, and for each article, it searches the entire
buffer until it finds the article.  But most of the article numbers
don't exist, and for them the *entire* buffer is searched each time...

I rewrote that part so that it first finds all article numbers (which
shouldn't take that much more time than searching for a single
non-existing article, since in each case the entire buffer has to be
scanned), then intersects that with the articles it should expire.
Back then, I had no idea that nnml did it that way.

Anyway, I wrote that code a year ago and promptly forgot all about
sending in a patch or anything.  But here it is, ported to pgnus 0.95,
in case anyone is interested.  It seems to work on my system, but I
can't guarantee that it will work on anyone else's or that I haven't
made stupid assumptions that won't hold all the time -- I'm not
exactly familiar with the internal workings of Gnus.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Faster expiry in nnfolder --]
[-- Type: text/x-patch, Size: 2878 bytes --]

--- nnfolder-original.el	Mon Jul 12 17:06:00 1999
+++ nnfolder.el	Mon Jul 12 17:33:28 1999
@@ -295,39 +295,64 @@
     (let ((nnmail-file-coding-system nnfolder-file-coding-system))
       (nnmail-find-file nnfolder-newsgroups-file))))
 
+;; Return a list consisting of all article numbers existing in the
+;; current folder.
+
+(defun nnfolder-existing-articles ()
+  (save-excursion
+    (when nnfolder-current-buffer
+      (set-buffer nnfolder-current-buffer)
+      (goto-char (point-min))
+      (let ((marker (concat "\n" nnfolder-article-marker))
+	    (number "[0-9]+")
+	    numbers)
+      
+	(while (and (search-forward marker nil t)
+		    (re-search-forward number nil t))
+	  (let ((newnum (string-to-number (match-string 0))))
+	    (if (nnmail-within-headers-p)
+		(push newnum numbers))))
+	numbers))))
+
 (deffoo nnfolder-request-expire-articles
   (articles newsgroup &optional server force)
   (nnfolder-possibly-change-group newsgroup server)
   (let* ((is-old t)
-	 rest)
+	 ;; The articles we have deleted so far.
+	 (deleted-articles nil)
+	 ;; The articles that really exist and will be expired if they are old enough.
+	 (maybe-expirable (gnus-intersection articles (nnfolder-existing-articles))))
     (nnmail-activate 'nnfolder)
 
     (save-excursion
       (set-buffer nnfolder-current-buffer)
-      (while (and articles is-old)
+      ;; Since messages are sorted in arrival order and expired in the
+      ;; same order, we can stop as soon as we find a message that is
+      ;; too old.
+      (while (and maybe-expirable is-old)
 	(goto-char (point-min))
-	(when (and (nnfolder-goto-article (car articles))
+	(when (and (nnfolder-goto-article (car maybe-expirable))
 		   (search-forward (concat "\n" nnfolder-article-marker)
 				   nil t))
 	  (forward-sexp)
-	  (if (setq is-old
+	  (when (setq is-old
 		    (nnmail-expired-article-p
 		     newsgroup
 		     (buffer-substring
 		      (point) (progn (end-of-line) (point)))
 		     force nnfolder-inhibit-expiry))
-	      (progn
 		(nnheader-message 5 "Deleting article %d..."
-				  (car articles) newsgroup)
-		(nnfolder-delete-mail))
-	    (push (car articles) rest)))
-	(setq articles (cdr articles)))
+			      (car maybe-expirable) newsgroup)
+	    (nnfolder-delete-mail)
+	    ;; Must remember which articles were actually deleted
+	    (push (car maybe-expirable) deleted-articles)))
+	(setq maybe-expirable (cdr maybe-expirable)))
       (unless nnfolder-inhibit-expiry
 	(nnheader-message 5 "Deleting articles...done"))
       (nnfolder-save-buffer)
       (nnfolder-adjust-min-active newsgroup)
       (nnfolder-save-active nnfolder-group-alist nnfolder-active-file)
-      (nconc rest articles))))
+      (gnus-sorted-complement articles (nreverse deleted-articles)))))
 
 (deffoo nnfolder-request-move-article (article group server
 					       accept-form &optional last)