From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/24197 Path: main.gmane.org!not-for-mail From: jonkv@ida.liu.se Newsgroups: gmane.emacs.gnus.general Subject: Re: Holes in article sequence Date: 12 Jul 1999 23:16:25 +0200 Sender: owner-ding@hpc.uh.edu Message-ID: References: <874sjhjitm.fsf@pc-hrvoje.srce.hr> NNTP-Posting-Host: coloc-standby.netfonds.no Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: main.gmane.org 1035161804 7382 80.91.224.250 (21 Oct 2002 00:56:44 GMT) X-Complaints-To: usenet@main.gmane.org NNTP-Posting-Date: Mon, 21 Oct 2002 00:56:44 +0000 (UTC) Cc: jonkv@ida.liu.se Return-Path: Original-Received: from farabi.math.uh.edu (farabi.math.uh.edu [129.7.128.57]) by sclp3.sclp.com (8.8.5/8.8.5) with ESMTP id RAA12906 for ; Mon, 12 Jul 1999 17:18:32 -0400 (EDT) Original-Received: from sina.hpc.uh.edu (lists@Sina.HPC.UH.EDU [129.7.3.5]) by farabi.math.uh.edu (8.9.1/8.9.1) with ESMTP id QAB18249; Mon, 12 Jul 1999 16:18:10 -0500 (CDT) Original-Received: by sina.hpc.uh.edu (TLB v0.09a (1.20 tibbs 1996/10/09 22:03:07)); Mon, 12 Jul 1999 16:19:02 -0500 (CDT) Original-Received: from sclp3.sclp.com (root@sclp3.sclp.com [204.252.123.139]) by sina.hpc.uh.edu (8.9.3/8.9.3) with ESMTP id QAA23526 for ; Mon, 12 Jul 1999 16:18:51 -0500 (CDT) Original-Received: from portofix.ida.liu.se (portofix.ida.liu.se [130.236.177.25]) by sclp3.sclp.com (8.8.5/8.8.5) with ESMTP id RAA12879 for ; Mon, 12 Jul 1999 17:17:47 -0400 (EDT) Original-Received: from localhost (ts2-2.ida.liu.se [130.236.178.202]) by portofix.ida.liu.se (8.8.8/8.8.8) with ESMTP id XAA28088; Mon, 12 Jul 1999 23:17:43 +0200 (MET DST) Original-Received: (from jk@localhost) by localhost (8.8.7/8.8.7) id XAA07741; Mon, 12 Jul 1999 23:17:26 +0200 Original-To: ding@gnus.org In-Reply-To: Lars Magne Ingebrigtsen's message of "09 Jul 1999 20:26:21 +0200" Original-Lines: 40 User-Agent: Gnus/5.070095 (Pterodactyl Gnus v0.95) XEmacs/21.1 (20 Minutes to Nikko) Precedence: list X-Majordomo: 1.94.jlt7 Xref: main.gmane.org gmane.emacs.gnus.general:24197 X-Report-Spam: http://spam.gmane.org/gmane.emacs.gnus.general:24197 --=-=-= (Apparently, I should write more followups to mailing list articles. Maybe then I'd finally learn to use f/F instead of r/R... I did NOT intend to send the first reply privately.) Lars Magne Ingebrigtsen writes: > Aha! nnml already does it the other way around -- it looks at the > actual articles in the group, and then works on the intersection > between the articles it's asked to expire and the ones it knows > exists. nnfolder, on the other hand, just iterates over the entire > list of expirable articles, which results in a call to > `nnfolder-goto-article' for each article in the list. That must be > slooow... I looked at this about a year ago, since I was using nnfolder for some large groups (for example, for the linux-kernel mailing list) and it was taking forever to expire those groups. I came to the same conclusion you just did. AFAICT, nnfolder-request-expire-articles is often called with a huge list of articles, because of holes or whatever. Then, it iterates through that entire list, and for each article, it searches the entire buffer until it finds the article. But most of the article numbers don't exist, and for them the *entire* buffer is searched each time... I rewrote that part so that it first finds all article numbers (which shouldn't take that much more time than searching for a single non-existing article, since in each case the entire buffer has to be scanned), then intersects that with the articles it should expire. Back then, I had no idea that nnml did it that way. Anyway, I wrote that code a year ago and promptly forgot all about sending in a patch or anything. But here it is, ported to pgnus 0.95, in case anyone is interested. It seems to work on my system, but I can't guarantee that it will work on anyone else's or that I haven't made stupid assumptions that won't hold all the time -- I'm not exactly familiar with the internal workings of Gnus. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=nnfolder.patch Content-Description: Faster expiry in nnfolder --- nnfolder-original.el Mon Jul 12 17:06:00 1999 +++ nnfolder.el Mon Jul 12 17:33:28 1999 @@ -295,39 +295,64 @@ (let ((nnmail-file-coding-system nnfolder-file-coding-system)) (nnmail-find-file nnfolder-newsgroups-file)))) +;; Return a list consisting of all article numbers existing in the +;; current folder. + +(defun nnfolder-existing-articles () + (save-excursion + (when nnfolder-current-buffer + (set-buffer nnfolder-current-buffer) + (goto-char (point-min)) + (let ((marker (concat "\n" nnfolder-article-marker)) + (number "[0-9]+") + numbers) + + (while (and (search-forward marker nil t) + (re-search-forward number nil t)) + (let ((newnum (string-to-number (match-string 0)))) + (if (nnmail-within-headers-p) + (push newnum numbers)))) + numbers)))) + (deffoo nnfolder-request-expire-articles (articles newsgroup &optional server force) (nnfolder-possibly-change-group newsgroup server) (let* ((is-old t) - rest) + ;; The articles we have deleted so far. + (deleted-articles nil) + ;; The articles that really exist and will be expired if they are old enough. + (maybe-expirable (gnus-intersection articles (nnfolder-existing-articles)))) (nnmail-activate 'nnfolder) (save-excursion (set-buffer nnfolder-current-buffer) - (while (and articles is-old) + ;; Since messages are sorted in arrival order and expired in the + ;; same order, we can stop as soon as we find a message that is + ;; too old. + (while (and maybe-expirable is-old) (goto-char (point-min)) - (when (and (nnfolder-goto-article (car articles)) + (when (and (nnfolder-goto-article (car maybe-expirable)) (search-forward (concat "\n" nnfolder-article-marker) nil t)) (forward-sexp) - (if (setq is-old + (when (setq is-old (nnmail-expired-article-p newsgroup (buffer-substring (point) (progn (end-of-line) (point))) force nnfolder-inhibit-expiry)) - (progn (nnheader-message 5 "Deleting article %d..." - (car articles) newsgroup) - (nnfolder-delete-mail)) - (push (car articles) rest))) - (setq articles (cdr articles))) + (car maybe-expirable) newsgroup) + (nnfolder-delete-mail) + ;; Must remember which articles were actually deleted + (push (car maybe-expirable) deleted-articles))) + (setq maybe-expirable (cdr maybe-expirable))) (unless nnfolder-inhibit-expiry (nnheader-message 5 "Deleting articles...done")) (nnfolder-save-buffer) (nnfolder-adjust-min-active newsgroup) (nnfolder-save-active nnfolder-group-alist nnfolder-active-file) - (nconc rest articles)))) + (gnus-sorted-complement articles (nreverse deleted-articles))))) (deffoo nnfolder-request-move-article (article group server accept-form &optional last) --=-=-=--