From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/66851 Path: news.gmane.org!not-for-mail From: Gareth McCaughan Newsgroups: gmane.emacs.gnus.general Subject: Re: Faster NOV braiding for large newsgroups with many cached articles Date: Sat, 19 Apr 2008 21:33:39 +0100 Organization: International Pedant Conspiracy Message-ID: <200804192133.39521.gareth.mccaughan@pobox.com> References: <200803302125.02240.gareth.mccaughan@pobox.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1208637317 21829 80.91.229.12 (19 Apr 2008 20:35:17 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sat, 19 Apr 2008 20:35:17 +0000 (UTC) Cc: ding@gnus.org To: Reiner Steib Original-X-From: ding-owner+M15333@lists.math.uh.edu Sat Apr 19 22:35:50 2008 connect(): Connection refused Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.50) id 1JnJmm-0001IX-Tz for ding-account@gmane.org; Sat, 19 Apr 2008 22:35:49 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1JnJlU-0007t3-1E; Sat, 19 Apr 2008 15:34:28 -0500 Original-Received: from mx1.math.uh.edu ([129.7.128.32]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1JnJlR-0007si-NZ for ding@lists.math.uh.edu; Sat, 19 Apr 2008 15:34:25 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx1.math.uh.edu with esmtp (Exim 4.67) (envelope-from ) id 1JnJlL-0005hn-4B for ding@lists.math.uh.edu; Sat, 19 Apr 2008 15:34:25 -0500 Original-Received: from smtp806.mail.ird.yahoo.com ([217.146.188.66]) by quimby.gnus.org with smtp (Exim 3.35 #1 (Debian)) id 1JnJlV-0007gE-00 for ; Sat, 19 Apr 2008 22:34:29 +0200 Original-Received: (qmail 61215 invoked from network); 19 Apr 2008 20:33:40 -0000 Original-Received: from unknown (HELO g.local) (gareth.mccaughan@btinternet.com@86.144.168.199 with plain) by smtp806.mail.ird.yahoo.com with SMTP; 19 Apr 2008 20:33:40 -0000 X-YMail-OSG: nHjMz4QVM1mjk.UhR0v8LNigfJcjHDMeBwboIUN1y8P9YAzdmtU4ancAEy5KElh5vzgpTUDQCg-- X-Yahoo-Newman-Property: ymail-3 Original-Received: from localhost.local ([127.0.0.1] helo=g) by g.local with esmtp (Exim 4.68 (FreeBSD)) (envelope-from ) id 1JnJki-000357-9Z; Sat, 19 Apr 2008 21:33:40 +0100 User-Agent: KMail/1.9.7 In-Reply-To: Content-Disposition: inline X-Spam-Score: -2.6 (--) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:66851 Archived-At: On Saturday 19 April 2008, Reiner Steib wrote: [me:] > > Specifically, gnus-cache-braid-nov takes several minutes to run, > > and much of this appears to be because all the insertions in the > > nntp-server-buffer are kinda slow. > > > > By building up the new buffer contents in a list of strings, > > assembling them into a single string, and then dumping that into > > the buffer where it belongs in a single operation, I can (on my > > machine, on one occasion -- I haven't tested this scientifically) > > speed up gnus-cache-braid-nov by a factor of about 20; 30 seconds > > instead of 10 minutes. ... > With which (X)Emacs and Gnus versions? Did you try other versions as > well? GNU Emacs (haven't tried XEmacs); several different versions of Gnus, namely the ones distributed with many recent releases of GNU Emacs. I'm afraid I don't have records of exactly which ones, but what I have at the moment is: (emacs-version) "GNU Emacs 22.1.1 (i386-pc-freebsd, GTK+ Version 2.10.14) of 2007-07-25 on g.local" (gnus-version) "Gnus v5.11" > It is easier for us if you don't post the modified function. Instead, > produce a diff (unified diff preferred: "-u") against the version you > use (preferably HEAD revision of the CVS trunk, else tell us the > version). [1] Looks like you've made the diff yourself in this case; thanks. (I posted the new code rather than the diff because I was basically replacing an entire function and it seemed friendlier to casual ding@-readers to have the code as readily readable as possible.) > Better use `gnus-message' here. Oops, yes. Didn't even notice its existence. > | + ;; reverse chunks and concatenate > | + (let ((n 0) (records new-records)) > | + (while records > | + (incf n (length (car records))) > | + (setq records (cdr records))) > | + (let ((new-content (make-string n ?.))) > | + (setq n 0) > | + (setq records (nreverse new-records)) > | + (setf new-records nil) ; help the GC a little > > Please explain why you use `setf' and why GC need help. I use "setf" because I write much more Common Lisp than elisp and occasionally forget myself. setq would be just as good, of course. I don't know whether the GC does in fact *need* help, but it can't do any harm to give it more opportunity to collect the garbage early if it wants. > On Fri, Apr 18 2008, Gareth McCaughan wrote on bugs@gnus.org: > [...] > > I posted a version of gnus-cache-braid-nov that works that way > > to ding@gnus.org. (No replies; fair enough.) > > Sorry, I didn't have time to look at your code. It would be better to > remind us by following-up to the original message instead of starting > a new thread on a different list. It occurred to me that perhaps I'd posted it to the wrong place originally and bugs@ might have been preferable. I'm sorry if that was less than maximally convenient. > > It might be better to look at the size of the group and of the cache > > and choose heuristically between the two implementations, so as not > > to pay the memory cost for large groups with few cached articles > > (where I think the speed should be OK with the old implementation, > > though I haven't measured it). > > Sounds useful. OK; I'll have a play and put together something that avoids gratuitously large time and memory costs in as wide a range of situations as I can. > > If there's any interest in improving this and my code is useful, > > I am happy to sign whatever papers are necessary. > > I'll send you the form off-list. Thanks. -- g