From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/62139 Path: news.gmane.org!not-for-mail From: Simon Josefsson Newsgroups: gmane.emacs.gnus.general Subject: Re: [PATCH] nnimap performance improvement for large or old groups Date: Fri, 03 Mar 2006 10:34:39 +0100 Message-ID: References: <87veuwxq70.fsf@rimspace.net> NNTP-Posting-Host: main.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: sea.gmane.org 1141378508 11092 80.91.229.2 (3 Mar 2006 09:35:08 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Fri, 3 Mar 2006 09:35:08 +0000 (UTC) Cc: ding@gnus.org Original-X-From: ding-owner+m10667@lists.math.uh.edu Fri Mar 03 10:35:06 2006 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from malifon.math.uh.edu ([129.7.128.13]) by ciao.gmane.org with esmtp (Exim 4.43) id 1FF6gj-0007Kl-0M for ding-account@gmane.org; Fri, 03 Mar 2006 10:35:05 +0100 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu ident=lists) by malifon.math.uh.edu with smtp (Exim 3.20 #1) id 1FF6gd-0002Ah-00; Fri, 03 Mar 2006 03:34:59 -0600 Original-Received: from nas01.math.uh.edu ([129.7.128.39]) by malifon.math.uh.edu with esmtp (Exim 3.20 #1) id 1FF6gT-0002Ac-00 for ding@lists.math.uh.edu; Fri, 03 Mar 2006 03:34:49 -0600 Original-Received: from quimby.gnus.org ([80.91.224.244]) by nas01.math.uh.edu with esmtp (Exim 4.52) id 1FF6gS-00082r-29 for ding@lists.math.uh.edu; Fri, 03 Mar 2006 03:34:49 -0600 Original-Received: from 178.230.13.217.in-addr.dgcsystems.net ([217.13.230.178] helo=yxa.extundo.com) by quimby.gnus.org with esmtp (Exim 3.35 #1 (Debian)) id 1FF6gQ-0001uo-00 for ; Fri, 03 Mar 2006 10:34:46 +0100 Original-Received: from latte.josefsson.org (jas@yxa.extundo.com [217.13.230.178]) (authenticated bits=0) by yxa.extundo.com (8.13.4/8.13.4/Debian-3) with ESMTP id k239YgL5025270 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 3 Mar 2006 10:34:44 +0100 Original-To: Daniel Pittman OpenPGP: id=B565716F; url=http://josefsson.org/key.txt X-Hashcash: 1:21:060303:daniel@rimspace.net::6xtQ8AICVt6GFAjq:3qrB X-Hashcash: 1:21:060303:ding@gnus.org::/vvKav4etxEdnLBu:6nyd In-Reply-To: <87veuwxq70.fsf@rimspace.net> (Daniel Pittman's message of "Fri, 03 Mar 2006 12:38:43 +1100") User-Agent: Gnus/5.110004 (No Gnus v0.4) Emacs/22.0.50 (gnu/linux) X-Spam-Status: No, score=-1.8 required=5.0 tests=AWL,BAYES_00, FORGED_RCVD_HELO,INFO_TLD autolearn=no version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on yxa-iv X-Virus-Scanned: ClamAV version 0.84, clamav-milter version 0.84e on yxa.extundo.com X-Virus-Status: Clean X-Spam-Score: -1.9 (-) Precedence: bulk Original-Sender: ding-owner@lists.math.uh.edu Xref: news.gmane.org gmane.emacs.gnus.general:62139 Archived-At: Daniel Pittman writes: > One of the nagging problems with using nnimap as my primary mail store > is that it takes absolutely forever to enter my INBOX, as well as > causing XEmacs to allocate around 150MB of memory. > > This is because I have been using the same INBOX for a long while now, > and the 'read' info range starts with '(1 . 695705)' > > The code in `nnimap-request-update-info-internal' called > `gnus-uncompress-range' on this, resulting in a list containing around > seven million numbers -- an awful lot of memory, and time spent working > through it. > > To address this I rewrote the code in that routine to work with > compressed ranges, rather than uncompressed, which gives me a huge > performance improvement (almost instant entry, vs ten to fifteen > seconds) and reduces the memory use significantly. > > > I don't think this is too performance-inefficient to include as is, and > it doesn't seem to adversely effect entry into small groups or anything > like that. Your patch seem correct. Installed, thanks! It should probably be tested for a while before being included in v5-10 though, the logic seem somewhat complicated and while I convinced myself that it does the same thing, I might be mistaken. > Daniel > > ? build.info > ? build.sh > ? nnimap-performance.patch > ? contrib/auto-autoloads.el > ? contrib/semantic.cache > Index: lisp/nnimap.el > =================================================================== > RCS file: /usr/local/cvsroot/gnus/lisp/nnimap.el,v > retrieving revision 7.30 > diff -u -u -r7.30 nnimap.el > --- lisp/nnimap.el 21 Feb 2006 07:14:23 -0000 7.30 > +++ lisp/nnimap.el 3 Mar 2006 01:37:36 -0000 > @@ -1183,18 +1183,12 @@ > (let (seen unseen) > ;; read info could contain articles marked unread by other > ;; imap clients! we correct this > - (setq seen (gnus-uncompress-range (gnus-info-read info)) > - unseen (imap-search "UNSEEN UNDELETED") > - seen (gnus-set-difference seen unseen) > - ;; seen might lack articles marked as read by other > - ;; imap clients! we correct this > - seen (append seen (imap-search "SEEN")) > - ;; remove dupes > - seen (sort seen '<) > - seen (gnus-compress-sequence seen t) > - ;; we can't return '(1) since this isn't a "list of ranges", > - ;; and we can't return '((1)) since g-list-of-unread-articles > - ;; is buggy so we return '((1 . 1)). > + (setq unseen (gnus-compress-sequence > + (imap-search "UNSEEN UNDELETED")) > + seen (gnus-range-difference (gnus-info-read info) unseen) > + seen (gnus-range-add seen > + (gnus-compress-sequence > + (imap-search "SEEN"))) > seen (if (and (integerp (car seen)) > (null (cdr seen))) > (list (cons (car seen) (car seen)))