From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/62136 Path: news.gmane.org!not-for-mail From: Daniel Pittman Newsgroups: gmane.emacs.gnus.general Subject: [PATCH] nnimap performance improvement for large or old groups Date: Fri, 03 Mar 2006 12:38:43 +1100 Message-ID: <87veuwxq70.fsf@rimspace.net> NNTP-Posting-Host: main.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: sea.gmane.org 1141357000 16944 80.91.229.2 (3 Mar 2006 03:36:40 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Fri, 3 Mar 2006 03:36:40 +0000 (UTC) Cc: Simon Josefsson Original-X-From: ding-owner+m10664@lists.math.uh.edu Fri Mar 03 04:36:27 2006 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from malifon.math.uh.edu ([129.7.128.13]) by ciao.gmane.org with esmtp (Exim 4.43) id 1FF15b-0007KP-Vm for ding-account@gmane.org; Fri, 03 Mar 2006 04:36:24 +0100 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu ident=lists) by malifon.math.uh.edu with smtp (Exim 3.20 #1) id 1FF15R-0001m2-00; Thu, 02 Mar 2006 21:36:13 -0600 Original-Received: from nas01.math.uh.edu ([129.7.128.39]) by malifon.math.uh.edu with esmtp (Exim 3.20 #1) id 1FEzc0-0001jY-00 for ding@lists.math.uh.edu; Thu, 02 Mar 2006 20:01:44 -0600 Original-Received: from quimby.gnus.org ([80.91.224.244]) by nas01.math.uh.edu with esmtp (Exim 4.52) id 1FEzby-0005iG-Pd for ding@lists.math.uh.edu; Thu, 02 Mar 2006 20:01:43 -0600 Original-Received: from main.gmane.org ([80.91.229.2] helo=ciao.gmane.org) by quimby.gnus.org with esmtp (Exim 3.35 #1 (Debian)) id 1FEzbv-0004Nq-00 for ; Fri, 03 Mar 2006 03:01:39 +0100 Original-Received: from list by ciao.gmane.org with local (Exim 4.43) id 1FEzbt-0007lW-2x for ding@gnus.org; Fri, 03 Mar 2006 03:01:37 +0100 Original-Received: from 203-217-29-45.perm.iinet.net.au ([203.217.29.45]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 03 Mar 2006 03:01:37 +0100 Original-Received: from daniel by 203-217-29-45.perm.iinet.net.au with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 03 Mar 2006 03:01:37 +0100 X-Injected-Via-Gmane: http://gmane.org/ Original-To: ding@gnus.org Original-Lines: 70 Original-X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 203-217-29-45.perm.iinet.net.au User-Agent: Gnus/5.110004 (No Gnus v0.4) XEmacs/21.5-b25 (eggplant, linux) Cancel-Lock: sha1:Ys9RrYqtCT527e0+sv2zEzdFs3c= X-Spam-Score: -1.3 (-) Precedence: bulk Original-Sender: ding-owner@lists.math.uh.edu Xref: news.gmane.org gmane.emacs.gnus.general:62136 Archived-At: --=-=-= One of the nagging problems with using nnimap as my primary mail store is that it takes absolutely forever to enter my INBOX, as well as causing XEmacs to allocate around 150MB of memory. This is because I have been using the same INBOX for a long while now, and the 'read' info range starts with '(1 . 695705)' The code in `nnimap-request-update-info-internal' called `gnus-uncompress-range' on this, resulting in a list containing around seven million numbers -- an awful lot of memory, and time spent working through it. To address this I rewrote the code in that routine to work with compressed ranges, rather than uncompressed, which gives me a huge performance improvement (almost instant entry, vs ten to fifteen seconds) and reduces the memory use significantly. I don't think this is too performance-inefficient to include as is, and it doesn't seem to adversely effect entry into small groups or anything like that. Daniel --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=nnimap-performance.patch ? build.info ? build.sh ? nnimap-performance.patch ? contrib/auto-autoloads.el ? contrib/semantic.cache Index: lisp/nnimap.el =================================================================== RCS file: /usr/local/cvsroot/gnus/lisp/nnimap.el,v retrieving revision 7.30 diff -u -u -r7.30 nnimap.el --- lisp/nnimap.el 21 Feb 2006 07:14:23 -0000 7.30 +++ lisp/nnimap.el 3 Mar 2006 01:37:36 -0000 @@ -1183,18 +1183,12 @@ (let (seen unseen) ;; read info could contain articles marked unread by other ;; imap clients! we correct this - (setq seen (gnus-uncompress-range (gnus-info-read info)) - unseen (imap-search "UNSEEN UNDELETED") - seen (gnus-set-difference seen unseen) - ;; seen might lack articles marked as read by other - ;; imap clients! we correct this - seen (append seen (imap-search "SEEN")) - ;; remove dupes - seen (sort seen '<) - seen (gnus-compress-sequence seen t) - ;; we can't return '(1) since this isn't a "list of ranges", - ;; and we can't return '((1)) since g-list-of-unread-articles - ;; is buggy so we return '((1 . 1)). + (setq unseen (gnus-compress-sequence + (imap-search "UNSEEN UNDELETED")) + seen (gnus-range-difference (gnus-info-read info) unseen) + seen (gnus-range-add seen + (gnus-compress-sequence + (imap-search "SEEN"))) seen (if (and (integerp (car seen)) (null (cdr seen))) (list (cons (car seen) (car seen))) --=-=-=--