From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/62159 Path: news.gmane.org!not-for-mail From: Simon Josefsson Newsgroups: gmane.emacs.gnus.general Subject: Re: [PATCH] nnimap performance improvement for large or old groups Date: Sun, 05 Mar 2006 23:10:42 +0100 Message-ID: References: <87veuwxq70.fsf@rimspace.net> <87zmk7xen8.fsf@rimspace.net> NNTP-Posting-Host: main.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: sea.gmane.org 1141596862 22294 80.91.229.2 (5 Mar 2006 22:14:22 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Sun, 5 Mar 2006 22:14:22 +0000 (UTC) Cc: ding@gnus.org Original-X-From: ding-owner+m10687@lists.math.uh.edu Sun Mar 05 23:14:18 2006 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from malifon.math.uh.edu ([129.7.128.13]) by ciao.gmane.org with esmtp (Exim 4.43) id 1FG1UF-00033W-JT for ding-account@gmane.org; Sun, 05 Mar 2006 23:14:00 +0100 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu ident=lists) by malifon.math.uh.edu with smtp (Exim 3.20 #1) id 1FG1UB-0007l5-00; Sun, 05 Mar 2006 16:13:55 -0600 Original-Received: from nas01.math.uh.edu ([129.7.128.39]) by malifon.math.uh.edu with esmtp (Exim 3.20 #1) id 1FG1RJ-0007kw-00 for ding@lists.math.uh.edu; Sun, 05 Mar 2006 16:10:57 -0600 Original-Received: from quimby.gnus.org ([80.91.224.244]) by nas01.math.uh.edu with esmtp (Exim 4.52) id 1FG1RH-0006yE-1V for ding@lists.math.uh.edu; Sun, 05 Mar 2006 16:10:56 -0600 Original-Received: from 178.230.13.217.in-addr.dgcsystems.net ([217.13.230.178] helo=yxa.extundo.com) by quimby.gnus.org with esmtp (Exim 3.35 #1 (Debian)) id 1FG1RF-0005vt-00 for ; Sun, 05 Mar 2006 23:10:53 +0100 Original-Received: from latte.josefsson.org (jas@yxa.extundo.com [217.13.230.178]) (authenticated bits=0) by yxa.extundo.com (8.13.4/8.13.4/Debian-3) with ESMTP id k25MAlOa026707 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 5 Mar 2006 23:10:48 +0100 Original-To: Daniel Pittman OpenPGP: id=B565716F; url=http://josefsson.org/key.txt X-Hashcash: 1:21:060305:daniel@rimspace.net::96RfOn87i9yU56LB:1xCK X-Hashcash: 1:21:060305:ding@gnus.org::4IRidXvtXPTC5IU4:6eRS In-Reply-To: <87zmk7xen8.fsf@rimspace.net> (Daniel Pittman's message of "Sat, 04 Mar 2006 11:00:27 +1100") User-Agent: Gnus/5.110004 (No Gnus v0.4) Emacs/22.0.50 (gnu/linux) X-Spam-Status: No, score=-2.4 required=5.0 tests=AWL,BAYES_00, FORGED_RCVD_HELO autolearn=ham version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on yxa-iv X-Virus-Scanned: ClamAV version 0.84, clamav-milter version 0.84e on yxa.extundo.com X-Virus-Status: Clean X-Spam-Score: -2.5 (--) Precedence: bulk Original-Sender: ding-owner@lists.math.uh.edu Xref: news.gmane.org gmane.emacs.gnus.general:62159 Archived-At: Daniel Pittman writes: > Simon Josefsson writes: >> Daniel Pittman writes: >> >>> One of the nagging problems with using nnimap as my primary mail store >>> is that it takes absolutely forever to enter my INBOX, as well as >>> causing XEmacs to allocate around 150MB of memory. >>> >>> This is because I have been using the same INBOX for a long while now, >>> and the 'read' info range starts with '(1 . 695705)' >>> >>> The code in `nnimap-request-update-info-internal' called >>> `gnus-uncompress-range' on this, resulting in a list containing around >>> seven million numbers -- an awful lot of memory, and time spent working >>> through it. >>> >>> To address this I rewrote the code in that routine to work with >>> compressed ranges, rather than uncompressed, which gives me a huge >>> performance improvement (almost instant entry, vs ten to fifteen >>> seconds) and reduces the memory use significantly. >>> >>> >>> I don't think this is too performance-inefficient to include as is, and >>> it doesn't seem to adversely effect entry into small groups or anything >>> like that. >> >> Your patch seem correct. Installed, thanks! >> >> It should probably be tested for a while before being included in >> v5-10 though, the logic seem somewhat complicated and while I >> convinced myself that it does the same thing, I might be mistaken. > > I completely agree that this should have long and thorough testing, for > all that I did very carefully check the way that it worked, and made > sure to verify every step of the new computation manually. There are some subtle situations where a message is marked unseen on the server (by another client), which should be propagated back into gnus, and also when an article is removed. But it looks OK to me. So if nobody complains, let's move it to v5-10 in 3 months. > Oh, but it surely is *nice* to have that in there, though. :) Yup.