From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/71974 Path: news.gmane.org!not-for-mail From: Simon Josefsson Newsgroups: gmane.emacs.gnus.general Subject: Re: nnimap issues Date: Mon, 27 Sep 2010 16:39:21 +0200 Message-ID: <87y6anw8rq.fsf@mocca.josefsson.org> References: <878w2olqoa.fsf@mocca.josefsson.org> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: dough.gmane.org 1285598413 30971 80.91.229.12 (27 Sep 2010 14:40:13 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Mon, 27 Sep 2010 14:40:13 +0000 (UTC) To: ding@gnus.org Original-X-From: ding-owner+M20347@lists.math.uh.edu Mon Sep 27 16:40:12 2010 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1P0EsF-0001iN-6m for ding-account@gmane.org; Mon, 27 Sep 2010 16:40:11 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1P0Es0-0000Zs-No; Mon, 27 Sep 2010 09:39:56 -0500 Original-Received: from mx1.math.uh.edu ([129.7.128.32]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1P0Ery-0000Zb-RL for ding@lists.math.uh.edu; Mon, 27 Sep 2010 09:39:54 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx1.math.uh.edu with esmtp (Exim 4.72) (envelope-from ) id 1P0Eru-0005mZ-O4 for ding@lists.math.uh.edu; Mon, 27 Sep 2010 09:39:54 -0500 Original-Received: from yxa-v.extundo.com ([83.241.177.39]) by quimby.gnus.org with esmtp (Exim 3.36 #1 (Debian)) id 1P0Ers-0003M0-00 for ; Mon, 27 Sep 2010 16:39:49 +0200 Original-Received: from mocca (c80-216-27-64.bredband.comhem.se [80.216.27.64]) (authenticated bits=0) by yxa-v.extundo.com (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id o8REdPIF030032 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Mon, 27 Sep 2010 16:39:35 +0200 OpenPGP: id=B565716F; url=http://josefsson.org/key.txt Mail-Copies-To: nobody X-Hashcash: 1:22:100927:ding@gnus.org::vT4niC8joJqzEUNB:8f/8 In-Reply-To: (Lars Magne Ingebrigtsen's message of "Sun, 26 Sep 2010 18:17:16 +0200") User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/23.2 (gnu/linux) X-Spam-Status: No, score=-0.8 required=5.0 tests=AWL,BAYES_00, DATE_IN_FUTURE_96_XX,RDNS_DYNAMIC,SPF_FAIL autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on yxa-v.extundo.com X-Virus-Scanned: clamav-milter 0.96.3 at yxa-v X-Virus-Status: Clean X-Spam-Score: -0.9 (/) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:71974 Archived-At: Lars Magne Ingebrigtsen writes: > Simon Josefsson writes: > >> The 'UID FETCH 1:*' command is an O(n) operation (both in terms of >> computation on client and server side but also on bandwidth) on the size >> of the mailbox, and for large mailboxes this becomes really painful. It >> can results in MB's of data being pushed every time I press 'g'. > > It only requests the last 100 flags... Then I wonder if it is the EXAMINE or the UID FETCH that is taken time, compared to the old approach of using STATUS. >> The old nnimap used the STATUS command to look for new mail instead. >> There are controversies over whether this is a good idea or not, but >> older IMAP RFCs support it and as far as I know, it is the fastest and >> most reliable way to check for new mail. Alternatively, doing a 'UID >> FETCH 1,*' should be an O(1) operation. >> >> Generally, why is it downloading all the flags on all messages over and >> over again? It should have all that cached anyway, except for possibly >> new messages. > > Before starting working on this, I was trolling the gnus-bug group, and > it seemed like the most constant complaint was "why is nnimap showing 30 > unread messages while there's really only 2"? As I recall, this isn't specific to nnimap, and happens for all backends if there are holes in the article range. > I think the only way to actually tell how many new messages have > actually arrived when UIDNEXT jumps from X to X+N is to do a UID FETCH > X:*, isn't it? (Please correct me if I'm mistaken -- I'm not an IMAP > expert). Yes, I suspect so. But does it have to EXAMINE+FETCH all groups unconditionally? I think it does that, right? It could use the old approach with STATUS to quickly see what mailboxes have changed at all, and then for those (hopefully few) mailboxes that have changed, do the EXAMINE+FETCH dance. Ah. I think I got it: the old code sent the STATUS in parallel, like this: 1554 STATUS "INBOX.sent-2010" (UIDVALIDITY UIDNEXT UNSEEN) 1555 STATUS "INBOX.sent-2009" (UIDVALIDITY UIDNEXT UNSEEN) 1556 STATUS "INBOX.fossgrupp" (UIDVALIDITY UIDNEXT UNSEEN) 1557 STATUS "INBOX.apel-en" (UIDVALIDITY UIDNEXT UNSEEN) and then waits for all the replies. The new code iterates through all groups and does EXAMINE+FETCH on each of them. What is biting now may be all the round trips. The roundtrips with the new code is O(2*N) in number of mailboxes, whereas the old was O(1+2*M) where M is the number of changed mailboxes (which is typically << N). > So Gnus requests (- X 100):*, to just double-check whether any of the > most recent messages have changed their flags, too. I see, that seems like a nice thing. > So the primary reason for the flag fetch isn't really to get the flags > (although that's a nice side-effect), but to find out how many new > articles there are in the mail box. > > If there's a faster way to do this, I'd love to hear it. I propose to do: 1) Parallel STATUS like the old code to discover which groups have been changed at all. 2) For those groups that have changed, do the EXAMINE+FETCH dance. > (Oh, there's QRESYNC, which might speed stuff up, but I haven't > implemented that yet.) Could be useful... >> 3) When imap-log is non-nil, the *imap-log* buffer no longer contains >> the output from the server, which makes debugging harder. > > I was planning on getting rid of the log buffer totally when it works > perfectly. Any day now! :-) Oh no, it has saved my day many times. Other mail clients, like Claws, has a IMAP network inspection tool. It would be a shame to see lose this feature in Gnus.. >> 4) Nnimap mail splitting is not working any more. All my e-mail sits in >> INBOX and there is no filtering going on. I don't see anything in >> GNUS-NEWS about any change in this area, so it must be a bug. :-) > > :-) I've now added a mention, but it's mostly related to the new > nnimap-inbox variable, and using the standard nnmail-split-methods. I'll take a look. > Yup; fixed. That idiom brings back memories. Glad to have you back working on Gnus, Lars! /Simon