From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/69452 Path: news.gmane.org!not-for-mail From: Martin Stjernholm Newsgroups: gmane.emacs.gnus.general Subject: Re: Patch: Support for non-ascii characters in imap group names Date: Sun, 21 Mar 2010 21:26:12 +0100 Message-ID: <7miq8pl8uz.fsf@kolon.stjernholm.org> References: <7mljev50dc.fsf@kolon.stjernholm.org> <7m635qutf5.fsf@kolon.stjernholm.org> <87zl23dmiy.fsf@lifelogs.com> <7m1vfdmwxk.fsf@kolon.stjernholm.org> <87iq8p7bni.fsf@lifelogs.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: dough.gmane.org 1269203281 26121 80.91.229.12 (21 Mar 2010 20:28:01 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Sun, 21 Mar 2010 20:28:01 +0000 (UTC) Cc: Ted Zlatanov To: ding@gnus.org Original-X-From: ding-owner+M17847@lists.math.uh.edu Sun Mar 21 21:27:52 2010 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1NtRkS-0006ae-Cx for ding-account@gmane.org; Sun, 21 Mar 2010 21:27:48 +0100 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1NtRjE-0004sd-J9; Sun, 21 Mar 2010 15:26:32 -0500 Original-Received: from mx1.math.uh.edu ([129.7.128.32]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1NtRjC-0004sP-VT for ding@lists.math.uh.edu; Sun, 21 Mar 2010 15:26:30 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx1.math.uh.edu with esmtp (Exim 4.69) (envelope-from ) id 1NtRix-00046E-IC for ding@lists.math.uh.edu; Sun, 21 Mar 2010 15:26:30 -0500 Original-Received: from mail.roxen.com ([212.247.29.220]) by quimby with esmtp (Exim 3.36 #1 (Debian)) id 1NtSf1-00050p-00 for ; Sun, 21 Mar 2010 22:26:15 +0100 Original-Received: from localhost (c-d5f0e255.710-9-64736c14.cust.bredbandsbolaget.se [85.226.240.213]) by mail.roxen.com (Postfix) with ESMTP id D62E0628144; Sun, 21 Mar 2010 21:26:13 +0100 (CET) In-Reply-To: <87iq8p7bni.fsf@lifelogs.com> (Ted Zlatanov's message of "Sun, 21 Mar 2010 13:49:37 -0500") User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/23.1 (gnu/linux) X-Spam-Score: -1.9 (-) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:69452 Archived-At: --=-=-= Ted Zlatanov wrote: > I haven't seen it (which is probably my fault as I was not paying > attention), can you please resend and CC me? Sure, it's attached below. For me it makes the startup time go down from 5 minutes or more to 10-20 seconds when I start up Gnus with my ~140 imap groups over a slow mobile network. The thing is that nnimap-retrieve-groups is very sensitive to high latency. The patch doesn't remedy that, but it cuts down the number of groups it polls. > Interesting. It sounds like we should get rid of the variable in code > completely and always use an accessor function with a "decoding" > parameter (so we can get the encoded or the unencoded name). I don't > like the current hidden behavior. Do you agree? You're right, using the variable instead in situations where the encoding isn't relevant is arguably a sort of micro-optimization. It shouldn't be a significant performance hit to use the function instead. Or if it is, the right solution would be to optimize in imap.el by keeping the decoded mailbox name in a variable. I don't really see the point with adding a "decoding" parameter though. The utf-7 encoding is just for transport in the imap protocol, and I can't see any reason why higher code would be interested in that. If anything, it'd be reasonable to change imap-current-mailbox (and possibly other variables) to contain decoded mailbox names instead, but that'd of course have compatibility implications which probably don't make it worth the hassle. --=-=-= Content-Type: message/rfc822 Content-Disposition: inline From: Martin Stjernholm To: ding@gnus.org Subject: Patch: Faster imap folder scanning at startup References: <7mljev50dc.fsf@kolon.stjernholm.org> X-Draft-From: ("prog.emacs" 121) Date: Sun, 14 Feb 2010 16:38:39 +0100 In-Reply-To: <7mljev50dc.fsf@kolon.stjernholm.org> (Martin Stjernholm's message of "Sun, 14 Feb 2010 15:59:43 +0100") Message-ID: <7mhbpj4ykg.fsf@kolon.stjernholm.org> User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/23.1 (gnu/linux) MIME-Version: 1.0 Hello again. This is a patch that significantly improves the Gnus startup time when there are many subscribed imap folders. It does not depend functionally on my previous patch to fix non-ascii imap group names, but it does require that one to apply cleanly. I've split it up as a patch sequence with atomic changes: First two patches to fix some bugs in the uidvalidity handling, then one that introduces a function to avoid code duplication, and last the real patch. See detailed descriptions below. ---------------------------------------- (nnimap-verify-uidvalidity): Fixed bug where uidvalidity wasn't updated after mismatch. --- a/lisp/nnimap.el +++ b/lisp/nnimap.el @@ -548,8 +548,9 @@ If SERVER is nil, uses the current server." (if old-uidvalidity (if (not (equal old-uidvalidity new-uidvalidity)) ;; uidvalidity clash - (gnus-delete-file file) - (gnus-group-set-parameter gnusgroup 'uidvalidity new-uidvalidity) + (progn + (gnus-group-set-parameter gnusgroup 'uidvalidity new-uidvalidity) + (gnus-delete-file file)) t) (gnus-group-add-parameter gnusgroup (cons 'uidvalidity new-uidvalidity)) t))) ---------------------------------------- (nnimap-verify-uidvalidity): Clear cached mailbox info correctly when uidvalidity changes. --- a/lisp/nnimap.el +++ b/lisp/nnimap.el @@ -550,9 +550,14 @@ If SERVER is nil, uses the current server." ;; uidvalidity clash (progn (gnus-group-set-parameter gnusgroup 'uidvalidity new-uidvalidity) + (gnus-sethash (gnus-group-prefixed-name group server) + nil nnimap-mailbox-info) (gnus-delete-file file)) t) (gnus-group-add-parameter gnusgroup (cons 'uidvalidity new-uidvalidity)) + (gnus-sethash ; Maybe not necessary here. + (gnus-group-prefixed-name group server) + nil nnimap-mailbox-info) t))) (defun nnimap-before-find-minmax-bugworkaround () ---------------------------------------- (nnimap-group-prefixed-name): New function to avoid some code duplication. (nnimap-verify-uidvalidity, nnimap-group-overview-filename, nnimap-request-group): Use it. --- a/lisp/nnimap.el +++ b/lisp/nnimap.el @@ -505,6 +505,12 @@ See also `nnimap-log'." (and group (mm-encode-coding-string group (gnus-group-name-charset nil group)))) +(defun nnimap-group-prefixed-name (group &optional server) + (gnus-group-prefixed-name group + (gnus-server-to-method + (format "nnimap:%s" + (or server nnimap-current-server))))) + (defsubst nnimap-get-server-buffer (server) "Return buffer for SERVER, if nil use current server." (cadr (assoc (or server nnimap-current-server) nnimap-server-buffer-alist))) @@ -525,9 +531,7 @@ If SERVER is nil, uses the current server." (defun nnimap-verify-uidvalidity (group server) "Verify stored uidvalidity match current one in GROUP on SERVER." - (let* ((gnusgroup (gnus-group-prefixed-name - group (gnus-server-to-method - (format "nnimap:%s" server)))) + (let* ((gnusgroup (nnimap-group-prefixed-name group server)) (new-uidvalidity (imap-mailbox-get 'uidvalidity)) (old-uidvalidity (gnus-group-get-parameter gnusgroup 'uidvalidity)) (dir (file-name-as-directory (expand-file-name nnimap-directory))) @@ -678,9 +682,7 @@ If EXAMINE is non-nil the group is selected read-only." "Make file name for GROUP on SERVER." (let* ((dir (file-name-as-directory (expand-file-name nnimap-directory))) (uidvalidity (gnus-group-get-parameter - (gnus-group-prefixed-name - group (gnus-server-to-method - (format "nnimap:%s" server))) + (nnimap-group-prefixed-name group server) 'uidvalidity)) (name (nnheader-translate-file-chars (concat nnimap-nov-file-name @@ -1034,8 +1036,7 @@ function is generally only called when Gnus is shutting down." (deffoo nnimap-request-group (group &optional server fast) (nnimap-request-update-info-internal group - (gnus-get-info (gnus-group-prefixed-name - group (gnus-server-to-method (format "nnimap:%s" server)))) + (gnus-get-info (nnimap-group-prefixed-name group server)) server) (when (nnimap-possibly-change-group group server) (nnimap-before-find-minmax-bugworkaround) ---------------------------------------- (nnimap-retrieve-groups, nnimap-verify-uidvalidity, nnimap-update-unseen): Significantly improved speed of Gnus startup with many imap folders. This is done by caching the group status from the imap server persistently in a group parameter `imap-status'. (This was cached before too if `nnimap-retrieve-groups-asynchronous' was set, but not persistently, so every Gnus startup was still very slow.) --- a/lisp/nnimap.el +++ b/lisp/nnimap.el @@ -554,11 +554,13 @@ If SERVER is nil, uses the current server." ;; uidvalidity clash (progn (gnus-group-set-parameter gnusgroup 'uidvalidity new-uidvalidity) + (gnus-group-remove-parameter gnusgroup 'imap-status) (gnus-sethash (gnus-group-prefixed-name group server) nil nnimap-mailbox-info) (gnus-delete-file file)) t) (gnus-group-add-parameter gnusgroup (cons 'uidvalidity new-uidvalidity)) + (gnus-group-remove-parameter gnusgroup 'imap-status) (gnus-sethash ; Maybe not necessary here. (gnus-group-prefixed-name group server) nil nnimap-mailbox-info) @@ -1060,8 +1062,7 @@ function is generally only called when Gnus is shutting down." nnimap-mailbox-info))) (list (nth 0 old) (nth 1 old) (imap-mailbox-status (nnimap-decode-group-name group) - 'unseen nnimap-server-buffer) - (nth 3 old))) + 'unseen nnimap-server-buffer))) nnimap-mailbox-info)) (defun nnimap-close-group (group &optional server) @@ -1167,9 +1168,9 @@ function is generally only called when Gnus is shutting down." (setq decoded-group (nnimap-decode-group-name group)) (gnus-message 9 "nnimap: Quickly checking mailbox %s" decoded-group) - (add-to-list (if (gnus-gethash-safe - (gnus-group-prefixed-name group server) - nnimap-mailbox-info) + (add-to-list (if (gnus-group-get-parameter + (nnimap-group-prefixed-name group) + 'imap-status) 'asyncgroups 'slowgroups) (list group (imap-mailbox-status-asynch @@ -1177,61 +1178,71 @@ function is generally only called when Gnus is shutting down." '(uidvalidity uidnext unseen) nnimap-server-buffer)))) (dolist (asyncgroup asyncgroups) - (let ((group (nth 0 asyncgroup)) - (tag (nth 1 asyncgroup)) - new old) + (let* ((group (nth 0 asyncgroup)) + (tag (nth 1 asyncgroup)) + (gnusgroup (nnimap-group-prefixed-name group)) + (saved-uidvalidity (gnus-group-get-parameter gnusgroup + 'uidvalidity)) + (saved-imap-status (gnus-group-get-parameter gnusgroup + 'imap-status)) + (saved-info (and saved-imap-status + (split-string saved-imap-status " ")))) (setq decoded-group (nnimap-decode-group-name group)) (when (imap-ok-p (imap-wait-for-tag tag nnimap-server-buffer)) - (if (or (not (string= - (nth 0 (gnus-gethash (gnus-group-prefixed-name - group server) - nnimap-mailbox-info)) + (if (or (not (equal + saved-uidvalidity (imap-mailbox-get 'uidvalidity decoded-group nnimap-server-buffer))) - (not (string= - (nth 1 (gnus-gethash (gnus-group-prefixed-name - group server) - nnimap-mailbox-info)) + (not (equal + (nth 0 saved-info) (imap-mailbox-get 'uidnext decoded-group nnimap-server-buffer)))) (push (list group) slowgroups) - (insert (nth 3 (gnus-gethash (gnus-group-prefixed-name - group server) - nnimap-mailbox-info)))))))) + (gnus-sethash + (gnus-group-prefixed-name group server) + (list (imap-mailbox-get 'uidvalidity + decoded-group nnimap-server-buffer) + (imap-mailbox-get 'uidnext + decoded-group nnimap-server-buffer) + (imap-mailbox-get 'unseen + decoded-group nnimap-server-buffer)) + nnimap-mailbox-info) + (insert (format "\"%s\" %s %s y\n" group + (nth 2 saved-info) + (nth 1 saved-info)))))))) (dolist (group slowgroups) (if nnimap-retrieve-groups-asynchronous (setq group (car group))) (setq decoded-group (nnimap-decode-group-name group)) (gnus-message 7 "nnimap: Mailbox %s modified" decoded-group) - (imap-mailbox-put 'uidnext nil decoded-group nnimap-server-buffer) (or (member "\\NoSelect" (imap-mailbox-get 'list-flags decoded-group nnimap-server-buffer)) - (let* ((info (nnimap-find-minmax-uid group 'examine)) - (str (format "\"%s\" %d %d y\n" group - (or (nth 2 info) 0) - (max 1 (or (nth 1 info) 1))))) + (let* ((gnusgroup (nnimap-group-prefixed-name group)) + (status (imap-mailbox-status + decoded-group '(uidvalidity uidnext unseen) + nnimap-server-buffer)) + (info (nnimap-find-minmax-uid group 'examine)) + (min-uid (max 1 (or (nth 1 info) 1))) + (max-uid (or (nth 2 info) 0))) (when (> (or (imap-mailbox-get 'recent decoded-group nnimap-server-buffer) 0) 0) (push (list (cons decoded-group 0)) nnmail-split-history)) - (insert str) - (when nnimap-retrieve-groups-asynchronous - (gnus-sethash - (gnus-group-prefixed-name group server) - (list (or (imap-mailbox-get - 'uidvalidity decoded-group nnimap-server-buffer) - (imap-mailbox-status - decoded-group 'uidvalidity nnimap-server-buffer)) - (or (imap-mailbox-get - 'uidnext decoded-group nnimap-server-buffer) - (imap-mailbox-status - decoded-group 'uidnext nnimap-server-buffer)) - (or (imap-mailbox-get - 'unseen decoded-group nnimap-server-buffer) - (imap-mailbox-status - decoded-group 'unseen nnimap-server-buffer)) - str) - nnimap-mailbox-info))))))) + (insert (format "\"%s\" %d %d y\n" group max-uid min-uid)) + (gnus-sethash + (gnus-group-prefixed-name group server) + status + nnimap-mailbox-info) + (if (not (equal (nth 0 status) + (gnus-group-get-parameter gnusgroup + 'uidvalidity))) + (nnimap-verify-uidvalidity group nnimap-current-server)) + ;; The imap-status parameter is a string on the form + ;; " ". + (gnus-group-add-parameter + gnusgroup + (cons 'imap-status + (format "%s %s %s" (nth 1 status) min-uid max-uid)))))))) (gnus-message 5 "nnimap: Checking mailboxes...done") 'active)) --=-=-=--