Re: Slow expiry with nnimap+Courier [patch]

[Jouni K Seppanen <jks@iki.fi>]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]

Simon Josefsson <jas@extundo.com> writes:

> Would you like to submit a patch for the manual as well?

Here's a patch. I also documented nnimap-nov-is-evil, or rather my
understanding of what it seems to do. It occurred to me that people to
whom slow expiry is a problem might go to the index of the manual and
look up "expiring mail". There was already one index entry for
"expiring IMAP mail", so I added this one, and in the process I added
entries on some other pages that might be relevant to people looking
up "expiring mail".

-- 
Jouni K Seppänen


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: doc-search-uid.patch --]
[-- Type: text/x-patch, Size: 7505 bytes --]

Index: texi/ChangeLog
===================================================================
RCS file: /usr/local/cvsroot/gnus/texi/ChangeLog,v
retrieving revision 7.184
diff -u -u -r7.184 ChangeLog
--- texi/ChangeLog	25 Oct 2005 19:00:24 -0000	7.184
+++ texi/ChangeLog	27 Oct 2005 18:28:33 -0000
@@ -1,3 +1,10 @@
+2005-10-27  Jouni K Seppanen  <jks@iki.fi>
+
+	* gnus.texi: Add index terms related to expiry.
+	(IMAP): Document variables nnimap-nov-is-evil and
+	nnimap-search-uids-not-since-is-evil. Mention latter in connection
+	with nnimap-expunge-search-string and in the section on expiring.
+	
 2005-10-25  Reiner Steib  <Reiner.Steib@gmx.de>
 
 	* gnus-coding.texi (Dependencies): Add some text.
Index: texi/gnus.texi
===================================================================
RCS file: /usr/local/cvsroot/gnus/texi/gnus.texi,v
retrieving revision 7.124
diff -u -u -r7.124 gnus.texi
--- texi/gnus.texi	17 Oct 2005 12:17:12 -0000	7.124
+++ texi/gnus.texi	27 Oct 2005 18:28:42 -0000
@@ -2822,6 +2822,7 @@
 
 @item auto-expire
 @cindex auto-expire
+@cindex expiring mail
 If the group parameter has an element that looks like @code{(auto-expire
 . t)}, all articles read will be marked as expirable.  For an
 alternative approach, @pxref{Expiring Mail}.
@@ -2830,6 +2831,7 @@
 
 @item total-expire
 @cindex total-expire
+@cindex expiring mail
 If the group parameter has an element that looks like
 @code{(total-expire . t)}, all read articles will be put through the
 expiry process, even if they are not marked as expirable.  Use with
@@ -2839,7 +2841,7 @@
 See also @code{gnus-total-expirable-newsgroups}.
 
 @item expiry-wait
-@cindex expiry-wait
+@vindex expiry-wait
 @vindex nnmail-expiry-wait-function
 If the group parameter has an element that looks like
 @code{(expiry-wait . 10)}, this value will override any
@@ -2849,7 +2851,7 @@
 symbols @code{never} or @code{immediate}.
 
 @item expiry-target
-@cindex expiry-target
+@vindex expiry-target
 Where expired messages end up.  This parameter overrides
 @code{nnmail-expiry-target}.
 
@@ -3419,6 +3421,7 @@
 @item C-c C-x
 @kindex C-c C-x (Group)
 @findex gnus-group-expire-articles
+@cindex expiring mail
 Run all expirable articles in the current group through the expiry
 process (if any) (@code{gnus-group-expire-articles}).  That is, delete
 all expirable articles in the group that have been around for a while.
@@ -3427,6 +3430,7 @@
 @item C-c C-M-x
 @kindex C-c C-M-x (Group)
 @findex gnus-group-expire-all-groups
+@cindex expiring mail
 Run all expirable articles in all groups through the expiry process
 (@code{gnus-group-expire-all-groups}).
 
@@ -3787,6 +3791,7 @@
 @item C-c C-x
 @kindex C-c C-x (Topic)
 @findex gnus-topic-expire-articles
+@cindex expiring mail
 Run all expirable articles in the current group or topic through the
 expiry process (if any)
 (@code{gnus-topic-expire-articles}).  (@pxref{Expiring Mail}).
@@ -10056,6 +10061,7 @@
 @item B e
 @kindex B e (Summary)
 @findex gnus-summary-expire-articles
+@cindex expiring mail
 Run all expirable articles in the current group through the expiry
 process (@code{gnus-summary-expire-articles}).  That is, delete all
 expirable articles in the group that have been around for a while.
@@ -10064,6 +10070,7 @@
 @item B C-M-e
 @kindex B C-M-e (Summary)
 @findex gnus-summary-expire-articles-now
+@cindex expiring mail
 Delete all the expirable articles in the group
 (@code{gnus-summary-expire-articles-now}).  This means that @strong{all}
 articles eligible for expiry in the current group will
@@ -14553,6 +14560,7 @@
 @node Expiring Mail
 @subsection Expiring Mail
 @cindex article expiry
+@cindex expiring mail
 
 Traditional mail readers have a tendency to remove mail articles when
 you mark them as read, in some way.  Gnus takes a fundamentally
@@ -16465,6 +16473,7 @@
 @item nnimap-expunge-search-string
 @cindex expunging
 @vindex nnimap-expunge-search-string
+@cindex expiring @acronym{IMAP} mail
 
 This variable contain the @acronym{IMAP} search command sent to server when
 searching for articles eligible for expiring.  The default is
@@ -16476,6 +16485,10 @@
 messages instead of the internal article date.  See section 6.4.4 of
 RFC 2060 for more information on valid strings.
 
+However, if @code{nnimap-search-uids-not-since-is-evil} 
+is true, the command is @code{"UID NOT SINCE @var{date}"},
+regardless of this variable.
+
 @item nnimap-authinfo-file
 @vindex nnimap-authinfo-file
 
@@ -16501,6 +16514,47 @@
 seem to need this under some circumstances; it was reported that
 Courier 1.7.1 did.
 
+@item nnimap-nov-is-evil
+@vindex nnimap-nov-is-evil
+@cindex Courier @acronym{IMAP} server
+@cindex @acronym{NOV}
+
+Never generate or use a local @acronym{NOV} database. Defaults to the
+value of @code{gnus-agent}.
+
+Using a @acronym{NOV} database usually makes header fetching much
+faster, but it uses the @code{UID SEARCH UID} command, which is very
+slow on some servers (notably some versions of Courier). Since the Gnus
+Agent caches the information in the @acronym{NOV} database without using
+the slow command, this variable defaults to true if the Agent is in use,
+and false otherwise.
+
+@item nnimap-search-uids-not-since-is-evil
+@vindex nnimap-search-uids-not-since-is-evil
+@cindex Courier @acronym{IMAP} server
+@cindex expiring @acronym{IMAP} mail
+
+Avoid the @code{UID SEARCH UID @var{message numbers} NOT SINCE
+@var{date}} command, which is slow on some @acronym{IMAP} servers
+(notably, some versions of Courier). Instead, use @code{UID SEARCH SINCE
+@var{date}} and prune the list of expirable articles within Gnus.
+
+When Gnus expires your mail (@pxref{Expiring Mail}), it starts with a
+list of expirable articles and asks the IMAP server questions like ``Of
+these articles, which ones are older than a week?'' While this seems
+like a perfectly reasonable question, some IMAP servers take a long time
+to answer it, since they seemingly go looking into every old article to
+see if it is one of the expirable ones. Curiously, the question ``Of
+@emph{all} articles, which ones are newer than a week?'' seems to be
+much faster to answer, so setting this variable causes Gnus to ask this
+question and figure out the answer to the real question itself.
+
+This problem can really sneak up on you: when you first configure Gnus,
+everything works fine, but once you accumulate a couple thousand
+messages, you start cursing Gnus for being so slow. On the other hand,
+if you get a lot of email within a week, setting this variable will
+cause a lot of network traffic between Gnus and the IMAP server.
+
 @end table
 
 @menu
@@ -16688,7 +16742,7 @@
 
 @node Expiring in IMAP
 @subsection Expiring in IMAP
-@cindex expiring imap mail
+@cindex expiring @acronym{IMAP} mail
 
 Even though @code{nnimap} is not a proper @code{nnmail} derived back
 end, it supports most features in regular expiring (@pxref{Expiring
@@ -16707,6 +16761,9 @@
 your server must support permanent storage of client specific flags on
 messages.  Most do, fortunately.
 
+If expiring @acronym{IMAP} mail seems very slow, try setting the server
+variable @code{nnimap-search-uids-not-since-is-evil}.
+
 @table @code
 
 @item nnmail-expiry-wait
@@ -18993,7 +19050,7 @@
 @findex gnus-agent-expire-group
 @cindex agent expiry
 @cindex Gnus agent expiry
-@cindex expiry
+@cindex expiry, in Gnus agent
 
 The Agent back end, @code{nnagent}, doesn't handle expiry.  Well, at
 least it doesn't handle it like other back ends.  Instead, there are