Slow expiry with nnimap+Courier [patch]

Slow expiry with nnimap+Courier [patch]

[Jouni K Seppanen]

[-- Attachment #1: Type: text/plain, Size: 1229 bytes --]

Hi,

I'm using nnimap with a Courier IMAP server (some version from 2004),
and expiry is taking longer and longer; currently exiting my INBOX
(with article numbers in the 6500s, perhaps half of those occupied by
actual articles) takes about 30 seconds. A friend pointed me to
<http://my.gnus.org/book/print/232>, which hints that Courier's UID
SEARCH implementation is sometimes slow.

What Gnus is sending to Courier is something like

  UID SEARCH UID 6512:6513,6515,6519,6526:6528 NOT SINCE 19-Oct-2005

and giving such commands by hand results in a long wait. In
comparison, the commands

  UID SEARCH NOT SINCE 19-Oct-2005
  UID SEARCH SINCE 19-Oct-2005

are almost instantaneous.

So it seems that Courier is doing something very suboptimal with the
Gnus command. The attached patch (against 5.10.6) seems to have
helped: now exiting the INBOX summary takes something like 2 seconds
instead of the previous 30.

I'm not very intimate with Emacs Lisp, Gnus or IMAP, so somebody
should probably check the code (and it looks like
nnimap-request-expire-articles could use some refactoring). However,
it seems to have worked correctly for me in the, um, three times I've
exited a summary buffer this far.

-- 
Jouni K Seppänen


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnimap-search-uids-not-since-is-evil.patch --]
[-- Type: text/x-patch, Size: 1534 bytes --]

--- gnus-5.10.6/lisp/nnimap.el.orig	2003-09-05 01:22:18.000000000 +0300
+++ gnus-5.10.6/lisp/nnimap.el	2005-10-26 15:20:45.366839000 +0300
@@ -312,6 +312,11 @@
 Unlike other backends, you do not need to take special care if you
 flip this variable.")
 
+(defvoo nnimap-search-uids-not-since-is-evil nil
+  "If non-nil, avoid \"UID SEARCH UID ... NOT SINCE\" queries when expiring.
+Instead, use \"UID SEARCH SINCE\" to prune the list of expirable
+articles within Gnus. This seems to be faster on Courier in some cases.")
+
 (defvoo nnimap-expunge-on-close 'always ; 'ask, 'never
   "Whether to expunge a group when it is closed.
 When a IMAP group with articles marked for deletion is closed, this
@@ -1411,6 +1416,21 @@
 			     (gnus-compress-sequence oldarts)) "\\Deleted")
 		       (setq articles (gnus-set-difference
 				       articles oldarts))))))
+		((and nnimap-search-uids-not-since-is-evil (numberp days))
+		 (let* ((all-new-articles 
+			 (gnus-compress-sequence
+			  (imap-search (format "SINCE %s"
+					       (nnimap-date-days-ago days)))))
+			(oldartseq
+			 (gnus-range-difference artseq all-new-articles))
+			(oldarts (gnus-uncompress-range oldartseq)))
+		   (when oldarts
+		     (nnimap-expiry-target oldarts group server)
+		     (when (imap-message-flags-add
+			    (imap-range-to-message-set oldartseq) 
+			    "\\Deleted")
+		       (setq articles (gnus-set-difference
+				       articles oldarts))))))
 		((numberp days)
 		 (let ((oldarts (imap-search
 				 (format nnimap-expunge-search-string

Re: Slow expiry with nnimap+Courier [patch]

[Simon Josefsson]

Jouni K Seppanen <jks@iki.fi> writes:

> So it seems that Courier is doing something very suboptimal with the
> Gnus command. The attached patch (against 5.10.6) seems to have
> helped: now exiting the INBOX summary takes something like 2 seconds
> instead of the previous 30.

Looks good to me.  I think this change is large enough so that
copyright papers are required, are you willing to transfer your
copyright to the FSF so we can install your patch in Gnus?  If so,
please fill out the form below and mail it.

Thanks!

Please email the following information to assign@gnu.org, and we
will send you the assignment form for your past and future changes.

Please use your full legal name (in ASCII characters) as the subject
line of the message.
----------------------------------------------------------------------
REQUEST: SEND FORM FOR PAST AND FUTURE CHANGES

[What is the name of the program or package you're contributing to?]


[Did you copy any files or text written by someone else in these changes?
Even if that material is free software, we need to know about it.]


[Do you have an employer who might have a basis to claim to own
your changes?  Do you attend a school which might make such a claim?]


[For the copyright registration, what country are you a citizen of?]


[What year were you born?]


[Please write your email address here.]


[Please write your postal address here.]





[Which files have you changed so far, and which new files have you written
so far?]

Re: Slow expiry with nnimap+Courier [patch]

[Simon Josefsson]

Simon Josefsson <jas@extundo.com> writes:

> Jouni K Seppanen <jks@iki.fi> writes:
>
>> So it seems that Courier is doing something very suboptimal with the
>> Gnus command. The attached patch (against 5.10.6) seems to have
>> helped: now exiting the INBOX summary takes something like 2 seconds
>> instead of the previous 30.
>
> Looks good to me.  I think this change is large enough so that
> copyright papers are required, are you willing to transfer your
> copyright to the FSF so we can install your patch in Gnus?  If so,
> please fill out the form below and mail it.

On second thought, the patch was mostly cut'n'paste from similar code
around it, so I have installed it in both No Gnus and Gnus 5.10.

Would you like to submit a patch for the manual as well?

Thanks!

Re: Slow expiry with nnimap+Courier [patch]

[Jouni K Seppanen]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]

Simon Josefsson <jas@extundo.com> writes:

> Would you like to submit a patch for the manual as well?

Here's a patch. I also documented nnimap-nov-is-evil, or rather my
understanding of what it seems to do. It occurred to me that people to
whom slow expiry is a problem might go to the index of the manual and
look up "expiring mail". There was already one index entry for
"expiring IMAP mail", so I added this one, and in the process I added
entries on some other pages that might be relevant to people looking
up "expiring mail".

-- 
Jouni K Seppänen


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: doc-search-uid.patch --]
[-- Type: text/x-patch, Size: 7505 bytes --]

Index: texi/ChangeLog
===================================================================
RCS file: /usr/local/cvsroot/gnus/texi/ChangeLog,v
retrieving revision 7.184
diff -u -u -r7.184 ChangeLog
--- texi/ChangeLog	25 Oct 2005 19:00:24 -0000	7.184
+++ texi/ChangeLog	27 Oct 2005 18:28:33 -0000
@@ -1,3 +1,10 @@
+2005-10-27  Jouni K Seppanen  <jks@iki.fi>
+
+	* gnus.texi: Add index terms related to expiry.
+	(IMAP): Document variables nnimap-nov-is-evil and
+	nnimap-search-uids-not-since-is-evil. Mention latter in connection
+	with nnimap-expunge-search-string and in the section on expiring.
+	
 2005-10-25  Reiner Steib  <Reiner.Steib@gmx.de>
 
 	* gnus-coding.texi (Dependencies): Add some text.
Index: texi/gnus.texi
===================================================================
RCS file: /usr/local/cvsroot/gnus/texi/gnus.texi,v
retrieving revision 7.124
diff -u -u -r7.124 gnus.texi
--- texi/gnus.texi	17 Oct 2005 12:17:12 -0000	7.124
+++ texi/gnus.texi	27 Oct 2005 18:28:42 -0000
@@ -2822,6 +2822,7 @@
 
 @item auto-expire
 @cindex auto-expire
+@cindex expiring mail
 If the group parameter has an element that looks like @code{(auto-expire
 . t)}, all articles read will be marked as expirable.  For an
 alternative approach, @pxref{Expiring Mail}.
@@ -2830,6 +2831,7 @@
 
 @item total-expire
 @cindex total-expire
+@cindex expiring mail
 If the group parameter has an element that looks like
 @code{(total-expire . t)}, all read articles will be put through the
 expiry process, even if they are not marked as expirable.  Use with
@@ -2839,7 +2841,7 @@
 See also @code{gnus-total-expirable-newsgroups}.
 
 @item expiry-wait
-@cindex expiry-wait
+@vindex expiry-wait
 @vindex nnmail-expiry-wait-function
 If the group parameter has an element that looks like
 @code{(expiry-wait . 10)}, this value will override any
@@ -2849,7 +2851,7 @@
 symbols @code{never} or @code{immediate}.
 
 @item expiry-target
-@cindex expiry-target
+@vindex expiry-target
 Where expired messages end up.  This parameter overrides
 @code{nnmail-expiry-target}.
 
@@ -3419,6 +3421,7 @@
 @item C-c C-x
 @kindex C-c C-x (Group)
 @findex gnus-group-expire-articles
+@cindex expiring mail
 Run all expirable articles in the current group through the expiry
 process (if any) (@code{gnus-group-expire-articles}).  That is, delete
 all expirable articles in the group that have been around for a while.
@@ -3427,6 +3430,7 @@
 @item C-c C-M-x
 @kindex C-c C-M-x (Group)
 @findex gnus-group-expire-all-groups
+@cindex expiring mail
 Run all expirable articles in all groups through the expiry process
 (@code{gnus-group-expire-all-groups}).
 
@@ -3787,6 +3791,7 @@
 @item C-c C-x
 @kindex C-c C-x (Topic)
 @findex gnus-topic-expire-articles
+@cindex expiring mail
 Run all expirable articles in the current group or topic through the
 expiry process (if any)
 (@code{gnus-topic-expire-articles}).  (@pxref{Expiring Mail}).
@@ -10056,6 +10061,7 @@
 @item B e
 @kindex B e (Summary)
 @findex gnus-summary-expire-articles
+@cindex expiring mail
 Run all expirable articles in the current group through the expiry
 process (@code{gnus-summary-expire-articles}).  That is, delete all
 expirable articles in the group that have been around for a while.
@@ -10064,6 +10070,7 @@
 @item B C-M-e
 @kindex B C-M-e (Summary)
 @findex gnus-summary-expire-articles-now
+@cindex expiring mail
 Delete all the expirable articles in the group
 (@code{gnus-summary-expire-articles-now}).  This means that @strong{all}
 articles eligible for expiry in the current group will
@@ -14553,6 +14560,7 @@
 @node Expiring Mail
 @subsection Expiring Mail
 @cindex article expiry
+@cindex expiring mail
 
 Traditional mail readers have a tendency to remove mail articles when
 you mark them as read, in some way.  Gnus takes a fundamentally
@@ -16465,6 +16473,7 @@
 @item nnimap-expunge-search-string
 @cindex expunging
 @vindex nnimap-expunge-search-string
+@cindex expiring @acronym{IMAP} mail
 
 This variable contain the @acronym{IMAP} search command sent to server when
 searching for articles eligible for expiring.  The default is
@@ -16476,6 +16485,10 @@
 messages instead of the internal article date.  See section 6.4.4 of
 RFC 2060 for more information on valid strings.
 
+However, if @code{nnimap-search-uids-not-since-is-evil} 
+is true, the command is @code{"UID NOT SINCE @var{date}"},
+regardless of this variable.
+
 @item nnimap-authinfo-file
 @vindex nnimap-authinfo-file
 
@@ -16501,6 +16514,47 @@
 seem to need this under some circumstances; it was reported that
 Courier 1.7.1 did.
 
+@item nnimap-nov-is-evil
+@vindex nnimap-nov-is-evil
+@cindex Courier @acronym{IMAP} server
+@cindex @acronym{NOV}
+
+Never generate or use a local @acronym{NOV} database. Defaults to the
+value of @code{gnus-agent}.
+
+Using a @acronym{NOV} database usually makes header fetching much
+faster, but it uses the @code{UID SEARCH UID} command, which is very
+slow on some servers (notably some versions of Courier). Since the Gnus
+Agent caches the information in the @acronym{NOV} database without using
+the slow command, this variable defaults to true if the Agent is in use,
+and false otherwise.
+
+@item nnimap-search-uids-not-since-is-evil
+@vindex nnimap-search-uids-not-since-is-evil
+@cindex Courier @acronym{IMAP} server
+@cindex expiring @acronym{IMAP} mail
+
+Avoid the @code{UID SEARCH UID @var{message numbers} NOT SINCE
+@var{date}} command, which is slow on some @acronym{IMAP} servers
+(notably, some versions of Courier). Instead, use @code{UID SEARCH SINCE
+@var{date}} and prune the list of expirable articles within Gnus.
+
+When Gnus expires your mail (@pxref{Expiring Mail}), it starts with a
+list of expirable articles and asks the IMAP server questions like ``Of
+these articles, which ones are older than a week?'' While this seems
+like a perfectly reasonable question, some IMAP servers take a long time
+to answer it, since they seemingly go looking into every old article to
+see if it is one of the expirable ones. Curiously, the question ``Of
+@emph{all} articles, which ones are newer than a week?'' seems to be
+much faster to answer, so setting this variable causes Gnus to ask this
+question and figure out the answer to the real question itself.
+
+This problem can really sneak up on you: when you first configure Gnus,
+everything works fine, but once you accumulate a couple thousand
+messages, you start cursing Gnus for being so slow. On the other hand,
+if you get a lot of email within a week, setting this variable will
+cause a lot of network traffic between Gnus and the IMAP server.
+
 @end table
 
 @menu
@@ -16688,7 +16742,7 @@
 
 @node Expiring in IMAP
 @subsection Expiring in IMAP
-@cindex expiring imap mail
+@cindex expiring @acronym{IMAP} mail
 
 Even though @code{nnimap} is not a proper @code{nnmail} derived back
 end, it supports most features in regular expiring (@pxref{Expiring
@@ -16707,6 +16761,9 @@
 your server must support permanent storage of client specific flags on
 messages.  Most do, fortunately.
 
+If expiring @acronym{IMAP} mail seems very slow, try setting the server
+variable @code{nnimap-search-uids-not-since-is-evil}.
+
 @table @code
 
 @item nnmail-expiry-wait
@@ -18993,7 +19050,7 @@
 @findex gnus-agent-expire-group
 @cindex agent expiry
 @cindex Gnus agent expiry
-@cindex expiry
+@cindex expiry, in Gnus agent
 
 The Agent back end, @code{nnagent}, doesn't handle expiry.  Well, at
 least it doesn't handle it like other back ends.  Instead, there are

Re: Slow expiry with nnimap+Courier [patch]

[Reiner Steib]

[-- Attachment #1: Type: text/plain, Size: 2197 bytes --]

On Thu, Oct 27 2005, Jouni K Seppanen wrote:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Would you like to submit a patch for the manual as well?
>
> Here's a patch. I also documented nnimap-nov-is-evil, or rather my
> understanding of what it seems to do. 

Can someone comment on this?  Is Jouni's documentation correct? [1]

> It occurred to me that people to whom slow expiry is a problem might
> go to the index of the manual and look up "expiring mail". There was
> already one index entry for "expiring IMAP mail", so I added this
> one, and in the process I added entries on some other pages that
> might be relevant to people looking up "expiring mail".

Thanks for your patch.  I'm sorry that we forgot to apply your patch
upto now.  I'd like to install it now in the stable branch.

I'm not sure about changing @cindex to @vindex, though:

,----
| @@ -2842,7 +2844,7 @@
|  See also @code{gnus-total-expirable-newsgroups}.
|  
|  @item expiry-wait
| -@cindex expiry-wait
| +@vindex expiry-wait
|  @vindex nnmail-expiry-wait-function
|  If the group parameter has an element that looks like
|  @code{(expiry-wait . 10)}, this value will override any
| @@ -2852,7 +2854,7 @@
|  symbols @code{never} or @code{immediate}.
|  
|  @item expiry-target
| -@cindex expiry-target
| +@vindex expiry-target
|  Where expired messages end up.  This parameter overrides
|  @code{nnmail-expiry-target}.
`----

I think we use @cindex for such symbols (as these are not variables).
Cf. M-x @cindex RET [^gmn][a-z]+-[a-z]+ RET (*) and M-x occur RET
@vindex [^gmn] RET:

,----
| 23 matches for "@cindex [^gmn][a-z]+-[a-z]+" in buffer: gnus.texi
|    1487:@cindex auto-save
|    2700:@cindex to-address		(*)
|    2723:@cindex to-list		(*)
|    2777:@cindex broken-reply-to	(*)
|    2786:@cindex to-group		(*)
|    2812:@cindex auto-expire		(*)
|    2821:@cindex total-expire		(*)
|    2859:@cindex admin-address		(*)
|    2926:@cindex ignored-charset	(*)
|    2934:@cindex posting-style
|    2953:@cindex post-method		(*)
|[...]
`----

I will install your patch without these @cindex -> @vindex changes.
Thanks for your contribution.

Bye, Reiner.

[1] For convenience, I include the patch once more:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: jouni-gnus-imap.texi.patch --]
[-- Type: text/x-patch, Size: 6798 bytes --]

Index: texi/gnus.texi
===================================================================
RCS file: /usr/local/cvsroot/gnus/texi/gnus.texi,v
retrieving revision 7.168
diff -u -r7.168 gnus.texi
--- texi/gnus.texi	5 Apr 2006 16:40:48 -0000	7.168
+++ texi/gnus.texi	6 Apr 2006 05:53:22 -0000
@@ -2825,6 +2825,7 @@
 
 @item auto-expire
 @cindex auto-expire
+@cindex expiring mail
 If the group parameter has an element that looks like @code{(auto-expire
 . t)}, all articles read will be marked as expirable.  For an
 alternative approach, @pxref{Expiring Mail}.
@@ -2833,6 +2834,7 @@
 
 @item total-expire
 @cindex total-expire
+@cindex expiring mail
 If the group parameter has an element that looks like
 @code{(total-expire . t)}, all read articles will be put through the
 expiry process, even if they are not marked as expirable.  Use with
@@ -2842,7 +2844,7 @@
 See also @code{gnus-total-expirable-newsgroups}.
 
 @item expiry-wait
-@cindex expiry-wait
+@vindex expiry-wait
 @vindex nnmail-expiry-wait-function
 If the group parameter has an element that looks like
 @code{(expiry-wait . 10)}, this value will override any
@@ -2852,7 +2854,7 @@
 symbols @code{never} or @code{immediate}.
 
 @item expiry-target
-@cindex expiry-target
+@vindex expiry-target
 Where expired messages end up.  This parameter overrides
 @code{nnmail-expiry-target}.
 
@@ -3462,6 +3464,7 @@
 @item C-c C-x
 @kindex C-c C-x (Group)
 @findex gnus-group-expire-articles
+@cindex expiring mail
 Run all expirable articles in the current group through the expiry
 process (if any) (@code{gnus-group-expire-articles}).  That is, delete
 all expirable articles in the group that have been around for a while.
@@ -3470,6 +3473,7 @@
 @item C-c C-M-x
 @kindex C-c C-M-x (Group)
 @findex gnus-group-expire-all-groups
+@cindex expiring mail
 Run all expirable articles in all groups through the expiry process
 (@code{gnus-group-expire-all-groups}).
 
@@ -3830,6 +3834,7 @@
 @item C-c C-x
 @kindex C-c C-x (Topic)
 @findex gnus-topic-expire-articles
+@cindex expiring mail
 Run all expirable articles in the current group or topic through the
 expiry process (if any)
 (@code{gnus-topic-expire-articles}).  (@pxref{Expiring Mail}).
@@ -10113,6 +10118,7 @@
 @item B e
 @kindex B e (Summary)
 @findex gnus-summary-expire-articles
+@cindex expiring mail
 Run all expirable articles in the current group through the expiry
 process (@code{gnus-summary-expire-articles}).  That is, delete all
 expirable articles in the group that have been around for a while.
@@ -10121,6 +10127,7 @@
 @item B C-M-e
 @kindex B C-M-e (Summary)
 @findex gnus-summary-expire-articles-now
+@cindex expiring mail
 Delete all the expirable articles in the group
 (@code{gnus-summary-expire-articles-now}).  This means that @strong{all}
 articles eligible for expiry in the current group will
@@ -14674,6 +14681,7 @@
 @node Expiring Mail
 @subsection Expiring Mail
 @cindex article expiry
+@cindex expiring mail
 
 Traditional mail readers have a tendency to remove mail articles when
 you mark them as read, in some way.  Gnus takes a fundamentally
@@ -16628,6 +16636,7 @@
 @item nnimap-expunge-search-string
 @cindex expunging
 @vindex nnimap-expunge-search-string
+@cindex expiring @acronym{IMAP} mail
 
 This variable contain the @acronym{IMAP} search command sent to server when
 searching for articles eligible for expiring.  The default is
@@ -16639,6 +16648,10 @@
 messages instead of the internal article date.  See section 6.4.4 of
 RFC 2060 for more information on valid strings.
 
+However, if @code{nnimap-search-uids-not-since-is-evil} 
+is true, the command is @code{"UID NOT SINCE @var{date}"},
+regardless of this variable.
+
 @item nnimap-authinfo-file
 @vindex nnimap-authinfo-file
 
@@ -16664,6 +16677,47 @@
 seem to need this under some circumstances; it was reported that
 Courier 1.7.1 did.
 
+@item nnimap-nov-is-evil
+@vindex nnimap-nov-is-evil
+@cindex Courier @acronym{IMAP} server
+@cindex @acronym{NOV}
+
+Never generate or use a local @acronym{NOV} database. Defaults to the
+value of @code{gnus-agent}.
+
+Using a @acronym{NOV} database usually makes header fetching much
+faster, but it uses the @code{UID SEARCH UID} command, which is very
+slow on some servers (notably some versions of Courier). Since the Gnus
+Agent caches the information in the @acronym{NOV} database without using
+the slow command, this variable defaults to true if the Agent is in use,
+and false otherwise.
+
+@item nnimap-search-uids-not-since-is-evil
+@vindex nnimap-search-uids-not-since-is-evil
+@cindex Courier @acronym{IMAP} server
+@cindex expiring @acronym{IMAP} mail
+
+Avoid the @code{UID SEARCH UID @var{message numbers} NOT SINCE
+@var{date}} command, which is slow on some @acronym{IMAP} servers
+(notably, some versions of Courier). Instead, use @code{UID SEARCH SINCE
+@var{date}} and prune the list of expirable articles within Gnus.
+
+When Gnus expires your mail (@pxref{Expiring Mail}), it starts with a
+list of expirable articles and asks the IMAP server questions like ``Of
+these articles, which ones are older than a week?'' While this seems
+like a perfectly reasonable question, some IMAP servers take a long time
+to answer it, since they seemingly go looking into every old article to
+see if it is one of the expirable ones. Curiously, the question ``Of
+@emph{all} articles, which ones are newer than a week?'' seems to be
+much faster to answer, so setting this variable causes Gnus to ask this
+question and figure out the answer to the real question itself.
+
+This problem can really sneak up on you: when you first configure Gnus,
+everything works fine, but once you accumulate a couple thousand
+messages, you start cursing Gnus for being so slow. On the other hand,
+if you get a lot of email within a week, setting this variable will
+cause a lot of network traffic between Gnus and the IMAP server.
+
 @end table
 
 @menu
@@ -16851,7 +16905,7 @@
 
 @node Expiring in IMAP
 @subsection Expiring in IMAP
-@cindex expiring imap mail
+@cindex expiring @acronym{IMAP} mail
 
 Even though @code{nnimap} is not a proper @code{nnmail} derived back
 end, it supports most features in regular expiring (@pxref{Expiring
@@ -16870,6 +16924,9 @@
 your server must support permanent storage of client specific flags on
 messages.  Most do, fortunately.
 
+If expiring @acronym{IMAP} mail seems very slow, try setting the server
+variable @code{nnimap-search-uids-not-since-is-evil}.
+
 @table @code
 
 @item nnmail-expiry-wait
@@ -19154,7 +19211,7 @@
 @findex gnus-agent-expire-group
 @cindex agent expiry
 @cindex Gnus agent expiry
-@cindex expiry
+@cindex expiry, in Gnus agent
 
 The Agent back end, @code{nnagent}, doesn't handle expiry.  Well, at
 least it doesn't handle it like other back ends.  Instead, there are

[-- Attachment #3: Type: text/plain, Size: 100 bytes --]

-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Slow expiry with nnimap+Courier [patch]

[Jouni K. Seppänen]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> Thanks for your patch.  I'm sorry that we forgot to apply your patch
> upto now.  I'd like to install it now in the stable branch.

Oops -- I had forgotten about it as well. There were some comments by
Simon Josefsson at <http://article.gmane.org/gmane.emacs.gnus.general/62896>;
I'll take another look at this once I have some more time.

> I'm not sure about changing @cindex to @vindex, though:

You're right, they shouldn't be changed.

-- 
Jouni K. Seppänen
http://www.iki.fi/jks

Re: Slow expiry with nnimap+Courier [patch]

[Jouni K. Seppänen]

[-- Attachment #1: Type: text/plain, Size: 128 bytes --]

jks@iki.fi (Jouni K. Seppänen) writes:

> I'll take another look at this once I have some more time.

Here's an updated patch.


[-- Attachment #2: nnimap-doc.patch --]
[-- Type: text/x-patch, Size: 8437 bytes --]

Index: lisp/ChangeLog
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/ChangeLog,v
retrieving revision 7.1442
diff -u -r7.1442 ChangeLog
--- lisp/ChangeLog	26 Dec 2006 20:08:05 -0000	7.1442
+++ lisp/ChangeLog	29 Dec 2006 14:04:35 -0000
@@ -1,3 +1,8 @@
+2006-12-29  Jouni K. Seppänen  <jks@iki.fi>
+
+	* nnimap.el (nnimap-expunge-search-string): Mention
+	nnimap-search-uids-not-since-is-evil in docstring.
+
 2006-12-26  Reiner Steib  <Reiner.Steib@gmx.de>
 
 	* gnus-msg.el (gnus-message-citation-mode): Fix autoload.
Index: lisp/nnimap.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnimap.el,v
retrieving revision 7.32
diff -u -r7.32 nnimap.el
--- lisp/nnimap.el	21 Mar 2006 19:58:50 -0000	7.32
+++ lisp/nnimap.el	29 Dec 2006 14:04:36 -0000
@@ -375,7 +375,10 @@
 One useful (and perhaps the only useful) value to change this to would
 be `UID %s NOT SENTSINCE %s' to make nnimap use the Date: header
 instead of the internal date of messages.  See section 6.4.4 of RFC
-2060 for more information on valid strings.")
+2060 for more information on valid strings.
+
+However, if `nnimap-search-uids-not-since-is-evil' is true, this
+variable has no effect since the search logic is reversed.")
 
 (defvoo nnimap-importantize-dormant t
   "If non-nil, mark \"dormant\" articles as \"ticked\" for other IMAP clients.
Index: texi/ChangeLog
===================================================================
RCS file: /usr/local/cvsroot/gnus/texi/ChangeLog,v
retrieving revision 7.295
diff -u -r7.295 ChangeLog
--- texi/ChangeLog	26 Dec 2006 17:10:01 -0000	7.295
+++ texi/ChangeLog	29 Dec 2006 14:04:36 -0000
@@ -1,3 +1,10 @@
+2006-12-29  Jouni K. Seppänen  <jks@iki.fi>
+
+	* gnus.texi: Add index terms related to expiry.
+	(IMAP): Document variables nnimap-nov-is-evil and
+	nnimap-search-uids-not-since-is-evil. Mention latter in connection
+	with nnimap-expunge-search-string and in the section on expiring.
+
 2006-12-26  Reiner Steib  <Reiner.Steib@gmx.de>
 
 	* gnus-news.texi: Add gnus-message-highlight-citation.
Index: texi/gnus.texi
===================================================================
RCS file: /usr/local/cvsroot/gnus/texi/gnus.texi,v
retrieving revision 7.201
diff -u -r7.201 gnus.texi
--- texi/gnus.texi	26 Dec 2006 17:10:01 -0000	7.201
+++ texi/gnus.texi	29 Dec 2006 14:04:42 -0000
@@ -2839,6 +2839,7 @@
 
 @item auto-expire
 @cindex auto-expire
+@cindex expiring mail
 If the group parameter has an element that looks like @code{(auto-expire
 . t)}, all articles read will be marked as expirable.  For an
 alternative approach, @pxref{Expiring Mail}.
@@ -2847,6 +2848,7 @@
 
 @item total-expire
 @cindex total-expire
+@cindex expiring mail
 If the group parameter has an element that looks like
 @code{(total-expire . t)}, all read articles will be put through the
 expiry process, even if they are not marked as expirable.  Use with
@@ -3477,6 +3479,7 @@
 @item C-c C-x
 @kindex C-c C-x (Group)
 @findex gnus-group-expire-articles
+@cindex expiring mail
 Run all expirable articles in the current group through the expiry
 process (if any) (@code{gnus-group-expire-articles}).  That is, delete
 all expirable articles in the group that have been around for a while.
@@ -3485,6 +3488,7 @@
 @item C-c C-M-x
 @kindex C-c C-M-x (Group)
 @findex gnus-group-expire-all-groups
+@cindex expiring mail
 Run all expirable articles in all groups through the expiry process
 (@code{gnus-group-expire-all-groups}).
 
@@ -3845,6 +3849,7 @@
 @item C-c C-x
 @kindex C-c C-x (Topic)
 @findex gnus-topic-expire-articles
+@cindex expiring mail
 Run all expirable articles in the current group or topic through the
 expiry process (if any)
 (@code{gnus-topic-expire-articles}).  (@pxref{Expiring Mail}).
@@ -10214,6 +10219,7 @@
 @item B e
 @kindex B e (Summary)
 @findex gnus-summary-expire-articles
+@cindex expiring mail
 Run all expirable articles in the current group through the expiry
 process (@code{gnus-summary-expire-articles}).  That is, delete all
 expirable articles in the group that have been around for a while.
@@ -10222,6 +10228,7 @@
 @item B C-M-e
 @kindex B C-M-e (Summary)
 @findex gnus-summary-expire-articles-now
+@cindex expiring mail
 Delete all the expirable articles in the group
 (@code{gnus-summary-expire-articles-now}).  This means that @strong{all}
 articles eligible for expiry in the current group will
@@ -14803,6 +14810,7 @@
 @node Expiring Mail
 @subsection Expiring Mail
 @cindex article expiry
+@cindex expiring mail
 
 Traditional mail readers have a tendency to remove mail articles when
 you mark them as read, in some way.  Gnus takes a fundamentally
@@ -16761,6 +16769,7 @@
 @item nnimap-expunge-search-string
 @cindex expunging
 @vindex nnimap-expunge-search-string
+@cindex expiring @acronym{IMAP} mail
 
 This variable contain the @acronym{IMAP} search command sent to server when
 searching for articles eligible for expiring.  The default is
@@ -16772,6 +16781,10 @@
 messages instead of the internal article date.  See section 6.4.4 of
 RFC 2060 for more information on valid strings.
 
+However, if @code{nnimap-search-uids-not-since-is-evil} 
+is true, this variable has no effect since the search logic
+is reversed, as described below.
+
 @item nnimap-authinfo-file
 @vindex nnimap-authinfo-file
 
@@ -16797,6 +16810,47 @@
 seem to need this under some circumstances; it was reported that
 Courier 1.7.1 did.
 
+@item nnimap-nov-is-evil
+@vindex nnimap-nov-is-evil
+@cindex Courier @acronym{IMAP} server
+@cindex @acronym{NOV}
+
+Never generate or use a local @acronym{NOV} database. Defaults to the
+value of @code{gnus-agent}.
+
+Using a @acronym{NOV} database usually makes header fetching much
+faster, but it uses the @code{UID SEARCH UID} command, which is very
+slow on some servers (notably some versions of Courier). Since the Gnus
+Agent caches the information in the @acronym{NOV} database without using
+the slow command, this variable defaults to true if the Agent is in use,
+and false otherwise.
+
+@item nnimap-search-uids-not-since-is-evil
+@vindex nnimap-search-uids-not-since-is-evil
+@cindex Courier @acronym{IMAP} server
+@cindex expiring @acronym{IMAP} mail
+
+Avoid the @code{UID SEARCH UID @var{message numbers} NOT SINCE
+@var{date}} command, which is slow on some @acronym{IMAP} servers
+(notably, some versions of Courier). Instead, use @code{UID SEARCH SINCE
+@var{date}} and prune the list of expirable articles within Gnus.
+
+When Gnus expires your mail (@pxref{Expiring Mail}), it starts with a
+list of expirable articles and asks the IMAP server questions like ``Of
+these articles, which ones are older than a week?'' While this seems
+like a perfectly reasonable question, some IMAP servers take a long time
+to answer it, since they seemingly go looking into every old article to
+see if it is one of the expirable ones. Curiously, the question ``Of
+@emph{all} articles, which ones are newer than a week?'' seems to be
+much faster to answer, so setting this variable causes Gnus to ask this
+question and figure out the answer to the real question itself.
+
+This problem can really sneak up on you: when you first configure Gnus,
+everything works fine, but once you accumulate a couple thousand
+messages, you start cursing Gnus for being so slow. On the other hand,
+if you get a lot of email within a week, setting this variable will
+cause a lot of network traffic between Gnus and the IMAP server.
+
 @end table
 
 @menu
@@ -16984,7 +17038,7 @@
 
 @node Expiring in IMAP
 @subsection Expiring in IMAP
-@cindex expiring imap mail
+@cindex expiring @acronym{IMAP} mail
 
 Even though @code{nnimap} is not a proper @code{nnmail} derived back
 end, it supports most features in regular expiring (@pxref{Expiring
@@ -17003,6 +17057,9 @@
 your server must support permanent storage of client specific flags on
 messages.  Most do, fortunately.
 
+If expiring @acronym{IMAP} mail seems very slow, try setting the server
+variable @code{nnimap-search-uids-not-since-is-evil}.
+
 @table @code
 
 @item nnmail-expiry-wait
@@ -19287,7 +19344,7 @@
 @findex gnus-agent-expire-group
 @cindex agent expiry
 @cindex Gnus agent expiry
-@cindex expiry
+@cindex expiry, in Gnus agent
 
 The Agent back end, @code{nnagent}, doesn't handle expiry.  Well, at
 least it doesn't handle it like other back ends.  Instead, there are

[-- Attachment #3: Type: text/plain, Size: 45 bytes --]


-- 
Jouni K. Seppänen
http://www.iki.fi/jks

Re: Slow expiry with nnimap+Courier [patch]

[Reiner Steib]

On Fri, Dec 29 2006, Jouni K. Seppänen wrote:

> Here's an updated patch.
[...]
> +2006-12-29  Jouni K. Seppänen  <jks@iki.fi>
> +
> +	* nnimap.el (nnimap-expunge-search-string): Mention
> +	nnimap-search-uids-not-since-is-evil in docstring.

Thanks.  I've installed this change (in v5-10; will be merge to the
trunk).

> +2006-12-29  Jouni K. Seppänen  <jks@iki.fi>

BTW, which spelling of your name in the ChangeLogs do you prefer?
"Jouni K Seppanen" or "Jouni K. Seppänen".  In the GNU copyright list
we have "Jouni Seppanen".

> +	* gnus.texi: Add index terms related to expiry.
> +	(IMAP): Document variables nnimap-nov-is-evil and
> +	nnimap-search-uids-not-since-is-evil. Mention latter in connection
> +	with nnimap-expunge-search-string and in the section on expiring.
[...]
> --- texi/gnus.texi	26 Dec 2006 17:10:01 -0000	7.201
> +++ texi/gnus.texi	29 Dec 2006 14:04:42 -0000

As your previous patch has already been applied in CVS, this patch
doesn't apply cleanly.  Could you please provide a patch against HEAD
(trunk or v5-10 branch).

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Slow expiry with nnimap+Courier [patch]

[Jouni K. Seppänen]

[-- Attachment #1: Type: text/plain, Size: 589 bytes --]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> BTW, which spelling of your name in the ChangeLogs do you prefer?
> "Jouni K Seppanen" or "Jouni K. Seppänen".  

It doesn't matter to me. The latter is more correct (in Finnish) but
the former tends to cause fewer encoding problems.

> As your previous patch has already been applied in CVS, this patch
> doesn't apply cleanly.  Could you please provide a patch against HEAD
> (trunk or v5-10 branch).

Attached. This patch also changes all occurrences of my name in the
ChangeLogs to be "Jouni K Seppanen" for the sake of uniformity.


[-- Attachment #2: nnimap-doc.patch --]
[-- Type: text/x-patch, Size: 3042 bytes --]

Index: lisp/ChangeLog
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/ChangeLog,v
retrieving revision 7.1446
diff -u -r7.1446 ChangeLog
--- lisp/ChangeLog	28 Dec 2006 19:58:19 -0000	7.1446
+++ lisp/ChangeLog	29 Dec 2006 16:17:22 -0000
@@ -1,3 +1,8 @@
+2006-12-29  Jouni K Seppanen  <jks@iki.fi>
+
+	* nnimap.el (nnimap-expunge-search-string): Mention
+	nnimap-search-uids-not-since-is-evil in docstring.
+
 2006-12-28  Reiner Steib  <Reiner.Steib@gmx.de>
 
 	* spam.el: Revert to make-obsolete-variable because
@@ -145,7 +150,7 @@
 	* gnus-sum.el (gnus-sort-threads, gnus-summary-limit-children):
 	Use `max' to avoid the value of `max-lisp-eval-depth' decreasing.
 
-2006-12-04  Jouni K. Seppänen  <jks@iki.fi>
+2006-12-04  Jouni K Seppanen  <jks@iki.fi>
 
 	* mm-url.el (mm-url-predefined-programs): Call curl with correct
 	options.
Index: lisp/nnimap.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnimap.el,v
retrieving revision 7.32
diff -u -r7.32 nnimap.el
--- lisp/nnimap.el	21 Mar 2006 19:58:50 -0000	7.32
+++ lisp/nnimap.el	29 Dec 2006 16:17:22 -0000
@@ -375,7 +375,10 @@
 One useful (and perhaps the only useful) value to change this to would
 be `UID %s NOT SENTSINCE %s' to make nnimap use the Date: header
 instead of the internal date of messages.  See section 6.4.4 of RFC
-2060 for more information on valid strings.")
+2060 for more information on valid strings.
+
+However, if `nnimap-search-uids-not-since-is-evil' is true, this
+variable has no effect since the search logic is reversed.")
 
 (defvoo nnimap-importantize-dormant t
   "If non-nil, mark \"dormant\" articles as \"ticked\" for other IMAP clients.
Index: texi/ChangeLog
===================================================================
RCS file: /usr/local/cvsroot/gnus/texi/ChangeLog,v
retrieving revision 7.297
diff -u -r7.297 ChangeLog
--- texi/ChangeLog	28 Dec 2006 14:59:17 -0000	7.297
+++ texi/ChangeLog	29 Dec 2006 16:17:23 -0000
@@ -1,3 +1,9 @@
+2006-12-29  Jouni K Seppanen  <jks@iki.fi>
+
+	* gnus.texi (IMAP): Fix incorrect explanation of
+	nnimap-search-uids-not-since-is-evil in documentation for
+	nnimap-expunge-search-string.
+
 2006-12-28  Reiner Steib  <Reiner.Steib@gmx.de>
 
 	* gnus.texi (Mail and Post): Index gnus-message-highlight-citation.
Index: texi/gnus.texi
===================================================================
RCS file: /usr/local/cvsroot/gnus/texi/gnus.texi,v
retrieving revision 7.203
diff -u -r7.203 gnus.texi
--- texi/gnus.texi	28 Dec 2006 14:59:17 -0000	7.203
+++ texi/gnus.texi	29 Dec 2006 16:17:28 -0000
@@ -16783,8 +16783,8 @@
 RFC 2060 for more information on valid strings.
 
 However, if @code{nnimap-search-uids-not-since-is-evil} 
-is true, the command is @code{"UID NOT SINCE @var{date}"},
-regardless of this variable.
+is true, this variable has no effect since the search logic
+is reversed, as described below.
 
 @item nnimap-authinfo-file
 @vindex nnimap-authinfo-file

[-- Attachment #3: Type: text/plain, Size: 45 bytes --]


-- 
Jouni K. Seppänen
http://www.iki.fi/jks

Re: Slow expiry with nnimap+Courier [patch]

[Reiner Steib]

On Fri, Dec 29 2006, Jouni K. Seppänen wrote:

> Reiner Steib <reinersteib+gmane@imap.cc> writes:
>
>> BTW, which spelling of your name in the ChangeLogs do you prefer?
>> "Jouni K Seppanen" or "Jouni K. Seppänen".  
>
> It doesn't matter to me. The latter is more correct (in Finnish) but
> the former tends to cause fewer encoding problems.

I think we should use the latter then, so I changed it to "Jouni
K. Seppänen".

>> Could you please provide a patch against HEAD (trunk or v5-10
>> branch).
>
> Attached. 

Installed (in v5-10).  Thanks.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Slow expiry with nnimap+Courier [patch]

[Ted Zlatanov]

On 26 Oct 2005, jks@iki.fi wrote:

> So it seems that Courier is doing something very suboptimal with the
> Gnus command. The attached patch (against 5.10.6) seems to have
> helped: now exiting the INBOX summary takes something like 2 seconds
> instead of the previous 30.

Is there any reason why the faster behavior should not be the default?
Is it slower on non-Courier servers?

Ted

Re: Slow expiry with nnimap+Courier [patch]

[Simon Josefsson]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> On 26 Oct 2005, jks@iki.fi wrote:
>
>> So it seems that Courier is doing something very suboptimal with the
>> Gnus command. The attached patch (against 5.10.6) seems to have
>> helped: now exiting the INBOX summary takes something like 2 seconds
>> instead of the previous 30.
>
> Is there any reason why the faster behavior should not be the default?
> Is it slower on non-Courier servers?

Yes, it may generate more network traffic, and may require servers to
search more messages for flag changes too.

Re: Slow expiry with nnimap+Courier [patch]

[Ted Zlatanov]

On  2 Nov 2005, jas@extundo.com wrote:

> "Ted Zlatanov" <tzz@lifelogs.com> writes:
>
>> On 26 Oct 2005, jks@iki.fi wrote:
>>
>>> So it seems that Courier is doing something very suboptimal with the
>>> Gnus command. The attached patch (against 5.10.6) seems to have
>>> helped: now exiting the INBOX summary takes something like 2 seconds
>>> instead of the previous 30.
>>
>> Is there any reason why the faster behavior should not be the default?
>> Is it slower on non-Courier servers?
>
> Yes, it may generate more network traffic, and may require servers to
> search more messages for flag changes too.

Can we set it on automatically if the server tells us it's a Courier
server?  Mine does in the first OK response, right after connecting.

Ted

Re: Slow expiry with nnimap+Courier [patch]

[Simon Josefsson]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> On  2 Nov 2005, jas@extundo.com wrote:
>
>> "Ted Zlatanov" <tzz@lifelogs.com> writes:
>>
>>> On 26 Oct 2005, jks@iki.fi wrote:
>>>
>>>> So it seems that Courier is doing something very suboptimal with the
>>>> Gnus command. The attached patch (against 5.10.6) seems to have
>>>> helped: now exiting the INBOX summary takes something like 2 seconds
>>>> instead of the previous 30.
>>>
>>> Is there any reason why the faster behavior should not be the default?
>>> Is it slower on non-Courier servers?
>>
>> Yes, it may generate more network traffic, and may require servers to
>> search more messages for flag changes too.
>
> Can we set it on automatically if the server tells us it's a Courier
> server?  Mine does in the first OK response, right after connecting.

Yes, I don't see how that would hurt anyone, except potentially
Courier users.  Some code to detect that it is the Courier server and
to set the variable would be needed, though.  Do Courier support the
ID extension?  There is support for ID in imap.el.

Re: Slow expiry with nnimap+Courier [patch]

[Jouni K Seppanen]

Simon Josefsson <jas@extundo.com> writes:

> "Ted Zlatanov" <tzz@lifelogs.com> writes:
>> Can we set it on automatically if the server tells us it's a Courier
>> server?  Mine does in the first OK response, right after connecting.
>
> Yes, I don't see how that would hurt anyone, except potentially
> Courier users.  Some code to detect that it is the Courier server and
> to set the variable would be needed, though.  Do Courier support the
> ID extension?  There is support for ID in imap.el.

At least the version I'm using doesn't support ID, so we'd have to
look at the first OK response. In my case it looks like this:

* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2004 Double Precision, Inc.  See COPYING for distribution information.

By the way, RFC 2971, which defines the ID extension, specifically
forbids using it to work around bugs or to optimize for a particular
client or server, with the rationale that the extension must not
"supplant IMAP's CAPABILITY mechanism with a ad-hoc approach". 
(So the standards-compliant way to do this would be to ask the Courier
developers to advertise a "X-SLOW-UID-SEARCH" capability so we can
detect it in Gnus?!)

-- 
Jouni K Seppänen

Re: Slow expiry with nnimap+Courier [patch]

[Bjørn Mork]

Jouni K Seppanen <jks@iki.fi> writes:

> (So the standards-compliant way to do this would be to ask the Courier
> developers to advertise a "X-SLOW-UID-SEARCH" capability so we can
> detect it in Gnus?!)

We should probably check whether this problem has been fixed in
Courier IMAP before doing so.

FWIW, I don't see this problem with any of the two servers I use,
although both are Courier IMAP with a 2004 copyright statement.  One
of them is the Debian stable package, based on Courier IMAP version
3.0.8.  Don't know the version of the other one.

I found at least one Changelog entry that might affect this behaviour
for 3.0.8 and later:

2004-09-01  Marc Horowitz <marc@mit.edu>

* search.c (fill_search_veryquick): Negative SEARCH UID does not
get short-circuited.


Could you verify whether this problem still exists in Courier IMAP
before implementing a workaround in Gnus?


Bjørn
-- 
You're a typical fascist person, totally repellent.

Re: Slow expiry with nnimap+Courier [patch]

[Jouni K Seppanen]

Bjørn Mork <bmork@dod.no> writes:

> I found at least one Changelog entry that might affect this behaviour
> for 3.0.8 and later:

The version I've had problems with seems to be 3.0.2, so the problem
could very well have been fixed in later versions. I think the
workaround would still be useful, since it is not always easy to
persuade sysadmins to update program versions, but perhaps it should
not be on by default.

-- 
Jouni K Seppänen