PATCH: Make IMAP mail sources more robust, was :Re: Working around IMAP server b0rk3n-ness

[Gaute B Strokkenes <biggaute@uwc.net>]

On 28 aug 2003, biggaute@uwc.net wrote:

> You're not alone, if it makes you feel any better.  I have a similar
> problem every morning when I try to get ~500 new messages.  Gnus
> constructs a line of ~2000 bytes; the process then hangs
> indefinitely.  I haven't quite tracked down what's going wrong, but
> the prime suspect is uw-imapd not being able to cope with long lines
> (we all know how robust c-client and friends are in that respect)
> with some sort of openssl / emacs sub-process brokenness as a more
> distant second.  Usually I just run fetchmail once every morning.

I have now determined that the problem definitely lies within openssl.
I can reproduce the problem by manually running openssl and logging
in; telnet on port 143 does not exhibit the problem.  openssl seems to
misbehave in an interesting way if you send it too much data at once;
this patch works around the problem by sending less data.

However, the openssl bug is not the only reason to apply the patch.
The UW imapd has an 8k long command buffer.  While this was not the
problem in this case, it would only take a couple of thousand messages
to reach that limit.  (For me, that's a couple of days' worth of
mail.)

> What's going on here is that remove is a list of message numbers
> that should be expunged.  The imap-range-to-message-set /
> gnus-compress-sequence combo is supposed to create a compact
> representation using the imap range syntax (i.e. "1:5" rather than
> "1,2,3,4,5").  However, gnus constructs remove by repeatedly pushing
> message numbers on to it--and since gnus fetches messages in
> sequence, the list of message numbers will be in reverse order,
> which defeats gnus-compress-sequence ("5,4,3,2,1").

The other part of the patch essentially partially undoes the changes
between revisions 6.32 and 6.33 of mail-source.el, made by Lars on 17
March.  As reported previously, the effect of the change is that
sometimes the wrong buffer is passed from mail-source-fetch to
imap-open.  This would occur when there were a " *imap source*" buffer
already present, such as that left over by interrupting
mail-source-fetch-imap with C-g.  (Because of the previous bug I have
done this quite frequently.)

2003-08-29  Gaute Strokkenes  <gs234@srcf.ucam.org>

	* mail-source.el (mail-source-fetch-imap): Pass correct buffer
	  to imap-open.  Reverse remove before calling
	  gnus-compress-sequence.

Index: mail-source.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/mail-source.el,v
retrieving revision 6.38
diff -u -r6.38 mail-source.el
--- mail-source.el	14 Jul 2003 08:03:13 -0000	6.38
+++ mail-source.el	29 Aug 2003 02:49:41 -0000
@@ -966,14 +966,13 @@
 (defun mail-source-fetch-imap (source callback)
   "Fetcher for imap sources."
   (mail-source-bind (imap source)
-    (let* ((from (format "%s:%s:%s" server user port))
-	   (found 0)
-	   (buffer-name " *imap source*")
-	   (buf (get-buffer-create (generate-new-buffer-name buffer-name)))
-	   (mail-source-string (format "imap:%s:%s" server mailbox))
-	   (imap-shell-program (or (list program) imap-shell-program))
-	   remove)
-      (if (and (imap-open server port stream authentication buffer-name)
+    (let ((from (format "%s:%s:%s" server user port))
+	  (found 0)
+	  (buf (generate-new-buffer " *imap source*"))
+	  (mail-source-string (format "imap:%s:%s" server mailbox))
+	  (imap-shell-program (or (list program) imap-shell-program))
+	  remove)
+      (if (and (imap-open server port stream authentication buf)
 	       (imap-authenticate
 		user (or (cdr (assoc from mail-source-password-cache))
 			 password) buf)
@@ -1006,6 +1005,7 @@
 	      (nnheader-ms-strip-cr))
 	    (incf found (mail-source-callback callback server))
 	    (when (and remove fetchflag)
+	      (setq remove (nreverse remove))
 	      (imap-message-flags-add
 	       (imap-range-to-message-set (gnus-compress-sequence remove))
 	       fetchflag nil buf))

-- 
Gaute Strokkenes                        http://www.srcf.ucam.org/~gs234/
Actually, what I'd like is a little toy spaceship!!