Working around IMAP server b0rk3n-ness

Working around IMAP server b0rk3n-ness

[David Z Maze]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

When using Gnus with an IMAP mail-source, it goes through, reads one
message at a time, sorts them away, and then tries to do a bulk delete
of all of the messages it's read.  This is all fine and within the
bounds of the IMAP spec; unfortunately, not all software is.  I
somewhat regularly wind up using an imtest binary with a 500-byte line
limit, and if I get around that, the local server has a 16K line
limit, which I do run into with enough mail.

It'd be nice if there was a switch or fallback mode or something such
that Gnus didn't always lose in exciting ways when it hit this.  The
"I'm away for a week so it's sure to die, slow and correct is better
than fast and broken" patch is:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 911 bytes --]

Index: mail-source.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/mail-source.el,v
retrieving revision 6.36
diff -u -r6.36 mail-source.el
--- mail-source.el      13 May 2003 22:58:29 -0000      6.36
+++ mail-source.el      27 Jun 2003 18:29:58 -0000
@@ -1005,9 +1005,10 @@
              (nnheader-ms-strip-cr))
            (incf found (mail-source-callback callback server))
            (when (and remove fetchflag)
-             (imap-message-flags-add
-              (imap-range-to-message-set (gnus-compress-sequence remove))
-              fetchflag nil buf))
+             (dolist (uid remove)
+               (imap-message-flags-add
+                (imap-list-to-message-set (list uid))
+                fetchflag nil buf)))
            (if dontexpunge
                (imap-mailbox-unselect buf)
              (imap-mailbox-close nil buf))

[-- Attachment #3: Type: text/plain, Size: 216 bytes --]


...but I don't actually want that all the time.

-- 
David Maze             dmaze@mit.edu          http://www.mit.edu/~dmaze/
"Theoretical politics is interesting.  Politicking should be illegal."
	-- Abra Mitchell

Re: Working around IMAP server b0rk3n-ness

[Gaute B Strokkenes]

On 27 jun 2003, dmaze@mit.edu wrote:

> When using Gnus with an IMAP mail-source, it goes through, reads one
> message at a time, sorts them away, and then tries to do a bulk
> delete of all of the messages it's read.  This is all fine and
> within the bounds of the IMAP spec; unfortunately, not all software
> is.  I somewhat regularly wind up using an imtest binary with a
> 500-byte line limit, and if I get around that, the local server has
> a 16K line limit, which I do run into with enough mail.
>
> It'd be nice if there was a switch or fallback mode or something
> such that Gnus didn't always lose in exciting ways when it hit this.

You're not alone, if it makes you feel any better.  I have a similar
problem every morning when I try to get ~500 new messages.  Gnus
constructs a line of ~2000 bytes; the process then hangs indefinitely.
I haven't quite tracked down what's going wrong, but the prime suspect
is uw-imapd not being able to cope with long lines (we all know how
robust c-client and friends are in that respect) with some sort of
openssl / emacs sub-process brokenness as a more distant second.
Usually I just run fetchmail once every morning.

> The "I'm away for a week so it's sure to die, slow and correct is
> better than fast and broken" patch is:
> 
> Index: mail-source.el
> ===================================================================
> RCS file: /usr/local/cvsroot/gnus/lisp/mail-source.el,v
> retrieving revision 6.36
> diff -u -r6.36 mail-source.el
> --- mail-source.el      13 May 2003 22:58:29 -0000      6.36
> +++ mail-source.el      27 Jun 2003 18:29:58 -0000
> @@ -1005,9 +1005,10 @@
> (nnheader-ms-strip-cr))
> (incf found (mail-source-callback callback server))
> (when (and remove fetchflag)
> -             (imap-message-flags-add
> -              (imap-range-to-message-set (gnus-compress-sequence remove))
> -              fetchflag nil buf))
> +             (dolist (uid remove)
> +               (imap-message-flags-add
> +                (imap-list-to-message-set (list uid))
> +                fetchflag nil buf)))
> (if dontexpunge
> (imap-mailbox-unselect buf)
> (imap-mailbox-close nil buf))
>
> ...but I don't actually want that all the time.

This may help:

*** mail-source.el.~6.38.~	1970-01-01 01:00:01.000000000 +0100
--- mail-source.el	2003-08-28 12:42:48.000000000 +0100
***************
*** 1006,1011 ****
--- 1005,1011 ----
  	      (nnheader-ms-strip-cr))
  	    (incf found (mail-source-callback callback server))
  	    (when (and remove fetchflag)
+ 	      (setq remove (nreverse remove))
  	      (imap-message-flags-add
  	       (imap-range-to-message-set (gnus-compress-sequence remove))
  	       fetchflag nil buf))

What's going on here is that remove is a list of message numbers that
should be expunged.  The imap-range-to-message-set /
gnus-compress-sequence combo is supposed to create a compact
representation using the imap range syntax (i.e. "1:5" rather than
"1,2,3,4,5").  However, gnus constructs remove by repeatedly pushing
message numbers on to it--and since gnus fetches messages in sequence,
the list of message numbers will be in reverse order, which defeats
gnus-compress-sequence ("5,4,3,2,1").

This won't help you much if the message numbers are not continuous,
but it's a start.

-- 
Gaute Strokkenes                        http://www.srcf.ucam.org/~gs234/
I'm DESPONDENT...  I hope there's something DEEP-FRIED under this
 miniature DOMED STADIUM...

PATCH: Make IMAP mail sources more robust, was :Re: Working around IMAP server b0rk3n-ness

[Gaute B Strokkenes]

On 28 aug 2003, biggaute@uwc.net wrote:

> You're not alone, if it makes you feel any better.  I have a similar
> problem every morning when I try to get ~500 new messages.  Gnus
> constructs a line of ~2000 bytes; the process then hangs
> indefinitely.  I haven't quite tracked down what's going wrong, but
> the prime suspect is uw-imapd not being able to cope with long lines
> (we all know how robust c-client and friends are in that respect)
> with some sort of openssl / emacs sub-process brokenness as a more
> distant second.  Usually I just run fetchmail once every morning.

I have now determined that the problem definitely lies within openssl.
I can reproduce the problem by manually running openssl and logging
in; telnet on port 143 does not exhibit the problem.  openssl seems to
misbehave in an interesting way if you send it too much data at once;
this patch works around the problem by sending less data.

However, the openssl bug is not the only reason to apply the patch.
The UW imapd has an 8k long command buffer.  While this was not the
problem in this case, it would only take a couple of thousand messages
to reach that limit.  (For me, that's a couple of days' worth of
mail.)

> What's going on here is that remove is a list of message numbers
> that should be expunged.  The imap-range-to-message-set /
> gnus-compress-sequence combo is supposed to create a compact
> representation using the imap range syntax (i.e. "1:5" rather than
> "1,2,3,4,5").  However, gnus constructs remove by repeatedly pushing
> message numbers on to it--and since gnus fetches messages in
> sequence, the list of message numbers will be in reverse order,
> which defeats gnus-compress-sequence ("5,4,3,2,1").

The other part of the patch essentially partially undoes the changes
between revisions 6.32 and 6.33 of mail-source.el, made by Lars on 17
March.  As reported previously, the effect of the change is that
sometimes the wrong buffer is passed from mail-source-fetch to
imap-open.  This would occur when there were a " *imap source*" buffer
already present, such as that left over by interrupting
mail-source-fetch-imap with C-g.  (Because of the previous bug I have
done this quite frequently.)

2003-08-29  Gaute Strokkenes  <gs234@srcf.ucam.org>

	* mail-source.el (mail-source-fetch-imap): Pass correct buffer
	  to imap-open.  Reverse remove before calling
	  gnus-compress-sequence.

Index: mail-source.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/mail-source.el,v
retrieving revision 6.38
diff -u -r6.38 mail-source.el
--- mail-source.el	14 Jul 2003 08:03:13 -0000	6.38
+++ mail-source.el	29 Aug 2003 02:49:41 -0000
@@ -966,14 +966,13 @@
 (defun mail-source-fetch-imap (source callback)
   "Fetcher for imap sources."
   (mail-source-bind (imap source)
-    (let* ((from (format "%s:%s:%s" server user port))
-	   (found 0)
-	   (buffer-name " *imap source*")
-	   (buf (get-buffer-create (generate-new-buffer-name buffer-name)))
-	   (mail-source-string (format "imap:%s:%s" server mailbox))
-	   (imap-shell-program (or (list program) imap-shell-program))
-	   remove)
-      (if (and (imap-open server port stream authentication buffer-name)
+    (let ((from (format "%s:%s:%s" server user port))
+	  (found 0)
+	  (buf (generate-new-buffer " *imap source*"))
+	  (mail-source-string (format "imap:%s:%s" server mailbox))
+	  (imap-shell-program (or (list program) imap-shell-program))
+	  remove)
+      (if (and (imap-open server port stream authentication buf)
 	       (imap-authenticate
 		user (or (cdr (assoc from mail-source-password-cache))
 			 password) buf)
@@ -1006,6 +1005,7 @@
 	      (nnheader-ms-strip-cr))
 	    (incf found (mail-source-callback callback server))
 	    (when (and remove fetchflag)
+	      (setq remove (nreverse remove))
 	      (imap-message-flags-add
 	       (imap-range-to-message-set (gnus-compress-sequence remove))
 	       fetchflag nil buf))

-- 
Gaute Strokkenes                        http://www.srcf.ucam.org/~gs234/
Actually, what I'd like is a little toy spaceship!!

Re: PATCH: Make IMAP mail sources more robust, was :Re: Working

[Gaute B Strokkenes]

On 29 aug 2003, biggaute@uwc.net wrote:
> On 28 aug 2003, biggaute@uwc.net wrote:
>
>> You're not alone, if it makes you feel any better.  I have a
>> similar problem every morning when I try to get ~500 new messages.
>> Gnus constructs a line of ~2000 bytes; the process then hangs
>> indefinitely.  I haven't quite tracked down what's going wrong, but
>> the prime suspect is uw-imapd not being able to cope with long
>> lines (we all know how robust c-client and friends are in that
>> respect) with some sort of openssl / emacs sub-process brokenness
>> as a more distant second.  Usually I just run fetchmail once every
>> morning.
>
> I have now determined that the problem definitely lies within
> openssl.  I can reproduce the problem by manually running openssl
> and logging in; telnet on port 143 does not exhibit the problem.
> openssl seems to misbehave in an interesting way if you send it too
> much data at once; this patch works around the problem by sending
> less data.

Interested parties can find my detailed analysis of the problem at:

  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=207840

-- 
Gaute Strokkenes                        http://www.srcf.ucam.org/~gs234/
I was in a HOT TUB!  I was NORMAL!  I was ITALIAN!!  I enjoyed th'
 EARTHQUAKE!

Re: PATCH: Make IMAP mail sources more robust, was :Re: Working

[Simon Josefsson]

Gaute B Strokkenes <biggaute@uwc.net> writes:

> Interested parties can find my detailed analysis of the problem at:
>
>   http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=207840

Perhaps you could try GNUTLS instead?  It might not have the fixed
size buffers that OpenSSL apparently has.  And you can forget about
installing ssl.el when you upgrade Emacs.

> 2003-08-29  Gaute Strokkenes  <gs234@srcf.ucam.org>
>
>	* mail-source.el (mail-source-fetch-imap): Pass correct buffer
>	  to imap-open.  Reverse remove before calling
>	  gnus-compress-sequence.

Thanks, applied.

Re: PATCH: Make IMAP mail sources more robust, was :Re: Working

[Gaute B Strokkenes]

On 30 aug 2003, jas@extundo.com wrote:
> Gaute B Strokkenes <biggaute@uwc.net> writes:
>
>> Interested parties can find my detailed analysis of the problem at:
>>
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=207840
>
> Perhaps you could try GNUTLS instead?  It might not have the fixed
> size buffers that OpenSSL apparently has.

Thanks, that's a good idea.  I've switched now.  (I actually used
GnuTLS originally, but gnutls-cli tripped over one of the previous
emacs bugs that we fixed more frequenly than openssl, so I switched.)

However, I just repeated the process in the bug report referenced
above, but with gnutls-cli rather than openssl s_client.  Guess what?
I get the exact same failure mode.  I suspect the problem lies with
uw-imapd.  (I should have known...)

> And you can forget about installing ssl.el when you upgrade Emacs.

Hmm?

gs234@belldandy:~$ locate ssl.el
/home/gs234/gnus/contrib/ssl.el
gs234@belldandy:~$ 

A cursory examination indicates that none of the ssl support in Gnus
is dependent on that file.  (load-library "ssl") throws an error for
me.

-- 
Gaute Strokkenes                        http://www.srcf.ucam.org/~gs234/
Your CHEEKS sit like twin NECTARINES above a MOUTH that knows no BOUNDS --

Re: PATCH: Make IMAP mail sources more robust, was :Re: Working

[Simon Josefsson]

Gaute B Strokkenes <biggaute@uwc.net> writes:


>> And you can forget about installing ssl.el when you upgrade Emacs.
>
> Hmm?
>
> gs234@belldandy:~$ locate ssl.el
> /home/gs234/gnus/contrib/ssl.el
> gs234@belldandy:~$ 
>
> A cursory examination indicates that none of the ssl support in Gnus
> is dependent on that file.  (load-library "ssl") throws an error for
> me.

Oops, I forgot ssl.el isn't necessary any more.

Re: PATCH: Make IMAP mail sources more robust, was :Re: Working

[Gaute B Strokkenes]

On 31 aug 2003, jas@extundo.com wrote:
> Gaute B Strokkenes <biggaute@uwc.net> writes:
> 
>>> And you can forget about installing ssl.el when you upgrade Emacs.
>>
>> Hmm?
>>
>> gs234@belldandy:~$ locate ssl.el
>> /home/gs234/gnus/contrib/ssl.el
>> gs234@belldandy:~$ 
>>
>> A cursory examination indicates that none of the ssl support in
>> Gnus is dependent on that file.  (load-library "ssl") throws an
>> error for me.
>
> Oops, I forgot ssl.el isn't necessary any more.

Good, so I'm not on drugs then. ^_^  Perhaps the file should be
removed from Gnus?

-- 
Gaute Strokkenes                        http://www.srcf.ucam.org/~gs234/
Was my SOY LOAF left out in th'RAIN?  It tastes REAL GOOD!!

Re: PATCH: Make IMAP mail sources more robust, was :Re: Working

[Simon Josefsson]

Gaute B Strokkenes <biggaute@uwc.net> writes:

>> Oops, I forgot ssl.el isn't necessary any more.
>
> Good, so I'm not on drugs then. ^_^  Perhaps the file should be
> removed from Gnus?

Perhaps, but people are used to finding it in contrib/, so maybe we
should wait a few years first.  Older Gnus still needs it.