nnimap hangs on connect plus Garbage:

nnimap hangs on connect plus Garbage:

[Mats Lidell]

Hi,

I have been living with a problem for some time now. I once posted to
the list, got a reply from Simon with some hints on what to look for,
thanks, but other things got higher priority. So I didn't do
anything. Now I'm ready. ;-) Let's start again. What is the problem
you might ask?

When I start gnus it won't connect to the IMAP-server. It times out
after a while. Trying again gives the same result.

Fortunately there is a work around. When gnus waits for the connection
stop it with a few ^G, turn on nnimap-debug, or have it on from the
start, make *nnimap-debug* visible in a window and restart gnus. (It
is not enough to just put on nnimap-debug. The buffer must be
visible.) Then I'll get the connection. This seems like a rather odd
race situation when Emacs seems to be to fast and needs to be slowed
down. But then again there might just be some other problem.

The other problem is that I get an occasional "Garbage:" written in
the message log when Gnus is splitting the messages into folders. It
is just to start gnus again and the splitting continues, as it seems,
from where it stopped. I'm not aware of any problem, like lost mail,
due to this problem.

I'm running: 
    emacs-version = "21.2  (beta43) \"Terspichore\" XEmacs Lucid"
    gnus-version = Oort Gnus v0.01
    imap-server = UW 2000.283
    nnimap-version = nnimap 0.131

But the actual versions doesn't seem to matter that much. I have seen
this for quite some versions of gnus, xemacs and I think also version
of UW-IMAP-servers.

Any clue? 

Yours
-- 
%% Mats

Re: nnimap hangs on connect plus Garbage:

[NAGY Andras]

Mats Lidell <Mats.Lidell@contactor.se> writes:

> The other problem is that I get an occasional "Garbage:" written in
> the message log when Gnus is splitting the messages into folders. It
> is just to start gnus again and the splitting continues, as it seems,
> from where it stopped. I'm not aware of any problem, like lost mail,
> due to this problem.

Are you using SSL?  The openssl program (which is being used for ssl
connections) interprets some characters as escape sequences and makes
gnus print Garbage: under certain circumstances, and gnus does not
make use of the switch which disables this behavior.  I've posted a
patch here [1], but it seems it hasn't been applied.

[1] news://sunsite.auc.dk/87snn5ho8t.fsf@lovi.inf.elte.hu

Re: nnimap hangs on connect plus Garbage:

[Mats Lidell]

>>>>> Nagy wrote:

Nagy> Are you using SSL?  

Well, yes but through a local port.

Nagy> The openssl program (which is being used for ssl connections)
Nagy> interprets some characters as escape sequences and makes gnus
Nagy> print Garbage: under certain circumstances, and gnus does not
Nagy> make use of the switch which disables this behavior.  I've
Nagy> posted a patch here [1], but it seems it hasn't been applied.

I'm using ssh to redirect the imap port to a local port and ding
connects to the local port just as if it was a normal network
connection. On the other hand I guess ssh, OpenSSH 2.1.1 that is, is
using openssl. Could this be it then?

Yours
-- 
%% Mats

Re: nnimap hangs on connect plus Garbage:

[NAGY Andras]

Mats Lidell <Mats.Lidell@contactor.se> writes:

> I'm using ssh to redirect the imap port to a local port and ding
> connects to the local port just as if it was a normal network
> connection. On the other hand I guess ssh, OpenSSH 2.1.1 that is, is
> using openssl. Could this be it then?

No, this can be a problem only if the actual openssl program is used.


Andras

Re: nnimap hangs on connect plus Garbage:

[Simon Josefsson]

Mats Lidell <Mats.Lidell@contactor.se> writes:

> When I start gnus it won't connect to the IMAP-server. It times out
> after a while. Trying again gives the same result.
> 
> Fortunately there is a work around. When gnus waits for the connection
> stop it with a few ^G, turn on nnimap-debug, or have it on from the
> start, make *nnimap-debug* visible in a window and restart gnus. (It
> is not enough to just put on nnimap-debug. The buffer must be
> visible.) Then I'll get the connection. This seems like a rather odd
> race situation when Emacs seems to be to fast and needs to be slowed
> down. But then again there might just be some other problem.

I've seen similar things (any Emacs 20 on AIX 4.[23]), and my
workaround has been to put

(debug-on-entry 'imap-kerberos4-open)

in my .gnus, and press 'c' directly when I see the debug window.  I
suspect the problem might apply to all external connection methods
(SSL, STARTTLS, GSSAPI, etc), but I've only seen it under AIX though.

What OS are you using?

Also, my experience is that if I don't use the workaround, Emacs locks
up completely (any number of C-g doesn't help).  I think it's a race
condition in Emacs process I/O.  I'm hoping that some emacs I/O guru
discover the same problem and fix it. :)

Your workaround probably causes enough delay after opening the
connection until something is read/written to make things work, just
as my workaround.  Perhaps imap.el could `sit-for' a second or so
after opening external processes.  Hm.

> The other problem is that I get an occasional "Garbage:" written in
> the message log when Gnus is splitting the messages into folders. It
> is just to start gnus again and the splitting continues, as it seems,
> from where it stopped. I'm not aware of any problem, like lost mail,
> due to this problem.

This may also be due using a external program, sometimes control
characters and stuff is inserted.  In these cases, please enable
`imap-log' and search it for whatever is printed in the "Garbage: "
message.  Things to look out for may also be long lines, sometimes
external programs have fixed size line buffers which gets overwritten
corrupting the IMAP commands.

If you don't use a external access method, I've no idea what would
cause this.

Re: nnimap hangs on connect plus Garbage:

[Mats Lidell]

>>>>> Simon wrote:

Simon> What OS are you using?

RedHat 7.0 what now that has to do with it.

Simon> Also, my experience is that if I don't use the workaround,
Simon> Emacs locks up completely (any number of C-g doesn't help).  I
Simon> think it's a race condition in Emacs process I/O.  I'm hoping
Simon> that some emacs I/O guru discover the same problem and fix
Simon> it. :)

Hmm... So what you are saying really is that this might be hard for us
fix.

Simon> Your workaround probably causes enough delay after opening the
Simon> connection until something is read/written to make things work,
Simon> just as my workaround.  Perhaps imap.el could `sit-for' a
Simon> second or so after opening external processes.  Hm.

I'd definitely would prefere that. Simple enough even for me to
experiment with. ;-)

Yours
-- 
%% Mats

Re: nnimap hangs on connect plus Garbage:

[Simon Josefsson]

Mats Lidell <Mats.Lidell@contactor.se> writes:

> Simon> Also, my experience is that if I don't use the workaround,
> Simon> Emacs locks up completely (any number of C-g doesn't help).  I
> Simon> think it's a race condition in Emacs process I/O.  I'm hoping
> Simon> that some emacs I/O guru discover the same problem and fix
> Simon> it. :)
> 
> Hmm... So what you are saying really is that this might be hard for us
> fix.

Yes.  Now that I try it, I'm not even able to reproduce it.

> Simon> Your workaround probably causes enough delay after opening the
> Simon> connection until something is read/written to make things work,
> Simon> just as my workaround.  Perhaps imap.el could `sit-for' a
> Simon> second or so after opening external processes.  Hm.
> 
> I'd definitely would prefere that. Simple enough even for me to
> experiment with. ;-)

Does the following have any effect?

--- imap.el.~6.3.~	Wed Dec 20 18:14:16 2000
+++ imap.el	Tue Feb  6 00:20:04 2001
@@ -857,13 +857,16 @@
 	  imap-current-message nil
 	  imap-state 'initial
 	  imap-process (condition-case ()
+			   (sit-for 1)
 			   (funcall (nth 2 (assq imap-stream 
 						 imap-stream-alist))
 				    "imap" buffer imap-server imap-port)
+			   (sit-for 1)
 			 ((error quit) nil)))
     (when imap-process
       (set-process-filter imap-process 'imap-arrival-filter)
       (set-process-sentinel imap-process 'imap-sentinel)
+      (sit-for 1)
       (while (and (eq imap-state 'initial)
 		  (memq (process-status imap-process) '(open run)))
 	(message "Waiting for response from %s..." imap-server)

Re: nnimap hangs on connect plus Garbage:

[Mats Lidell]

>>>>> Simon wrote:

Simon> Does the following have any effect?

Simon> --- imap.el.~6.3.~	Wed Dec 20 18:14:16 2000
Simon> +++ imap.el	Tue Feb  6 00:20:04 2001
Simon> @@ -857,13 +857,16 @@
Simon>  	  imap-current-message nil
Simon>  	  imap-state 'initial
Simon>  	  imap-process (condition-case ()
Simon> +			   (sit-for 1)
Simon>  			   (funcall (nth 2 (assq imap-stream 
Simon>  						 imap-stream-alist))
Simon>  				    "imap" buffer imap-server imap-port)
Simon> +			   (sit-for 1)
Simon>  			 ((error quit) nil)))
Simon>      (when imap-process
Simon>        (set-process-filter imap-process 'imap-arrival-filter)
Simon>        (set-process-sentinel imap-process 'imap-sentinel)
Simon> +      (sit-for 1)
Simon>        (while (and (eq imap-state 'initial)
Simon>  		  (memq (process-status imap-process) '(open run)))
Simon>  	(message "Waiting for response from %s..." imap-server)

Doesn't work. Is condition-case supposed to work that way?

Anyway. I looked down the funcall way and found that it hangs in
imap-network-open: (My messages) (Good old debug by print you know)

      (while (and (memq (process-status process) '(open run))
		  (goto-char (point-min))
		  (message "imap-network-open 3")
		  (not (imap-parse-greeting)))
	(accept-process-output process 1)
	(message "imap-network-open 4")
	(sit-for 1))

Endless sequence of "imap-network-open 3" and "imap-network-open 4"
seen in minibuffer.

When I break the loop with C-g the "*nnimap host" buffer contains a
valid greeting line. imap-parse-greeting returns t anyway.

Does this suggest that the "*nnimap host" buffer isn't updated so that
looking-at (in imap-parse-greeting) can see it? 

Removing the delay in accept-process-output stops Emacs completely
which suggests to me that all possible output was received but
imap-parse-greeting didn't see it.

Suggestions are welcome.

Yours
-- 
%% Mats

Re: nnimap hangs on connect plus Garbage:

[Kai Großjohann]

On 06 Feb 2001, Mats Lidell wrote:

>       (while (and (memq (process-status process) '(open run))
> 		  (goto-char (point-min))
> 		  (message "imap-network-open 3")
> 		  (not (imap-parse-greeting)))
> 	(accept-process-output process 1)
> 	(message "imap-network-open 4")
> 	(sit-for 1))
> 
> Endless sequence of "imap-network-open 3" and "imap-network-open 4"
> seen in minibuffer.

Hm.  Hmmm...  When you interrupt it, can you then go to point-min and
issue M-: (imap-parse-greeting) RET and does it correctly parse the
greeting?

Maybe point-min is off due to narrowing?  You might want to print the
value of (point-min) in the "imap-network-open 3" message.

What happens when you single-step through this code using debug or
edebug?  Probably it works then, since this is timing related...

If your suggestion of the buffer containing the wrong stuff is true,
maybe it helps to say (redraw-display).  But I doubt this.

kai
-- 
Be indiscrete.  Do it continuously.

Re: nnimap hangs on connect plus Garbage:

[Mats Lidell]

>>>>> Kai wrote:

Kai> Hm.  Hmmm...  When you interrupt it, can you then go to point-min
Kai> and issue M-: (imap-parse-greeting) RET and does it correctly
Kai> parse the greeting?

Yes!

Kai> Maybe point-min is off due to narrowing?  You might want to print
Kai> the value of (point-min) in the "imap-network-open 3" message.

It seems that it is not looking-at the right buffer! I added a message
to imap-parse-greeting that writes the buffer name and to my surprise
it was " *nnimap <host>" the FIRST time. Forever after it is
looking-at the "*Group*" buffer.

So somehow the current buffer is changed from " *nnimap <host>* to
"*Group" during establishing the connection.

Kai> If your suggestion of the buffer containing the wrong stuff is
Kai> true, maybe it helps to say (redraw-display).  But I doubt this.

I was wrong. The contents is right. It is the buffer that is wrong.

I tried to force the right buffer by doing a set-buffer before calling
imap-parse-greeting. This makes the parsing right but I will then get
the error

        Signaling: (error "Buffer *Group* has no process")
          process-status(nil)
          imap-open-1(" *nnimap* <host>")
          ...

Wrong buffer again. Any ideas?

Yours
-- 
%% Mats