Exchange, nnimap, and lock-ups

Exchange, nnimap, and lock-ups

[Michael Welsh Duggan]

I run a gnus demon, with the following parameters:

(gnus-demon-add-handler 'gnus-demon-scan-news 10 1)

Most of the time this works, but every now and then it will lock up
mid-fetch, and I will have to C-g to get things to continue.  Also,
messages in my INBOX will get split appropriately, but will not be
deleted or expunged out of the INBOX after they have been copied.

Everything seemed to work well before, and I believe I have traced the
problem to commit 8943cf1ad32a11652fe72ef4d34025f9618176ec:

    (nnimap-wait-for-response): Wait for results in a more secure
    manner.

I have tried to debug this, but the problems never happen when I am
stepping through, and toggle-debug-on-quit doesn't appear to work in the
lock-up case (it continues processing instead).  I would appreciate any
help in debugging this problem.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Exchange, nnimap, and lock-ups

[Tassilo Horn]

Michael Welsh Duggan <md5i@md5i.com> writes:

Hi Michael,

> I run a gnus demon, with the following parameters:
>
> (gnus-demon-add-handler 'gnus-demon-scan-news 10 1)
>
> Most of the time this works, but every now and then it will lock up
> mid-fetch, and I will have to C-g to get things to continue.  Also,
> messages in my INBOX will get split appropriately, but will not be
> deleted or expunged out of the INBOX after they have been copied.

Have a look at my message with Message-ID
<87ipww6pzc.fsf@member.fsf.org>.  I think we suffer the same problem.
Can you verify my observation that in the *imap log* buffer there're
always 2 NOOP ops when the hang occurs?

Bye,
Tassilo
-- 
Sent from my Emacs

Re: Exchange, nnimap, and lock-ups

[Michael Welsh Duggan]

Tassilo Horn <tassilo@member.fsf.org> writes:

> Michael Welsh Duggan <md5i@md5i.com> writes:
>
> Hi Michael,
>
>> I run a gnus demon, with the following parameters:
>>
>> (gnus-demon-add-handler 'gnus-demon-scan-news 10 1)
>>
>> Most of the time this works, but every now and then it will lock up
>> mid-fetch, and I will have to C-g to get things to continue.  Also,
>> messages in my INBOX will get split appropriately, but will not be
>> deleted or expunged out of the INBOX after they have been copied.
>
> Have a look at my message with Message-ID
> <87ipww6pzc.fsf@member.fsf.org>.  I think we suffer the same problem.
> Can you verify my observation that in the *imap log* buffer there're
> always 2 NOOP ops when the hang occurs?

I can verify this:

14:29:09 958 EXAMINE "shared.teams.netsa.logs.svn"
14:29:09 959 UID FETCH 14536:* FLAGS
14:29:09 960 NOOP
14:29:09 961 NOOP

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Exchange, nnimap, and lock-ups

[Michael Welsh Duggan]

Michael Welsh Duggan <md5i@md5i.com> writes:

> Tassilo Horn <tassilo@member.fsf.org> writes:
>
>> Michael Welsh Duggan <md5i@md5i.com> writes:
>>
>> Hi Michael,
>>
>>> I run a gnus demon, with the following parameters:
>>>
>>> (gnus-demon-add-handler 'gnus-demon-scan-news 10 1)
>>>
>>> Most of the time this works, but every now and then it will lock up
>>> mid-fetch, and I will have to C-g to get things to continue.  Also,
>>> messages in my INBOX will get split appropriately, but will not be
>>> deleted or expunged out of the INBOX after they have been copied.
>>
>> Have a look at my message with Message-ID
>> <87ipww6pzc.fsf@member.fsf.org>.  I think we suffer the same problem.
>> Can you verify my observation that in the *imap log* buffer there're
>> always 2 NOOP ops when the hang occurs?
>
> I can verify this:
>
> 14:29:09 958 EXAMINE "shared.teams.netsa.logs.svn"
> 14:29:09 959 UID FETCH 14536:* FLAGS
> 14:29:09 960 NOOP
> 14:29:09 961 NOOP

I should note that I am connected to two separate IMAP servers, an
Exchange server and a Cyrus server.  In this particular case, it makes
it all the way through the exchange server, and it locks up _after_ the
last group of the Cyrus server.  I.e., it did make it through a full
scan of both servers before the hanging NOOPs.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Exchange, nnimap, and lock-ups

[Tassilo Horn]

Michael Welsh Duggan <md5i@md5i.com> writes:

Hi Michael,

>> I can verify this:
>>
>> 14:29:09 958 EXAMINE "shared.teams.netsa.logs.svn"
>> 14:29:09 959 UID FETCH 14536:* FLAGS
>> 14:29:09 960 NOOP
>> 14:29:09 961 NOOP
>
> I should note that I am connected to two separate IMAP servers, an
> Exchange server and a Cyrus server.  In this particular case, it makes
> it all the way through the exchange server, and it locks up _after_
> the last group of the Cyrus server.  I.e., it did make it through a
> full scan of both servers before the hanging NOOPs.

Ditto here.  And the problematic server on my side is also a Cyrus
server:

  * OK mail Cyrus IMAP4 v2.2.12 server ready

Bye,
Tassilo
-- 
Sent from my Emacs

Re: Exchange, nnimap, and lock-ups

[Michael Welsh Duggan]

Tassilo Horn <tassilo@member.fsf.org> writes:

> Michael Welsh Duggan <md5i@md5i.com> writes:
>
> Hi Michael,
>
>>> I can verify this:
>>>
>>> 14:29:09 958 EXAMINE "shared.teams.netsa.logs.svn"
>>> 14:29:09 959 UID FETCH 14536:* FLAGS
>>> 14:29:09 960 NOOP
>>> 14:29:09 961 NOOP
>>
>> I should note that I am connected to two separate IMAP servers, an
>> Exchange server and a Cyrus server.  In this particular case, it makes
>> it all the way through the exchange server, and it locks up _after_
>> the last group of the Cyrus server.  I.e., it did make it through a
>> full scan of both servers before the hanging NOOPs.
>
> Ditto here.  And the problematic server on my side is also a Cyrus
> server:
>
>   * OK mail Cyrus IMAP4 v2.2.12 server ready

As a further data point, I did the following:

(cancel-timer nnimap-keepalive-timer)
(setq nnimap-keepalive-timer t)

After this, I have not seen any problems, so it's very likely a problem
of sorts with the NOOP processing.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Exchange, nnimap, and lock-ups

[Lars Ingebrigtsen]

Michael Welsh Duggan <md5i@md5i.com> writes:

> After this, I have not seen any problems, so it's very likely a problem
> of sorts with the NOOP processing.

I found a bug in the code that would register when commands are
executing, and this might lead to NOOPs being issued prematurely.  Could
you check whether the current git Gnus fixes this?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Exchange, nnimap, and lock-ups

[Michael Welsh Duggan]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael Welsh Duggan <md5i@md5i.com> writes:
>
>> After this, I have not seen any problems, so it's very likely a problem
>> of sorts with the NOOP processing.
>
> I found a bug in the code that would register when commands are
> executing, and this might lead to NOOPs being issued prematurely.  Could
> you check whether the current git Gnus fixes this?

It does appear to, yes.  Thank you.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: Exchange, nnimap, and lock-ups

[Michael Welsh Duggan]

Michael Welsh Duggan <md5i@md5i.com> writes:

> I run a gnus demon, with the following parameters:
>
> (gnus-demon-add-handler 'gnus-demon-scan-news 10 1)
>
> Most of the time this works, but every now and then it will lock up
> mid-fetch, and I will have to C-g to get things to continue.  Also,
> messages in my INBOX will get split appropriately, but will not be
> deleted or expunged out of the INBOX after they have been copied.

Here is the portion of the imap-log where a message is copied from my
INBOX folder via mail splitting.  Note the absence of an mark
deleted/expunge clause.  I have no idea why, as this definitely used to
work (and sometimes still does).

16:56:05 3312 LIST "" "*"
16:56:05 3313 SELECT "INBOX"
16:56:05 3314 UID FETCH 1:* FLAGS
16:56:05 3315 UID FETCH 25302:25305 (UID BODY.PEEK[HEADER] BODY.PEEK[1])
16:56:05 3316 UID COPY 25305 "INBOX/echo"
16:56:05 3317 EXAMINE "INBOX/junk-2010-11-19"

> Everything seemed to work well before, and I believe I have traced the
> problem to commit 8943cf1ad32a11652fe72ef4d34025f9618176ec:
>
>     (nnimap-wait-for-response): Wait for results in a more secure
>     manner.

-- 
Michael Welsh Duggan
(md5i@md5i.com)