error In imap-parse-body 2 (Exchange 2007)

error In imap-parse-body 2 (Exchange 2007)

[Jake Colman]

I am trying to enter an IMAP folder.  After hanging a while, if I press
C-g, I get the following:

(error/warning) Error in process filter: (error In imap-parse-body 2)

In my imap-log, I see the following:

1165 OK FETCH completed

so it would seem that the fetch completed.

The actual fetch looks like this:

1165 UID FETCH 515 (UID RFC822.SIZE BODY BODY.PEEK[HEADER.FIELDS (Subject
From Date Message-Id References In-Reply-To Xref To Newsgroups)])
* 515 FETCH (UID 515 RFC822.SIZE 2928 BODY (("text" "plain" ("charset"
  "iso-8859-1") NIL NIL "quoted-printable" 47 7)("message" "rfc822" NIL NIL
  NIL NIL 2128 ("Thu, 3 Apr 2008 10:34:35 -0400" "On Leave Tomorrow (4th
  April 2008)" (("Rafiq Ahmed" NIL "rafiq.ahmed" "Xoriant.Com")) NIL NIL
  (("Jake Colman" NIL "colman" "ppllc.com")) NIL NIL NIL
  "<FA3D7C54DA3A1540B2C92C93ACD2156002A32D6E@XOREXCH.India.XoriantCorp.com>")("text"
  "plain" ("charset" "iso-8859-1") NIL NIL "quoted-printable" 702 22)
  54)("text" "plain" ("name" "ATT00001.txt") NIL "ATT00001.txt" "base64" 252
  4) "mixed") BODY[HEADER.FIELDS (Subject From Date Message-Id References
  In-Reply-To Xref To Newsgroups)] {214}
From: Jake Colman <colman@ppllc.com>
To: Hector Fierro <fierro@ppllc.com>
Date: Thu, 3 Apr 2008 11:27:02 -0400
Subject: Fwd: On Leave Tomorrow (4th April 2008)
Message-ID: <76fxu3dkmx.fsf@dev-d01.ppllc.com>

)
1165 OK FETCH completed.

So where do I go from here?  What other information can I provide?

TIA!

...Jake

-- 
Jake Colman
Director of Software Development
Principia Partners LLC
101 West Elm Street
Suite 620
Conshohocken, PA  19428
+1 (610) 755-9786
www.principiapartners.com

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Re: error In imap-parse-body 2 (Exchange 2007)

[James Cloos]

>>>>> "Jake" == Jake Colman <colman@ppllc.com> writes:

Jake> (error/warning) Error in process filter: (error In imap-parse-body 2)

Jake> In my imap-log, I see the following:

Jake>   "iso-8859-1") NIL NIL "quoted-printable" 47 7)("message" "rfc822" NIL NIL
                                                        ^^^^^^^   ^^^^^^

I posted a bug report about message/rfc822 parts a while back.
Including a debug trace.

I wasn't able to debug why it happens, but nnimap is unable to deal with them.

(I should note that message/rfc822 parts have a significantly different
representation in an imap FETCH responce than any other mime type.  And
that code in imap.el/nnimap.el *looks* OK; it seems to have the right
logic.  And the abf in the comments matches what my imapd sends.
But something is nonetheless wrong.)

I was forced to route all of my incoming mail which has a message/rfc822
part to a non-imap folder using this procmail recipe:

,----[ excerpt from ~/.procmailrc ]
| # push message/rfc822 mail to a nnmh folder since nnimap.el eats them.
| :0HBw
| * ^Content-Type: message/rfc822
|   RFC822/.
`----

I had confirmed that other imap clients like mutt and seamonkey had no
problem with such messages on my imapd, but didn't have any other imap
server to test against.  That you see it as well proves that it is not
specific to dbmail-imapd.

I wouldn't ming seeing this one squashed.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: error In imap-parse-body 2 (Exchange 2007)

[Jake Colman]

>>>>> "JC" == James Cloos <cloos@jhcloos.com> writes:

>>>>> "Jake" == Jake Colman <colman@ppllc.com> writes:

   Jake> (error/warning) Error in process filter: (error In imap-parse-body 2)

   Jake> In my imap-log, I see the following:

   Jake> "iso-8859-1") NIL NIL "quoted-printable" 47 7)("message" "rfc822" NIL
   Jake> NIL
   JC>                                                         ^^^^^^^   ^^^^^^

   JC> I posted a bug report about message/rfc822 parts a while back.
   JC> Including a debug trace.

   JC> I wasn't able to debug why it happens, but nnimap is unable to deal with
   JC> them.

   JC> (I should note that message/rfc822 parts have a significantly different
   JC> representation in an imap FETCH responce than any other mime type.  And
   JC> that code in imap.el/nnimap.el *looks* OK; it seems to have the right
   JC> logic.  And the abf in the comments matches what my imapd sends.
   JC> But something is nonetheless wrong.)

   JC> I was forced to route all of my incoming mail which has a message/rfc822
   JC> part to a non-imap folder using this procmail recipe:

   JC> ,----[ excerpt from ~/.procmailrc ]
   JC> | # push message/rfc822 mail to a nnmh folder since nnimap.el eats them.
   JC> | :0HBw
   JC> | * ^Content-Type: message/rfc822
   JC> |   RFC822/.
   JC> `----

   JC> I had confirmed that other imap clients like mutt and seamonkey had no
   JC> problem with such messages on my imapd, but didn't have any other imap
   JC> server to test against.  That you see it as well proves that it is not
   JC> specific to dbmail-imapd.

   JC> I wouldn't ming seeing this one squashed.

Interesting.  I assumed that this is part and parcel of my my company's
"upgrade" from Exchange 5.5 to Exchange 2007.  But I guess other servers
are broken too.

-- 
Jake Colman
Director of Software Development
Principia Partners LLC
101 West Elm Street
Suite 620
Conshohocken, PA  19428
+1 (610) 755-9786
www.principiapartners.com

Re: error In imap-parse-body 2 (Exchange 2007)

[Jake Colman]

>>>>> "JC" == James Cloos <cloos@jhcloos.com> writes:

   JC> I posted a bug report about message/rfc822 parts a while back.
   JC> Including a debug trace.

   JC> I wasn't able to debug why it happens, but nnimap is unable to deal with
   JC> them.

   JC> (I should note that message/rfc822 parts have a significantly different
   JC> representation in an imap FETCH responce than any other mime type.  And
   JC> that code in imap.el/nnimap.el *looks* OK; it seems to have the right
   JC> logic.  And the abf in the comments matches what my imapd sends.
   JC> But something is nonetheless wrong.)

   JC> I was forced to route all of my incoming mail which has a message/rfc822
   JC> part to a non-imap folder using this procmail recipe:

   JC> ,----[ excerpt from ~/.procmailrc ]
   JC> | # push message/rfc822 mail to a nnmh folder since nnimap.el eats them.
   JC> | :0HBw
   JC> | * ^Content-Type: message/rfc822
   JC> |   RFC822/.
   JC> `----

   JC> I had confirmed that other imap clients like mutt and seamonkey had no
   JC> problem with such messages on my imapd, but didn't have any other imap
   JC> server to test against.  That you see it as well proves that it is not
   JC> specific to dbmail-imapd.

   JC> I wouldn't ming seeing this one squashed.

James,

Can you think of any way I can avoid the problem?  I don't (think I)
have the option of using a procmailrc recipe since my email is delivered
directly to the IMAP server and I use Gnus as an IMAP client.  Is there
something I can do to tell Gnus to avoid the problematic email?

...Jake


-- 
Jake Colman
Director of Software Development
Principia Partners LLC
101 West Elm Street
Suite 620
Conshohocken, PA  19428
+1 (610) 755-9786
www.principiapartners.com

Re: error In imap-parse-body 2 (Exchange 2007)

[Jake Colman]

>>>>> "Jake" == Jake Colman <colman@ppllc.com> writes:

>>>>> "JC" == James Cloos <cloos@jhcloos.com> writes:
   JC> I posted a bug report about message/rfc822 parts a while back.
   JC> Including a debug trace.

   JC> I wasn't able to debug why it happens, but nnimap is unable to deal
   JC> with
   JC> them.

   JC> (I should note that message/rfc822 parts have a significantly
   JC> different
   JC> representation in an imap FETCH responce than any other mime type.
   JC> And
   JC> that code in imap.el/nnimap.el *looks* OK; it seems to have the right
   JC> logic.  And the abf in the comments matches what my imapd sends.
   JC> But something is nonetheless wrong.)

   JC> I was forced to route all of my incoming mail which has a
   JC> message/rfc822
   JC> part to a non-imap folder using this procmail recipe:

   JC> ,----[ excerpt from ~/.procmailrc ]
   JC> | # push message/rfc822 mail to a nnmh folder since nnimap.el eats
   JC> them.
   JC> | :0HBw
   JC> | * ^Content-Type: message/rfc822
   JC> |   RFC822/.
   JC> `----

   JC> I had confirmed that other imap clients like mutt and seamonkey had no
   JC> problem with such messages on my imapd, but didn't have any other imap
   JC> server to test against.  That you see it as well proves that it is not
   JC> specific to dbmail-imapd.

   JC> I wouldn't ming seeing this one squashed.

   Jake> James,

   Jake> Can you think of any way I can avoid the problem?  I don't (think I)
   Jake> have the option of using a procmailrc recipe since my email is delivered
   Jake> directly to the IMAP server and I use Gnus as an IMAP client.  Is there
   Jake> something I can do to tell Gnus to avoid the problematic email?

   Jake> ...Jake

James,

Actually, even more important in the short-term, is there a way to force
Gnus to keep going if it encounters this error?  If I try to enter a
group that has this one of these messages, it simply hangs.  Then when I
hit C-g, I see the warning.  The only way to continue is for me to use a
different IMAP client to kill the offending message.  Can you think of a
better way?

...Jake


-- 
Jake Colman
Director of Software Development
Principia Partners LLC
101 West Elm Street
Suite 620
Conshohocken, PA  19428
+1 (610) 755-9786
www.principiapartners.com

Re: error In imap-parse-body 2 (Exchange 2007)

[James Cloos]

>>>>> "Jake" == Jake Colman <colman@ppllc.com> writes:

Jake> Is there something I can do to tell Gnus to avoid the problematic email?

I didn't discover one.  And it is an insidious bug; when it hits not
only does that email not show up in the *Summary* buffer, but nothing
that the server sent in its FETCH reply which is after the first rfc/822
mail will show up either.

The best I managed was to log in in parallel with mutt, delete the
offending mail and re-enter the group in gnus.  Since I have procmail in
between postfix and my imap store anyway, filtering those out to a
folder where gnus can handle them was the easiest option.

(I did post a bug report, but never saw any replies, and my attempts to
debug it didn't elucidate the actual cause....)

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: error In imap-parse-body 2 (Exchange 2007)

[James Cloos]

>>>>> "Jake" == Jake Colman <colman@ppllc.com> writes:

Jake> Actually, even more important in the short-term, is there a way to
Jake> force Gnus to keep going if it encounters this error?

I tried commenting the assert out but that didn't help.  At that point
gnus is too confused about how to parse the FETCH reply.

I *think* what gnus would need to do is to forget that message, skip
past the next end-of-line and if that looks like the start of the next
message's data to continue parsing there as if the offending message
were not included in the reply.

It would take me a while -- perhaps a few hours -- to figure out how to
do that in elisp.  My only exposure to parsing in elisp has been trying
to debug this problem, and that was weeks (maybe months by now) ago.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: error In imap-parse-body 2 (Exchange 2007)

[Jake Colman]

>>>>> "JC" == James Cloos <cloos@jhcloos.com> writes:

>>>>> "Jake" == Jake Colman <colman@ppllc.com> writes:

   Jake> Actually, even more important in the short-term, is there a way
   Jake> to force Gnus to keep going if it encounters this error?

   JC> I tried commenting the assert out but that didn't help.  At that
   JC> point gnus is too confused about how to parse the FETCH reply.

   JC> I *think* what gnus would need to do is to forget that message,
   JC> skip past the next end-of-line and if that looks like the start
   JC> of the next message's data to continue parsing there as if the
   JC> offending message were not included in the reply.

   JC> It would take me a while -- perhaps a few hours -- to figure out
   JC> how to do that in elisp.  My only exposure to parsing in elisp
   JC> has been trying to debug this problem, and that was weeks (maybe
   JC> months by now) ago.

I wish I could help you but, although I can read elisp, I cannot really
write it.  Far be it from me to tell you how to spend your time but it
would certainly bail me out of a tough spot if you would find a way to
avoid this problem.

I'll be happy to help you test this or debug it anyway possible.

Thanks.

...Jake

-- 
Jake Colman
Director of Software Development
Principia Partners LLC
101 West Elm Street
Suite 620
Conshohocken, PA  19428
+1 (610) 755-9786
www.principiapartners.com

Re: error In imap-parse-body 2 (Exchange 2007)

[Jake Colman]

>>>>> "JC" == James Cloos <cloos@jhcloos.com> writes:

>>>>> "Jake" == Jake Colman <colman@ppllc.com> writes:

   Jake> Is there something I can do to tell Gnus to avoid the
   Jake> problematic email?

   JC> I didn't discover one.  And it is an insidious bug; when it hits
   JC> not only does that email not show up in the *Summary* buffer, but
   JC> nothing that the server sent in its FETCH reply which is after
   JC> the first rfc/822 mail will show up either.

   JC> The best I managed was to log in in parallel with mutt, delete
   JC> the offending mail and re-enter the group in gnus.  Since I have
   JC> procmail in between postfix and my imap store anyway, filtering
   JC> those out to a folder where gnus can handle them was the easiest
   JC> option.

   JC> (I did post a bug report, but never saw any replies, and my
   JC> attempts to debug it didn't elucidate the actual cause....)


That's pretty much what I've been doing as well.  I have Outlook up and
running on my PC and I manually delete any messages that Gnus cannot
handle.


-- 
Jake Colman
Director of Software Development
Principia Partners LLC
101 West Elm Street
Suite 620
Conshohocken, PA  19428
+1 (610) 755-9786
www.principiapartners.com

Re: error In imap-parse-body 2 (Exchange 2007)

[Ted Zlatanov]

On Thu, 10 Apr 2008 10:46:25 -0400 James Cloos <cloos@jhcloos.com> wrote: 

>>>>>> "Jake" == Jake Colman <colman@ppllc.com> writes:
Jake> Actually, even more important in the short-term, is there a way to
Jake> force Gnus to keep going if it encounters this error?

JC> I tried commenting the assert out but that didn't help.  At that point
JC> gnus is too confused about how to parse the FETCH reply.

JC> I *think* what gnus would need to do is to forget that message, skip
JC> past the next end-of-line and if that looks like the start of the next
JC> message's data to continue parsing there as if the offending message
JC> were not included in the reply.

JC> It would take me a while -- perhaps a few hours -- to figure out how to
JC> do that in elisp.  My only exposure to parsing in elisp has been trying
JC> to debug this problem, and that was weeks (maybe months by now) ago.

Jake, James,

I tried to duplicate this bug but couldn't.  Can each of you please
forward a message that shows the problem to me, and tell me what IMAP
server you use when you see the bug?

I'll work on fixing this once I can duplicate it.

Thanks
Ted

Re: error In imap-parse-body 2 (Exchange 2007)

[Ted Zlatanov]

On Thu, 10 Apr 2008 14:09:07 -0500 Ted Zlatanov <tzz@lifelogs.com> wrote: 

TZ> On Thu, 10 Apr 2008 10:46:25 -0400 James Cloos <cloos@jhcloos.com> wrote: 
>>>>>>> "Jake" == Jake Colman <colman@ppllc.com> writes:
Jake> Actually, even more important in the short-term, is there a way to
Jake> force Gnus to keep going if it encounters this error?

JC> I tried commenting the assert out but that didn't help.  At that point
JC> gnus is too confused about how to parse the FETCH reply.

JC> I *think* what gnus would need to do is to forget that message, skip
JC> past the next end-of-line and if that looks like the start of the next
JC> message's data to continue parsing there as if the offending message
JC> were not included in the reply.

JC> It would take me a while -- perhaps a few hours -- to figure out how to
JC> do that in elisp.  My only exposure to parsing in elisp has been trying
JC> to debug this problem, and that was weeks (maybe months by now) ago.

TZ> I tried to duplicate this bug but couldn't.  Can each of you please
TZ> forward a message that shows the problem to me, and tell me what IMAP
TZ> server you use when you see the bug?

I spent time putting the exact message that's causing you trouble into
my IMAP server (tried Exchange and Courier) and didn't have any luck
triggering the assert.  You'll have to spend more time trying to trigger
and fix this, unfortunately.  As a first step, apply this patch so the
assert call will tell us where in the IMAP buffer we are and what's
around us.  This will be a LARGE debug statement so do it right after
you start Gnus, to diminish the amount of data that's getting sent out,
and then don't use it anymore (remove it and restart Gnus).

As an aside, I am doing C-x C-e to redefine imap-parse-body and it's not
taking effect.  Anyone know why that would happen?

Thanks
Ted

Index: imap.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/imap.el,v
retrieving revision 7.45
diff -r7.45 imap.el
2903,2904c2903,2904
< 
<       (assert (eq (char-after) ?\)) nil "In imap-parse-body 2")
---
>       
>       (assert (eq (char-after) ?\)) nil "position %d in %s" (point) (buffer-string))

Re: error In imap-parse-body 2 (Exchange 2007)

[Jake Colman]

>>>>> "TZ" == Ted Zlatanov <tzz@lifelogs.com> writes:

   TZ> I spent time putting the exact message that's causing you trouble into
   TZ> my IMAP server (tried Exchange and Courier) and didn't have any luck
   TZ> triggering the assert.  You'll have to spend more time trying to trigger
   TZ> and fix this, unfortunately.  As a first step, apply this patch so the
   TZ> assert call will tell us where in the IMAP buffer we are and what's
   TZ> around us.  This will be a LARGE debug statement so do it right after
   TZ> you start Gnus, to diminish the amount of data that's getting sent out,
   TZ> and then don't use it anymore (remove it and restart Gnus).

Ted,

I have not yet had a chance to try this patch but perhaps you can try
the following:

1) Forward an email that you have received but with a gcc-self.
2) In the folder that is the target of the gcc-self, you will get a copy
   of the forwarded email.  This email should be formatted as an
   attachment.
3) Try reading that email.

This sequence of steps will always trigger the problem on my Exchange
2007 server.

James - can you try this too?

...Jake


-- 
Jake Colman
Director of Software Development
Principia Partners LLC
101 West Elm Street
Suite 620
Conshohocken, PA  19428
+1 (610) 755-9786
www.principiapartners.com

Re: error In imap-parse-body 2 (Exchange 2007)

[James Cloos]

I forwarded that message to myself, bypassing the procmail rule which
would otherwise have kept it out of the dbmail store.

The exact packet dbmail-imapd sent to imap.el looks like:

=============================================================================
* 1 FETCH (RFC822.SIZE 5872 UID 1 BODY (("text" "plain" NIL NIL NIL "7BIT" 13 3)("message" "rfc822" NIL NIL NIL "7BIT" 4551 ("Thu, 24 Apr 2008 10:54:20 -0400" "Re: error In imap-parse-body 2 (Exchange 2007)" (("Jake Colman" NIL "colman" "ppllc.com"))(("Jake Colman" NIL "colman" "ppllc.com"))(("Jake Colman" NIL "colman" "ppllc.com"))(("Ted Zlatanov" NIL "tzz" "lifelogs.com"))(("James Cloos" NIL "cloos" "jhcloos.com")(NIL NIL "ding" "gnus.org")) NIL {91}
<86ej95lfdp.fsf@lifelogs.com> (Ted Zlatanov's message of "Wed, 16	Apr 2008 15:21:54 -0500") "<764p9r9ucj.fsf@dev-d01.ppllc.com>")("text" "plain" NIL NIL NIL "7BIT" 1339 41) 119) "mixed") BODY[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref To Newsgroups)] {227}
to: cloos@freeshell.org
subject: [Jake Colman] Re: error In imap-parse-body 2 (Exchange 2007)
message-id: <m3mynjqcbn.fsf@lugabout.jhcloos.org>
from: James Cloos <cloos@jhcloos.com>
date: Thu, 24 Apr 2008 15:31:17 -0400

)

=============================================================================

(OK, I did replace the CRLF line endings with LF line endings.)

The first interesting thing I see is that the tab in the In-Reply-To
header's value (from a continuation line) is still there.  Dbmail
dropped the CRLF but did not convert the tab to a space.

I'll have to try it again with the patch.

(At least since I upgraded pgsql from 8.2 to 8.3 -- which included a
full dump and restore of the db's data -- and switched from having
xinetd run dbmail-imapd -n to having emacs do so -- which eliminated
the 50% idle time during startup -- it now takes somewhere between
10 and 20 minutes to startup rather than 3 hours or so....)

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: error In imap-parse-body 2 (Exchange 2007)

[Ted Zlatanov]

On Thu, 24 Apr 2008 10:54:20 -0400 Jake Colman <colman@ppllc.com> wrote: 

>>>>>> "TZ" == Ted Zlatanov <tzz@lifelogs.com> writes:
TZ> I spent time putting the exact message that's causing you trouble into
TZ> my IMAP server (tried Exchange and Courier) and didn't have any luck
TZ> triggering the assert.  You'll have to spend more time trying to trigger
TZ> and fix this, unfortunately.  As a first step, apply this patch so the
TZ> assert call will tell us where in the IMAP buffer we are and what's
TZ> around us.  This will be a LARGE debug statement so do it right after
TZ> you start Gnus, to diminish the amount of data that's getting sent out,
TZ> and then don't use it anymore (remove it and restart Gnus).

Jake> 1) Forward an email that you have received but with a gcc-self.
Jake> 2) In the folder that is the target of the gcc-self, you will get a copy
Jake>    of the forwarded email.  This email should be formatted as an
Jake>    attachment.
Jake> 3) Try reading that email.

Jake> This sequence of steps will always trigger the problem on my Exchange
Jake> 2007 server.

It doesn't do it for me.  Very strange.

Ted