nnimap compatibility issues (UTF-7, msg count, SEARCH vs. group select)

nnimap compatibility issues (UTF-7, msg count, SEARCH vs. group select)

[Matthias Andree]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 3698 bytes --]

Hi,

I have some compatibility issues with using imap.web.de. First of all:
Mozilla 1.3 and Sylpheed 0.8.11claws get it right, so it seems doable
with some effort. Whether it's worth it, is up to others to decide, but
I'll report it all the same, I have IMAP traces and profile or backtrace
where needed.

I don't speak IMAP4r1 very well, but I hope I got the relevant
sections. I will refine my logs if needed.

Contents:

1. UTF-7 decoding mangled
2. message count hosed
3. selecting group fails and hogs CPU to 100% load

--------------------

1. UTF-7 decoding mangled

Here's the IMAP excerpt:

1 CAPABILITY
* CAPABILITY IMAP4 IMAP4rev1 LITERAL+ UIDPLUS X-NETSCAPE
1 OK Completed
2 LOGIN "hohohoimnottellingyou" "youdlovetoknow"
2 OK User logged in
3 LIST "" "*%"
* LIST (\Noinferiors ) "/" "Trash"
* LIST (\Noinferiors ) "/" "Unerw&APw-nscht"
...
3 OK Completed

The "Unerw&APw-nscht" string is misdisplayed in the server buffer, it
prints as "Unerw.ünscht"; UTF-7 decoding is on. However, it appears
correctly as "Unerwünscht" in other client software, namely Sylpheed
0.8.11claws and Mozilla 1.3. Just in case: CVS 5.10.x Gnus, emacs 21.3.

--------------------

2. Next, the INBOX message count in the server buffer is apparently hosed:

K      0: Gesendet
 107062715: INBOX
K      0: Papierkorb

Evidently, the INBOX has only 365 messages.

18 EXAMINE "INBOX"
* 365 EXISTS
* OK [UNSEEN 7] Message 359 is first unseen
* OK [PERMANENTFLAGS (\Deleted \Seen \Answered)]
* OK [UIDVALIDITY 1]
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
18 OK Completed [READ-ONLY]
19 FETCH 1,* UID
* 1 FETCH (UID 643933627)
* 365 FETCH (UID 700320820)
19 OK Completed

--------------------

3. Group selection fails with 100% CPU loop (the server is agentized, FWIW):

26 SELECT "INBOX"
* 365 EXISTS
* OK [UNSEEN 7] Message 359 is first unseen
* OK [PERMANENTFLAGS (\Deleted \Seen \Answered)]
* OK [UIDVALIDITY 1]
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
26 OK Completed [READ-WRITE]
27 UID SEARCH UNSEEN UNDELETED
27 BAD Function not implemented
28 UID SEARCH SEEN
28 BAD Function not implemented
...

Gnus then tries SEARCH ANSWERED, SEARCH RECENT (all BAD), again FETCH,
and these four SEARCH commands again and then goes into a 100% CPU
loop. Here's the ELP trace:

Function Name                            Call Count  Elapsed Time  Average Time
=======================================  ==========  ============  ============
nnimap-request-group                     2           16.718578     8.359289
nnimap-request-update-info-internal      2           16.645619     8.3228095
nnimap-find-minmax-uid                   1           0.071981      0.071981
nnimap-mark-permanent-p                  28          0.001458      5.207...e-05
nnimap-possibly-change-group             4           0.00096       0.00024
nnimap-mark-to-flag                      30          0.0006149999  2.049...e-05
nnimap-mark-to-flag-1                    30          0.0001290000  4.300...e-06
nnimap-server-opened                     2           7.8e-05       3.9e-05

And here's a backtrace, just in case:

Debugger entered--Lisp error: (quit)
  gnus-list-of-unread-articles("nnimap+imap.web.de:INBOX")
  gnus-select-newsgroup("nnimap+imap.web.de:INBOX" nil nil)
  gnus-summary-read-group-1("nnimap+imap.web.de:INBOX" nil nil nil nil nil)
  gnus-summary-read-group("nnimap+imap.web.de:INBOX" nil nil nil nil nil nil)
  gnus-group-read-group(nil)
  gnus-mouse-pick-group((mouse-2 (#<window 20 on *Group*> 37 (324 . 13) 56961915)))
  call-interactively(gnus-mouse-pick-group)
...

-- 
Matthias Andree

Re: nnimap compatibility issues (UTF-7, msg count, SEARCH vs. group

[Dave Love]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 1298 bytes --]

Matthias Andree <ma@dt.e-technik.uni-dortmund.de> writes:

> 1. UTF-7 decoding mangled
>
> Here's the IMAP excerpt:
>
> 1 CAPABILITY
> * CAPABILITY IMAP4 IMAP4rev1 LITERAL+ UIDPLUS X-NETSCAPE
> 1 OK Completed
> 2 LOGIN "hohohoimnottellingyou" "youdlovetoknow"
> 2 OK User logged in
> 3 LIST "" "*%"
> * LIST (\Noinferiors ) "/" "Trash"
> * LIST (\Noinferiors ) "/" "Unerw&APw-nscht"
> ...
> 3 OK Completed
>
> The "Unerw&APw-nscht" string is misdisplayed in the server buffer, it
> prints as "Unerw.ünscht";

I'm not sure what that means and what exactly goes wrong.  Where is it
decoded from the utf-7?

> UTF-7 decoding is on. However, it appears
> correctly as "Unerwünscht" in other client software, namely Sylpheed
> 0.8.11claws and Mozilla 1.3. Just in case: CVS 5.10.x Gnus, emacs 21.3.

Did this work before I changed utf7.el recently to cope with more than
Latin-1?  If so, it's probably a coding conversion confusion, and
perhaps the dot is a mangled \201 or similar.  Previously the decoder
returned a unibyte string, which doesn't make much sense.  As far as I
could tell, that just happened to work by automatic conversion (in a
Latin-1 session).  You could test it by commenting out
`mm-enable-multibyte' in `utf7-decode'.  I can't, I'm afraid.

Re: nnimap compatibility issues (UTF-7, msg count, SEARCH vs. group

[Matthias Andree]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 1972 bytes --]

Dave Love <d.love@dl.ac.uk> writes:

>> 1. UTF-7 decoding mangled
>>
>> Here's the IMAP excerpt:
>>
>> 1 CAPABILITY
>> * CAPABILITY IMAP4 IMAP4rev1 LITERAL+ UIDPLUS X-NETSCAPE
>> 1 OK Completed
>> 2 LOGIN "hohohoimnottellingyou" "youdlovetoknow"
>> 2 OK User logged in
>> 3 LIST "" "*%"
>> * LIST (\Noinferiors ) "/" "Trash"
>> * LIST (\Noinferiors ) "/" "Unerw&APw-nscht"
>> ...
>> 3 OK Completed
>>
>> The "Unerw&APw-nscht" string is misdisplayed in the server buffer, it
>> prints as "Unerw.ünscht";
>
> I'm not sure what that means and what exactly goes wrong.  Where is it
> decoded from the utf-7?

I don't know. Can you formulate a more specific question?

>> UTF-7 decoding is on. However, it appears
>> correctly as "Unerwünscht" in other client software, namely Sylpheed
>> 0.8.11claws and Mozilla 1.3. Just in case: CVS 5.10.x Gnus, emacs 21.3.
>
> Did this work before I changed utf7.el recently to cope with more than
> Latin-1?  If so, it's probably a coding conversion confusion, and
> perhaps the dot is a mangled \201 or similar.  Previously the decoder
> returned a unibyte string, which doesn't make much sense.  As far as I
> could tell, that just happened to work by automatic conversion (in a
> Latin-1 session).  You could test it by commenting out
> `mm-enable-multibyte' in `utf7-decode'.  I can't, I'm afraid.

If I comment this one out, then the Unerw.*nscht folder disappears in
the folder list when entering the buffer. And yes indeed, the original
buffer showed \201 before the other unprintable character.

So somehow the character set the folder name was decoded to and the
buffer/display's character set didn't match?

But my bigger concern however is that I can't fetch messages from that
server. See my previous mail.

I can't really help myself out because my Elisp isn't even good enough
to fully understand the code, and MULE isn't my strong point either.

-- 
Matthias Andree

Re: nnimap compatibility issues (UTF-7, msg count, SEARCH vs. group

[Dave Love]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 2486 bytes --]

Matthias Andree <ma@dt.e-technik.uni-dortmund.de> writes:

> Dave Love <d.love@dl.ac.uk> writes:
>
>>> 1. UTF-7 decoding mangled
>>>
>>> Here's the IMAP excerpt:
>>>
>>> 1 CAPABILITY
>>> * CAPABILITY IMAP4 IMAP4rev1 LITERAL+ UIDPLUS X-NETSCAPE
>>> 1 OK Completed
>>> 2 LOGIN "hohohoimnottellingyou" "youdlovetoknow"
>>> 2 OK User logged in
>>> 3 LIST "" "*%"
>>> * LIST (\Noinferiors ) "/" "Trash"
>>> * LIST (\Noinferiors ) "/" "Unerw&APw-nscht"
>>> ...
>>> 3 OK Completed
>>>
>>> The "Unerw&APw-nscht" string is misdisplayed in the server buffer, it
>>> prints as "Unerw.ünscht";
>>
>> I'm not sure what that means and what exactly goes wrong.  Where is it
>> decoded from the utf-7?
>
> I don't know. Can you formulate a more specific question?

You said `misdisplayed in the server buffer', but in the text above
which appears to be from the server, it isn't decoded, and shouldn't
be.

>> Did this work before I changed utf7.el recently to cope with more than
>> Latin-1?  If so, it's probably a coding conversion confusion, and
>> perhaps the dot is a mangled \201 or similar.  Previously the decoder
>> returned a unibyte string, which doesn't make much sense.  As far as I
>> could tell, that just happened to work by automatic conversion (in a
>> Latin-1 session).  You could test it by commenting out
>> `mm-enable-multibyte' in `utf7-decode'.  I can't, I'm afraid.
>
> If I comment this one out, then the Unerw.*nscht folder disappears in
> the folder list when entering the buffer.

Do you mean it is still wrong?

> And yes indeed, the original
> buffer showed \201 before the other unprintable character.
>
> So somehow the character set the folder name was decoded to and the
> buffer/display's character set didn't match?

Something has decoded the multibyte text produced by the utf-7
decoding (i.e. it has been doubly decoded).  I assume this is in the
imap code, but I can't test that.

> But my bigger concern however is that I can't fetch messages from that
> server. See my previous mail.
>
> I can't really help myself out because my Elisp isn't even good enough
> to fully understand the code, and MULE isn't my strong point either.

I still don't know if it worked before I made changes to utf7.el.  Did
it work?  If so, I guess the change should be reverted until someone
can look at the imap code, but that means that mailbox names can only
be Latin-1 even when your Emacs can support more.

Re: nnimap compatibility issues (UTF-7, msg count, SEARCH vs. group

[Matthias Andree]

On Sun, 25 May 2003, Dave Love wrote:

> You said `misdisplayed in the server buffer', but in the text above
> which appears to be from the server, it isn't decoded, and shouldn't
> be.

Emacs 21.3.1 here.

This is how the IMAP server buffer looks like with unmodified utf7.el
revisions 6.1 and 6.2 from CVS, the X after the \216 is a square box
with dashed line style.

K      0: Entwurf
K      0: Gesendet
U      0: INBOX
K      0: Papierkorb
K      0: Postausgang
K      0: Trash
K      0: Unerw\216Xnscht

Now I apply this patch to utf7.el and reinstall things and restart
Emacs:

Index: utf7.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/utf7.el,v
retrieving revision 6.2
diff -u -r6.2 utf7.el
--- utf7.el	9 May 2003 20:37:56 -0000	6.2
+++ utf7.el	27 May 2003 14:15:19 -0000
@@ -219,7 +219,7 @@
     (with-temp-buffer
       (insert string)
       (utf7-decode-internal for-imap)
-      (mm-enable-multibyte)
+;      (mm-enable-multibyte)
       (buffer-string))))
 
 (provide 'utf7)

This is what the IMAP server buffer looks like now, with the patch applied:

K      0: Entwurf
K      0: Gesendet
U      0: INBOX
K      0: Papierkorb
K      0: Postausgang
K      0: Trash

This is the complete server buffer, the "Unerwünscht" folder is missing.

So removing the mm-enable-multibyte clearly makes things worse, by
hiding a folder.

> > So somehow the character set the folder name was decoded to and the
> > buffer/display's character set didn't match?
> 
> Something has decoded the multibyte text produced by the utf-7
> decoding (i.e. it has been doubly decoded).  I assume this is in the
> imap code, but I can't test that.

Assuming it isn't the utf7-decoder, where do I look?

> I still don't know if it worked before I made changes to utf7.el.  Did
> it work?  If so, I guess the change should be reverted until someone
> can look at the imap code, but that means that mailbox names can only
> be Latin-1 even when your Emacs can support more.

I believe the 6.1 -> 6.2 changes in utf7.el are fine. They make no
visible difference AFAICT. Unless my emacs used the wrong files.

I've dug a little deeper. If I try to choose this group (any group
besides INBOX), I get a "can't select" complaint.

So there is something deep within imap.el or nnimap.el that doesn't
cooperate with this server anyways.

-- 
Matthias Andree

Re: nnimap compatibility issues (UTF-7, msg count, SEARCH vs. group

[Kai Großjohann]

Matthias Andree <matthias.andree@gmx.de> writes:

> On Sun, 25 May 2003, Dave Love wrote:
>
>> You said `misdisplayed in the server buffer', but in the text above
>> which appears to be from the server, it isn't decoded, and shouldn't
>> be.
>
> Emacs 21.3.1 here.
>
> This is how the IMAP server buffer looks like with unmodified utf7.el
> revisions 6.1 and 6.2 from CVS, the X after the \216 is a square box
> with dashed line style.

Matthias,

I think what you mean with "IMAP server buffer" is the result of
hitting ^ in the group buffer, then hitting RET on the nnimap line.
Is this correct?

I believe that Dave thought you were talking about the " *nnimap foo"
buffer which is an internal buffer used by Gnus where the IMAP
protocol exchange takes place.  (Note the leading space in the buffer
name -- that signifies hidden, internal, buffers.)
-- 
This line is not blank.

Re: nnimap compatibility issues (UTF-7, msg count, SEARCH vs. group

[Matthias Andree]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> I think what you mean with "IMAP server buffer" is the result of
> hitting ^ in the group buffer, then hitting RET on the nnimap line.
> Is this correct?

Exactly.

> I believe that Dave thought you were talking about the " *nnimap foo"
> buffer which is an internal buffer used by Gnus where the IMAP
> protocol exchange takes place.  (Note the leading space in the buffer
> name -- that signifies hidden, internal, buffers.)

Hum. How should I call this buffer then to make clear I mean the server
browser? "server browser"?

-- 
Matthias Andree

Re: nnimap compatibility issues (UTF-7, msg count, SEARCH vs. group

[Kai Großjohann]

Matthias Andree <ma@dt.e-technik.uni-dortmund.de> writes:

> Hum. How should I call this buffer then to make clear I mean the server
> browser? "server browser"?

Well, the server buffer is what you get after hitting ^ in the group
buffer.  But what you were talking about is not that buffer.  You were
talking about the buffer you get after hitting RET in the server
buffer.

Anyhow, if there is no single word that describes it, then use
multiple words.
-- 
This line is not blank.

Re: nnimap compatibility issues (UTF-7, msg count, SEARCH vs. group

[Dave Love]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> I believe that Dave thought you were talking about the " *nnimap foo"
> buffer which is an internal buffer used by Gnus where the IMAP
> protocol exchange takes place.

Yes, sorry.

[Not necessarily to Kai.]  If the recent changes I made to utf7.el
aren't the problem, I'm afraid I can't help.  Do utf-7 mailbox names
basically work, i.e. can mail be read from them, regardless of what
server mode does?  Is no-one else using non-ASCII mailbox names?