cannot display folder with non-latin character using imap

cannot display folder with non-latin character using imap

[sssslang]

Hi, i recently want to use imap instead of pop for easy sync between my PC and 
laptop, but I found I cannot make gnus work under my locale setting. I use 
Gmail' imap server, and the Gmail's UI language is Chinese. Then my gnus didn't 
display folders whose name contain Chinese characters. I tried mutt and kmail, 
both of them are ok. If I change Gmail's UI language to English, Gnus works. Is 
there something wrong with Gnus's imap decode method? I look though the code 
and see the imap use utf7 to encode folder name. But I know little about lisp. 
Could someone can help me?

Re: cannot display folder with non-latin character using imap

[Ted Zlatanov]

On Sun, 29 Jun 2008 13:27:59 +0000 (UTC) sssslang <sssslang@gmail.com> wrote: 

s> Hi, i recently want to use imap instead of pop for easy sync between my PC and 
s> laptop, but I found I cannot make gnus work under my locale setting. I use 
s> Gmail' imap server, and the Gmail's UI language is Chinese. Then my gnus didn't 
s> display folders whose name contain Chinese characters. I tried mutt and kmail, 
s> both of them are ok. If I change Gmail's UI language to English, Gnus works. Is 
s> there something wrong with Gnus's imap decode method? I look though the code 
s> and see the imap use utf7 to encode folder name. But I know little about lisp. 
s> Could someone can help me?

Can you generate an IMAP and nnimap log?  Set imap-debug and
nnimap-debug to t, then take the contents of the two log buffers and
attach it.

Can you quote the exact folder name for us, as well?  I can try random
UCS characters but it's good to test with the exact same parameters.

Thanks
Ted

Re: cannot display folder with non-latin character using imap

[ssSslang]

Thank you for your reply, Ted. This is a long message, please read
patiently.

Ted Zlatanov <tzz@lifelogs.com> writes:

> Can you generate an IMAP and nnimap log?  Set imap-debug and
> nnimap-debug to t, then take the contents of the two log buffers and
> attach it.

here's imap log

--8<---------------cut here---------------start------------->8---
11 LIST "" "*%"
* LIST (\HasNoChildren) "." "[Gmail].&g0l6Pw-"
* LIST (\HasNoChildren) "." "[Gmail].&XfJSoGYfaAc-"
* LIST (\HasNoChildren) "." "[Gmail].&XfJT0ZCuTvY-"
* LIST (\HasNoChildren) "." "[Gmail].&V4NXPpCuTvY-"
* LIST (\HasNoChildren) "." "[Gmail].&YkBnCZCuTvY-"
* LIST (\HasNoChildren) "." "[Gmail].&XfJSIJZkkK5O9g-"
* LIST (\HasNoChildren) "." "testlabel"
* LIST (\HasNoChildren) "." "INBOX"
11 OK List completed.
12 EXAMINE "INBOX"
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS ()] Read-only mailbox.
* 0 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1214734941] UIDs valid
* OK [UIDNEXT 1] Predicted next UID
12 OK [READ-ONLY] Select completed.
13 EXAMINE "[Gmail].&XfJSoGYfaAc"
13 NO Mailbox doesn't exist: [Gmail].&XfJSoGYfaAc
14 EXAMINE "testlabel"
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS ()] Read-only mailbox.
* 1 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1214735045] UIDs valid
* OK [UIDNEXT 2] Predicted next UID
14 OK [READ-ONLY] Select completed.
15 FETCH 1,* UID
* 1 FETCH (UID 1)
15 OK Fetch completed.
16 EXAMINE "[Gmail].&XfJSIJZkkK5O9g"
16 NO Mailbox doesn't exist: [Gmail].&XfJSIJZkkK5O9g
17 EXAMINE "[Gmail].&g0l6Pw"
17 NO Mailbox doesn't exist: [Gmail].&g0l6Pw
18 EXAMINE "[Gmail].&V4NXPpCuTvY"
18 NO Mailbox doesn't exist: [Gmail].&V4NXPpCuTvY
19 EXAMINE "[Gmail].&YkBnCZCuTvY"
19 NO Mailbox doesn't exist: [Gmail].&YkBnCZCuTvY
20 EXAMINE "[Gmail].&XfJT0ZCuTvY"
20 NO Mailbox doesn't exist: [Gmail].&XfJT0ZCuTvY
21 NOOP
21 OK NOOP completed.
--8<---------------cut here---------------end--------------->8---

and nnimap log

--8<---------------cut here---------------start------------->8---
======================================================================
1 -> nnimap-request-list: server="GMail"
| 2 -> nnimap-possibly-change-server: server="GMail"
| 2 <- nnimap-possibly-change-server: " *nnimap* GMail"
| 2 -> nnimap-before-find-minmax-bugworkaround: 
| 2 <- nnimap-before-find-minmax-bugworkaround: t
| 2 -> nnimap-pattern-to-list-arguments: pattern="*"
| 2 <- nnimap-pattern-to-list-arguments: ((nil . "*"))
| 2 -> nnimap-find-minmax-uid: group="INBOX" examine=examine
| 2 <- nnimap-find-minmax-uid: (0 nil nil)
| 2 -> nnimap-find-minmax-uid: group="[Gmail].已加星标" examine=examine
| 2 <- nnimap-find-minmax-uid: nil
| 2 -> nnimap-find-minmax-uid: group="testlabel" examine=examine
| 2 <- nnimap-find-minmax-uid: (1 1 1)
| 2 -> nnimap-find-minmax-uid: group="[Gmail].已删除邮件" examine=examine
| 2 <- nnimap-find-minmax-uid: nil
| 2 -> nnimap-find-minmax-uid: group="[Gmail].草稿" examine=examine
| 2 <- nnimap-find-minmax-uid: nil
| 2 -> nnimap-find-minmax-uid: group="[Gmail].垃圾邮件" examine=examine
| 2 <- nnimap-find-minmax-uid: nil
| 2 -> nnimap-find-minmax-uid: group="[Gmail].所有邮件" examine=examine
| 2 <- nnimap-find-minmax-uid: nil
| 2 -> nnimap-find-minmax-uid: group="[Gmail].已发邮件" examine=examine
| 2 <- nnimap-find-minmax-uid: nil
1 <- nnimap-request-list: t
======================================================================
--8<---------------cut here---------------end--------------->8---


> Can you quote the exact folder name for us, as well?  I can try random
> UCS characters but it's good to test with the exact same parameters.

last night i did some test. see line 2 of imap.log, it list a folder
named "[Gmail].&g0l6Pw-", but at line 34, it examine a folder called
"[Gmail].&g0l6Pw"(without the following '-'). the characters "&g0l6Pw-"
is "草稿" in Chinese(means draft, sorry if you cannot display the
Chinese characters). So in nnimap log, the function
nnimap-find-minmax-uid which call imap examine returns nil.

Then i examine the two functions in imap.el: imap-utf7-encode and
imap-utf7-decode. i found imap-utf7-encode is strange(i don't know
lisp, so i just test and guess). it encode "草稿" to "&g0l6Pw" without
the last "-". I tried to make it do so by this:

--8<---------------cut here---------------start------------->8---
--- imap-org.el	2008-06-30 01:37:05.000000000 +0800
+++ imap.el	2008-06-30 00:51:39.000000000 +0800
@@ -464,7 +464,7 @@
   (if imap-use-utf7
       (and string
 	   (condition-case ()
-	       (utf7-encode string t)
+	       (concat (utf7-encode string t) "-")
 	     (error (message
 		     "imap: Could not UTF7 encode `%s', using it unencoded..."
 		     string)
--8<---------------cut here---------------end--------------->8---

Then I can see the right folder when i enter server buffer, while still
cannot select the buffer. I think this maybe something wrong in the
nnimap backend. So, if you find something, please correct it. Thanks.
-- 
ssSslang

Re: cannot display folder with non-latin character using imap

[Ted Zlatanov]

On Wed, 02 Jul 2008 09:08:09 +0800 ssSslang <sssslang@gmail.com> wrote: 

s> Ted Zlatanov <tzz@lifelogs.com> writes:

>> Can you generate an IMAP and nnimap log?  Set imap-debug and
>> nnimap-debug to t, then take the contents of the two log buffers and
>> attach it.

s> here's imap log
s> 11 LIST "" "*%"
s> * LIST (\HasNoChildren) "." "[Gmail].&g0l6Pw-"
s> * LIST (\HasNoChildren) "." "[Gmail].&XfJSoGYfaAc-"
s> * LIST (\HasNoChildren) "." "[Gmail].&XfJT0ZCuTvY-"
s> * LIST (\HasNoChildren) "." "[Gmail].&V4NXPpCuTvY-"
s> * LIST (\HasNoChildren) "." "[Gmail].&YkBnCZCuTvY-"
s> * LIST (\HasNoChildren) "." "[Gmail].&XfJSIJZkkK5O9g-"
s> * LIST (\HasNoChildren) "." "testlabel"
s> * LIST (\HasNoChildren) "." "INBOX"
s> 11 OK List completed.
s> 12 EXAMINE "INBOX"
s> * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
s> * OK [PERMANENTFLAGS ()] Read-only mailbox.
s> * 0 EXISTS
s> * 0 RECENT
s> * OK [UIDVALIDITY 1214734941] UIDs valid
s> * OK [UIDNEXT 1] Predicted next UID
s> 12 OK [READ-ONLY] Select completed.
s> 13 EXAMINE "[Gmail].&XfJSoGYfaAc"
s> 13 NO Mailbox doesn't exist: [Gmail].&XfJSoGYfaAc
s> 14 EXAMINE "testlabel"
s> * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
s> * OK [PERMANENTFLAGS ()] Read-only mailbox.
s> * 1 EXISTS
s> * 0 RECENT
s> * OK [UIDVALIDITY 1214735045] UIDs valid
s> * OK [UIDNEXT 2] Predicted next UID
s> 14 OK [READ-ONLY] Select completed.
s> 15 FETCH 1,* UID
s> * 1 FETCH (UID 1)
s> 15 OK Fetch completed.
s> 16 EXAMINE "[Gmail].&XfJSIJZkkK5O9g"
s> 16 NO Mailbox doesn't exist: [Gmail].&XfJSIJZkkK5O9g
s> 17 EXAMINE "[Gmail].&g0l6Pw"
s> 17 NO Mailbox doesn't exist: [Gmail].&g0l6Pw
s> 18 EXAMINE "[Gmail].&V4NXPpCuTvY"
s> 18 NO Mailbox doesn't exist: [Gmail].&V4NXPpCuTvY
s> 19 EXAMINE "[Gmail].&YkBnCZCuTvY"
s> 19 NO Mailbox doesn't exist: [Gmail].&YkBnCZCuTvY
s> 20 EXAMINE "[Gmail].&XfJT0ZCuTvY"
s> 20 NO Mailbox doesn't exist: [Gmail].&XfJT0ZCuTvY
s> 21 NOOP
s> 21 OK NOOP completed.

s> and nnimap log
s> ======================================================================
s> 1 -> nnimap-request-list: server="GMail"
s> | 2 -> nnimap-possibly-change-server: server="GMail"
s> | 2 <- nnimap-possibly-change-server: " *nnimap* GMail"
s> | 2 -> nnimap-before-find-minmax-bugworkaround: 
s> | 2 <- nnimap-before-find-minmax-bugworkaround: t
s> | 2 -> nnimap-pattern-to-list-arguments: pattern="*"
s> | 2 <- nnimap-pattern-to-list-arguments: ((nil . "*"))
s> | 2 -> nnimap-find-minmax-uid: group="INBOX" examine=examine
s> | 2 <- nnimap-find-minmax-uid: (0 nil nil)
s> | 2 -> nnimap-find-minmax-uid: group="[Gmail].已加星标" examine=examine
s> | 2 <- nnimap-find-minmax-uid: nil
s> | 2 -> nnimap-find-minmax-uid: group="testlabel" examine=examine
s> | 2 <- nnimap-find-minmax-uid: (1 1 1)
s> | 2 -> nnimap-find-minmax-uid: group="[Gmail].已删除邮件" examine=examine
s> | 2 <- nnimap-find-minmax-uid: nil
s> | 2 -> nnimap-find-minmax-uid: group="[Gmail].草稿" examine=examine
s> | 2 <- nnimap-find-minmax-uid: nil
s> | 2 -> nnimap-find-minmax-uid: group="[Gmail].垃圾邮件" examine=examine
s> | 2 <- nnimap-find-minmax-uid: nil
s> | 2 -> nnimap-find-minmax-uid: group="[Gmail].所有邮件" examine=examine
s> | 2 <- nnimap-find-minmax-uid: nil
s> | 2 -> nnimap-find-minmax-uid: group="[Gmail].已发邮件" examine=examine
s> | 2 <- nnimap-find-minmax-uid: nil
s> 1 <- nnimap-request-list: t
s> ======================================================================


>> Can you quote the exact folder name for us, as well?  I can try random
>> UCS characters but it's good to test with the exact same parameters.

s> last night i did some test. see line 2 of imap.log, it list a folder
s> named "[Gmail].&g0l6Pw-", but at line 34, it examine a folder called
s> "[Gmail].&g0l6Pw"(without the following '-'). the characters "&g0l6Pw-"
s> is "草稿" in Chinese(means draft, sorry if you cannot display the
s> Chinese characters). So in nnimap log, the function
s> nnimap-find-minmax-uid which call imap examine returns nil.

s> Then i examine the two functions in imap.el: imap-utf7-encode and
s> imap-utf7-decode. i found imap-utf7-encode is strange(i don't know
s> lisp, so i just test and guess). it encode "草稿" to "&g0l6Pw" without
s> the last "-". I tried to make it do so by this:
s> --- imap-org.el	2008-06-30 01:37:05.000000000 +0800
s> +++ imap.el	2008-06-30 00:51:39.000000000 +0800
s> @@ -464,7 +464,7 @@
s>    (if imap-use-utf7
s>        (and string
s>  	   (condition-case ()
s> -	       (utf7-encode string t)
s> +	       (concat (utf7-encode string t) "-")
s>  	     (error (message
s>  		     "imap: Could not UTF7 encode `%s', using it unencoded..."
s>  		     string)

s> Then I can see the right folder when i enter server buffer, while still
s> cannot select the buffer. I think this maybe something wrong in the
s> nnimap backend. So, if you find something, please correct it. Thanks.
[reordered]
s> Thank you for your reply, Ted. This is a long message, please read
s> patiently.

Can you please avoid top-posting?  Thanks.  Also, your report was very
helpful and I appreciate all the effort you put into it.

I don't see the problem here.  I do get an error but it's not due to the
utf7 ancoding, and I think it's registry-related (everything works until
AFTER the message is spooled, and the article doesn't show up until
after a refresh).  Here's me entering the group:

nnimap log:

1 -> nnimap-close-group: group="INBOX.\345\267\262\345\212\240\346\230\237\346\240\207" server="blockstar.com"
| 2 -> nnimap-possibly-change-group: group="INBOX.\345\267\262\345\212\240\346\230\237\346\240\207" server="blockstar.com"
| | 3 -> nnimap-possibly-change-server: server="blockstar.com"
| | 3 <- nnimap-possibly-change-server: " *nnimap* blockstar.com"
| | 3 -> nnimap-verify-uidvalidity: group="INBOX.\345\267\262\345\212\240\346\230\237\346\240\207" server="blockstar.com"
| | 3 <- nnimap-verify-uidvalidity: t
| 2 <- nnimap-possibly-change-group: "INBOX.&ACAAIAAgACAAIAAgACAAIAAgACAAIAAg-"
1 <- nnimap-close-group: nil

IMAP log:

2177 EXAMINE "INBOX.&ACAAIAAgACAAIAAgACAAIAAgACAAIAAg-"
* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS ()] No permanent flags permitted
* 3 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1215118658] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
2177 OK [READ-ONLY] Ok
2178 FETCH 1,* UID
* 1 FETCH (UID 1)
* 3 FETCH (UID 3)
2178 OK FETCH completed.
2179 NOOP
2179 OK NOOP completed
2180 SELECT "INBOX.&ACAAIAAgACAAIAAgACAAIAAgACAAIAAg-"
* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited
* 3 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1215118658] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
2180 OK [READ-WRITE] Ok
2181 STATUS "INBOX.&ACAAIAAgACAAIAAgACAAIAAgACAAIAAg-" (UNSEEN)
* STATUS "INBOX.&ACAAIAAgACAAIAAgACAAIAAgACAAIAAg-" (UNSEEN 3)
2181 OK STATUS Completed.
2182 EXPUNGE
2182 OK EXPUNGE completed
2183 NOOP
2183 OK NOOP completed

So for your specific problem I see the '-' at the end of the encoded
string in my IMAP and nnimap logs.  Also, utf7-encode-internal inserts
the last '-' explicitly, so what you observed shouldn't be possible
unless something is eating that last character.  Really strange.  I used
the group name 已加星标.  Maybe the problem is caused by the [Gmail]
string at the beginning of the group name.

The Gnus utf7.el is pretty outdated.  Emacs offers newer facilities for
UTF-7 encoding in lisp/international/utf-7.el and Gnus should probably
use them (in imap.el).  Reiner, Katsumi, WDYT?  Feel free to cc
emacs-devel if necessary.

Ted

Re: cannot display folder with non-latin character using imap

[William Xu]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Can you please avoid top-posting?

Except he is not...  

-- 
William

http://williamxu.net9.org

You never gain something but that you lose something.
		-- Thoreau

Re: cannot display folder with non-latin character using imap

[Ted Zlatanov]

On Fri, 04 Jul 2008 12:31:20 +0900 William Xu <william.xwl@gmail.com> wrote: 

WX> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Can you please avoid top-posting?

WX> Except he is not...  

My mistake, sorry.

Ted

Re: cannot display folder with non-latin character using imap

[ssSslang]

Ted Zlatanov <tzz@lifelogs.com> writes:

> I don't see the problem here.  I do get an error but it's not due to the
> utf7 ancoding, and I think it's registry-related (everything works until
> AFTER the message is spooled, and the article doesn't show up until
> after a refresh). 

> So for your specific problem I see the '-' at the end of the encoded
> string in my IMAP and nnimap logs.  Also, utf7-encode-internal inserts
> the last '-' explicitly, so what you observed shouldn't be possible
> unless something is eating that last character.  Really strange.  I used
> the group name 已加星标.  Maybe the problem is caused by the [Gmail]
> string at the beginning of the group name.

It's the encode-coding-string's problem. I checkout the lastest cvs
code, this function works as we expected.

But the real problem still exist. I cannot select the groups with
Chinese characters. I mean I cannot enter these groups, I cannot see the
articles inside them. So, is it the nnimap backend's bug? I found some
weiqrd strings in the IMAP log, "ACAAIAAg". It repeats the same time as
the total Chinese characters(if group name contains two Chinese chars,
the name described in imap log will contains "ACAAIAAgACAAIAAg").

-- 
ssSslang

Re: cannot display folder with non-latin character using imap

[Ted Zlatanov]

On Mon, 07 Jul 2008 09:18:34 +0800 ssSslang <sssslang@gmail.com> wrote: 

s> It's the encode-coding-string's problem. I checkout the lastest cvs
s> code, this function works as we expected.

s> But the real problem still exist. I cannot select the groups with
s> Chinese characters. I mean I cannot enter these groups, I cannot see the
s> articles inside them. So, is it the nnimap backend's bug? I found some
s> weiqrd strings in the IMAP log, "ACAAIAAg". It repeats the same time as
s> the total Chinese characters(if group name contains two Chinese chars,
s> the name described in imap log will contains "ACAAIAAgACAAIAAg").

Hi,

a recent checkin to support encoding/decoding the group name in the IMAP
code in Gnus may have fixed your issue with Chinese characters.  If you
have the time, try it out.

Ted