Unknown charset: gbk

Unknown charset: gbk

[jidanni]

 Content-Type: text/plain; charset=GBK
caused
 Unknown charset: gbk
so I had to use
 munpack -t && iconv -f gbk part1
to read the message. gnus-version "Gnus v5.11".

Re: Unknown charset: gbk

[Reiner Steib]

On Mon, Oct 08 2007, jidanni@jidanni.org wrote:

>  Content-Type: text/plain; charset=GBK
> caused
>  Unknown charset: gbk
> so I had to use
>  munpack -t && iconv -f gbk part1
> to read the message. gnus-version "Gnus v5.11".

To reproduce the problem and to test a fix, it would help if you'd
give us the Message-ID (and newsgroup) of such articles.

Hopefully fixed in CVS (trunk and v5-10) by this patch (at least in
Emacs 22):

--8<---------------cut here---------------start------------->8---
--- mm-util.el	04 Oct 2007 20:22:22 +0200	7.67
+++ mm-util.el	08 Oct 2007 20:06:31 +0200	
@@ -253,6 +253,12 @@
     ,@(when (and (not (mm-coding-system-p 'windows-31j))
 		 (mm-coding-system-p 'cp932))
 	'((windows-31j . cp932)))
+    ;; Charset name: GBK, Charset aliases: CP936, MS936, windows-936
+    ;; http://www.iana.org/assignments/charset-reg/GBK
+    ;; Emacs 22.1 has cp936, but not gbk, so we alias it:
+    ,@(when (and (not (mm-coding-system-p 'gbk))
+		 (mm-coding-system-p 'cp936))
+	'((gbk . cp936)))
     ;; ISO8859-1 is ISO-8859-1
     ,@(when (and (not (mm-coding-system-p 'iso8859-1))
 		 (mm-coding-system-p 'iso-8859-1))
--8<---------------cut here---------------end--------------->8---

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Unknown charset: gbk

[Reiner Steib]

On Mon, Oct 08 2007, Reiner Steib wrote:

> To reproduce the problem and to test a fix, it would help if you'd
> give us the Message-ID (and newsgroup) of such articles.
[...]
> Hopefully fixed in CVS (trunk and v5-10) by this patch (at least in
> Emacs 22):

off-list, jidanni@jidanni.org wrote:

| Here's one for you to test with. Thanks.
| It should look like
| $ iconv -f gbk part1
| 我建議 也可以去 gcin 的 mailing list 問問看...

It looks like this for me:

: 我建\327h 也可以去 gcin 的 mailing list \206\226\206\226看

I don't know the reason for the \NNN chars.  Maybe only a problem with
the forwarded mail?  If not, you might try to produce some correct GBK
file and open it in Emacs to see if Emacs can handle it correctly.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Unknown charset: gbk

[Reiner Steib]

On Wed, Oct 10 2007, Reiner Steib wrote:

> It looks like this for me:
>
> : 我建\327h 也可以去 gcin 的 mailing list \206\226\206\226看
>
> I don't know the reason for the \NNN chars.  

With your forwarded mail, I get a similar result: Chinese characters,
but also \nnn.

So, I'd still suggest...

> Maybe only a problem with the forwarded mail?  If not, you might try
> to produce some correct GBK file and open it in Emacs to see if
> Emacs can handle it correctly.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Unknown charset: gbk

[William Xu]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> off-list, jidanni@jidanni.org wrote:
>
> | Here's one for you to test with. Thanks.
> | It should look like
> | $ iconv -f gbk part1
> | 我建議 也可以去 gcin 的 mailing list 問問看...
>
> It looks like this for me:
>
> : 我建\327h 也可以去 gcin 的 mailing list \206\226\206\226看
>
> I don't know the reason for the \NNN chars.  Maybe only a problem with
> the forwarded mail?  If not, you might try to produce some correct GBK
> file and open it in Emacs to see if Emacs can handle it correctly.

jidanni's gbk texts displays correctly here. (emacs cvs trunk)

So i think that you can' display all the characters may be simply a
missing font issue.

-- 
William

http://williamxu.net9.org

Re: Unknown charset: gbk

[Katsumi Yamaoka]

>>>>> William Xu <william.xwl@gmail.com> wrote:

> Reiner Steib <reinersteib+gmane@imap.cc> writes:

>> off-list, jidanni@jidanni.org wrote:
>>
>>| Here's one for you to test with. Thanks.
>>| It should look like
>>| $ iconv -f gbk part1
>>| 我建議 也可以去 gcin 的 mailing list 問問看...
>>
>> It looks like this for me:
>>
>> : 我建\327h 也可以去 gcin 的 mailing list \206\226\206\226看
>>
>> I don't know the reason for the \NNN chars.  Maybe only a problem with
>> the forwarded mail?  If not, you might try to produce some correct GBK
>> file and open it in Emacs to see if Emacs can handle it correctly.

> jidanni's gbk texts displays correctly here. (emacs cvs trunk)

> So i think that you can' display all the characters may be simply a
> missing font issue.

Aren't they displayed with boxes if the lack of font is the problem?
I think the best is to post a gzip'd example message if possible.

BTW, I have the mule-gbk package[1] installed (for checking
whether emacs-w3m decodes gbk web pages correctly).  It provides
the gbk coding system, moreover modifies `mm-charset-synonym-alist'!
Though I don't know which of mule-gbk, Emacs' cp936 and Unicode-2's
gbk is better, since I'm ignorant in Chinese.

[1] http://mule-gbk.sourceforge.net/

;; ~/.emacs
(if (and (not (featurep 'xemacs))
	 (>= emacs-major-version 22)
	 (string-match "\\`5" mule-version)) ;; Exclude Unicode-2.
    (require 'mule-gbk))

Re: Unknown charset: gbk

[William Xu]

[-- Attachment #1: Type: text/plain, Size: 502 bytes --]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Aren't they displayed with boxes if the lack of font is the problem?

Oh, right. And that why jidanni's chinese in Reiner's mail displays
correctly here is because Reiner sent them encoded with utf-8..

> I think the best is to post a gzip'd example message if possible.

I created `gbk_test' file(attached) with `iconv -f utf-8 -t gbk'. Now if
i try to open it inside emacs, i got exactly same results as Reiner's..

-- 
William

http://williamxu.net9.org

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gbk_test --]
[-- Type: text/plain; charset=utf-8, Size: 47 bytes --]

ÎÒ½¨×h Ò²¿ÉÒÔÈ¥ gcin µÄ mailing list †–†–¿´...

Re: Unknown charset: gbk

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 933 bytes --]

>>>>> William Xu wrote:

> I created `gbk_test' file(attached) with `iconv -f utf-8 -t gbk'. Now if
> i try to open it inside emacs, i got exactly same results as Reiner's..

Er, utf-8 converts the charset of text.  For instance, the
following line is displayed with all the Japanese letters in
my Emacs (`current-language-environment' is "Japanese"):

>>>>> Reiner Steib <reinersteib+gmane@imap.cc> wrote:
>| 我建議 也可以去 gcin 的 mailing list 問問看...

(split-char ?我) => (japanese-jisx0208 50 102)
(split-char ?建) => (japanese-jisx0208 55 122)

You get different results?

It suggests that it will probably be meaningless to examine the
Chinese text that was encoded by utf-8 once.  So, I think what
we need is the original message.

The gbk text that I made using Emacs Unicode-2 is in the next
page.  It looks fine in Unicode-2, but is displayed with all
boxes in Emacs 23.0.50. ;-)
\f

[-- Attachment #2: HELLO --]
[-- Type: text/plain, Size: 16 bytes --]

你好,元气,开发

Re: Unknown charset: gbk

[Katsumi Yamaoka]

>>>>> Katsumi Yamaoka wrote:

> The gbk text that I made using Emacs Unicode-2 is in the next
> page.  It looks fine in Unicode-2, but is displayed with all
> boxes in Emacs 23.0.50. ;-)

The gbk text was displayed correctly if Emacs 23.0.50 did not
load the mule-gbk package.  So, maybe it's better to use Emacs's
cp936 than mule-gbk, and Reiner's solution was right.

(I'll be inactive until Monday.  Sorry.)

Re: Unknown charset: gbk

[William Xu]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Er, utf-8 converts the charset of text.  For instance, the
> following line is displayed with all the Japanese letters in
> my Emacs (`current-language-environment' is "Japanese"):

Here: current-language-environment => "Chinese-GB"

>>>>>> Reiner Steib <reinersteib+gmane@imap.cc> wrote:
>>| 我建議 也可以去 gcin 的 mailing list 問問看...
>
> (split-char ?我) => (japanese-jisx0208 50 102)
> (split-char ?建) => (japanese-jisx0208 55 122)
>
> You get different results?

Yes, it depends on the language environment.

(split-char ?我) => (chinese-gb2312 78 82)
(split-char ?建) => (chinese-gb2312 61 40)

(split-char ?議) => (chinese-big5-1 92 63)

So here's the interesting part. Instead decoding with gbk, it has chosen
chinese-big5-1..(the traditional chinese encoding system)

> It suggests that it will probably be meaningless to examine the
> Chinese text that was encoded by utf-8 once.  So, I think what
> we need is the original message.
>
> The gbk text that I made using Emacs Unicode-2 is in the next
> page.  It looks fine in Unicode-2, but is displayed with all
> boxes in Emacs 23.0.50. ;-)

> 你好,元气,开发

Like your next post, it displays okay in 23.0.50.

While, actually the text you made are all covered by gb2312. gbk is
somehow is superset of gb2312. So `C-u C-x =' would say that their
charset is chinese-gb2312..

So to test gbk, maybe should test against some charcters not covered by
gb2312. Usually, traditional chinese characters are not covered by
gb2312. Like this one, (split-char ?議) => (chinese-big5-1 92 63)
or 開発(kaihatu, in japanese).

-- 
William

http://williamxu.net9.org

Re: Unknown charset: gbk

[jidanni]

OK, here's another GBK message, wrapped super safely with shar(1).

#!/bin/sh
# This is a shell archive (produced by GNU sharutils 4.6.3).
# To extract the files from this archive, save it to some FILE, remove
# everything before the `#!/bin/sh' line above, then type `sh FILE'.
#
lock_dir=_sh10243
# Made on 2007-10-20 05:48 CST by <jidanni@jidanni.org>.
# Source directory was `/tmp'.
#
# Existing files will *not* be overwritten, unless `-c' is specified.
#
# This shar contains:
# length mode       name
# ------ ---------- ------------------------------------------
#   2206 -rw-r--r-- gbk_message
#
MD5SUM=${MD5SUM-md5sum}
f=`${MD5SUM} --version | egrep '^md5sum .*(core|text)utils'`
test -n "${f}" && md5check=true || md5check=false
${md5check} || \
  echo 'Note: not verifying md5sums.  Consider installing GNU coreutils.'
save_IFS="${IFS}"
IFS="${IFS}:"
gettext_dir=FAILED
locale_dir=FAILED
first_param="$1"
for dir in $PATH
do
  if test "$gettext_dir" = FAILED && test -f $dir/gettext \
     && ($dir/gettext --version >/dev/null 2>&1)
  then
    case `$dir/gettext --version 2>&1 | sed 1q` in
      *GNU*) gettext_dir=$dir ;;
    esac
  fi
  if test "$locale_dir" = FAILED && test -f $dir/shar \
     && ($dir/shar --print-text-domain-dir >/dev/null 2>&1)
  then
    locale_dir=`$dir/shar --print-text-domain-dir`
  fi
done
IFS="$save_IFS"
if test "$locale_dir" = FAILED || test "$gettext_dir" = FAILED
then
  echo=echo
else
  TEXTDOMAINDIR=$locale_dir
  export TEXTDOMAINDIR
  TEXTDOMAIN=sharutils
  export TEXTDOMAIN
  echo="$gettext_dir/gettext -s"
fi
if (echo "testing\c"; echo 1,2,3) | grep c >/dev/null
then if (echo -n test; echo 1,2,3) | grep n >/dev/null
     then shar_n= shar_c='
'
     else shar_n=-n shar_c= ; fi
else shar_n= shar_c='\c' ; fi
f=shar-touch.$$
st1=200112312359.59
st2=123123592001.59
st2tr=123123592001.5 # old SysV 14-char limit
st3=1231235901

if touch -am -t ${st1} ${f} >/dev/null 2>&1 && \
   test ! -f ${st1} && test -f ${f}; then
  shar_touch='touch -am -t $1$2$3$4$5$6.$7 "$8"'

elif touch -am ${st2} ${f} >/dev/null 2>&1 && \
   test ! -f ${st2} && test ! -f ${st2tr} && test -f ${f}; then
  shar_touch='touch -am $3$4$5$6$1$2.$7 "$8"'

elif touch -am ${st3} ${f} >/dev/null 2>&1 && \
   test ! -f ${st3} && test -f ${f}; then
  shar_touch='touch -am $3$4$5$6$2 "$8"'

else
  shar_touch=:
  echo
  ${echo} 'WARNING: not restoring timestamps.  Consider getting and'
  ${echo} 'installing GNU `touch'\'', distributed in GNU coreutils...'
  echo
fi
rm -f ${st1} ${st2} ${st2tr} ${st3} ${f}
#
if test ! -d ${lock_dir}
then : ; else ${echo} 'lock directory '${lock_dir}' exists'
  exit 1
fi
if mkdir ${lock_dir}
then ${echo} 'x - created lock directory `'${lock_dir}\''.'
else ${echo} 'x - failed to create lock directory `'${lock_dir}\''.'
  exit 1
fi
# ============= gbk_message ==============
if test -f 'gbk_message' && test "$first_param" != -c; then
  ${echo} 'x -SKIPPING gbk_message (file already exists)'
else
${echo} 'x - extracting gbk_message (text)'
  sed 's/^X//' << 'SHAR_EOF' > 'gbk_message' &&
XFrom nobody Sat Oct 20 05:36:24 2007
Delivery-date: Sat, 20 Oct 2007 02:56:54 +0800
XX-Original-To: jidanni@jidanni.org
XX-Apparently-To: chewing-devel@googlegroups.com
XFrom: Keimoto <me@keimoto.net>
To: Chewing IM Development <chewing-devel@googlegroups.com>
Subject: Re: =?ISO-2022-JP?B?GyRCNFRNLUMvOl8bKEI=?= =?ISO-2022-JP?B?TGludXgbJEI+ZU8uGyhC?= =?ISO-2022-JP?B?Q2hld2luZxskQkUqISkbKEI=?=
Date: Fri, 19 Oct 2007 02:44:33 -0700
Message-ID: <1192787073.155425.167590@e34g2000pro.googlegroups.com>
User-Agent: G2/1.0
Mime-Version: 1.0
Content-Type: text/plain; charset=GBK
Content-Transfer-Encoding: base64
Reply-To: chewing-devel@googlegroups.com
Sender: chewing-devel@googlegroups.com
Mailing-List: list chewing-devel@googlegroups.com;
X	contact chewing-devel-owner@googlegroups.com
List-Id: <chewing-devel.googlegroups.com>
List-Post: <mailto:chewing-devel@googlegroups.com>
List-Help: <mailto:chewing-devel-help@googlegroups.com>
List-Unsubscribe: <http://googlegroups.com/group/chewing-devel/subscribe>,
X	<mailto:chewing-devel-unsubscribe@googlegroups.com>
Lines: 18
X
X
T24gMTDUwjE5yNUsIMnPzuc0lXIyObfWLCBLYW5ydSA8Y2thLi4uQGdtYWlsLmNvbT4gd3JvdGU6
Cj4gucDTi8rHIHByZWVkaXQgYXJlYSDpTLbIo6y68t9419bUfpS1wb+1xNOLy+PT0NVgLi4uCj4K
PiC/ydLUs6/fQLe9w+bW+MrWIDpwCgrO0tLRvZvV+4KAY2hvaWNlLmOhomNoYXIuY6GiY2hld2lu
Z2lvLmOhomNoZXdpbmd1dGlsLmOhomF1eF9tb2QuY7a8v7TBy9K7senT1tK7semjrLK7lOC12LjE
oaIK1IehorjEoaLUhy4uLt+AysebXdPQ1dK1vaGjCgrO0tNYtcPX7sbmuda1xMrHo6xzeW1ib2wv
mMv8Y7XE3ZTI68rHIM3ijJG1xKOsm13T0N9AgoCGlu59o7vQobXcsruyxaOs1/e24Le9w+axyN1e
yNTOtMTcveKbUaGjCgrP69WIhpZ3aW4zMrDmtcTfgJX+0/a1vd9AgoCGlu59IKO/CgpLZWltb3Rv
IKSxpKSk4qTICgoKLS1+LS1+LS0tLS0tLS0tfi0tfi0tLS1+LS0tLS0tLS0tLS0tfi0tLS0tLS1+
LS1+LS0tLX4KxPrK1bW9tMvgXbz+o6zKx9LynunE+tOG6YbByyBHb29nbGUgvlfJz9WTia+1xKG4
Q2hld2luZyBJTSBEZXZlbG9wbWVudKG5yLoKvU2howogyOfSqtTatMvIur1Nj4jZTsH00dSjrNWI
vMTritfT4F28/tbBIGNoZXdpbmctZGV2ZWxAZ29vZ2xlZ3JvdXBzLmNvbQogyOfSqsihz/vThumG
tMvIur1No6zViLzE64rX0+BdvP7WwSBjaGV3aW5nLWRldmVsLXVuc3Vic2NyaWJlQGdvb2dsZWdy
b3Vwcy5jb20KIMjn0Oi4/Lbg33jtl6Os1YjU7NRMtMvIur1No7podHRwOi8vZ3JvdXBzLmdvb2ds
ZS5jb20udHcvZ3JvdXAvY2hld2luZy1kZXZlbD9obD16aC1UVwqhowotfi0tLS0tLS0tLS1+LS0t
LX4tLS0tfi0tLS1+LS0tLS0tfi0tLS1+LS0tLS0tfi0tfi0tLQoK
X
X
SHAR_EOF
  (set 20 07 10 20 05 37 59 'gbk_message'; eval "$shar_touch") &&
  chmod 0644 'gbk_message'
if test $? -ne 0
then ${echo} 'restore of gbk_message failed'
fi
  if ${md5check}
  then (
       ${MD5SUM} -c >/dev/null 2>&1 || ${echo} 'gbk_message: MD5 check failed'
       ) << SHAR_EOF
96edfb6ef5c5e2f19c673165fcecb7dc  gbk_message
SHAR_EOF
  else
test `LC_ALL=C wc -c < 'gbk_message'` -ne 2206 && \
  ${echo} 'restoration warning:  size of gbk_message is not 2206'
  fi
fi
if rm -fr ${lock_dir}
then ${echo} 'x - removed lock directory `'${lock_dir}\''.'
else ${echo} 'x - failed to remove lock directory `'${lock_dir}\''.'
  exit 1
fi
exit 0

Re: Unknown charset: gbk

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

>>>>> jidanni@jidanni.org wrote:

> OK, here's another GBK message, wrapped super safely with shar(1).

At least for this message, making `gbk' be an alias to `cp936'
in Gnus is a bad idea.  Reiner?  I could reproduce \NNN using
Emacs 22.1 and the current Emacs trunk.  The cp936 coding system
in those versions of Emacsen seems to be incomplete for gbk text.

OTOH, Unicode 2 (i.e. Emacs 23.0.60) and the iconv command (both
support gbk) look good.  Therefore, I tried creating the gbk
coding system for Mule version 5 (i.e. Emacs 21.1-23.0.50) using
iconv.  I use:

$ iconv --version
iconv (GNU libc) 2.6

and Mule-UCS for Emacs 21.x.  If you try this module, you have
to load (or require) it before loading Gnus.


[-- Attachment #2: mule5-gbk.el --]
[-- Type: application/emacs-lisp, Size: 2018 bytes --]

Re: Unknown charset: gbk

[Reiner Steib]

On Mon, Oct 22 2007, Katsumi Yamaoka wrote:

> At least for this message, making `gbk' be an alias to `cp936'
> in Gnus is a bad idea.  Reiner?  I could reproduce \NNN using
> Emacs 22.1 and the current Emacs trunk.  

But isn't it still better than "Unknown charset: gbk"?  (It's a no-op
if the coding system gbk is defined, e.g. with your `mule5-gbk.el',
isn't it?)

> The cp936 coding system in those versions of Emacsen seems to be
> incomplete for gbk text.

Would you please report this on emacs-devel (maybe Cc Handa-san)?
Maybe it can be fixed for Emacs 22.2.

> OTOH, Unicode 2 (i.e. Emacs 23.0.60) 

Does Emacs unicode define an alias for gbk?  If not, it probably
should.

> and the iconv command (both support gbk) look good.  

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Unknown charset: gbk

[Katsumi Yamaoka]

>>>>> Reiner Steib wrote:
> On Mon, Oct 22 2007, Katsumi Yamaoka wrote:

>> At least for this message, making `gbk' be an alias to `cp936'
>> in Gnus is a bad idea.  Reiner?  I could reproduce \NNN using
>> Emacs 22.1 and the current Emacs trunk.

> But isn't it still better than "Unknown charset: gbk"?  (It's a no-op
> if the coding system gbk is defined, e.g. with your `mule5-gbk.el',
> isn't it?)

Okay. :)

>> The cp936 coding system in those versions of Emacsen seems to be
>> incomplete for gbk text.

> Would you please report this on emacs-devel (maybe Cc Handa-san)?
> Maybe it can be fixed for Emacs 22.2.

I'll do so.

>> OTOH, Unicode 2 (i.e. Emacs 23.0.60)

> Does Emacs unicode define an alias for gbk?  If not, it probably
> should.

In Emacs Unicode-2, gbk, cp936, and windows-936 are aliases to
chinese-gbk, not to chinese-iso-8bit.  (language/chinese.el)