Status of XEmacs' (21.5) UTF-8 capabilities

Status of XEmacs' (21.5) UTF-8 capabilities

[Reiner Steib]

Hi,

users in the German Gnus newsgroup are asking from time to time if
Gnus (5.10) is able to handle UTF-8 articles.  "handle" includes
reading and posting, i.e. followup to utf-8 articles.

While GNU Emacs 21.3 (latest stable release) can handle at least the
"Western European" part (Latin1, windows-1252) quite well, it seems
that XEmacs 21.5 produces multipart/mixed posting (where some
characters are incorrect [1]) or "windows-1252 labeled as Latin-1" (b)
instead of utf-8.  See the followups to my test articles in
<news:gmane.test> (<URL:http://thread.gmane.org/gmane.test/586>):

(a) <news://news.gmane.org/utf-8-demo.fsf@marauder.physik.uni-ulm.de>
    Markus Kuhn's utf-8 demo file

    XEmacs produces multipart/mixed with 726 (sic!) parts here!
    (a1) <news:85ptdweo5q.fsf@lorch-net.stlo.de>
    
(b) <news://news.gmane.org/windows-1252-cs-w1252@marauder.physik.uni-ulm.de>
    Latin-1 and windows-1252 w/ charset=windows-1252

    XEmacs produces "windows-1252 labeled as Latin-1"
    (b1) <news:85y8skeo7w.fsf_-_@lorch-net.stlo.de>

(c) <news://news.gmane.org/windows-1252-cs-utf-8@marauder.physik.uni-ulm.de>
    Latin-1 and windows-1252 w/ charset=utf-8

    XEmacs produces multipart/mixed with 29 parts.
    (c1) <news:85u138eo6x.fsf_-_@lorch-net.stlo.de>

(d) <news://news.gmane.org/latin-1-plus-1-cs-utf-8@marauder.physik.uni-ulm.de>
    Latin-1 and some windows-1252 chars w/ charset=utf-8

    XEmacs produces multipart/mixed with 20 parts.
    (d1) private mail from Steffen.

Is Steffen Lorch's (he did the tests with XEmacs) and my assumption
correct, that (Gnus with) XEmacs cannot post UTF-8 articles?  If not,
could you followup to the abovementioned articles in <news:gmane.test>
(with a full-quote of my text!) to prove me wrong and give the XEmacs
Users some instructions for their setup?

(IIRC, Steffen already has the latin-unity package installed.)

Bye, Reiner.

[1] <news:85u138eo6x.fsf_-_@lorch-net.stlo.de>

;; 82 = U+201A : SINGLE LOW-9 QUOTATION MARK
;; 83 = U+0192 : LATIN SMALL LETTER F WITH HOOK
;; 84 = U+201E : DOUBLE LOW-9 QUOTATION MARK
;; 88 = U+02C6 : MODIFIER LETTER CIRCUMFLEX ACCENT
;; 8B = U+2039 : SINGLE LEFT-POINTING ANGLE QUOTATION MARK
(;; 96 = U+2013 : EN DASH ?)
(;; 97 = U+2014 : EM DASH ?)
;; 98 = U+02DC : SMALL TILDE
;; 9B = U+203A : SINGLE RIGHT-POINTING ANGLE QUOTATION MARK

(and maybe others)
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Status of XEmacs' (21.5) UTF-8 capabilities

[Mike FABIAN]

[-- Attachment #1: Type: text/plain, Size: 346 bytes --]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> さんは書きました:

> Is Steffen Lorch's (he did the tests with XEmacs) and my assumption
> correct, that (Gnus with) XEmacs cannot post UTF-8 articles?

You need to patch Gnus v5.10.6 a little bit. Attached is the small patch
for Gnus v5.10.6 which I use to make XEmacs be able to post in UTF-8:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: mm-util.el.patch --]
[-- Type: text/x-patch, Size: 634 bytes --]

Index: mm-util.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/mm-util.el,v
retrieving revision 6.53
diff -u -r6.53 mm-util.el
--- mm-util.el	6 Jan 2003 04:31:23 -0000	6.53
+++ mm-util.el	31 Jan 2003 21:35:17 -0000
@@ -541,7 +541,9 @@
     (if (and (memq 'iso-2022-jp-2 charsets)
 	     (memq 'iso-2022-jp-2 hack-charsets))
 	(setq charsets (delq 'iso-2022-jp charsets)))
-    charsets))
+    (if (> (length charsets) 1)
+	'(utf-8) 
+      charsets)))
 
 (defmacro mm-with-unibyte-buffer (&rest forms)
   "Create a temporary buffer, and evaluate FORMS there like `progn'.

[-- Attachment #3: Type: text/plain, Size: 100 bytes --]


-- 
Mike FABIAN   <mfabian@suse.de>   http://www.suse.de/~mfabian
睡眠不足はいい仕事の敵だ。

Re: Status of XEmacs' (21.5) UTF-8 capabilities

[Mike FABIAN]

Mike FABIAN <mfabian@suse.de> さんは書きました:

> You need to patch Gnus v5.10.6 a little bit. Attached is the small patch
> for Gnus v5.10.6 which I use to make XEmacs be able to post in UTF-8:

OK, with my patch I can post in UTF-8.

But now I would like to know how I can disable this for certain
recipients.

Unfortunately most web-mail services cannot yet handle UTF-8.
If I write a mail containing Japanese *and* German to somebody
with a

    @yahoo.co.jp, @hotmail.com, ...

address, and I don't remember that I must not use
real German umlauts (äüöÄÜÖß) but rather the ASCII replacements
ae, ue, oe, Ae, Ue, Oe, ss, then XEmacs/Gnus sends it as UTF-8
and the Japanese becomes completely unreadable for the recipient.

Therefore I wonder whether it is already possible with
Gnus to force a certain encoding depending on the address of the
recipient. For example, if the To: address matches one
of the above mentioned web-mailers, use ISO-2022-JP encoding,
no matter what.

This would of course destroy German Umlauts, but that usually doesn't
make the German completely unreadable (contrary to Japanese which
becomes complete garbage for the reader when his MUA doesn't support
the encoding).

Does anything like this already exist in Gnus?

-- 
Mike FABIAN   <mfabian@suse.de>   http://www.suse.de/~mfabian
睡眠不足はいい仕事の敵だ。