UTF-8 support has stopped working

UTF-8 support has stopped working

[Steinar Bang]

Platform: Intel PII,
	  debian testing/unstable (sarge),
	  XEmacs 21.4 (patch 6) "Common Lisp" [Lucid] (i386-debian-linux, Mule)
	  Oort Gnus v0.18 (CVS update done today)
	  mule-ucs 0.84.99rc3

I am no longer able to see correct characters in articles with:
	Content-Type: text/plain; charset=utf-8
I see UTF-8 characters rendered as ISO 8859-1 instead.

An example of such an article, is <m3znocweb7.fsf@bfnet.com>, posted
earlier on this list.

Does anyone have an idea of how to debug this?

I suspect that it happened after a recent debian apt-get update of my
system.  I noticed that the mule-ucs package was updated.  The
mule-ucs emacs lisp code seems to still be byte compiled for
XEmacs(*), because the library 40mule-ucs is loaded during startup.

Thanx


- Steinar

(*) emacs lisp packages in debian are byte compiled separately for all
    emacsen that can use them, and don't already have them included in
    the upstream tarball

Re: UTF-8 support has stopped working

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> I am no longer able to see correct characters in articles with:
> 	Content-Type: text/plain; charset=utf-8
> I see UTF-8 characters rendered as ISO 8859-1 instead.

It seems to be working OK in Emacs 21...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: UTF-8 support has stopped working

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Steinar Bang <sb@dod.no> writes:
>> I am no longer able to see correct characters in articles with:
>> 	Content-Type: text/plain; charset=utf-8
>> I see UTF-8 characters rendered as ISO 8859-1 instead.

> It seems to be working OK in Emacs 21...

On a similar setup?  Ie. debian testing/unstable, mule-ucs from the
debian package with the same name, Gnus from CVS?

As I said, it used to work for me in XEmacs 21 as well, up until
recently.  I'm suspecting it's something a recent upgrade of the
mule-ucs package did, and I'm looking for where to start looking for
the problem.

Thanx!


- Steinar

Re: UTF-8 support has stopped working

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> Platform: Intel PII,
> 	  debian testing/unstable (sarge),
> 	  XEmacs 21.4 (patch 6) "Common Lisp" [Lucid] (i386-debian-linux, Mule)
> 	  Oort Gnus v0.18 (CVS update done today)
> 	  mule-ucs 0.84.99rc3

> I am no longer able to see correct characters in articles with:
> 	Content-Type: text/plain; charset=utf-8
> I see UTF-8 characters rendered as ISO 8859-1 instead.

[snip!]
> I suspect that it happened after a recent debian apt-get update of
> my system.  I noticed that the mule-ucs package was updated.  The
> mule-ucs emacs lisp code seems to still be byte compiled for
> XEmacs(*), because the library 40mule-ucs is loaded during startup.

40mule-ucs does not by itself load mule-ucs.  This is the file
/etc/emacs/site-start.d/40mule-ucs.el, which only sets load-path and
stuff. 

According to /usr/share/doc/mule-ucs/changelog.Debian.gz, un-define is
no longer always loaded (40mule-ucs used to take a long time, during
Emacs and XEmacs start on my old machines).

So I tried `M-x load-library RET un-define RET', but UTF-8 was still
not decoded.  locate found that there are two un-define.elc for XEmacs
21, on my debian system:
	/usr/share/xemacs21/mule-packages/lisp/mule-ucs/un-define.elc
	/usr/share/xemacs21/site-lisp/mule-ucs/un-define.elc

The first one belongs to the xemacs21-mulesupport deb package.  The
second one doesn't belong to any package, but I'm guessing this is a
file that's byte compiled from the 
	/usr/share/emacs/site-lisp/mule-ucs/lisp/un-define.el
file of the mule-ucs deb package.

But I'm unable to find out which un-define.elc file that was loaded.
The `M-x locate-library RET' command doesn't work for me on XEmacs 21,
and never has.

Thanx!


- Steinar

Re: UTF-8 support has stopped working

[Kai Großjohann]

Steinar Bang <sb@dod.no> writes:

> But I'm unable to find out which un-define.elc file that was loaded.
> The `M-x locate-library RET' command doesn't work for me on XEmacs 21,
> and never has.

Well, C-h v load-path RET tells you where Emacs looks, and in which
order.
-- 
file-error; Data: (Opening input file no such file or directory ~/.signature)

Re: UTF-8 support has stopped working

[Steinar Bang]

>>>>> kai.grossjohann@gmx.net (Kai Großjohann):

> Well, C-h v load-path RET tells you where Emacs looks, and in which
> order.

I know that.  But the load-path in XEmacs on debian is loooong.
I prefer to let emacs itself do the job of searching through the
load-path, instead of my fallible human mind.

Unfortunately I'm unable to do that with XEmacs on debian. Is
locate-library supposed to work in XEmacs?

BTW FWIW the lack of mule-ucs support in Gnus isn't XEmacs specific.
UTF-8 encoded text/plain in Gnus, doesn't work in GNU Emacs 20.7.2
either, so I don't think the duplicate un-define.elc is the culprit.

Then again, I'm not sure if my emacs 20 has been compiled with or
without mule support.  Is that an option with emacs 20?  There doesn't
seem to be a separate emacs mule binary debian package, the way it is
for XEmacs...?

Re: UTF-8 support has stopped working

[Simon Josefsson]

Steinar Bang <sb@dod.no> writes:

> Unfortunately I'm unable to do that with XEmacs on debian. Is
> locate-library supposed to work in XEmacs?

It works here, XEmacs 21 (with mule) and mule-ucs from unstable says:

Library is file /usr/share/xemacs21/site-lisp/mule-ucs/un-define.elc

> BTW FWIW the lack of mule-ucs support in Gnus isn't XEmacs specific.
> UTF-8 encoded text/plain in Gnus, doesn't work in GNU Emacs 20.7.2
> either, so I don't think the duplicate un-define.elc is the culprit.

I tried mule-ucs in unstable now with both Emacs 20 and XEmacs 21 and
had no problems with UTF-8.  XEmacs 21 without MULE does not work
though, even in a UTF-8 locale, but I'm not sure if XEmacs without
MULE is supposed to cater for charset locales, anyone?

> Then again, I'm not sure if my emacs 20 has been compiled with or
> without mule support.  Is that an option with emacs 20?  There doesn't
> seem to be a separate emacs mule binary debian package, the way it is
> for XEmacs...?

MULE is not an option for Emacs, only XEmacs.

Re: UTF-8 support has stopped working

[Steinar Bang]

>>>>> Simon Josefsson <jas@extundo.com>:

> I tried mule-ucs in unstable now with both Emacs 20 and XEmacs 21
> and had no problems with UTF-8.

xemacs21-mule in testing is 21.4.6-8, while unstable is 21.4.12-1(*).
mule-ucs in testing is 0.84.99rc3-1, while unstable is 0.84.99rc3-2.

Perhaps I should yank a new xemacs21 and mule-ucs in from unstable?
Can you confirm that the above deb package version numbers for
unstable, are the same as the ones that works for you?

Thanx!


- Steinar


(*) BTW is the third version number of the upstream version, ie. the
    number following the second dot, a real version number?  Or is it like
    GNU Emacs in the old days, the number of times emacs is built from the
    same unpacked and configured source?

Re: UTF-8 support has stopped working

[Simon Josefsson]

Steinar Bang <sb@dod.no> writes:

>>>>>> Simon Josefsson <jas@extundo.com>:
>
>> I tried mule-ucs in unstable now with both Emacs 20 and XEmacs 21
>> and had no problems with UTF-8.
>
> xemacs21-mule in testing is 21.4.6-8, while unstable is 21.4.12-1(*).
> mule-ucs in testing is 0.84.99rc3-1, while unstable is 0.84.99rc3-2.

I used Emacs 20 and XEmacs 21 from stable, actually, only mule-ucs was
unstable.

> Perhaps I should yank a new xemacs21 and mule-ucs in from unstable?
> Can you confirm that the above deb package version numbers for
> unstable, are the same as the ones that works for you?

I was using XEmacs 21.4.6-8 and mule-ucs 0.84.99rc3-2.

> (*) BTW is the third version number of the upstream version, ie. the
>     number following the second dot, a real version number?  Or is it like
>     GNU Emacs in the old days, the number of times emacs is built from the
>     same unpacked and configured source?

Emacs uses the old version scheme, but for XEmacs it is real upstream
version numbers.

Re: UTF-8 support has stopped working

[Steinar Bang]

>>>>> Simon Josefsson <jas@extundo.com>:

> Steinar Bang <sb@dod.no> writes:

>> xemacs21-mule in testing is 21.4.6-8, while unstable is 21.4.12-1(*).
>> mule-ucs in testing is 0.84.99rc3-1, while unstable is 0.84.99rc3-2.

> I used Emacs 20 and XEmacs 21 from stable, actually, only mule-ucs
> was unstable.
[snip]
> I was using XEmacs 21.4.6-8 and mule-ucs 0.84.99rc3-2.

OK.  Now I've upgraded mule-ucs to 0.84.99.rc3-2.

An improvement is that UTF-8 in message headers now display
correctly, both in the summary buffer, and in the article buffer,
instead of displaying the undecoded RFC2047 coding, as these headers
did before I upgraded.

But the contents of text/plain; charset=utf-8 message parts, still
display like UTF-8 rendered with ISO 8859-1.

Hm... could it have something to do with a locale setting?  Strange
that it should work for headers, though...?

Re: UTF-8 support has stopped working

[Steinar Bang]

>>>>> Steinar Bang <sb@dod.no>:

> OK.  Now I've upgraded mule-ucs to 0.84.99.rc3-2.

> An improvement is that UTF-8 in message headers now display
> correctly, both in the summary buffer, and in the article buffer,
> instead of displaying the undecoded RFC2047 coding, as these headers
> did before I upgraded.

> But the contents of text/plain; charset=utf-8 message parts, still
> display like UTF-8 rendered with ISO 8859-1.

Er... by mistake I started GNU Emacs 20 instead of XEmacs 21.  What's
described above, is the behaviour I see with GNU Emacs 20.

With XEmacs 21, it displays both UTF-8 encoded headers and message
parts correctly. 

Thanx!


- Steinar

Re: UTF-8 support has stopped working

[Reiner Steib]

On Mon, Apr 14 2003, Steinar Bang wrote:

> With XEmacs 21, it displays both UTF-8 encoded headers and message
> parts correctly.

Really?  Can you see Markus Kuhn's utf-8 demo file in XEmacs 21.4 more
or less correctly?  See
<news://news.gmane.org/utf-8-demo.fsf@marauder.physik.uni-ulm.de>.

I tried and I saw lots of `?' and other errors, e.g.:

  ‚deutsche‘ „Anführungszeichen“, the euro symbol: 14.95 €

was shown as:

  ?deutsche‘ ?Anführungszeichen“, the euro symbol: 14.95 $
  ^          ^                                           ^

But this is not a Gnus issue (I get the same opening the demo file
directly in XEmacs).  

In Emacs 21.3, I see some rectangles (I guess this is due to missing
fonts) in: Mathematics and sciences, APL, Georgian, Thai (UCS Level
2), Ethiopian, Compact font selection example text and Greetings in
various languages.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: UTF-8 support has stopped working

[Steinar Bang]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

[Note! Gnus insists making this message 7 parts because of charset
       issues.  Should it automatically have selected utf-8?  I tried
       doing `C-x RET f utf-8 RET', but it didn't seem to make any
       difference]

>>>>> Reiner Steib <4.uce.03.r.s@nurfuerspam.de>:

> On Mon, Apr 14 2003, Steinar Bang wrote:

>> With XEmacs 21, it displays both UTF-8 encoded headers and message
>> parts correctly.

> Really?  Can you see Markus Kuhn's utf-8 demo file in XEmacs 21.4 more
> or less correctly?  See
> <news://news.gmane.org/utf-8-demo.fsf@marauder.physik.uni-ulm.de>.

I see a suprising amount of stuff seemingly correctly.  Ie. I wasn't
aware that I had the fonts to see it correctly

> I tried and I saw lots of `?' and other errors, e.g.:

>   ?deutsche‘ 

[-- Attachment #2: Type: text/plain, Size: 18 bytes --]

?Anführungszeichen

[-- Attachment #3: Type: text/plain, Size: 33 bytes --]

“, the euro symbol: 14.95 

[-- Attachment #4: Type: text/plain, Size: 348 bytes --]

€

I see question marks before "deutsche" and "Anführungszeichen", I see
a rectangle immediately following "deutsche".  I see a double quote
character with the thickest end at the bottom and curling towards the
right (is this a "german quote"?) just before the comma, and I see the
Euro symbol (ie. not the dollar symbol you see).

> was shown as:

[-- Attachment #5: Type: text/plain, Size: 16 bytes --]


>   ?deutsche‘ 

[-- Attachment #6: Type: text/plain, Size: 18 bytes --]

?Anführungszeichen

[-- Attachment #7: Type: text/plain, Size: 2164 bytes --]

“, the euro symbol: 14.95 $
>   ^          ^                                             ^

> But this is not a Gnus issue (I get the same opening the demo file
> directly in XEmacs).  

> In Emacs 21.3, I see some rectangles (I guess this is due to missing
> fonts)

I would have guessed that missing fonts was the root of the entire
problem.  You're saying it's not?

> in: Mathematics and sciences,

Some symbols shown correctly.  Some just shown as question marks.
No rectangles.  In Linguistics and dictionaries, I see some characters
correctly, and some as tilde ie. "~".

> APL,

Some OK.  Some as question marks.  In the Nicer typography in plain
text files, I see some rectangles for the first time, for the single
quotes.   Greek looks... well,... Greek to me.

> Georgian,

All question marks, except for the text "Unicode" and some numbers.
Russian looks OK.  Cyrillic characters, and no unknown characters.

> Thai (UCS Level 2),

I see only tilde characters.

> Ethiopian,

I see only tilde characters.  Runes and Braille are all question
marks. 

> Compact font selection example text and Greetings in various
> languages.

Both look OK.

In any case, we're not seeing the same thing with the same software.
So the difference in what we're seeing either comes from different
versions of the software, or a different selection of fonts installed.

I'm guessing at fonts.  See the end of the message for the missing
font warnings I got.

Why some missing characters showed up as rectangles, some as question
mark, and some as tilde, I have no idea.


- Steinar

Missing font warnings when opening Markus Kuhn's utf-8 demo file:

(1) (font/warning) Unable to instantiate font for face default, charset ipa

(2) (font/warning) Unable to instantiate font for face default, charset chinese-cns11643-1

(3) (font/warning) Unable to instantiate font for face default, charset chinese-big5-1

(4) (font/warning) Unable to instantiate font for face default, charset japanese-jisx0212

(5) (font/warning) Unable to instantiate font for face default, charset thai-tis620

(6) (font/warning) Unable to instantiate font for face default, charset ethiopic

Re: UTF-8 support has stopped working

[Reiner Steib]

On Tue, Apr 15 2003, Steinar Bang wrote:

> [Note! Gnus insists making this message 7 parts because of charset
>        issues.  Should it automatically have selected utf-8?  

Yes.  I'm neither an expert on XEmacs nor on Unicode.  But XEmacs'
decoding of my article seems buggy.  Part 2 and part 7 of your message
are charset=iso-2022-jp (Japanese), but mine didn't contain Japanese
characters.

>>>>>> Reiner Steib <4.uce.03.r.s@nurfuerspam.de>:
>
>> On Mon, Apr 14 2003, Steinar Bang wrote:
>
>>> With XEmacs 21, it displays both UTF-8 encoded headers and message
>>> parts correctly.
>
>> Really?  Can you see Markus Kuhn's utf-8 demo file in XEmacs 21.4 more
>> or less correctly?  See
>> <news://news.gmane.org/utf-8-demo.fsf@marauder.physik.uni-ulm.de>.
>
> I see a suprising amount of stuff seemingly correctly.  Ie. I wasn't
> aware that I had the fonts to see it correctly

> I see question marks before "deutsche" and "Anführungszeichen", I
> see a rectangle immediately following "deutsche".  I see a double
> quote character with the thickest end at the bottom and curling
> towards the right (is this a "german quote"?)  just before the
> comma,

XEmacs uses a Japanese character for the low (i.e. left) double
quotation mark (after "Anführungszeichen"), which looks similar to the
correct character (but much more fat; and it's a `wide' character).

> and I see the Euro symbol (ie. not the dollar symbol you see).

Strange: Somebody in dcsg (German Gnus newsgroup) reported to see a
dollar too with Oort Gnus v0.10 / XEmacs 21.4 (Common Lisp).  And I
see a dollar with XEmacs 21.4.12 and a fresh CVS Gnus (or with 5.8.8),
too.  <news:yxzvfxogjqj.fsf@d79qxd0j.bifab.de>

>> But this is not a Gnus issue (I get the same opening the demo file
>> directly in XEmacs).  
>
>> In Emacs 21.3, I see some rectangles (I guess this is due to missing
>> fonts)
>
> I would have guessed that missing fonts was the root of the entire
> problem.  You're saying it's not?
[...]
> In any case, we're not seeing the same thing with the same software.

That's only the case for the EUR<->$ issue, isn't it?  (XEmacs and
Emacs are _not_ the same software of course.)

> So the difference in what we're seeing either comes from different
> versions of the software, or a different selection of fonts installed.
>
> I'm guessing at fonts.  See the end of the message for the missing
> font warnings I got.

I use a font server here with lots of fonts.  Although I get no font
warnings at all with XEmacs, I don't see the correct quotation marks
and EUR sign.  An the conversion to Japanese characters is very
strange, IMHO.  My impression is, that this is due to a buggy (or
incomplete) UTF-8 implementation in XEmacs.

BTW: Someone using XEmacs 21.5.11 and Oort Gnus 0.15 reported to see a
big, fat equal signs instead of the low quotaion marks and other
characters.  He also had to split the response in many parts, see...
<v91y0chhra.fsf@marauder.physik.uni-ulm.de> (test with some UTF-8
chars) and it followups:
<87of3fmuh0.fsf@520075220525-0001.dialin.t-online.de> (split)
<v97ka28j2q.fsf@marauder.physik.uni-ulm.de> (`M-x describe-char-after
RET'-analysis of some of the CJK chars of the previous message).

> Why some missing characters showed up as rectangles, some as question
> mark, and some as tilde, I have no idea.

I guess you should ask this in some XEmacs list or group.  The same
problems also happen when you look at Markus Kuhn's utf-8 demo file
(without Gnus) in XEmacs.
<URL:http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt>

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: UTF-8 support has stopped working

[Simon Josefsson]

Steinar Bang <sb@dod.no> writes:

>>>>>> Steinar Bang <sb@dod.no>:
>
>> OK.  Now I've upgraded mule-ucs to 0.84.99.rc3-2.
>
>> An improvement is that UTF-8 in message headers now display
>> correctly, both in the summary buffer, and in the article buffer,
>> instead of displaying the undecoded RFC2047 coding, as these headers
>> did before I upgraded.
>
>> But the contents of text/plain; charset=utf-8 message parts, still
>> display like UTF-8 rendered with ISO 8859-1.
>
> Er... by mistake I started GNU Emacs 20 instead of XEmacs 21.  What's
> described above, is the behaviour I see with GNU Emacs 20.
>
> With XEmacs 21, it displays both UTF-8 encoded headers and message
> parts correctly. 

FWIW, utf-8 worked for me in Emacs 20 (from stable) with mule-ucs
(from unstable).  I only tested bodies, not headers.