charset=macintosh

charset=macintosh

[Karl Eichwalder]

[-- Attachment #1: Type: text/plain, Size: 653 bytes --]

Oort Gnus v0.16
GNU Emacs 21.3.50.1 (i686-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2003-03-05 on tux
200 gnu.franken.de InterNetNews server INN 2.3.2 ready

I wrote to attached message using the TeX iput method (C-x RET c TeX RET).
Especially I added these quotes: \ldq \rdq and then \ldots and \euro .

Sending the message, Gnus choose this encoding:

MIME-Version: 1.0
Content-Type: text/plain; charset=macintosh
Content-Transfer-Encoding: quoted-printable

I would have expected UTF-8 since UTF-8 is more standard than
"macintosh"; I know you can set your preferences somehow -- but I'd
like to see Gnus doing the right thing out of the box.


[-- Attachment #2: Type: message/rfc822, Size: 601 bytes --]

From: Karl Eichwalder <keichwa@gmx.net>
To: ke@gnu.franken.de
Subject: encoding test
Date: Fri, 07 Mar 2003 18:28:35 +0100
Message-ID: <shzno73x30.fsf@tux.gnu.franken.de>

“quotes”
and more…
Grüß Gott.

€ 1.

-- 
ke@suse.de (work) / keichwa@gmx.net (home):              |
http://www.gnu.franken.de/ke/                            |      ,__o
Free Translation Project:                                |    _-\_<,
http://www.iro.umontreal.ca/contrib/po/HTML/             |   (*)/'(*)


[-- Attachment #3: Type: text/plain, Size: 271 bytes --]

-- 
ke@suse.de (work) / keichwa@gmx.net (home):              |
http://www.gnu.franken.de/ke/                            |      ,__o
Free Translation Project:                                |    _-\_<,
http://www.iro.umontreal.ca/contrib/po/HTML/             |   (*)/'(*)

Re: charset=macintosh

[Simon Josefsson]

Karl Eichwalder <keichwa@gmx.net> writes:

> Oort Gnus v0.16
> GNU Emacs 21.3.50.1 (i686-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
>  of 2003-03-05 on tux
> 200 gnu.franken.de InterNetNews server INN 2.3.2 ready
>
> I wrote to attached message using the TeX iput method (C-x RET c TeX RET).
> Especially I added these quotes: \ldq \rdq and then \ldots and \euro .
>
> Sending the message, Gnus choose this encoding:
>
> MIME-Version: 1.0
> Content-Type: text/plain; charset=macintosh
> Content-Transfer-Encoding: quoted-printable
>
> I would have expected UTF-8 since UTF-8 is more standard than
> "macintosh"; I know you can set your preferences somehow -- but I'd
> like to see Gnus doing the right thing out of the box.

Me too, but I'm not sure if Gnus should override Emacs' coding system
ordering, and emacs prefers mac-roman over utf-8.  The reason Emacs
doesn't list UTF-8 higher than macintosh is, supposedly, that the
UTF-8 support is not complete.  Once it is complete, the emacs
maintainers will probably change the preference ordering.

Fortunately it is easy to override the priorities within Gnus:

 '(mm-coding-system-priorities (quote (iso-latin-1 iso-latin-9 utf-8)))

Of course, the die-hard solution is to run emacs in a UTF-8 locale,
then UTF-8 is preferred everywhere.  I use this without problems, but
I do use the above line to downgrade to 8859-1/8859-15 in mail,
whenever possible.

Hm.  The UTF-8 discussion in Emacs 21.2 PROBLEMS is not present in
Emacs 21.3 prerelease, so perhaps it has been fixed.  I'll ask on
emacs-devel whether utf-8 should be preferred over mac-roman.

Re: charset=macintosh

[Karl Eichwalder]

Simon Josefsson <jas@extundo.com> writes:

> Me too, but I'm not sure if Gnus should override Emacs' coding system
> ordering, and emacs prefers mac-roman over utf-8.  The reason Emacs
> doesn't list UTF-8 higher than macintosh is, supposedly, that the
> UTF-8 support is not complete.  Once it is complete, the emacs
> maintainers will probably change the preference ordering.

Thanks for explanation!

> Fortunately it is easy to override the priorities within Gnus:
>
>  '(mm-coding-system-priorities (quote (iso-latin-1 iso-latin-9 utf-8)))
>
> Of course, the die-hard solution is to run emacs in a UTF-8 locale,
> then UTF-8 is preferred everywhere.  I use this without problems, but
> I do use the above line to downgrade to 8859-1/8859-15 in mail,
> whenever possible.

Last week I also did Emacs in a UTF-8 locale for some days and most of
the time, it was a pleasure.  You're right one should set:

'(mm-coding-system-priorities (quote (iso-latin-1 iso-latin-9 utf-8)))

If you do not try to post using latin1/15, many a lot readers are not
that happy with your postings.

I also observed that even more users don't set the Mime headers at
all!  And Gnus assumes UTF-8 for those postings and that's always
wrong -- as you might imagine ;)

> Hm.  The UTF-8 discussion in Emacs 21.2 PROBLEMS is not present in
> Emacs 21.3 prerelease, so perhaps it has been fixed.  I'll ask on
> emacs-devel whether utf-8 should be preferred over mac-roman.

Thanks for your help!

-- 
ke@suse.de (work) / keichwa@gmx.net (home):              |
http://www.gnu.franken.de/ke/                            |      ,__o
Free Translation Project:                                |    _-\_<,
http://www.iro.umontreal.ca/contrib/po/HTML/             |   (*)/'(*)

Re: charset=macintosh

[Jesper Harder]

Karl Eichwalder <keichwa@gmx.net> writes:

> If you do not try to post using latin1/15, many a lot readers are not
> that happy with your postings.

And rightly so.  RFC 2046 (section 4.1.2) says that you should use »the
"lowest common denominator" character set possible.« 

Clearly, UTF-8 doesn't comply with this.

> I also observed that even more users don't set the Mime headers at
> all!  And Gnus assumes UTF-8 for those postings

That sounds like a bug.  How can I reproduce it?

Re: charset=macintosh

[Simon Josefsson]

Karl Eichwalder <keichwa@gmx.net> writes:

> I also observed that even more users don't set the Mime headers at
> all!  And Gnus assumes UTF-8 for those postings and that's always
> wrong -- as you might imagine ;)

On decoding it should be smarter since non-ASCII and non-UTF-8 texts
typically is invalid UTF-8, so it is possible to detect and recover.
I posted some tests to gnus.test on quimby and they don't work well.
If the emacs mule engine handles the magic, it should be simple to
make Gnus do the right thing, otherwise it might require some work.

Maybe a mm-decoding-system-priorities.  Hm.

Re: charset=macintosh

[Karl Eichwalder]

Jesper Harder <harder@myrealbox.com> writes:

> And rightly so.  RFC 2046 (section 4.1.2) says that you should use »the
> "lowest common denominator" character set possible.« 

When I know, I can post UTF-8 I always do stupid things like adding
some ♥'s or a € and things like that…  Thus, wuickliy the lowest common
deniminator is UTF-8 ⌣ or ⌢

> Clearly, UTF-8 doesn't comply with this.

This depends ‡‡‡ — otherwise you are completely right ;-)

>> I also observed that even more users don't set the Mime headers at
>> all!  And Gnus assumes UTF-8 for those postings
>
> That sounds like a bug.  How can I reproduce it?

Start Emacs in a UTF-8 locale:

    LANG=en_US.UTF-8 emacs

and page thru some German newsgroup like de.rec.buecher (books) or
de.rec.fahrrad (bikes).  Every 10th or 20th article will feature those
nice \321, \345, \432 decorations :)

-- 
ke@suse.de (work) / keichwa@gmx.net (home):              |
http://www.gnu.franken.de/ke/                            |      ,__o
Free Translation Project:                                |    _-\_<,
http://www.iro.umontreal.ca/contrib/po/HTML/             |   (*)/'(*)

Re: charset=macintosh

[Karl Eichwalder]

Simon Josefsson <jas@extundo.com> writes:

> Fortunately it is easy to override the priorities within Gnus:
>
>  '(mm-coding-system-priorities (quote (iso-latin-1 iso-latin-9 utf-8)))

Yes, works nice.  Even under LANG=de_DE.UTF-8 as you said.

Simon Josefsson <jas@extundo.com> writes:

> On decoding it should be smarter since non-ASCII and non-UTF-8 texts
> typically is invalid UTF-8, so it is possible to detect and recover.
> I posted some tests to gnus.test on quimby and they don't work well.
> If the emacs mule engine handles the magic, it should be simple to
> make Gnus do the right thing, otherwise it might require some work.
>
> Maybe a mm-decoding-system-priorities.  Hm.

Yes.  Below de.* for most cases ISO-8859-1 schould work, some postings
will require ISO-8859-15 and cp-1252.

-- 
ke@suse.de (work) / keichwa@gmx.net (home):              |
http://www.gnu.franken.de/ke/                            |      ,__o
Free Translation Project:                                |    _-\_<,
http://www.iro.umontreal.ca/contrib/po/HTML/             |   (*)/'(*)

Re: charset=macintosh

[Simon Josefsson]

Karl Eichwalder <keichwa@gmx.net> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> On decoding it should be smarter since non-ASCII and non-UTF-8 texts
>> typically is invalid UTF-8, so it is possible to detect and recover.
>> I posted some tests to gnus.test on quimby and they don't work well.
>> If the emacs mule engine handles the magic, it should be simple to
>> make Gnus do the right thing, otherwise it might require some work.
>>
>> Maybe a mm-decoding-system-priorities.  Hm.
>
> Yes.  Below de.* for most cases ISO-8859-1 schould work, some postings
> will require ISO-8859-15 and cp-1252.

Heuristically deciding between those is more difficult than for UTF-8,
so `1 g' (Article -> Display -> View as different encoding), or
setting up the coding priorities in emacs the way you want, is
probably the only solution.

Re: charset=macintosh

[Jesper Harder]

Karl Eichwalder <keichwa@gmx.net> writes:

> Jesper Harder <harder@myrealbox.com> writes:
>
>>> I also observed that even more users don't set the Mime headers at
>>> all!  And Gnus assumes UTF-8 for those postings
>>
>> That sounds like a bug.  How can I reproduce it?
>
> Start Emacs in a UTF-8 locale:
>
>     LANG=en_US.UTF-8 emacs
>
> and page thru some German newsgroup like de.rec.buecher (books) or
> de.rec.fahrrad (bikes).  Every 10th or 20th article will feature those
> nice \321, \345, \432 decorations :)

OK.  I see a couple of ways to fix it:

1. gnus-default-charset's default value is deduced from the language
   environment:

      (defcustom gnus-default-charset (mm-guess-mime-charset)

   I don't think this is necessarily a good idea.  

   Just because you're using UTF-8, mac-roman, EBCDIC or whatever on
   your local system doesn't mean that it's a good guess that you'll
   receive mail and news using this charset.  

   So maybe `mm-guess-mime-charset' shouldn't choose some charsets --
   UTF-8, UTF-16 and mac-roman would probably be likely candidates to
   exclude.

2. We could add more entries to `gnus-group-charset-alist'.

Re: TeX input method? (Was: charset=macintosh)

[Jinhyok Heo]

Though I googled, I couldn't find about TeX input method on the
Internet. Is it a new input method?

Could you let me know where I can find it? Can it be used on XEmacs?

-- 
| Jinhyok Heo (novembre @ ournature.org || http://ournature.org/~novembre/)
|--------------------------------------------------------------------------
| "We are still reaching for the sky. In the developed countries people
|  are coming back down, saying, `It's empty up there.'" --- a Ladakhi monk

Re: TeX input method?

[Karl Eichwalder]

Jinhyok Heo <novembre+dated+1047522742.3d77e9@ournature.org> writes:

> Though I googled, I couldn't find about TeX input method on the
> Internet. Is it a new input method?

It comes with "leim", which will be bundled with Emacs starting with
the version coming from Emacs CVS HEAD (ATM AKA 21.3.50).

Once "leim" is installed press "C-x RET C-\ TeX RET" to switch to this
method.  With C-\ you can toggle back to your default input method.

> Could you let me know where I can find it? Can it be used on XEmacs?

Probably yes, but I don't know for sure.

-- 
ke@suse.de (work) / keichwa@gmx.net (home):              |
http://www.gnu.franken.de/ke/                            |      ,__o
Free Translation Project:                                |    _-\_<,
http://www.iro.umontreal.ca/contrib/po/HTML/             |   (*)/'(*)

Re: charset=macintosh

[Kai Großjohann]

Jesper Harder <harder@myrealbox.com> writes:

>    Just because you're using UTF-8, mac-roman, EBCDIC or whatever on
>    your local system doesn't mean that it's a good guess that you'll
>    receive mail and news using this charset.  

I think this guess is better than no guess at all.

Do you have a better guess?
-- 
A preposition is not a good thing to end a sentence with.

Re: charset=macintosh

[Jorge Godoy]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Jesper Harder <harder@myrealbox.com> writes:
>
>>    Just because you're using UTF-8, mac-roman, EBCDIC or whatever on
>>    your local system doesn't mean that it's a good guess that you'll
>>    receive mail and news using this charset.  
>
> I think this guess is better than no guess at all.
>
> Do you have a better guess?

Isn't it obligatory to fill in the charset at message headers?
Instead of guessing, Gnus should --- I think it already does --- use
this information to choose the charset.

If nothing is filled, them use the locale the use is in and add a
notice that the message has an obligatory information missing.

-- 
Godoy.    <godoy@metalab.unc.edu>

Re: charset=macintosh

[Simon Josefsson]

Jorge Godoy <godoy@metalab.unc.edu> writes:

> kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
>
>> Jesper Harder <harder@myrealbox.com> writes:
>>
>>>    Just because you're using UTF-8, mac-roman, EBCDIC or whatever on
>>>    your local system doesn't mean that it's a good guess that you'll
>>>    receive mail and news using this charset.  
>>
>> I think this guess is better than no guess at all.
>>
>> Do you have a better guess?
>
> Isn't it obligatory to fill in the charset at message headers?
> Instead of guessing, Gnus should --- I think it already does --- use
> this information to choose the charset.

Yes, and Gnus does this.  I think the thread is about when this
information isn't available, which in some communities can be a large
proportion.

> If nothing is filled, them use the locale the use is in and add a
> notice that the message has an obligatory information missing.

Gnus should be more helpful, and already is since it allows certain
hierarchies to use other charsets (see `gnus-group-charset-alist').

For articles without MIME tags, in groups not in g-g-c-a, it would be
nice if Gnus could guess better -- like trying to UTF-8 decode it,
which typically only fails when data wasn't UTF-8 encoded, and then go
on and try other encodings.  Emacs' decoding functions behave a little
strange, but onces fixed Gnus should be able to do this.

Re: charset=macintosh

[Kai Großjohann]

Jorge Godoy <godoy@metalab.unc.edu> writes:

> If nothing is filled, them use the locale the use is in and add a
> notice that the message has an obligatory information missing.

I agree.  That's what I was trying to say.  (Though perhaps without
that note.)

Btw, us-ascii may be sent without MIMEy stuff.
-- 
A preposition is not a good thing to end a sentence with.

Re: charset=macintosh

[Frank Schmitt]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Jesper Harder <harder@myrealbox.com> writes:
>
>>    Just because you're using UTF-8, mac-roman, EBCDIC or whatever on
>>    your local system doesn't mean that it's a good guess that you'll
>>    receive mail and news using this charset.  
>
> I think this guess is better than no guess at all.
>
> Do you have a better guess?

I think somewhere in Gnus' code there's a list of Usenet hierarchies
with default charsets, IMO we should use this.

-- 
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.

Re: charset=macintosh

[Simon Josefsson]

Frank Schmitt <usereplyto@Frank-Schmitt.net> writes:

> kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
>
>> Jesper Harder <harder@myrealbox.com> writes:
>>
>>>    Just because you're using UTF-8, mac-roman, EBCDIC or whatever on
>>>    your local system doesn't mean that it's a good guess that you'll
>>>    receive mail and news using this charset.  
>>
>> I think this guess is better than no guess at all.
>>
>> Do you have a better guess?
>
> I think somewhere in Gnus' code there's a list of Usenet hierarchies
> with default charsets, IMO we should use this.

Yup, isn't it used?  For groups not in gnus-group-charset-alist, Gnus
should ideally be more helpful than simply using the locale default.
If Gnus can discover that the locale default cannot possibly be
applicable to an article, it should try something else that might
work.

Re: charset=macintosh

[Jesper Harder]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Jesper Harder <harder@myrealbox.com> writes:
>
>>    Just because you're using UTF-8, mac-roman, EBCDIC or whatever on
>>    your local system doesn't mean that it's a good guess that you'll
>>    receive mail and news using this charset.  
>
> I think this guess is better than no guess at all.

Well, I think experience shows that UTF-8 isn't a good guess.  It's a
bad guess for dk.*, and as Karl says also for de.*.  

Previously `gnus-group-charset-alist' had (".*" iso-8859-1) as the last
entry -- this was probably better for the majority of users.

I haven't got any statistics to back it up, but I suspect that clients
which send UTF-8 are probably newer, and thus more likely to include a
proper MIME charset declaration.

So, I think it's much more likely that the charset of a message with an
undeclared charset is iso-8859-x rather than UTF-8.

Re: charset=macintosh

[Jesper Harder]

Simon Josefsson <jas@extundo.com> writes:

> For articles without MIME tags, in groups not in g-g-c-a, it would be
> nice if Gnus could guess better -- like trying to UTF-8 decode it,
> which typically only fails when data wasn't UTF-8 encoded, and then go
> on and try other encodings.  Emacs' decoding functions behave a little
> strange, but onces fixed Gnus should be able to do this.

Currently they're not good enough, IMHO.  Here's an example:

(detect-coding-string (encode-coding-string "dk.test.utf8-æøå" 'utf-8))

=> (iso-latin-1 iso-latin-1 raw-text japanese-shift-jis 
    chinese-big5 no-conversion mule-utf-8)

The correct answer is last in the list.

Re: charset=macintosh

[Simon Josefsson]

Jesper Harder <harder@myrealbox.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> For articles without MIME tags, in groups not in g-g-c-a, it would be
>> nice if Gnus could guess better -- like trying to UTF-8 decode it,
>> which typically only fails when data wasn't UTF-8 encoded, and then go
>> on and try other encodings.  Emacs' decoding functions behave a little
>> strange, but onces fixed Gnus should be able to do this.
>
> Currently they're not good enough, IMHO.  Here's an example:
>
> (detect-coding-string (encode-coding-string "dk.test.utf8-æøå" 'utf-8))
>
> => (iso-latin-1 iso-latin-1 raw-text japanese-shift-jis 
>     chinese-big5 no-conversion mule-utf-8)
>
> The correct answer is last in the list.

Doesn't that function use the preference order configured by the user?
For me, who runs emacs in a UTF-8 locale, it returns mule-utf-8 first.
Released emacs versions have incomplete UTF-8 support (see PROBLEMS)
and UTF-8 have a very low priority so any potential bug aren't
triggered unless they, err, really must be triggered.  This is
reasonable, I think.

I've asked on emacs-devel that emacs in CVS (both 21.3 and HEAD),
which supposedly has complete Unicode support, since the PROBLEMS
entry is removed, should prefer UTF-8 more often.  This would be the
best solution IMHO, as Gnus wouldn't have to contain magic charset
prioritizing code.  It also seems like a reasonable solution, assuming
the Unicode stuff actually is working.

The simplest would probably be that people who likes Unicode run emacs
in a UTF-8 locale though, then they would not have any of these
problems.

Your RFC quote was interesting though, I think it suggests that Gnus
should downgrade UTF-8 to ISO-8859-X whenever possible, even if the
user uses a UTF-8 locale, since ISO-8859-X is more widely supported.
That would probably be a contentious decision though: What if a
Japanese user, in a UTF-8 locale, enters text that happens to be
downgradable to ISO-8859-1?  Downgrading in this case is probably
never a good idea.  OTOH if this situation is purely hypothetical, it
doesn't matter if downgrading happens.

Re: charset=macintosh

[Simon Josefsson]

Jesper Harder <harder@myrealbox.com> writes:

> kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:
>
>> Jesper Harder <harder@myrealbox.com> writes:
>>
>>>    Just because you're using UTF-8, mac-roman, EBCDIC or whatever on
>>>    your local system doesn't mean that it's a good guess that you'll
>>>    receive mail and news using this charset.  
>>
>> I think this guess is better than no guess at all.
>
> Well, I think experience shows that UTF-8 isn't a good guess.  It's a
> bad guess for dk.*, and as Karl says also for de.*.  
>
> Previously `gnus-group-charset-alist' had (".*" iso-8859-1) as the last
> entry -- this was probably better for the majority of users.
>
> I haven't got any statistics to back it up, but I suspect that clients
> which send UTF-8 are probably newer, and thus more likely to include a
> proper MIME charset declaration.

Seems reasonable.

> So, I think it's much more likely that the charset of a message with an
> undeclared charset is iso-8859-x rather than UTF-8.

Or ISO-2022.  Or Big5.  The problem with choosing ISO-8859-X is that
it is western centric, and probably rightly be considered offensive.

But what if you are saying about UTF-8 clients being MIME capable is
true, and since UTF-8 is typically never preferred by current emacsen,
doesn't emacs' current guessing works the best we can hope for?
Doesn't it detect among ISO-8859-X, ISO-2022 and Big5 properly?  I
think it does.

So there is only a problem where:

  1) UTF-8 is used without MIME tagging since emacs never guess this
     correctly, but you argue (and I agree) that this case is unlikely.

  2) Users with emacs in UTF-8 prefers UTF-8 too often, even when the
     data is invalid UTF-8 and another encoding should be selected.

The second situation is a bug, and I hope we can fix this.  Maybe I
missed something.

Re: charset=macintosh

[Jesper Harder]

Simon Josefsson <jas@extundo.com> writes:

> But what if you are saying about UTF-8 clients being MIME capable is
> true, and since UTF-8 is typically never preferred by current emacsen,
> doesn't emacs' current guessing works the best we can hope for?
> Doesn't it detect among ISO-8859-X, ISO-2022 and Big5 properly?

No.  I was hoping we could do something like this (for headers):

(let ((coding-systems (detect-coding-string string)))
  (if (memq default coding-systems)
      (decode-coding-string string default)
    (decode-coding-string string (car coding-systems))))

i.e. if the default coding system is valid for the string, then use
that; otherwise use whatever Emacs thinks is the most likely coding
system.  I think this would be ideal.

But unfortunately `detect-coding-string' _doesn't_ return a complete
list of possible coding systems.  Consider this scenario: 

  I'm using Emacs in a Latin-1 locale.  dk.* newsgroups work fine
  because latin-1 is the default.  But I also subscribe to, say, a few
  Korean newsgroups.  The entry in `gnus-groups-charset-alist':

         ("\\(^\\|:\\)han\\>" euc-kr)

  should take care of selecting the proper default charset.  But *oops*,
  `detect-coding-string' doesn't think that euc-kr is a possible charset
  for a Korean string encoded in euc-kr:

       (detect-coding-string (encode-coding-string "안녕" 'euc-kr))
       => (iso-latin-1 iso-latin-1 raw-text japanese-shift-jis 
           chinese-big5 no-conversion)

So the above approach would fail.

>   2) Users with emacs in UTF-8 prefers UTF-8 too often, even when the
>      data is invalid UTF-8 and another encoding should be selected.
>
> The second situation is a bug, and I hope we can fix this.

Yep, 2) is the most serious problem.  Especially because more and more
people are (often unknowingly) using an UTF-8 locale because Redhat 8
switched to UTF-8 by default.  Those people would experience Gnus as
broken when reading hierarchies like dk.* or de.*.

Re: charset=macintosh

[Simon Josefsson]

Jesper Harder <harder@myrealbox.com> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> But what if you are saying about UTF-8 clients being MIME capable is
>> true, and since UTF-8 is typically never preferred by current emacsen,
>> doesn't emacs' current guessing works the best we can hope for?
>> Doesn't it detect among ISO-8859-X, ISO-2022 and Big5 properly?
>
> No.  I was hoping we could do something like this (for headers):
>
> (let ((coding-systems (detect-coding-string string)))
>   (if (memq default coding-systems)
>       (decode-coding-string string default)
>     (decode-coding-string string (car coding-systems))))
>
> i.e. if the default coding system is valid for the string, then use
> that; otherwise use whatever Emacs thinks is the most likely coding
> system.  I think this would be ideal.

Yes.  (Except for the current UTF-8 bug, of course)

> But unfortunately `detect-coding-string' _doesn't_ return a complete
> list of possible coding systems.  Consider this scenario: 
>
>   I'm using Emacs in a Latin-1 locale.  dk.* newsgroups work fine
>   because latin-1 is the default.  But I also subscribe to, say, a few
>   Korean newsgroups.  The entry in `gnus-groups-charset-alist':
>
>          ("\\(^\\|:\\)han\\>" euc-kr)
>
>   should take care of selecting the proper default charset.  But *oops*,
>   `detect-coding-string' doesn't think that euc-kr is a possible charset
>   for a Korean string encoded in euc-kr:
>
>        (detect-coding-string (encode-coding-string "안녕" 'euc-kr))
>        => (iso-latin-1 iso-latin-1 raw-text japanese-shift-jis 
>            chinese-big5 no-conversion)
>
> So the above approach would fail.

I wonder if this isn't another bug.  Maybe you could report it as a
bug?

>>   2) Users with emacs in UTF-8 prefers UTF-8 too often, even when the
>>      data is invalid UTF-8 and another encoding should be selected.
>>
>> The second situation is a bug, and I hope we can fix this.
>
> Yep, 2) is the most serious problem.  Especially because more and more
> people are (often unknowingly) using an UTF-8 locale because Redhat 8
> switched to UTF-8 by default.  Those people would experience Gnus as
> broken when reading hierarchies like dk.* or de.*.

Fortunately we can blame it on emacs, and the PROBLEMS entry.  Not so
for Emacs 21.3 though, which has removed the entry but the behaviour
is the same.  But this should be fixed..

Re: charset=macintosh

[Jesper Harder]

Simon Josefsson <jas@extundo.com> writes:

> Jesper Harder <harder@myrealbox.com> writes:
>
>> But unfortunately `detect-coding-string' _doesn't_ return a complete
>> list of possible coding systems.
>
> I wonder if this isn't another bug.  Maybe you could report it as a
> bug?

OK, it appears that `detect-coding-string' isn't supposed to return a
complete list.  But fortunately Kenichi Handa suggested another way to
do it.

mm-coding-system-priorities (was: charset=macintosh)

[Reiner Steib]

On Fri, Mar 07 2003, Simon Josefsson wrote:

> The reason Emacs doesn't list UTF-8 higher than macintosh is,
> supposedly, that the UTF-8 support is not complete.  Once it is
> complete, the emacs maintainers will probably change the preference
> ordering.

Apparently it has already been changed in Emacs 21.3, see below.

> Fortunately it is easy to override the priorities within Gnus:
>  '(mm-coding-system-priorities (quote (iso-latin-1 iso-latin-9 utf-8)))
>
> Hm.  The UTF-8 discussion in Emacs 21.2 PROBLEMS is not present in
> Emacs 21.3 prerelease, so perhaps it has been fixed.  I'll ask on
> emacs-devel whether utf-8 should be preferred over mac-roman.

Sample text: “quotes”

With Emacs 21.3.1 and current Oort [1] I get (in mml-preview):

,----
| (setq mm-coding-system-priorities nil)
| ==> charset=utf-8
| 
| (setq mm-coding-system-priorities '(macintosh utf-8))
| ==> charset=utf-8
| 
| (setq mm-coding-system-priorities '(mac-roman utf-8))
| ==> charset=macintosh
`----

Adding windows-1252 [2] gives:

,----
| (setq mm-coding-system-priorities nil)
| ==> charset=utf-8
| 
| (setq mm-coding-system-priorities '(windows-1252 macintosh utf-8))
| ==> charset=windows-1252
| 
| (setq mm-coding-system-priorities '(utf-8 macintosh windows-1252))
| ==> charset=windows-1252
| 
| (setq mm-coding-system-priorities '(mule-utf-8 windows-1252))
| ==> charset=utf-8
`----

Apparently, we need to write `mule-utf-8' and `mac-roman' instead of
`utf-8' and `macintosh', respectively.

,----[ C-h v mm-coding-system-priorities RET ]
| mm-coding-system-priorities's value is nil
| 
| Documentation:
| Preferred coding systems for encoding outgoing mails.
`----

So `mm-coding-system-priorities' expects an Emacs coding system and
not a MIME charset.  But OTOH...

,----[ M-x describe-coding-system utf-8 RET ]
| u -- utf-8 (alias of mule-utf-8)
`----

Hm, does it even expect the original name of the coding-system and
*not* an alias?  If so, I think this should be made more clear in the
documentation.

Bye, Reiner.

[1] Oort Gnus v0.18
In GNU Emacs 21.3.1 (i586-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
configured using `configure  --prefix=/soft/local/upp/emacs'
Important settings:
  value of $LC_CTYPE: en_US.ISO_8859-1
  value of $LANG: C
  locale-coding-system: iso-latin-1
  default-enable-multibyte-characters: t

[2] http://theotp1.physik.uni-ulm.de/~ste/comp/emacs/gnus/rs-windows-1252.el
(require 'rs-windows-1252)
(rs-use-windows-1252)

-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: mm-coding-system-priorities

[Lars Magne Ingebrigtsen]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:

> So `mm-coding-system-priorities' expects an Emacs coding system and
> not a MIME charset.  But OTOH...
>
> ,----[ M-x describe-coding-system utf-8 RET ]
> | u -- utf-8 (alias of mule-utf-8)
> `----
>
> Hm, does it even expect the original name of the coding-system and
> *not* an alias?  If so, I think this should be made more clear in the
> documentation.

utf-8 is both a coding system and a charset, I think.  (And I don't
think it should matter whether you use the name or an alias.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: mm-coding-system-priorities

[Reiner Steib]

On Sat, Apr 12 2003, Lars Magne Ingebrigtsen wrote:

> Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:
>> ,----[ M-x describe-coding-system utf-8 RET ]
>> | u -- utf-8 (alias of mule-utf-8)
>> `----
>>
>> Hm, does it even expect the original name of the coding-system and
>> *not* an alias?  If so, I think this should be made more clear in the
>> documentation.
>
> utf-8 is both a coding system and a charset, I think.

AFAICS, we need to distinguish between "base coding-system",
"aliased/subsidiary coding-system" and mime-charset.  utf-8 is both,
an "aliased/subsidiary coding-system" and a mime-charset, but not a
"base coding-system".  The problem is, that using utf-8 in
`mm-coding-system-priorities' has no effect[1] at all: You need to use
a non-aliased coding-system, as returned by `coding-system-base':

,----[ C-h f coding-system-base RET ]
| coding-system-base is a compiled Lisp function in `international/mule'.
| (coding-system-base CODING-SYSTEM)
| 
| Return the base coding system of CODING-SYSTEM.
| A base coding system is what made by `make-coding-system'.
| Any alias nor subsidiary coding systems are not base coding system.
`----

The doc-string of `mm-coding-system-priorities' doesn't mention this.
Even the default value for users with "current-language-environment =
Japanese" contains the bogus entry utf-8.

> (And I don't think it should matter whether you use the name or an
> alias.)

Changing `mm-sort-coding-systems-predicate' should achieve it (I have
committed this):

(defun mm-sort-coding-systems-predicate (a b)
  (let ((priorities
	 (mapcar (lambda (cs)
		   ;; Note: invalid entries are dropped silently
		   (and (coding-system-p cs)
			(coding-system-base cs)))
		 mm-coding-system-priorities)))
    (> (length (memq a priorities))
       (length (memq b priorities)))))

Bye, Reiner.

[1] See my test in <news:v965pyw1gv.fsf@marauder.physik.uni-ulm.de>
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: mm-coding-system-priorities

[Lars Magne Ingebrigtsen]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:

> | A base coding system is what made by `make-coding-system'.
> | Any alias nor subsidiary coding systems are not base coding system.

Ah, right; I didn't know that.

> Changing `mm-sort-coding-systems-predicate' should achieve it (I have
> committed this):

Cool.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: charset=macintosh

[Jesper Harder]

Simon Josefsson <jas@extundo.com> writes:

> Jesper Harder <harder@myrealbox.com> writes:

>>   But *oops*, `detect-coding-string' doesn't think that euc-kr is a
>>   possible charset for a Korean string encoded in euc-kr:
>>
>>        (detect-coding-string (encode-coding-string "안녕" 'euc-kr))
>>        => (iso-latin-1 iso-latin-1 raw-text japanese-shift-jis 
>>            chinese-big5 no-conversion)
>>
>> So the above approach would fail.
>
> I wonder if this isn't another bug.  

I don't know.  A lot of the MULE stuff is so under-documented.  The
documentation for `detect-coding-string' doesn't actually promise to
return _all_ possible coding systems.  It turns out that it _can_ detect
euc-kr (AKA korean-iso-8bit) if you choose the right language
environment:

   (set-language-environment "Korean")
   (detect-coding-string (encode-coding-string "안녕" 'euc-kr))
   => (korean-iso-8bit iso-latin-1 raw-text chinese-big5 no-conversion)


> Maybe you could report it as a bug?

I'll do that.