Setup additional charsets on demand

Setup additional charsets on demand

[Reiner Steib]

Hi,

in reaction to Matthias Andree's posting[1]...

,----[ http://thread.gmane.org/gmane.emacs.gnus.general/59971 ]
| What's up here? Does Gnus lack Windows-1251? If so, why does it not
| replace everything by dots, X or ?. If it supports Windows-1251, why
| doesn't it see it? [...] (Emacs 21.3 with rm'd movemail)
`----

... I added `mm-charset-eval-alist' to the trunk now.

If nobody complains (i.e. if it seems to work correctly for other
people than me), I'd like to add to v5-10 too.  I consider it as a
bug, that Gnus doesn't use Windows-1251 although Emacs 21 supports it.

Suggestion for better variable names or documentation welcome!

Bye, Reiner.

[1] <news:m34qflors2.fsf@merlin.emma.line.org>
    http://thread.gmane.org/gmane.emacs.gnus.general/59971
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Display unknown non-ASCII chars as `.', `X' or `?' (was: Setup additional charsets on demand)

[Reiner Steib]

On Tue, Oct 11 2005, Reiner Steib wrote:

> in reaction to Matthias Andree's posting[1]...
>
> ,----[ http://thread.gmane.org/gmane.emacs.gnus.general/59971 ]
> | What's up here? Does Gnus lack Windows-1251? If so, why does it not
> | replace everything by dots, X or ?. If it supports Windows-1251, why
> | doesn't it see it? [...] (Emacs 21.3 with rm'd movemail)
> `----
>
> ... I added `mm-charset-eval-alist' to the trunk now.

While at it...
| If so, why does it not replace everything by dots, X or ?.

I.e. if the charset is not available: How can we configure Gnus to
display every non-ASCII chars as `.', `X' or `?'?

One solution is to define a "dummy coding system" where all non-ASCII
chars are replaced by some place holder:

(defun rs-ucs-ascii-8bit-dumb (&optional char)
  "Create dummy coding system `rs-ucs-ascii-8bit-dumb'."
  (unless char (setq char ?\?))
  ;; We need cp-make-coding-system (Emacs 22)
  (unless (fboundp 'cp-make-coding-system)
    (error "Function requires `cp-make-coding-system'"))
  (cp-make-coding-system
   rs-ucs-ascii-8bit-dumb
   (make-vector 128 char)
   (format "Dummy coding system.  Replace all non-ASCII chars as `%c'." char)
   ?\?))
(rs-ucs-ascii-8bit-dumb ?¿)
(add-to-list 'mm-charset-override-alist
	     '(undecided . rs-ucs-ascii-8bit-dumb))

With this, also non-ASCII chars in postings without proper MIME
headers[1] will be displayed as `¿'.  (IIRC, slrn and tin replace
unknown characters with `?'.  Some people like this and turn off
fallbacks like iso-8859-1.)

Can we achieve the same without using functions from Emacs 22
(`cp-make-coding-system')?

Bye, Reiner.

[1] Some examples: <news:Usenet.qaremkmn@localhost>,
    <news:Usenet.cfkoelmf@localhost>
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Display unknown non-ASCII chars as `.', `X' or `?'

[Bjørn Mork]

Reiner Steib <reinersteib+gmane@imap.cc> writes:
> On Tue, Oct 11 2005, Reiner Steib wrote:
>
>> in reaction to Matthias Andree's posting[1]...
>>
>> ,----[ http://thread.gmane.org/gmane.emacs.gnus.general/59971 ]
>> | What's up here? Does Gnus lack Windows-1251? If so, why does it not
>> | replace everything by dots, X or ?. If it supports Windows-1251, why
>> | doesn't it see it? [...] (Emacs 21.3 with rm'd movemail)
>> `----
>>
>> ... I added `mm-charset-eval-alist' to the trunk now.
>
> While at it...
> | If so, why does it not replace everything by dots, X or ?.
>
> I.e. if the charset is not available: How can we configure Gnus to
> display every non-ASCII chars as `.', `X' or `?'?

This is a violation of RFC2049, which gives the following criteria for
a MIME-conformant MUA:

   -- Treat material in an unknown character set as if it were
      "application/octet-stream".



Bjørn
-- 
Unpleasant feminist arsehole!  

Re: Display unknown non-ASCII chars as `.', `X' or `?'

[Reiner Steib]

On Wed, Oct 12 2005, Bjørn Mork wrote:

> Reiner Steib <reinersteib+gmane@imap.cc> writes:
[...]
>> While at it...
>> | If so, why does it not replace everything by dots, X or ?.
>>
>> I.e. if the charset is not available: How can we configure Gnus to
>> display every non-ASCII chars as `.', `X' or `?'?
>
> This is a violation of RFC2049, which gives the following criteria for
> a MIME-conformant MUA:
>
>    -- Treat material in an unknown character set as if it were
>       "application/octet-stream".

I fail to see the connection (Disclaimer: I didn't read this RFC).
"application/octet-stream" is a content type, whereas I was talking
about the charset.

BTW: Yesterday, in the German newsgroup de.comm.software.newsreader a
thread titled "Zeichensatz erraten"[1] started about this topic.  Some
people claim that the receiver _must_ assume `us-ascii' when MIME
headers are absent.

In case it wasn't clear: I don't want to change Gnus default behavior.
But maybe it makes sense to replace unknown chars by `?' for some
users so it should be possible to do this with Gnus.

Bye, Reiner.

[1] http://www.google.de/groups?as_umsgid=9fk6Itd7fjB%40helmut.hullen.de&hl=en
    http://groups.google.de/group/de.comm.software.newsreader/browse_frm/thread/7c7ed4e3c7e63560/cede6121a1685dda#cede6121a1685dda
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Display unknown non-ASCII chars as `.', `X' or `?'

[Bjørn Mork]

Reiner Steib <reinersteib+gmane@imap.cc> writes:
> On Wed, Oct 12 2005, Bjørn Mork wrote:
>> Reiner Steib <reinersteib+gmane@imap.cc> writes:
> [...]
>>> While at it...
>>> | If so, why does it not replace everything by dots, X or ?.
>>>
>>> I.e. if the charset is not available: How can we configure Gnus to
>>> display every non-ASCII chars as `.', `X' or `?'?
>>
>> This is a violation of RFC2049, which gives the following criteria for
>> a MIME-conformant MUA:
>>
>>    -- Treat material in an unknown character set as if it were
>>       "application/octet-stream".
>
> I fail to see the connection (Disclaimer: I didn't read this RFC).
> "application/octet-stream" is a content type, whereas I was talking
> about the charset.

Sorry, my fault.  I didn't quote enough to make it understandable.
The part above comes from 

2.  MIME Conformance
[..]
   A mail user agent that is MIME-conformant MUST:
[..]
    (6)   Explicitly handle the following media type values, to
          at least the following extents:

          Text:
[..]
            -- Treat material in an unknown character set as if
            it were "application/octet-stream".


> BTW: Yesterday, in the German newsgroup de.comm.software.newsreader a
> thread titled "Zeichensatz erraten"[1] started about this topic.  Some
> people claim that the receiver _must_ assume `us-ascii' when MIME
> headers are absent.

That's not true.  RFC2049 says 

    (8)   Conformant user agents are required, if they provide
          non-standard support for non-MIME messages employing
          character sets other than US-ASCII, to do so on
          received messages only. Conforming user agents must not
          send non-MIME messages containing anything other than
          US-ASCII text.

          In particular, the use of non-US-ASCII text in mail
          messages without a MIME-Version field is strongly
          discouraged as it impedes interoperability when sending
          messages between regions with different localization
          conventions. Conforming user agents MUST include proper
          MIME labelling when sending anything other than plain
          text in the US-ASCII character set.

          In addition, non-MIME user agents should be upgraded if
          at all possible to include appropriate MIME header
          information in the messages they send even if nothing
          else in MIME is supported.  This upgrade will have
          little, if any, effect on non-MIME recipients and will
          aid MIME in correctly displaying such messages.  It
          also provides a smooth transition path to eventual
          adoption of other MIME capabilities.



> In case it wasn't clear: I don't want to change Gnus default behavior.
> But maybe it makes sense to replace unknown chars by `?' for some
> users so it should be possible to do this with Gnus.

I should have known:  That's the proper Gnus way to do things. 

I guess that makes the proposal OK. 



Bjørn
-- 
I'll bet you think that random frobs are disgusting.