\201 problem reproduced

\201 problem reproduced

[Lars Magne Ingebrigtsen]

So, I managed to reproduce the \201 problem if I started Emacs with
--unibyte.  0.20 attempts to fix these problems.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: \201 problem reproduced

[Vladimir Volovich]

"LMI" == Lars Magne Ingebrigtsen writes:

 LMI> So, I managed to reproduce the \201 problem if I started Emacs
 LMI> with --unibyte.  0.20 attempts to fix these problems.

Well. Haven't received any reply on my problems from the semi-gnus
people (which i reported to the ding list too), i've tried pgnus. i
downloaded the pgnus-0.23.  Unfortunately, it was unable to show
messages with

Content-Type: text/plain;  charset=koi8-r
Content-Transfer-Encoding: 8bit

I.e. they contained 8-bit cyrillic letters in a koi8-r
encoding. Instead of `normal' letters i got combinations like \214
followed by some non-latin letters (but not from koi8-r correct as
well). all cyrillic letters were prefixed with \214

The same happens in the summary buffer for subjects in koi8-r.

Also, for multipart mime messages, which gnus tried to show, i got the
following results. if the part had type:

Content-Type: text/plain;
	charset="koi8-r"
Content-Transfer-Encoding: quoted-printable

then the characters were shown as e.g.

\317\326\305\324...

(i haven't touched the standard-display-8bit). I have the following in
my .emacs:

(set-default-font "fontset-standard")
(set-language-environment "Cyrillic-KOI8")
(setq default-mime-charset 'koi8-r)

	Best regards, -- Vladimir.
-- 
When you go out to buy, don't show your silver.

Re: \201 problem reproduced

[Lars Magne Ingebrigtsen]

Vladimir Volovich <vvv@vvv.vsu.ru> writes:

> Unfortunately, it was unable to show messages with
> 
> Content-Type: text/plain;  charset=koi8-r
> Content-Transfer-Encoding: 8bit

Could you send me such a message?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: \201 problem reproduced

[Vladimir Volovich]

"LMI" == Lars Magne Ingebrigtsen writes:

 >> Content-Type: text/plain; charset=koi8-r
 >> Content-Transfer-Encoding: 8bit

 LMI> Could you send me such a message?

Done (in a private mail). BTW, i've tried to send a message with
russian letters using pgnus-0.24, and it set a charset to iso-8895-5
regardless of the settings in my .emacs:

(set-language-environment "Cyrillic-KOI8")

which implies that e.g. default-mime-charset's value is koi8-r.

I'd like my messages with russian letters to have charset=KOI8-R
if i use (set-language-environment "Cyrillic-KOI8").

	Best regards, -- Vladimir.
-- 
All's well that ends.

Re: \201 problem reproduced

[Lars Magne Ingebrigtsen]

Vladimir Volovich <vvv@vvv.vsu.ru> writes:

> Done (in a private mail). BTW, i've tried to send a message with
> russian letters using pgnus-0.24, and it set a charset to iso-8895-5
> regardless of the settings in my .emacs:
> 
> (set-language-environment "Cyrillic-KOI8")
> 
> which implies that e.g. default-mime-charset's value is koi8-r.

I don't have such a variable defined in my Emacs.  How should a
program find out (based on the default language environment) which
charsets should be used?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: \201 problem reproduced

[Vladimir Volovich]

"LMI" == Lars Magne Ingebrigtsen writes:

 >> (set-language-environment "Cyrillic-KOI8")
 >> which implies that e.g. default-mime-charset's value is koi8-r.

 LMI> I don't have such a variable defined in my Emacs.

Sorry, that was a semi-gnus thing.

 LMI> How should a program find out (based on the default language
 LMI> environment) which charsets should be used?

I do not know, but there are e.g. three cyrillic language environments
defined in MULE: Cyrillic-KOI8, Cyrillic-ISO, Cyrillic-ALT. May be,
person who uses Cyrillic-ISO would prefer his letters to be sent
encoded iso-8859-5, but those of us using Cyrillic-KOI8, would like to
send messages encoded with koi8-r. Semi-gnus somehow knew about this,
and did not try to use iso-8859-5 at all. :-)

BTW, the problem with \214 seems to be present only in
nnmbox:mail.misc `newsgroup'. In nnml:archive and in nntp newsgroups
the situation is much better. Very strange.

	Best regards, -- Vladimir.

Re: \201 problem reproduced

[Lars Magne Ingebrigtsen]

Vladimir Volovich <vvv@vvv.vsu.ru> writes:

> I do not know, but there are e.g. three cyrillic language environments
> defined in MULE: Cyrillic-KOI8, Cyrillic-ISO, Cyrillic-ALT. May be,
> person who uses Cyrillic-ISO would prefer his letters to be sent
> encoded iso-8859-5, but those of us using Cyrillic-KOI8, would like to
> send messages encoded with koi8-r. Semi-gnus somehow knew about this,
> and did not try to use iso-8859-5 at all. :-)

Could you try to investigate how semi-gnus finds this out?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: \201 problem reproduced

[Vladimir Volovich]

"LMI" == Lars Magne Ingebrigtsen writes:

 >> I do not know, but there are e.g. three cyrillic language
 >> environments defined in MULE: Cyrillic-KOI8, Cyrillic-ISO,
 >> Cyrillic-ALT. May be, person who uses Cyrillic-ISO would prefer
 >> his letters to be sent encoded iso-8859-5, but those of us using
 >> Cyrillic-KOI8, would like to send messages encoded with
 >> koi8-r. Semi-gnus somehow knew about this, and did not try to use
 >> iso-8859-5 at all. :-)
 LMI> Could you try to investigate how semi-gnus finds this out?

I'll try, but for now, i'd like to ask one question. ;-)

The mm-util.el from pgnus contains the following:

(defvar mm-mime-mule-charset-alist
  '((us-ascii ascii)
    (iso-8859-1 latin-iso8859-1)
    (iso-8859-2 latin-iso8859-2)
    (iso-8859-3 latin-iso8859-3)
    (iso-8859-4 latin-iso8859-4)
    (iso-8859-5 cyrillic-iso8859-5)
    (koi8-r cyrillic-iso8859-5)
    (iso-8859-6 arabic-iso8859-6)
    (iso-8859-7 greek-iso8859-7)
[the rest skipped]

Note the presence of two lines corresponding to cyrillic-iso8859-5
mule charset:

    (iso-8859-5 cyrillic-iso8859-5)
    (koi8-r cyrillic-iso8859-5)

I wonder, does the first line override the second one??? I suppose
that if i'll remove the line

    (iso-8859-5 cyrillic-iso8859-5)

then gnus will use koi8-r by befault (and btw koi8-r is a de-facto
standard for russian e-mail exchange anyway; the presence of two
mappings for one mule encoding does not seem useful anyway btw, --- is
it a bug?).

	Best regards, -- Vladimir.
-- 
Stability itself is nothing else than a more sluggish motion.

Re: \201 problem reproduced

[Vladimir Volovich]

"VV" == Vladimir Volovich writes:

 VV> I wonder, does the first line override the second one??? I
 VV> suppose that if i'll remove the line

 VV>     (iso-8859-5 cyrillic-iso8859-5)

 VV> then gnus will use koi8-r by befault

This did not help, because the purpose of that variable is
different. After removing the line "(iso-8859-5 cyrillic-iso8859-5)",
pgnus continued to send my messages in Russian encoded with
iso-8859-5, but this setting `fooled' pgnus, and it began to thought
that this charset does not need to be encoded in subjects, because my
rfc2047-default-charset was set to koi8-r.

I've looked a bit into semi-gnus, and here are some results.

In the file emu-20.el (emulation module for Emacs-20) from the `apel'
package there is the definition of the charsets-mime-charset-alist:

(defvar charsets-mime-charset-alist
  '(((ascii)						. us-ascii)
    ((ascii latin-iso8859-1)				. iso-8859-1)
    ((ascii latin-iso8859-2)				. iso-8859-2)
    ((ascii latin-iso8859-3)				. iso-8859-3)
    ((ascii latin-iso8859-4)				. iso-8859-4)
;;; ((ascii cyrillic-iso8859-5)				. iso-8859-5)
    ((ascii cyrillic-iso8859-5)				. koi8-r)
    ((ascii arabic-iso8859-6)				. iso-8859-6)
    ((ascii greek-iso8859-7)				. iso-8859-7)
    ((ascii hebrew-iso8859-8)				. iso-8859-8)
    ((ascii latin-iso8859-9)				. iso-8859-9)
    ((ascii latin-jisx0201
	    japanese-jisx0208-1978 japanese-jisx0208)	. iso-2022-jp)
    ((ascii latin-jisx0201
	    katakana-jisx0201 japanese-jisx0208)	. shift_jis)
    ((ascii korean-ksc5601)				. euc-kr)
    ((ascii chinese-gb2312)				. cn-gb-2312)
    ((ascii chinese-big5-1 chinese-big5-2)		. cn-big5)
    ((ascii latin-iso8859-1 greek-iso8859-7
	    latin-jisx0201 japanese-jisx0208-1978
	    chinese-gb2312 japanese-jisx0208
	    korean-ksc5601 japanese-jisx0212)		. iso-2022-jp-2)
    ((ascii latin-iso8859-1 greek-iso8859-7
	    latin-jisx0201 japanese-jisx0208-1978
	    chinese-gb2312 japanese-jisx0208
	    korean-ksc5601 japanese-jisx0212
	    chinese-cns11643-1 chinese-cns11643-2)	. iso-2022-int-1)
    ((ascii latin-iso8859-1 latin-iso8859-2
	    cyrillic-iso8859-5 greek-iso8859-7
	    latin-jisx0201 japanese-jisx0208-1978
	    chinese-gb2312 japanese-jisx0208
	    korean-ksc5601 japanese-jisx0212
	    chinese-cns11643-1 chinese-cns11643-2
	    chinese-cns11643-3 chinese-cns11643-4
	    chinese-cns11643-5 chinese-cns11643-6
	    chinese-cns11643-7)				. iso-2022-int-1)
    ))

Here is a bit of documentation for this variable (from `semi'
package):

If you want to use non-ISO-8859-1 charset in Emacs 19 or XEmacs
without mule, please set variable `default-mime-charset'.  This
variable must be symbol of which name is a MIME charset.

If you want to add more charsets in mule, please set variable
`charsets-mime-charset-alist'.  This variable must be alist of which
key is list of charset and value is symbol of MIME charset.  If name
of coding-system is different as MIME charset, please set variable
`mime-charset-coding-system-alist'.  This variable must be alist of
which key is MIME charset and value is coding-system.

I.e., this variable provides a customizable way of choosing the
preferred mime charsets for lists of mule charsets. E.g., if the
buffer contains symbols from ascii or cyrillic-iso8859-5, then the
mime-charset (and encoding of the message part) should be chosen as
koi8-r, and so on. Very nice imho.

The `apel' package contains also emulation modules for different
versions of emacsen (including xemacs, mule, nemacs, etc), and it
provides a uniform interface independent of the emacs version/flavor.
Apel is a stand-alone package, but it must be installed if one uses
semi-gnus (the packages are: apel, flim, semi, semi-gnus).
ftp://ftp.jaist.ac.jp/pub/GNU/elisp/apel/apel-8.17.tar.gz

I hope that pgnus will provide the similar functionality (or will
utilize apel).

	Best regards, -- Vladimir.

Re: \201 problem reproduced

[Vladimir Volovich]

"LMI" == Lars Magne Ingebrigtsen writes:

 LMI> Could you try to investigate how semi-gnus finds this out?

One more observation. :) I've looked at the standard emacs
multilingual support and found that for many (each?) language
environments there is a big framework of defined info. In particular,
for the Russian language, the corresponding stuff is defined in the
file lisp/language/cyrillic.el. There are e.g. settings of
language-info-alists for each language environment from which gnus can
get the needed info (like the default mime charset for the language
environment Cyrillic-KOI and functions/tables needed to decode/encode
to/from that charset). Also, try "M-x apropos enter koi8 enter", and
there will be a few interesting variables (defined mainly in
cyrillic.el).

Thanks a lot.

	Best regards, -- Vladimir.

Re: \201 problem reproduced

[Hallvard B Furuseth]

Right.  The closest you can get with stuff like this in mm-utils:

(defun mm-mime-charset (charset b e)
  (if (fboundp 'coding-system-get)
      (or
>>>    (coding-system-get
>>>     (get-charset-property charset 'prefered-coding-system)
        'mime-charset)
       (car (memq charset (find-coding-systems-region
                           (point-min) (point-max)))))
    (mm-mule-charset-to-mime-charset charset)))

is probably to first check if the session's preferred coding system or
whatever supports <charset> and has a 'mime-charset.  Apparently
something like this (unless MULE has a function to do it already):

    (if (let ((safe (coding-system-get <preferred-MIME-coding-system>
				       'safe-charsets)))
	  (or (eq safe t) (memq charset safe)))
	(coding-system-get <preferred-MIME-coding-system> 'mime-charset)

...except that we want the inverse of 'safe-charsets':-( safe-charsets
means the charset can encode the coding system, not that the coding
system can encode the charset.  koi8 can't encode all of iso8859-5.

The only real solution is to *not* deal with MULE charsets, but go
directly from MULE coding system to MIME charset.  But maybe that's more
work and won't happen that fast.

-- 
Hallvard

Re: \201 problem reproduced

[Lars Magne Ingebrigtsen]

Hallvard B Furuseth <h.b.furuseth@usit.uio.no> writes:

> is probably to first check if the session's preferred coding system or
> whatever supports <charset> and has a 'mime-charset.  Apparently
> something like this (unless MULE has a function to do it already):
> 
>     (if (let ((safe (coding-system-get <preferred-MIME-coding-system>
> 				       'safe-charsets)))
> 	  (or (eq safe t) (memq charset safe)))
> 	(coding-system-get <preferred-MIME-coding-system> 'mime-charset)

Right.  I've now tried this, and added a new variable
`mm-default-coding-system.  Vladimir, could you try setting this to
`koi8' and see what happens?

I still don't quite understand all this...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: \201 problem reproduced

[Vladimir Volovich]

"LMI" == Lars Magne Ingebrigtsen writes:

 LMI> Right.  I've now tried this, and added a new variable
 LMI> `mm-default-coding-system.

Does it sound reasonable to get the default for this variable
automagically from language-alist?

 LMI> Vladimir, could you try setting this to `koi8' and see what
 LMI> happens?

Yes! It works! Thanks!!! :-)))

One minor glitch: when i try to use russian letters in headers, i'm
unable to send my messages: when i press C-c C-c, Emacs writes "End of
buffer", and my message looks like:

----------------------------------------------------------------
To: root
Subject: test =D4=C5=D3=D4 test
From: Vladimir Volovich <vvv@vvv.vsu.ru>
Date: 13 Sep 1998 12:56:32 +0400
Message-ID: <m3hfyc1jcv.fsf@vvv.vsu.ru>
Lines: 7
User-Agent: Gnus/5.07003 (Pterodactyl Gnus v0.30) Emacs/20.3
--text follows this line--
Проба пера

	Best regards, -- Vladimir.
----------------------------------------------------------------

	Best regards, -- Vladimir.
-- 
Think of your family tonight.  Try to crawl home after the computer crashes.

Re: \201 problem reproduced

[Lars Magne Ingebrigtsen]

Vladimir Volovich <vvv@vvv.vsu.ru> writes:

> One minor glitch: when i try to use russian letters in headers, i'm
> unable to send my messages: when i press C-c C-c, Emacs writes "End of
> buffer", and my message looks like:

`(setq debug-on-error t)', repeat the bug and mail me the resulting
backtrace.

You may also have to `(setq debug-ignored-errors nil)'.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: \201 problem reproduced

[Vladimir Volovich]

"LMI" == Lars Magne Ingebrigtsen writes:

 >> One minor glitch: when i try to use russian letters in headers,
 >> i'm unable to send my messages: when i press C-c C-c, Emacs writes
 >> "End of buffer", and my message looks like:

 LMI> `(setq debug-on-error t)', repeat the bug and mail me the
 LMI> resulting backtrace.

Signaling: (end-of-buffer)
  rfc2047-q-encode-region(42 46)
  rfc2047-encode(42 46 cyrillic-iso8859-5)
  rfc2047-encode-region(28 52)
  mail-encode-encoded-word-buffer()
  message-send-mail(nil)
  message-send-via-mail(nil)
  message-send(nil)
  message-send-and-exit(nil)
* call-interactively(message-send-and-exit)

After an error, the *mail* buffer appeared and it contained only one
line:
=D4=C5=D3=D4.

Perhaps, gnus cannot encode-word into koi8-r (and i'd like it to
encode to koi8-r if i set mm-default-coding-system to koi8)?

 LMI> You may also have to `(setq debug-ignored-errors nil)'.

Yes. ;-)

BTW, why gnus does not generate mime-version and content-type header
for us-ascii text/plain messages? And: is it legal to not generate the
mime-version header if some of the headers are encoded using
encoded-word, but the bidy is still us-ascii? imho, the presence of
mime-version and c-t is good in all cases.

	Best regards, -- Vladimir.
-- 
Chairman of the Bored.

Re: \201 problem reproduced

[Lars Magne Ingebrigtsen]

Vladimir Volovich <vvv@vvv.vsu.ru> writes:

> Signaling: (end-of-buffer)
>   rfc2047-q-encode-region(42 46)

Yup.  Fix in Pterodactyl Gnus v0.31.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: \201 problem reproduced

[Vladimir Volovich]

"LMI" == Lars Magne Ingebrigtsen writes:

 >> Signaling: (end-of-buffer) rfc2047-q-encode-region(42 46)
 LMI> Yup.  Fix in Pterodactyl Gnus v0.31.

Nope. Now, if i (setq mm-default-coding-system 'koi8) and compose a
message which contains russian in headers (Subject, From, etc), and
press C-c C-c, gnus prints "Sending..." and hangs forever. Backtrace
after pressing C-g:

Signaling: (quit)
  search-backward("=" nil 19337)
  rfc2047-q-encode-region(24 28)
  rfc2047-encode(24 28 cyrillic-iso8859-5)
  rfc2047-encode-region(10 34)
  mail-encode-encoded-word-buffer()
  message-send-mail(nil)
  message-send-via-mail(nil)
  message-send(nil)
  message-send-and-exit(nil)
* call-interactively(message-send-and-exit)

	Best regards, -- Vladimir.

Re: \201 problem reproduced

[Lars Magne Ingebrigtsen]

Vladimir Volovich <vvv@vvv.vsu.ru> writes:

> Signaling: (quit)
>   search-backward("=" nil 19337)
>   rfc2047-q-encode-region(24 28)

Yup.  Fix in Pterodactyl Gnus v0.32.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen