mm-coding-system-priorities per hierarchy

mm-coding-system-priorities per hierarchy

[Reiner Steib]

Hi,

I've added this explanation about setting
`mm-coding-system-priorities' per hierarchy to texi/emacs-mime.texi.
Corrections and additions are welcome!

--8<---------------cut here---------------start------------->8---
@c Corrections about preferred charsets are welcome.  de, fr and fj
@c should be correct, I don't know about the rest (so these are only
@c examples):
@lisp
(add-to-list 'gnus-newsgroup-variables 'mm-coding-system-priorities)
(setq gnus-parameters
      (nconc
       ;; Some charsets are just examples!
       '(("^cn\\." ;; Chinese
	  (mm-coding-system-priorities
	   '(iso-8859-1 cn-big5 chinese-iso-7bit utf-8)))
	 ("^cz\\.\\|^pl\\." ;; Central and Eastern European
	  (mm-coding-system-priorities '(iso-8859-2 utf-8)))
	 ("^de\\." ;; German language
	  (mm-coding-system-priorities '(iso-8859-1 iso-8859-15 utf-8)))
	 ("^fr\\." ;; French
	  (mm-coding-system-priorities '(iso-8859-15 iso-8859-1 utf-8)))
	 ("^fj\\." ;; Japanese
	  (mm-coding-system-priorities
	   '(iso-8859-1 iso-2022-jp iso-2022-jp-2 shift_jis utf-8)))
	 ("^ru\\." ;; Cyrillic
	  (mm-coding-system-priorities
	   '(koi8-r iso-8859-5 iso-8859-1 utf-8))))
       gnus-parameters))
--8<---------------cut here---------------end--------------->8---

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: mm-coding-system-priorities per hierarchy

[Katsumi Yamaoka]

>>>>> In <v9hcz21obx.fsf@marauder.physik.uni-ulm.de>
>>>>>	Reiner Steib wrote:

> I've added this explanation about setting
> `mm-coding-system-priorities' per hierarchy to texi/emacs-mime.texi.
> Corrections and additions are welcome!

This has to be used with care.  I verified that the Japanese
mail message written when the de.test group was selected or just
after exiting the de.test group was sent with the utf-8 encoding
even if iso-2022-jp is better to be used.

Re: mm-coding-system-priorities per hierarchy

[Reiner Steib]

On Thu, Sep 21 2006, Katsumi Yamaoka wrote:

> This has to be used with care.  I verified that the Japanese
> mail message written when the de.test group was selected or just
> after exiting the de.test group was sent with the utf-8 encoding
> even if iso-2022-jp is better to be used.

I think the documentation of `gnus-newsgroup-variables' should
actually be improved and it should describe what
`gnus-summary-make-local-variables' does [1] (or maybe I just didn't
read the doc string carefully enough [2]):

When using this expression...

  (add-to-list 'gnus-newsgroup-variables 'mm-coding-system-priorities)

... `gnus-summary-make-local-variables' sets the global value of
`mm-coding-system-priorities' to nil.  I'd guess nil leads to utf-8
instead of iso-2022-jp in your environment.

Do you get the expected behavior with this expression?

  (add-to-list 'gnus-newsgroup-variables
	       '(mm-coding-system-priorities . global))

(Thank for fixing my commit.)

Bye, Reiner.

[1]
--8<---------------cut here---------------start------------->8---
(defun gnus-summary-make-local-variables ()
  "Make all the local summary buffer variables."
  (let (global)
    (dolist (local gnus-summary-local-variables)
      (if (consp local)
	  (progn
	    (if (eq (cdr local) 'global)
		;; Copy the global value of the variable.
		(setq global (symbol-value (car local)))
	      ;; Use the value from the list.
	      (setq global (eval (cdr local))))
	    (set (make-local-variable (car local)) global))
	;; Simple nil-valued local variable.
	(set (make-local-variable local) nil)))))
--8<---------------cut here---------------end--------------->8---

,----[ <f1> v gnus-newsgroup-variables RET ]
| gnus-newsgroup-variables is a variable defined in `gnus-sum'.
| Its value is nil
| 
| Documentation:
| A list of variables that have separate values in different newsgroups.
| A list of newsgroup (summary buffer) local variables, or cons of
| variables and their default expressions to be evalled (when the default
| values are not nil), that should be made global while the summary buffer
| is active.
| 
| Note: The default expressions will be evaluated (using function `eval')
| before assignment to the local variable rather than just assigned to it.
| If the default expression is the symbol `global', that symbol will not
| be evaluated but the global value of the local variable will be used
| instead.
| 
| These variables can be used to set variables in the group parameters
| while still allowing them to affect operations done in other buffers.
| For example:
| 
| (setq gnus-newsgroup-variables
|      '(message-use-followup-to
|        (gnus-visible-headers .
| 	 "^From:\\|^Newsgroups:\\|^Subject:\\|^Date:\\|^To:")))
`----

-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: mm-coding-system-priorities per hierarchy

[Katsumi Yamaoka]

>>>>> In <v9odt92u9c.fsf@marauder.physik.uni-ulm.de>
>>>>>	Reiner Steib wrote:

> On Thu, Sep 21 2006, Katsumi Yamaoka wrote:

>> This has to be used with care.  I verified that the Japanese
>> mail message written when the de.test group was selected or just
>> after exiting the de.test group was sent with the utf-8 encoding
>> even if iso-2022-jp is better to be used.

> I think the documentation of `gnus-newsgroup-variables' should
> actually be improved and it should describe what
> `gnus-summary-make-local-variables' does [1] (or maybe I just didn't
> read the doc string carefully enough [2]):

I had learned how those group-local variables work by reading
the source code.

> When using this expression...

>   (add-to-list 'gnus-newsgroup-variables 'mm-coding-system-priorities)

> ... `gnus-summary-make-local-variables' sets the global value of
> `mm-coding-system-priorities' to nil.  I'd guess nil leads to utf-8
> instead of iso-2022-jp in your environment.

No, it is not caused by the nil value.

> Do you get the expected behavior with this expression?

>   (add-to-list 'gnus-newsgroup-variables
> 	       '(mm-coding-system-priorities . global))

That is not good for me.  It should be the following:

  (add-to-list
   'gnus-newsgroup-variables
   '(mm-coding-system-priorities
     . '(iso-8859-1 iso-2022-jp iso-2022-jp-2 shift_jis utf-8)))

The value here is the default value of `mm-coding-system-priorities'
in the Japanese language environment.  Though it is insufficient
for me even if I do so.

If I set `gnus-parameters' (as you suggested) in addition to
this, the value of `mm-coding-system-priorities' is changed into
(iso-8859-1 iso-8859-15 utf-8) when I enter to a DE newsgroup.
It is useful for posting German messages there (it is highly
unlikely with me, though ;-).  However, I should not send
Japanese mails in there even if I recall an urgent business.  It
is because `iso-2022-jp', which is used for Japanese messages
traditionally, is not preferred when encoding Japanese text.

It is possible to send Japanese messages safely after I exit the
DE group, isn't it?  *No*.  `mm-coding-system-priorities' still
has the value (iso-8859-1 iso-8859-15 utf-8) at that time.  The
value will not contain `iso-2022-jp' until I enter to a FJ group
or a group that is not listed in `gnus-parameters'.

What I'd like to say is that a similar problem will probably
happen with people who are not in the Japanes language
environment as well.  Group parameter is useful to things
associated to group, however messages are not always associated
to group.

Regards,

Re: mm-coding-system-priorities per hierarchy

[Reiner Steib]

On Fri, Sep 22 2006, Katsumi Yamaoka wrote:

>>>>>>	Reiner Steib wrote:
>> When using this expression...
>
>>   (add-to-list 'gnus-newsgroup-variables 'mm-coding-system-priorities)
>
>> ... `gnus-summary-make-local-variables' sets the global value of
>> `mm-coding-system-priorities' to nil.  I'd guess nil leads to utf-8
>> instead of iso-2022-jp in your environment.
>
> No, it is not caused by the nil value.

IIRC, if `mm-coding-system-priorities' is nil, Gnus uses Emacs'
`sort-coding-systems' to find the suitable charset.  So my guess was
that utf-8 is preferred to iso-2022-jp...

,----
| $ LC_ALL=C emacs -nw -Q -f ielm
|
| ELISP> current-language-environment
| "ASCII"
| ELISP> (sort-coding-systems '(iso-8859-1 iso-2022-jp iso-2022-jp-2 shift_jis utf-8))
| (iso-8859-1 iso-2022-jp shift_jis utf-8 iso-2022-jp-2)
| 
| ELISP> (set-language-environment "Japanese")
| t
| ELISP> (sort-coding-systems '(iso-8859-1 iso-2022-jp iso-2022-jp-2 shift_jis utf-8))
| (iso-2022-jp shift_jis iso-8859-1 utf-8 iso-2022-jp-2)
`----

But maybe...

,----
| $ LC_ALL=jp_JP.utf8 emacs -nw -Q -f ielm
| 
| ELISP> (sort-coding-systems '(iso-8859-1 iso-2022-jp iso-2022-jp-2 shift_jis utf-8))
| (utf-8 iso-2022-jp shift_jis iso-8859-1 iso-2022-jp-2)
`----

>> Do you get the expected behavior with this expression?
>
>>   (add-to-list 'gnus-newsgroup-variables
>> 	       '(mm-coding-system-priorities . global))
>
> That is not good for me.  It should be the following:
>
>   (add-to-list
>    'gnus-newsgroup-variables
>    '(mm-coding-system-priorities
>      . '(iso-8859-1 iso-2022-jp iso-2022-jp-2 shift_jis utf-8)))
>
> The value here is the default value of `mm-coding-system-priorities'
> in the Japanese language environment.  

My understanding is that `global' is supposed to take this value.  Is
it a bug (in gnus-sum.el) or am I missing something?

> Though it is insufficient for me even if I do so.
>
> If I set `gnus-parameters' (as you suggested) in addition to
> this, the value of `mm-coding-system-priorities' is changed into
> (iso-8859-1 iso-8859-15 utf-8) when I enter to a DE newsgroup.
> It is useful for posting German messages there (it is highly
> unlikely with me, though ;-).  However, I should not send
> Japanese mails in there even if I recall an urgent business.  It
> is because `iso-2022-jp', which is used for Japanese messages
> traditionally, is not preferred when encoding Japanese text.
>
> It is possible to send Japanese messages safely after I exit the
> DE group, isn't it?  *No*.  `mm-coding-system-priorities' still
> has the value (iso-8859-1 iso-8859-15 utf-8) at that time.  The
> value will not contain `iso-2022-jp' until I enter to a FJ group
> or a group that is not listed in `gnus-parameters'.
>
> What I'd like to say is that a similar problem will probably
> happen with people who are not in the Japanes language
> environment as well.  Group parameter is useful to things
> associated to group, however messages are not always associated
> to group.

I think it is (also?) a bug if entering a group modifies the
default-value of a variable.  It probably should only modify the local
values in the summary, article and message buffers for the matching
groups.

The code of `gnus-summary-make-local-variables' seems to be correct.
Maybe `gnus-set-global-variables' or some other function is the
culprit?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: mm-coding-system-priorities per hierarchy

[Katsumi Yamaoka]

>>>>> In <v9slihtn7g.fsf@marauder.physik.uni-ulm.de>
>>>>>	Reiner Steib wrote:

> On Fri, Sep 22 2006, Katsumi Yamaoka wrote:

>>>>>>>	Reiner Steib wrote:
>>> When using this expression...
>>
>>>   (add-to-list 'gnus-newsgroup-variables 'mm-coding-system-priorities)
>>
>>> ... `gnus-summary-make-local-variables' sets the global value of
>>> `mm-coding-system-priorities' to nil.  I'd guess nil leads to utf-8
>>> instead of iso-2022-jp in your environment.
>>
>> No, it is not caused by the nil value.

> IIRC, if `mm-coding-system-priorities' is nil, Gnus uses Emacs'
> `sort-coding-systems' to find the suitable charset.  So my guess was
> that utf-8 is preferred to iso-2022-jp...

Yes, you're right in the generality of cases.  I'm sorry I forgot
to mention that I use the ja_JP.eucjp locale (I use it to keep
the compatibility with the old programs even though the recent
Linux system uses ja_JP.utf8 by default).  In that case, euc-jp
is preferred to iso-2022-jp and utf-8.  So, utf-8 should never be
chosen to encode non-special Japanese text if
`mm-coding-system-priorities' is nil.  Here are examples observed
in my Linux system:

(mapcar 'symbol-value coding-category-list)
 => (japanese-iso-8bit iso-2022-jp japanese-shift-jis
     iso-2022-jp-2 iso-latin-1 mule-utf-8
     mule-utf-16be-with-signature mule-utf-16le-with-signature
     iso-2022-7bit iso-2022-8bit-ss2 emacs-mule raw-text
     chinese-big5 nil no-conversion)

(with-temp-buffer
  (insert "日本語")
  (let ((mm-coding-system-priorities nil))
    (mm-find-mime-charset-region (point-min) (point-max))))
 => (euc-jp)

(with-temp-buffer
  (insert "日本語")
  (let ((mm-coding-system-priorities
	 '(iso-8859-1 iso-8859-15 utf-8)))
    (mm-find-mime-charset-region (point-min) (point-max))))
 => (utf-8)

>>> Do you get the expected behavior with this expression?
>>
>>>   (add-to-list 'gnus-newsgroup-variables
>>> 	       '(mm-coding-system-priorities . global))
>>
>> That is not good for me.  It should be the following:
>>
>>   (add-to-list
>>    'gnus-newsgroup-variables
>>    '(mm-coding-system-priorities
>>      . '(iso-8859-1 iso-2022-jp iso-2022-jp-2 shift_jis utf-8)))
>>
>> The value here is the default value of `mm-coding-system-priorities'
>> in the Japanese language environment.

> My understanding is that `global' is supposed to take this value.

In the case where I use `global', `mm-coding-system-priorities'
is changed into (iso-8859-1 iso-8859-15 utf-8) if I enter to a
DE group even once, and it is unchanged after I exit the group.
In other words, the `global' value is changed into (iso-8859-1
iso-8859-15 utf-8).  There is no means to save the initial
(global) value.  So, it keeps the value suitable to DE groups
even if I enter for the next time to a group that is not listed
in `gnus-parameters'.

> Is it a bug (in gnus-sum.el) or am I missing something?

I'm not quite sure of it but it might be a bug in the sense that
the global value is not always the initial value.

[...]

>> What I'd like to say is that a similar problem will probably
>> happen with people who are not in the Japanes language
>> environment as well.  Group parameter is useful to things
>> associated to group, however messages are not always associated
>> to group.

> I think it is (also?) a bug if entering a group modifies the
> default-value of a variable.  It probably should only modify the local
> values in the summary, article and message buffers for the matching
> groups.

Well, I think it will not go well.  The values specific to the
group that is currently focused by a user should be seen in any
buffers (except for the other summary buffers which visit the
other groups).  It is because a program which works for an
article or a summary associated with the group might run in a
temp buffer, a process buffer, etc.

> The code of `gnus-summary-make-local-variables' seems to be correct.
> Maybe `gnus-set-global-variables' or some other function is the
> culprit?

I think the group parameter was designed for reading articles
and displaying summaries but did not intend to use with messages
to send (though it is actually useful only for sending messages
to the current newsgroup).  The other framework will be necessary.
It reads Newsgroups header, To header, etc. in a message buffer
and decides what to do.

Regards,