with pgnus 0.55 and 0.56 i'm unable to see text in russian

with pgnus 0.55 and 0.56 i'm unable to see text in russian

[Vladimir Volovich]

Hi,

as i reported some time before, pgnus 0.55 broke displaying of
articles in russian. 0.54 worked fine. here is what happens:

(1) in nnmbox, russian articles with C-T-E=8bit are displayed Ok

(2) in nnmbox, russian articles with C-T-E=QP are displayed buggy
    (\214<8-bit-char>)

(3) in nnmbox, russian multipart/alternative articles which have html
    part with C-T-E=QP (which are rendered with w3), are displayed Ok
    (i.e. russian characters look fine) except that background color
    is only at positions with letters; empty lines and space at the
    end of lines is not colored with background color, but this is
    another issue.

(4) in nntp newsgroups, i see latin1 instead of koi8-r 8-bit articles
    without C-T-E (regardless of setting of mm-default-coding-system
    to 'koi8)

Some observations: i tried to debug the function mm-display-inline;
and in pgnus-0.55 the text which finally appears in the *Article*
buffer gets corrupted when the function mm-encode-coding-region is
called (before that, the *Article* buffer already contains russian
text properly encoded, so this is the second encoding which breaks
things).

	Best regards, -- Vladimir.

Re: with pgnus 0.55 and 0.56 i'm unable to see text in russian

[Shenghuo ZHU]

>>>>> "VVV" == Vladimir Volovich <vvv@vvv.vsu.ru> writes:

VVV> Hi,
VVV> as i reported some time before, pgnus 0.55 broke displaying of
VVV> articles in russian. 0.54 worked fine. here is what happens:

VVV> (1) in nnmbox, russian articles with C-T-E=8bit are displayed Ok

VVV> (2) in nnmbox, russian articles with C-T-E=QP are displayed buggy
VVV>     (\214<8-bit-char>)

VVV> (3) in nnmbox, russian multipart/alternative articles which have html
VVV>     part with C-T-E=QP (which are rendered with w3), are displayed Ok
VVV>     (i.e. russian characters look fine) except that background color
VVV>     is only at positions with letters; empty lines and space at the
VVV>     end of lines is not colored with background color, but this is
VVV>     another issue.

VVV> (4) in nntp newsgroups, i see latin1 instead of koi8-r 8-bit articles
VVV>     without C-T-E (regardless of setting of mm-default-coding-system
VVV>     to 'koi8)

VVV> Some observations: i tried to debug the function mm-display-inline;
VVV> and in pgnus-0.55 the text which finally appears in the *Article*
VVV> buffer gets corrupted when the function mm-encode-coding-region is
VVV> called (before that, the *Article* buffer already contains russian
VVV> text properly encoded, so this is the second encoding which breaks
VVV> things).

I can not reproduce the bug in Emacs 20.3. 

I guess the problem is that "quoted-printable-decode-region" runs in a
non-"unibyte" buffer. To verify, do the following steps,

1. set breakpoint at quoted-printable-decode-region;
2. select a QP Russian article;
3. run quoted-printable-decode-region by pressing "c" in *Backtrace*;
4. check whether Russian characters are shown in *Article <blah>* 
   buffer

If true, the bug is "quoted-printable-decode-region" runs in a
non-"unibyte" buffer.

BTW, what is difference between koi8-r and koi8? In XEmacs, XEmacs
20.4 can not show Russian characters in koi8-r article.

-- 
Shenghuo

Re: with pgnus 0.55 and 0.56 i'm unable to see text in russian

[Vladimir Volovich]

"ZSH" == Shenghuo ZHU writes:

 ZSH> I can not reproduce the bug in Emacs 20.3.

My previous test article was incorrect, sorry. The text/plain messages
with one part, even if they are QP-encoded, are displayed fine. The
problem appears in multipart messages. E.g multipart/alternative in
which the text part is QP-encoded, is displayed buggily. I'm sending a
sample message (which _definitely_ is viewed buggy for me) to you in a
private mail. And: the 8-bit articles in koi8-r which do not have C-T
header, are displayed incorrectly: i see iso-8859-1 characters instead
of russian characters, but i have

  (setq mm-default-coding-system 'koi8)

in my ~/.emacs file. This worked fine with 0.54, but was broken in 0.55.

 ZSH> I guess the problem is that "quoted-printable-decode-region"
 ZSH> runs in a non-"unibyte" buffer. To verify, do the following
 ZSH> steps,

i'll try this, and will report soon.

 ZSH> BTW, what is difference between koi8-r and koi8?

No difference at all. koi8-r is the correct name. Koi8 is an
alias. There are some variants of koi8-r which are not yet
standardized, --- e.g. koi8-u, koi8-ru, koi8-rub (extensions for the
Ukrainian and Byelorussian languages). But koi8-r is a standard
charset specified in RFC.

 ZSH> In XEmacs, XEmacs 20.4 can not show Russian characters in koi8-r
 ZSH> article.

That's very strange. What does it show? Does adding of the line e.g.
;-*-coding: koi8-r;-*-
help?

	Best regards, -- Vladimir.

Re: with pgnus 0.55 and 0.56 i'm unable to see text in russian

[Shenghuo ZHU]

>>>>> "VVV" == Vladimir Volovich <vvv@vvv.vsu.ru> writes:

VVV> My previous test article was incorrect, sorry. The text/plain messages
VVV> with one part, even if they are QP-encoded, are displayed fine. The
VVV> problem appears in multipart messages. E.g multipart/alternative in
VVV> which the text part is QP-encoded, is displayed buggily. I'm sending a
VVV> sample message (which _definitely_ is viewed buggy for me) to you in a
VVV> private mail. And: the 8-bit articles in koi8-r which do not have C-T
VVV> header, are displayed incorrectly: i see iso-8859-1 characters instead
VVV> of russian characters, but i have

VVV>   (setq mm-default-coding-system 'koi8)

VVV> in my ~/.emacs file. This worked fine with 0.54, but was broken in 0.55.

Fixed. A patch is attached.

ChangeLog:
Mon Nov 30 13:52:50 1998  Shenghuo ZHU  <zsh@cs.rochester.edu>

	* mm-bodies.el (mm-decode-string): New function.
	* mm-view.el (mm-inline-text): Use mm-decode-string.


Reason: (cited from "C-h f insert")

If the current buffer is multibyte, unibyte strings are converted
to multibyte for insertion (see `unibyte-char-to-multibyte').
If the current buffer is unibyte, multibyte strings are converted
to unibyte for insertion.

ZSH> I guess the problem is that "quoted-printable-decode-region"
ZSH> runs in a non-"unibyte" buffer. To verify, do the following
ZSH> steps,

VVV> i'll try this, and will report soon.

There is no bug.

ZSH> BTW, what is difference between koi8-r and koi8?

VVV> No difference at all. koi8-r is the correct name. Koi8 is an
VVV> alias. There are some variants of koi8-r which are not yet
VVV> standardized, --- e.g. koi8-u, koi8-ru, koi8-rub (extensions for the
VVV> Ukrainian and Byelorussian languages). But koi8-r is a standard
VVV> charset specified in RFC.

ZSH> In XEmacs, XEmacs 20.4 can not show Russian characters in koi8-r
ZSH> article.

VVV> That's very strange. What does it show? Does adding of the line e.g.
VVV> ;-*-coding: koi8-r;-*-
VVV> help?

I guess there is no coding-system named "koi8-r" in XEmacs 20.4. So,
mm-mime-mule-charset-alist needs to be fixed.

-- 
Shenghuo


:- cut ----------------------------
--- mm-bodies.el	1998/11/30 18:45:04	1.1
+++ mm-bodies.el	1998/11/30 19:01:23
@@ -163,6 +163,18 @@
 		       (setq mule-charset rfc2047-default-charset)))
 	  (mm-decode-coding-region (point-min) (point-max) mule-charset))))))
 
+(defun mm-decode-string (string charset)
+  "Decode STRING with CHARSET."
+  (setq charset (or charset rfc2047-default-charset))
+  (when (featurep 'mule)
+    (let (mule-charset)
+      (when (and charset
+		 (setq mule-charset (mm-charset-to-coding-system charset))
+		 enable-multibyte-characters
+		 (or (not (eq mule-charset 'ascii))
+		     (setq mule-charset rfc2047-default-charset)))
+	(mm-decode-coding-string string mule-charset)))))
+
 (provide 'mm-bodies)
 
 ;; mm-bodies.el ends here

--- mm-view.el	1998/11/30 18:29:57	1.1
+++ mm-view.el	1998/11/30 18:51:51
@@ -54,13 +54,12 @@
     (cond
      ((equal type "plain")
       (setq text (mm-get-part handle))
-      (let ((b (point)))
-	(insert text)
+      (let ((b (point))
+	    (charset (mail-content-type-get
+		      (mm-handle-type handle) 'charset)))
+	(insert (mm-decode-string text charset))
 	(save-restriction
 	  (narrow-to-region b (point))
-	  (let ((charset (mail-content-type-get
-			  (mm-handle-type handle) 'charset)))
-	    (mm-decode-body charset nil))
 	  (mm-handle-set-undisplayer
 	   handle
 	   `(lambda ()

Re: with pgnus 0.55 and 0.56 i'm unable to see text in russian

[Vladimir Volovich]

"ZSH" == Shenghuo ZHU writes:

 ZSH> Fixed. A patch is attached.

Thanks a lot, i'll try it. BTW, i just noticed that in 0.55 there was
a change in mm-view.el (mm-inline-text), which in effect changed
with-temp-buffer to mm-with-unibyte-buffer (which is used in
mm-get-part function). Maybe, changing mm-with-unibyte-buffer to
with-temp-buffer in the definition of mm-get-part will fix this
problem, too (which seems like a simpler fix)?

 ZSH> I guess the problem is that "quoted-printable-decode-region"
 ZSH> runs in a non-"unibyte" buffer. To verify, do the following
 ZSH> steps,
 VVV> i'll try this, and will report soon.
 ZSH> There is no bug.

You mean, there is no bug in quoted-printable-decode-region?
Perhaps. But there is still a bug in displaying multipart QP messages
(i sent tests to the list several minutes ago).

	Best regards, -- Vladimir.

Re: with pgnus 0.55 and 0.56 i'm unable to see text in russian

[Shenghuo ZHU]

>>>>> "VVV" == Vladimir Volovich <vvv@vvv.vsu.ru> writes:

VVV> Thanks a lot, i'll try it. BTW, i just noticed that in 0.55 there
VVV> was a change in mm-view.el (mm-inline-text), which in effect
VVV> changed with-temp-buffer to mm-with-unibyte-buffer (which is used
VVV> in mm-get-part function). Maybe, changing mm-with-unibyte-buffer
VVV> to with-temp-buffer in the definition of mm-get-part will fix
VVV> this problem, too (which seems like a simpler fix)?

I did not find the change in ChangeLog. I did the last change in 0.53.

VVV> You mean, there is no bug in quoted-printable-decode-region?
VVV> Perhaps. But there is still a bug in displaying multipart QP
VVV> messages (i sent tests to the list several minutes ago).

I can see the nice boxes (I did not install Russian fonts) after
patching.

-- 
Shenghuo

Re: with pgnus 0.55 and 0.56 i'm unable to see text in russian

[Vladimir Volovich]

[-- Attachment #1: Type: text/plain, Size: 588 bytes --]

"ZSH" == Shenghuo ZHU writes:

 VVV> Thanks a lot, i'll try it. BTW, i just noticed that in 0.55
 VVV> there was a change in mm-view.el (mm-inline-text), which in
 VVV> effect changed with-temp-buffer to mm-with-unibyte-buffer (which
 VVV> is used in mm-get-part function). Maybe, changing
 VVV> mm-with-unibyte-buffer to with-temp-buffer in the definition of
 VVV> mm-get-part will fix this problem, too (which seems like a
 VVV> simpler fix)?

 ZSH> I did not find the change in ChangeLog. I did the last change in
 ZSH> 0.53.

there is the following hunk in the pgnus-0.54-0.55.diff:


[-- Attachment #2: Type: text/plain, Size: 1081 bytes --]


  (defun mm-inline-text (handle)
    (let ((type (cadr (split-string (car (mm-handle-type handle)) "/")))
  	text buffer-read-only)
      (cond
       ((equal type "plain")
!       (with-temp-buffer
! 	(insert-buffer-substring (mm-handle-buffer handle))
! 	(mm-decode-content-transfer-encoding
! 	 (mm-handle-encoding handle)
! 	 (car (mm-handle-type handle)))
! 	(setq text (buffer-string)))
        (let ((b (point)))
  	(insert text)
  	(save-restriction
--- 39,59 ----
      (set-extent-property annot 'mm t)
      (set-extent-property annot 'duplicable t)))
  
+ (defvar mm-w3-setup nil)
+ (defun mm-setup-w3 ()
+   (unless mm-w3-setup
+     (w3-do-setup)
+     (require 'url)
+     (require 'w3-vars)
+     (url-register-protocol 'cid nil 'url-identity-expander)
+     (setq mm-w3-setup t)))
+ 
  (defun mm-inline-text (handle)
    (let ((type (cadr (split-string (car (mm-handle-type handle)) "/")))
  	text buffer-read-only)
      (cond
       ((equal type "plain")
!       (setq text (mm-get-part handle))
        (let ((b (point)))
  	(insert text)
  	(save-restriction

[-- Attachment #3: Type: text/plain, Size: 702 bytes --]



i.e. it changes

!       (with-temp-buffer
! 	(insert-buffer-substring (mm-handle-buffer handle))
! 	(mm-decode-content-transfer-encoding
! 	 (mm-handle-encoding handle)
! 	 (car (mm-handle-type handle)))
! 	(setq text (buffer-string)))

to (setq text (mm-get-part handle)) in the definition of
mm-inline-text. But the definition of mm-get-part in mm-decode.el
contains mm-with-unibyte-buffer instead of with-temp-buffer. So, this
change is not 100% `compatible' with older behavior, and i guess that
if i will change mm-with-unibyte-buffer to with-temp-buffer in the
definition of mm-get-part, or if i will revert this hunk, the correct
behavior will be restored. Lars?

	Best regards, -- Vladimir.