bug in displaying inline message/rfc822

bug in displaying inline message/rfc822

[vvv]

[-- Attachment #1: Type: text/plain, Size: 370 bytes --]

Hi there,

this is a test message that demonstrates a bug in gnus. when i receive
this message (in nnmbox backend, if that matters), gnus tries to show the
message/rfc822 parts inline, but it fails: i see \214<8-bit-char> instead
of real characters. perhaps, this is because the CTT of message/rfc822
parts is 8bit, and gnus encodes the message buffer several times...


[-- Attachment #2: Type: message/rfc822, Size: 322 bytes --]

From: Vladimir Volovich <vvv@vvv.vsu.ru>
To: ding@gnus.org
Subject: Re: Разрыв формул
Date: 12 Apr 1999 21:23:01 +0400
Message-ID: <m3zp4d69e2.fsf@vvv.vsu.ru>

Это тест. Проба пера.

	Best regards, -- Vladimir.

[-- Attachment #3: Type: message/rfc822, Size: 390 bytes --]

From: Vladimir Volovich <vvv@vvv.vsu.ru>
To: ding@gnus.org
Subject: test áüöè test
Date: 18 Apr 1999 00:07:34 +0400
Message-ID: <m3r9pj9fjt.fsf@vvv.vsu.ru>

test áüöè test ÿåæ

	Best regards, -- Vladimir.
-- 
Music in the soul can be heard by the universe.
		-- Lao Tsu

Re: bug in displaying inline message/rfc822

[Kai.Grossjohann]

vvv@vvv.vsu.ru writes:

  > this is a test message that demonstrates a bug in gnus.

This is interesting.  In the first subject, I see boxes only (so maybe
the font is missing).  In the first body, I see \201 followed by
cyrillic (?) characters (but ascii characters are not preceded by
\201).  In the second subject, I see iso-8859-1 characters.  In the
second body, every non-ascii character is preceded by \201, as well.

And you see \214 and I presume that \214 is the internal representation
of koi8-r or iso-8859-* (what's the number for Cyrillic?).

Using nnml with pgnus 0.80.

kai
-- 
Abort this operation?   [Abort]  [Cancel]

Re: bug in displaying inline message/rfc822

[Hrvoje Niksic]

vvv@vvv.vsu.ru writes:

> this is a test message that demonstrates a bug in gnus. [...]

Both messages show up correctly in XEmacs.

> From: Vladimir Volovich <vvv@vvv.vsu.ru>
> Subject: Re: Разрыв формул
> To: ding@gnus.org
> Date: 12 Apr 1999 21:23:01 +0400
> Message-ID: <m3zp4d69e2.fsf@vvv.vsu.ru>
> MIME-Version: 1.0
> Content-Type: text/plain; charset=koi8-r
> Content-Transfer-Encoding: base64
> X-MIME-Autoconverted: from 8bit to base64 by regoc.srce.hr id WAA24757
> 
> Это тест. Проба пера.

Similarities between Russian and Croatian always amaze me.  In
Croatian it would mean something like "Here is a test.  A test of the
pen."  :-)

(I went to school back when Serbian Cyrillic script was being taught,
so I can read some of it.)

Re: bug in displaying inline message/rfc822

[Vladimir Volovich]

"Hrvoje" == Hrvoje Niksic writes:

 >> this is a test message that demonstrates a bug in gnus. [...]
 Hrvoje> Both messages show up correctly in XEmacs.

i'm afraid that this is not completely true, --- see below.

 >> From: Vladimir Volovich <vvv@vvv.vsu.ru> Subject: Re: Разрыв
 >> формул To: ding@gnus.org Date: 12 Apr 1999 21:23:01 +0400
 >> Message-ID: <m3zp4d69e2.fsf@vvv.vsu.ru> MIME-Version: 1.0
 >> Content-Type: text/plain; charset=koi8-r
 >> Content-Transfer-Encoding: base64 X-MIME-Autoconverted: from 8bit

here is it! the message was translated to 7-bit encoding (base64) and
appeared in your mbox in that form. if the message was not changed
(i.e. it should have contained _8-bit_ chars, with CTT=8-bit), then i
guess the bug will show even in XEmacs, because it is bug in gnus.

 >> to base64 by regoc.srce.hr id WAA24757
 >> 
 >> Это тест. Проба пера.

 Hrvoje> Similarities between Russian and Croatian always amaze me.
 Hrvoje> In Croatian it would mean something like "Here is a test.  A
 Hrvoje> test of the pen."  :-)

and how it would be in Croatian? :-)

 Hrvoje> (I went to school back when Serbian Cyrillic script was being
 Hrvoje> taught, so I can read some of it.)

	Best regards, -- Vladimir.
-- 
Even a cabbage may look at a king.

Re: bug in displaying inline message/rfc822

[Vladimir Volovich]

"KG" == Kai Grossjohann writes:

 KG> This is interesting.  In the first subject, I see boxes only (so
 KG> maybe the font is missing).

perhaps.

 KG> In the first body, I see \201 followed by cyrillic (?) characters

if you see cyrillic characters, you should have the cyrillic font, and
the first subject should look fine (without boxes). strange..

 KG> (but ascii characters are not preceded by \201). In the second
 KG> subject, I see iso-8859-1 characters.

the rule is: if the character is represented in the raw message on
disk in a 7-bit-safe form (e.g. inside base64, or encode-word or QP in
subject, etc), then it is not affected by "double encoding" of buffer
which leads to these \201s and \204s. But if the character in the raw
message is represented in some 8-bit charset using byte value above
128, then these characters sometime (in places where gnus have bugs ;)
appear as \201s, \204s, etc (when double encoding takes place).

In this case, subjects were 7-bit-safe (encode-word or QP), so they
look right, but

 KG> In the second body, every non-ascii character is preceded by
 KG> \201, as well.

message bodies were 8-bit, so they were "corrupted" by gnus.

 KG> And you see \214 and I presume that \214 is the internal
 KG> representation of koi8-r or iso-8859-* (what's the number for
 KG> Cyrillic?).

the number of what? :-) in unicode (ucs-2) it is 0x04??
in the Emacs' internal representation it is 0xe?? or so...

	Best regards, -- Vladimir.
-- 
Renning's Maxim:
	Man is the highest animal.  Man does the classifying.

Re: bug in displaying inline message/rfc822

[Hrvoje Niksic]

Vladimir Volovich <vvv@vvv.vsu.ru> writes:

> here is it! the message was translated to 7-bit encoding (base64)
> and appeared in your mbox in that form. if the message was not
> changed (i.e. it should have contained _8-bit_ chars, with
> CTT=8-bit), then i guess the bug will show even in XEmacs, because
> it is bug in gnus.

Hmm, yes.  And reading my answer to you, I discovered found another
bug in Gnus, which failed to handle spurious newlines at eob nicely
(base64-decode-region didn't like the extra whitespace.)  Patch
a-coming.

>  >> to base64 by regoc.srce.hr id WAA24757
>  >> 
>  >> Это тест. Проба пера.
> 
>  Hrvoje> Similarities between Russian and Croatian always amaze me.
>  Hrvoje> In Croatian it would mean something like "Here is a test.  A
>  Hrvoje> test of the pen."  :-)
> 
> and how it would be in Croatian? :-)

Eto testa.  Proba pisma.

This is assuming that what I read "pen" actually meant "script".

Re: bug in displaying inline message/rfc822

[Peter von der Ahé]

[-- Attachment #1: Type: text/plain, Size: 263 bytes --]

Hi!

The attached patch will correct the problem in Emacs 20.3.

I'm not sure that this is the right way to do stuff, it will probably
fail on non-Mule Emacsen, but it should give someone with more insight
an idea on howto solve the problem.

Kind Regards
Peter


[-- Attachment #2: correct inline message/rfc822 --]
[-- Type: application/x-patch, Size: 549 bytes --]

[-- Attachment #3: Type: text/plain, Size: 157 bytes --]


-- 
		  · Peter von der Ahé · Systems programmer ·
	      · Office phone: +45 89 42 31 41 · Office: 540.030 ·
		    · <URL:http://www.daimi.au.dk/~pahe/> ·

Re: bug in displaying inline message/rfc822

[Lars Magne Ingebrigtsen]

vvv@vvv.vsu.ru writes:

> this is a test message that demonstrates a bug in gnus. when i receive
> this message (in nnmbox backend, if that matters), gnus tries to show the
> message/rfc822 parts inline, but it fails: i see \214<8-bit-char> instead
> of real characters. perhaps, this is because the CTT of message/rfc822
> parts is 8bit, and gnus encodes the message buffer several times...

[...]

> Content-Type: text/plain; charset=koi8-r
> Content-Transfer-Encoding: 8bit
> 
> Это тест. Проба пера.

As you can see, this displayed OK for me using nnml.  Uhm -- you use
Emacs, not XEmacs, right?  I'd better test with Emacs as well.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bug in displaying inline message/rfc822

[Lars Magne Ingebrigtsen]

pahe@daimi.au.dk (Peter von der Ahé) writes:

> The attached patch will correct the problem in Emacs 20.3.
> 
> I'm not sure that this is the right way to do stuff, it will probably
> fail on non-Mule Emacsen, but it should give someone with more insight
> an idea on howto solve the problem.

Yup.  This pinpointed the problem, and I did something with a similar
effect. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bug in displaying inline message/rfc822

[Vladimir Volovich]

"LMI" == Lars Magne Ingebrigtsen writes:

 >> Это тест. Проба пера.

 LMI> As you can see, this displayed OK for me using nnml.  Uhm -- you
 LMI> use Emacs, not XEmacs, right?  I'd better test with Emacs as
 LMI> well.

my original message was not translated to some 7-bit CTE (e.g., QP or
base64) by MTAs, was it? i.e., does the raw message on disk contain
8-bit chars?

	Best regards, -- Vladimir.

Re: bug in displaying inline message/rfc822

[Kai.Grossjohann]

Vladimir Volovich <vvv@vvv.vsu.ru> writes:

  >  KG> And you see \214 and I presume that \214 is the internal
  >  KG> representation of koi8-r or iso-8859-* (what's the number for
  >  KG> Cyrillic?).
  > 
  > the number of what? :-) in unicode (ucs-2) it is 0x04??
  > in the Emacs' internal representation it is 0xe?? or so...

I meant, which number should go in iso-8859-* instead of the "*"?  Or
is Cyrillic not specified at all in iso-8859?

kai
-- 
Abort this operation?   [Abort]  [Cancel]

Re: bug in displaying inline message/rfc822

[Vladimir Volovich]

"KG" == Kai Grossjohann writes:

 KG> I meant, which number should go in iso-8859-* instead of the "*"?
 KG> Or is Cyrillic not specified at all in iso-8859?

cyrillic is represented in iso-8859-5 (sorry it was late when i wrote
my prev. letter :)

	Best regards, -- Vladimir.

Re: bug in displaying inline message/rfc822

[Vladimir Volovich]

"LMI" == Lars Magne Ingebrigtsen writes:

 >> The attached patch will correct the problem in Emacs 20.3.
 LMI> Yup.  This pinpointed the problem, and I did something with a
 LMI> similar effect.

Thanks Peter and Lars, --- in pgnus 0.83 the bug seems to be fixed.

	Best regards, -- Vladimir.
-- 
Statistics means never having to say you're certain.