Charset encoding brokenness

Charset encoding brokenness

[Jesper Harder]

Charset encoding is currently broken in quite a few places, viz.

· Forwarding.  `C-u N C-c C-f' for N=1,2,4 is broken if the CTE of
  the forwarded message is 8bit.

· Digesting.  `S o p' and `S o m' is similarly broken if
  `message-forward-as-mime' is nil.

· Resending.  `S D e' is also broken.

· Bouncing.  Ditto for `S D b'.

The last two problems were introduced by me :-(

In all cases the bug occurs because the body of the
forwarded/bounced/etc message is inserted without being decoded.

This means that the 8bit characters get the charset
`eight-bit-graphic' which breaks the message in various ways.

It's easiest to observe for utf-8 (with CTE 8bit) -- for Latin-1
you'll probably need to do `C-u C-x =' to see it immediately.

Unfortunately, I can't think of any obvious way to fix it.  Ideas?

Re: Charset encoding brokenness

[Simon Josefsson]

Jesper Harder <harder@ifa.au.dk> writes:

> Charset encoding is currently broken in quite a few places, viz.
>
> · Forwarding.  `C-u N C-c C-f' for N=1,2,4 is broken if the CTE of
>   the forwarded message is 8bit.
>
> · Digesting.  `S o p' and `S o m' is similarly broken if
>   `message-forward-as-mime' is nil.

Did my C-c C-f rewrite cause this?  I suppose not, but would just like
to make sure.

Re: Charset encoding brokenness

[Jesper Harder]

Simon Josefsson <jas@extundo.com> writes:

> Jesper Harder <harder@ifa.au.dk> writes:
>
>> Charset encoding is currently broken in quite a few places, viz.
>>
>> · Forwarding.  `C-u N C-c C-f' for N=1,2,4 is broken if the CTE of
>>   the forwarded message is 8bit.
>>
>> · Digesting.  `S o p' and `S o m' is similarly broken if
>>   `message-forward-as-mime' is nil.
>
> Did my C-c C-f rewrite cause this?

N=1 appears to work in a Gnus from 2003-07-31, so it probably broke
that.  N=2,4 was also broken before the rewrite.

Re: Charset encoding brokenness

[Jesper Harder]

Jesper Harder <harder@ifa.au.dk> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Jesper Harder <harder@ifa.au.dk> writes:
>>
>>> Charset encoding is currently broken in quite a few places, viz.
>>>
>>> · Forwarding.  `C-u N C-c C-f' for N=1,2,4 is broken if the CTE of
>>>   the forwarded message is 8bit.
>>>
>>> · Digesting.  `S o p' and `S o m' is similarly broken if
>>>   `message-forward-as-mime' is nil.
>>
>> Did my C-c C-f rewrite cause this?
>
> N=1 appears to work in a Gnus from 2003-07-31, so it probably broke
> that.

Right.  Before the rewrite N=1 used `mime-to-mml' -- it doesn't now,
which explains why it's broken.

So the question is: how to forward/digest/etc correctly without
converting to mml?

I think `mime-to-mml' is the wrong thing for most purposes for a
variety of reasons:

1. When I ask someone to forward a message, I usually want a
   _verbatim_ copy, not an interpretation by Gnus.

2. `mime-to-mml' assumes that the message is actually valid MIME.

3. It's broken for pgp/mime.

The last problem could probably be fixed by some special-casing.  But
the other two remain.

Re: Charset encoding brokenness

[Vasily Korytov]

On Tue, 28 Oct 2003 04:01:22 +0100, Jesper Harder wrote:

> So the question is: how to forward/digest/etc correctly without
> converting to mml?
>
> I think `mime-to-mml' is the wrong thing for most purposes for a
> variety of reasons:
>
> 1. When I ask someone to forward a message, I usually want a
> _verbatim_ copy, not an interpretation by Gnus.

Hm. Then we really should have different options to have both
interpretation and verbatim plaintext forwards.

In fact, the interpretation is much more often desired. What the use do
I have of base64 undecoded? And if that wasn't me -- but some poor
Outlook or even fido users?

-- 
       I accept RFC3156 and RFC2440-compatible encrypted mail.
PGP key fingerprint: 123A 7CCE 6E26 6233 0D87 E01A A0F8 3524 FCD8 1841

Re: Charset encoding brokenness

[Jesper Harder]

deskpot@despammed.com (Vasily Korytov) writes:

> On Tue, 28 Oct 2003 04:01:22 +0100, Jesper Harder wrote:
>
>> I think `mime-to-mml' is the wrong thing for most purposes for a
>> variety of reasons:
>>
>> 1. When I ask someone to forward a message, I usually want a
>> _verbatim_ copy, not an interpretation by Gnus.
>
> Hm. Then we really should have different options to have both
> interpretation and verbatim plaintext forwards.

We already have that ... the problem is that verbatim forward is
currently broken.

> In fact, the interpretation is much more often desired. What the use do
> I have of base64 undecoded?

If you just want to read the message, then it doesn't matter if it's
base64 encoded -- Gnus decodes it automatically, so you won't even
notice unless you view the raw message.

> And if that wasn't me -- but some poor Outlook or even fido users?

Outlook has no problem with viewing base64.  I don't know about Fido,
but I'd consider any client which doesn't do base64 to be extremely
obsolete.

Re: Charset encoding brokenness

[Vasily Korytov]

On Tue, 28 Oct 2003 17:41:35 +0100, Jesper Harder wrote:

> deskpot@despammed.com (Vasily Korytov) writes:
>
>>> 1. When I ask someone to forward a message, I usually want a
>>> _verbatim_ copy, not an interpretation by Gnus.
>>
>> Hm. Then we really should have different options to have both
>> interpretation and verbatim plaintext forwards.
>
> We already have that ... the problem is that verbatim forward is
> currently broken.

Heh. The ``interpretation'' forward _is_ broken. I've just updated Gnus
from CVS. See lower for details.

>> In fact, the interpretation is much more often desired. What the use do
>> I have of base64 undecoded?
>
> If you just want to read the message, then it doesn't matter if it's
> base64 encoded -- Gnus decodes it automatically, so you won't even
> notice unless you view the raw message.

You didn't get it: I set message-forward-as-mime to nil and press C-c
C-f and get a block like:

-------------------- Start of forwarded message --------------------
Path: main.gmane.org!not-for-mail
Message-ID: <20031019222253.40d11690.bulldog@kaluga.ru>
From: Oleg Sheremetinsky <bulldog@kaluga.ru>
Newsgroups: gmane.emacs.xemacs.user.russian
Subject: Re: Oort Gnus
Date: Sun, 19 Oct 2003 22:22:53 +0400
References: <20031017214524.0687b646.bulldog@kaluga.ru>
	<878ynj66to.fsf@unix.home>
	<20031018204829.630c8e47.bulldog@kaluga.ru>
	<87vfqmbd72.fsf@unix.home>
Lines: 17
Organization: =?KOI8-R?Q?=F4=EF=EF_"=F2=D5=CC=C5=DA"?=
Sender: xemacs-users-ru-admin@xemacs.org
Approved: news@gmane.org
NNTP-Posting-Host: deer.gmane.org
Mime-Version: 1.0
Content-Type: text/plain; charset=KOI8-R
Content-Transfer-Encoding: base64
X-Trace: sea.gmane.org 1066587859 15862 80.91.224.253 (19 Oct 2003 18:24:19 GMT)
X-Complaints-To: usenet@sea.gmane.org
NNTP-Posting-Date: Sun, 19 Oct 2003 18:24:19 +0000 (UTC)
Original-X-From: xemacs-users-ru-admin@xemacs.org Sun Oct 19 20:24:17 2003
Return-path: <xemacs-users-ru-admin@xemacs.org>
Original-Received: from gwyn.tux.org ([199.184.165.135])
	by deer.gmane.org with esmtp (Exim 3.35 #1 (Debian))
	id 1ABIDw-0005Js-00
	for <gexur-xemacs-users-ru@gmane.org>; Sun, 19 Oct 2003 20:24:17 +0200
Original-Received: from gwyn.tux.org (localhost.localdomain [127.0.0.1])
	by gwyn.tux.org (8.11.6p2/8.9.1) with ESMTP id h9JIO2v28821;
	Sun, 19 Oct 2003 14:24:02 -0400
Original-Received: (from turnbull@localhost)
	by gwyn.tux.org (8.11.6p2/8.9.1) id h9JIN4x28617
	for xemacs-users-ru-mailman@xemacs.org; Sun, 19 Oct 2003 14:23:04 -0400
Original-Received: (from mail@localhost)
	by gwyn.tux.org (8.11.6p2/8.9.1) id h9JIN4n28607
	for turnbull@tux.org; Sun, 19 Oct 2003 14:23:04 -0400
Original-Received: from lancia.kaluga.ru (lancia.kaluga.ru [62.148.128.2])
	by gwyn.tux.org (8.11.6p2/8.9.1) with ESMTP id h9JIN3v28600
	for <xemacs-users-ru@xemacs.org>; Sun, 19 Oct 2003 14:23:03 -0400
Original-Received: from localhost.localdomain (14.pool-157.kaluga.ru [62.148.157.14])
	by lancia.kaluga.ru (8.12.10/8.12.10) with ESMTP id h9JIMxcl089459
	for <xemacs-users-ru@xemacs.org>; Sun, 19 Oct 2003 22:23:00 +0400 (MSD)
Original-Received: by localhost.localdomain (Postfix on, from userid 500)
	id AB68F2454A; Sun, 19 Oct 2003 22:22:53 +0400 (MSD)
Original-To: xemacs-users-ru@xemacs.org
In-Reply-To: <87vfqmbd72.fsf@unix.home>
X-Mailer: Sylpheed version 0.9.5claws (GTK+ 1.2.10; i386-asplinux-linux-gnu)
X-XEmacs-List: users-ru
Errors-To: xemacs-users-ru-admin@xemacs.org
X-BeenThere: xemacs-users-ru@xemacs.org
X-Mailman-Version: 2.0.13
Precedence: bulk
List-Help: <mailto:xemacs-users-ru-request@xemacs.org?subject=help>
List-Post: <mailto:xemacs-users-ru@xemacs.org>
List-Subscribe: <http://lists.xemacs.org/lists/listinfo/xemacs-users-ru>,
	<mailto:xemacs-users-ru-request@xemacs.org?subject=subscribe>
List-Id: Russian XEmacs Users <xemacs-users-ru.xemacs.org>
List-Unsubscribe: <http://lists.xemacs.org/lists/listinfo/xemacs-users-ru>,
	<mailto:xemacs-users-ru-request@xemacs.org?subject=unsubscribe>
List-Archive: <http://lists.xemacs.org/pipermail/xemacs-users-ru/>
X-MIME-Autoconverted: from 8bit to base64 by gwyn.tux.org id h9JIO2v28821
X-Report-Spam: http://spam.gmane.org/gmane.emacs.xemacs.user.russian:747
Xref: unix.home gmane.emacs.xemacs.user.russian:457

T24gU2F0LCAxOCBPY3QgMjAwMyAyMjo0MTo1MyArMDQwMA0KZGVza3BvdEBkZXNwYW1tZWQu
Y29tIChWYXNpbHkgS29yeXRvdikgd3JvdGU6DQoNCg0KPiA+PiD1x9UuIO7PINrB3sXNINTF
wsUgyc3Fzs7PIE9vcnQ/IDUuMTAg1dbFINDPzMfPxMEgy8HLINfZ28XMLiDpzMkg1NkNCj4g
Pj4gyM/exdvYIMnNxdTYIMzPx88g0yDU1c3Bzs/NPyDkz9PUwdTP3s7PINfZ1MHdydTYIM/E
yc4gxsHKzCDJ2iBPb3J0Lg0KPiA+IPTV1CDJztTF0sXTzsHRINfF3dggz8LOwdLV1snMwdPY
IC0gZ251cyDJ2iBjdnMnwSDQ0skg0M/T1MnOx8HIDQo+ID4gKMbP0tfB0sTByCkgxM/Xz8zY
zs8gxM/Mx88gz9TQ0sHXzNHF1CDT1MHU2Mkgx8TFIMvPzMnexdPU188g09TSz8sgPiAxMDAN
Cj4gPiAo0NLJzcXSzs8g1MHLKS4gT29ydCBnbnVzINzUycgg1M/Szc/az9cgzsUgyc3FxdQu
IPEg0M/Oyc3BwCwg3tTPINzUzyDOxQ0KPiA+IMLPzMXFLCDexc0i0M/E2sXNztnKINPU1csi
LCDOzyDX06Mg1sUuLi4NCj4gDQo+IOjNLiDxINTP1sUg2sHNxdTJzCDc1M8sIM7PIMfSxdvJ
zCDOwSDT18/KIMTPwsHXzMXOztnKINDSycLMydrJ1MXM2M7PINcNCj4g1M8g1sUg19LFzdEg
yNXLLg0KDQr3IMTP0M/MzsXOycUgyyDQ0sXE2cTV3cXN1SDQydPYzdUgLSBHbnVzIMnaIGN2
cyfBINDPxCBYRW1hY3Mnz80g0M/exc3VLdTPDQrX08Ug09TB1NjJIM/U0NLB18zRxdQgwsXa
INDSz8LMxc0uIPfP1CDUwcvB0SDXz9Qg2sHHz8fVzMnOwSA6LSkNCg0KLS0gDQrzINXXwdbF
zsnFzSwg+8XSxc3F1MnO08vJyiDvzMXHICD308sg78vUIDE5IDIyOjIwOjI1IE1TRCAyMDAz
DQpFLW1haWw6IGJ1bGxkb2c8YXQ+a2FsdWdhLnJ1ICjHLiDrwczVx8EpDQoNCg==


-------------------- End of forwarded message --------------------

>> And if that wasn't me -- but some poor Outlook or even fido users?
>
> Outlook has no problem with viewing base64.  I don't know about Fido,
> but I'd consider any client which doesn't do base64 to be extremely
> obsolete.

I know no MUA, that handle such constructions properly. Even Gnus.

-- 
       I accept RFC3156 and RFC2440-compatible encrypted mail.
PGP key fingerprint: 123A 7CCE 6E26 6233 0D87 E01A A0F8 3524 FCD8 1841

Re: Charset encoding brokenness

[Vasily Korytov]

On Wed, 29 Oct 2003 10:03:00 +0300, Vasily Korytov wrote:

>> Outlook has no problem with viewing base64.  I don't know about Fido,
>> but I'd consider any client which doesn't do base64 to be extremely
>> obsolete.
>
> I know no MUA, that handle such constructions properly. Even Gnus.

What a shame on my head. =))

But I still consider this behaviour somewhat broken.

-- 
       I accept RFC3156 and RFC2440-compatible encrypted mail.
PGP key fingerprint: 123A 7CCE 6E26 6233 0D87 E01A A0F8 3524 FCD8 1841

Re: Charset encoding brokenness

[Simon Josefsson]

deskpot@despammed.com (Vasily Korytov) writes:

> You didn't get it: I set message-forward-as-mime to nil and press C-c
> C-f and get a block like:

What you want appear to correspond to my 'Forward-What-I-See, ASCII
purist' mode in my other message.  I think it is the second most
important forward mode to support (pure MIME mode being the first).

>>> And if that wasn't me -- but some poor Outlook or even fido users?
>>
>> Outlook has no problem with viewing base64.  I don't know about Fido,
>> but I'd consider any client which doesn't do base64 to be extremely
>> obsolete.
>
> I know no MUA, that handle such constructions properly. Even Gnus.

Gnus does, but I don't think it is relevant whether any MUA support
displaying a forward like that -- when you forward a RFC 822 message
verbatim, it is for debugging, and it should be possible to send
forwards like that.  It shouldn't be the default though.

Re: Charset encoding brokenness

[Simon Josefsson]

Jesper Harder <harder@ifa.au.dk> writes:

> So the question is: how to forward/digest/etc correctly without
> converting to mml?
>
> I think `mime-to-mml' is the wrong thing for most purposes for a
> variety of reasons:
>
> 1. When I ask someone to forward a message, I usually want a
>    _verbatim_ copy, not an interpretation by Gnus.
>
> 2. `mime-to-mml' assumes that the message is actually valid MIME.
>
> 3. It's broken for pgp/mime.
>
> The last problem could probably be fixed by some special-casing.  But
> the other two remain.

Yes; mime-to-mml shouldn't be used unless MML will be shown in the
forwarded message buffer.  The C-c C-f docstring is quite
disappointing, so I experimented some, and I came up with more
detailed descriptions of the modes:

if ARG is nil,

  If m-f-a-m=t and m-f-s-m=best, this selects between ARG=2 and ARG=3
  below depending on if MML is safe.  I haven't thought through what
  it should do for the remaining 2*3-1=5 cases.

if ARG is 1,

  "Gnus specific ASCII purist".  This wrap a MML decoded message in a
  ASCII header/footer.  I can't begin to understand why anyone would
  want this.  For any non-trivial MIME message, I doubt any MUA but
  Gnus would render this in any understandable way.

if ARG is 2,

  "MIME purist".  Wrap raw body in MML application/rfc822 part.  No
  MML decoding within body.

if ARG is 3,

  "MML purist". Wrap MML-decoded body in MML application/rfc822 part.

if ARG is 4,

  "ASCII purist in debug mode". Wrap gnus-original-article-buffer
  within ASCII forward headers.

if ARG is 5 (not currently supported)

  "Forward-What-I-See, ASCII purist".  This should use whatever is in
  the Gnus *Article* buffer.  It should NOT try to decode the message
  in gnus-original-article-buffer again.  No MML tags whatsoever, the
  forward header should be pure ASCII.

The problem in implementing 5 appear to be getting the *Article*
buffer down into message.el, since currently it only has access to
gnus-original-article-buffer which isn't sufficient.

We might consider swapping ARG=1 and ARG=5, since the ARG=1 above is
quite weird, but this wouldn't be backwards compatible so maybe we
shouldn't.

Of course, then there is the problem of \345 characters instead of
decoded data.  Will think about that once I understand more.

Re: Charset encoding brokenness

[Simon Josefsson]

Jesper Harder <harder@ifa.au.dk> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Jesper Harder <harder@ifa.au.dk> writes:
>>
>>> Charset encoding is currently broken in quite a few places, viz.
>>>
>>> · Forwarding.  `C-u N C-c C-f' for N=1,2,4 is broken if the CTE of
>>>   the forwarded message is 8bit.
>>>
>>> · Digesting.  `S o p' and `S o m' is similarly broken if
>>>   `message-forward-as-mime' is nil.
>>
>> Did my C-c C-f rewrite cause this?
>
> N=1 appears to work in a Gnus from 2003-07-31, so it probably broke
> that.  N=2,4 was also broken before the rewrite.

I fixed N=1 which was a cut'n'paste bug in the 2003-07-31 commit.  I
think I tested all modes, and they should now behave the same as
before.  (Still not correct though, but that is a separate problem..)

Re: Charset encoding brokenness

[Simon Josefsson]

Jesper Harder <harder@ifa.au.dk> writes:

> Charset encoding is currently broken in quite a few places, viz.
>
> · Forwarding.  `C-u N C-c C-f' for N=1,2,4 is broken if the CTE of
>   the forwarded message is 8bit.

I believe I fixed N=1 before, but cannot find any problem with N=4?  I
can reproduce the problem with N=2 though, and it could be worked
around by using

<#part type=message/rfc822 disposition=inline raw=t buffer=" *Original Article*">
<#/part>

although one has to be careful so the buffer isn't modified between
when you press C-c C-f and when you post the message.  So it should
create a new buffer and copy gnus-original-article-buffer there,
instead of referncing g-o-a-b directly.  Perhaps that buffer even
should be marked read-only, because if you add a some non-ASCII text
you will likely break the message, since it is not MIME encoded, which
novices might not realize (and experts know how to disable the
read-only flag).  What do you think?

I tested using <ilur80tmlcq.fsf@latte.josefsson.org> in gnus.test.

Re: Charset encoding brokenness

[Jesper Harder]

Simon Josefsson <jas@extundo.com> writes:

> Jesper Harder <harder@ifa.au.dk> writes:
>
>> Charset encoding is currently broken in quite a few places, viz.
>>
>> · Forwarding.  `C-u N C-c C-f' for N=1,2,4 is broken if the CTE of
>>   the forwarded message is 8bit.
>
> I believe I fixed N=1 before,

Yup.

> but cannot find any problem with N=4?

He, you had me really confused for a moment -- I was sure that N=4
didn't work.

You're right that N=4 "works" now, but that's because there's a bug in
the logic of `message-forward-make-body', i.e. that N=1 and N=4 do
exactly the same thing.

In both cases `message-forward-make-body-plain' is called, which is
wrong for N=4 where the message isn't supposed to be converted to MML.

> <#part type=message/rfc822 disposition=inline raw=t buffer=" *Original Article*">
> <#/part>
>
> although one has to be careful so the buffer isn't modified between
> when you press C-c C-f and when you post the message.  So it should
> create a new buffer and copy gnus-original-article-buffer there,
> instead of referncing g-o-a-b directly.  Perhaps that buffer even
> should be marked read-only, because if you add a some non-ASCII text
> you will likely break the message, since it is not MIME encoded, which
> novices might not realize (and experts know how to disable the
> read-only flag).  What do you think?

I think it sounds like a good plan.