Another problem with rfc2047 encoding

Another problem with rfc2047 encoding

[Reiner Steib]

Hi,

when posting a subject like...

| ¿Qué significa =?ISO-8859-1?Q?=F6?= en el Asunto?

... or...

| Ich hab' da mal 'ne Fräge: Was bedeuten denn diese Zeichen =?ISO-8859-1?Q?=F6?= ?

... Oort Gnus (CVS) doesn't treat "=?ISO-8859-1?Q?=F6?=" literal as it
should.  "=?" or/and "?=" would have to be encoded, probably.

,----[ (1) <v9k7deg3aj.fsf@marauder.physik.uni-ulm.de> / Oort ]
| Subject: =?iso-8859-1?q?=BFQu=E9?= significa =?ISO-8859-1?Q?=F6?= en el
|  Asunto?
`----

I.e. this will be decoded to "¿Qué significa ö en el Asunto?", which
is wrong.

Gnus 5.9 (from Emacs 21.3) does it correctly:

,----[ (2) <v9k7dea87n.fsf@theotp2.physik.uni-ulm.de> / Gnus 5.9 ]
| Subject: =?iso-8859-1?q?=BFQu=E9_significa_=3D=3FISO-8859-1=3FQ=3F=3DF6?=
|  =?iso-8859-1?q?=3F=3D_en_el_Asunto=3F__(Gnus_5.9)?=
`----

I didn't follow the discussion about "rfc2047 still bogus" on
bugs@gnus.org closely, but this might be related:

,----[ `rfc2047.el' from Oort ]
| ELISP> (load ".../share/emacs/site-lisp/gnus-Oort/rfc2047.elc")
| t
| ELISP> (with-temp-buffer
|   (insert "Subject: ¿Qué significa =?ISO-8859-1?Q?=F6?= en el Asunto?\n")
|   (rfc2047-encode-message-header)
|   (buffer-string))
| "Subject: =?iso-8859-1?q?=BFQu=E9?= significa =?ISO-8859-1?Q?=F6?= en el\n Asunto?\n"
`----

,----[ `rfc2047.el' from Emacs 21.3 ]
| ELISP> (load ".../share/emacs/21.3/lisp/gnus/rfc2047.elc")
| t
| ELISP> (with-temp-buffer
|   (insert "Subject: ¿Qué significa =?ISO-8859-1?Q?=F6?= en el Asunto?\n")
|   (rfc2047-encode-message-header)
|   (buffer-string))
| "Subject: =?iso-8859-1?q?=BFQu=E9_significa_=3D=3FISO-8859-1=3FQ=3F=3DF?=\n =?iso-8859-1?q?6=3F=3D_en_el_Asunto=3F?=\n"
`----

Bye, Reiner.

(1) nntp://news.gmane.org/gmane.test/620
(2) nntp://news.gmane.org/gmane.test/624
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Another problem with rfc2047 encoding

[Reiner Steib]

On Mon, Apr 28 2003, Reiner Steib wrote:

> "=?" or/and "?=" would have to be encoded, probably.

Yes:

,----[ rfc2047; last paragraph in section 6 ]
| Use of 'encoded-word's to represent strings of purely ASCII characters
| is allowed, but discouraged.  In rare cases it may be necessary to
| encode ordinary text that looks like an 'encoded-word'.
`----

See (sorry, in German) the discussion in de.comm.software.newsreader,
starting with <news:8kb90y1ACHB@michael.heydekamp.dialin.t-online.de>.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: Another problem with rfc2047 encoding

[Lars Magne Ingebrigtsen]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:

> ... Oort Gnus (CVS) doesn't treat "=?ISO-8859-1?Q?=F6?=" literal as it
> should.  "=?" or/and "?=" would have to be encoded, probably.

Yes, they should.  But I think we'll leave fixing this bug until No
Gnus. 

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Another problem with rfc2047 encoding

[Thomas Hühn]

Also sprach Reiner Steib <4.uce.03.r.s@nurfuerspam.de>:

> ... Oort Gnus (CVS) doesn't treat "=?ISO-8859-1?Q?=F6?=" literal as it
> should.  "=?" or/and "?=" would have to be encoded, probably.

And perhaps related:

In the following article...

From: Ralf =?ISO-8859-1?Q?D=F6blitz?= <=?iso-8859-1?q?d=F6blitz?=@doeblitz.net>
Newsgroups: de.admin.news.misc
Subject: Re: Reflektoren und RFC822
Date: 16 Apr 2003 19:01:20 GMT
Message-ID: <b7k9a0$he6$3@msgids.netzverwaltung.net>

...the localpart seems to be interpreted, though it shouldn't.

RFC 2047 states in section 5(3):
"An 'encoded-word' MUST NOT appear in any portion of an 'addr-spec'."

Thomas

Re: Another problem with rfc2047 encoding

[Lars Magne Ingebrigtsen]

ding@daphne.gnuu.de (Thomas Hühn) writes:

> From: Ralf =?ISO-8859-1?Q?D=F6blitz?= <=?iso-8859-1?q?d=F6blitz?=@doeblitz.net>

[...]

> Message-ID: <b7k9a0$he6$3@msgids.netzverwaltung.net>
>
> ...the localpart seems to be interpreted, though it shouldn't.

But that's not a Gnus message, is it?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Another problem with rfc2047 encoding

[Thomas Hühn]

Also sprach Lars Magne Ingebrigtsen <larsi@gnus.org>:

>> From: Ralf =?ISO-8859-1?Q?D=F6blitz?= <=?iso-8859-1?q?d=F6blitz?=@doeblitz.net>
>
> [...]
>
>> Message-ID: <b7k9a0$he6$3@msgids.netzverwaltung.net>
>>
>> ...the localpart seems to be interpreted, though it shouldn't.
>
> But that's not a Gnus message, is it?

Yes, what I mean is: If I want to reply, Gnus tries to send a mail to
<döblitz@doeblitz.net>.

OK, you could see it as the user's problem, but a warning would be nice.
And perhaps just don't "decode" the localpart. That way the user has a
chance that the "undecoded" localpart exists at the receiving mail
server. At least he sees that there is something strange going on. If
you're used to umlaute you don't catch them at first glance in the mail
address.

BTW, Ralf seems to interpret the RfC that way, that mail should be sent
to <=?iso-8859-1?q?d=F6blitz?=@doeblitz.net>, literally. I'm no expert
in terms of the RfC, so I'm not sure, whether that "must not" really
means, that no such string of characters may appear in the localpart, or
whether it means, that this string must not be interpreted as an encoded
word, thus being legal.

Thomas

Re: Another problem with rfc2047 encoding

[Lars Magne Ingebrigtsen]

thomas_mail@daphne.gnuu.de (Thomas Hühn) writes:

> Yes, what I mean is: If I want to reply, Gnus tries to send a mail to
> <döblitz@doeblitz.net>.

Yes, that's a bug.  It should be fixed after Gnus 5.10 has been
released.  Say -- 5.10.2.

Dave Love has done work on the version of rfc2047.el in Emacs
21.3.50.  It should be examined whether that version can just be
dropped into Gnus 5.10.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Another problem with rfc2047 encoding

[Christoph Garbers]

* thomas_mail@daphne.gnuu.de (Thomas Hühn):

> BTW, Ralf seems to interpret the RfC that way, that mail should be sent
> to <=?iso-8859-1?q?d=F6blitz?=@doeblitz.net>, literally. I'm no expert
> in terms of the RfC, so I'm not sure, whether that "must not" really
> means, that no such string of characters may appear in the localpart, or
> whether it means, that this string must not be interpreted as an encoded
> word, thus being legal.

The localpart should be taken as it is. 
<=?iso-8859-1?q?d=F6blitz?=@doeblitz.net> would be the right address to
send mail to. And I would be happy if Gnus would handle this that way in
the future.

Christoph

Re: Another problem with rfc2047 encoding

[Dave Love]

Christoph Garbers <cgarbers@gmx.de> writes:

> The localpart should be taken as it is. 
> <=?iso-8859-1?q?d=F6blitz?=@doeblitz.net> would be the right address to
> send mail to. And I would be happy if Gnus would handle this that way in
> the future.

The version in CVS does so, if I understand the requirement correctly.

Re: Another problem with rfc2047 encoding

[Christoph Garbers]

* Dave Love <d.love@dl.ac.uk>:

>> The localpart should be taken as it is. 
>> <=?iso-8859-1?q?d=F6blitz?=@doeblitz.net> would be the right address to
>> send mail to. And I would be happy if Gnus would handle this that way in
>> the future.
> 
> The version in CVS does so, if I understand the requirement correctly.

It does not work for me here. Gnus still decodes the localpart.

Christoph

Re: Another problem with rfc2047 encoding

[Dave Love]

Christoph Garbers <cgarbers@gmx.de> writes:

> It does not work for me here. Gnus still decodes the localpart.

Could you send me an example of exactly what it's doing wrong, then,
please.  I probably misunderstood the thread.

Re: Another problem with rfc2047 encoding

[Christoph Garbers]

* Dave Love <d.love@dl.ac.uk>:

>> It does not work for me here. Gnus still decodes the localpart.
> 
> Could you send me an example of exactly what it's doing wrong, then,
> please.  I probably misunderstood the thread.

Have a look at <b7k9a0$he6$3@msgids.netzverwaltung.net> (it is in the
german newsgroup de.admin.news.misc about four weeks ago).

Christoph

Re: Another problem with rfc2047 encoding

[Dave Love]

Christoph Garbers <cgarbers@gmx.de> writes:

> Have a look at <b7k9a0$he6$3@msgids.netzverwaltung.net> (it is in the
> german newsgroup de.admin.news.misc about four weeks ago).

Sorry, I can't.

Re: Another problem with rfc2047 encoding

[Bjørn Mork]

ding@daphne.gnuu.de (Thomas Hühn) writes:

> In the following article...
>
> From: Ralf =?ISO-8859-1?Q?D=F6blitz?= <=?iso-8859-1?q?d=F6blitz?=@doeblitz.net>
> Newsgroups: de.admin.news.misc
> Subject: Re: Reflektoren und RFC822
> Date: 16 Apr 2003 19:01:20 GMT
> Message-ID: <b7k9a0$he6$3@msgids.netzverwaltung.net>
>
> ...the localpart seems to be interpreted, though it shouldn't.
>
> RFC 2047 states in section 5(3):
> "An 'encoded-word' MUST NOT appear in any portion of an 'addr-spec'."

True, but I wonder... How should Gnus handle an illegal address like
that?  Quote it, warn the user and try to send anyway if the user
insists?  Or just refuse it unconitionally to avoid having people
believe they can get away with such addresses?

But there are cases where Gnus will happily violate RFC2047 even when
fed valid input.  For example, if I reply to myself (not that I really
*do* that :-), Gnus will create an In-Reply-To field like this:

In-Reply-To: <hvsmrzf65l.fsf@rasputin.ws.nextra.no>
 =?iso-8859-1?q?(Bj=F8rn?= Mork's message of "Thu, 01 May 2003 10:00:22
 +0200")

The In-Reply-To field is a structured field, and the text added after
the message-id is a comment.  Which is perfectly OK, given that CFWS
is allowed there.  But when encoding it, Gnus screws up and the result
is a violation of RFC2047 and RFC2822.  The comment delimiters should
not be part of the encoded word.

One strange thing to notice is that Gnus will correctly encode a
comment in the From field (and possibly other structured fields?)  I
don't know why.


Bjørn
-- 
It takes a homosexual sexist like you to say that you're religious.  

Re: Another problem with rfc2047 encoding

[Lars Magne Ingebrigtsen]

Bjørn Mork <bmork@dod.no> writes:

> In-Reply-To: <hvsmrzf65l.fsf@rasputin.ws.nextra.no>
>  =?iso-8859-1?q?(Bj=F8rn?= Mork's message of "Thu, 01 May 2003 10:00:22
>  +0200")
>
> The In-Reply-To field is a structured field, and the text added after
> the message-id is a comment.  Which is perfectly OK, given that CFWS
> is allowed there.  But when encoding it, Gnus screws up and the result
> is a violation of RFC2047 and RFC2822.  The comment delimiters should
> not be part of the encoded word.
>
> One strange thing to notice is that Gnus will correctly encode a
> comment in the From field (and possibly other structured fields?)  I
> don't know why.

When rfc2047.el was written, In-Reply-To was not a structured field.
:-)  I think that happened in RFC2822.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Another problem with rfc2047 encoding

[Simon Josefsson]

Bjørn Mork <bmork@dod.no> writes:

> ding@daphne.gnuu.de (Thomas Hühn) writes:
>
>> In the following article...
>>
>> From: Ralf =?ISO-8859-1?Q?D=F6blitz?= <=?iso-8859-1?q?d=F6blitz?=@doeblitz.net>
>> Newsgroups: de.admin.news.misc
>> Subject: Re: Reflektoren und RFC822
>> Date: 16 Apr 2003 19:01:20 GMT
>> Message-ID: <b7k9a0$he6$3@msgids.netzverwaltung.net>
>>
>> ...the localpart seems to be interpreted, though it shouldn't.
>>
>> RFC 2047 states in section 5(3):
>> "An 'encoded-word' MUST NOT appear in any portion of an 'addr-spec'."
>
> True, but I wonder... How should Gnus handle an illegal address like
> that?  Quote it, warn the user and try to send anyway if the user
> insists?  Or just refuse it unconitionally to avoid having people
> believe they can get away with such addresses?

How does Gnus know it is an illegal address?  Perhaps someone has an
local-part that is =?iso-8859-1?q?d=F6blitz?=, I believe it is a legal
RFC 822 local-part.

IMHO Gnus should not alter local-parts when replying to a message.

> But there are cases where Gnus will happily violate RFC2047 even when
> fed valid input.  For example, if I reply to myself (not that I really
> *do* that :-), Gnus will create an In-Reply-To field like this:
>
> In-Reply-To: <hvsmrzf65l.fsf@rasputin.ws.nextra.no>
>  =?iso-8859-1?q?(Bj=F8rn?= Mork's message of "Thu, 01 May 2003 10:00:22
>  +0200")
>
> The In-Reply-To field is a structured field, and the text added after
> the message-id is a comment.  Which is perfectly OK, given that CFWS
> is allowed there.  But when encoding it, Gnus screws up and the result
> is a violation of RFC2047 and RFC2822.  The comment delimiters should
> not be part of the encoded word.
>
> One strange thing to notice is that Gnus will correctly encode a
> comment in the From field (and possibly other structured fields?)  I
> don't know why.

Apparently the rfc2047.el encoder is buggy, Dave Love reported this
and he had a fix for it too but it doesn't apply any more.

Re: Another problem with rfc2047 encoding

[Bjørn Mork]

Simon Josefsson <jas@extundo.com> writes:
> Bjørn Mork <bmork@dod.no> writes:
>
>> True, but I wonder... How should Gnus handle an illegal address like
>> that?  Quote it, warn the user and try to send anyway if the user
>> insists?  Or just refuse it unconitionally to avoid having people
>> believe they can get away with such addresses?
>
> How does Gnus know it is an illegal address?  Perhaps someone has an
> local-part that is =?iso-8859-1?q?d=F6blitz?=, I believe it is a legal
> RFC 822 local-part.

Ah, sorry.  I didn't check the article well enough (pressing RET C-u g
on the message-id caused "No such article", although RET displayed the
article just fine).  I assumed Thomas were referring to the problem
where Gnus encodes the localpart.  Try replying to 
<hvbrymhkv7.fsf@rasputin.ws.nextra.no> and you'll see...

Gnus will also normally encode the From field in the same way on
outgoing messages.

> IMHO Gnus should not alter local-parts when replying to a message.

Agreed, the wording in RFC2047 only makes sense for the encoding done
by mail clients.  If someone creates a localpart that looks like it's
RFC2047-encoded then that should be used unaltered.  In fact, I
believe Gnus should never alter any local-part at all.  It does now.

And isn't it wrong to decode the local-part when displaying the
message Thomas referred to?


Bjørn
-- 
Don't you realise that trees are crass?  

Re: Another problem with rfc2047 encoding

[Lars Magne Ingebrigtsen]

Bjørn Mork <bmork@dod.no> writes:

> And isn't it wrong to decode the local-part when displaying the
> message Thomas referred to?

Yes, it is.  On the other hand, making the decoder much slower for
pretty marginal gain isn't really a good idea, either.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen