Reply misinterprets Commas in encoded-words (Gnus violates RFC 2047)

Reply misinterprets Commas in encoded-words (Gnus violates RFC 2047)

[Reiner Steib]

Hi,

back in September 2004, Stefan Wiens reported the following bug in
gnus.gnus-bug but there was no response or fix yet:

,----
| From: =?iso-8859-1?q?Wiens=2C_Stef=AAn?= <s.wi@gmx.net>
| Subject: Reply misinterprets Commas in encoded-words
| Newsgroups: gnus.gnus-bug
| Date: Sat, 18 Sep 2004 03:34:06 +0200
| Message-ID: <87pt4kbbsh.fsf@xenon.eswe.dyndns.org>
| 
| To reproduce, please try to "reply" or "wide reply" to this message
| and have a look at the "To" and "Cc" headers in your Message buffer.
`----

The problem is that Gnus decodes »=?iso-8859-1?q?Wiens=2C_Stef=AAn?=«
to »Wiens, Stefªn <s.wi@gmx.net>« in the article buffer and when doing
a (wide) reply, this leads to:

| To: Wiens
| Cc: Stefªn <s.wi@gmx.net>,  bugs@gnus.org

Note that this problem is not artifical (as the »ª« might suggest):
Many people in Germany or other countries have non-ascii in their
names.  If the mail software of the sender writes the full name in the
From/Cc header in the form »Lastname, Fírstname« the user cannot reply
correctly without manual correction.

In the German Gnus newsgroup this has been discussed several times:
(a) http://www.google.de/groups?as_umsgid=uoek8x11f.fsf@M.web.de&hl=en
(b) http://www.google.de/groups?as_umsgid=87brbdhc4d.fsf@ABG3595C.abg.fsc.net&hl=en

According to Stefan Wiens and Dirk Nimmich (tin developer), Gnus'
behavior violates RFC 2047 (see thread (a)).  Could somebody look at
this problem, please?  Stefan, maybe you could summarize this and the
other problems with Gnus RFC-2047 en/decoder again if necessary.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Reply misinterprets Commas in encoded-words (Gnus violates RFC 2047)

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 2226 bytes --]

>>>>> In <v9sm4n6ls5.fsf@marauder.physik.uni-ulm.de> Reiner Steib wrote:

> Hi,

> back in September 2004, Stefan Wiens reported the following bug in
> gnus.gnus-bug but there was no response or fix yet:

> ,----
>| From: =?iso-8859-1?q?Wiens=2C_Stef=AAn?= <s.wi@gmx.net>
>| Subject: Reply misinterprets Commas in encoded-words
>| Newsgroups: gnus.gnus-bug
>| Date: Sat, 18 Sep 2004 03:34:06 +0200
>| Message-ID: <87pt4kbbsh.fsf@xenon.eswe.dyndns.org>
>| 
>| To reproduce, please try to "reply" or "wide reply" to this message
>| and have a look at the "To" and "Cc" headers in your Message buffer.
> `----

> The problem is that Gnus decodes »=?iso-8859-1?q?Wiens=2C_Stef=AAn?=«
> to »Wiens, Stefªn <s.wi@gmx.net>« in the article buffer and when doing
> a (wide) reply, this leads to:

>| To: Wiens
>| Cc: Stefªn <s.wi@gmx.net>,  bugs@gnus.org

> Note that this problem is not artifical (as the »ª« might suggest):
> Many people in Germany or other countries have non-ascii in their
> names.  If the mail software of the sender writes the full name in the
> From/Cc header in the form »Lastname, Fírstname« the user cannot reply
> correctly without manual correction.

> In the German Gnus newsgroup this has been discussed several times:
> (a) http://www.google.de/groups?as_umsgid=uoek8x11f.fsf@M.web.de&hl=en
> (b) http://www.google.de/groups?as_umsgid=87brbdhc4d.fsf@ABG3595C.abg.fsc.net&hl=en

Though I cannot read German articles, I may be able somewhat to
help.

> According to Stefan Wiens and Dirk Nimmich (tin developer), Gnus'
> behavior violates RFC 2047 (see thread (a)).  Could somebody look at
> this problem, please?  Stefan, maybe you could summarize this and the
> other problems with Gnus RFC-2047 en/decoder again if necessary.

> Bye, Reiner.

Well, it seems to be difficult to improve the
message-get-reply-headers function, etc. so that it may parse
correctly any words including commas.  But it is easy to make
Gnus' rfc2047 encoder include the quotes in the encoded words
when the quoted text contains non-ASCII characters and commas.
An example of the encoded word is in the Cc header of this
message.  The patch for the Gnus CVS trunk is here:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 2006 bytes --]

--- rfc2047.el~	2004-11-23 21:49:55 +0000
+++ rfc2047.el	2005-02-14 07:13:18 +0000
@@ -35,6 +35,7 @@
 
 (require 'qp)
 (require 'mm-util)
+(require 'ietf-drums)
 ;; Fixme: Avoid this (used for mail-parse-charset) mm dependence on gnus.
 (require 'mail-prsvr)
 (require 'base64)
@@ -337,13 +338,23 @@
 		  ;; Does it need encoding?
 		  (goto-char start)
 		  (if (re-search-forward encodable-regexp end 'move)
-		      ;; It needs encoding.  Strip the quotes first,
-		      ;; since encoded words can't occur in quotes.
-		      (progn
-			(goto-char end)
-			(delete-backward-char 1)
-			(goto-char start)
-			(delete-char 1)
+		      ;; It needs encoding.
+		      (let ((tspecials (progn
+					 (goto-char (1+ start))
+					 (re-search-forward
+					  (concat "[" ietf-drums-tspecials "]")
+					  (1- end) t))))
+			(if tspecials
+			    ;; Don't strip the quotes which should be
+			    ;; included in the encoded word, since there
+			    ;; are special characters.
+			    (goto-char start)
+			  ;; Strip the quotes first, since encoded words
+			  ;; can't occur in quotes.
+			  (goto-char end)
+			  (delete-backward-char 1)
+			  (goto-char start)
+			  (delete-char 1))
 			(when last-encoded
 			  ;; There was a preceding quoted word.  We need
 			  ;; to include any separating whitespace in this
@@ -354,7 +365,9 @@
 			  (setq start (point)
 				end (1+ end)))
 			;; Adjust the end position for the deleted quotes.
-			(rfc2047-encode start (- end 2))
+			(rfc2047-encode start (if tspecials
+						  end
+						(- end 2)))
 			(setq last-encoded t)) ; record that it was encoded
 		    (setq last-encoded  nil)))
 		 ((eq ?. csyntax)
@@ -766,8 +779,7 @@
   (let* ((rfc2047-encoding-type 'mime)
 	 (rfc2047-encode-max-chars nil)
 	 (string (rfc2047-encode-string value)))
-    (if (string-match "[][()<>@,;:\\\"/?=]" ;; tspecials
-		      string)
+    (if (string-match (concat "[" ietf-drums-tspecials "]") string)
 	(format "%s=%S" param string)
       (concat param "=" string))))
 

Re: Reply misinterprets Commas in encoded-words (Gnus violates RFC 2047)

[Katsumi Yamaoka]

>>>>> In <b9ywttbwr0l.fsf@jpl.org> Katsumi Yamaoka wrote:

> Well, it seems to be difficult to improve the
> message-get-reply-headers function, etc. so that it may parse
> correctly any words including commas.  But it is easy to make
> Gnus' rfc2047 encoder include the quotes in the encoded words
> when the quoted text contains non-ASCII characters and commas.
> An example of the encoded word is in the Cc header of this
> message.  The patch for the Gnus CVS trunk is here:

> --- rfc2047.el~	2004-11-23 21:49:55 +0000
> +++ rfc2047.el	2005-02-14 07:13:18 +0000

That might have been insufficient to explain how it is effective,
sorry.  Gnus parses the decoded header contents when making the
reply header.  If the header is as follows at that time,

From: Wiens, Stefªn <s.wi@example.com>

I wrote that it seems to be difficult to parse.  I also wrote
that it's easy to solve if what does such an encoding is only
Gnus.  The patch makes Gnus encode the quotes together with the
full name so as to be decoded as follows:

From: "Wiens, Stefªn" <s.wi@example.com>

It seems that any software will be able to parse it correctly.
However, if it is general that there are other softwares which
do the encoding like Gnus, the patch might be useless.

Re: Reply misinterprets Commas in encoded-words (Gnus violates RFC

[Reiner Steib]

On Thu, Feb 17 2005, Katsumi Yamaoka wrote:

>>>>>> In <b9ywttbwr0l.fsf@jpl.org> Katsumi Yamaoka wrote:
>
>> Well, it seems to be difficult to improve the
>> message-get-reply-headers function, etc. so that it may parse
>> correctly any words including commas.  But it is easy to make
>> Gnus' rfc2047 encoder include the quotes in the encoded words
>> when the quoted text contains non-ASCII characters and commas.

I'm afraid that this doesn't help much in the situation in question:
The Gnus user *receives* a message with a From (or Cc) header like...

| From: =?iso-8859-1?q?Wiens=2C_Stef=AAn?= <s.wi@gmx.net>

... and Gnus fails to build the correct reply headers.  The Gnus user
has no influence on how the From (or Cc) header looks like.

> Gnus parses the decoded header contents when making the reply
> header.  If the header is as follows at that time,
>
> From: Wiens, Stefªn <s.wi@example.com>
>
> I wrote that it seems to be difficult to parse.

Would it be possible to build the (Reply) headers from the original
(undecoded) article?  My impression is that this would be necessary in
order to fix the problem.

> I also wrote that it's easy to solve if what does such an encoding
> is only Gnus.  The patch makes Gnus encode the quotes together with
> the full name so as to be decoded as follows:
>
> From: "Wiens, Stefªn" <s.wi@example.com>
>
> It seems that any software will be able to parse it correctly.
> However, if it is general that there are other softwares which
> do the encoding like Gnus, the patch might be useless.

There are some problems with this patch.  After applying your patch, I
see the *encoded* From in the Summary and Article buffer:

| From: =?iso-8859-1?q?Wiens=2C_Stef=AAn?= <s.wi@gmx.net>

Also in the To header and the attribution line in the reply I get the
encoded form.  Surprisingly, the From in the reply of the sent message
is encoded:

  I've put another test message in gmane.test[1] and after `R' (using
  No Gnus including your patch), I received:

  | To: =?iso-8859-1?q?Steib=2C_Re=EDner?= <reinersteib+gmane@imap.cc>

Thanks for looking into this problem.

Bye, Reiner.

[1] <news:v9k6p7jp8p.fsf@marauder.physik.uni-ulm.de>
    http://article.gmane.org/gmane.test/2087
    BTW, Gmane also problems with such From lines.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Reply misinterprets Commas in encoded-words (Gnus violates RFC

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 1586 bytes --]

>>>>> In <v97jl7th3s.fsf@marauder.physik.uni-ulm.de> Reiner Steib wrote:

> On Thu, Feb 17 2005, Katsumi Yamaoka wrote:

>> From: Wiens, Stefªn <s.wi@example.com>
>>
>> I wrote that it seems to be difficult to parse.

> Would it be possible to build the (Reply) headers from the original
> (undecoded) article?

That is what I just wrote that it seems to be difficult. ;-)

> My impression is that this would be necessary in order to fix the
> problem.

I agree.  Today I found a better way to fix the problem without
modifying the rfc2047 encoder.  It doesn't require to modify the
way to build the reply headers either.  The solution is to quote
decoded words in the " *gnus article copy*" buffer (in which the
reply headers are extracted) if there are special characters.
The new patch for the trunk is below.

> There are some problems with this patch.  After applying your patch, I
> see the *encoded* From in the Summary and Article buffer:

>| From: =?iso-8859-1?q?Wiens=2C_Stef=AAn?= <s.wi@gmx.net>

Eh?, that's strange.  I've modified neither the decoder nor the
core of the encoder, hmm..., but now, let's junk it anyway.  I'm
sorry to have waste your time.

	* gnus-msg.el (gnus-copy-article-buffer): Quote decoded words
	containing special characters.

	* rfc2047.el (rfc2047-encode-parameter): Use ietf-drums-tspecials.
	(rfc2047-quote-decoded-words-containing-tspecials): New variable.
	(rfc2047-decode-region): Quote decoded words containing special
	characters when rfc2047-quote-decoded-words-containing-tspecials
	is non-nil.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 3424 bytes --]

--- gnus-msg.el~	2005-02-13 21:48:03 +0000
+++ gnus-msg.el	2005-02-18 07:23:24 +0000
@@ -878,7 +878,8 @@
 	    ;; Decode charsets.
 	    (let ((gnus-article-decode-hook
 		   (delq 'article-decode-charset
-			 (copy-sequence gnus-article-decode-hook))))
+			 (copy-sequence gnus-article-decode-hook)))
+		  (rfc2047-quote-decoded-words-containing-tspecials t))
 	      (run-hooks 'gnus-article-decode-hook)))))
       gnus-article-copy)))
 
--- rfc2047.el~	2004-11-23 21:49:55 +0000
+++ rfc2047.el	2005-02-18 07:23:24 +0000
@@ -35,6 +35,7 @@
 
 (require 'qp)
 (require 'mm-util)
+(require 'ietf-drums)
 ;; Fixme: Avoid this (used for mail-parse-charset) mm dependence on gnus.
 (require 'mail-prsvr)
 (require 'base64)
@@ -766,8 +767,7 @@
   (let* ((rfc2047-encoding-type 'mime)
 	 (rfc2047-encode-max-chars nil)
 	 (string (rfc2047-encode-string value)))
-    (if (string-match "[][()<>@,;:\\\"/?=]" ;; tspecials
-		      string)
+    (if (string-match (concat "[" ietf-drums-tspecials "]") string)
 	(format "%s=%S" param string)
       (concat param "=" string))))
 
@@ -780,6 +780,9 @@
     "=\\?\\([^][\000-\040()<>@,\;:*\\\"/?.=]+\\)\\(?:\\*[^?]+\\)?\
 \\?\\(B\\|Q\\)\\?\\([!->@-~ ]*\\)\\?="))
 
+(defvar rfc2047-quote-decoded-words-containing-tspecials nil
+  "If non-nil, quote decoded words containing special characters.")
+
 ;; Fixme: This should decode in place, not cons intermediate strings.
 ;; Also check whether it needs to worry about delimiting fields like
 ;; encoding.
@@ -814,14 +817,56 @@
 	  (insert (rfc2047-parse-and-decode
 		   (prog1
 		       (match-string 0)
-		     (delete-region (match-beginning 0) (match-end 0)))))
-	  ;; Remove newlines between decoded words, though such things
-	  ;; essentially must not be there.
+		     (delete-region e (match-end 0)))))
+	  (while (looking-at rfc2047-encoded-word-regexp)
+	    (insert (rfc2047-parse-and-decode
+		     (prog1
+			 (match-string 0)
+		       (delete-region (point) (match-end 0))))))
 	  (save-restriction
 	    (narrow-to-region e (point))
 	    (goto-char e)
+	    ;; Remove newlines between decoded words, though such
+	    ;; things essentially must not be there.
 	    (while (re-search-forward "[\n\r]+" nil t)
 	      (replace-match " "))
+	    (when rfc2047-quote-decoded-words-containing-tspecials
+	      ;; Quote decoded words if there are special characters
+	      ;; which might violate RFC2822.
+	      (let (quoted)
+		(goto-char e)
+		(skip-chars-forward " \t")
+		(setq start (point))
+		(setq quoted (eq (char-after) ?\"))
+		(goto-char (point-max))
+		(skip-chars-backward " \t")
+		(if (setq quoted (and quoted
+				      (> (point) (1+ start))
+				      (eq (char-before) ?\")))
+		    (progn
+		      (backward-char)
+		      (setq start (1+ start)
+			    end (point-marker)))
+		  (setq end (point-marker)))
+		(goto-char start)
+		(while (search-forward "\"" end t)
+		  (when (prog2
+			    (backward-char)
+			    (zerop (% (skip-chars-backward "\\\\") 2))
+			  (goto-char (match-beginning 0)))
+		    (insert "\\"))
+		  (forward-char))
+		(when (and (not quoted)
+			   (progn
+			     (goto-char start)
+			     (re-search-forward
+			      (concat "[" ietf-drums-tspecials "]")
+			      end t)))
+		  (goto-char start)
+		  (insert "\"")
+		  (goto-char end)
+		  (insert "\""))
+		(set-marker end nil)))
 	    (goto-char (point-max)))
 	  (when (and (mm-multibyte-p)
 		     mail-parse-charset

Re: Reply misinterprets Commas in encoded-words (Gnus violates RFC

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 544 bytes --]

>>>>> In <b9yis4ql3wu.fsf@jpl.org> Katsumi Yamaoka wrote:

> I agree.  Today I found a better way to fix the problem without
> modifying the rfc2047 encoder.  It doesn't require to modify the
> way to build the reply headers either.  The solution is to quote
> decoded words in the " *gnus article copy*" buffer (in which the
> reply headers are extracted) if there are special characters.
> The new patch for the trunk is below.

Please apply the following additional patch.  It is needed so as
not to quote words in the Subject header, etc.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 937 bytes --]

--- rfc2047.el~	2005-02-18 07:31:04 +0000
+++ rfc2047.el	2005-02-18 08:40:05 +0000
@@ -830,9 +830,19 @@
 	    ;; things essentially must not be there.
 	    (while (re-search-forward "[\n\r]+" nil t)
 	      (replace-match " "))
-	    (when rfc2047-quote-decoded-words-containing-tspecials
-	      ;; Quote decoded words if there are special characters
-	      ;; which might violate RFC2822.
+	    ;; Quote decoded words if there are special characters
+	    ;; which might violate RFC2822.
+	    (when (and rfc2047-quote-decoded-words-containing-tspecials
+		       (let ((regexp (car (rassq
+					   'address-mime
+					   rfc2047-header-encoding-alist))))
+			 (when regexp
+			   (save-restriction
+			     (widen)
+			     (beginning-of-line)
+			     (while (and (memq (char-after) '(?  ?\t))
+					 (zerop (forward-line -1))))
+			     (looking-at regexp)))))
 	      (let (quoted)
 		(goto-char e)
 		(skip-chars-forward " \t")

Re: Reply misinterprets Commas in encoded-words (Gnus violates RFC

[Reiner Steib]

On Fri, Feb 18 2005, Katsumi Yamaoka wrote:

>>>>>> In <b9yis4ql3wu.fsf@jpl.org> Katsumi Yamaoka wrote:
>
>> I agree.  Today I found a better way to fix the problem without
>> modifying the rfc2047 encoder.  It doesn't require to modify the
>> way to build the reply headers either.  The solution is to quote
>> decoded words in the " *gnus article copy*" buffer (in which the
>> reply headers are extracted) if there are special characters.
>> The new patch for the trunk is below.
>
> Please apply the following additional patch.  It is needed so as
> not to quote words in the Subject header, etc.

The patch seems to work correctly for me (cursorily tested).  It
requires some manual adaption for v5-10 (one hunk fails).

Could you please apply it to both, trunk and v5-10 (if you think it is
stable enough)?  Thanks.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Reply misinterprets Commas in encoded-words (Gnus violates RFC

[Katsumi Yamaoka]

>>>>> In <v9k6p66i86.fsf@marauder.physik.uni-ulm.de> 
>>>>>	Reiner Steib <reinersteib+gmane@imap.cc> wrote:

> The patch seems to work correctly for me (cursorily tested).  It
> requires some manual adaption for v5-10 (one hunk fails).

> Could you please apply it to both, trunk and v5-10 (if you think it
> is stable enough)?  Thanks.

Yes.  I've committed it to both just now adding two additional
changes.  One is for editing articles, the other is for
forwarding articles.  Because things like "Wiens, Stefªn" in
address headers should be quoted so as to be encoded together.