bug: \201 in summary buffer

bug: \201 in summary buffer

[Vladimir Volovich]

Hi,

i've sent two test messages containing digests (with subjects "bug
testing: first digest" and "bug testing: second digest").

i use the nnmbox backend to receive my messages, so i could not say
whether this bug is present in other backends (note: this is not a
"mule suxx" kind of bug, but a bug in gnus because of double encoding
the buffer :).

mark these messages in a summary buffer with a hash #, and then press
M-C-d to see the digests. the prepared summary buffer contains \201s
instead of 8-bit iso-8859-1 characters (the message bodies are okay).

previewing these messages alone (with C-d) does not show the bug, ---
only M-C-d shows the bug.

	Best regards, -- Vladimir.

Re: bug: \201 in summary buffer

[Kai.Grossjohann]

Vladimir Volovich <vvv@vvv.vsu.ru> writes:

  > i use the nnmbox backend to receive my messages, so i could not say
  > whether this bug is present in other backends (note: this is not a
  > "mule suxx" kind of bug, but a bug in gnus because of double encoding
  > the buffer :).

Using nnml with pgnus-0.80, I see the same problem.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bug: \201 in summary buffer

[Lars Magne Ingebrigtsen]

Kai.Grossjohann@CS.Uni-Dortmund.DE writes:

>   > i use the nnmbox backend to receive my messages, so i could not say
>   > whether this bug is present in other backends (note: this is not a
>   > "mule suxx" kind of bug, but a bug in gnus because of double encoding
>   > the buffer :).
> 
> Using nnml with pgnus-0.80, I see the same problem.

I seem to be unable to reproduce this bug.  Could you uuencode and
forward me a digest that displays this bug?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bug: \201 in summary buffer

[Kai.Grossjohann]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

  > I seem to be unable to reproduce this bug.  Could you uuencode and
  > forward me a digest that displays this bug?

Hm.  I don't have this mail anymore.  It's still in the mailing list
archive, but of course in the wrong format.  Hm.  Sorry.

kai
-- 
Abort this operation?   [Abort]  [Cancel]

patch: \201 when cache (was: bug: \201 in summary buffer)

[Shenghuo ZHU]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]


When I enter an article with non-ASCII in subject to cache, \201
appears in summary buffer when I reenter the group. The reason is that
gnus-cache-possibly-enter-article uses the existing headers which may
contain mule characters. I guess this may also cause Kai's bug.

Here are two patches. The first forces
gnus-cache-possibly-enter-article to re-parse the headers. The second
does a deep clean job, removing parameter `headers' of
gnus-cache-possibly-enter-article.

-- 
Shenghuo ZHU

Sun May  2 00:55:54 1999  Shenghuo ZHU  <zsh@cs.rochester.edu>

	* gnus-cache.el (gnus-cache-possibly-enter-article): Re-parse header.

[-- Attachment #2: 84-zsh1.diff --]
[-- Type: application/octet-stream, Size: 435 bytes --]

--- gnus-cache.el	1999/05/02 04:41:44	1.1
+++ gnus-cache.el	1999/05/02 04:50:14
@@ -183,6 +183,8 @@
 	      (gnus-request-article-this-buffer number group))
 	    (when (> (buffer-size) 0)
 	      (gnus-write-buffer file)
+	      (setq headers (nnheader-parse-head t))
+	      (mail-header-set-number headers number)
 	      (gnus-cache-change-buffer group)
 	      (set-buffer (cdr gnus-cache-buffer))
 	      (goto-char (point-max))

[-- Attachment #3: Type: text/plain, Size: 359 bytes --]


Sun May  2 01:00:02 1999  Shenghuo ZHU  <zsh@cs.rochester.edu>

	* gnus-cache.el (gnus-cache-possibly-enter-article): Remove
 	parameter `headers'.
	(gnus-cache-enter-article): Ditto.
	(gnus-cache-update-article): Ditto.  
	* gnus-sum.el (gnus-summary-move-article): Ditto.
	(gnus-summary-mark-article-as-unread): Ditto.
	(gnus-summary-mark-article): Ditto.

[-- Attachment #4: 84-zsh2.diff --]
[-- Type: application/octet-stream, Size: 2871 bytes --]

--- gnus-sum.el	1999/05/02 05:02:05	1.1
+++ gnus-sum.el	1999/05/02 05:03:51
@@ -7233,10 +7233,6 @@
 	      (when gnus-use-cache
 		(gnus-cache-possibly-enter-article
 		 to-group to-article
-		 (let ((header (copy-sequence
-				(gnus-summary-article-header article))))
-		   (mail-header-set-number header to-article)
-		   header)
 		 (memq article gnus-newsgroup-marked)
 		 (memq article gnus-newsgroup-dormant)
 		 (memq article gnus-newsgroup-unreads)))
@@ -7904,7 +7900,6 @@
 	     (save-excursion
 	       (gnus-cache-possibly-enter-article
 		gnus-newsgroup-name article
-		(gnus-summary-article-header article)
 		(= mark gnus-ticked-mark)
 		(= mark gnus-dormant-mark) (= mark gnus-unread-mark))))
 
@@ -7950,7 +7945,6 @@
 	     (save-excursion
 	       (gnus-cache-possibly-enter-article
 		gnus-newsgroup-name article
-		(gnus-summary-article-header article)
 		(= mark gnus-ticked-mark)
 		(= mark gnus-dormant-mark) (= mark gnus-unread-mark))))
 
--- gnus-cache.el	1999/05/02 04:57:16	1.2
+++ gnus-cache.el	1999/05/02 05:09:20
@@ -145,20 +145,17 @@
       (setq gnus-cache-buffer nil))))
 
 (defun gnus-cache-possibly-enter-article
-  (group article headers ticked dormant unread &optional force)
+  (group article ticked dormant unread &optional force)
   (when (and (or force (not (eq gnus-use-cache 'passive)))
 	     (numberp article)
-	     (> article 0)
-	     (vectorp headers))		; This might be a dummy article.
-    ;; If this is a virtual group, we find the real group.
-    (when (gnus-virtual-group-p group)
-      (let ((result (nnvirtual-find-group-art
-		     (gnus-group-real-name group) article)))
-	(setq group (car result)
-	      headers (copy-sequence headers))
-	(mail-header-set-number headers (cdr result))))
-    (let ((number (mail-header-number headers))
-	  file)
+	     (> article 0))		; This might be a dummy article.
+    (let ((number article) file headers)
+      ;; If this is a virtual group, we find the real group.
+      (when (gnus-virtual-group-p group)
+	(let ((result (nnvirtual-find-group-art
+		       (gnus-group-real-name group) article)))
+	  (setq group (car result)
+		number (cdr result))))
       (when (and number
 		 (> number 0)		; Reffed article.
 		 (or force
@@ -332,7 +329,6 @@
       (if (natnump article)
 	  (when (gnus-cache-possibly-enter-article
 		 gnus-newsgroup-name article
-		 (gnus-summary-article-header article)
 		 nil nil nil t)
 	    (push article out))
 	(gnus-message 2 "Can't cache article %d" article))
@@ -428,7 +424,7 @@
   (when (gnus-cache-possibly-remove-article article nil nil nil t)
     (let ((gnus-use-cache nil))
       (gnus-cache-possibly-enter-article
-       gnus-newsgroup-name article (gnus-summary-article-header article)
+       gnus-newsgroup-name article
        nil nil nil t))))
 
 (defun gnus-cache-possibly-remove-article (article ticked dormant unread

Re: patch: \201 when cache (was: bug: \201 in summary buffer)

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Shenghuo ZHU <zsh@cs.rochester.edu>  on Sun, 02 May 1999
| When I enter an article with non-ASCII in subject to cache, \201
| appears in summary buffer when I reenter the group.

Just FYI, raw 8-bit characters are not allowed in mail or news article
headers.  They *must* be encoded into a 7-bit format.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3LFBMgl+vIlSVSNkRAs0VAJ41znvNQwsR6Vkj13I08ewe0RVVBwCg4HFj
AfROPDEKtY44xdR5ZoskSJk=
=Xx4M
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Do not taunt Happy Fun Ball.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 

Re: patch: \201 when cache (was: bug: \201 in summary buffer)

[François Pinard]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> * Shenghuo ZHU <zsh@cs.rochester.edu>  on Sun, 02 May 1999
> | When I enter an article with non-ASCII in subject to cache, \201
> | appears in summary buffer when I reenter the group.

> Just FYI, raw 8-bit characters are not allowed in mail or news article
> headers.  They *must* be encoded into a 7-bit format.

Allowed or not, such characters often occur in practice.  I've been told
that some countries consider normal and very practical, despite standards
may say, to just let 8-bit clean transport of email, and then assume their
national charset.  I quite understand such reasons, and for email which
is not internationally aimed, encoding headers is just unneeded complexity.

In my personal case, I much prefer international-aware headers.  But we
should not be utterly dogmatic for others, and be prepared to cope with
8-bit characters in headers, whatever standards may say.

-- 
François Pinard   http://www.iro.umontreal.ca/~pinard

Re: patch: \201 when cache (was: bug: \201 in summary buffer)

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* François Pinard <pinard@iro.umontreal.ca>  on Sun, 02 May 1999
| Allowed or not, such characters often occur in practice.  I've been told
| that some countries consider normal and very practical, despite standards
| may say, to just let 8-bit clean transport of email, and then assume their
| national charset.  I quite understand such reasons, and for email which
| is not internationally aimed, encoding headers is just unneeded complexity.

It is still broken.  There are pleny of MUAs out there still in operation
that cannot deal with 8-bit data in mail messages, because it is not allowed.

| In my personal case, I much prefer international-aware headers.  But we
| should not be utterly dogmatic for others, and be prepared to cope with
| 8-bit characters in headers, whatever standards may say.

There is being liberal in what you accept, and then there is accepting
something that is certifiably wrong.  While there is a need to work around
the bugs that the FSF seems unwilling to admit even exist, this is not a
bug.  8-bit data is not allowed in Internet mail headers, period.  Short of
rejecting invalid data outright, the closest thing to the Right Thing Gnus
can do is to convert such headers to the ISO-style escaped 7-bit format.
This has the two-fold benefit of avoiding \201 entirely and rendering
messages in the standard format so that other MUAs will not barf on them.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3LGSMgl+vIlSVSNkRAo+1AKCfZDU1v/gFhPYjrzFbE0g8+SIqKgCdHbXk
2Xfacq4IZM4UoqcMYJKsMHM=
=imLL
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Warning: pregnant women, the elderly, and
Minion of Nathan - Nathan says Hi! \ children under 10 should avoid prolonged
PGP Key: at a key server near you!  \ exposure to Happy Fun Ball.

Re: patch: \201 when cache (was: bug: \201 in summary buffer)

[Hrvoje Niksic]

François Pinard <pinard@iro.umontreal.ca> writes:

> Allowed or not, such characters often occur in practice.  I've been told
> that some countries consider normal and very practical, despite standards
> may say, to just let 8-bit clean transport of email, and then assume their
> national charset.  I quite understand such reasons, and for email which
> is not internationally aimed, encoding headers is just unneeded complexity.
> 
> In my personal case, I much prefer international-aware headers.  But we
> should not be utterly dogmatic for others, and be prepared to cope with
> 8-bit characters in headers, whatever standards may say.

Being from such a country, I (for once) fully agree with everything
Francois said.

Speaking of "national charsets", we are still waging a battle to make
iso-8859-2 the standardly used charset, instead of the Windows "code
page 1250".

Re: patch: \201 when cache (was: bug: \201 in summary buffer)

[Shenghuo ZHU]

>>>>> "Rat" == Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

Rat> * Shenghuo ZHU <zsh@cs.rochester.edu>  on Sun, 02 May 1999
Rat> | When I enter an article with non-ASCII in subject to cache, \201
Rat> | appears in summary buffer when I reenter the group.

Rat> Just FYI, raw 8-bit characters are not allowed in mail or news article
Rat> headers.  They *must* be encoded into a 7-bit format.

This bug also exists in some RFC2047 encoded subjects.

-- 
Shenghuo ZHU

Re: patch: \201 when cache (was: bug: \201 in summary buffer)

[Per Abrahamsen]

François Pinard <pinard@iro.umontreal.ca> writes:

> Stainless Steel Rat <ratinox@peorth.gweep.net> writes:
> 
> > * Shenghuo ZHU <zsh@cs.rochester.edu>  on Sun, 02 May 1999
> > | When I enter an article with non-ASCII in subject to cache, \201
> > | appears in summary buffer when I reenter the group.
> 
> > Just FYI, raw 8-bit characters are not allowed in mail or news article
> > headers.  They *must* be encoded into a 7-bit format.
> 
> Allowed or not, such characters often occur in practice. 

More than that, it is recommended practice for many "cooperating
subnets", including the one the main Gnus author is located in.
Mostly because just-pass-8bit gave less trouble than QP encoded
headers.

Gnus should -- when possible -- allow for messages that conform to
local conventions, even when they override international standards.

Re: patch: \201 when cache (was: bug: \201 in summary buffer)

[François Pinard]

Stainless Steel Rat <ratinox@peorth.gweep.net> écrit:

> | [...] for email which is not internationally aimed, encoding headers
> | is just unneeded complexity.

> It is still broken.  There are pleny of MUAs out there still in operation
> that cannot deal with 8-bit data in mail messages, because it is not
> allowed.

I prefer to think such MUAs are themselves very broken, would it be only
by utter chauvinism :-).[1]

> the closest thing to the Right Thing Gnus can do is to convert such
> headers to the ISO-style escaped 7-bit format.  This has the two-fold
> benefit of avoiding \201 entirely and rendering messages in the standard
> format so that other MUAs will not barf on them.

The right thing to do is to leave incoming mail undisturbed and unconverted,
that is, as close as the original as possible, yet within reason.
Some practical concerns may bring Lars to choose otherwise and to do
slightly less than the right thing.  But since he shown good judgement so
far, I'm rather confident that he will do something sensible here as well.

For \201 problems, for one, I would feel better if they were solved instead
of avoided, because `avoiding' is just one form of `postponing'.

For how Gnus emits outgoing messages, I may well compose new messages and
directly edit 8-bit characters in headers, and when I do so, I expect Gnus
to act and do something reasonable over them.  This is not much related
to how it receives messages, nor to how it stocks them on disk.

--------------------
[1] The smiley applies to me, not the sentence, as I'm happy nevertheless.
MUAs which are pedantic enough to refuse 8-bit email are surely no fun.

-- 
François Pinard                            mailto:pinard@iro.umontreal.ca
Join the free Translation Project!    http://www.iro.umontreal.ca/~pinard

Re: patch: \201 when cache (was: bug: \201 in summary buffer)

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > Just FYI, raw 8-bit characters are not allowed in mail or news article
  > headers.  They *must* be encoded into a 7-bit format.

ESMTP contains 8BITMIME which allows eight-bit characters.  Thus, if
both ends of a connection agree on speaking 8BITMIME, eight-bit
characters may be transferred.

Presumably, the recipient may just drop the message into the maildrop
as-is, without further processing, or may encode the message using
quoted-printable.  I'm not sure whether the action of the recipient is
specified anywhere, though.  [time passes]  Ah, the RFC 1652 specifies
that a compliant server must deliver the content without loss of
information, meaning it must deliver it as-is, or transform (without
loss of information) into ASCII or report an error.  From this I infer
that a server conforming to RFC 1652 may drop a message into the
maildrop which contains eight-bit characters.

RFC 977 specifies ASCII only, though.

kai
-- 
Abort this operation?   [Abort]  [Cancel]

Re: patch: \201 when cache (was: bug: \201 in summary buffer)

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* François Pinard <pinard@IRO.UMontreal.CA>  on Sun, 02 May 1999
| I prefer to think such MUAs are themselves very broken, would it be only
| by utter chauvinism :-).[1]

It is not that they are broken, because they are not.  That they are old,
running on legacy systems, and cannot be be updated.

[...]
| The right thing to do is to leave incoming mail undisturbed and
| unconverted, that is, as close as the original as possible, yet within
| reason.

An 8-bit clean MTA is required to convert 8-bit data to a standard 7-bit
representation when delivering to an MTA that is not 8-bit clean.  If it
fails to do that, chances are the message has already been irecoverably
mangled.  At that point all the MUA (Gnus) can do is try to pick up the
pieces to prevent any further damage.  A non-destructive transformation
like ISO escaping falls "within reason" for this purpose.

[...]
| For \201 problems, for one, I would feel better if they were solved
| instead of avoided, because `avoiding' is just one form of `postponing'.

A workaround in Gnus, regardless of how it is implemented, is not a
solution.  The posted workaround fixes the problem for Gnus.  My suggestion 
does that as well as prevents problems with other MUAs which do not or
cannot deal with 8-bit data.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3LckCgl+vIlSVSNkRAhsjAJ9ostxhye8jAA8XCPgTUcn5qat+jgCgsyic
CkbeTXDq8paPZAjpGG3yeRc=
=35LI
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ If Happy Fun Ball begins to smoke, get
Minion of Nathan - Nathan says Hi! \ away immediately. Seek shelter and cover
PGP Key: at a key server near you!  \ head.

Re: patch: \201 when cache (was: bug: \201 in summary buffer)

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Shenghuo ZHU <zsh@cs.rochester.edu>  on Sun, 02 May 1999
Rat> Just FYI, raw 8-bit characters are not allowed in mail or news article
Rat> headers.  They *must* be encoded into a 7-bit format.

| This bug also exists in some RFC2047 encoded subjects.

If I understand RFC 2047 correctly, this should be impossible unless there
are serious bugs in FSF Emacs (which there are :P).
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3Lcljgl+vIlSVSNkRAqaOAKD1zC8ZzYXViYYckF4G9euN6g2qLACeIaw5
pVUay4LLFkJHZalv/lDry/M=
=cVlB
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Do not taunt Happy Fun Ball.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 

Re: patch: \201 when cache (was: bug: \201 in summary buffer)

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Kai.Grossjohann@CS.Uni-Dortmund.DE  on Sun, 02 May 1999
| Ah, the RFC 1652 specifies that a compliant server must deliver the
| content without loss of information, meaning it must deliver it as-is, or
| transform (without loss of information) into ASCII or report an error.
| From this I infer that a server conforming to RFC 1652 may drop a message
| into the maildrop which contains eight-bit characters.

Acceptable behaviour, yes.  But it is safest to perform a non-destructive
7-bit transformation on the data.

| RFC 977 specifies ASCII only, though.

That is why.  Vanilla ASCII is the most portable data format in the world.
It works everywhere, even with legacy systems that for whatever reason
cannot be upgraded.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3Lcs8gl+vIlSVSNkRAvBxAJoDzCxQjOdMnv4pPdxjRdywHiXRGwCguHXb
gf9r9bHbSu/M//m4aXAXHQ0=
=I3/A
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Do not taunt Happy Fun Ball.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \