`hard-newline' changes trigger format=flowed in Gnus

`hard-newline' changes trigger format=flowed in Gnus

[Reiner Steib]

Hi,

the changes related to `hard-newline' seem to provoke undesired
behavior in Gnus.

I can't reproduce the problem with an older CVS build (before
`hard-newline' was introduced): "GNU Emacs 22.0.50.1
(i386-mingw-nt5.1.2600) of 2005-06-26 on NONIQPC".  I can work around
the problem by setting (setq hard-newline "\n") so I'm quite sure the
problem is triggered by the `hard-newline' changes.

Bug description:

When copying some lines from the mail header (in Gnus' message mode)
into the mail body, some of the newlines seem to be marked as `hard'.
When sending (or previewing) the message, the function
`fill-flowed-encode' detects the hard newlines and the whole message
is encodes as format=flowed which leads to an unexpected output.

To reproduce, save the following lines to `flow-fill_test.el' and
start emacs with "emacs -Q -l flow-fill_test.el".

--8<---------------cut here---------------start------------->8---
(gnus-slave-no-server 1)
(setq temp-address "foo@bar.invalid")
(gnus-msg-mail temp-address
	       "test on hard-newlines"
	       `(("From" ,temp-address)))
(with-current-buffer (get-buffer (format "*mail to %s*" temp-address))
  (insert-buffer-substring (current-buffer)
			   (point-min) (- (point-max) 13))
  (insert "...\nfoo\nbar\nbaz\nzab\n")
  (mml-preview))
--8<---------------cut here---------------end--------------->8---

The message buffer looks like this:

,----
| From: foo@bar.invalid
| To: foo@bar.invalid
| Subject: test on hard-newlines
| --text follows this line--
| From: foo@bar.invalid
| To: foo@bar.invalid
| Subject: test on hard-newlines
| --text follows...
| foo
| bar
| baz
| zab
`----

Whereas in the preview buffer and the sent message, I get:

,----
| From: foo@bar.invalid
| Subject: test on hard-newlines
| To: foo@bar.invalid
| Date: Tue, 29 Nov 2005 21:21:36 +0100
| 
| From: foo@bar.invalid
| To: foo@bar.invalid Subject: test on hard-newlines
| --text follows...  foo bar baz zab
`----

Apparently the newlines in the From and Subject lines are hard and the
newline in the To line and all other lines aren't.

I don't know if (and how) this bug should be fixed Gnus
(`gnus/flow-fill.el' or `gnus/message.el'?) or in Emacs.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: `hard-newline' changes trigger format=flowed in Gnus

[Romain Francoise]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> the changes related to `hard-newline' seem to provoke undesired
> behavior in Gnus.

Yes, see also <URL: http://bugs.debian.org/340988>.

> I don't know if (and how) this bug should be fixed Gnus
> (`gnus/flow-fill.el' or `gnus/message.el'?) or in Emacs.

I think these changes should just be reverted...

-- 
Romain Francoise <romain@orebokech.com> | The sea! the sea! the open
it's a miracle -- http://orebokech.com/ | sea! The blue, the fresh, the
                                        | ever free! --Bryan W. Procter

Re: `hard-newline' changes trigger format=flowed in Gnus

[David Koppelman]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> Hi,
>
> the changes related to `hard-newline' seem to provoke undesired
> behavior in Gnus.
[snip]
> When copying some lines from the mail header (in Gnus' message mode)
> into the mail body, some of the newlines seem to be marked as `hard'.
> When sending (or previewing) the message, the function
> `fill-flowed-encode' detects the hard newlines and the whole message
> is encodes as format=flowed which leads to an unexpected output.
[snip]
> I don't know if (and how) this bug should be fixed Gnus
> (`gnus/flow-fill.el' or `gnus/message.el'?) or in Emacs.

This problem is particularly insidious because the sender does not see
the flowed message, only the recipient who might not realize anything
is wrong.

Re: `hard-newline' changes trigger format=flowed in Gnus

[Chong Yidong]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> When copying some lines from the mail header (in Gnus' message mode)
> into the mail body, some of the newlines seem to be marked as `hard'.
> When sending (or previewing) the message, the function
> `fill-flowed-encode' detects the hard newlines and the whole message
> is encodes as format=flowed which leads to an unexpected output.

What is the intended purpose of fill-flowed-encode?  What is the
reason for message-mode to do anything special when there are hard
newlines in the buffer?

Re: `hard-newline' changes trigger format=flowed in Gnus

[Per Abrahamsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> What is the intended purpose of fill-flowed-encode?

To propagate the difference between hard and soft newlines to the
recipient.

Re: `hard-newline' changes trigger format=flowed in Gnus

[Chong Yidong]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> the changes related to `hard-newline' seem to provoke undesired
> behavior in Gnus.

I found out that it's not necessary to use hard-newline in
those places after all.  I've reverted the 2005-11-29 changes to 
mailheader.el

Thanks.

Re: `hard-newline' changes trigger format=flowed in Gnus

[Reiner Steib]

On Tue, Nov 29 2005, Chong Yidong wrote:

> Reiner Steib <reinersteib+gmane@imap.cc> writes:
>
>> When copying some lines from the mail header (in Gnus' message mode)
>> into the mail body, some of the newlines seem to be marked as `hard'.
>> When sending (or previewing) the message, the function
>> `fill-flowed-encode' detects the hard newlines and the whole message
>> is encodes as format=flowed which leads to an unexpected output.
>
> What is the intended purpose of fill-flowed-encode?  What is the
> reason for message-mode to do anything special when there are hard
> newlines in the buffer?

There is some information in emacs-mime.texi:

,----[ (info "(emacs-mime)Flowed text") ]
| The Emacs MIME library will respect the `use-hard-newlines' variable
| (*note Hard and Soft Newlines: (emacs)Hard and Soft Newlines.) when
| encoding a message, and the "format=flowed" Content-Type parameter when
| decoding a message.
| 
|    On encoding text, regardless of `use-hard-newlines', lines
| terminated by soft newline characters are filled together and wrapped
| after the column decided by `fill-flowed-encode-column'.  Quotation
| marks (matching `^>* ?') are respected.  The variable controls how the
| text will look in a client that does not support flowed text, the
| default is to wrap after 66 characters.  If hard newline characters are
| not present in the buffer, no flow encoding occurs.
| [...]
`----

IMHO, it won't hurt to add a short note and reference to this
explanation in gnus.texi and message.texi.  The only occurrence is in
the "New Features" section (info "(gnus)Oort Gnus").  Suggestions?


On Wed, Nov 30 2005, Chong Yidong wrote:

> Reiner Steib <reinersteib+gmane@imap.cc> writes:
>
>> the changes related to `hard-newline' seem to provoke undesired
>> behavior in Gnus.
>
> I found out that it's not necessary to use hard-newline in those
> places after all.  I've reverted the 2005-11-29 changes to
> mailheader.el

Thanks.  AFAICS this fixed the problem.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: `hard-newline' changes trigger format=flowed in Gnus

[Sam Steingold]

> * Reiner Steib <ervarefgrvo+tznar@vznc.pp> [2005-11-30 16:10:28 +0100]:
>
> |    On encoding text, regardless of `use-hard-newlines', lines
> | terminated by soft newline characters are filled together and wrapped
> | after the column decided by `fill-flowed-encode-column'.  Quotation
> | marks (matching `^>* ?') are respected.  The variable controls how the
> | text will look in a client that does not support flowed text, the
> | default is to wrap after 66 characters.  If hard newline characters are
> | not present in the buffer, no flow encoding occurs.

during the short period while message.el was broken, this was _NOT_ the
case, see, e.g., <http://article.gmane.org/gmane.emacs.devel:46314>
where my message is filled _TOGETHER_ with the message I quoted into a
single paragraph.

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
http://www.camera.org http://www.openvotingconsortium.org/
http://www.iris.org.il http://www.palestinefacts.org/ http://www.memri.org/
I haven't lost my mind -- it's backed up on tape somewhere.

Re: `hard-newline' changes trigger format=flowed in Gnus

[Reiner Steib]

On Wed, Nov 30 2005, Sam Steingold wrote:

>> * Reiner Steib <ervarefgrvo+tznar@vznc.pp> [2005-11-30 16:10:28 +0100]:
>>
>> |    On encoding text, regardless of `use-hard-newlines', lines
>> | terminated by soft newline characters are filled together and wrapped
>> | after the column decided by `fill-flowed-encode-column'.  [...]
>> | If hard newline characters are not present in the buffer, no flow
>> | encoding occurs.
>
> during the short period while message.el was broken, this was _NOT_ the
> case, see, e.g., <http://article.gmane.org/gmane.emacs.devel:46314>
> where my message is filled _TOGETHER_ with the message I quoted into a
> single paragraph.

But I guess that there _where_ hard newlines in your message buffer?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: `hard-newline' changes trigger format=flowed in Gnus

[Sam Steingold]

> * Reiner Steib <ervarefgrvo+tznar@vznc.pp> [2005-11-30 21:23:16 +0100]:
>
> On Wed, Nov 30 2005, Sam Steingold wrote:
>
>>> * Reiner Steib <ervarefgrvo+tznar@vznc.pp> [2005-11-30 16:10:28 +0100]:
>>>
>>> |    On encoding text, regardless of `use-hard-newlines', lines
>>> | terminated by soft newline characters are filled together and wrapped
>>> | after the column decided by `fill-flowed-encode-column'.  [...]
>>> | If hard newline characters are not present in the buffer, no flow
>>> | encoding occurs.
>>
>> during the short period while message.el was broken, this was _NOT_ the
>> case, see, e.g., <http://article.gmane.org/gmane.emacs.devel:46314>
>> where my message is filled _TOGETHER_ with the message I quoted into a
>> single paragraph.
>
> But I guess that there _where_ hard newlines in your message buffer?

The honest answer, I am afraid, is "I don't know".
I did not know what is a hard-newline then
(and I am sorry that I do know that now),
and I certainly did nothing special to get it inserted into the message.

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
http://www.iris.org.il http://pmw.org.il/ http://www.memri.org/
http://truepeace.org http://ffii.org/ http://www.dhimmi.com/
People hear what they want to hear and discard the rest.

Re: `hard-newline' changes trigger format=flowed in Gnus

[Reiner Steib]

On Wed, Nov 30 2005, Sam Steingold wrote:

>> * Reiner Steib <ervarefgrvo+tznar@vznc.pp> [2005-11-30 21:23:16 +0100]:
>>
>> On Wed, Nov 30 2005, Sam Steingold wrote:
[...]
>>> during the short period while message.el was broken, this was _NOT_ the
>>> case, see, e.g., <http://article.gmane.org/gmane.emacs.devel:46314>
>>> where my message is filled _TOGETHER_ with the message I quoted into a
>>> single paragraph.
>>
>> But I guess that there _where_ hard newlines in your message buffer?
>
> The honest answer, I am afraid, is "I don't know".
> I did not know what is a hard-newline then
> (and I am sorry that I do know that now),

(info "(emacs)Hard and Soft Newlines") should tell you.  If not, the
manual node should be improved.

> and I certainly did nothing special to get it inserted into the message.

I guess it was caused by those "hard-newline" commits in gnus/*.el at
that time or the stuff Chong Yidong reverted now.  I hope everything
is fixed now.  To be sure, you may advice `fill-flowed-encode' with a
`y-or-no' query or a `debug'.

[ See also my reply to David Koppelman
  <news:v9vey84spy.fsf@marauder.physik.uni-ulm.de> ]

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: `hard-newline' changes trigger format=flowed in Gnus

[David Koppelman]

Reiner Steib <reinersteib+gmane@imap.cc> writes:

> There is some information in emacs-mime.texi:
>
> ,----[ (info "(emacs-mime)Flowed text") ]
> | The Emacs MIME library will respect the `use-hard-newlines' variable
> | (*note Hard and Soft Newlines: (emacs)Hard and Soft Newlines.) when
> | encoding a message, and the "format=flowed" Content-Type parameter when
> | decoding a message.
> | 
> |    On encoding text, regardless of `use-hard-newlines', lines
> | terminated by soft newline characters are filled together and wrapped
> | after the column decided by `fill-flowed-encode-column'.  Quotation
> | marks (matching `^>* ?') are respected.  The variable controls how the
> | text will look in a client that does not support flowed text, the
> | default is to wrap after 66 characters.  If hard newline characters are
> | not present in the buffer, no flow encoding occurs.
> | [...]
> `----
>
> IMHO, it won't hurt to add a short note and reference to this
> explanation in gnus.texi and message.texi.  The only occurrence is in
> the "New Features" section (info "(gnus)Oort Gnus").  Suggestions?

If I understand things correctly if soft and hard newline characters
are found then it will be assumed that a soft newline is what was
intended and so text will be flowed, no questions asked. But there are
many cases in which the soft newline character is there by
"accident". The sender won't know until the recipient gets back to the
sender and then the sender will have to delve deeply into lisp or the
documentation to diagnose the problem and develop a workaround.

Therefore one must be much more careful before flowing. Perhaps some
combination of the following:

 Some indication that text will be assumed flowable, perhaps in the
 modeline or perhaps by highlighting around the soft newlines.

 If flowability has been assumed then ask the user if that's correct and
 take yes, yes always, no, and no never responses.

Re: `hard-newline' changes trigger format=flowed in Gnus

[Reiner Steib]

On Wed, Nov 30 2005, David Koppelman wrote:

> Reiner Steib <reinersteib+gmane@imap.cc> writes:
>
>> There is some information in emacs-mime.texi:
>>
>> ,----[ (info "(emacs-mime)Flowed text") ]
>> | The Emacs MIME library will respect the `use-hard-newlines' variable
>> | (*note Hard and Soft Newlines: (emacs)Hard and Soft Newlines.) when
>> | encoding a message, and the "format=flowed" Content-Type parameter when
>> | decoding a message.
[...]
>> IMHO, it won't hurt to add a short note and reference to this
>> explanation in gnus.texi and message.texi.  The only occurrence is in
>> the "New Features" section (info "(gnus)Oort Gnus").  Suggestions?
>
> If I understand things correctly if soft and hard newline characters
> are found 

The code (see `mml-generate-mime-1') checks for hard newlines:

  (text-property-any (point-min) (point-max) 'hard 't)

The soft newlines are the normal '\n'-s without any special text
property.   (At least that is my understanding, CMIIW.)

> then it will be assumed that a soft newline is what was intended and
> so text will be flowed, no questions asked. But there are many cases
> in which the soft newline character is there by "accident". 

Maybe you meant "hard newline" in this paragraph?

> The sender won't know until the recipient gets back to the sender
> and then the sender will have to delve deeply into lisp or the
> documentation to diagnose the problem and develop a workaround.

ACK.  I only noticed this bug because I was sending mail to some other
account of mine while testing /etc/postfix/virtual (therefore copying
the headers of the message into the body).

> Therefore one must be much more careful before flowing. Perhaps some
> combination of the following:
>
>  Some indication that text will be assumed flowable, perhaps in the
>  modeline or perhaps by highlighting around the soft newlines.

Probably highlighting hard newlines is possible.  I guess people on
emacs-devel can tell you how.  Performing a "flowable" test while
writing a message doesn't make much sense IMHO.

>  If flowability has been assumed then ask the user if that's correct and
>  take yes, yes always, no, and no never responses.

Maybe something like this?

--8<---------------cut here---------------start------------->8---
--- mml.el	01 Nov 2005 14:18:40 +0100	7.38
+++ mml.el	01 Dec 2005 18:27:37 +0100	
@@ -413,6 +413,8 @@
 				      cont)))
 	(buffer-string)))))
 
+(defvar mml-confirm-flow-fill t)
+
 (defun mml-generate-mime-1 (cont)
   (let ((mm-use-ultra-safe-encoding
 	 (or mm-use-ultra-safe-encoding (assq 'sign cont))))
@@ -481,16 +483,21 @@
 		    ;; parts where there either isn't a format parameter
 		    ;; in the mml tag or it says "flowed" and there
 		    ;; actually are hard newlines in the text.
-		    (let (use-hard-newlines)
+		    (let (found-hard-newlines)
 		      (when (and (string= type "text/plain")
 				 (not (string= (cdr (assq 'sign cont)) "pgp"))
 				 (or (null (assq 'format cont))
 				     (string= (cdr (assq 'format cont))
 					      "flowed"))
-				 (setq use-hard-newlines
+				 (setq found-hard-newlines
 				       (text-property-any
 					(point-min) (point-max) 'hard 't)))
-			(fill-flowed-encode)
+			(when (and found-hard-newlines
+				   (or mml-confirm-flow-fill
+				       (y-or-n-p
+					"Hard newlines found.  Send as \"format=flowed\"? ")))
+			  (let ((use-hard-newlines found-hard-newlines))
+			    (fill-flowed-encode)))
 			;; Indicate that `mml-insert-mime-headers' should
 			;; insert a "; format=flowed" string unless the
 			;; user has already specified it.
--8<---------------cut here---------------end--------------->8---

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/