[flow-fill.el] Filling of quoted blank lines

[flow-fill.el] Filling of quoted blank lines

[Ralf Angeli]

Hello everybody,

there are quite some people out there using Mozilla with
`format=flowed' enabled.  This is handled nicely by `flow-fill.el'
except when it comes to quoted blank lines.  Mozilla is quoting them
with `> \n' where `flow-fill.el' sees soft line breaks at the end. The
lines are filled and disappear.  You can have a look at an example in
comp.text.tex; the MID is <news:bqia0o$2tm$02$1@news.t-online.com>.
Compare the display of the message using `C-u g' and `g'.  In the
latter case some lines are missing.

Is this the intended behaviour?  I find it a little bit irritating and
sometimes don't even want to answer such posts as I am afraid that
Gnus messes up the text.

I looked into `flow-fill.el' and made some changes which now take
lines with `> ' into account and won't fill them.  They are attached
at the bottom of this article.  I am not sure if I understood the code
correctly so there might be some changes needed in the respective
regular expressions.

It would be nice if the behaviour of Gnus could be changed in the
respective manner.

Regards,
Ralf

-- 
 Moo!
     `~~_.
______',,/______ No gnus is bad news.


--8<---------------cut here---------------start------------->8---
Index: flow-fill.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/flow-fill.el,v
retrieving revision 6.17
diff -u -r6.17 flow-fill.el
--- flow-fill.el	23 Oct 2003 05:52:49 -0000	6.17
+++ flow-fill.el	2 Dec 2003 17:14:01 -0000
@@ -112,7 +112,7 @@
 	      (beginning-of-line)
 	      (looking-at "^\\(>*\\)\\( ?\\)"))
 	(let ((quote (match-string 1))
-	      sig)
+	      except)
 	  (if (string= quote "")
 	      (setq quote nil))
 	  (when (and quote (string= (match-string 2) ""))
@@ -124,8 +124,10 @@
 	  ;; XXX slightly buggy handling of "-- "
 	  (while (and (save-excursion
 			(ignore-errors (backward-char 3))
-			(setq sig (looking-at "-- "))
-			(looking-at "[^-][^-] "))
+			(setq except (or (looking-at "-- ")
+                                         (looking-at ">[ >]*$")))
+			(looking-at "[^-][^-] ")
+                        (looking-at "[^>] $"))
 		      (save-excursion
 			(unless (eobp)
 			  (forward-char 1)
@@ -136,7 +138,7 @@
 				 "" "\\2")))
 	    (backward-delete-char -1)
 	    (end-of-line))
-	  (unless sig
+	  (unless except
 	    (condition-case nil
 		(let ((fill-prefix (when quote (concat quote " ")))
 		      (fill-column (eval fill-flowed-display-column))
--8<---------------cut here---------------end--------------->8---

Re: [flow-fill.el] Filling of quoted blank lines

[Simon Josefsson]

Ralf Angeli <dev.null@iwi.uni-sb.de> writes:

> Hello everybody,
>
> there are quite some people out there using Mozilla with
> `format=flowed' enabled.  This is handled nicely by `flow-fill.el'
> except when it comes to quoted blank lines.  Mozilla is quoting them
> with `> \n' where `flow-fill.el' sees soft line breaks at the end. The
> lines are filled and disappear.  You can have a look at an example in
> comp.text.tex; the MID is <news:bqia0o$2tm$02$1@news.t-online.com>.
> Compare the display of the message using `C-u g' and `g'.  In the
> latter case some lines are missing.

I could not find that MID, but I found another message that looked like:

> 
> foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar
> foo bar

This was rendered by Gnus as:

> foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo
> bar
> foo bar

(This message should illustrate this.)

This is suboptimal, but I believe it is compliant given the weird
input.

Is this what you mean by "disappear"?

> I looked into `flow-fill.el' and made some changes which now take
> lines with `> ' into account and won't fill them.  They are attached
> at the bottom of this article.  I am not sure if I understood the code
> correctly so there might be some changes needed in the respective
> regular expressions.

I applied the patch, but the above message looked the same.  Perhaps
there are two separate problems.

Re: [flow-fill.el] Filling of quoted blank lines

[Ralf Angeli]

* Simon Josefsson (2003-12-02) writes:

> I could not find that MID, but I found another message that looked like:
>
>>
>> foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar
>> foo bar
>
> This was rendered by Gnus as:
>
>> foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo
>> bar
>> foo bar
>
> (This message should illustrate this.)
>
> This is suboptimal, but I believe it is compliant given the weird
> input.
>
> Is this what you mean by "disappear"?

Not exactly.  Here is a copy of the contents of the message with the
MID given in my last article:

--8<---------------cut here---------------start------------->8---
From: Herbert Voss <Herbert.Voss@alumni.TU-Berlin.DE>
Newsgroups: comp.text.tex
Subject: Re: Collection of standard mathematical formulas
Date: Tue, 02 Dec 2003 16:10:18 +0100
Organization: T-Online
Message-ID: <bqia0o$2tm$02$1@news.t-online.com>
References: <107028676.4Wq83rlaq2@ID-187157.news.dfncis.de> <slrnbsmr6r.s85.cswingle@nika.frontier.iarc.uaf.edu> <20585475.PkF9EzsXAW@ID-187157.news.dfncis.de> <pan.2003.12.02.15.02.53.985000@hotmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
X-Trace: news.t-online.com 1070377817 02 2998 rZ2ZG4COnIJTpNi 031202 15:10:17
X-Complaints-To: usenet-abuse@t-online.de
X-ID: TFLzxvZvgelGGK4mOw0S1FIeeYMZH1UXoRXOFqjqBYwDhffJjaRUgt
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624 Netscape/7.1
X-Accept-Language: en-us, en, de
In-Reply-To: <pan.2003.12.02.15.02.53.985000@hotmail.com>

jblazi wrote:
> On Tue, 02 Dec 2003 08:53:15 +0100, Tristan Miller wrote:
> 
> 
>>  http://www.tug.org/texshowcase/
> 
> 
> How can I generate the PS output on Windows XP?

have a look at adobe for PS printer drivers

Herbert
--8<---------------cut here---------------end--------------->8---

Notice the blanks after the `>' characters.  The text will be rendered
as:

--8<---------------cut here---------------start------------->8---
jblazi wrote:
> On Tue, 02 Dec 2003 08:53:15 +0100, Tristan Miller wrote:
>
>>  http://www.tug.org/texshowcase/
> How can I generate the PS output on Windows XP?

have a look at adobe for PS printer drivers

Herbert
--8<---------------cut here---------------end--------------->8---

So every line with with a `^> \n$' will be deleted by the filling
function.

> I applied the patch, but the above message looked the same.  Perhaps
> there are two separate problems.

It might be.  Try to find an article produced by Mozilla which
encloses quoted blank lines and is of type `format=flowed'.

I found another one from comp.text.tex here:
<news:bq2tln$97m$04$1@news.t-online.com> This one is a bit older.

BTW: I don't know if the patch does not break filling `format=flowed'
messages completely as a change of `fill-column' did not have any
effect.

Regards,
Ralf

-- 
 Moo!
     `~~_.
______',,/______ No gnus is bad news.

Re: [flow-fill.el] Filling of quoted blank lines

[Ralf Angeli]

* Simon Josefsson (2003-12-02) writes:

> I could not find that MID, but I found another message that looked like:
>
>> 
>> foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar
>> foo bar
>
> This was rendered by Gnus as:
>
>> foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo
>> bar
>> foo bar
>
> (This message should illustrate this.)
>
> This is suboptimal, but I believe it is compliant given the weird
> input.

AFAICS this is a bug of my patch and was not intended to happen.
Without the patch your example should render something like this:

> foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo bar foo
> bar foo bar

I will see if I can find the bug.  Maybe somebody else has a solution?

Regards,
Ralf

-- 
 Moo!
     `~~_.
______',,/______ No gnus is bad news.

Re: [flow-fill.el] Filling of quoted blank lines

[Ralf Angeli]

* Ralf Angeli (2003-12-03) writes:

> I will see if I can find the bug.

Okay, here is a new patch which works correctly AFAICS.  I know it is
not very elegant as I just copied and adapted the code for finding the
signature but I did not want to change too much of the existing code.

It would be nice if somebody could have a look at it and maybe check
it in if it is acceptable.

Regards,
Ralf

-- 
 Moo!
     `~~_.
______',,/______ No gnus is bad news.


--8<---------------cut here---------------start------------->8---
Index: flow-fill.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/flow-fill.el,v
retrieving revision 6.17
diff -u -r6.17 flow-fill.el
--- flow-fill.el	23 Oct 2003 05:52:49 -0000	6.17
+++ flow-fill.el	3 Dec 2003 21:43:13 -0000
@@ -112,7 +112,7 @@
 	      (beginning-of-line)
 	      (looking-at "^\\(>*\\)\\( ?\\)"))
 	(let ((quote (match-string 1))
-	      sig)
+	      except)
 	  (if (string= quote "")
 	      (setq quote nil))
 	  (when (and quote (string= (match-string 2) ""))
@@ -124,8 +124,12 @@
 	  ;; XXX slightly buggy handling of "-- "
 	  (while (and (save-excursion
 			(ignore-errors (backward-char 3))
-			(setq sig (looking-at "-- "))
+			(setq except (looking-at "-- "))
 			(looking-at "[^-][^-] "))
+                      (save-excursion
+			(ignore-errors (backward-char 2))
+			(setq except (or except (looking-at "> ")))
+			(looking-at "[^>] "))
 		      (save-excursion
 			(unless (eobp)
 			  (forward-char 1)
@@ -136,7 +140,7 @@
 				 "" "\\2")))
 	    (backward-delete-char -1)
 	    (end-of-line))
-	  (unless sig
+	  (unless except
 	    (condition-case nil
 		(let ((fill-prefix (when quote (concat quote " ")))
 		      (fill-column (eval fill-flowed-display-column))
--8<---------------cut here---------------end--------------->8---

Re: [flow-fill.el] Filling of quoted blank lines

[Simon Josefsson]

Ralf Angeli <dev.null@iwi.uni-sb.de> writes:

> * Ralf Angeli (2003-12-03) writes:
>
>> I will see if I can find the bug.
>
> Okay, here is a new patch which works correctly AFAICS.  I know it is
> not very elegant as I just copied and adapted the code for finding the
> signature but I did not want to change too much of the existing code.
>
> It would be nice if somebody could have a look at it and maybe check
> it in if it is acceptable.

I hope this is slightly more robust, please test it.

It also fixes a long-standing problem in that:

foo 
-- 
bar

would be flowed together, when it shouldn't.

There really need to be a regression test harness for flow-fill, it is
difficult to tell what consequences even a minor change would have.

--- flow-fill.el.~6.17.~	2003-12-02 21:05:15.000000000 +0100
+++ flow-fill.el	2003-12-04 00:18:00.000000000 +0100
@@ -111,8 +111,7 @@
       (when (save-excursion
 	      (beginning-of-line)
 	      (looking-at "^\\(>*\\)\\( ?\\)"))
-	(let ((quote (match-string 1))
-	      sig)
+	(let ((quote (match-string 1)) skiponeline)
 	  (if (string= quote "")
 	      (setq quote nil))
 	  (when (and quote (string= (match-string 2) ""))
@@ -121,11 +120,18 @@
 	      (beginning-of-line)
 	      (when (> (skip-chars-forward ">") 0)
 		(insert " "))))
-	  ;; XXX slightly buggy handling of "-- "
+	  (unless (save-excursion
+		    (beginning-of-line)
+		    (looking-at (format "^%s ?-- $" (or quote " ?"))))
 	  (while (and (save-excursion
-			(ignore-errors (backward-char 3))
-			(setq sig (looking-at "-- "))
-			(looking-at "[^-][^-] "))
+			  (ignore-errors (forward-char 1))
+			  (not (setq skiponeline
+				     ;; Next line contains signature separator.
+				     (or (looking-at (format "^%s ?-- $"
+							     (or quote "")))
+					 ;; Next line is empty, preserve it.
+					 (looking-at (format "^%s * $"
+							     (or quote "")))))))
 		      (save-excursion
 			(unless (eobp)
 			  (forward-char 1)
@@ -136,7 +142,6 @@
 				 "" "\\2")))
 	    (backward-delete-char -1)
 	    (end-of-line))
-	  (unless sig
 	    (condition-case nil
 		(let ((fill-prefix (when quote (concat quote " ")))
 		      (fill-column (eval fill-flowed-display-column))
@@ -147,7 +152,10 @@
 			       'left 'nosqueeze))
 	      (error
 	       (forward-line 1)
-	       nil))))))))
+	       nil))
+	    (when skiponeline
+	      (forward-line 1)
+	      (setq skiponeline nil))))))))
 
 (provide 'flow-fill)
 

Re: [flow-fill.el] Filling of quoted blank lines

[Ralf Angeli]

* Simon Josefsson (2003-12-04) writes:

> I hope this is slightly more robust, please test it.

Hm, doesn't seem to work correctly.

I am testing it with an article in de.comp.text.tex (MID:
<news:3fcaafbd$0$22267$9b622d9e@news.freenet.de>) which encloses the
following lines:

,----
| >> unterscheidet einen "Heinrich Paul Mueller (H.P.Mueller)" vom
| >> "Hans-Peter Mueller (H.-P. Mueller").
| > 
| > Wenn das in der neuesten verfügbaren Version so ist, wäre wohl ein
| > Bug-Report an den Autor des custom-bib-Pakets angebracht.
`----

If I understand correctly, your code will first search forward until
it finds ` $'.  In the above example point will then be at the end of
the "empty" line.  Then (forward-char 1) is called and point will be
at the beginning of `> Wenn das ...'.  So the following test regarding
`skiponeline' will never see the the "empty" line.

There is another article on Gmane I am using for testing.  You can
find it at
<URL:http://article.gmane.org/gmane.linux.debian.ports.powerpc/14615>.

With the new code e.g. the line starting with "... Seriously, folks,
..." in the second paragraph will get filled with the first paragraph.
I haven't looked into it and cannot tell, where the problem is though.

> It also fixes a long-standing problem in that:
>
> foo
> -- 
> bar
>
> would be flowed together, when it shouldn't.

Yes, I saw this as well and it is good that you address it with the
new code because the solution I proposed was even more error prone as
a line ending with `some text> ' is more common than a dash at the
end.

> There really need to be a regression test harness for flow-fill, it is
> difficult to tell what consequences even a minor change would have.

If you want to, I could post some test articles to this thread,
e.g. like the two mentioned above.  I just have to figure out how I
can add `format=flowed' to the header and what test cases would be
needed exactly.

Regards,
Ralf

-- 
 Moo!
     `~~_.
______',,/______ No gnus is bad news.

Re: [flow-fill.el] Filling of quoted blank lines

[Simon Josefsson]

Ralf Angeli <dev.null@iwi.uni-sb.de> writes:

> * Simon Josefsson (2003-12-04) writes:
>
>> I hope this is slightly more robust, please test it.
>
> Hm, doesn't seem to work correctly.

I noticed it too.

>> There really need to be a regression test harness for flow-fill, it is
>> difficult to tell what consequences even a minor change would have.
>
> If you want to, I could post some test articles to this thread,
> e.g. like the two mentioned above.  I just have to figure out how I
> can add `format=flowed' to the header and what test cases would be
> needed exactly.

Write <#part format=flowed> on top of the article.  But please send
the messages to me privately, preferably as a digest.

I think it is best to first create a test harness for format=flowed,
and then start to make changes.  I can add your test articles to the
test harness.  I'm not installing the patch in the previous messages.

Re: [flow-fill.el] Filling of quoted blank lines

[Ralf Angeli]

* Ralf Angeli (2003-12-02) writes:

> there are quite some people out there using Mozilla with
> `format=flowed' enabled.  This is handled nicely by `flow-fill.el'
> except when it comes to quoted blank lines.  Mozilla is quoting them
> with `> \n' where `flow-fill.el' sees soft line breaks at the end. The
> lines are filled and disappear.  You can have a look at an example in
> comp.text.tex; the MID is <news:bqia0o$2tm$02$1@news.t-online.com>.
> Compare the display of the message using `C-u g' and `g'.  In the
> latter case some lines are missing.
>
> Is this the intended behaviour?  I find it a little bit irritating and
> sometimes don't even want to answer such posts as I am afraid that
> Gnus messes up the text.

I'd like to revitalize this rather old thread as I am occasionally
still seeing this.  An example were Gnus really messes up the display
is [1].  The raw message looks like this:

,----
| Frank Küster wrote:
| > "Tino Langer" <tino.langer@tournex.de> schrieb:
| > 
| > 
| >>Wie heißt denn der Zähler für die Tabellen - ich habe das leider nicht
| >>gefunden.
| > 
| > 
| > Damit kannst du höchstens das Symptom kurieren, aber nicht das Problem
| > lösen. Und früher oder später treffen dich dann andere Symptome des
| > gleichen Problems.
| 
| natürlich löst er damit das longtable-Problem, so es das ist.
| 
| Herbert
`----

And the result displayed by Gnus looks like this:

,----
| Frank Küster wrote:
| > "Tino Langer" <tino.langer@tournex.de> schrieb:
| > 
| >>Wie heißt denn der Zähler für die Tabellen - ich habe das leider nicht
| >>gefunden.
| > Damit kannst du höchstens das Symptom kurieren, aber nicht das
| > Problem
| > lösen. Und früher oder später treffen dich dann andere Symptome des
| > gleichen Problems.
| 
| natürlich löst er damit das longtable-Problem, so es das ist.
| 
| Herbert
`----

The original message was produced by Mozilla.  I managed to find a bug
description[2] in Mozilla's bug management system where the developers
express their views upon this kind of quoting, i.e. their
interpretation of RFC2646.  The important note is comment #5 made by
Ben Bucksch:

,----
| I would interpret
| <quote src="ftp://venera.isi.edu/in-notes/rfc2646.txt">
| 4.4.   Space-Stuffing
| [...]
|    On reception, if the
|    first character of a line is a space, it is logically deleted.  This
|    occurs after the test for a quoted line, and before the test for a
|    flowed line.
|    [...]
|    Other lines MAY be space-stuffed as desired.
| [...]
| 4.5.  Quoting
| [...]
|    Note that because of space-stuffing, the lines
|        >> Exit, Stage Left
|    and
|        >>Exit, Stage Left
|    are semantically identical; both have a quote-depth of two, and a
|    content of "Exit, Stage Left".
| </quote>
| in the way that ">> " is a space-stuffed, quoted and empty line and thus not
| flowed.
`----

I am sharing this view.

As it is currently a bit annoying that `flow-fill.el' doesn't behave
like this I thought about an option to disable it completely as a
quick fix.

If I understand it correctly this could be done by adding another
check to `mm-view.el' (mm-inline-text) in the `when' statement just
before `fill-flowed' is called (line 348).  But this would need a new
defcustom or defvar. (First I thought about abusing
`fill-flowed-encode-column' but it doesn't seem suitable due to its
intention mentioned in the doc string and because it is not available
before `flow-fill.el' is loaded.)

Regards,
Ralf

Footnotes:

[1] <URL:http://groups.google.com/groups?as_umsgid=%3Cbu3nuh%24g2d%2407%241@news.t-online.com%3E>

[2] <URL:http://bugzilla.mozilla.org/show_bug.cgi?id=43455>

-- 
 Moo!
     `~~_.
______',,/______ No Gnus is good news.

Re: [flow-fill.el] Filling of quoted blank lines

[Ralf Angeli]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

* Ralf Angeli (2004-01-19) writes:

> * Ralf Angeli (2003-12-02) writes:
>
>> there are quite some people out there using Mozilla with
>> `format=flowed' enabled.  This is handled nicely by `flow-fill.el'
>> except when it comes to quoted blank lines.  Mozilla is quoting them
>> with `> \n' where `flow-fill.el' sees soft line breaks at the end. The
>> lines are filled and disappear.  You can have a look at an example in
>> comp.text.tex; the MID is <news:bqia0o$2tm$02$1@news.t-online.com>.
>> Compare the display of the message using `C-u g' and `g'.  In the
>> latter case some lines are missing.
>>
>> Is this the intended behaviour?  I find it a little bit irritating and
>> sometimes don't even want to answer such posts as I am afraid that
>> Gnus messes up the text.

[...]

> As it is currently a bit annoying that `flow-fill.el' doesn't behave
> like this I thought about an option to disable it completely as a
> quick fix.

Here is a tiny patch which adds this functionality:


[-- Attachment #2: mm-view.diff --]
[-- Type: text/plain, Size: 906 bytes --]

Index: mm-view.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/mm-view.el,v
retrieving revision 7.4
diff -u -r7.4 mm-view.el
--- mm-view.el	15 Jan 2004 10:59:39 -0000	7.4
+++ mm-view.el	27 Jan 2004 14:35:06 -0000
@@ -64,6 +64,12 @@
     (html2text  html2text))
   "The attributes of washer types for text/html.")
 
+(defcustom mm-fill-flowed t
+  "If non-nil an format=flowed article will be displayed flowed."
+  :type 'boolean
+  :group 'mime-display)
+
+
 ;;; Internal variables.
 
 ;;;
@@ -345,7 +351,8 @@
 	  (mm-insert-part handle)
 	  (goto-char (point-max)))
       (insert (mm-decode-string (mm-get-part handle) charset)))
-    (when (and (equal type "plain")
+    (when (and mm-fill-flowed
+               (equal type "plain")
 	       (equal (cdr (assoc 'format (mm-handle-type handle)))
 		      "flowed"))
       (save-restriction

[-- Attachment #3: Type: text/plain, Size: 10 bytes --]


-- 
Ralf

Re: [flow-fill.el] Filling of quoted blank lines

[Simon Josefsson]

Ralf Angeli <dev.null@iwi.uni-sb.de> writes:

>> As it is currently a bit annoying that `flow-fill.el' doesn't behave
>> like this I thought about an option to disable it completely as a
>> quick fix.
>
> Here is a tiny patch which adds this functionality:

Added, thanks!

Re: [flow-fill.el] Filling of quoted blank lines

[Ralf Angeli]

* Simon Josefsson (2004-01-27) writes:

> Ralf Angeli <dev.null@iwi.uni-sb.de> writes:
>
>>> As it is currently a bit annoying that `flow-fill.el' doesn't behave
>>> like this I thought about an option to disable it completely as a
>>> quick fix.
>>
>> Here is a tiny patch which adds this functionality:
>
> Added, thanks!

Thank you, too.  I just saw a small typo in the docstring.  It is
supposed to read "a format=flowed article" and not "an format=...".
And you can use the email address `angeli@iwi...'.  The
`dev.null@iwi...'  address literally gets routed to /dev/null.  Just
in case this is relevant ...

-- 
Ralf

Re: [flow-fill.el] Filling of quoted blank lines

[Simon Josefsson]

Ralf Angeli <dev.null@iwi.uni-sb.de> writes:

> * Simon Josefsson (2004-01-27) writes:
>
>> Ralf Angeli <dev.null@iwi.uni-sb.de> writes:
>>
>>>> As it is currently a bit annoying that `flow-fill.el' doesn't behave
>>>> like this I thought about an option to disable it completely as a
>>>> quick fix.
>>>
>>> Here is a tiny patch which adds this functionality:
>>
>> Added, thanks!
>
> Thank you, too.  I just saw a small typo in the docstring.  It is
> supposed to read "a format=flowed article" and not "an format=...".
> And you can use the email address `angeli@iwi...'.  The
> `dev.null@iwi...'  address literally gets routed to /dev/null.  Just
> in case this is relevant ...

Done.  Thanks again.