Double Newlines

Double Newlines

[David Abrahams]

Everything was working just fine for years, but now many of my
outgoing messages are coming out with an extra blank line after each
line I actually write.  It only seems to happen when I reply to a
message with a "Content-Transfer-Encoding: 8bit" header.

Can anyone offer some small clue as to what I might do to fix the problem?

Several earlier posts about this issue are at:

  http://news.gmane.org/find-root.php?message_id=%3cubqru8k1j.fsf%5f%2d%5f%40boost%2dconsulting.com%3e

Incidentally, this appears not to be the first time I've experienced
something like this

  http://news.gmane.org/find-root.php?message_id=%3c84bs4wt67d.fsf%40crybaby.uni%2dduisburg.de%3e

but I can't for the life of me remember what the actual fix was.  I
currently have no customization of file-coding-system-alist.

Thanks in advance,

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: Double Newlines

[Katsumi Yamaoka]

>>>>> In <uhd16p32o.fsf@boost-consulting.com> David Abrahams wrote:

> Everything was working just fine for years, but now many of my
> outgoing messages are coming out with an extra blank line after each
> line I actually write.  It only seems to happen when I reply to a
> message with a "Content-Transfer-Encoding: 8bit" header.

> Can anyone offer some small clue as to what I might do to fix the problem?

The following code saves in the ~/TESTMAIL file all mails you
send.

(setq message-send-mail-function
      (lambda nil
	(let ((coding-system-for-write 'binary))
	  (write-region (point-min) (point-max) "~/TESTMAIL"))))

What do you see there?  If newlines are doubled already, the
cause might be in the Gnus' encoder.  Otherwise, the program to
send mails or the mail server might do something strange.
Whether or no, I cannot imagine what does it, though.

> Several earlier posts about this issue are at:

>   http://news.gmane.org/find-root.php?message_id=%3cubqru8k1j.fsf%5f%2d%5f%40boost%2dconsulting.com%3e

> Incidentally, this appears not to be the first time I've experienced
> something like this

>   http://news.gmane.org/find-root.php?message_id=%3c84bs4wt67d.fsf%40crybaby.uni%2dduisburg.de%3e

> but I can't for the life of me remember what the actual fix was.  I
> currently have no customization of file-coding-system-alist.

> Thanks in advance,

> --
> Dave Abrahams
> Boost Consulting
> www.boost-consulting.com

Re: Double Newlines

[David Abrahams]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

>>>>>> In <uhd16p32o.fsf@boost-consulting.com> David Abrahams wrote:
>
>> Everything was working just fine for years, but now many of my
>> outgoing messages are coming out with an extra blank line after each
>> line I actually write.  It only seems to happen when I reply to a
>> message with a "Content-Transfer-Encoding: 8bit" header.
>
>> Can anyone offer some small clue as to what I might do to fix the problem?

Yamaoka-san,

Thank you *very* much for your reply...

> The following code saves in the ~/TESTMAIL file all mails you
> send.
>
> (setq message-send-mail-function
>       (lambda nil
> 	(let ((coding-system-for-write 'binary))
> 	  (write-region (point-min) (point-max) "~/TESTMAIL"))))
>
> What do you see there?  

This is a test=0D
This is only a test=0D
If this had been an actual message, we'd tell you.=0D

> If newlines are doubled already, 

I don't know for sure how to interpret the above, but my first guess
is that they're not (right?)

> the cause might be in the Gnus' encoder.  Otherwise, the program to
> send mails or the mail server might do something strange.  Whether
> or no, I cannot imagine what does it, though.

I'm running a local Exim on this machine to send the mail out.  Maybe
I should be looking at that.  On the other hand, this particular
instance of the phenomenon showed up in a newsgroup posting, not a
mail message.  I don't think NG postings are handled like regular
mail, are they?  If not, then they're not going through Exim and Gnus
might still be involved.  Does message-send-mail-function get used for
NG postings, or just email?  If it's just for email, then our TESTMAIL
test doesn't tell us very much.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: Double Newlines

[Katsumi Yamaoka]

>>>>> In <ubqrdoizi.fsf@boost-consulting.com> David Abrahams wrote:

>> (setq message-send-mail-function
>>       (lambda nil
>> 	(let ((coding-system-for-write 'binary))
>> 	  (write-region (point-min) (point-max) "~/TESTMAIL"))))
>>
>> What do you see there?

> This is a test=0D
> This is only a test=0D
> If this had been an actual message, we'd tell you.=0D

>> If newlines are doubled already,

> I don't know for sure how to interpret the above, but my first guess
> is that they're not (right?)

That looks strange to me.  `=0D' is the one that the CR character
(0x0d) was encoded by quoted-printable.  What does the following
Lisp form return? (To do that, copy it to the *scratch* buffer
and type `C-j' at the end of the last line.) [1]

(let ((file (make-temp-file "testing"))
      (coding-system-for-write 'iso-8859-1)
      (coding-system-for-read 'binary))
  (with-temp-file file
    (insert "foo\n"))
  (prog1
      (with-temp-buffer
	(insert-file-contents file)
	(buffer-string))
    (delete-file file)))

If you see

"foo^M
"
, your Emacs 22 has a bug.  Since Kenichi Handa fixed the bug[2]
recently[3], updating your Emacs 22 might fix the problem[4].

[1] It might probably be able to reproduce simply by the form:

(encode-coding-string "foo\n" 'iso-8859-1)

[2] Under the system in which the system_eol_type is CRLF (i.e.,
Windows), Emacs 22 encodes text by the coding system FOO as if
it is FOO-dos.

[3] (lisp/ChangeLog)
2006-07-20  Kenichi Handa  <handa@m17n.org>

	* international/mule-cmds.el (select-safe-coding-system): Fix the
	way of deciding eol-type of the coding system.

[4] I'm not quite sure that `=0D' is decoded into a newline.

>> the cause might be in the Gnus' encoder.  Otherwise, the program to
>> send mails or the mail server might do something strange.  Whether
>> or no, I cannot imagine what does it, though.

> I'm running a local Exim on this machine to send the mail out.  Maybe
> I should be looking at that.  On the other hand, this particular
> instance of the phenomenon showed up in a newsgroup posting, not a
> mail message.  I don't think NG postings are handled like regular
> mail, are they?  If not, then they're not going through Exim and Gnus
> might still be involved.  Does message-send-mail-function get used for
> NG postings, or just email?  If it's just for email, then our TESTMAIL
> test doesn't tell us very much.

If the problem is concerned with the charset encoding, it will
happen with both mail and news.

Re: Double Newlines

[David Abrahams]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

>>>>>> In <ubqrdoizi.fsf@boost-consulting.com> David Abrahams wrote:
>
>>> (setq message-send-mail-function
>>>       (lambda nil
>>> 	(let ((coding-system-for-write 'binary))
>>> 	  (write-region (point-min) (point-max) "~/TESTMAIL"))))
>>>
>>> What do you see there?
>
>> This is a test=0D
>> This is only a test=0D
>> If this had been an actual message, we'd tell you.=0D
>
>>> If newlines are doubled already,
>
>> I don't know for sure how to interpret the above, but my first guess
>> is that they're not (right?)
>
> That looks strange to me.  `=0D' is the one that the CR character
> (0x0d) was encoded by quoted-printable.  What does the following
> Lisp form return? (To do that, copy it to the *scratch* buffer
> and type `C-j' at the end of the last line.) [1]
>
> (let ((file (make-temp-file "testing"))
>       (coding-system-for-write 'iso-8859-1)
>       (coding-system-for-read 'binary))
>   (with-temp-file file
>     (insert "foo\n"))
>   (prog1
>       (with-temp-buffer
> 	(insert-file-contents file)
> 	(buffer-string))
>     (delete-file file)))
>
> If you see
>
> "foo^M
> "
> , your Emacs 22 has a bug.  

That's what I see.

> Since Kenichi Handa fixed the bug[2]
> recently[3], updating your Emacs 22 might fix the problem[4].

Okay, that's my next step, thanks.

> [1] It might probably be able to reproduce simply by the form:
>
> (encode-coding-string "foo\n" 'iso-8859-1)

yep.

> [2] Under the system in which the system_eol_type is CRLF (i.e.,
> Windows), Emacs 22 encodes text by the coding system FOO as if
> it is FOO-dos.
>
> [3] (lisp/ChangeLog)
> 2006-07-20  Kenichi Handa  <handa@m17n.org>
>
> 	* international/mule-cmds.el (select-safe-coding-system): Fix the
> 	way of deciding eol-type of the coding system.
>
> [4] I'm not quite sure that `=0D' is decoded into a newline.
>
>>> the cause might be in the Gnus' encoder.  Otherwise, the program to
>>> send mails or the mail server might do something strange.  Whether
>>> or no, I cannot imagine what does it, though.
>
>> I'm running a local Exim on this machine to send the mail out.  Maybe
>> I should be looking at that.  On the other hand, this particular
>> instance of the phenomenon showed up in a newsgroup posting, not a
>> mail message.  I don't think NG postings are handled like regular
>> mail, are they?  If not, then they're not going through Exim and Gnus
>> might still be involved.  Does message-send-mail-function get used for
>> NG postings, or just email?  If it's just for email, then our TESTMAIL
>> test doesn't tell us very much.
>
> If the problem is concerned with the charset encoding, it will
> happen with both mail and news.

Thanks, I'll try rebuilding my emacs now.


-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: Double Newlines

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 1645 bytes --]

Hi,

I don't know whether it is related to David Abrahams' problem,
but I heard that the most recent Emacs 22 built on Windows
behaves unlike the old ones.

When saving a file in the case where `coding-system-for-write'
is bound to the coding system `raw-text' for example, CRLF is
used as the line break code.  It is said that it happens as well
when communicating with an external process.

Because the default value of `nnmail-file-coding-system' and
`nnmail-active-file-coding-system' is `raw-text', the nnml back
end saves mail files, NOV files and the ACTIVE file as if the
coding-system is set to `raw-text-dos'.  Hideyuki SHIRAI said
that LF was used for the line break code formerly.  It might be
generally a good idea to use CRLF when saving files on Windows,
though.

You might not seem that it causes a problem since Gnus reads
those files using `raw-text' and CRLF is converted into LF again.
However, OHASHI Akira reported that a problem occurs actually.
He tried setting `gnus-nov-is-evil' to t and saw all summary
lines in a nnml group appeared as:

   [   ?: nobody                 ] (none)

I guess it always happens in nnmh groups.  Could anyone who uses
the most recent Emacs 22 built on Windows verify it?

The cause is that the `nnheader-insert-head' function doesn't
work properly.  In that function, it is expected that
`(insert-file-contents FILE nil BEG END)' returns `(FILE CHARS)',
where CHARS is the number in which BEG is subtracted from END.
However, it fails because of the CRLF-to-LF conversion.

I don't use Emacs on Windows, so I hope someone follows it up.
Here's a solution, though I haven't tested it yet.


[-- Attachment #2: Type: application/emacs-lisp, Size: 992 bytes --]

[-- Attachment #3: Type: text/plain, Size: 10 bytes --]


Regards,

Re: Double Newlines

[David Abrahams]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

>>>>>> In <ubqrdoizi.fsf@boost-consulting.com> David Abrahams wrote:
>
>>> (setq message-send-mail-function
>>>       (lambda nil
>>> 	(let ((coding-system-for-write 'binary))
>>> 	  (write-region (point-min) (point-max) "~/TESTMAIL"))))
>>>
>>> What do you see there?
>
>> This is a test=0D
>> This is only a test=0D
>> If this had been an actual message, we'd tell you.=0D
>
>>> If newlines are doubled already,
>
>> I don't know for sure how to interpret the above, but my first guess
>> is that they're not (right?)
>
> That looks strange to me.  `=0D' is the one that the CR character
> (0x0d) was encoded by quoted-printable.  

Rebuilding emacs made that behavior go away (thank you!)...

> What does the following
> Lisp form return? (To do that, copy it to the *scratch* buffer
> and type `C-j' at the end of the last line.) [1]
>
> (let ((file (make-temp-file "testing"))
>       (coding-system-for-write 'iso-8859-1)
>       (coding-system-for-read 'binary))
>   (with-temp-file file
>     (insert "foo\n"))
>   (prog1
>       (with-temp-buffer
> 	(insert-file-contents file)
> 	(buffer-string))
>     (delete-file file)))
>
> If you see
>
> "foo^M
> "
> , your Emacs 22 has a bug.  Since Kenichi Handa fixed the bug[2]
> recently[3], updating your Emacs 22 might fix the problem[4].

...but curiously, _that_ behavior remains!

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: Double Newlines

[Katsumi Yamaoka]

>>>>> In <u7j20qb6w.fsf@boost-consulting.com> David Abrahams wrote:

> Rebuilding emacs made that behavior go away (thank you!)...

Great.

>> What does the following
>> Lisp form return? (To do that, copy it to the *scratch* buffer
>> and type `C-j' at the end of the last line.) [1]
>>
>> (let ((file (make-temp-file "testing"))
>>       (coding-system-for-write 'iso-8859-1)
>>       (coding-system-for-read 'binary))
>>   (with-temp-file file
>>     (insert "foo\n"))
>>   (prog1
>>       (with-temp-buffer
>> 	(insert-file-contents file)
>> 	(buffer-string))
>>     (delete-file file)))
>>
>> If you see
>>
>> "foo^M
>> "
>> , your Emacs 22 has a bug.  Since Kenichi Handa fixed the bug[2]
>> recently[3], updating your Emacs 22 might fix the problem[4].

> ...but curiously, _that_ behavior remains!

Yes, it does.  Hideyuki SHIRAI who uses Windows told me
yesterday that Kenichi Handa fixed only the following problem:

(encode-coding-string "foo\n" 'iso-8859-1)
"foo^M
"
Fixing this is enough to solve your problem, I think.  OTOH,
that the result of the above "testing" code contains ^M seems to
be a spec of the most recent Emacs 22 which runs on Windows.  I
don't know whether it is good or bad, though.  I posted a report
about the possibility that it is harmful to Gnus[1].

[1] <b4m3bcotw52.fsf@jpl.org> or
http://article.gmane.org/gmane.emacs.gnus.general/63502

Re: Double Newlines

[Katsumi Yamaoka]

>>>>> In <b4m3bcotw52.fsf@jpl.org> Katsumi Yamaoka wrote:

> However, OHASHI Akira reported that a problem occurs actually.
> He tried setting `gnus-nov-is-evil' to t and saw all summary
> lines in a nnml group appeared as:

>    [   ?: nobody                 ] (none)

> I guess it always happens in nnmh groups.

OHASHI-san confirmed it surely happens in nnmh groups with the
recent Emacs 22 which runs on Windows.  He also verified
improving of the `nnheader-insert-head' function does the trick.

> (defun nnheader-insert-head (file)

I slightly modified and installed it to both the trunk and the
v5-10 branch.

Re: Double Newlines

[Reiner Steib]

On Thu, Jul 27 2006, Katsumi Yamaoka wrote:

> Yes, it does.  Hideyuki SHIRAI who uses Windows told me
> yesterday that Kenichi Handa fixed only the following problem:
>
> (encode-coding-string "foo\n" 'iso-8859-1)
> "foo^M
> "
> Fixing this is enough to solve your problem, I think.  OTOH,
> that the result of the above "testing" code contains ^M seems to
> be a spec of the most recent Emacs 22 which runs on Windows.  I
> don't know whether it is good or bad, though.  I posted a report
> about the possibility that it is harmful to Gnus[1].
>
> [1] <b4m3bcotw52.fsf@jpl.org> or
> http://article.gmane.org/gmane.emacs.gnus.general/63502

I didn't follow this discussion in detail, but shouldn't this problem
be reported to the Emacs developers?  I don't think that Kenichi Handa
uses Windows so he might not be aware of the problem.  Is the changed
behavior documented e.g. in etc/NEWS?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Double Newlines

[Katsumi Yamaoka]

>>>>> In <v9ejw7udsj.fsf@marauder.physik.uni-ulm.de>
>>>>>	Reiner Steib wrote:

> On Thu, Jul 27 2006, Katsumi Yamaoka wrote:

[...]

>> [1] <b4m3bcotw52.fsf@jpl.org> or
>> http://article.gmane.org/gmane.emacs.gnus.general/63502

> I didn't follow this discussion in detail, but shouldn't this problem
> be reported to the Emacs developers?  I don't think that Kenichi Handa
> uses Windows so he might not be aware of the problem.  Is the changed
> behavior documented e.g. in etc/NEWS?

Once OHASHI Akira reported to the mule-ja mailing list the
problem caused by the difference between the recent Emacs 22 on
Windows and the old one, but Kenichi Handa hasn't responded yet
so far.  I'll repeat it again.  Thanks.

Re: Double Newlines

[Katsumi Yamaoka]

>>>>> In <b4m8xmfflcp.fsf@jpl.org> Katsumi Yamaoka wrote:

>> (defun nnheader-insert-head (file)

> I slightly modified and installed it to both the trunk and the
> v5-10 branch.

I forgot to make it work with Mac as well as Windows.  Fixed.