mail-header-separator starting with whitespace fails

mail-header-separator starting with whitespace fails

[Russ Allbery]

Hello all,

I just upgraded to Emacs 24 from Emacs 23, with the corresponding change
in Gnus versions.  For the most part this went smoothly (thank you!).  But
there was one rather annoying hiccup that caused some problems, and I
thought I'd mention it somewhere so that people could consider whether
it's worth fixing.

I've never liked the default mail-header-separator in Emacs.  I think it
blends in with the header and the start of the body text.  So for years
I've used:

    (setq mail-header-separator (concat (make-string 39 ? ) "."))

to create a separator that's much less obtrusive.

Unfortunately, when upgrading from Emacs 23 to Emacs 24, something in Gnus
or in the underneath modes changed so that this mail-header-separator no
longer works.  It's appended to the Content-Type header (causing message
corruption if it has a charset setting), and then the first line of the
body is lifted into the headers and disappears.

Removing this setting fixes the problem, but brings back the default
message separator.

I did a little bit of poking around, and it looks like something inside
the Gnus message code isn't actually using mail-header-separator but
instead is looking for the first line in the buffer that doesn't look like
a header.  Since my message separator looked like a continuation line, it
skipped right past it and kept going.  This seems surprising to me.  I
think whatever is in mail-header-separator should always work, even if it
looks like a header.

-- 
Russ Allbery (eagle@eyrie.org)              <http://www.eyrie.org/~eagle/>

Re: mail-header-separator starting with whitespace fails

[Katsumi Yamaoka]

On Tue, 13 May 2014 09:31:04 -0700, Russ Allbery wrote:
> (setq mail-header-separator (concat (make-string 39 ? ) "."))

I tried this, confirmed it causes a mail corruption, and found
at least one cause.  Could you try replacing the function definition
of `rfc822-goto-eoh' with this?  (I.e., eval this form?)

--8<---------------cut here---------------start------------->8---
(defun rfc822-goto-eoh ()
  "If the buffer starts with a mail header, move point to the header's end.
Otherwise, moves to `point-min'.
The end of the header is the start of the next line, if there is one,
else the end of the last line.  This function obeys RFC822."
  (goto-char (point-min))
  (when (re-search-forward
	 (concat "^" (regexp-quote mail-header-separator) "\n\\|"
		 "^\\([:\n]\\|[^: \t\n]+[ \t\n]\\)")
	 nil 'move)
    (goto-char (match-beginning 0))))
--8<---------------cut here---------------end--------------->8---

This function is used by `smtpmail-send-it' by way of
`mail-sendmail-undelimit-header' to delete the separator, but
the original one doesn't respect `mail-header-separator'.

But I'm not quite sure that it is a cause of your problem, since
those functions seem not to have been modified for years.  Didn't
you use a separator that does not begin with a SPC, like this?

.                                       .

Re: mail-header-separator starting with whitespace fails

[Russ Allbery]

Katsumi Yamaoka <yamaoka@jpl.org> writes:
> On Tue, 13 May 2014 09:31:04 -0700, Russ Allbery wrote:

>> (setq mail-header-separator (concat (make-string 39 ? ) "."))

> I tried this, confirmed it causes a mail corruption, and found
> at least one cause.  Could you try replacing the function definition
> of `rfc822-goto-eoh' with this?  (I.e., eval this form?)

> (defun rfc822-goto-eoh ()
>   "If the buffer starts with a mail header, move point to the header's end.
> Otherwise, moves to `point-min'.
> The end of the header is the start of the next line, if there is one,
> else the end of the last line.  This function obeys RFC822."
>   (goto-char (point-min))
>   (when (re-search-forward
> 	 (concat "^" (regexp-quote mail-header-separator) "\n\\|"
> 		 "^\\([:\n]\\|[^: \t\n]+[ \t\n]\\)")
> 	 nil 'move)
>     (goto-char (match-beginning 0))))

> This function is used by `smtpmail-send-it' by way of
> `mail-sendmail-undelimit-header' to delete the separator, but
> the original one doesn't respect `mail-header-separator'.

Sorry about the delay in verifying this.  That change does indeed fix my
problem.  Thank you!

> But I'm not quite sure that it is a cause of your problem, since
> those functions seem not to have been modified for years.  Didn't
> you use a separator that does not begin with a SPC, like this?

> .                                       .

No, the mail-header-separator setting I was using had been unchanged since
I started using Gnus back in 1994.

I was previously using the Gnus included in Emacs 23.4.  Something must
have changed from that version to Emacs 24.3 that caused this corruption,
although I'm not sure what.

-- 
Russ Allbery (eagle@eyrie.org)              <http://www.eyrie.org/~eagle/>

Re: mail-header-separator starting with whitespace fails

[Lars Ingebrigtsen]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> I tried this, confirmed it causes a mail corruption, and found
> at least one cause.  Could you try replacing the function definition
> of `rfc822-goto-eoh' with this?  (I.e., eval this form?)
>
> (defun rfc822-goto-eoh ()
>   "If the buffer starts with a mail header, move point to the header's end.
> Otherwise, moves to `point-min'.
> The end of the header is the start of the next line, if there is one,
> else the end of the last line.  This function obeys RFC822."
>   (goto-char (point-min))
>   (when (re-search-forward
> 	 (concat "^" (regexp-quote mail-header-separator) "\n\\|"
> 		 "^\\([:\n]\\|[^: \t\n]+[ \t\n]\\)")
> 	 nil 'move)
>     (goto-char (match-beginning 0))))
>
> This function is used by `smtpmail-send-it' by way of
> `mail-sendmail-undelimit-header' to delete the separator, but
> the original one doesn't respect `mail-header-separator'.

The mail header separator should have been stripped before the message
was passed to smtpmail, though, so I think something else must be wrong
somewhere...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: mail-header-separator starting with whitespace fails

[Katsumi Yamaoka]

On Wed, 28 Jan 2015 16:49:25 +1100, Lars Ingebrigtsen wrote:
>> (defun rfc822-goto-eoh ()
[...]
>> This function is used by `smtpmail-send-it' by way of
>> `mail-sendmail-undelimit-header' to delete the separator, but
>> the original one doesn't respect `mail-header-separator'.

> The mail header separator should have been stripped before the message
> was passed to smtpmail, though, so I think something else must be wrong
> somewhere...

Gnus seems to have been not deleting the mail-header-separator
before passing a message to message-send-mail-function for over
ten years.  I've verified it using message-mail of Ma Gnus v0.1,
No Gnus v0.1, and Oort Gnus v0.01 (2001-04-14). ;-)

Re: mail-header-separator starting with whitespace fails

[Lars Ingebrigtsen]

Katsumi Yamaoka <yamaoka@jpl.org> writes:

> Gnus seems to have been not deleting the mail-header-separator
> before passing a message to message-send-mail-function for over
> ten years.  I've verified it using message-mail of Ma Gnus v0.1,
> No Gnus v0.1, and Oort Gnus v0.01 (2001-04-14). ;-)

Heh.

I've now made message.el remove the separator when using smtpmail, too.
That should probably fix the problem the user had.

Should perhaps be backported to emacs-24 if this doesn't lead to any
problems.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: mail-header-separator starting with whitespace fails

[Katsumi Yamaoka]

On Thu, 29 Jan 2015 12:47:30 +1100, Lars Ingebrigtsen wrote:
> I've now made message.el remove the separator when using smtpmail, too.
> That should probably fix the problem the user had.

> Should perhaps be backported to emacs-24 if this doesn't lead to any
> problems.

Thanks.  That shouldn't cause any problem, I believe.  So, I've
merged it to both the trunk and the emacs-24 branch.