Trailing whitespace and PGP/MIME

Trailing whitespace and PGP/MIME

[Jesper Harder]

[-- Attachment #1: Type: text/plain, Size: 1041 bytes --]

Currently Gnus violates this requirement in RFC 3156 (MIME Security
with OpenPGP):

   implementations MUST make sure that no trailing whitespace is
   present after the MIME encoding has been applied.

Fixing it the obvious way would, however, break the intention of this
change:

2003-10-27  Simon Josefsson  <jas@extundo.com>

	* mm-bodies.el (mm-body-encoding): Don't use QP when message body
	only consists of short lines and ASCII, when
	mm-use-ultra-safe-encoding.  Refer to 'About foo' thread in
	gnus-bug, e.g. <ilullrg4k7p.fsf@extundo.com>, for more discussion.
	This make it possible to pipe the raw RFC 822 message into 'gpg'
	and have the signature work.  Potential problem: what if message
	contain data that would be dash-escaped by OpenPGP
	implementations? Then PGP 2.x might not be able to parse the raw
	RFC 822 message correctly.  If that problem is worth fixing, it
	should be fixed by detecting the situation, instead of applying QP
	to everything.  Based on discussion with "John A. Martin"
	<jam@jamux.com>.

-- 
foo

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]

Re: Trailing whitespace and PGP/MIME

[Ivan Boldyrev]

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]

On 8608 day of my life Jesper Harder wrote:
> Currently Gnus violates this requirement in RFC 3156 (MIME Security
> with OpenPGP):
>
>    implementations MUST make sure that no trailing whitespace is
>    present after the MIME encoding has been applied.

Note: RFC 3156 describes PGP/MIME only.

> Fixing it the obvious way would, however, break the intention of this
> change:
>
> 2003-10-27  Simon Josefsson  <jas@extundo.com>
>
> 	* mm-bodies.el (mm-body-encoding): Don't use QP when message body
> 	only consists of short lines and ASCII, when
> 	mm-use-ultra-safe-encoding.  Refer to 'About foo' thread in
> 	gnus-bug, e.g. <ilullrg4k7p.fsf@extundo.com>, for more discussion.
> 	This make it possible to pipe the raw RFC 822 message into 'gpg'
> 	and have the signature work.

It's true only for Plain PGP.  PGP/MIME uses detached signatures, that
cannot be verified with piping.

My opinion is that whitespaces must be removed from PGP/MIME-encoded
messages but not from Plain PGP-encoded ones.   But it is current
state of Gnus, isn't it?

BTW, is gnus-bug archived somewhere?

-- 
Ivan Boldyrev

                        Today is the first day of the rest of your life.

[-- Attachment #2: Type: application/pgp-signature, Size: 480 bytes --]

Re: Trailing whitespace and PGP/MIME

[Simon Josefsson]

Jesper Harder <harder@ifa.au.dk> writes:

> Currently Gnus violates this requirement in RFC 3156 (MIME Security
> with OpenPGP):
>
>    implementations MUST make sure that no trailing whitespace is
>    present after the MIME encoding has been applied.
>
> Fixing it the obvious way would, however, break the intention of this
> change:
>
> 2003-10-27  Simon Josefsson  <jas@extundo.com>
>
> 	* mm-bodies.el (mm-body-encoding): Don't use QP when message body
> 	only consists of short lines and ASCII, when
> 	mm-use-ultra-safe-encoding.  Refer to 'About foo' thread in
> 	gnus-bug, e.g. <ilullrg4k7p.fsf@extundo.com>, for more discussion.
> 	This make it possible to pipe the raw RFC 822 message into 'gpg'
> 	and have the signature work.  Potential problem: what if message
> 	contain data that would be dash-escaped by OpenPGP
> 	implementations? Then PGP 2.x might not be able to parse the raw
> 	RFC 822 message correctly.  If that problem is worth fixing, it
> 	should be fixed by detecting the situation, instead of applying QP
> 	to everything.  Based on discussion with "John A. Martin"
> 	<jam@jamux.com>.

I think detecting if there are trailing SPC, and using QP in that case
is reasonable.  Does this work?

--- mm-bodies.el.~6.29.~	2003-12-26 06:32:11.000000000 +0100
+++ mm-bodies.el	2003-12-30 12:05:33.000000000 +0100
@@ -132,7 +132,8 @@
     (cond
      ((and (not longp)
 	   (not (and mm-use-ultra-safe-encoding
-		     (save-excursion (re-search-forward "^From " nil t))))
+		     (save-excursion (or (re-search-forward "^From " nil t)
+					 (re-search-forward " $" nil t)))))
 	   (eq bits '7bit))
       bits)
      ((and (not mm-use-ultra-safe-encoding)

Re: Trailing whitespace and PGP/MIME

[Simon Josefsson]

Ivan Boldyrev <boldyrev+nospam@cgitftp.uiggm.nsc.ru> writes:

>> 2003-10-27  Simon Josefsson  <jas@extundo.com>
>>
>> 	* mm-bodies.el (mm-body-encoding): Don't use QP when message body
>> 	only consists of short lines and ASCII, when
>> 	mm-use-ultra-safe-encoding.  Refer to 'About foo' thread in
>> 	gnus-bug, e.g. <ilullrg4k7p.fsf@extundo.com>, for more discussion.
>> 	This make it possible to pipe the raw RFC 822 message into 'gpg'
>> 	and have the signature work.
>
> It's true only for Plain PGP.

Yes, that was the intent.

> PGP/MIME uses detached signatures, that cannot be verified with
> piping.

Well... technically, it can be verified by piping after glueing the
detached signature together with the data.  I believe Gnus uses this
if you use Mailcrypt as the backend, because mailcrypt can't verify
detached signatures.

> My opinion is that whitespaces must be removed from PGP/MIME-encoded
> messages but not from Plain PGP-encoded ones.

I agree for PGP/MIME, and probably for plain PGP as well (although the
number of problems with plain PGP and CTE's have almost made me
consider it a lost cause).

> But it is current state of Gnus, isn't it?

I don't think so, I was able to reproduce the problem with PGP/MIME
too.  However, this patch should fix it.

--- mm-bodies.el.~6.29.~	2003-12-26 06:32:11.000000000 +0100
+++ mm-bodies.el	2003-12-30 12:26:57.000000000 +0100
@@ -132,7 +132,8 @@
     (cond
      ((and (not longp)
 	   (not (and mm-use-ultra-safe-encoding
-		     (save-excursion (re-search-forward "^From " nil t))))
+		     (save-excursion (or (re-search-forward "^From " nil t)
+					 (re-search-forward " $" nil t)))))
 	   (eq bits '7bit))
       bits)
      ((and (not mm-use-ultra-safe-encoding)

> BTW, is gnus-bug archived somewhere?

See <nntp://quimby.gnus.org/gnus.gnus-bug>.

Re: Trailing whitespace and PGP/MIME

[Jesper Harder]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Simon Josefsson <jas@extundo.com> writes:

> Jesper Harder <harder@ifa.au.dk> writes:
>
>>    implementations MUST make sure that no trailing whitespace is
>>    present after the MIME encoding has been applied.
>>
>> Fixing it the obvious way would, however, break the intention of this
>> change:
>>
>> 2003-10-27  Simon Josefsson  <jas@extundo.com>
>>
>> 	* mm-bodies.el (mm-body-encoding): Don't use QP when message body
>> 	only consists of short lines and ASCII, when
>> 	mm-use-ultra-safe-encoding.
>
> I think detecting if there are trailing SPC, and using QP in that case
> is reasonable.  Does this work?
>
> -		     (save-excursion (re-search-forward "^From " nil t))))
> +		     (save-excursion (or (re-search-forward "^From " nil t)
> +					 (re-search-forward " $" nil t)))))

Yes, but this was what I meant by "the obvious way" :-)

It'll fix pgp/mime, but it will also force QP for inline pgp if you
use a signature -- which for a lot of people means _always_ -- thus
more or less reverting the intention of your previous change.

- -- 
foo
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.0 (GNU/Linux)

iD8DBQE/8Xd9zEN/MEcathkRApd2AKCjjbupG6lNaFYJvE7XofYFhGbEJwCggqrG
Ywl7sIQ0GmFpFu8JGG+3oXY=
=BRl1
-----END PGP SIGNATURE-----

Re: Trailing whitespace and PGP/MIME

[Ivan Boldyrev]

On 8608 day of my life Simon Josefsson wrote:
>> PGP/MIME uses detached signatures, that cannot be verified with
>> piping.
>
> Well... technically, it can be verified by piping after glueing the
> detached signature together with the data.  I believe Gnus uses this
> if you use Mailcrypt as the backend, because mailcrypt can't verify
> detached signatures.

If you do preprocessing such as gluing, you can do MIME decoding as
well :)

>> My opinion is that whitespaces must be removed from PGP/MIME-encoded
>> messages but not from Plain PGP-encoded ones.
>
> I agree for PGP/MIME, and probably for plain PGP as well (although the
> number of problems with plain PGP and CTE's have almost made me
> consider it a lost cause).

Plain PGP is considered harmful :) So it is up to message composer
responsibility to deal with "^From" and " $" in Plain PGP messages.

BTW, some MTA can decode one-part MIME-encoded messages and deliver
them as 8bit, for example.  But multipart messages (and PGP/MIME) are
never decoded by correct MTA.  So, even if we will encode Plain PGP
messages, it will be probably decoded anyway.  So, it is useless.

Plain PGP must be supported, but with note: NO WARRANTY.  THERE IS NO
WARRANTY OF CORRECT DELIVERY OF PLAIN PGP MESSAGES.  USE IT ON YOU OWN
RISK.

>> But it is current state of Gnus, isn't it?
>
> I don't think so, I was able to reproduce the problem with PGP/MIME
> too.  However, this patch should fix it.
>
> --- mm-bodies.el.~6.29.~	2003-12-26 06:32:11.000000000 +0100
> +++ mm-bodies.el	2003-12-30 12:26:57.000000000 +0100
> @@ -132,7 +132,8 @@
>      (cond
>       ((and (not longp)
>  	   (not (and mm-use-ultra-safe-encoding
> -		     (save-excursion (re-search-forward "^From " nil t))))
> +		     (save-excursion (or (re-search-forward "^From " nil t)
> +					 (re-search-forward " $" nil t)))))

It's is better to do encoding always if we use PGP/MIME.  Or even use

          (not mm-use-ultra-safe-encoding)

>> BTW, is gnus-bug archived somewhere?
>
> See <nntp://quimby.gnus.org/gnus.gnus-bug>.

Cool :)

I use leafnode, so I don't see that articles at quimby never expire...

-- 
Ivan Boldyrev

                      У женщины легче получить прощение, чем разрешение.

Re: Trailing whitespace and PGP/MIME

[Simon Josefsson]

Jesper Harder <harder@ifa.au.dk> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Jesper Harder <harder@ifa.au.dk> writes:
>>
>>>    implementations MUST make sure that no trailing whitespace is
>>>    present after the MIME encoding has been applied.
>>>
>>> Fixing it the obvious way would, however, break the intention of this
>>> change:
>>>
>>> 2003-10-27  Simon Josefsson  <jas@extundo.com>
>>>
>>> 	* mm-bodies.el (mm-body-encoding): Don't use QP when message body
>>> 	only consists of short lines and ASCII, when
>>> 	mm-use-ultra-safe-encoding.
>>
>> I think detecting if there are trailing SPC, and using QP in that case
>> is reasonable.  Does this work?
>>
>> -		     (save-excursion (re-search-forward "^From " nil t))))
>> +		     (save-excursion (or (re-search-forward "^From " nil t)
>> +					 (re-search-forward " $" nil t)))))
>
> Yes, but this was what I meant by "the obvious way" :-)

Ah.

> It'll fix pgp/mime, but it will also force QP for inline pgp if you
> use a signature -- which for a lot of people means _always_ -- thus
> more or less reverting the intention of your previous change.

Ah.  Hm.  Er.  So what IS the right thing?  The MUST above is for
PGP/MIME, yes, but the _reason_ the MUST is there in the document is
about as valid for plain PGP as it is for PGP/MIME, I think, arguing
that the obvious approach is the right.

One (non-)solution is to warn the user whenever the message content
may not work reliably with plain PGP and ask the user if she wants to
continue, or use PGP/MIME instead.  There are many things on this list
now, e.g., non-ASCII, trailing unencoded SPC, data that look dash
escaped.  This would be a useful feature, but just move the problem
into the face of the user instead of solving it, and I doubt most
users understand nor want to understand the problem.

Re: Trailing whitespace and PGP/MIME

[Jesper Harder]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Simon Josefsson <jas@extundo.com> writes:

> Jesper Harder <harder@ifa.au.dk> writes:
>
>> It'll fix pgp/mime, but it will also force QP for inline pgp if you
>> use a signature -- which for a lot of people means _always_ -- thus
>> more or less reverting the intention of your previous change.
>
> Ah.  Hm.  Er.  So what IS the right thing?  The MUST above is for
> PGP/MIME, yes, but the _reason_ the MUST is there in the document is
> about as valid for plain PGP as it is for PGP/MIME, I think, arguing
> that the obvious approach is the right.

I'm not sure all the reasons are valid for cleartext signatures.

In RFC 2440 textmode is used for cleartext signatures.  PGP/MIME
allows you to use either a textmode or a binary mode detached
signature.

I think that's why they need the extra restrictions.  Trailing SPC
doesn't matter if you're only using textmode, because it's ignored
when computing the signature.

> One (non-)solution is to warn the user whenever the message content
> may not work reliably with plain PGP and ask the user if she wants to
> continue, or use PGP/MIME instead.

This is similar to Mutt (or at least what Mutt used to do -- my
version is fairly old):

,----
|   6.3.115 pgp_create_traditional
| 
|   Type: quadoption
|   Default: no
| 
|   This option controls whether Mutt generates old-style PGP encrypted or
|   signed messages under certain circumstances.
| 
|   Note that PGP/MIME will be used automatically for messages which have
|   a character set different from us-ascii, or which consist of more than
|   a single MIME part.
| 
|   Also note that using the old-style PGP message format is *strongly
|   depreceated*
`----

> There are many things on this list now, e.g., non-ASCII,

Yup, I agree that inline signatures are unsuitable for non-ASCII.

> trailing unencoded SPC,

I don't think that's a problem.

> data that look dash escaped.

But gpg seems to handle dash-escapes just fine:

- - -

Also, the "PGP sign part" and "PGP encrypt part" commands should
probably be removed, since Gnus itself isn't even able to handle the
result.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.0 (GNU/Linux)

iD8DBQE/8gpCzEN/MEcathkRAsyuAJ0ZYZAW43Vomkvgy1USXy7csgC76wCg0cpN
dUEzVV66ZtZolVLRv6RQ/6I=
=Sl27
-----END PGP SIGNATURE-----

Re: Trailing whitespace and PGP/MIME

[Russ Allbery]

Simon Josefsson <jas@extundo.com> writes:

> Ah.  Hm.  Er.  So what IS the right thing?  The MUST above is for
> PGP/MIME, yes, but the _reason_ the MUST is there in the document is
> about as valid for plain PGP as it is for PGP/MIME, I think, arguing
> that the obvious approach is the right.

It shouldn't be as valid for inline PGP.  PGP implementations generally
handle trailing spaces without difficulty when processing attached
signatures (by stripping them before processing the message).  The
behavior is different than when they process detached signatures, but I'm
not sure that we care.

> One (non-)solution is to warn the user whenever the message content
> may not work reliably with plain PGP and ask the user if she wants to
> continue, or use PGP/MIME instead.

This is going to get really annoying for people who use inline signatures
with a sig delimiter.

The real solution is for people to stop using the broken inline PGP
signature standard and switch to PGP/MIME so that the true structure of
their messages is exposed to software, but it's so hard to convince people
of that....

> There are many things on this list now, e.g., non-ASCII, trailing
> unencoded SPC, data that look dash escaped.  This would be a useful
> feature, but just move the problem into the face of the user instead of
> solving it, and I doubt most users understand nor want to understand the
> problem.

Yup.

-- 
Russ Allbery (rra@stanford.edu)             <http://www.eyrie.org/~eagle/>

Re: Trailing whitespace and PGP/MIME

[Simon Josefsson]

Jesper Harder <harder@ifa.au.dk> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Jesper Harder <harder@ifa.au.dk> writes:
>>
>>> It'll fix pgp/mime, but it will also force QP for inline pgp if you
>>> use a signature -- which for a lot of people means _always_ -- thus
>>> more or less reverting the intention of your previous change.
>>
>> Ah.  Hm.  Er.  So what IS the right thing?  The MUST above is for
>> PGP/MIME, yes, but the _reason_ the MUST is there in the document is
>> about as valid for plain PGP as it is for PGP/MIME, I think, arguing
>> that the obvious approach is the right.
>
> I'm not sure all the reasons are valid for cleartext signatures.
>
> In RFC 2440 textmode is used for cleartext signatures.  PGP/MIME
> allows you to use either a textmode or a binary mode detached
> signature.
>
> I think that's why they need the extra restrictions.  Trailing SPC
> doesn't matter if you're only using textmode, because it's ignored
> when computing the signature.

Yes.  I believe pre-OpenPGP implementations did not ignore trailing
SPC, even in "textmode", though.  I think that's one of the reason
PGP/MIME require that they shouldn't be present -- so that pre-OpenPGP
tools compute the same hash.

One alternative would be for Gnus to require an OpenPGP
implementation.  This would solve some other problems as well, such as
QP of dash escaped text for RFC 1991 compatibility.  But it would mean
dropping support for PGP 2.x, and perhaps some of 5.x/6.x/etc too.  I
don't particularly care about those, but perhaps some do.  Opinions?

> |   6.3.115 pgp_create_traditional

Interesting, thanks for the reference.

>> There are many things on this list now, e.g., non-ASCII,
>
> Yup, I agree that inline signatures are unsuitable for non-ASCII.

However, raw 8-bit with plain PGP can work well.

>> trailing unencoded SPC,
>
> I don't think that's a problem.

See above, it might be.

>> data that look dash escaped.
>
> But gpg seems to handle dash-escapes just fine:
>
> - -

See above, RFC 1991 implementation does not understand how to revert
dash escaped text.  That's why QP encoding could be used for text that
might be dash escaped.  But that doesn't work if QP isn't already used
for other reasons, such as non-ASCII -- reverting to QP just for this
seems excessive.

> Also, the "PGP sign part" and "PGP encrypt part" commands should
> probably be removed, since Gnus itself isn't even able to handle the
> result.

Yes...  ideally they should be fixed to generate inline PGP within
MIME parts, which is what most Outlook PGP users appear to generate
(and parse).

Re: Trailing whitespace and PGP/MIME

[Russ Allbery]

Simon Josefsson <jas@extundo.com> writes:

> Yes.  I believe pre-OpenPGP implementations did not ignore trailing SPC,
> even in "textmode", though.  I think that's one of the reason PGP/MIME
> require that they shouldn't be present -- so that pre-OpenPGP tools
> compute the same hash.

I'm fairly certain that PGP 2.x did, provided that you gave it the -t flag
to force textmode.  This is something I fought with a bunch when writing
PGP::Sign.

> One alternative would be for Gnus to require an OpenPGP implementation.

This sounds like a great idea to me.  I'm not sure why anyone would still
use PGP 2.x voluntarily rather than GnuPG or the current PGP version on
Windows these days, given that GnuPG can now generate RSA signatures if
you just have to have them.

-- 
Russ Allbery (rra@stanford.edu)             <http://www.eyrie.org/~eagle/>

Re: Trailing whitespace and PGP/MIME

[Simon Josefsson]

Russ Allbery <rra@stanford.edu> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Ah.  Hm.  Er.  So what IS the right thing?  The MUST above is for
>> PGP/MIME, yes, but the _reason_ the MUST is there in the document is
>> about as valid for plain PGP as it is for PGP/MIME, I think, arguing
>> that the obvious approach is the right.
>
> It shouldn't be as valid for inline PGP.  PGP implementations generally
> handle trailing spaces without difficulty when processing attached
> signatures (by stripping them before processing the message).

Even pre-OpenPGP implementations?  E.g., PGP 2.x.  RFC 1991 does not
mention "whitespace", "trailing", or "strip".  Gnus currently claim to
support them, although I'm not sure this is a good idea, nor if anyone
actually uses PGP 2.x.

>> One (non-)solution is to warn the user whenever the message content
>> may not work reliably with plain PGP and ask the user if she wants to
>> continue, or use PGP/MIME instead.
>
> This is going to get really annoying for people who use inline signatures
> with a sig delimiter.
>
> The real solution is for people to stop using the broken inline PGP
> signature standard and switch to PGP/MIME so that the true structure of
> their messages is exposed to software, but it's so hard to convince people
> of that....

So maybe an annoying blurb showing up every time users try to use
inline PGP with non-ASCII, trailing whitespace, or dash escaped text,
is the best we can hope to do.  If we annoy people with the blurb,
maybe they stop annoying us by using inline PGP.

For pure-ASCII, no trailing whitespace, no possibly dash escaped text,
use of inline isn't that bad, especially for announcements stored in
files -- PGP/MIME is too e-mail specific to be used in that case.

Re: Trailing whitespace and PGP/MIME

[Russ Allbery]

Simon Josefsson <jas@extundo.com> writes:
> Russ Allbery <rra@stanford.edu> writes:

>> It shouldn't be as valid for inline PGP.  PGP implementations generally
>> handle trailing spaces without difficulty when processing attached
>> signatures (by stripping them before processing the message).

> Even pre-OpenPGP implementations?  E.g., PGP 2.x.

I can't speak for all of them, but PGP 2.x does indeed do that as long as
you give it the -t flag.  Whether the standard tools for checking such
messages do that or not, I don't know (but they should).

> RFC 1991 does not mention "whitespace", "trailing", or "strip".  Gnus
> currently claim to support them, although I'm not sure this is a good
> idea, nor if anyone actually uses PGP 2.x.

It's still relatively widely used for newsgroup control messages just
because that's the PGP that tale originally used, nearly all the keys are
RSA, and GnuPG couldn't handle RSA keys until about a year ago.  Outside
of that, I'm seeing rapidly diminishing popularity.

> For pure-ASCII, no trailing whitespace, no possibly dash escaped text,
> use of inline isn't that bad, especially for announcements stored in
> files -- PGP/MIME is too e-mail specific to be used in that case.

Point.

-- 
Russ Allbery (rra@stanford.edu)             <http://www.eyrie.org/~eagle/>