Default encryption for Message

Default encryption for Message

[Jens Lechtenboerger]

Hi there,

my stance on encryption is simple: The Internet is a hostile place,
where all personal communication must be encrypted.  Network
software should perform encryption by default and require a
conscious decision on the user’s part to send out plaintext.
Based on this conviction I have been maintaining DefaultEncrypt (and
ExtendSMIME) for some time [1]:
http://www.emacswiki.org/emacs/DefaultEncrypt

Essentially, that package extends Gnus Message to add MML encryption
tags if public keys for all recipients are available.

I’d like to integrate that code into Gnus, but I see a couple of
challenges, which I’d like to discuss with you.  (This message is
long, be warned ;)

I prepared a new version of that package with the aim to simplify
discussion, where I replaced the key rebinding with add-hooks and
renamed functions to indicate where they might belong:
http://informationelle-selbstbestimmung-im-internet.de/emacs/jl-encrypt-4.1-discussion.el
http://informationelle-selbstbestimmung-im-internet.de/emacs/jl-encrypt-4.1-discussion.el.asc
(No new functionality is added.  This is not a regular release.)

1. Bcc handling.
Bcc handling in Gnus is broken.  Messages are encrypted to
all recipients, giving away the key IDs (thus, in general the
identities) of all recipients, including the “blind” ones.
The Right Thing is explained there:
http://lists.gnupg.org/pipermail/gnupg-users/2014-April/049394.html

In DefaultEncrypt I added a test to warn against such cases.  In the
version linked above, mml-secure-bcc-is-safe implements that test.
I suggest to copy that function (and its prerequisites) into
mml-sec.el.  Then, mml-secure-bcc-is-safe can be added as
message-send-hook, which I suggest as default until proper Bcc
handling is implemented.

2. mml-default-encrypt-method
The variable mml-default-encrypt-method combines two separate
concepts: First, whether to use OpenPGP or S/MIME.  Second, if
OpenPGP is chosen whether to use MIME Security or not.  For me, this
is not good enough.  Suppose I preferred S/MIME in general (which I
do not).  Then, I would need to customize mml-default-encrypt-method
for S/MIME.  If I then send an e-mail to someone who does not use
S/MIME but OpenPGP, my code kicks in, detects that OpenPGP should be
used, but does not know whether to use MIME Security or not.  Thus,
I’m using an additional variable that focuses on MIME Security for
OpenPGP: mml-secure-pgp-without-mime

If (a variant of) that variable was added to Gnus, mml-encrypt-alist
probably would need to change to include a generic pgp function,
which decides whether to use MIME or something else.  My own variable
mml-secure-pgp-without-mime is currently just a Boolean, so it would
not be good enough to distinguish pgp, pgpmime, and pgpauto.  When
and what is pgpauto good for, anyways?

3. Creation of signatures
If I’m not mistaken, Gnus only supports the creation of signatures
for reply messages (gnus-message-replysign and
gnus-message-replysignencrypted).  I added a variable
mml-secure-insert-signature with the following doc string:
Control whether MML signature tags should be produced.
If it is nil, no signatures are created.  If it is `always',
signature tags are inserted for new messages (with their type
determined by `mml-default-sign-method').  If is is `encrypted',
messages that are encrypted are also signed.

If this variable is set to nil, my code sets
gnus-message-replysignencrypted to nil, which is a hack.  Maybe my
variable should really be called gnus-message-sign and supersede
gnus-message-replysign and gnus-message-replysignencrypted.  Maybe
more values would be necessary then to cover all situations where
MML sign tags should be added automatically (always, only in replies
to signed messages, only in encrypted replies to signed messages,
only in encrypted replies to signed and encrypted messages, in every
encrypted message, never).
I’m not sure about user expectations and the necessity of backwards
compatibility, though.

4. mm-encrypt-option
I wrote code around mm-encrypt-option, via the variable
mml-secure-i-am-aware-of-mm-encrypt-option with the following
doc string:
In general, multiple public keys may exist for a single e-mail
address (e.g., for different security levels or before one key
expires).  In such a case, EasyPG and gpg automatically select a
key, while gpgsm reports an ambiguity.  This variable is intended as
safety check against the default setting of `mm-encrypt-option'.  If
that variable has its default value of nil, some key is selected
automatically, without any documentation as to what selection
process is applied.
If you are really sure about the appropriateness of that process,
set this variable to t.  If you are unsure about that process, do
not touch the default value of this variable, and you will be asked
to select keys manually if multiple ones are available.
If you configure `mm-encrypt-option' to `guided', this variable does
not have any effect.  (However, in that case you will *always* be
asked to choose keys, even if only one per recipient is available.
Quite likely, that is not what you want.)

Clearly, my variable is a hack.  A better approach might be a third
value for mm-encrypt-option, say guided-if-multiple, to only enter
guided mode if multiple keys are available.  That would require,
however, to modify code in mml1991.el, mml2015.el, and mml-smime.el,
which brings me to the next point.

5. Refactoring
mml1991.el, mml2015.el, and mml-smime.el share code, forcing
copy-and-paste on updates.

For example, in the past I used mml2015-epg-find-usable-key to find
usable keys.  With Gnus v5.13, mml2015-epg-find-usable-key and
mml1991-epg-find-usable-key were exact duplicates, while
mml-smime-epg-find-usable-key was similar but did not test for
disabled keys.  I went for mml2015-epg-find-usable-key in all cases.

With Ma Gnus v0.7, mml2015-epg-find-usable-key was rewritten with an
incompatible list of arguments to use the new function
mml2015-epg-check-sub-key, while mml1991-epg-find-usable-key
remained unchanged.  Why actually?

To be honest I don’t understand the comment in
mml2015-epg-find-usable-key about “Non e-mail user-id” with
mml-signers and mml2015-encrypt-to-self.  In fact, I don’t
understand mml2015-encrypt-to-self (which is not mentioned in the
manual) at all.  If I set that to true I cannot send encrypted
messages but receive the error: “Neither message sender nor
mml2015-signers are set”
For reasons I don’t understand the code seems to bind
encrypt-to-self to signing.  I like the freedom of encryption
independently of the controlling chains of signatures (also,
signatures require the presence of private keys, which is not the
case for encryption).  I’m simply using encrypt-to and
hidden-encrypt-to in gpg.conf.  Alternatively, messages could be
encrypted to the From address.  Alternatively, why is gpg’s
encrypt-to not good enough?
(Also, currently users probably need to configure three variables:
mml2015-encrypt-to-self, mml1991-encrypt-to-self,
mml-smime-encrypt-to-self)

In any case, I’d like to use a stable interface into epg and suggest
to provide that in a new file, say mml-epg.el.  Then, we could move
either mml2015-epg-find-usable-key or mml1991-epg-find-usable-key
with the new name mml-epg-find-usable-keys into that file and use
that in mml1991, mml2015, and mml-smime.  Actually, that function
should have an additional Boolean argument specifying whether to
return just one key or all of them.  Also, the functions named
mml-epg-... in jl-encrypt.el could go into that file.

I’d be happy to write that file once we discussed the change in
mml2015-epg-find-usable-key about “Non email user-id” mentioned
above.

What’s your opinion?

Best wishes
Jens

[1] Actually, much more could be done.
(a) Emacs or Gnus could check whether GnuPG is installed.  If not,
    recommend its installation.
The following is what https://www.mailpile.is/ seems to do.
(b) For every message Gnus could check whether a secret key exists
    for the From address.  If not, (recommend to) create one.  (With
    customizable exceptions.)
(c) For every plaintext message, attach the own public key, and
    point to a good guide, e.g.: https://emailselfdefense.fsf.org/
    (With customizable exceptions.)
(d) Automatically import public keys.
(e) Support search over encrypted messages.

Re: Default encryption for Message

[Ted Zlatanov]

On Sun, 21 Sep 2014 12:08:44 +0200 Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> wrote: 

JL> 1. Bcc handling.

JL> Bcc handling in Gnus is broken.  Messages are encrypted to
JL> all recipients, giving away the key IDs (thus, in general the
JL> identities) of all recipients, including the “blind” ones.
JL> The Right Thing is explained there:
JL> http://lists.gnupg.org/pipermail/gnupg-users/2014-April/049394.html

JL> In DefaultEncrypt I added a test to warn against such cases.  In the
JL> version linked above, mml-secure-bcc-is-safe implements that test.
JL> I suggest to copy that function (and its prerequisites) into
JL> mml-sec.el.  Then, mml-secure-bcc-is-safe can be added as
JL> message-send-hook, which I suggest as default until proper Bcc
JL> handling is implemented.

I don't think warning is enough.  If it can be fixed, it should be fixed.

JL> 2. mml-default-encrypt-method

No opinion.

JL> 3. Creation of signatures
...
JL> I’m not sure about user expectations and the necessity of backwards
JL> compatibility, though.

No opinion, except IMO the Right Thing should override backwards
compatibility when it comes to security.

JL> 4. mm-encrypt-option
...
JL> Clearly, my [solution] is a hack.  A better approach might be a third
JL> value for mm-encrypt-option, say guided-if-multiple, to only enter
JL> guided mode if multiple keys are available.  That would require,
JL> however, to modify code in mml1991.el, mml2015.el, and mml-smime.el,
JL> which brings me to the next point.

Ask once, then save the preference in a Customize-controlled variable.

JL> 5. Refactoring

No opinion.

Ted

Re: Default encryption for Message

[Jens Lechtenboerger]

On 2014-09-23, Ted Zlatanov wrote:

> JL> 4. mm-encrypt-option
> ...
> JL> Clearly, my [solution] is a hack.  A better approach might be a third
> JL> value for mm-encrypt-option, say guided-if-multiple, to only enter
> JL> guided mode if multiple keys are available.  That would require,
> JL> however, to modify code in mml1991.el, mml2015.el, and mml-smime.el,
> JL> which brings me to the next point.
>
> Ask once, then save the preference in a Customize-controlled variable.

Before this was saved permanently the user should probably be asked,
as there might be addresses where the “correct” key depends on the
message’s context.

Also, an implementation of such a feature requires more code.
Currently, I’m setting “guided” on a per message basis to reuse the
code around epa-select-keys in the three files mentioned above.
There, epa-select-keys is invoked for *all* recipients of the
message, not just those with multiple keys.

For your proposal, the code in those three files would need to be
changed to iterate over the recipients and invoke epa-select-keys
for each recipient with multiple keys.  Another reason for
refactoring.

Best wishes
Jens

Re: Default encryption for Message

[Ted Zlatanov]

On Wed, 24 Sep 2014 15:59:24 +0200 Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> wrote: 

JL> For your proposal, the code in those three files would need to be
JL> changed to iterate over the recipients and invoke epa-select-keys
JL> for each recipient with multiple keys.  Another reason for
JL> refactoring.

I don't know the internals there (and don't normally use those
facilities) and am happy to let you and others lead the effort.

If you're considering refactoring, please consider adding tests to
support the code.

Ted

Re: Default encryption for Message

[Daiki Ueno]

Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> writes:

> 5. Refactoring
> mml1991.el, mml2015.el, and mml-smime.el share code, forcing
> copy-and-paste on updates.

Unifying those code would be a good idea.

> With Ma Gnus v0.7, mml2015-epg-find-usable-key was rewritten with an
> incompatible list of arguments to use the new function
> mml2015-epg-check-sub-key, while mml1991-epg-find-usable-key
> remained unchanged.  Why actually?

I guess it's a left-over.

> To be honest I don’t understand the comment in
> mml2015-epg-find-usable-key about “Non e-mail user-id” with
> mml-signers and mml2015-encrypt-to-self.  In fact, I don’t
> understand mml2015-encrypt-to-self (which is not mentioned in the
> manual) at all.  If I set that to true I cannot send encrypted
> messages but receive the error: “Neither message sender nor
> mml2015-signers are set”

mml2015-encrypt-to-self requires that either mml2015-sign-with-sender or
mml2015-signers is set.  If the former variable is set, it looks up a
key by a sender's e-mail address.  Otherwise, it looks for keys listed
in mml2015-signers, which could be:

(setq mml2015-signers '("D7E69871" ...))

Here, "D7E69871" is a "Non e-mail user-id"[1].  Perhaps we could enable
mml2015-sign-with-sender by default, if mml2015-encrypt-to-self is set.

> I’m simply using encrypt-to and hidden-encrypt-to in gpg.conf.

This was the idea why epg.el dropped the options like
pgg-encrypt-to-self.  However, in the end we realized that the Gnus
usage of GPG needed more flexibility.  Some people use multiple keys
associated with a single e-mail address.  For example, I have several
keys associated with ueno@gnu.org for different purposes than encrypting
or signing mails, e.g. signing tarballs.

> In any case, I’d like to use a stable interface into epg and suggest
> to provide that in a new file, say mml-epg.el.  Then, we could move
> either mml2015-epg-find-usable-key or mml1991-epg-find-usable-key
> with the new name mml-epg-find-usable-keys into that file and use
> that in mml1991, mml2015, and mml-smime.  Actually, that function
> should have an additional Boolean argument specifying whether to
> return just one key or all of them.  Also, the functions named
> mml-epg-... in jl-encrypt.el could go into that file.

Agreed.

Footnotes: 
[1]  The acceptable patterns are described in: https://www.gnupg.org/documentation/manuals/gnupg/Specify-a-User-ID.html#Specify-a-User-ID

Re: Default encryption for Message

[Jens Lechtenboerger]

On 2014-09-24, Daiki Ueno wrote:

> Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> writes:
>
>> To be honest I don’t understand the comment in
>> mml2015-epg-find-usable-key about “Non e-mail user-id” with
>> mml-signers and mml2015-encrypt-to-self.
>
> mml2015-encrypt-to-self requires that either mml2015-sign-with-sender or
> mml2015-signers is set.  If the former variable is set, it looks up a
> key by a sender's e-mail address.  Otherwise, it looks for keys listed
> in mml2015-signers, which could be:
>
> (setq mml2015-signers '("D7E69871" ...))
>
> Here, "D7E69871" is a "Non e-mail user-id"[1].

Thanks for the clarification.  When and why would you set this
variable to a list?

> Perhaps we could enable mml2015-sign-with-sender by default, if
> mml2015-encrypt-to-self is set.

I fail to see why encrypt-to-self should be tied to sign-related
variables.

>> I’m simply using encrypt-to and hidden-encrypt-to in gpg.conf.
>
> This was the idea why epg.el dropped the options like
> pgg-encrypt-to-self.  However, in the end we realized that the Gnus
> usage of GPG needed more flexibility.  Some people use multiple keys
> associated with a single e-mail address.  For example, I have several
> keys associated with ueno@gnu.org for different purposes than encrypting
> or signing mails, e.g. signing tarballs.

I’m taken by surprise.  Do you sign tarballs in Gnus?

I understand that flexible schemes are necessary, but I still fail
to see why the current mml2015 approach is desirable.  If highly
customizable encrypt-to-self options are needed, I would recommend
Bcc.

Best wishes
Jens

Re: Default encryption for Message

[Daiki Ueno]

Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> writes:

> I fail to see why encrypt-to-self should be tied to sign-related
> variables.

The variable name might not be too intuitive, but maybe you could
imagine how GPG signing and encryption work.  Decryption keys are
usually same as signing keys.  I didn't want to add a separate variable
for this particular feature.

> I’m taken by surprise.  Do you sign tarballs in Gnus?

No.  I'm saying that I don't want Gnus to pick the key I use for singing
tarballs, outside of Gnus.

> I understand that flexible schemes are necessary, but I still fail
> to see why the current mml2015 approach is desirable.  If highly
> customizable encrypt-to-self options are needed, I would recommend
> Bcc.

I'm not sure what you understand nor what you are proposing.  Here is
the detailed use-case:

Suppose that one has three keys associated with his e-mail address.  Key
A is for encrypted e-mail communication with his colleagues, key B is
for communicating with friends, and key C is for automatic signing of
tarballs.

Here, those keys are more securely protected in order of A, B, and C.
He doesn't want to use key C for any e-mail encryption.  He doesn't want
to use key B for confidential communication within his company.

This might sound hypothetical but I do have a similar setting.  How do
you address this situation with ~/.gnupg/gpg.conf options and Bcc?

Re: Default encryption for Message

[Jens Lechtenboerger]

On 2014-09-25, Daiki Ueno wrote:

> Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> writes:
>
>> I fail to see why encrypt-to-self should be tied to sign-related
>> variables.
>
> The variable name might not be too intuitive, but maybe you could
> imagine how GPG signing and encryption work.  Decryption keys are
> usually same as signing keys.

I’m not sure about that, but this may be a side issue.  GnuPG
creates an encryption subkey by default.

> Suppose that one has three keys associated with his e-mail address.  Key
> A is for encrypted e-mail communication with his colleagues, key B is
> for communicating with friends, and key C is for automatic signing of
> tarballs.
>
> Here, those keys are more securely protected in order of A, B, and C.
> He doesn't want to use key C for any e-mail encryption.  He doesn't want
> to use key B for confidential communication within his company.
>
> This might sound hypothetical but I do have a similar setting.  How do
> you address this situation with ~/.gnupg/gpg.conf options and Bcc?

Apparently, we have been talking about different things.

I was interested in mml2015-encrypt-to-self and looked at its doc
string.  Setting to t fails.  You explained that
mml2015-sign-with-sender or mml2015-signers are necessary.
However, I don’t want to sign and suggested gpg.conf/Bcc instead.

In your scenario, I see two choices, say concerning Alice with those
three keys.

(1) Friend Bob wants to send a message to Alice and is supposed to
choose the correct key B (but neither A nor C).

(2) Alice wants to send an encrypted message to colleague Charlie,
and she wants to make sure that the message is encrypted to her key
A (but neither B nor C) in addition to Charlie’s key.

For (1), my code sets mm-encrypt-option to 'guided to enforce a
manual choice so that Bob learns about the existence of those three
public keys in the first place.  (Ted Zlatanov suggested to save
that choice.)

For (2), I think that another variable would be necessary to
customize what key to use for what recipient.  (Gnus could ask if it
detects multiple encryption keys.)

I don’t see how either case is supported by mml2015, though.  What
is your configuration?

Besides, what is the intended meaning for signatures if
mml2015-signers contains multiple keys?

Best wishes
Jens

Re: Default encryption for Message

[Daiki Ueno]

Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> writes:

> I was interested in mml2015-encrypt-to-self and looked at its doc
> string.  Setting to t fails.  You explained that
> mml2015-sign-with-sender or mml2015-signers are necessary.
> However, I don’t want to sign and suggested gpg.conf/Bcc instead.

So, maybe improving the documentation of mml2015-encrypt-to-self would
be a good start?

> In your scenario, I see two choices, say concerning Alice with those
> three keys.
>
> (1) Friend Bob wants to send a message to Alice and is supposed to
> choose the correct key B (but neither A nor C).
>
> (2) Alice wants to send an encrypted message to colleague Charlie,
> and she wants to make sure that the message is encrypted to her key
> A (but neither B nor C) in addition to Charlie’s key.
>
> For (1), my code sets mm-encrypt-option to 'guided to enforce a
> manual choice so that Bob learns about the existence of those three
> public keys in the first place.  (Ted Zlatanov suggested to save
> that choice.)

I don't think this is a good idea.  I know some people ("security" fans)
like this kind of verbose behavior, but others don't even want to
remember key IDs and are prone to choose wrong keys.

There are criticisms about a related topic:
http://www.superlectures.com/guadec2013/more-secure-with-less-security

> For (2), I think that another variable would be necessary to
> customize what key to use for what recipient.  (Gnus could ask if it
> detects multiple encryption keys.)

No objection on that.

> I don’t see how either case is supported by mml2015, though.  What
> is your configuration?

It can be done by dynamically setting mml2015-signers upon sending.  For
some reason I use hooks (namely message-header-hook), but it can be done
more easily by using group parameters or BBDB.

(info "(gnus) Group Parameters")

> Besides, what is the intended meaning for signatures if
> mml2015-signers contains multiple keys?

This is certainly for the case when one wants to sign a message with
multiple keys.

Re: Default encryption for Message

[Jens Lechtenboerger]

On 2014-09-28, Daiki Ueno wrote:

> Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> writes:
>
> So, maybe improving the documentation of mml2015-encrypt-to-self would
> be a good start?

Yes, there is room for improvement ;)
Maybe that variable should not be t or nil but instead a list of key
IDs (empty by default).

>> (1) Friend Bob wants to send a message to Alice and is supposed to
>> choose the correct key B (but neither A nor C).
>>
>> (2) Alice wants to send an encrypted message to colleague Charlie,
>> and she wants to make sure that the message is encrypted to her key
>> A (but neither B nor C) in addition to Charlie’s key.
>>
>> For (1), my code sets mm-encrypt-option to 'guided to enforce a
>> manual choice so that Bob learns about the existence of those three
>> public keys in the first place.  (Ted Zlatanov suggested to save
>> that choice.)
>
> I don't think this is a good idea.  I know some people ("security" fans)
> like this kind of verbose behavior, but others don't even want to
> remember key IDs and are prone to choose wrong keys.
>
> There are criticisms about a related topic:
> http://www.superlectures.com/guadec2013/more-secure-with-less-security

Thank you for the link.  The idea of using intentions is a good one.
I don’t see how to specify intentions without questions in this
case, though.  Maybe I just didn’t get the point of the video.

Bob’s intention is simple: Encrypt a message to Alice.  However, in
the scenario outlined by you, Alice’s intention is much more
complex: Have different parties use different keys in messages sent
to her.  How do those parties and their software learn about her
intention?  As you just pointed out, Bob quite likely does not know
about key IDs.  In contrast, his software does.

I think of a question like this (to be answered by Bob): “Multiple
encryption keys for <Alice@example.org> are available.  Which one(s)
would you like to use?  (Type ?/a/c/m; ? for help.)”

? displays explanation for the other keys:
- a: Abort.  I don’t know what that means, and I’ll ask the
  recipient offline.
- c: Continue.  I don’t know what that means.  Use the default key
  and insert a question for the recipient.
  (This would insert a customizable text like the following into the
  message: “P.S. My software told me that I could use several
  encryption keys, namely <list of key IDs inserted here>, but I
  don’t know how to choose.  Please explain your intention.”)
- m: Menu.  Display a menu (see mm-encrypt-option 'guided), where
  Bob chooses.  Once Bob has chosen, he is asked whether he’d like
  to choose anew for every message sent to Alice or to record this
  choice permanently.

>> Besides, what is the intended meaning for signatures if
>> mml2015-signers contains multiple keys?
>
> This is certainly for the case when one wants to sign a message with
> multiple keys.

My fault.  I wasn’t aware of that idea.  I don’t see why people
would do this, but it works.

Best wishes
Jens

Refactoring of mml1991.el, mml2015.el, mml-smime.el (was: Default encryption for Message)

[Jens Lechtenboerger]

On 2014-09-21, at 12:08, Jens Lechtenboerger wrote:

> 5. Refactoring
> mml1991.el, mml2015.el, and mml-smime.el share code, forcing
> copy-and-paste on updates.

Hi there,

this is a continuation of a one year old thread.  Sorry for the delay.
Essentially, I proposed a refactoring of mml1991.el, mml2015.el, and
mml-smime.el such that common code can be maintained in a single file.

I did that now, with a new file mml-epg.el that is accompanied by
test cases, mml-epg-test.el.

My changes are as follows, please let me know what you think.


* Common code moved to mml-epg.el (from all three previous files)

Functionality concerning passphrase callback, code to check user IDs and
sub-keys for validity, code to find usable public and private keys,
common code from sign and encrypt functions.


* Treatment of common variables

There are variables in mml1991.el and mml2015.el for which I doubt that
users will really set them differently.  In fact, both files concern
OpenPGP, most likely (or for sure?) with GnuPG, and I fail to see
reasons to set them differently.  These are the variables I’m referring
to:
mml1991-cache-passphrase and mml2015-cache-passphrase,
mml1991-passphrase-cache-expiry and mml2015-passphrase-cache-expiry,
mml1991-signers and mml2015-signers,
mml1991-encrypt-to-self and mml2015-encrypt-to-self.

In my code, each pair is replaced with a single variable with the prefix
“mml-epg-openpgp”, e.g., “mml-epg-openpgp-cache-passphrase”.

Concerning S/MIME, I introduced the same variables with prefix
“mml-epg-smime”.

Also, two variables from mml2015.el and mml-smime.el were missing in
mml1991.el, whose functionality is present now with the common prefix
“mml-epg-openpgp”:
mml-epg-openpgp-sign-with-sender and mml-epg-openpgp-always-trust


* Treatment of encrypt-to-self

As discussed last year, encrypt-to-self in mml2015.el is implicitly
based on signature-related variables (based on undocumented use of
mml2015-sign-with-sender), which I find counter-intuitive.  I changed
the encrypt-to-self variables into independent ones: If the value is t,
then the message is also encrypted to the sender’s key.  Alternatively,
the value can also be a list of encryption keys (or corresponding e-mail
addresses), in which case the message is also encrypted to those keys.


* Minor (?) differences among previous files, now unified

In mml-smime-epg-passphrase-callback, password-cache-key-id is
not passed as argument to password-read, while such an argument is
passed in the corresponding functions in mml2015.el and mml1991.el.
Now that argument is passed also for S/MIME.
Is there a reason not to pass that argument?

In mml1991.el, prior to encryption, recipients were not enclosed in
ankle brackets "<" and ">", while such brackets were added in
mml2015.el.  Now they are added in all cases.


* Passphrase caching

If I’m not mistaken, with GnuPG 2.x (and gpgsm) passphrases cannot be
cached within Emacs as gpg-agent is started automatically and tries to
invoke a pinentry program.  So caching-related variables only apply to
GnuPG 1.x and S/MIME with openssl (I suggest not to advertise the
latter any longer, see separate point below).

In fact, one of my test cases
(mml-epg-en-decrypt-passphrase-no-cache-smime-todo)
has an expected failure for gpgsm, as the cache is not inspected.
Another test case (mml-epg-en-decrypt-passphrase-no-cache-openpgp-todo)
is skipped for GnuPG 2.x.
Is there a way to cache passphrases within Emacs, even with gpgsm and
GnuPG 2.x?

Note that GnuPG 2.x is where the development happens, and according to
GnuPG’s README, “2.0 is the current stable version for general use”,
while “1.4 is the old standalone version which is most suitable for
older or embedded platforms.”
So, if we cannot cache for the current stable version, should we not
recommend caching inside gpg-agent in general and get rid of the
associated code?  Even if there was a way, should we not recommend
caching inside gpg-agent in general?


* Selecting “usable” keys

In mml2015-epg-find-usable-key the test mml2015-epg-check-user-id is not
applied if the condition (string-match "\\`<" name) is true.
I don’t understand this, but I still copied the code to
mml-epg-find-usable-keys.  What is special about names starting with "<"
(except that they are used to indicate exact matches for e-mail
addresses in GnuPG)?
Why should revoked or expired user IDs be OK if they match that pattern?

In fact, one of my test cases (mml-epg-find-usable-keys-todo) is an
expected failure due to this behavior, where a revoked user IDs is
returned (also expired ones could be returned, if one knows how to
create them).

Moreover, the function mml2015-epg-find-usable-secret-key does not check
validity of subkeys requested by key ID.  Two of my test cases
(mml-epg-find-usable-keys-todo-2 and -3) fail due to this issue, when an
expired subkey is requested and the mainkey is returned.
Also, test case mml-epg-sign-verify-todo behaves in the wrong way due to
this issue.

Finally, function mml-epg-find-usable-key has an argument USAGE, which
can be `sign'.  However, as main keys can typically be used to sign
(by the key owner), almost every key is “usable” for USAGE `sign'
according to this function, although no sign operation can be performed
when the private key is missing.

I’ve written new functions mml-epg-check-sub-key-2 and
mml-epg-find-usable-keys-2 to address the above issues (with
corresponding test cases).  Note that mml-epg-find-usable-keys-2
subsumes mml-epg-find-usable-keys and mml-epg-find-usable-secret-keys
(which are based on mml2015.el).  The doc string explains the
differences.

Essentially, I propose to use mml-epg-find-usable-keys-2 (and changed
helper functions) instead of mml-epg-find-usable-keys and
mml-epg-find-usable-secret-keys.


* Choice of “usable” keys, now with customizable preferences

Previously, the “first” usable key (based on an undocumented order) was
chosen if multiple keys were available (for encryption as well as
signing).  My new code asks the user which one to use and allows to
customize that choice (new variable mml-epg-key-preferences).



* Questions while coding

In mml-epg-test.el two TODO comments occur, marking sections that need
verification.  One is discussed above (passphrase caching).  The second
one:

In my test code, I use several fixtures for test setup, sometimes
calling one fixture from another.  I ran into infinite loops, when I
called variables representing function bodies “body” in several
fixtures.  By renaming variables to have different names in different
fixtures, the code runs as I expected.  Why do those infinite loops
occur?  Missing understanding on my part or an Emacs bug?

Moreover, I added several TODO comments in mml-epg.el, related to above
points as well as to additional questions:

In mml-epg-select-keys-1, I use y-or-n-p to display a question
(let-binding message-truncate-lines to nil), but the question gets
truncated if called from an encryption context.
To see this, invoke mml-epg-select-preferred-keys-todo from
mml-epg-test.el (see README.txt for detailed instructions), mark one or
two keys, and press `C-c C-c'.  You’ll be asked whether to save that
choice, but the question is truncated.
How can I resize the echo area to display the entire message?
(With mml-epg-select-preferred-keys-ok you can see the entire message,
but outside an encryption context.)

Towards the end of mml-epg.jl three basic functions are defined, which
might be of independent interest.  Where should they go?

In mml-epg.el I use define-obsolete-variable-alias, probably in an
incorrect way.  Where/how should that be done to make sure that
potential user customizations of the obsolete variables carry over to
the new ones?


* Preference for openssl vs epg (gpgsm)

Currently, Gnus prefers openssl over epg (gpgsm), via
(defcustom mml-smime-use (if (featurep 'epg) 'epg 'openssl))
in mml-smime.el.  However, epg does not get loaded on its own even if it
is present.  Thus, users need to set mml-smime-use or require epg in
their ~/.emacs, but the manual does not mention gpgsm at all.
Where would be an appropriate place to (require 'epg)?
I propose to change that preference in favor of epg (gpgsm) as:
** Gpgsm manages certificates (storage, expiry, revocation).
   Users need to perform those tasks manually with openssl.
** Openssl has bugs as documented in the BUGS section of man smime(1).
   In particular: SMIMECapabilities are ignored, no revocation checking
   is done on the signer's certificate.
** Advertised SMIMECapabilities include broken encryption algorithms.
   With the precompiled openssl 1.0.1f on my system RC2 is advertised,
   which should have been dropped since S/MIME 3.x, see:
   https://tools.ietf.org/html/rfc5751#appendix-B


* Next steps

A tarball with my files is available over there (README.txt explains its
contents):
http://informationelle-selbstbestimmung-im-internet.de/emacs/mml-epg-2015-10-16.tar.bz2
http://informationelle-selbstbestimmung-im-internet.de/emacs/mml-epg-2015-10-16.tar.bz2.asc

What are your suggestions to continue from here?

Best wishes
Jens

Re: Refactoring of mml1991.el, mml2015.el, mml-smime.el

[Peter Münster]

On Fri, Oct 16 2015, Jens Lechtenboerger wrote:

> So, if we cannot cache for the current stable version, should we not
> recommend caching inside gpg-agent in general and get rid of the
> associated code?  Even if there was a way, should we not recommend
> caching inside gpg-agent in general?

+1

-- 
           Peter

Re: Refactoring of mml1991.el, mml2015.el, mml-smime.el

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 3469 bytes --]


Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> writes:

[I have not yet absorbed your entire message and am commenting selectively.]

> * Passphrase caching
>
> If I’m not mistaken, with GnuPG 2.x (and gpgsm) passphrases cannot be
> cached within Emacs as gpg-agent is started automatically and tries to
> invoke a pinentry program.  So caching-related variables only apply to
> GnuPG 1.x and S/MIME with openssl (I suggest not to advertise the
> latter any longer, see separate point below).

I'm not sure that it is "cannot", as it seems that gpg2 should be
configurable not to use gpg-agent.  But, I can't find that in the docs,
and it seems clear that gpg-agent is the gnpug-recommended method.

Also, one can use gpg-agent with gnupg1 with the "use-agent" option.

> In fact, one of my test cases
> (mml-epg-en-decrypt-passphrase-no-cache-smime-todo)
> has an expected failure for gpgsm, as the cache is not inspected.
> Another test case (mml-epg-en-decrypt-passphrase-no-cache-openpgp-todo)
> is skipped for GnuPG 2.x.
> Is there a way to cache passphrases within Emacs, even with gpgsm and
> GnuPG 2.x?

Are you saying that you think it is a good idea to cache passphrases
within emacs, for general use, or to enable tests?

> Note that GnuPG 2.x is where the development happens, and according to
> GnuPG’s README, “2.0 is the current stable version for general use”,
> while “1.4 is the old standalone version which is most suitable for
> older or embedded platforms.”
> So, if we cannot cache for the current stable version, should we not
> recommend caching inside gpg-agent in general and get rid of the
> associated code?  Even if there was a way, should we not recommend
> caching inside gpg-agent in general?

I'm having a little trouble following "should we not".

So if you mean:

  within elisp code in gnus/epg/etc. we should remove the code that
  caches passphrases

  we should expect people to use gnupg's gpg-agent(1) program to deal
  with caching if they want to cache, including telling people to do
  that with gnupg1.

  we should realize that openssl-based mime will not have any caching in
  emacs or externally, but that's ok because that code is or will be
  deprecated soon

then that sounds good.    

> * Preference for openssl vs epg (gpgsm)
>
> Currently, Gnus prefers openssl over epg (gpgsm), via
> (defcustom mml-smime-use (if (featurep 'epg) 'epg 'openssl))
> in mml-smime.el.  However, epg does not get loaded on its own even if it
> is present.  Thus, users need to set mml-smime-use or require epg in
> their ~/.emacs, but the manual does not mention gpgsm at all.
> Where would be an appropriate place to (require 'epg)?
> I propose to change that preference in favor of epg (gpgsm) as:
> ** Gpgsm manages certificates (storage, expiry, revocation).
>    Users need to perform those tasks manually with openssl.
> ** Openssl has bugs as documented in the BUGS section of man smime(1).
>    In particular: SMIMECapabilities are ignored, no revocation checking
>    is done on the signer's certificate.
> ** Advertised SMIMECapabilities include broken encryption algorithms.
>    With the precompiled openssl 1.0.1f on my system RC2 is advertised,
>    which should have been dropped since S/MIME 3.x, see:
>    https://tools.ietf.org/html/rfc5751#appendix-B

Having tried to use both for S/MIME, I concur with recommending gpgsm.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 180 bytes --]

Re: Refactoring of mml1991.el, mml2015.el, mml-smime.el

[Jens Lechtenboerger]

On 2015-10-18, at 10:09, Greg Troxel wrote:

> Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> writes:

>> In fact, one of my test cases
>> (mml-epg-en-decrypt-passphrase-no-cache-smime-todo)
>> has an expected failure for gpgsm, as the cache is not inspected.
>> Another test case (mml-epg-en-decrypt-passphrase-no-cache-openpgp-todo)
>> is skipped for GnuPG 2.x.
>> Is there a way to cache passphrases within Emacs, even with gpgsm and
>> GnuPG 2.x?
>
> Are you saying that you think it is a good idea to cache passphrases
> within emacs, for general use, or to enable tests?

I tried to enable that for testing, but failed.

>> Note that GnuPG 2.x is where the development happens, and according to
>> GnuPG’s README, “2.0 is the current stable version for general use”,
>> while “1.4 is the old standalone version which is most suitable for
>> older or embedded platforms.”
>> So, if we cannot cache for the current stable version, should we not
>> recommend caching inside gpg-agent in general and get rid of the
>> associated code?  Even if there was a way, should we not recommend
>> caching inside gpg-agent in general?
>
> I'm having a little trouble following "should we not".

I’m sorry for that wording.  Dropping the “not” seems better.

> So if you mean:
>
>   within elisp code in gnus/epg/etc. we should remove the code that
>   caches passphrases

I’d like to hear opinions on that.

>   we should expect people to use gnupg's gpg-agent(1) program to deal
>   with caching if they want to cache, including telling people to do
>   that with gnupg1.

That would be my preference, yes.

>   we should realize that openssl-based mime will not have any caching in
>   emacs or externally, but that's ok because that code is or will be
>   deprecated soon

I’m not sure how many people use caching inside Emacs.  Let’s wait
for comments here.

Best wishes
Jens

Re: Refactoring of mml1991.el, mml2015.el, mml-smime.el

[Daiki Ueno]

Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> writes:

> On 2014-09-21, at 12:08, Jens Lechtenboerger wrote:
>
>> 5. Refactoring
>> mml1991.el, mml2015.el, and mml-smime.el share code, forcing
>> copy-and-paste on updates.
>
> Hi there,
>
> this is a continuation of a one year old thread.  Sorry for the delay.
> Essentially, I proposed a refactoring of mml1991.el, mml2015.el, and
> mml-smime.el such that common code can be maintained in a single file.

Thanks for working on that.  I am afraid I don't have time at the moment
to go through the patch, but maybe you can go ahead and we could fix it
later on if any problem arises?

> In my code, each pair is replaced with a single variable with the prefix
> “mml-epg-openpgp”, e.g., “mml-epg-openpgp-cache-passphrase”.
>
> Concerning S/MIME, I introduced the same variables with prefix
> “mml-epg-smime”.

The prefix "mml-epg" sounds a bit too much implementation detail.  How
about renaming those prefixes to mml{,-secure}-pgpmime /
mml{,-secure}-smime and moving to mml-sec.el?

(I would use the word "pgpmime" to distinguish PGP/MIME signatures from
inline PGP signatures.  I remember there is already a use of the word in
Gnus).

> A tarball with my files is available over there (README.txt explains its
> contents):
> http://informationelle-selbstbestimmung-im-internet.de/emacs/mml-epg-2015-10-16.tar.bz2
> http://informationelle-selbstbestimmung-im-internet.de/emacs/mml-epg-2015-10-16.tar.bz2.asc

I saw a few functions prefixed with "jl-".  Maybe they can be moved to
mm-util.el or gnus-util.el?

Thanks,
-- 
Daiki Ueno

Re: Refactoring of mml1991.el, mml2015.el, mml-smime.el

[Jens Lechtenboerger]

On 2015-11-06, at 11:10, Daiki Ueno wrote:

> Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> writes:
>
>> On 2014-09-21, at 12:08, Jens Lechtenboerger wrote:

>> In my code, each pair is replaced with a single variable with the prefix
>> “mml-epg-openpgp”, e.g., “mml-epg-openpgp-cache-passphrase”.
>>
>> Concerning S/MIME, I introduced the same variables with prefix
>> “mml-epg-smime”.
>
> The prefix "mml-epg" sounds a bit too much implementation detail.  How
> about renaming those prefixes to mml{,-secure}-pgpmime /
> mml{,-secure}-smime and moving to mml-sec.el?
>
> (I would use the word "pgpmime" to distinguish PGP/MIME signatures from
> inline PGP signatures.  I remember there is already a use of the word in
> Gnus).

Yes, I can move that to mml-sec.el.  However, note that I’m not
restricting functionality to pgpmime.  Inline PGP is still available
(via mml1991.el).  The variables whose names include “openpgp” in my
code cover pgpmime (mml2015.el) as well as inline PGP (mml1991.el).
Therefore, I’d like to use “openpgp” instead of “pgpmime”.

Moreover, in mml-sec.el the variables mml-secure-cache-passphrase
and mml-secure-passphrase-cache-expiry are defined, which have
specialized versions in mml-smime.el, mml1991.el, and mml2015.el,
which I meant to delete or to supersede by two new variables (for
OpenPGP and S/MIME).  So far, I didn’t receive strong feedback on
the related issue of my e-mail (passphrase caching in Emacs vs
gpg-agent).  Now I wonder whether to get rid of all of them except
those in mml-secure.el (with a recommendation to use gpg-agent).

How should I proceed from here?  Ask for opinions on other mailing
lists such as the following two?
https://lists.gnu.org/mailman/listinfo/info-gnus-english
https://lists.gnu.org/mailman/listinfo/help-gnu-emacs

> I saw a few functions prefixed with "jl-".  Maybe they can be moved to
> mm-util.el or gnus-util.el?

Indeed, gnus-util.el seems appropriate.

Best wishes
Jens

Re: Refactoring of mml1991.el, mml2015.el, mml-smime.el

[Daiki Ueno]

Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> writes:

> Yes, I can move that to mml-sec.el.  However, note that I’m not
> restricting functionality to pgpmime.  Inline PGP is still available
> (via mml1991.el).  The variables whose names include “openpgp” in my
> code cover pgpmime (mml2015.el) as well as inline PGP (mml1991.el).
> Therefore, I’d like to use “openpgp” instead of “pgpmime”.

Right, then "openpgp" would be fine.

> Moreover, in mml-sec.el the variables mml-secure-cache-passphrase
> and mml-secure-passphrase-cache-expiry are defined, which have
> specialized versions in mml-smime.el, mml1991.el, and mml2015.el,
> which I meant to delete or to supersede by two new variables (for
> OpenPGP and S/MIME).  So far, I didn’t receive strong feedback on
> the related issue of my e-mail (passphrase caching in Emacs vs
> gpg-agent).  Now I wonder whether to get rid of all of them except
> those in mml-secure.el (with a recommendation to use gpg-agent).

It's tempting to remove them, but I guess we still need to care users
who keep using gpg1 without gpg-agent.

> How should I proceed from here?  Ask for opinions on other mailing
> lists such as the following two?
> https://lists.gnu.org/mailman/listinfo/info-gnus-english
> https://lists.gnu.org/mailman/listinfo/help-gnu-emacs

If there is any incompatible change in the patch, then it would be good
to announce on those lists.  On the other hand, if you are doing only
refactoring, I doubt it will be useful to ask on the lists and it might
be easier to directly push the changes to the git repo and gather
practical feedbacks.

Do you have copyright assignment for Emacs?  If not, consider filing
one.  Once the process has done, I think you can ask Lars or Ted for the
commit access.

Regards,
-- 
Daiki Ueno

Re: Refactoring of mml1991.el, mml2015.el, mml-smime.el

[Jens Lechtenboerger]

On 2015-11-11, at 15:20, Daiki Ueno wrote:

> If there is any incompatible change in the patch, then it would be
> good to announce on those lists.

There will be incompatible changes.  I plan to go for changes such
that my test cases do not fail.

> On the other hand, if you are doing only refactoring, I doubt it
> will be useful to ask on the lists and it might be easier to
> directly push the changes to the git repo and gather practical
> feedbacks.

Let’s try that.

> Do you have copyright assignment for Emacs?  If not, consider
> filing one.  Once the process has done, I think you can ask Lars
> or Ted for the commit access.

I do have that assignment, and I sent an e-mail to Ted (he is
suggested on the git web page).

Many thanks for your input!

Best wishes
Jens

in defense of GitLab or something (was: Refactoring of mml1991.el, mml2015.el, mml-smime.el)

[Ted Zlatanov]

On Sat, 14 Nov 2015 16:44:07 +0100 Jens Lechtenboerger <jens.lechtenboerger@fsfe.org> wrote: 

JL> I do have that assignment, and I sent an e-mail to Ted (he is
JL> suggested on the git web page).

Sorry, I've been swamped with work. One of the reasons I proposed GitLab
here is that it wouldn't make it necessary for me to spin up a special
VM with a static IP address, log into the Gnus Git server, then add your
key manually.  I could just go to the admin interface and do it in 5
minutes.  Lars, WDYT?

Jens, I'll e-mail you separately.

Ted