Refactoring of mml1991.el, mml2015.el, mml-smime.el (was: Default encryption for Message)

[Jens Lechtenboerger <jens.lechtenboerger@fsfe.org>]

On 2014-09-21, at 12:08, Jens Lechtenboerger wrote:

> 5. Refactoring
> mml1991.el, mml2015.el, and mml-smime.el share code, forcing
> copy-and-paste on updates.

Hi there,

this is a continuation of a one year old thread.  Sorry for the delay.
Essentially, I proposed a refactoring of mml1991.el, mml2015.el, and
mml-smime.el such that common code can be maintained in a single file.

I did that now, with a new file mml-epg.el that is accompanied by
test cases, mml-epg-test.el.

My changes are as follows, please let me know what you think.


* Common code moved to mml-epg.el (from all three previous files)

Functionality concerning passphrase callback, code to check user IDs and
sub-keys for validity, code to find usable public and private keys,
common code from sign and encrypt functions.


* Treatment of common variables

There are variables in mml1991.el and mml2015.el for which I doubt that
users will really set them differently.  In fact, both files concern
OpenPGP, most likely (or for sure?) with GnuPG, and I fail to see
reasons to set them differently.  These are the variables I’m referring
to:
mml1991-cache-passphrase and mml2015-cache-passphrase,
mml1991-passphrase-cache-expiry and mml2015-passphrase-cache-expiry,
mml1991-signers and mml2015-signers,
mml1991-encrypt-to-self and mml2015-encrypt-to-self.

In my code, each pair is replaced with a single variable with the prefix
“mml-epg-openpgp”, e.g., “mml-epg-openpgp-cache-passphrase”.

Concerning S/MIME, I introduced the same variables with prefix
“mml-epg-smime”.

Also, two variables from mml2015.el and mml-smime.el were missing in
mml1991.el, whose functionality is present now with the common prefix
“mml-epg-openpgp”:
mml-epg-openpgp-sign-with-sender and mml-epg-openpgp-always-trust


* Treatment of encrypt-to-self

As discussed last year, encrypt-to-self in mml2015.el is implicitly
based on signature-related variables (based on undocumented use of
mml2015-sign-with-sender), which I find counter-intuitive.  I changed
the encrypt-to-self variables into independent ones: If the value is t,
then the message is also encrypted to the sender’s key.  Alternatively,
the value can also be a list of encryption keys (or corresponding e-mail
addresses), in which case the message is also encrypted to those keys.


* Minor (?) differences among previous files, now unified

In mml-smime-epg-passphrase-callback, password-cache-key-id is
not passed as argument to password-read, while such an argument is
passed in the corresponding functions in mml2015.el and mml1991.el.
Now that argument is passed also for S/MIME.
Is there a reason not to pass that argument?

In mml1991.el, prior to encryption, recipients were not enclosed in
ankle brackets "<" and ">", while such brackets were added in
mml2015.el.  Now they are added in all cases.


* Passphrase caching

If I’m not mistaken, with GnuPG 2.x (and gpgsm) passphrases cannot be
cached within Emacs as gpg-agent is started automatically and tries to
invoke a pinentry program.  So caching-related variables only apply to
GnuPG 1.x and S/MIME with openssl (I suggest not to advertise the
latter any longer, see separate point below).

In fact, one of my test cases
(mml-epg-en-decrypt-passphrase-no-cache-smime-todo)
has an expected failure for gpgsm, as the cache is not inspected.
Another test case (mml-epg-en-decrypt-passphrase-no-cache-openpgp-todo)
is skipped for GnuPG 2.x.
Is there a way to cache passphrases within Emacs, even with gpgsm and
GnuPG 2.x?

Note that GnuPG 2.x is where the development happens, and according to
GnuPG’s README, “2.0 is the current stable version for general use”,
while “1.4 is the old standalone version which is most suitable for
older or embedded platforms.”
So, if we cannot cache for the current stable version, should we not
recommend caching inside gpg-agent in general and get rid of the
associated code?  Even if there was a way, should we not recommend
caching inside gpg-agent in general?


* Selecting “usable” keys

In mml2015-epg-find-usable-key the test mml2015-epg-check-user-id is not
applied if the condition (string-match "\\`<" name) is true.
I don’t understand this, but I still copied the code to
mml-epg-find-usable-keys.  What is special about names starting with "<"
(except that they are used to indicate exact matches for e-mail
addresses in GnuPG)?
Why should revoked or expired user IDs be OK if they match that pattern?

In fact, one of my test cases (mml-epg-find-usable-keys-todo) is an
expected failure due to this behavior, where a revoked user IDs is
returned (also expired ones could be returned, if one knows how to
create them).

Moreover, the function mml2015-epg-find-usable-secret-key does not check
validity of subkeys requested by key ID.  Two of my test cases
(mml-epg-find-usable-keys-todo-2 and -3) fail due to this issue, when an
expired subkey is requested and the mainkey is returned.
Also, test case mml-epg-sign-verify-todo behaves in the wrong way due to
this issue.

Finally, function mml-epg-find-usable-key has an argument USAGE, which
can be `sign'.  However, as main keys can typically be used to sign
(by the key owner), almost every key is “usable” for USAGE `sign'
according to this function, although no sign operation can be performed
when the private key is missing.

I’ve written new functions mml-epg-check-sub-key-2 and
mml-epg-find-usable-keys-2 to address the above issues (with
corresponding test cases).  Note that mml-epg-find-usable-keys-2
subsumes mml-epg-find-usable-keys and mml-epg-find-usable-secret-keys
(which are based on mml2015.el).  The doc string explains the
differences.

Essentially, I propose to use mml-epg-find-usable-keys-2 (and changed
helper functions) instead of mml-epg-find-usable-keys and
mml-epg-find-usable-secret-keys.


* Choice of “usable” keys, now with customizable preferences

Previously, the “first” usable key (based on an undocumented order) was
chosen if multiple keys were available (for encryption as well as
signing).  My new code asks the user which one to use and allows to
customize that choice (new variable mml-epg-key-preferences).



* Questions while coding

In mml-epg-test.el two TODO comments occur, marking sections that need
verification.  One is discussed above (passphrase caching).  The second
one:

In my test code, I use several fixtures for test setup, sometimes
calling one fixture from another.  I ran into infinite loops, when I
called variables representing function bodies “body” in several
fixtures.  By renaming variables to have different names in different
fixtures, the code runs as I expected.  Why do those infinite loops
occur?  Missing understanding on my part or an Emacs bug?

Moreover, I added several TODO comments in mml-epg.el, related to above
points as well as to additional questions:

In mml-epg-select-keys-1, I use y-or-n-p to display a question
(let-binding message-truncate-lines to nil), but the question gets
truncated if called from an encryption context.
To see this, invoke mml-epg-select-preferred-keys-todo from
mml-epg-test.el (see README.txt for detailed instructions), mark one or
two keys, and press `C-c C-c'.  You’ll be asked whether to save that
choice, but the question is truncated.
How can I resize the echo area to display the entire message?
(With mml-epg-select-preferred-keys-ok you can see the entire message,
but outside an encryption context.)

Towards the end of mml-epg.jl three basic functions are defined, which
might be of independent interest.  Where should they go?

In mml-epg.el I use define-obsolete-variable-alias, probably in an
incorrect way.  Where/how should that be done to make sure that
potential user customizations of the obsolete variables carry over to
the new ones?


* Preference for openssl vs epg (gpgsm)

Currently, Gnus prefers openssl over epg (gpgsm), via
(defcustom mml-smime-use (if (featurep 'epg) 'epg 'openssl))
in mml-smime.el.  However, epg does not get loaded on its own even if it
is present.  Thus, users need to set mml-smime-use or require epg in
their ~/.emacs, but the manual does not mention gpgsm at all.
Where would be an appropriate place to (require 'epg)?
I propose to change that preference in favor of epg (gpgsm) as:
** Gpgsm manages certificates (storage, expiry, revocation).
   Users need to perform those tasks manually with openssl.
** Openssl has bugs as documented in the BUGS section of man smime(1).
   In particular: SMIMECapabilities are ignored, no revocation checking
   is done on the signer's certificate.
** Advertised SMIMECapabilities include broken encryption algorithms.
   With the precompiled openssl 1.0.1f on my system RC2 is advertised,
   which should have been dropped since S/MIME 3.x, see:
   https://tools.ietf.org/html/rfc5751#appendix-B


* Next steps

A tarball with my files is available over there (README.txt explains its
contents):
http://informationelle-selbstbestimmung-im-internet.de/emacs/mml-epg-2015-10-16.tar.bz2
http://informationelle-selbstbestimmung-im-internet.de/emacs/mml-epg-2015-10-16.tar.bz2.asc

What are your suggestions to continue from here?

Best wishes
Jens