[REVIEW] 16350 SHA1 may produce incorrect hashes if given a 2^29 byte input

[REVIEW] 16350 SHA1 may produce incorrect hashes if given a 2^29 byte input

[Bill Sommerfeld]

Issue: https://www.illumos.org/issues/16350
CR: https://code.illumos.org/c/illumos-gate/+/3451
Diff: 
https://code.illumos.org/~diff/86a2cb5a7f91f4cf23c522637a11a5e6c029e69e

A couple months ago I fixed a couple bugs in SHA2Update relating to
bookkeeping of large input blocks.  It turns out that SHA1Update had
similar code and had a subset of the bugs covered by #16317: the
64-bit build could miscompute the bit count if given a very large
input segment.

I'd like to submit a change that fixes this problem.

Background:

Like its ancestors MD4 and MD5, and its successor SHA2, the SHA1 hash
function is built on top of a compression function that takes a
starting hash state and a fixed-size input block and produces the next
hash state.

To accomodate inputs that aren't a multiple of the input block size
and to use the input length as part of the hash input, the input is
padded to a multiple of the input block size (64 bytes for SHA1) with
fixed pattern that ends with the length of the input, and fed through
a compression function one input block at a time to produce the
output.

The underlying cryptographic function is defined in terms of an input
bitstring rather than bytestring, and so the length encoded in the
padding is in bits, not bytes.

As the API is defined in terms of 8-bit bytes, the impementation must
do some shifts and multi-word arithmetic to convert from byte counts
to bit counts.

How we did that accounting was wrong when SHA1 was built for the
64-bit ABI as the low-half computation needs to be explicitly
truncated to 32 bits before checking for a carry out.

Testing notes are in the bug.