Currently, there are two different directories which contain internal
header files of libcrypto which are meant to be shared internally:

While header files in 'include/internal' are intended to be shared
between libcrypto and libssl, the files in 'crypto/include/internal'
are intended to be shared inside libcrypto only.

To make things complicated, the include search path is set up in such
a way that the directive #include "internal/file.h" could refer to
a file in either of these two directoroes. This makes it necessary
in some cases to add a '_int.h' suffix to some files to resolve this
ambiguity:

  #include "internal/file.h"      # located in 'include/internal'
  #include "internal/file_int.h"  # located in 'crypto/include/internal'

This commit moves the private crypto headers from

  'crypto/include/internal'  to  'include/crypto'

As a result, the include directives become unambiguous

  #include "internal/file.h"       # located in 'include/internal'
  #include "crypto/file.h"         # located in 'include/crypto'

hence the superfluous '_int.h' suffixes can be stripped.

The files 'store_int.h' and 'store.h' need to be treated specially;
they are joined into a single file.
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9681)

They were documented to take an EVP_MD pointer, when they really take
an EVP_MD_CTX pointer.

Fixes #9993
Reviewed-by: NDmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9997)

Found by OSS-Fuzz
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>

GH: #9959
(cherry picked from commit a6105ef40d65b35818f2b8ae8ca9e57ca6956d1d)

Currently the pkcs12 app will only ever print the first value of a multi-value
attribute. This is OK for some attributes (e.g. friendlyName, localKeyId) but
may miss values for other attributes.
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/9751)

(cherry picked from commit dbcc7b45670483cc89428afe1d3c363ef83d76df)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9889)

Reviewed-by: NDmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9830)

(cherry picked from commit 6ef03ea98fac501e6d6e33bac6ad3c92ea074712)

CLA: trivial
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9894)

(cherry picked from commit 64c1e74572f16a3e7c225f66fe85a3451ad39e68)

Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/9812)

(cherry picked from commit bd9faf117db1e53b2ad2ee9cbfe8def8c98ecb7b)

[extended tests]
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9888)

[extended tests]
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9886)

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9833)

(cherry picked from commit f28bc7d386b25fb75625d0c62c6b2e6d21de0d09)

There can be data to write in output buffer and data to read that were
not yet read in the input stream.

Fixes #9866
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9877)

(cherry picked from commit 6beb8b39ba8e4cb005c1fcd2586ba19e17f04b95)

Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9860)

Fixes #9858
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9860)

Reviewed-by: NRichard Levitte <levitte@openssl.org>

Reviewed-by: NRichard Levitte <levitte@openssl.org>

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9847)

Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9844)

An attack is simple, if the first CMS_recipientInfo is valid but the
second CMS_recipientInfo is chosen ciphertext. If the second
recipientInfo decodes to PKCS #1 v1.5 form plaintext, the correct
encryption key will be replaced by garbage, and the message cannot be
decoded, but if the RSA decryption fails, the correct encryption key is
used and the recipient will not notice the attack.

As a work around for this potential attack the length of the decrypted
key must be equal to the cipher default key length, in case the
certifiate is not given and all recipientInfo are tried out.

The old behaviour can be re-enabled in the CMS code by setting the
CMS_DEBUG_DECRYPT flag.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9777)

(cherry picked from commit 5840ed0cd1e6487d247efbc1a04136a41d7b3a37)

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9841)

Due to the dynamic allocation that was added to rand_pool_add_begin
this function could now return a null pointer where it was previously
guaranteed to succeed. But the return value of this function does
not need to be checked by design.

Move rand_pool_grow from rand_pool_add_begin to rand_pool_bytes_needed.
Make an allocation error persistent to avoid falling back to less secure
or blocking entropy sources.

Fixes: a6a66e4511ee ("Make rand_pool buffers more dynamic in their sizing.")
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9687)

(cherry picked from commit fa3eb248e29ca8031e6a14e8a2c6f3cd58b5450e)

There was a warning about unused variables in this config:
./config --strict-warnings --with-rand-seed=rdcpu
Reviewed-by: NMatthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9687)

(cherry picked from commit e301c147a763f67dcc5ba63eb7e2ae40d83a68aa)

Since commit 7c226dfc a chained DRBG does not add additional
data anymore when reseeding from its parent. The reason is that
the size of the additional data exceeded the allowed size when
no derivation function was used.

This commit provides an alternative fix: instead of adding the
entire DRBG's complete state, we just add the DRBG's address
in memory, thereby providing some distinction between the different
DRBG instances.
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9802)

Provides a little extra fork-safety on UNIX systems, adding to the
fact that all DRBGs reseed automatically when the fork_id changes.
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9802)

When the new OpenSSL CSPRNG was introduced in version 1.1.1,
it was announced in the release notes that it would be fork-safe,
which the old CSPRNG hadn't been.

The fork-safety was implemented using a fork count, which was
incremented by a pthread_atfork handler. Initially, this handler
was enabled by default. Unfortunately, the default behaviour
had to be changed for other reasons in commit b5319bdb, so
the new OpenSSL CSPRNG failed to keep its promise.

This commit restores the fork-safety using a different approach.
It replaces the fork count by a fork id, which coincides with
the process id on UNIX-like operating systems and is zero on other
operating systems. It is used to detect when an automatic reseed
after a fork is necessary.

To prevent a future regression, it also adds a test to verify that
the child reseeds after fork.

CVE-2019-1549
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9802)

Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NNicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/9821)

(cherry picked from commit 1d3cd983f56e0a580ee4216692ee3c9c7bf14de9)

crypto/rand/rand_win.c(70) : error C2065: 'BCRYPT_USE_SYSTEM_PREFERRED_RNG' : undeclared identifier
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9827)

(cherry picked from commit d3a1128bc25ec8bf835c81821e1be68fba39ab4b)

Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9796)

(cherry picked from commit fa01370f7dc8f0a379483bbe74de11225857e5fe)

so results were undefined.
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9796)

(cherry picked from commit 2b95e8efcf8b99892106070d9ac745a0a369f503)

For for G=2 and 5 DH_generate_parameters will continue to generate
the order 2q subgroup for compatibility with previous versions.

For G=3 DH_generate_parameters generates an order q subgroup, but it
will not pass the check in DH_check with previous OpenSSL versions.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9820)

(cherry picked from commit 65936a56461fe09e8c81bca45122af5adcfabb00)
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9813)

Description
-----------

Upon `EC_GROUP_new_from_ecparameters()` check if the parameters match any
of the built-in curves. If that is the case, return a new
`EC_GROUP_new_by_curve_name()` object instead of the explicit parameters
`EC_GROUP`.

This affects all users of `EC_GROUP_new_from_ecparameters()`:
- direct calls to `EC_GROUP_new_from_ecparameters()`
- direct calls to `EC_GROUP_new_from_ecpkparameters()` with an explicit
  parameters argument
- ASN.1 parsing of explicit parameters keys (as it eventually
  ends up calling `EC_GROUP_new_from_ecpkparameters()`)

A parsed explicit parameter key will still be marked with the
`OPENSSL_EC_EXPLICIT_CURVE` ASN.1 flag on load, so, unless
programmatically forced otherwise, if the key is eventually serialized
the output will still be encoded with explicit parameters, even if
internally it is treated as a named curve `EC_GROUP`.

Before this change, creating any `EC_GROUP` object using
`EC_GROUP_new_from_ecparameters()`, yielded an object associated with
the default generic `EC_METHOD`, but this was never guaranteed in the
documentation.
After this commit, users of the library that intentionally want to
create an `EC_GROUP` object using a specific `EC_METHOD` can still
explicitly call `EC_GROUP_new(foo_method)` and then manually set the
curve parameters using `EC_GROUP_set_*()`.

Motivation
----------

This has obvious performance benefits for the built-in curves with
specialized `EC_METHOD`s and subtle but important security benefits:
- the specialized methods have better security hardening than the
  generic implementations
- optional fields in the parameter encoding, like the `cofactor`, cannot
  be leveraged by an attacker to force execution of the less secure
  code-paths for single point scalar multiplication
- in general, this leads to reducing the attack surface

Check the manuscript at https://arxiv.org/abs/1909.01785 for an in depth
analysis of the issues related to this commit.

It should be noted that `libssl` does not allow to negotiate explicit
parameters (as per RFC 8422), so it is not directly affected by the
consequences of using explicit parameters that this commit fixes.
On the other hand, we detected external applications and users in the
wild that use explicit parameters by default (and sometimes using 0 as
the cofactor value, which is technically not a valid value per the
specification, but is tolerated by parsers for wider compatibility given
that the field is optional).
These external users of `libcrypto` are exposed to these vulnerabilities
and their security will benefit from this commit.

Related commits
---------------

While this commit is beneficial for users using built-in curves and
explicit parameters encoding for serialized keys, commit
b783beeadf6b80bc431e6f3230b5d5585c87ef87 (and its equivalents for the
1.0.2, 1.1.0 and 1.1.1 stable branches) fixes the consequences of the
invalid cofactor values more in general also for other curves
(CVE-2019-1547).

The following list covers commits in `master` that are related to the
vulnerabilities presented in the manuscript motivating this commit:

- d2baf88c43 [crypto/rsa] Set the constant-time flag in multi-prime RSA too
- 311e903d84 [crypto/asn1] Fix multiple SCA vulnerabilities during RSA key validation.
- b783beeadf [crypto/ec] for ECC parameters with NULL or zero cofactor, compute it
- 724339ff44 Fix SCA vulnerability when using PVK and MSBLOB key formats

Note that the PRs that contributed the listed commits also include other
commits providing related testing and documentation, in addition to
links to PRs and commits backporting the fixes to the 1.0.2, 1.1.0 and
1.1.1 branches.

This commit includes a partial backport of
https://github.com/openssl/openssl/pull/8555
(commit 8402cd5f75f8c2f60d8bd39775b24b03dd8b3b38)
for which the main author is Shane Lontis.

Responsible Disclosure
----------------------

This and the other issues presented in https://arxiv.org/abs/1909.01785
were reported by Cesar Pereida García, Sohaib ul Hassan, Nicola Tuveri,
Iaroslav Gridin, Alejandro Cabrera Aldaya and Billy Bob Brumley from the
NISEC group at Tampere University, FINLAND.

The OpenSSL Security Team evaluated the security risk for this
vulnerability as low, and encouraged to propose fixes using public Pull
Requests.

_______________________________________________________________________________
Co-authored-by: NShane Lontis <shane.lontis@oracle.com>

(Backport from https://github.com/openssl/openssl/pull/9808)
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9809)

While gcc ignores unknown options of the type '-Wno-xxx', clang by default issues
a warning [-Wunknown-warning-option] (see [3]), which together with '-Werror'
causes the build to fail. This turned out to be a problem on the 1.0.2 stable branch
in the case of the '-Wextended-offsetof' option, which was removed in version 6.0.0,
but needs to be kept here in order to support older clang versions, too (see #9446).

Incidentally, master and 1.1.1 branch already contained the -Wno-unknown-warning-option
option. Due to its special role and its importance, this commit adds an explaining
commit message and moves the option to the front.

[extended tests]
Reviewed-by: NRichard Levitte <levitte@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
Reviewed-by: NPaul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/9447)

(cherry picked from commit 03e5668343078b963cc6544ad7270743de13e514)

This amends the entry added in a6186f39
with the relevant CVE.
Reviewed-by: NMatt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9800)

This leaves VPAES and AESNI support.
The VPAES performance is comparable but BSAES is not
completely constant time. There are table lookups
using secret key data in AES_set_encrypt/decrypt_key
and in ctr mode short data uses the non-constant
time AES_encrypt function instead of bit-slicing.
Furthermore the AES_ASM is by far outperformed
by recent GCC versions.
Since BSAES calls back to AES_ASM for short
data blocks the performance on those is also
worse than the pure software implementaion.

Fixes: #9640
Reviewed-by: NRichard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9675)

Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NTomas Mraz <tmraz@fedoraproject.org>
Reviewed-by: NNicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/9781)

Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NTomas Mraz <tmraz@fedoraproject.org>
Reviewed-by: NNicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/9781)

The cofactor argument to EC_GROUP_set_generator is optional, and SCA
mitigations for ECC currently use it. So the library currently falls
back to very old SCA-vulnerable code if the cofactor is not present.

This PR allows EC_GROUP_set_generator to compute the cofactor for all
curves of cryptographic interest. Steering scalar multiplication to more
SCA-robust code.

This issue affects persisted private keys in explicit parameter form,
where the (optional) cofactor field is zero or absent.

It also affects curves not built-in to the library, but constructed
programatically with explicit parameters, then calling
EC_GROUP_set_generator with a nonsensical value (NULL, zero).

The very old scalar multiplication code is known to be vulnerable to
local uarch attacks, outside of the OpenSSL threat model. New results
suggest the code path is also vulnerable to traditional wall clock
timing attacks.

CVE-2019-1547
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NTomas Mraz <tmraz@fedoraproject.org>
Reviewed-by: NNicola Tuveri <nic.tuv@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/9781)

Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9511)

(cherry picked from commit 4fe2ee3a449a8ca2886584e221f34ff0ef5de119)

Replace flip_endian() by using the little endian specific
BN_bn2lebinpad() and BN_lebin2bn().
Reviewed-by: NMatt Caswell <matt@openssl.org>
Reviewed-by: NBernd Edlinger <bernd.edlinger@hotmail.de>
(Merged from https://github.com/openssl/openssl/pull/9511)

(cherry picked from commit e0b660c27d8d97b4ad9e2098cc957de26872c0ef)