FIPS: abort on rsa_keygen_pairwise_test failure

ISO 19790 AS10.09 says the module shall not perform any cryptographic
operations or output data in an error state, but OpenSSL does not have
checks for the module state in EVP_DigestUpdate() and
EVP_EncryptUpdate().

Upstream and their certification lab says these checks aren't needed,
our lab disagrees. We asked for clarification from CMVP. While we are
waiting for that, add a change that will allow us to submit. We will
drop this patch one we found a solution together with upstream.

See #22506 for the discussion upstream.

Resolves: RHEL-17104

0044-FIPS-140-3-keychecks.patch

@@ -374,3 +374,17 @@ diff -up openssl-3.0.7/providers/implementations/signature/rsa_sig.c.pairwise op
  const OSSL_DISPATCH ossl_rsa_signature_functions[] = {
      { OSSL_FUNC_SIGNATURE_NEWCTX, (void (*)(void))rsa_newctx },
      { OSSL_FUNC_SIGNATURE_SIGN_INIT, (void (*)(void))rsa_sign_init },
+diff --git a/crypto/rsa/rsa_gen.c b/crypto/rsa/rsa_gen.c
+index e0d139d..35f23b2 100644
+--- a/crypto/rsa/rsa_gen.c
++++ b/crypto/rsa/rsa_gen.c
+@@ -463,6 +463,9 @@ static int rsa_keygen(OSSL_LIB_CTX *libctx, RSA *rsa, int bits, int primes,
+             rsa->dmp1 = NULL;
+             rsa->dmq1 = NULL;
+             rsa->iqmp = NULL;
++#ifdef FIPS_MODULE
++            abort();
++#endif /* defined(FIPS_MODULE) */
+         }
+     }
+     return ok;

openssl.spec

@@ -532,6 +532,8 @@ ln -s /etc/crypto-policies/back-ends/openssl_fips.config $RPM_BUILD_ROOT%{_sysco
 * Tue Nov 21 2023 Dmitry Belyavskiy <dbelyavs@redhat.com> - 1:3.0.7-26
 - Avoid implicit function declaration when building openssl
   Related: RHEL-1780
+- In FIPS mode, prevent any other operations when rsa_keygen_pairwise_test fails
+  Resolves: RHEL-17104
 
 * Mon Oct 16 2023 Dmitry Belyavskiy <dbelyavs@redhat.com> - 1:3.0.7-25
 - Provide relevant diagnostics when FIPS checksum is corrupted