CVE-2026-42770

Issue summary: When EVP_PKEY_derive_set_peer() is called with a DHX (X9.42)
peer key, the peer key is not properly checked for the subgroup membership.

Impact summary: A malicious peer which presents an X9.42 key carrying the
victim's p and g parameters, a forged q = r (a small prime factor of the
cofactor (p−1)/q_local), and a public value Y of order r can recover the
victim's private key after a small number of key exchange attempts.

When EVP_PKEY_derive_set_peer() is called with a DHX (X9.42) peer key, the
subgroup membership check Y^q ≡ 1 (mod p) is performed using the peer's
own q parameter, not the local key's q. The peer's domain parameters are
then matched against the domain parameters of the private key, but the value
of q is not compared.

A malicious peer who presents an X9.42 key carrying the victim's p, g,
a forged q = r (a small prime factor of the cofactor), and a public
value Y of order r passes all checks. The shared secret then takes only
r distinct values, leaking priv mod r. Repeating for each small-prime
factor of the cofactor and combining via CRT recovers the full private
key (Lim–Lee / small-subgroup-confinement attack).

The realistic attack surface is narrow: principally CMP deployments with
long-lived RA/CA DHX keys and bespoke enterprise or government applications
using X9.42 DHX static keys with interactive protocols and therefore this
issue was assigned Low severity.

The FIPS modules in 4.0, 3.6, 3.5, 3.4, 3.1.2 and 3.0 are affected by this
issue.