SSLHandshakeException or ClassCastException when using an HSM and Java 11 with ForgeRock products

Symptoms

In DS, you will see the following in the logs:

If you enable SSL debug logging, you will see one of the following errors depending on whether you are using TLS 1.3 or storing EC keys:

• Using TLS 1.3 with Java 11: javax.net.ssl.SSLHandshakeException: Cannot produce CertificateVerify signature    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:128)     at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)     at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)     at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:905)     at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.onProduceCertificateVerify(CertificateVerify.java:1077)    at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.produce(CertificateVerify.java:1070)     at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)     at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1224)     at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1160)     at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:849)     at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:810)     at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)     at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)     at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)     at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)     at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)     at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)     at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)     at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)     at TLSServer$ServerThread.run(TLSServer.java:92)  Caused by: java.security.InvalidKeyException: No installed provider supports this key: sun.security.pkcs11.P11Key$P11PrivateKey     at java.base/java.security.Signature$Delegate.chooseProvider(Signature.java:1163)     at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1204)     at java.base/java.security.Signature.initSign(Signature.java:546)     at java.base/sun.security.ssl.SignatureScheme.getSignature(SignatureScheme.java:473)     at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:895)
• Storing EC keys in the PKCS11 module and using Java 11: java.lang.ClassCastException: class sun.security.pkcs11.P11Key$P11PrivateKey cannot be cast to class java.security.interfaces.ECPrivateKey (sun.security.pkcs11.P11Key$P11PrivateKey is in module jdk.crypto.cryptoki of loader 'platform'; java.security.interfaces.ECPrivateKey is in module java.base of loader 'bootstrap')    at java.base/sun.security.ssl.SignatureScheme.getPreferableAlgorithm(SignatureScheme.java:436)     at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:867)     at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.onProduceCertificateVerify(CertificateVerify.java:1077)    at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.produce(CertificateVerify.java:1070)     at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)     at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1224)     at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1160)     at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:849)     at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:810)

You can enable SSL debugging as indicated in the following articles depending on your product:

• AM: FAQ: SSL/TLS secured connections in AM and Agents (Q. How do I debug SSL connection issues?)
• DS: How do I troubleshoot connection via LDAPS issues in DS (All versions)? (SSL debug logs section)
• IDM: FAQ: SSL certificates and secured connections in IDM (Q. How do I enable SSL debug logging?)

Recent Changes

Upgraded to Java 11.

Implemented an HSM via the PKCS11 module and kept the default protocol (TLS 1.3 is the default protocol used for a PKCS11 HSM).

Used EC keys stored in PKCS11.

Causes

There are two JDK bugs that are causing these issues:

• The SSLHandshakeException error occurs because TLS 1.3 does not work with PKCS11 in Java 11: JDK-8217611 (PKCS11 module unable to negotiate TLSv1.3 with RSASSA-PSS) (Fixed in Java 11.0.6)
• The ClassCastException error occurs because EC keys do not work with PKCS11 in Java 11, regardless of the TLS version used: JDK-8217610 (TLSv1.3 fail with ClassException when EC keys are stored in PKCS11) (Fixed in Java 11.0.6)

These errors are a result of bugs in the JDK rather than being issues with ForgeRock products.

Solution

This issue can be resolved by upgrading to Java 11.0.6 or later.

Workaround

If you cannot upgrade Java, you can work around these issues as follows:

• SSLHandshakeException error - you can either use TLS 1.2 or you can downgrade Java to JDK 1.8.
• ClassCastException error - the only workaround for this issue is to downgrade Java to JDK 1.8.

See Also

Hardware Security Modules (HSM) secret stores (AM)

Cryptographic keys (DS)

Hardware security module (HSM) (IDM)

Related Training

N/A

Related Issue Tracker IDs

OPENDJ-5761 (java.security.InvalidKeyException during SSL Handshake with keys in SoftHSM)