Solutions

SSLHandshakeException or ClassCastException when using an HSM and Java 11 with ForgeRock products

Last updated Jan 24, 2019

The purpose of this article is to provide assistance if you encounter an SSLHandshakeException or ClassCastException when using a hardware security module (HSM) via the PKCS11 module with Java® 11. This issue affects all ForgeRock products that are compatible with Java 11 (AM, DS, IDM and IG 6.5 and later).


Symptoms

In DS/OpenDJ, you will see the following in the logs:

Unable to connect to the server: 91 (Connect Error) Additional Information: The LDAP connection has failed because an error occurred during the SSL handshake: java.io.EOFException

If you enable SSL debug logging, you will see one of the following errors depending on whether you are using TLS 1.3 or storing EC keys:

  • Using TLS 1.3 with Java 11:
    javax.net.ssl.SSLHandshakeException: Cannot produce CertificateVerify signature 
       at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:128) 
       at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) 
       at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) 
       at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:905) 
       at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.onProduceCertificateVerify(CertificateVerify.java:1077)
       at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.produce(CertificateVerify.java:1070) 
       at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436) 
       at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1224) 
       at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1160) 
       at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:849) 
       at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:810) 
       at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) 
       at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) 
       at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) 
       at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178) 
       at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) 
       at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) 
       at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063) 
       at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) 
       at TLSServer$ServerThread.run(TLSServer.java:92) 
    Caused by: java.security.InvalidKeyException: No installed provider supports this key: sun.security.pkcs11.P11Key$P11PrivateKey 
       at java.base/java.security.Signature$Delegate.chooseProvider(Signature.java:1163) 
       at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1204) 
       at java.base/java.security.Signature.initSign(Signature.java:546) 
       at java.base/sun.security.ssl.SignatureScheme.getSignature(SignatureScheme.java:473) 
       at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:895) 
    
  • Storing EC keys in the PKCS11 module and using Java 11:
    java.lang.ClassCastException: class sun.security.pkcs11.P11Key$P11PrivateKey cannot be cast to class java.security.interfaces.ECPrivateKey (sun.security.pkcs11.P11Key$P11PrivateKey is in module jdk.crypto.cryptoki of loader 'platform'; java.security.interfaces.ECPrivateKey is in module java.base of loader 'bootstrap') 
       at java.base/sun.security.ssl.SignatureScheme.getPreferableAlgorithm(SignatureScheme.java:436) 
       at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:867) 
       at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.onProduceCertificateVerify(CertificateVerify.java:1077)
       at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.produce(CertificateVerify.java:1070) 
       at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436) 
       at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1224) 
       at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1160) 
       at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:849) 
       at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:810) 
    

You can enable SSL debugging as indicated in the following articles depending on your product:

Recent Changes

Upgraded to Java 11.

Implemented an HSM via the PKCS11 module and kept the default protocol (TLS 1.3 is the default protocol used for a PKCS11 HSM).

Used EC keys stored in PKCS11.

Causes

There are two JDK bugs that are causing these issues:

These errors are a result of bugs in the JDK rather than being issues with ForgeRock products.

Solution

Until the JDK bugs are fixed, you can workaround these issues as follows:

  • SSLHandshakeException error - you can either use TLS 1.2 or you can downgrade Java to JDK 1.8.
  • ClassCastException error - the only workaround for this issue is to downgrade Java to JDK 1.8.

See Also

Security Guide › Using a Hardware Security Module

Integrator's Guide › Configuring IDM For a Hardware Security Module (HSM) Device

How do I configure AM/OpenAM (All versions) to use a Hardware Security Module (HSM) for signing SAML assertions?

Federation related pages do not display in the console with a java.lang.NoClassDefFoundError: sun/misc/CharacterEncoder error in AM 6.5.x

Related Training

N/A

Related Issue Tracker IDs

OPENDJ-5761 (java.security.InvalidKeyException during SSL Handshake with keys in SoftHSM)



Copyright and TrademarksCopyright © 2019 ForgeRock, all rights reserved.
Loading...