SSL in IDM

How do I limit the supported secure protocols and cipher suites in IDM 5.x and 6.x?

The purpose of this article is to provide information on disabling specific secure protocols and cipher suites in IDM. You may need to do this to remove an insecure protocol or address findings from a vulnerability scan.

Overview

Changes have been made in IDM 7 which mean only TLSv1.2 and TLSv1.3 are enabled by default. Additionally, the process for enabling and disabling protocols and ciphers has changed. You should refer to the documentation for IDM 7 and later: Installation Guide › Enable and Disable Secure Protocols and Cipher Suites.

Pre-IDM 7

By default, Jetty® supports a number of protocols and cipher suites.

To review what protocols and ciphers are supported, you can use tools such as sslscan or sslyze. These are third-party tools that we suggest can be used for information purposes but are not supported by ForgeRock.

Disabling secure protocols

The SSLv3 protocol is disabled by default because it is considered an obsolete and insecure protocol: This POODLE Bites: Exploiting The SSL 3.0 Fallback.

Support for the TLSv1.0 protocol has been removed in IDM 6; see the following PDF: Migrating from SSL and Early TLS from the PCI Security Standards Council for further information. Support for the TLSv1.1 protocol is deprecated in IDM 6 (and removed in IDM 7) due to a potential vulnerability: see CVE-2011-3389 from the National Vulnerability Database from the US National Institute of Standards and Technology for further information.

You can exclude protocols as follows:

Add any protocols you want to exclude to the ExcludeProtocols sections of sslContextFactory and sslContextFactoryMutualAuth in the jetty.xml file (located in the /path/to/idm/conf directory).
Restart IDM to apply these changes.

Example

The following example demonstrates excluding the TLSv1 and TLSv1.1 protocols in addition to the SSLv3 protocol:

Disabling cipher suites

You can exclude cipher suites as follows:

Add any cipher suites you want to exclude to the <Array id="excludedCipherSuites" type="java.lang.String"> section in the jetty.xml file (located in the /path/to/idm/conf directory). There are numerous excluded cipher suites by default.
Restart IDM to apply these changes.

Example

The following example demonstrates excluding the TLS_RSA_WITH_AES_128_CBC_SHA cipher suite:

<Array id="excludedCipherSuites" type="java.lang.String"> <Item>TLS_RSA_WITH_AES_128_CBC_SHA</Item>  <Item>SSL_RSA_EXPORT_WITH_RC4_40_MD5</Item> <Item>SSL_DH_anon_EXPORT_WITH_RC4_40_MD5</Item> <Item>TLS_KRB5_EXPORT_WITH_RC4_40_MD5</Item>  <Item>SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA</Item> ... </Array>

Related Training

N/A

Related Issue Tracker IDs

N/A

How do I disable TLS Client-Initiated Renegotiation in IDM (All versions)?

The purpose of this article is to provide information on disabling secure client-initiated renegotiation in IDM. You may want to do this to address findings from a vulnerability scan.

Disabling TLS renegotiation

Java® includes the jdk.tls.rejectClientInitiatedRenegotiation system property, which controls TLS client-initiated renegotiation.

You can disable client-initiated renegotiation using the OPENIDM_OPTS environment variable:

On Unix® and Linux® systems: $ cd /path/to/idm/ $ export OPENIDM_OPTS="-Djdk.tls.rejectClientInitiatedRenegotiation=true" $ ./startup.sh
On Microsoft® Windows® systems: C:\> cd \path\to\idm C:\path\to\idm> set OPENIDM_OPTS=-Djdk.tls.rejectClientInitiatedRenegotiation=true C:\path\to\idm> startup.bat

Note

You can also edit the startup.sh or startup.bat files to update the default OPENIDM_OPTS values.

Related Training

N/A

Related Issue Tracker IDs

N/A

How do I connect to IDM (All versions) with mutual SSL authentication from IG (All versions)?

The purpose of this article is to provide information on connecting to IDM with mutual SSL authentication from IG using the client certificate module (CLIENT_CERT).

Connecting to IDM with mutual SSL authentication

The following process guides you through connecting to IDM with mutual SSL authentication from IG, where IDM is the server and IG is the client:

Configure the CLIENT_CERT module in IDM per Security Guide › Authentication and Session Modules ensuring you update your authentication.json file (located in the /path/to/idm/conf directory) to include the module.
Create a keystore for IG that contains the IG private key and certificate. See Gateway Guide › Configuring IG For HTTPS (Client-Side) for further information.
Import the IG certificate into the IDM truststore to allow IDM to trust IG. See Security Guide › Test Client Certificate Authentication for an example.
Import the IDM server certificate (openidm-localhost) into the IG truststore to allow IG to trust IDM.
Create a custom endpoint for mutual SSL authentication purposes. See Scripting Guide › Create Custom Endpoints to Launch Scripts for further information on custom endpoints.
Update the access.json file (IDM 7 and later - located in /path/to/idm/conf) or the access.js file (Pre-IDM 7 - located in /path/to/idm/script) to allow the internal/role/openidm-cert role to access the custom endpoint you created. The internal/role/openidm-cert role is the default role for any user authenticated with mutual SSL authentication. For example, if your custom endpoint is endpoint/echo, the addition needed in the access.json file or the access.js file would be similar to this: { "pattern" : "endpoint/echo", "roles" : "internal/role/openidm-cert", "methods" : "*", "actions" : "*" }, In pre-IDM 6.5, you do not need to include the full path; you can just refer to the role name, for example, openidm-cert.
Create a route in IG that includes a ClientTlsOptions (IG 7 and later), TlsOptions (IG 6.5) or a keyManager (pre-IG 6.5) for the ClientHandler that is configured with the IG keystore. For example, your route may look similar to this with a ClientTlsOptions object: { "handler": { "type": "Chain", "config": { "filters" : [ { "name": "OpenIDMuserName", "type": "HeaderFilter", "config": { "messageType": "REQUEST", "remove": ["X-OpenIDM-Username"], "add" : { "X-OpenIDM-Username": [ "anonymous" ] } } }, { "name": "OpenIDMpassword", "type": "HeaderFilter", "config": { "messageType": "REQUEST", "remove": ["X-OpenIDM-Password"], "add" : { "X-OpenIDM-Password": [ "anonymous" ] } } } ], "handler": { "type" : "ClientHandler", "config": { "hostnameVerifier": "STRICT", "tls": { "type": "ClientTlsOptions", "config": { "sslContextAlgorithm": "TLSv1.2", "keyManager": { "type": "KeyManager", "config": { "keystore": { "type": "KeyStore", "config": { "url": "file://${env['HOME']}/keystore.jks", "passwordSecretId": "keymanager.keystore.secret.id", "secretsProvider": "SystemAndEnvSecretStore" } }, "passwordSecretId": "keymanager.secret.id", "secretsProvider": "SystemAndEnvSecretStore" } } } } } }, "capture" : "all" } }, "condition": "${matches(request.uri.path, '^/openidm/endpoint/echo')}", "baseURI": "https://idm.example.com:18444", "capture" : "all", "timer" : true } See Gateway Guide › Configuring Routes for further information.
Update the openidm.auth.clientauthonlyport property in the boot.properties file (located in install-dir/resolver/ (IDM 6 and later) or /path/to/idm/conf/boot/ (pre-IDM 6)) to include the IG port (and ports for any other clients that are authorized to perform authentication). You can have a list of ports, for example: openidm.auth.clientauthonlyports=18444,9090
Restart IDM.

Header host filter

As an alternative to specifying authorized ports in step 8, you can add a filter to the IG route to replace the header host with the IDM host and port, for example:

{ "name": "HostFilter", "type": "HeaderFilter", "config": { "messageType": "REQUEST", "remove": "host", "add": { "host": "idm.example.com:18444" } }

It is up to you whether you choose to make the configuration change in IDM or IG.

Related Training

N/A

Related Issue Tracker IDs

N/A

Frequently Asked Questions

FAQ: SSL certificates and secured connections in IDM

The purpose of this FAQ is to provide answers to commonly asked questions regarding SSL certificate management and SSL/TLS secured connections in IDM.

Frequently asked questions

Q. What is the difference between the keystore and truststore?
Q. How do I import a DS certificate into the IDM truststore?
Q. Why am I seeing SLHandshakeException: unable to find valid certification path errors?
Q. How do I enable SSL debug logging?
Q. What are the javax.net.ssl.keyStore and javax.net.ssl.trustStore properties used for?
Q. Can I change which SSL/TLS ciphers are used by IDM?

Q. What is the difference between the keystore and truststore?

A. Truststores are used for storing trusted public certificate entries, CA and Self-Signed Certificates that you trust to validate the identity of the certificate presented to the server, and keystores are used for storing private keys; the truststore is used to find the certificates of trusted servers/clients when a remote party presents its certificate to the IDM server, whereas the keystore contains the encryption keys used by IDM.

Q. How do I import a DS certificate into the IDM truststore?

A. You can import a DS certificate into the IDM truststore by following the steps in Password Synchronization Plugin Guide › Enable IDM to trust DS Certificates.

Q. Why am I seeing SLHandshakeException: unable to find valid certification path errors?

A. If you see certificate errors such as the following in your logs:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This means that:

IDM is not pointing to the truststore, OR
The certificate presented by the client/server does not match the one in the truststore.

To resolve this issue, you should verify that the relevant certificate, and any intermediary and root certificates from the CA have been imported into the truststore location that's defined in the secrets.json file (IDM 6.5 and later) or the boot.properties file (pre-IDM 6.5).

See Security Guide › Managing Secret Stores, Certificates and Keys for further information.

If the error persists once you have verified your setup appears correct, you should enable SSL debug logging to investigate this issue further.

Q. How do I enable SSL debug logging?

A. SSL debugging traces the SSL handshaking phase. You can enable SSL debugging via the OPENIDM_OPTS environment variable.

On Unix® and Linux® systems:

$ cd /path/to/idm/ $ export OPENIDM_OPTS="-Djavax.net.debug=all" $ ./startup.sh

On Microsoft® Windows® systems:

C:\> cd \path\to\idm C:\path\to\idm> set OPENIDM_OPTS=-Djavax.net.debug=all C:\path\to\idm> startup.bat

Note

You can also edit the startup.sh or startup.bat files to update the default OPENIDM_OPTS values.

Q. What are the javax.net.ssl.keyStore and javax.net.ssl.trustStore properties used for?

A. These properties allow you to specify the exact location of the keystore or truststore. These properties are set by IDM, but there can be a timing issue where the connection is made ahead of the properties being set, which can prevent the server finding the necessary certificates or keys. Therefore, it is sometimes necessary to set these properties, especially if you are experiencing certificate errors but have already verified that the certificate exists in the correct location and IDM is pointing to it.

You can set these properties as follows:

Add the following to the system.properties file (located in the /path/to/idm/conf directory) depending on which store is needed:
- Specify the location of the keystore: javax.net.ssl.keyStore=[keystore_location] javax.net.ssl.keyStorePassword=[keystore_passwword]
- Specify the location of the truststore: javax.net.ssl.trustStore=[truststore_location] javax.net.ssl.trustStorePassword=[truststore_passwword]
Restart IDM.

Note

You can also edit the startup.sh or startup.bat files to update the JAVA_OPTS value to include these properties, for example, using the format: -Djavax.net.ssl.keyStore etc.

Q. Can I change which SSL/TLS ciphers are used by IDM?

A. Yes you can. See Installation Guide › Enable and Disable Secure Protocols and Cipher Suites (IDM 7 and later) or How do I limit the supported secure protocols and cipher suites in IDM 5.x and 6.x? for further information.

Related Training

N/A

Related Issue Tracker IDs

N/A

Known Issues

AM 5, 5.5, 5.5.1 and 6.0.0.x, IDM 6.x and Rest2LDAP cannot connect to DS 5, DS 5.5, DS 5.5.1, DS 5.5.2 or 6 after restricting DS cipher suites or Java upgrade

The purpose of this article is to provide assistance if connections to DS fail with a java.io.EOFException after restricting cipher suites in DS, installing DS using Production mode or upgrading to Java® JDK 11 or JDK 1.8.0_192 (or later). This issue affects various use cases including AM connecting to a DS configuration store, identity store, CTS store and so on; IDM connecting to an external DS repository or connections to DS using the Rest2LDAP gateway servlet.

Symptoms

A java.io.EOFException occurs before an SSL connection is established. You will see this error occurring in different logs depending on what is trying to connect to DS.

AM

If you are using DS as an identity store, you will notice that you cannot see any identities in the repository.

You will also see the following Connect Errors in the CoreSystem, IdRepo or Session debug logs depending on what type of stores you are using DS for (configuration, identity or CTS):

Connect Error: No operational connection factories available at org.forgerock.opendj.ldap.LdapException.newLdapException(LdapException.java:206) at org.forgerock.opendj.ldap.LdapException.newLdapException(LdapException.java:144) at org.forgerock.opendj.ldap.LdapException.newLdapException(LdapException.java:113) at org.forgerock.opendj.ldap.LoadBalancer.getMonitoredLdapClient(LoadBalancer.java:411) at org.forgerock.opendj.ldap.LoadBalancer.access$3300(LoadBalancer.java:68) at org.forgerock.opendj.ldap.LoadBalancer$LdapClientSocketImpl.getLoadBalancedContext(LoadBalancer.java:787) at org.forgerock.opendj.ldap.LoadBalancer$LdapClientSocketImpl.lambda$getLoadBalancedContext$4(LoadBalancer.java:797) ... Caused by: Connect Error: The LDAP connection has failed because an error occurred during the SSL handshake: java.io.EOFException at org.forgerock.opendj.ldap.LdapException.newLdapException(LdapException.java:206) at org.forgerock.opendj.ldap.LdapException.newLdapException(LdapException.java:144) at org.forgerock.opendj.ldap.LdapException.newLdapException(LdapException.java:113) at org.forgerock.opendj.grizzly.GrizzlyLdapSocketConnector$CompletionHandlerAdapter$1.failed(GrizzlyLdapSocketConnector.java:274) ... 28 more Caused by: java.io.EOFException ... 27 more

IDM

You will see the following errors in openidm0.log after starting IDM:

May 28, 2019 2:27:01 PM org.glassfish.grizzly.filterchain.DefaultFilterChain execute FEIN: GRIZZLY0013: Exception during FilterChain execution java.io.EOFException at org.glassfish.grizzly.nio.transport.TCPNIOTransport.read(TCPNIOTransport.java:630) at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:75) at org.glassfish.grizzly.filterchain.TransportFilter.handleRead(TransportFilter.java:173) at org.glassfish.grizzly.ssl.SSLBaseFilter$SSLTransportFilterWrapper.handleRead(SSLBaseFilter.java:1145) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) ... May 28, 2019 2:27:01 PM org.forgerock.openidm.repo.opendj.impl.OpenDJRepoService waitForConnection WARNUNG: Unable to get DS connection. Next attempt at Tues May 28 14:28:01 CEST 2019

Rest2LDAP

You will see connect errors similar to the following on the Rest2LDAP side if logging has been enabled:

29-May-2019 15:07:42.780 WARNING [OpenDJ LDAP SDK Client Selector(1) SelectorRunner] org.forgerock.i18n.slf4j.LocalizedLogger.warn Connection factory 'CachedConnectionPool(size=0[in:0 + out:0 + pending:0], maxSize=24, blocked=0, ldapClient=org.forgerock.opendj.ldap.LdapClientImpl@2517fc05)' is no longer operational: Connect Error

You can enable logging in the Rest2LDAP gateway servlet (if it runs in the Apache Tomcat™ web container) by specifying logging properties in the logging.properties file (located in the WEB-INF/classes directory where the gateway servlet is deployed).

SSL debugging - AM

If you enable SSL debug logging for AM, you will just see that the secure ciphers are being ignored, for example:

Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

You can enable SSL debug logging as described in FAQ: SSL/TLS secured connections in AM and Agents (Q. How do I debug SSL connection issues?).

SSL debugging - DS

If you enable SSL debug logging for DS, you may see errors similar to the following in the DS server.out log:

%% Initialized: [Session-7, SSL_NULL_WITH_NULL_NULL] LDAPS 0.0.0.0 port 40636(2) SelectorRunner, fatal error: 40: no cipher suites in common javax.net.ssl.SSLHandshakeException: no cipher suites in common %% Invalidated: [Session-7, SSL_NULL_WITH_NULL_NULL] LDAPS 0.0.0.0 port 40636(2) SelectorRunner, SEND TLSv1.2 ALERT: fatal, description = handshake_failure LDAPS 0.0.0.0 port 40636(2) SelectorRunner, WRITE: TLSv1.2 Alert, length = 2 LDAPS 0.0.0.0 port 40636(2) SelectorRunner, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: no cipher suites in common Using SSLEngineImpl.

You can enable SSL debug logging as described in FAQ: SSL certificate management in DS 5.x or 6.x (Q. How do I debug a SSL handshake error?).

Additionally, the server.out log does not show the more secure cipher suites that have been enabled.

Recent Changes

Upgraded to JDK 1.8.0_192 or later, or JDK 11.

Restricted cipher suites in DS to more secure ones such as TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, which use protocol version TLSv1.2.

Installed DS using the --productionMode setup option (which restricts the available ciphers and forces the protocol version to TLSv1.2).

Causes

AM connects to DS using the embedded LDAP SDK and IDM connects to DS using the Grizzly LDAP connector (which in turn uses the Rest2LDAP servlet).

These mechanisms for connecting to DS securely do not correctly support the TLSv1.2 protocol, whereas using the more secure ciphers and installing DS in production mode forces the protocol version to TLSv1.2. In Java versions prior to JDK 1.8.0_192, unsupported cipher suites were simply ignored, which is why these connections continued to work despite the mismatch in protocol versions.

Changes were introduced in JDK 1.8.0_192 (JDK-8162362 : Introduce system property to control enabled ciphersuites), which changed how the JDK determined which cipher suites (and resulting protocol) to use. As a consequence of these Java changes, AM, IDM and the Rest2LDAP cannot communicate with DS using a SSL/TLS secured connection because it uses different cipher suites and protocol to the DS server; both the client and server must support the same cipher suites and protocol agreed upon when attempting to establish a secure connection.

Solution

This issue can be resolved by as follows:

If you are experiencing this specific issue with AM connecting to DS, this can be resolved by upgrading to AM 6.5 or later, or AM 5.5.2; you can download this from BackStage.
If you are experiencing this specific issue with IDM or Rest2LDAP connecting to DS, this can be resolved by upgrading to DS 6.5 or later, or DS 5.5.3; you can download this from BackStage.

Workaround

You can workaround this issue using one of the following options:

Downgrade to JDK 1.8.0_191 or earlier.
Add one or more cipher suites that use the TLSv1.1 or TLSv1.0 protocol version, for example: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, and so on. See Security Guide › TLS Protocols and Cipher Suites for further information. You should review any security risks of using alternative cipher suites before proceeding.

Note

Java 8 does not support keys with 256-bit AES encryption by default, so you may need to install the Oracle® Java JCE unlimited strength jars if you want to use keys with 256-bit AES encryption. See Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files Download for further information and links to download the jars.

Related Training

N/A

Related Issue Tracker IDs

OPENAM-14986 (AM 6 Cannot connect to TLSv1.2 DJ server (production mode) after JDK 8 update 192)

OPENIDM-11152 (Unable to connect to External DS datastore via TLS/SSL)

OPENDJ-5553 (Rest2Ldap cannot connect to TLSv1.2 servers)

Authentication fails with IDM 5.x or 6.x integrated with AM when session-jwt cookie size exceeds browser limits

The purpose of this article is to provide assistance if authentication fails in an environment where IDM is integrated with AM using the OAUTH_CLIENT module (IDM 5.5 and later) or the OPENID_CONNECT module (IDM 5).

Symptoms

Authentication fails.

If you examine network traffic using your browser's Developer Tools or by capturing a HAR file, you will notice the following:

The /openidm/authentication POST call is successful (200 OK response) and you can see the response to this request contains a Set-Cookie for session-jwt.
The session-jwt cookie set is larger than 4096 bytes.
The subsequent /openidm/managed/user/userID GET call fails (403 Forbidden response) and the session-jwt cookie is not set (null).

You can capture a HAR file as described in: How do I create a HAR file for troubleshooting IDM (All versions)?

Recent Changes

Changed the default alias (openidm-localhost) for the encryption key used by the Jetty® web server to service SSL requests.

Assigned additional authorization roles to the user(s) who cannot authenticate.

Causes

The session-jwt cookie is larger than 4096 bytes, which is the limit for most browsers per RFC 6265. As a result, IDM sends the session-jwt cookie correctly and it's received by the browser (shown in the POST call), but the browser does not save the cookie nor send it in the next request (shown in the GET call), which causes the authentication flow to fail.

The session-jwt cookie contains a lot of information and certain items can cause it to be too big, including (but not limited to):

Public key length, particularly if the public key in use exceeds the default length of 2048 bit.
Number of authorization roles, particularly if the user has more than one authorization role assigned to them.
Length of realm names and/or use of sub-realms as this results in longer URLs within the cookie.

Public key length

If you're unsure what the public key length is in your environment, you can check it as follows providing you have Java® keytool and openssl installed:

Check the alias of the certificate currently in use as follows depending on your version (it's set to openidm-localhost in a default installation):
- IDM 6.5.x: look at the alias specified for the idm.jwt.session.module.encryption secret in the secrets.json file (located in /path/to/idm/conf): { "secretId" : "idm.jwt.session.module.encryption", "types": [ "ENCRYPT", "DECRYPT" ], "aliases": [ "&{openidm.https.keystore.cert.alias|openidm-localhost}" ] },
- Pre-IDM 6.5: look for the value of the following property in the boot.properties file: openidm.https.keystore.cert.alias
Navigate to the /path/to/idm/security directory.
Run the following keytool command to return the public key, ensuring you substitute your alias, password and keystore details as needed: $ keytool -list -rfc -storetype JCEKS -keystore keystore.jceks -storepass changeit -alias openidm-localhost | openssl x509 -inform pem -pubkey This will give you an output similar to this: -----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAkOgv6oQT5bKk... kwIDAQAB <SNIP> -----END PUBLIC KEY-----
Save this output (including the headers) to a text file.
Run the following command referencing the file you just created: $ openssl rsa -noout -text -inform PEM -in pubkey.txt -pubin This will give you an output similar to this, where the public key length is shown (2048 bit in this example): Public-Key: (2048 bit) Modulus: 00:90:e8:2f:ea:84:13:e5:b2:a4:5a:81:56:05:04: <SNIP> 5d:93 Exponent: 65537 (0x10001)

Solution

This issue can be resolved by reducing the length of the session-jwt cookie to below 4096 bytes.

Depending on what caused the large session-jwt cookie, the following suggestions may help:

Authentication and Single Sign-On Guide › Planning for Client-Based Sessions
Public key length - use a 2048 bit public key.
Number of authorization roles - use only a single authorization role.
Length of realm names and/or use of sub-realms - use the AM DNS alias functionality and then use the DNS alias in all URLs on the IDM side.

Related Training

N/A

Related Issue Tracker IDs

OPENIDM-11055 (Full Stack: in some configurations, may need to increase the default header size)

OPENAM-13165 (Full Stack: in some configurations, may need to increase the default header size)

SSLHandshakeException or ClassCastException when using an HSM and Java 11 with ForgeRock products

The purpose of this article is to provide assistance if you encounter an SSLHandshakeException or ClassCastException when using a Hardware Security Module (HSM) via the PKCS11 module with Java® 11 (11.0.1 to 11.0.5). This issue affects all ForgeRock products that are compatible with Java 11 (AM, DS, IDM and IG 6.5 and later).

Symptoms

In DS/OpenDJ, you will see the following in the logs:

Unable to connect to the server: 91 (Connect Error) Additional Information: The LDAP connection has failed because an error occurred during the SSL handshake: java.io.EOFException

If you enable SSL debug logging, you will see one of the following errors depending on whether you are using TLS 1.3 or storing EC keys:

Using TLS 1.3 with Java 11: javax.net.ssl.SSLHandshakeException: Cannot produce CertificateVerify signature at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:128) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:905) at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.onProduceCertificateVerify(CertificateVerify.java:1077) at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.produce(CertificateVerify.java:1070) at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436) at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1224) at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1160) at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:849) at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:810) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) at TLSServer$ServerThread.run(TLSServer.java:92) Caused by: java.security.InvalidKeyException: No installed provider supports this key: sun.security.pkcs11.P11Key$P11PrivateKey at java.base/java.security.Signature$Delegate.chooseProvider(Signature.java:1163) at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1204) at java.base/java.security.Signature.initSign(Signature.java:546) at java.base/sun.security.ssl.SignatureScheme.getSignature(SignatureScheme.java:473) at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:895)
Storing EC keys in the PKCS11 module and using Java 11: java.lang.ClassCastException: class sun.security.pkcs11.P11Key$P11PrivateKey cannot be cast to class java.security.interfaces.ECPrivateKey (sun.security.pkcs11.P11Key$P11PrivateKey is in module jdk.crypto.cryptoki of loader 'platform'; java.security.interfaces.ECPrivateKey is in module java.base of loader 'bootstrap') at java.base/sun.security.ssl.SignatureScheme.getPreferableAlgorithm(SignatureScheme.java:436) at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyMessage.<init>(CertificateVerify.java:867) at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.onProduceCertificateVerify(CertificateVerify.java:1077) at java.base/sun.security.ssl.CertificateVerify$T13CertificateVerifyProducer.produce(CertificateVerify.java:1070) at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436) at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1224) at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1160) at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:849) at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:810)

You can enable SSL debugging as indicated in the following articles depending on your product:

AM/OpenAM: FAQ: SSL/TLS secured connections in AM and Agents (Q. How do I debug SSL connection issues?)
DS/OpenDJ: FAQ: SSL certificate management in DS 5.x or 6.x (Q. How do I debug a SSL handshake error?)
IDM/OpenIDM: FAQ: SSL certificates and secured connections in IDM (Q. How do I enable SSL debug logging?)

Recent Changes

Upgraded to Java 11.

Implemented an HSM via the PKCS11 module and kept the default protocol (TLS 1.3 is the default protocol used for a PKCS11 HSM).

Used EC keys stored in PKCS11.

Causes

There are two JDK bugs that are causing these issues:

The SSLHandshakeException error occurs because TLS 1.3 does not work with PKCS11 in Java 11: JDK-8217611 (PKCS11 module unable to negotiate TLSv1.3 with RSASSA-PSS) (Fixed in Java 11.0.6)
The ClassCastException error occurs because EC keys do not work with PKCS11 in Java 11, regardless of the TLS version used: JDK-8217610 (TLSv1.3 fail with ClassException when EC keys are stored in PKCS11) (Fixed in Java 11.0.6)

These errors are a result of bugs in the JDK rather than being issues with ForgeRock products.

Solution

This issue can be resolved by upgrading to Java 11.0.6 or later.

Workaround

If you cannot upgrade Java, you can workaround these issues as follows:

SSLHandshakeException error - you can either use TLS 1.2 or you can downgrade Java to JDK 1.8.
ClassCastException error - the only workaround for this issue is to downgrade Java to JDK 1.8.

Related Training

N/A

Related Issue Tracker IDs

OPENDJ-5761 (java.security.InvalidKeyException during SSL Handshake with keys in SoftHSM)

¯\_(ツ)_/¯

Page not found

Overview

Disabling secure protocols

Disabling cipher suites

See Also

Related Training

Related Issue Tracker IDs

Disabling TLS renegotiation

Note

See Also

Related Training

Related Issue Tracker IDs

Connecting to IDM with mutual SSL authentication

See Also

Related Training

Related Issue Tracker IDs

Frequently Asked Questions

Frequently asked questions

Q. What is the difference between the keystore and truststore?

Q. How do I import a DS certificate into the IDM truststore?

Q. Why am I seeing SLHandshakeException: unable to find valid certification path errors?

Q. How do I enable SSL debug logging?

Note

Q. What are the javax.net.ssl.keyStore and javax.net.ssl.trustStore properties used for?

Note

Q. Can I change which SSL/TLS ciphers are used by IDM?

See Also

Related Training

Related Issue Tracker IDs

Known Issues

Symptoms

Recent Changes

Causes

Solution

Note

See Also

Related Training

Related Issue Tracker IDs

Symptoms

Recent Changes

Causes

Solution

See Also

Related Training

Related Issue Tracker IDs

Symptoms

Recent Changes

Causes

Solution

See Also

Related Training

Related Issue Tracker IDs