No secret with id storepass for purpose storepass errors after upgrading to AM 6.5.x or 7.x

Symptoms

After successfully upgrading or installing AM, your servers start up but you encounter the following error on one or more of your upgraded/new servers: Caused by: org.forgerock.secrets.NoSuchSecretException: No secret with id storepass for purpose storepass   at org.forgerock.secrets.propertyresolver.PropertyResolverSecretStore.lambda$getNamed$2(PropertyResolverSecretStore.java:109)    at java.base/java.util.Optional.orElseGet(Optional.java:369)    at org.forgerock.secrets.propertyresolver.PropertyResolverSecretStore.getNamed(PropertyResolverSecretStore.java:107)    at org.forgerock.secrets.propertyresolver.PropertyResolverSecretStore.getActive(PropertyResolverSecretStore.java:96)

You may also notice this error occurring in a number of situations, such as when trying to retrieve an OIDC token or an OAuth2 access token.

A similar issue can occur that causes SAML2 federation to fail after upgrading or installing AM, and you see errors such as the following in your debug logs:

• No secret configured for purpose am.global.services.saml2.client.storage.jwt.encryption:org.forgerock.openam.secrets.SecretsProviderFacade:21/12/2021 12:28:34:530 PM GMT: Thread[https-jsse-nio-8443-exec-27,5,main]: TransactionId[c9ce698b-4a32-5057-8f10-2b1854f6f67e-613] Not resolved at realm level: Purpose{secretType=DataEncryptionKey, label='am.global.services.saml2.client.storage.jwt.encryption'} org.forgerock.secrets.NoSuchSecretException: No secret configured for purpose am.global.services.saml2.client.storage.jwt.encryption   at org.forgerock.secrets.keystore.KeyStoreSecretStore.getActive(KeyStoreSecretStore.java:249)    at org.forgerock.secrets.SecretsProvider.lambda$getActiveSecret$3(SecretsProvider.java:138)    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
• Failed to load secret store default-keystore using the currently available secrets:org.forgerock.openam.secrets.Secrets:21/12/2021 12:55:28:964 PM GMT: Thread[http-bio-17171-exec-12,5,main]: TransactionId[822fc530-43fc-963e-b660-dc0c1c20b827-101100] Failed to load secret store default-keystore using the currently available secrets java.lang.IllegalArgumentException: Unable to initialise keystore using provided parameters   at org.forgerock.secrets.keystore.KeyStoreSecretStore.<init>(KeyStoreSecretStore.java:104)    at org.forgerock.openam.secrets.config.KeyStoreSecretStore.createStore(KeyStoreSecretStore.java:162)    at org.forgerock.openam.secrets.config.KeyStoreBasedSecretStoreProvider.getStore(KeyStoreBasedSecretStoreProvider.java:50)    at org.forgerock.openam.secrets.config.KeyStoreBasedSecretStoreProvider.getStore(KeyStoreBasedSecretStoreProvider.java:38)    at org.forgerock.openam.secrets.Secrets.resolveSecretStores(Secrets.java:245)    at org.forgerock.openam.secrets.Secrets.loadSecretStores(Secrets.java:227)    at org.forgerock.openam.secrets.Secrets.loadGlobalSecretStores(Secrets.java:192) ... Caused by: org.forgerock.secrets.NoSuchSecretException: No secret with id storepass for purpose storepass
• Unable to load all configured secret stores:org.forgerock.openam.secrets.Secrets:21/12/2021 12:46:25:245 PM GMT: Thread[http-bio-17171-exec-3,5,main]: TransactionId[3908158e-cefb-b125-4842-e64f6cfb3ff8-13782] ERROR: Unable to load all configured secret stores, last failure was: org.forgerock.openam.secrets.SecretInitialisationException: Unable to load keystore   at org.forgerock.openam.secrets.config.KeyStoreSecretStore.lambda$createStore$4(KeyStoreSecretStore.java:147)    at org.forgerock.secrets.GenericSecret.lambda$revealAsText$0(GenericSecret.java:120)    at org.forgerock.secrets.GenericSecret.reveal(GenericSecret.java:82) ... Caused by: java.lang.IllegalStateException: Unable to load keystore

Recent Changes

Upgraded to AM 6.5 or later.

Installed AM 6.5 or later.

Causes

AM cannot load secrets from the keystore on one or more of your upgraded/new servers. Specifically, for SAML federation, the missing secret is am.global.services.saml2.client.storage.jwt.encryption, which exists in a new /path/to/am/secrets/encrypted_base64 directory, which is created on upgrade/install.

Typically, this issue occurs after an upgrade if secret stores have not been redeployed to all upgraded servers within your site. The upgrade process only creates the relevant secret store files on the AM instance where you performed the upgrade; it does not copy the secret stores to other servers in your site. After upgrading, you will need to make secrets available to other servers in the site as described in the Configure secret stores after upgrade.

This can also occur on a new install if you did not follow all the steps in the Add a server to a site to make the keystore and secret store directory infrastructure available.

Occasionally, this issue can occur when your keystore password is incorrect.

Solution

This issue can be resolved as follows depending on whether you are doing an upgrade or a new install:

• Upgrade: Follow the steps in the Configure secret stores after upgrade. In essence, you must:
  1. Copy the following keystores and directories from the server on which you performed the upgrade to all other servers in the site:/path/to/am/am/keystore.jceks /path/to/am/am/.storepass /path/to/am/am/.keypass /path/to/am/secrets/encrypted/storepass /path/to/am/secrets/encrypted/entrypass /path/to/am/secrets/encrypted*/*By default, the file permissions for .storepass and .keypass are read only. You can temporarily relax the permissions to allow the files to be overwritten and then re-secure them afterwards.
  2. Check the permissions to make sure that the user who starts AM can also read these files/directories.
  3. Restart the web application container in which AM runs to apply these changes. ​
• New install: Follow the steps in the Add a server to a site and ensure you complete steps 10 and 11 to make all the keystore and secret store infrastructure available.

Checking keystore passwords

If the above steps do not resolve your issues, you should check your keystore passwords are correct. For example, for the default keystore, you could use a command such as the following to check the storepass password:$ keytool -list -v -keystore /path/to/am/security/keystores/keystore.jceks -storetype JCEKS -storepass:file /path/to/am/security/secrets/default/.storepassIf the password in the .storepass file is correct, you will see a listing of all the certificates in your keystore returned. If the password is incorrect, you will see an error message.

See The AM keystore for further information.

See Also

Secret store fails to start with Label must match regex exception in AM 6.5.0.x, 6.5.1 and 6.5.2.x

Upgrade

Configure sites and add servers

Secrets, certificates, and keys

Related Training

N/A

Related Issue Tracker IDs

N/A