CoreSystem debug log grows rapidly in OpenAM 13.0 and 13.5 when CTS connection fails

Last updated Jan 5, 2021

The purpose of this article is to provide assistance if the CoreSystem debug log fills up quickly in OpenAM 13.0 and 13.5 when the CTS connection fails. You will see "Cannot start task executor" and "Diagnostic Message: No operational connection factories available" errors repeating continually in the debug log. This log growth can fill the disk up, bypassing the log rotation maximum value and eventually bring OpenAM down.

2 readers recommend this article

Archived

This article has been archived and is no longer maintained by ForgeRock.

Symptoms

The following error is shown in the CoreSystem debug log when the log fills up:

amThreadManager:01/26/2016 02:14:47:633 PM GMT: Thread[amThreadManager-1,5,ServerService ThreadGroup]: TransactionId[300c2122-7500-4b31-abdc-39a19681ff30-2] ERROR: ThreadMonitor: Thread WatchDog detected error, restarting java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Cannot start task executor at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.forgerock.openam.shared.concurrency.ThreadMonitor$WatchDog.run(ThreadMonitor.java:231) at org.forgerock.openam.audit.context.AuditRequestContextPropagatingRunnable.run(AuditRequestContextPropagatingRunnable.java:42) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: Cannot start task executor at org.forgerock.openam.sm.datalayer.impl.SeriesTaskExecutorThread.run(SeriesTaskExecutorThread.java:85) ... 6 more Caused by: org.forgerock.openam.sm.datalayer.api.LdapOperationFailedException: CTS: Operation failed: Result Code: Connect Error Diagnostic Message: No operational connection factories available Matched DN: at org.forgerock.openam.sm.datalayer.providers.LdapConnectionFactoryProvider$LdapConnectionFactory.create(LdapConnectionFactoryProvider.java:158) at org.forgerock.openam.sm.datalayer.providers.LdapConnectionFactoryProvider$LdapConnectionFactory.create(LdapConnectionFactoryProvider.java:126) at org.forgerock.openam.cts.monitoring.impl.connections.MonitoredCTSConnectionFactory.create(MonitoredCTSConnectionFactory.java:71) at org.forgerock.openam.sm.datalayer.impl.SimpleTaskExecutor.start(SimpleTaskExecutor.java:59) at org.forgerock.openam.sm.datalayer.impl.SeriesTaskExecutorThread.run(SeriesTaskExecutorThread.java:83) ... 6 more Caused by: org.forgerock.opendj.ldap.ConnectionException: Connect Error

The final Caused by statement varies depending on the root cause, but you may see one of the following causes instead:

Caused by: org.forgerock.opendj.ldap.ConnectionException: Connect Error: Connection refused Caused by: org.forgerock.opendj.ldap.ConnectionException: Server Connection Closed Caused by: org.forgerock.opendj.ldap.AuthenticationException: Invalid Credentials

Note

You will still encounter this issue even if you have debug log rotation set up since OpenAM does not currently delete old log files once they are rotated.

Recent Changes

Upgraded to, or installed OpenAM 13.0 or 13.5.

Made configuration changes to CTS.

CTS Configuration is set to incorrect value for "Login Id".

Causes

OpenAM cannot connect to the CTS token store; this may be because CTS is misconfigured or there is a connection issue, such as the CTS token store is down or a firewall is refusing the connection between the two servers.

There is a known issue where this situation results in the errors being continually logged in the CoreSystem debug log, which causes it to grow rapidly: OPENAM-8202 (If the "Login Id" in the External Store Configuration(CTS) is set to incorrect value,CoreSystem debug log is full of duplicate error).

Solution

This issue can be resolved by upgrading to OpenAM 13.5.1 or later; you can download this from BackStage.

Workaround

Alternatively, this issue can be resolved as follows:

Clear your CoreSystem debug log.
Correct the root cause.

Clear your CoreSystem debug log: You can clear the debug file as detailed in How do I clear debug logs in AM (All versions)? This should make the system responsive again, although the CoreSystem debug log will immediately start filling again so you need to be quick. If you do not have time to rectify your issue before the log fills up again, you can run the following command to continually clear the debug file on a loop to give you more time:

$ while true; do cat /dev/null > CoreSystem; done

Alternatively, you can temporarily switch OpenAM back to using the default embedded CTS token store (rather than external) using the following ssoadm command to give you more time:

$ ./ssoadm update-server-cfg -s [serverName] -u [adminID] -f [passwordfile] -a org.forgerock.services.cts.store.location=default

replacing [serverName], [adminID] and [passwordfile] with appropriate values. You can use default for [serverName] if you want to change the Default Server Settings.

Correct the root cause: Depending on whether you made any configuration changes or not, you should:

Ensure the CTS token store is up and running prior to starting OpenAM.
Ensure OpenAM can communicate with the OpenDJ server running CTS.
Correct your misconfiguration / revert the recent configuration change you made. A good way of doing this is via ssoadm using the commands detailed in How do I configure an external CTS token store in AM (All versions) using Amster or ssoadm? or by restoring a backup taken prior to making your configuration changes.

Related Training

N/A

Related Issue Tracker IDs

OPENAM-8202 (If the "Login Id" in the External Store Configuration(CTS) is set to incorrect value,CoreSystem debug log is full of duplicate error)