Solutions

CoreSystem debug log grows rapidly in OpenAM 13.0 and 13.5 when CTS connection fails

Last updated Jul 9, 2018

The purpose of this article is to provide assistance if the CoreSystem debug log fills up quickly in OpenAM 13.0 and 13.5 when the CTS connection fails. You will see "Cannot start task executor" and "Diagnostic Message: No operational connection factories available" errors repeating continually in the debug log. This log growth can fill the disk up, bypassing the log rotation maximum value and eventually bring OpenAM down.


2 readers recommend this article

Symptoms

The following error is shown in the CoreSystem debug log when the log fills up:

amThreadManager:01/26/2016 02:14:47:633 PM GMT: Thread[amThreadManager-1,5,ServerService ThreadGroup]: TransactionId[300c2122-7500-4b31-abdc-39a19681ff30-2]
ERROR: ThreadMonitor: Thread WatchDog detected error, restarting
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Cannot start task executor
   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
   at java.util.concurrent.FutureTask.get(FutureTask.java:192)
   at org.forgerock.openam.shared.concurrency.ThreadMonitor$WatchDog.run(ThreadMonitor.java:231)
   at org.forgerock.openam.audit.context.AuditRequestContextPropagatingRunnable.run(AuditRequestContextPropagatingRunnable.java:42)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Cannot start task executor
   at org.forgerock.openam.sm.datalayer.impl.SeriesTaskExecutorThread.run(SeriesTaskExecutorThread.java:85)
   ... 6 more
Caused by: org.forgerock.openam.sm.datalayer.api.LdapOperationFailedException: 
CTS: Operation failed:
Result Code: Connect Error
Diagnostic Message: No operational connection factories available
Matched DN: 
   at org.forgerock.openam.sm.datalayer.providers.LdapConnectionFactoryProvider$LdapConnectionFactory.create(LdapConnectionFactoryProvider.java:158)
   at org.forgerock.openam.sm.datalayer.providers.LdapConnectionFactoryProvider$LdapConnectionFactory.create(LdapConnectionFactoryProvider.java:126)
   at org.forgerock.openam.cts.monitoring.impl.connections.MonitoredCTSConnectionFactory.create(MonitoredCTSConnectionFactory.java:71)
   at org.forgerock.openam.sm.datalayer.impl.SimpleTaskExecutor.start(SimpleTaskExecutor.java:59)
   at org.forgerock.openam.sm.datalayer.impl.SeriesTaskExecutorThread.run(SeriesTaskExecutorThread.java:83)
   ... 6 more
Caused by: org.forgerock.opendj.ldap.ConnectionException: Connect Error 

The final Caused by statement varies depending on the root cause, but you may see one of the following causes instead:

Caused by: org.forgerock.opendj.ldap.ConnectionException: Connect Error: Connection refused

Caused by: org.forgerock.opendj.ldap.ConnectionException: Server Connection Closed 

Caused by: org.forgerock.opendj.ldap.AuthenticationException: Invalid Credentials
Note

You will still encounter this issue even if you have debug log rotation set up since OpenAM does not currently delete old log files once they are rotated.

Recent Changes

Upgraded to, or installed OpenAM 13.0 or 13.5.

Made configuration changes to CTS.

CTS Configuration is set to incorrect value for "Login Id".

Causes

OpenAM cannot connect to the CTS token store; this may be because CTS is misconfigured or there is a connection issue, such as the CTS token store is down or a firewall is refusing the connection between the two servers.

There is a known issue where this situation results in the errors being continually logged in the CoreSystem debug log, which causes it to grow rapidly: OPENAM-8202 (If the "Login Id" in the External Store Configuration(CTS) is set to incorrect value,CoreSystem debug log is full of duplicate error).

Solution

This issue can be resolved by upgrading to OpenAM 13.5.1 or later; you can download this from BackStage.

Workaround

Alternatively, this issue can be resolved as follows:

  1. Clear your CoreSystem debug log.
  2. Correct the root cause.

Clear your CoreSystem debug log: You can clear the debug file as detailed in How do I clear debug logs in AM/OpenAM (All versions)? This should make the system responsive again, although the CoreSystem debug log will immediately start filling again so you need to be quick. If you do not have time to rectify your issue before the log fills up again, you can run the following command to continually clear the debug file on a loop to give you more time:

$ while true; do cat /dev/null > CoreSystem; done

Alternatively, you can temporarily switch OpenAM back to using the default embedded CTS token store (rather than external) using the following ssoadm command to give you more time: 

$ ./ssoadm update-server-cfg -s [serverName] -u [adminID] -f [passwordfile] -a org.forgerock.services.cts.store.location=default

replacing [serverName], [adminID] and [passwordfile] with appropriate values. You can use default for [serverName] if you want to change the Default Server Settings. 

Correct the root cause: Depending on whether you made any configuration changes or not, you should:

  • Ensure the CTS token store is up and running prior to starting OpenAM.
  • Ensure OpenAM can communicate with the OpenDJ server running CTS.
  • Correct your misconfiguration / revert the recent configuration change you made. A good way of doing this is via ssoadm using the commands detailed in How do I configure an external CTS token store in AM/OpenAM (All versions) using ssoadm? or by restoring a backup taken prior to making your configuration changes.

See Also

How do I configure an external CTS token store in AM/OpenAM (All versions) using ssoadm?

Best practice for configuring an external DS/OpenDJ instance for the Core Token Service (CTS) in AM/OpenAM (All versions)

FAQ: Core Token Service (CTS) and session high availability in AM/OpenAM

Installation Guide › Implementing the Core Token Service

Related Training

N/A

Related Issue Tracker IDs

OPENAM-8202 (If the "Login Id" in the External Store Configuration(CTS) is set to incorrect value,CoreSystem debug log is full of duplicate error)



Copyright and TrademarksCopyright © 2018 ForgeRock, all rights reserved.
Loading...