Solutions

CTS: Timed out whilst waiting for result errors in AM/OpenAM (All versions)

Last updated Nov 14, 2019

The purpose of this article is to provide assistance if you encounter errors such as "CTS: Timed out whilst waiting for result" in AM/OpenAM. Users are likely to experience issues logging in, with failed authentications or slow response times when this happens.


Symptoms

Users are encountering issues logging in such as failed authentications and/or slow response times. When this happens, you will see "CTS: Timed out whilst waiting on queue" messages in your logs. This message is associated with various different errors, which all have the same root cause.

The following examples show the type of errors that are associated with this message that you might see in your Session debug logs:

  • Unable to persist session to data store:
    amSession:04/10/2019 10:29:06:631 AM BST: Thread[http-nio-8080-exec-22,5,main]: TransactionId[d8983d6b-42f6-4f81-ba9e-03dfe7a9c39a-942469]
    ERROR: Unable to persist session to data store, check documentation for CTS configuration to ensure reads and writes may occur, and all appropriate virtual attributes are enabled.
    org.forgerock.openam.cts.exceptions.CoreTokenException:
    CTS: Timed out whilst waiting for result
       at org.forgerock.openam.cts.impl.queue.AsyncResultHandler.getResults(AsyncResultHandler.java:81)
       at org.forgerock.openam.cts.monitoring.impl.queue.TokenMonitoringResultHandler.getResults(TokenMonitoringResultHandler.java:42)
       at org.forgerock.openam.cts.monitoring.impl.queue.TokenMonitoringResultHandler.getResults(TokenMonitoringResultHandler.java:19)
       at org.forgerock.openam.cts.CTSPersistentStoreImpl.create(CTSPersistentStoreImpl.java:118)
       at org.forgerock.openam.cts.CTSPersistentStoreImpl.create(CTSPersistentStoreImpl.java:105)
       at org.forgerock.openam.session.cts.SessionPersistenceStore.create(SessionPersistenceStore.java:114)
       at org.forgerock.openam.session.cts.CtsSessionPersistenceStore.store(CtsSessionPersistenceStore.java:54)
       at org.forgerock.openam.session.cts.CtsSessionStoreChain$ChainIterator.store(CtsSessionStoreChain.java:62)
    ...
    Caused by: org.forgerock.openam.sm.datalayer.api.QueueTimeoutException:
    CTS: Timed out whilst waiting on queue.
    
  • SessionService.saveForFailover: exception encountered:
    amSession:04/10/2019 10:29:06:645 AM BST: Thread[https-executor-threads - 76,5,https-executor-threads]: TransactionId[d8983d6b-42f6-4f81-ba9e-03dfe7a9c39a-942469]
    ERROR: SessionService.saveForFailover: exception encountered
    org.forgerock.openam.cts.exceptions.CoreTokenException: 
    CTS: Error in data layer
       at org.forgerock.openam.cts.impl.queue.TaskDispatcher.update(TaskDispatcher.java:139)
       at org.forgerock.openam.cts.impl.CoreTokenAdapter.updateOrCreate(CoreTokenAdapter.java:137)
       at org.forgerock.openam.cts.CTSPersistentStoreImpl.updateAsync(CTSPersistentStoreImpl.java:103)
    ...
    Caused by: org.forgerock.openam.sm.datalayer.api.QueueTimeoutException: 
    CTS: Timed out whilst waiting on queue.
    
  • SessionService : failed deleting session:
    amSession:04/10/2019 10:29:06:645 AM BST: Thread[SystemTimerPool,5,ServerService ThreadGroup]: TransactionId[d8983d6b-42f6-4f81-ba9e-03dfe7a9c39a-942469]
    ERROR: SessionService : failed deleting session
    org.forgerock.openam.cts.exceptions.CoreTokenException:
    CTS: Error in data layer
       at org.forgerock.openam.cts.impl.queue.TaskDispatcher.delete(TaskDispatcher.java:160)
       at org.forgerock.openam.cts.impl.CoreTokenAdapter.delete(CoreTokenAdapter.java:150)
       at org.forgerock.openam.cts.CTSPersistentStoreImpl.deleteAsync(CTSPersistentStoreImpl.java:127)
    ...
    Caused by: org.forgerock.openam.sm.datalayer.api.QueueTimeoutException:
    CTS: Timed out whilst waiting on queue.
    
  • SessionCount.getSessionsFromRepository: Session repository is not available:
    amSession:04/10/2019 10:29:06:645 AM BST: Thread[catalina-exec-11,5,main]: TransactionId[d8983d6b-42f6-4f81-ba9e-03dfe7a9c39a-942469] 
    ERROR: SessionCount.getSessionsFromRepository: Session repository is not available org.forgerock.openam.cts.exceptions.QueryFailedException:
    CTS: Query operation Failed: Error: CTS: Error in data layer Filter: TokenFilter: Filter: [coreTokenUserId eq "id=amadmin,ou=user,dc=openam,dc=forgerock,dc=org"]
    ...
    Caused by: org.forgerock.openam.sm.datalayer.api.QueueTimeoutException:
    CTS: Timed out whilst waiting on queue.
    
  • SAML2CTSPersistentStore.retrieveSAML2Token(): failed to retrieve SAML2 token using primary key:
    amCoreTokenService:04/10/2019 10:29:06:631 AM BST: Thread[http-nio-8080-exec-67,5,main]: TransactionId[d8983d6b-42f6-4f81-ba9e-03dfe7a9c39a-942469]
    ERROR: SAML2CTSPersistentStore.retrieveSAML2Token(): failed to retrieve SAML2 token using primary key:663938356273323732336565633362303364343764363637346130351386536633364656639666562323
    org.forgerock.openam.cts.exceptions.CoreTokenException: 
    CTS: Error in data layer
       at org.forgerock.openam.cts.impl.queue.TaskDispatcher.read(TaskDispatcher.java:131)
       at org.forgerock.openam.cts.impl.CoreTokenAdapter.read(CoreTokenAdapter.java:101)
       at org.forgerock.openam.cts.CTSPersistentStoreImpl.read(CTSPersistentStoreImpl.java:142)
       at org.forgerock.openam.cts.CTSPersistentStoreImpl.read(CTSPersistentStoreImpl.java:137)
       at org.forgerock.openam.cts.impl.SAML2CTSPersistentStore.retrieveSAML2Token(SAML2CTSPersistentStore.java:77)
    ...
    Caused by: org.forgerock.openam.sm.datalayer.api.QueueTimeoutException: 
    CTS: Timed out whilst waiting on queue.
    

You may also see errors in other logs affecting different types of tokens, for example from the OAuth2Provider debug log:

OAuth2Provider:04/10/2019 11:37:32:359 AM BST: Thread[http-nio-8080-exec-100,5,main]: TransactionId[c6f89e30-1704-4522-b377-17ece7d4e5c8-1089384991]
ERROR: StatefulTokenStore::Unable to read access token corresponding to id: 27_y2lzukYZoc-3emUokZeZT9VM
org.forgerock.openam.cts.exceptions.ReadFailedException: 
CTS: Failed to read Token: 27_y2lzukYZoc-3emUokZeZT9VM
   at org.forgerock.openam.cts.impl.CoreTokenAdapter.read(CoreTokenAdapter.java:113)
   at org.forgerock.openam.cts.CTSPersistentStoreImpl.read(CTSPersistentStoreImpl.java:142)
   at org.forgerock.openam.cts.CTSPersistentStoreImpl.read(CTSPersistentStoreImpl.java:137)
   at org.forgerock.openam.rest.router.CTSPersistentStoreProxy.read(CTSPersistentStoreProxy.java:97)
   at org.forgerock.openam.oauth2.token.cts.OAuthTokenStore.read(OAuthTokenStore.java:86)
   at org.forgerock.openam.oauth2.token.cts.StatefulTokenStore.readAccessToken(StatefulTokenStore.java:173)
   at org.forgerock.openam.oauth2.token.OpenAMTokenStore.readAccessToken(OpenAMTokenStore.java:169)
...
Caused by: org.forgerock.openam.cts.exceptions.CoreTokenException: 
CTS: Timed out whilst waiting for result

Recent Changes

N/A

Causes

The Core Token Service (CTS) store is designed to automatically throttle throughput when the buffer (queue) fills up with requests and then continue processing as normal when the queue frees up.

This error indicates that AM/OpenAM has timed out the connection to the CTS because it's taking too long to get results. This typically happens for one of the following reasons:

  • The CTS queue is full. Once the queue is full, all new requests are "blocked" before being placed in the queue; if the queue doesn't free up in the allotted time or it exceeds the allowed queue size, it will time out. By default, the queue size is 5000 and the timeout is 2 minutes, but these defaults need tuning in a production system to handle the expected load.
  • The connection pools have been exhausted, which cause the queue to become unresponsive.

Solution

This issue can be resolved by tuning your CTS instance appropriately for the number of requests it receives as discussed in the Installation Guide › CTS Tuning Considerations.

Given the nature of the error, you should pay particular attention to the following settings:

Once you have made changes to the tuning settings, you must restart the web application container in which AM/OpenAM runs to apply the changes. 

Note

Each deployment environment is unique and has its own specific set of requirements. The CTS can be different depending on its function as many different components of AM use CTS to store data. Sizing and/or tuning recommendations are outside the scope of ForgeRock support. if you want more tailored advice, consider engaging Deployment Support Services.

See Also

Knowledge Base

Documentation

Related Training

N/A

Related Issue Tracker IDs

N/A



Copyright and TrademarksCopyright © 2019 ForgeRock, all rights reserved.

Recommended Books

Loading...