Tuning the CTS

Reaper Cache Size

The size of the AM reaper cache is controlled by the org.forgerock.services.cts.reaper.cache.size advanced property. The default size is 500000 tokens.

If an AM server is under sustained heavy load, the reaper cache may reach capacity, causing degraded performance due to the additional slower searches of the CTS store. If the reaper cache is full, messages are logged in the Session debug log, such as the following:

The CTS token reaper cache is full. This will result in degraded performance.
You should increase the cache size by setting the advanced server property
'org.forgerock.services.cts.reaper.cache.size' to a number higher than 500000.

If this debug message appears frequently in the debug logs, increase the value of the org.forgerock.services.cts.reaper.cache.size property. To alter the value, in the AM console, navigate to Configure > Server Defaults > Advanced, and add the property and increased value to the list.

Increasing the size of the reaper cache causes higher memory usage on the AM server. If a cache of the default size of 500000 entries is nearly full, the server memory used could be up to approximately 100 megabytes.

Reaper Search Size

The reaper uses its cache to ensure that expired tokens are removed from the CTS token store, but it also has a built-in mechanism to search for expired tokens that may have not been purged. This could happen, for example, due to an AM instance crash.

When the reaper searches for expired tokens, DS returns a page of records that then the reaper will delete.

To configure the maximum amount of records that DS will return, use the org.forgerock.services.cts.reaper.search.tokenLimit advanced server property The default is 5000.

To configure how often the search runs, and the grace period of the tokens that will be reaped, configure the following properties:

To tune the reaper searches, run load tests in your environment with the default reaper parameters, and then watch if the expired tokens build up. If there is none, then the reaper is sufficiently tuned. If you see token build-up, tune the reaper cache first, since it is the less expensive of the two reaping mechanisms.

It is fine if there is a small amount of expired tokens left on each iteration of the reaper search as long as tokens are cleared up during lower activity periods.

Keeping a large amount of expired tokens in the CTS store will result in increased disk space usage, and in AM not sending expiration notifications in a timely fashion.

Setting the token limit too high could negatively affect the performance of the DS instance. It is better to launch more runs of the reaper search with smaller amounts of records, than fewer runs with larger amounts of records.

Queue Size and Timeout

One tuning consideration is to manage the CTS queue size and timeout for efficient throughput. AM makes CTS requests from the following components:

Every create, update, and delete requests to CTS are placed into an asynchronous buffer before being handled by an asynchronous processor. This ensures that callers performing write operations can continue without waiting for CTS to complete processing.

Once the queue is full, all new operations are "blocked" before being placed in the queue. Once the queue is frees up, the caller can continue as normal.

CTS is designed to automatically throttle throughput when the buffer fills up with requests. Therefore, if you require a balance between performance versus system memory, AM provides two properties that can be used to tune CTS, queue size and queue timeout.

To set the queue size and timeout properties in the AM console, navigate to Configure > Server Defaults > Advanced, enter the key name and value, and then click Add.

For additional information on tuning CTS, see "Tuning CTS Store LDAP Connections".

Virtual Attributes

DS supports a number of virtual attributes, which dynamically generate entry values that are not persisted in the token store. By default, the following DS virtual attributes are enabled at install:

To improve CTS throughput, you can disable your virtual attributes using the dsconfig command, except for the Entity Tag virtual attribute, which must remain enabled else it will lead to an inoperable server.

To disable a virtual attribute, use the dsconfig command on DS to disable, for example, the Collective Attribute Subentries virtual attribute:

./dsconfig set-virtual-attribute-prop \
--name=collective-attribute-subentries-virtual-attribute \
--set enabled: false \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPassword password \
--no-prompt

OAuth 2.0 CTS Storage Scheme

AM 6.5 introduced a new scheme for storing OAuth 2.0 tokens in the CTS store, called the grant-set scheme.

The grant-set scheme groups multiple authorizations for a given OAuth 2.0 client and resource owner pair and stores them in a single CTS OAUTH2_GRANT_SET entry. This implementation reduces the size and quantity of entries stored, as well as the number of calls required to perform OAuth 2.0 operations.

The one-to-one scheme stores the state of multiple authorizations for a given OAuth 2.0 client and resource owner pair across multiple entries, and is more resource intensive. You should upgrade to the grant-set scheme once all the servers on your environment have been upgraded to AM 6.5 or later.

The grant-set scheme is backwards-compatible with existing entries stored in the CTS store. Therefore, any access or refresh token issued before configuring the grant-set scheme is still valid. Existing tokens will be retained in their original form until the refresh token expires or it is actively revoked.

Users will not notice any change in the tokens they receive, and there is no change to the OAuth 2.0 endpoints.

To enable the grant-set scheme, navigate to Configure > Global Services > OAuth2 Provider > Global Attributes and set the CTS Storage Scheme drop-down to Grant-Set Storage Scheme. Then, save your changes.

New OAuth 2.0 tokens stored in the CTS after the change will use the new scheme automatically.