Tune the CTS
There are several tuning considerations that you can make for the efficient processing of your CTS token store:
| Task | Resources |
|---|---|
Configure the reaper cache size The reaper process prunes tokens in the store that have reached their maximum time-to-live. |
|
Manage the reaper search size In addition to the cache, the reaper also uses a search to find expired tokens. You can tune the maximum amount of tokens that the reaper can find and delete in a single invocation. |
|
Manage the CTS queue size and timeout Every create, update, and delete operation sent to the CTS token store is placed in an asynchronous queue before being handled. When the queue is full, new operations are blocked until they get into the queue or time out. |
|
Disable virtual attributes To improve CTS throughput, disable unnecessary virtual DS attributes. |
|
Change the OAuth 2.0 Storage Scheme If the CTS token store was created on a version of AM earlier than 6.5,
configure the |
Reaper cache size
The size of the AM reaper cache is controlled by
the org.forgerock.services.cts.reaper.cache.size advanced property.
The default size is 500000 tokens.
If an AM server is under sustained heavy load, the reaper cache may reach capacity,
causing degraded performance due to the additional slower searches of the CTS store.
If the reaper cache is full, messages are logged in the Session debug log, such as the following:
The CTS token reaper cache is full. This will result in degraded performance. You should increase the cache size by setting the advanced server property 'org.forgerock.services.cts.reaper.cache.size' to a number higher than 500000.
If this debug message appears frequently in the debug logs,
increase the value of the org.forgerock.services.cts.reaper.cache.size property.
To alter the value, in the AM admin UI, go to Configure > Server Defaults > Advanced,
and add the property and increased value to the list.
Increasing the size of the reaper cache causes higher memory usage on the AM server. If a cache of the default size of 500000 entries is nearly full, the server memory used could be up to approximately 100 megabytes.
|
Tune the AM reaper cache size only if
the |
Reaper search size
The reaper uses its cache to ensure that expired tokens are removed from the CTS token store, but it also has a built-in mechanism to search for expired tokens that may have not been purged. This could happen, for example, due to an AM instance crash.
When the reaper searches for expired tokens, DS returns a page of records that then the reaper will delete.
To configure the maximum amount of records that DS will return,
use the org.forgerock.services.cts.reaper.search.tokenLimit advanced server property.
The default is 5000.
To configure how often the search runs, and the grace period of the tokens that will be reaped, configure the following properties:
-
org.forgerock.services.cts.reaper.search.pollFrequencyMilliseconds.Specifies how often to perform a search for expired tokens. The default is
5000milliseconds.The value of this setting should never be higher than a few seconds, since the goal is to launch the search fair often so that the expired tokens do not pile up.
-
org.forgerock.services.cts.reaper.search.gracePeriodMilliseconds.Specifies the grace period used when searching for expired tokens. Any tokens that expired more than the specified duration ago are returned. The default is
300000milliseconds.The grace period should be larger than the value controlled by the
org.forgerock.services.cts.reaper.cache.pollFrequencyMillisecondsadvanced property. This allows an AM instance sufficient time to delete the token using its cache, rather than search.Deleting from the cache is preferred as it avoids expensive searches against the CTS store.
To tune the reaper searches, run load tests in your environment with the default reaper parameters, and then watch if the expired tokens build up. If there is none, then the reaper is sufficiently tuned. If you see token build-up, tune the reaper cache first, since it is the less expensive of the two reaping mechanisms.
It is fine if there is a small amount of expired tokens left on each iteration of the reaper search as long as tokens are cleared up during lower activity periods.
Keeping a large amount of expired tokens in the CTS store will result in increased disk space usage, and in AM not sending expiration notifications in a timely fashion.
Setting the token limit too high could negatively affect the performance of the DS instance. It is better to launch more runs of the reaper search with smaller amounts of records, than fewer runs with larger amounts of records.
Queue size and timeout
One tuning consideration is to manage the CTS queue size and timeout for efficient throughput. AM makes CTS requests from the following components:
-
AM server-side sessions and authentication sessions
-
OAuth 2.0
-
UMA
-
Session denylist (if enabled for client-side sessions)
-
Authentication session allowlist (if enabled for client-side sessions)
-
SAML v2.0 (if enabled for Security Token Service token validation and cancellation)
-
OAuth 2.0 server-side tokens and client-side token denylist
-
Push notification during authentication
-
Cluster-wide notification
Every create, update, and delete requests to CTS are placed into an asynchronous buffer before being handled by an asynchronous processor. This ensures that callers performing write operations can continue without waiting for CTS to complete processing.
Once the queue is full, all new operations are "blocked" before being placed in the queue. Once the queue is frees up, the caller can continue as normal.
CTS is designed to automatically throttle throughput when the buffer fills up with requests. Therefore, if you require a balance between performance versus system memory, AM provides two properties that can be used to tune CTS, queue size and queue timeout.
org.forgerock.services.cts.async.queue.size-
Default size: 5000. Determines the amount of request operations that can be buffered before the queue size becomes full, after which the caller will be required to wait for the buffered requests to complete processing. All CRUDQ operations are converted to tasks, which are placed in the queue, ensuring that operations happen in the correct sequence.
org.forgerock.services.cts.async.queue.timeout-
Default timeout is 120 seconds. Determines the length of time a caller will wait when the buffer is full. If the timeout expires, the caller receives an error. The timeout property is used in any system configuration where the LDAP server throughput is considerably slower than the AM server, which can result in blocked requests as the backlog increases.
To set the queue size and timeout properties:
-
In the AM admin UI, go to Configure > Server Defaults > Advanced.
-
Enter the key name and value.
-
Click Add.
For additional information on tuning the CTS, see Tune the CTS store LDAP connections.
Virtual attributes
DS supports a number of virtual attributes, which dynamically generate entry values that are not persisted in the token store. By default, the following DS virtual attributes are enabled at install:
-
collective-attribute-subentries -
entity-tag -
entry-dn -
entry-uuid -
governing-structure-rule -
has-subordinates -
is-member-of -
num-subordinates -
password-expiration-time -
password-policy-subentry -
structural-object-class -
subschema-subentry
To improve CTS throughput, you can disable your virtual attributes using the dsconfig command,
except for the Entity Tag virtual attribute, which must remain enabled else it will lead to an inoperable server.
To disable a virtual attribute, use the dsconfig command on DS to disable, for example,
the Collective Attribute Subentries virtual attribute:
./dsconfig set-virtual-attribute-prop \
--name=collective-attribute-subentries-virtual-attribute \
--set enabled: false \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPasswordpassword\
--no-prompt|
The DS |
OAuth 2.0 CTS storage scheme
AM 6.5 introduced a new scheme for storing OAuth 2.0 tokens in the CTS store, called the grant-set scheme.
The grant-set scheme groups multiple authorizations for a given OAuth 2.0 client and resource owner pair
and stores them in a single CTS OAUTH2_GRANT_SET entry.
This implementation reduces the size and quantity of entries stored,
as well as the number of calls required to perform OAuth 2.0 operations.
The one-to-one scheme stores the state of multiple authorizations
for a given OAuth 2.0 client and resource owner pair across multiple entries, and is more resource intensive.
You should upgrade to the grant-set scheme once all the servers on your environment
have been upgraded to AM 6.5 or later.
The grant-set scheme is backwards-compatible with existing entries stored in the CTS store.
Therefore, any access or refresh token issued before configuring the grant-set scheme is still valid.
Existing tokens will be retained in their original form until the refresh token expires or it is actively revoked.
Users will not notice any change in the tokens they receive, and there is no change to the OAuth 2.0 endpoints.
To enable the grant-set scheme:
-
Go to Configure > Global Services > OAuth2 Provider > Global Attributes.
-
In the CTS Storage Scheme drop-down, choose Grant-Set Storage Scheme.
-
Click Save Changes.
New OAuth 2.0 tokens stored in the CTS after the change will use the new scheme automatically.