Best practice for configuring sessions in AM (All versions) to reduce the impact on the CTS store
The purpose of this article is to provide best practice advice for configuring sessions in AM to reduce the impact on the Core Token Service (CTS) store. This article includes information on how sessions impact the CTS store and the resulting impact on the DS changelog.
2 readers recommend this article
The impact of sessions on the CTS store depends on session type (where it's stored), the number of writes (which depends on how busy your site is and how sessions are configured) and the storage scheme in use (
Session data is stored in the CTS store as follows depending on type:
- Server-side sessions (called CTS-based in pre-AM 7.2) - all session data is stored in the CTS store.
- Client-side sessions (called Client-based in pre-AM 7.2) - all session data is contained within the session cookie (called iPlanetDirectoryPro by default).
You should refer to Choose where to store sessions for help with determining which session types are the most appropriate for your environment.
See FAQ: Cookies in AM and Introduction to sessions for further information.
Summary of recommendations
The following list provides a high-level summary of recommendations to consider for reducing the impact on the CTS store and in turn, the DS changelog; the subsequent sections provide more detail on these points where applicable:
- Upgrade to AM 6.5 or later and use the Grant-Set storage scheme.
- Configure sessions appropriately, in particular:
- Only add necessary session properties.
- Streamline your authentication process; consider whether you require things like ForceAuth and ensure efficient use of PAPs.
- Use the
&refresh=falseaction when validating sessions.
- Set the Maximum Session Cache Size and Latest Access Time Update Frequency properties appropriately for your environment.
- Size replication servers appropriately to avoid disk space issues. See the DS changelog section for further information.
It’s important to note that sizing and configuring sessions for CTS is a tuning process. Each deployment environment is unique and has its own specific set of requirements. The CTS can be different depending on its function as many different components of AM use CTS to store data. Sizing and/or tuning recommendations are outside the scope of ForgeRock support. if you want more tailored advice, consider engaging Deployment Support Services.
AM 6.5 introduced a new grant-set scheme for storing OAuth 2.0 tokens in the CTS store. This scheme groups multiple authorizations for a given OAuth 2.0 client and resource owner pair, and stores them in a single CTS OAUTH2_GRANT_SET entry. This scheme reduces the amount of data stored in the CTS by removing duplication and reduces the number of operations to the CTS.
See How do I improve performance with OAuth 2.0 tokens in AM 6.5.x and 7.x? for best practice advice when using the grant-set storage scheme.
This scheme is not the default in AM 6.5, but it is highly recommended that you use it once all AM instances are upgraded to AM 6.5. See OAuth 2.0 CTS storage scheme for further information.
There is no choice of scheme in AM 6 and each OAuth2 token is mapped to an individual CTS entry.
The following lists some important session considerations that can increase the number of writes to the CTS. You don't need to try to address all of these things; they are just listed to make you aware of them when you are considering what is needed in your environment, and whether you have inefficient processes or settings that are contributing to excessive CTS load:
- Adding session properties - each extra session property contributes another MODIFY operation to the CTS plus increases the size of the session token that is being written and stored (CoreTokenObject). There is a RFE to batch session properties to reduce the number of writes to the CTS: OPENAM-13788 (AM Session properties can only be added one at a time which is highly inefficient).
- Using a Post-Authentication Plugin (PAP) - any steps that modify or add session properties will increase the number of writes to the CTS.
- Using the ForceAuth process - Out of the box the ForceAuth process incurs at least eight writes to the CTS (max idle time, max session time, chain, service, upgraded session type set to true, chain, service and role). Introducing a Post-Authentication Plugin further increases the number of writes. A single session may hit ForceAuth up to five times; with an assumed total of 10 writes per ForceAuth journey that means 50 writes of the large CoreTokenObject for authentication per session (and 50 entries in the changelogDB).
Resetting AM session idle times - each time an idle session time is reset, AM writes to the CTS. By default, the session idle time is reset when the session is validated. You can use the
&refresh=falseaction to mitigate this. See Validate sessions using REST for further information.
- Setting the Maximum Session Cache Size property incorrectly (see below for further information).
- Setting the Latest Access Time Update Frequency property incorrectly (see below for further information).
Maximum Session Cache Size
This setting specifies the maximum number of sessions to cache in each AM server’s internal session cache. The cache is an LRU (Least Recently Used) cache, which evicts the least used session (usually the first session to be added to the cache) when this maximum session cache size value is reached. For example, with the default value of 5000: if there are 5000 active sessions within a single AM, when session 5001 arrives, session 1 is evicted. Until then, all sessions remain in the cache until modified and/or deleted.
This setting applies to each server and assuming sticky load balancing has been applied, a guide to setting this appropriately is to take the maximum number of active sessions at any one time divided by the number of available AM instances.
See Session settings for further information on this setting.
It is critical that the value chosen is thoroughly tested as this cache does consume JVM heap space, more so with additional session properties. If the value is set too high, there is a risk of instability or Out Of Memory exceptions.
Latest Access Time Update Frequency
This setting defines the write interval for the following two timers within the CTS entry that represent the SSO session token:
- CoreTokenExpirationDate - defines the token expiry date.
- coreTokenString04 - defines the token last access time and is used to calculate idle time.
The default for this parameter is 60 seconds, which means:
- If a call to
/json/sessions?_action=validateis made within 60 seconds of coreTokenString04 - no write operations occur for that entry in CTS, just optimized reads.
- If the call is made outside of this 60 second period - two MODIFY operations occur to replace coreTokenExpirationDate (for example, 20181023090859.390+0100) and coreTokenString04 (for example, 1540280339390 - Unix Epoch Time).
You can increase this write interval to reduce the number of writes to the CTS and reduce the frequency of the two MODIFY operations. This write interval should not exceed 10 minutes.
See Session for further information on this setting.
Each time a coreTokenObject is updated, the entire object is written to the changelog. The coreTokenObject is a relatively large JSON object with nested arrays of attributes that are stored as a single entry. The entries are stored base64 encoded, but neither this encoding or the raw changelogDb offer any compression.
It is possible to enable token compression, which can potentially reduce the size of token entries in the backend and changelog by half with minimal increase in CPU. However, this can undermine the security of encryption and should only be considered as a secondary option once session configuration has been optimized. See Manage CTS tokens for further information.
Replication changes are kept per the replication purge delay (three days by default), so the size of the changelog is dependent on the actual number of changes received within that time period. You can reduce the replication purge delay to reduce the size of the changelog, but you must ensure it is set appropriately to keep data long enough for replication and data recovery purposes. Information about replication changes is permanently gone once it has been purged from the changelog. See How do I control how long replication changes are retained in DS (All versions)? for further information.
Once you have set a replication purge delay that is appropriate to your needs, you should do some load testing to determine expected changelog sizes and then size your replication servers accordingly to prevent issues with disk space.
If you run out of disk space, you will likely see errors similar to the following in the DS Errors logs, which indicate write errors to the CTS:[20/Feb/2019:10:17:51 +1000] category=SYNC severity=ERROR msgID=26 msg=Error trying to use the underlying database. The Replication Server is going to shut down: ChangelogException: Could not add record 'Record [00000165c86da14665cc008e2763:ModifyMsg content: protocolVersion: 8 dn: coreTokenId=kOrkxaDZ6fYcUrcE0c3PEMFIGNk=,dc=am-cts,dc=forgerock,dc=org csn: 0000016613b6f7e40e79074d33e3 uniqueId: 561a1ab7-4bc2-4741-a8ef-ff96624854b9 assuredFlag: false assuredMode: SAFE_DATA_MODE safeDataLevel: 1]' in log file '/opt/ds/changelogDb/1.dom/1000.server/head.log' (BlockLogWriter.java:120 LogFile.java:250 Log.java:422 FileReplicaDB.java:150 FileChangelogDB.java:805 ReplicationServerDomain.java:503 ReplicationServerDomain.java:325 ServerHandler.java:1131 ServerReader.java:103)
FAQ: Core Token Service (CTS) and session high availability in AM
How do I generate sample user data for performance testing in DS (All versions)?
Core Token Service (CTS) and sessions in AM
Understanding CTS token types in AM
Performance tuning and monitoring ForgeRock products
Related Issue Tracker IDs
OPENAM-13788 (AM Session properties can only be added one at a time which is highly inefficient)