FAQ

FAQ: Session crosstalk and the Core Token Service (CTS) in OpenAM

Last updated Jul 9, 2018

The purpose of this FAQ is to provide answers to commonly asked questions regarding session crosstalk and the CTS in OpenAM. Crosstalk only applies to OpenAM 13.5.x and earlier. As of AM 5.0, crosstalk and the concept of a home server have been removed.


4 readers recommend this article

Frequently asked questions

Q. What is session crosstalk?

A. When a user authenticates, the session is created on the OpenAM server and is cached in memory on that server; this means that a session can only be validated on the originating (Home) server. In a multi-server deployment, a session may need to be validated on a different server to the one receiving the request. In this situation, the server receiving the request will make a HTTP request via a back channel to the Home server to retrieve the session. This mode of communication between servers for the purpose of validating sessions is referred to as session crosstalk. 

Any requests which update session objects such as log out, reset idle times or set a session attribute always use crosstalk to ensure the integrity. 

Q. What is the Home server?

A. The Home server is the server on which the session was created (the originating server). This may be the local server or a remote server depending on which server is receiving the request.

Note

As of AM 5.0, the concept of a Home server no longer applies; see Release Notes › What's New › New Features for further information on these changes. 

Q. How does OpenAM identify the Home server?

A. OpenAM uses the information contained in the session cookie (iPlanetDirectoryPro cookie) to identify the originating site and server as described in FAQ: Cookies in AM/OpenAM (Q. What information is contained in the AM/OpenAM session cookie?).

Q. When is session crosstalk used?

A. Session crosstalk happens when the session originates from a different OpenAM instance to one receiving the request.

Typically, OpenAM tries to resolve sessions in the following order upon receiving a request:

  1. Retrieves the session from the Home server as follows, in order:
    1. Looks up the cached session in memory if either it originates from the OpenAM instance receiving the request or it has been cached from a previous crosstalk attempt.
    2. Uses crosstalk to resolve the session on the remote server where the session originated if the remote server is up and running. The check of whether a server instance is alive is done by ClusterStateService. If the remote server is not running, OpenAM will try to find the next candidate. The retrieved session is then cached on the requesting server (non-Home server) to enable future requests to be resolved quickly. This session is stored per the Maximum Caching Time setting as detailed in How do I change the Maximum Caching Time in AM 5.x, 6 and OpenAM 12.x, 13.x? Additionally, a callback listener is registered with the Home server to enable notifications about session changes to be received to ensure the cached session does not become out-of-date.
  2. Looks up the session in the CTS store if all the other remote servers are down. If the other remote servers are running, AM/OpenAM will try to find the next candidate and perform crosstalk. This behavior is different when the Reduced Crosstalk option is set:
    1. When Reduced Crosstalk option is set, AM/OpenAM first tries to retrieve the session from the CTS for read operations. Any operation that requires the session state to be changed (e.g. Logout, set property , or reset idle time) will need to be delegated to the authoritative server (home server) and this is achieved using crosstalk.
    2. You should not disable reduced crosstalk unless advised to do so by ForgeRock Support. 
Note

Requests to update sessions, such as requests to log out, reset the session idle time, or set a session attribute, always use crosstalk to ensure the integrity of the update requests. See  Request Forward Caveats for further details including load balancer considerations. 

Q. How do OpenAM servers communicate with each other for crosstalk?

A. OpenAM uses the server URL for back channel communication. You should ensure that all OpenAM instances can communicate with each other via this URL. The URIs are:

  • /sessionservice for session related requests.
  • /namingservice for ClusterStateService.

You can check what port etc they are communicating on as follows:

  • OpenAM 13.5 console: navigate to Deployment > Servers.
  • Pre-OpenAM 13.5 console: navigate to Configuration > Servers and Sites.

For crosstalk, OpenAM performs a HTTP request against the remote Home server's sessionservice PLL endpoint (sent as XML) and receives back the session details in XML format (SessionInfo).

Q. How can I tell if crosstalk is being used?

A. You will see POST requests being made to the sessionservice endpoint from one OpenAM server to another. For example, you will see something similar to the following in the Apache Tomcat™ access logs, which in this example shows the call originating from IP address: 198.51.100.20:

198.51.100.20 - - [21/Jun/2017:10:07:11 +0100] "POST /openam/sessionservice HTTP/1.1" 200 403

You will also see requests being sent and received in the Session logs, for example:

PLLClient:03/06/2017 09:22:34:656 PM GMT: Thread[http-bio-8443-exec-23,5,main]
Send RequestSet Session XML :
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<RequestSet vers="1.0" svcid="session" reqid="198">
<Request><![CDATA[<SessionRequest vers="1.0" reqid="127">
<GetSession reset="true">
<SessionID>AQIC5wM2LY4Sfcxwo1cOJAMRkbIsn0bS8Pm1IB623MLBCnM.*AAJTSQACMDIAAlNLABMxOTM5OTEyMDI1MjY3NzE4OTc0AAJTMQACMDE.*</SessionID>
</GetSession>
</SessionRequest>]]></Request>
</RequestSet>
PLLClient:03/06/2017 09:22:34:656 PM GMT Thread[http-bio-8443-exec-23,5,main]
Response XML :
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ResponseSet vers="1.0" svcid="session" reqid="198">
<Response dtdid=""><![CDATA[<SessionResponse vers="1.0" reqid="127">
<GetSession>
<Session sid="AQIC5wM2LY4Sfcxwo1cOJAMRkbIsn0bS8Pm1IB623MLBCnM.*AAJTSQACMDIAAlNLABMxOTM5OTEyMDI1MjY3NzE4OTc0AAJTMQACMDE.*" stype="user" cid="uid=user1, ou=employees, ou=People, dc=example, dc=com" cdomain="o=users,ou=services,dc=amconfig,dc=example,dc=com" maxtime="900" maxidle="120" maxcaching="3" timeidle="2" timeleft="53980" state="valid">
<Property name="CharSet" value="UTF-8"></Property>
<Property name="UserId" value="user1"></Property>
<Property name="FullLoginURL" value="/openam/XUI/#login"></Property>
<Property name="successURL" value="/openam/XUI/#profile/></Property>
<Property name="cookieSupport" value="true"></Property>
<Property name="AuthLevel" value="0"></Property>
<Property name="SessionHandle" value="shandle:AQIC5wM2LY4Sfcxwo1cOJAMRkbIsn0bS8Pm1IB623MLBCnM.*AAJTSQACMDIAAlNLABMxOTM5OTEyMDI1MjY3NzE4OTc0AAJTMQACMDE.*"></Property>
<Property name="UserToken" value="user1"></Property>
<Property name="loginURL" value="/openam/XUI/#login"></Property>
<Property name="maxTokenLifeMinutes" value="10080"></Property>
<Property name="IndexType" value="service"></Property>
<Property name="Principals" value="uid=user1, ou=employees, ou=People, dc=example, dc=com"></Property>
<Property name="Service" value="ldap"></Property>
<Property name="PostAuthProcessInstance" value="org.forgerock.openam.authentication.modules.persistentcookie.PersistentCookieAuthModule"></Property>
<Property name="sun.am.UniversalIdentifier" value="id=user1,ou=user,o=users,ou=services,dc=amconfig,dc=example,dc=com"></Property>
<Property name="amlbcookie" value="02"></Property>
<Property name="Organization" value="o=users,ou=services,dc=amconfig,dc=example,dc=com"></Property>
<Property name="Locale" value="en_GB"></Property>

A key line to note in the above log extract is the following, since reset="true" is what triggers crosstalk:

<GetSession reset="true">

If you see reset="false" instead, this means the CTS store will be used.

Q. How can I tell if the session is being retrieved from a local or remote server?

A. You can tell from the Session logs if the session is local or remote as follows:

  • Local:
    amSession:06/11/2017 03:29:01:534 AM GMT: Thread[tomcat-http--13,5,main]: TransactionId[d9f5bb96-71e6-463e-11755-2f7ea3950543-63]
    Local fetch SessionInfo for AQIC5wM2LY4Sfcxwo1cOJAMRkbIsn0bS8Pm1IB623MLBCnM.*AAJTSQACMDIAAlNLABMxOTM5OTEyMDI1MjY3NzE4OTc0AAJTMQACMDE.*
    
  • Remote:
    amSession:06/11/2017 03:29:01:534 AM GMT: Thread[tomcat-http--13,5,main]: TransactionId[d9f5bb96-71e6-463e-11755-2f7ea3950543-63]
    Remote fetch SessionInfo for AQIC5wM2LY4Sfcxwo1cOJAMRkbIsn0bS8Pm1IB623MLBCnM.*AAJTSQACMDIAAlNLABMxOTM5OTEyMDI1MjY3NzE4OTc0AAJTMQACMDE.*
    

Q. What impact does crosstalk have on network traffic?

A. Crosstalk generates network traffic, which can have an impact on performance. The HTTP requests made between servers are blocking calls, so you have an outgoing and incoming request when crosstalk is happening, and potentially a third request if the crosstalk is occurring between sites as well. This means a single user request can keep two or three request processing threads busy at a time. 

It is important to tune your timeout settings according to your network speed or latency; in particular, you need to set the following advanced server properties (in milliseconds) appropriately for your deployment:

com.sun.identity.url.readTimeout              (default is 30000 ms)
org.forgerock.openam.url.connectTimeout       (default is 10000 ms)

You can set these properties using either the console or ssoadm:

  • OpenAM 13.5 console: navigate to Deployment > Servers > [Server Name] > Advanced and add the required property and value.
  • Pre-OpenAM 13.5 console: navigate to Configuration > Servers and Sites > [Server Name] > Advanced and add the required property and value.
  • ssoadm: enter the following command:
    $ ./ssoadm update-server-cfg -s [serverName] -u [adminID] -f [passwordfile] -a [property=value]
    replacing [serverName], [adminID], [passwordfile] and [property=value] with appropriate values.

If these timeouts are set too low to allow a remote server to respond, crosstalk will fail and you will see errors such as the following in your logs:

  • CoreSystem log:
    amXMLHandler:06/11/2017 03:29:01:532 AM GMT: Thread[http-nio-8080-exec-41,5,main]
    Error Response String : <Exception errorCode="Read timed out"></Exception>
    PLLClient:06/11/2017 03:29:01:532 AM GMT: Thread[http-nio-8080-exec-34,5,main]
    WARNING: PLLClient.send: exception: 
    java.net.SocketTimeoutException: Read timed out
  • Sessions log:
    amSession:06/11/2017 03:29:01:532 AM GMT: Thread[SystemTimer,5,main]: TransactionId[d9f5bb96-71e6-463e-11755-2f7ea3950543-63]
    ERROR: 20218:ClusterStateService.checkServerUp():: timeout too small 10000
    
    amSession:06/11/2017 03:29:01:534 AM GMT: Thread[SystemTimer,5,main]: TransactionId[d9f5bb96-71e6-463e-11755-2f7ea3950543-63]
    ERROR: 20218:ClusterStateService.checkServerUp():: error while checking server 01
    java.net.SocketTimeoutException: connect timed out
    amSession:06/11/2017 03:29:01:534 AM GMT: Thread[tomcat-http--13,5,main]: TransactionId[d9f5bb96-71e6-463e-11755-2f7ea3950543-63]
    processSessionRequest caught exception: connect timed out
    com.iplanet.dpro.session.SessionException: connect timed out

Q. What happens if the CTS store is down?

A. This depends on whether the Reduced Crosstalk option is set or not: 

  • If it is not set, OpenAM should retrieve the session from the Home server as described in Q. When is session crosstalk used? 
  • If it is set, OpenAM will try to retrieve the session from the CTS store but will fail.

Q. Are policy agent sessions stored in the CTS store?

A. No, policy agent sessions (Application SSOSession) are not stored in the CTS store by default as they are not subject to session failover. Therefore, crosstalk is always used with policy agent sessions.

However, if the Reduced Crosstalk option is set, policy agent session validation will always fail since the session does not exist in the CTS store and crosstalk is not used for agent sessions when this option is set. When this happens, you will see errors such as the following in the agent logs (debug.log or amAgent):

2017-06-09 11:03:47.002 Error 4786:7fc8f789bae0 AM_SSO_SERVICE: SSOTokenService::getSessionInfo(): Error 35 for sso token ID AQIC5wM2LY4Sfcxwo1cOJAMRkbIsn0bS8Pm1IB623MLBCnM.*AAJTSQACMDIAAlNLABMxOTM5OTEyMDI1MjY3NzE4OTc0AAJTMQACMDE.* 
2015-08-12 12:59:59.002 Error 4786:7fc8f789bae0 PolicyEngine: am_policy_evaluate: InternalException in Service::update_policy with error message:Session query failed. and code:35

Storing policy agent sessions in the CTS store

You can change the configuration to store policy agent sessions in the CTS store by setting the com.iplanet.am.session.agentSessionIdleTime property to force idle policy agent sessions to expire.

The default is 0 (policy agent sessions never expire) but you can also set it to a value of 30 or above (no maximum) to indicate the number of minutes a session can be idle. When set to 30 or above, the agent application token is stored in the CTS token store rather than locally. 

You can set this property using either the console or ssoadm:

  • AM / OpenAM 13.5 console: navigate to: Configure > Server Defaults > Advanced > com.iplanet.am.session.agentSessionIdleTime and amend the required number of minutes.
  • Pre-OpenAM 13.5 console: navigate to: Configuration > Servers and Sites > Default Server Settings > Advanced > com.iplanet.am.session.agentSessionIdleTime and enter the required number of minutes.
  • ssoadm: enter the following command:
    $ ./ssoadm update-server-cfg -s default -u [adminID] -f [passwordfile] -a com.iplanet.am.session.agentSessionIdleTime=[minutes]
    replacing [adminID], [passwordfile] and [minutes] with appropriate values.
Note

You must restart the web application container in which AM/OpenAM runs to apply these configuration changes. 

See Also

FAQ: Core Token Service (CTS) and session high availability in AM/OpenAM

Best practice for configuring an external DS/OpenDJ instance for the Core Token Service (CTS) in AM/OpenAM (All versions)

Sessions

Related Training

N/A

Related Issue Tracker IDs

OPENAM-9731 (Occasional failure to find server from ID in Webtop/Naming )

OPENAM-5632 (Occasional failure of OpenAM configurator tool)



Copyright and TrademarksCopyright © 2018 ForgeRock, all rights reserved.
Loading...