Troubleshooting AM/OpenAM and Policy Agents

This book provides information on troubleshooting various issues in AM/OpenAM and Policy Agents, including collecting useful troubleshooting information such as logs, heap dumps and stack traces.

## Sending troubleshooting data to ForgeRock Support for analysis

#### Sending data for analysis

You can attach troubleshooting and diagnostic data to tickets in BackStage. When using BackStage to access tickets, there are no limits to the attachment size; this is the recommended method of sending us troubleshooting and diagnostic data.

AM/OpenAM

DS/OpenDJ

IDM/OpenIDM

IG/OpenIG

## How do I record troubleshooting information in AM (All versions) and OpenAM 13.x?

The purpose of this article is to provide information on recording troubleshooting information in AM/OpenAM. You can record troubleshooting information using ssoadm or the REST API. Troubleshooting information includes: debug logs, thread dumps (similar output to a JStack stack trace), important run-time properties and the AM/OpenAM configuration.

#### Overview

The following sections demonstrate how to manually start and stop recording using REST or ssoadm:

However, recordings can stop automatically if:

• You start a new recording that has a different issueID specified in the recording control file.
• The maximum recording time specified in the recording control file has been reached (autoStop and time properties).
• The maximum debug log file size specified in the recording control file has been reached (autoStop and fileSize properties).

See The Recording Control File for further information on these properties.

Known issues

You should be aware of the following known issues and workarounds:

• If your recording exceeds the time set in the autoStop property, your recording may be empty or show content from a previous recording. This is a known issue: OPENAM-11629 (ssoadm recording show previous recording content ). You can workaround this by increasing the recording time (autoStop) from the default 15 seconds to ensure it exceeds your recording time, for example, by setting it to 5 minutes. For example, if you use the ssoadm stop-recording command to stop the recording after 1 minute, but autoStop is set to 15 seconds, you will see the following error:
{"code":400,"reason":"Bad Request","message":"No record or it's already stopped."}
• If you immediately make a second recording with the same referenceID, the second recording doesn't generate any debug logs. This is a known issue: OPENAM-8848 (Recording OpenAM failed if using the same reference ID twice in a row), which is fixed in OpenAM 13.5; you can workaround this in OpenAM 13.0 by ensuring you always use a different referenceID.
• If you disable the threadDump section, you also need to remove the delay section as shown in the examples below, else the recording will fail.
• If you disable module based authentication, the ssoadm start-recording command will fail with a 401 Unauthorized: Access denied response. See Some ssoadm commands fail with Service URL not found:session error when module based authentication is disabled in AM/OpenAM (All versions) for further information and the solution.

#### Using the REST API to record troubleshooting information

This section provides information on manually starting and stopping the recording using the REST API.

##### Note

You can check the status of a recording using the json/records?_action=status endpoint as described in Getting the Status of a Recording Event.

You can use the REST API to record troubleshooting information as follows depending on your version:

AM 5 and later

1. Authenticate as amadmin (you cannot use a delegated administrator). You must use the actual AM server URL (not lb). For example:
$curl -v -X POST -H "X-OpenAM-Username: amadmin" -H "X-OpenAM-Password: cangetinam" -H "Content-Type: application/json" -H "Accept-API-Version: resource=2.0, protocol=1.0" http://host1.example.com:8080/openam/json/realms/root/authenticate  Example response: { "tokenId": "AQIC5wM2LY4SfcxsuvGEjcsppDSFR8H8DYBSouTtz3m64PI.*AAJTSQACMDIAAlNLABQtNTQwMTU3NzgxODI0NzE3OTIwNAEwNDU2NjE0*", "successUrl": "/openam/console", "realm": "/" }  2. Start recording the troubleshooting information using the following curl command against the actual AM server URL (not lb), where the iPlanetDirectoryPro header (default AM session cookie name) is set to the token returned when you authenticated. This curl command includes very simple example recording properties (including thread dumps enabled), but you should change them according to what data you need to collect: $ curl -v -X POST -H "iPlanetDirectoryPro: AQIC5..." -H "Content-Type: application/json" -H "Accept-API-Version: resource=1.0, protocol=1.0" -d ' {
"issueID": 103572,
"referenceID": "policyEvalFails",
"description": "Troubleshooting artifacts in support of case 103572",
"zipEnable": true,
"configExport": {
"enable": true,
},
"debugLogs": {
"debugLevel": "MESSAGE"
},
"enable": true,
"delay" :  {
"timeUnit": "SECONDS",
"value": 5
}
}
}' http://host1.example.com:8080/openam/json/realms/root/records?_action=start

Example response:
{"recording":true,"record":{"issueID":103572,"referenceID":"policyEvalFails","description":"Troubleshooting artifacts in support of case 103572","zipEnable":true,"threadDump":{"enable":false},"configExport":{"enable":true,"password":"xxxxxx","sharePassword":false},"debugLogs":{"debugLevel":"message","autoStop":{"time":{"timeUnit":"MINUTES","value":5},"fileSize":{"sizeUnit":"KB","value":1048576}}},"status":"RUNNING","folder":"/home/forgerock/am1/openam/debug/record/103572/policyEvalFails/"}}


See The Recording Control File for further information on the properties in this curl command.
3. Reproduce the issue for which you want to record troubleshooting information.
4. Stop recording the troubleshooting information (unless it will stop automatically) using the following curl command against the actual AM server URL (not lb), where the iPlanetDirectoryPro header (default AM session cookie name) is set to the token returned when you authenticated:
$curl -v -X POST -H "iPlanetDirectoryPro: AQIC5..." -H "Content-Type: application/json" -H "Accept-API-Version: resource=1.0, protocol=1.0" http://host1.example.com:8080/openam/json/realms/root/records?_action=stop 5. Find your recording, which is located in the /debug/record directory (by default,$HOME/[am_instance]/openam/debug/record) and is stored according to the issueID and referenceID as explained in Retrieving Recording Information.
6. Attach the recording to your support request as detailed in Sending troubleshooting data to ForgeRock Support for analysis.

OpenAM 13.x

1. Authenticate as amadmin (you cannot use a delegated administrator). You must use the actual OpenAM server URL (not lb). For example:
$curl -v -X POST -H "X-OpenAM-Username: amadmin" -H "X-OpenAM-Password: cangetinam" -H "Content-Type: application/json" http://host1.example.com:8080/openam/json/authenticate Example response: { "tokenId": "AQIC5wM2LY4SfcxsuvGEjcsppDSFR8H8DYBSouTtz3m64PI.*AAJTSQACMDIAAlNLABQtNTQwMTU3NzgxODI0NzE3OTIwNAEwNDU2NjE0*", "successUrl": "/openam/console", "realm": "/" }  2. Start recording the troubleshooting information using the following curl command against the actual OpenAM server URL (not lb), where the iPlanetDirectoryPro header (default OpenAM session cookie name) is set to the token returned when you authenticated. This curl command includes very simple example recording properties (including thread dumps enabled), but you should change them according to what data you need to collect: $ curl -v -X POST -H "iPlanetDirectoryPro: AQIC5..." -H "Content-Type: application/json" -d ' {
"issueID": 103572,
"referenceID": "policyEvalFails",
"description": "Troubleshooting artifacts in support of case 103572",
"zipEnable": true,
"configExport": {
"enable": true,
},
"debugLogs": {
"debugLevel": "MESSAGE"
},
"enable": true,
"delay" :  {
"timeUnit": "SECONDS",
"value": 5
}
}
}' http://host1.example.com:8080/openam/json/records?_action=start

Example response:
{"recording":true,"record":{"issueID":103572,"referenceID":"policyEvalFails","description":"Troubleshooting artifacts in support of case 103572","zipEnable":true,"threadDump":{"enable":false},"configExport":{"enable":true,"password":"xxxxxx","sharePassword":false},"debugLogs":{"debugLevel":"message","autoStop":{"time":{"timeUnit":"MINUTES","value":5},"fileSize":{"sizeUnit":"KB","value":1048576}}},"status":"RUNNING","folder":"/home/forgerock/am1/openam/debug/record/103572/policyEvalFails/"}}


See The Recording Control File for further information on the properties in this curl command.
3. Reproduce the issue for which you want to record troubleshooting information.
4. Stop recording the troubleshooting information (unless it will stop automatically) using the following curl command against the actual OpenAM server URL (not lb), where the iPlanetDirectoryPro header (default OpenAM session cookie name) is set to the token returned when you authenticated:
$curl -v -X POST -H "iPlanetDirectoryPro: AQIC5..." -H "Content-Type: application/json" http://host1.example.com:8080/openam/json/records?_action=stop 5. Find your recording, which is located in the /debug/record directory (by default,$HOME/[openam_instance]/openam/debug/record) and is stored according to the issueID and referenceID as explained in Retrieving Recording Information.
6. Attach the recording to your support request as detailed in Sending troubleshooting data to ForgeRock Support for analysis.

#### Using ssoadm to record troubleshooting information

This section provides information on manually starting and stopping the recording using ssoadm.

##### Note

You can check the status of a recording using the ssoadm get-recording-status command, which takes the same format as the stop-recording command detailed in step 4 below.

You can use ssoadm to record troubleshooting information as follows:

1. Create a JSON recording control file to specify what should be recorded and how. An example file looks like this (with autoStop properties specified and thread dumps disabled):
{
"issueID": 103572,
"referenceID": "policyEvalFails",
"description": "Troubleshooting artifacts in support of case 103572",
"zipEnable": true,
"configExport": {
"enable": true,
},
"debugLogs": {
"debugLevel": "MESSAGE",
"autoStop": {
"time":  {
"timeUnit": "MINUTES",
"value": 5
},
"fileSize": {
"sizeUnit": "GB",
"value": 1
}
}
},
"enable": false
}
}


See The Recording Control File for further information on the properties in this file.
2. Start recording the troubleshooting information using the following ssoadm command:
$./ssoadm start-recording -s [serverName] -u [adminID] -f [passwordfile] -J [controlFile]  replacing [serverName], [adminID], [passwordfile] and [controlFile] with appropriate values, where [serverName] is the URL with fully qualified domain name (FQDN) of the server for which you want to record information and [controlFile] is the path and filename of the recording control file you created in step 1. An example command and response looks like this: $ ./ssoadm start-recording -s http://host1.example.com:8080/openam -u amadmin -f pwd.txt -J /path/to/recording.json


3. Reproduce the issue for which you want to record troubleshooting information.
4. Stop recording the troubleshooting information using the following ssoadm command (unless it will stop automatically):
$./ssoadm stop-recording -s [serverName] -u [adminID] -f [passwordfile] replacing [serverName], [adminID] and [passwordfile] with appropriate values. 5. Find your recording, which is located in the /debug/record directory (by default,$HOME/[am_instance]/openam/debug/record) and is stored according to the issueID and referenceID as explained in Retrieving Recording Information.
6. Attach the recording to your support request as detailed in Sending troubleshooting data to ForgeRock Support for analysis

## Best practice for recording troubleshooting information in AM (All versions) and OpenAM 13.x

The purpose of this article is to provide best practice advice on using the recording feature in AM/OpenAM. You can use this to record troubleshooting information, which includes: debug logs, thread dumps (similar output to a JStack stack trace), important run-time properties and the AM/OpenAM configuration.

#### Recording AM/OpenAM

As of OpenAM 13, you can record AM/OpenAM. For further information on this feature, refer to How do I record troubleshooting information in AM (All versions) and OpenAM 13.x? and Setup and Maintenance Guide › Troubleshooting › Recording Troubleshooting Information.

This article provides a set of recording examples, which will help you understand this feature and investigate AM/OpenAM during everyday usage. We recommend using Postman to manage the REST requests; this tool is easy to use and gives you quick access to your requests to make them easier to re-use.

##### Note

Postman uses your browser's cookies; therefore you may need to logout and re-authenticate between Postman tests if you are using the same browser for testing different scenarios.

#### Installing Postman

First you need to install Postman and then import the required collections.

1. Install Postman.
2. Select the Manage Environments option from the Settings cog (No environment drop-down in the top right-hand corner of Postman if you do not have any environments configured). You can then add one or more environments or import this example one to get you started: ForgeRockRecordingExampleEnv.postman_environment. You should avoid including specific environment properties to make it easier to switch between environments without modifying your request. If you have multiple AM/OpenAM environments, it is recommended that you create a Postman environment for each one. You can do this by importing the example environment multiple times and then adjusting the values needed. The following examples show four AM/OpenAM environments, where only the port number differs:

1. Import one of the following collections of example requests: (Record AM5.postman_collection.json for AM 5 and later; Record OpenAM.json.postman_collection for OpenAM 13.x) by clicking the Import button in the top-left hand corner of Postman. You should now have a new collection on the left-hand side of Postman, like this:

By importing this Postman collection and having the correct environment configured, you should be able to use these example requests immediately. If you want to modify one of these requests, it is recommended that you duplicate the request prior to making changes in order to keep these examples intact for future reference.

#### Using the collection

This section walks through an example scenario using requests within the Postman collection; you can use the other requests in the collection in a similar way. Each request in this collection has a description that outlines what you should know about it. You can copy the example requests you are interested in and modify the request with your data.

This example investigates a configuration issue by comparing the problematic configuration to a similar working configuration. This is a typical use case you might encounter when setting up a policy: You are configuring policy A, which doesn't seem to work yet, but is very similar to policy B which is working. By recording both policy A and B, you can compare AM/OpenAM debug logs to help you understand why policy A isn't working.

1. Authenticate as the amadmin user by running the following request: Record AM/OpenAM > Basic Commands > Authenticate and changing the X-OpenAM-Password value. On the Body tab, you should ensure the raw option is selected before you send your request:

1. Start recording the working policy B by running the following request: Record AM/OpenAM > Configuration issue > Success conf and changing the issueID value; it is recommended that you use the ZenDesk Ticket ID as the issueID when working with ForgeRock Support:

1. Test policy B. AM/OpenAM records logs for 5 minutes after you have run this request. You can manually stop it after you have tested the policy by running the following request: Record AM/OpenAM > Basic commands > Stop recording or wait until the 5 minutes have elapsed.
##### Note

You can check the current status of the recording manager by running the request: Record AM/OpenAM > Basic commands > Check status.

1. Start recording the problematic policy A by running the following request: Record AM/OpenAM > Configuration issue > Failing conf and changing the issueID value.
2. Test policy A. Again, this records logs for 5 minutes; you can stop it manually after you have tested your policy or wait until the 5 minutes have elapsed.

After recording these two use cases, you should have two folders in your AM/OpenAM /debug directory: record/8888/PolicySuccess and record/8888/PolicyFailure, where 8888 is the issueID and PolicySuccess / PolicyFailure are the referenceIDs used in this example:

##### Note

If you immediately make a second recording with the same referenceID, the second recording doesn't generate any debug logs. This is a known issue: OPENAM-8848 (Recording OpenAM failed if using the same reference ID twice in a row), which is fixed in OpenAM 13.5; you can workaround this in OpenAM 13.0 by ensuring you always use a different referenceID.

Once you have located your recordings, you can compare the debug logs and configurations to investigate your problematic policy. If you need further help from ForgeRock Support, you can attach the recording to your support request as detailed in Sending troubleshooting data to ForgeRock Support for analysis.

## How do I check if AM/OpenAM (All versions) is up and running?

The purpose of this article is to provide information on checking if AM/OpenAM is up and running; this information applies to standalone servers and servers running in site behind a load balancer. This can include checking connections from AM/OpenAM to the configuration store and user data store (also known as the identity repository, which is the data store that contains user profile information).

#### Checking if AM/OpenAM is up and running

You can perform a couple of checks to see if AM/OpenAM is up and running, depending on your requirements.

AM/OpenAM server and configuration store

You can send a request to the isAlive.jsp endpoint; this request can either be sent to the server URL or the load balancer URL, for example:

http://host1.example.com:8080/openam/isAlive.jsp
http://lb.example.com:8080/openam/isAlive.jsp


If you receive a HTTP 200 OK response, the status page will return Server is ALIVE: at this point you have verified AM/OpenAM is up and running, and can communicate with its configuration store; otherwise, this currently throws an exception, which causes the web server to return an error code of 500. If AM/OpenAM is not running you will receive a message saying the browser could not connect.

If AM/OpenAM is running and the directory server used for the configuration store is not reachable, the status page will return Server is DOWN.

User data store

You can perform a login operation using a REST call followed by a logout operation to remove the created session. If you can authenticate successfully, you know that your user store is reachable; this also means that AM/OpenAM is up and running, so you can use this authentication check instead of sending a request to the isAlive.jsp endpoint for a more comprehensive check.

##### Note

The core authentication settings for the realm you are authenticating against must have been set to Profile Required, and you must have an authentication module in your authentication chain that utilizes username and password authentication to be able to use this approach.

Alternatively, you could perform a login operation using invalid credentials. The failed authentication due to invalid credentials verifies AM/OpenAM and the user data store are communicating and functioning correctly, but means that a logout operation to destroy the created session is not needed.

See Development Guide › Developing with the REST API › Authentication and Logout for further information on using REST calls to authenticate.

N/A

## How do I export and import Service configurations for AM/OpenAM (All versions)?

#### Amster (AM 5 and later)

In AM 5 and later, you can also use Amster to export and import configurations as detailed in:

With Amster, it is also possible to import a Service configuration that was created in a different version of AM/OpenAM. For example, you can export a configuration from AM 5 and import it into AM 5.5.

##### Note

If you have custom modules, you must ensure you have installed and registered the custom module first as detailed in How do I import Service configurations in AM (All versions) using Amster when there are custom modules?

#### Exporting Service configurations using ssoadm

$./ssoadm export-svc-cfg -u [adminID] -f [passwordfile] -e [secretkey] -o [outputfile] replacing [adminID], [passwordfile], [secretkey] and [outputfile] with appropriate values. ##### Note The secret key can be any value and is used to encrypt passwords from the configuration; the same key must be used again when importing the configuration through ssoadm import-svc-cfg. Known issues The following known issues may prevent you from exporting Service configurations using ssoadm: #### Importing Service configurations using ssoadm ##### Caution You cannot import a Service configuration that was created in a different version of AM/OpenAM using ssoadm. You can import your Service configurations using the following ssoadm command: $ ./ssoadm import-svc-cfg -u [adminID] -f [passwordfile] -e [secretkey] -X [XMLfile]

Restart the web application container in which AM/OpenAM runs to update the configuration.

Known issues

The following known issues may prevent you from importing Service configurations using ssoadm:

#### Back up and Restore

In testing environments, you can use the ssoadm import and export commands to back up and restore your service configuration. In production environments, you should back up the service configuration using file system utilities or the export-ldif command. The export-ldif  and import-ldif commands must be on the same deployment as the one where the encryption keys are located.

## How do I export configuration details for the Policy Agent (All versions)?

The purpose of this article is to provide information on exporting configuration details for the Policy Agent. Exporting configuration details for the policy agent is a common step in troubleshooting as it allows you (or someone else) to check all your configuration settings.

#### Exporting policy agent configuration details

You can export the policy agent configuration details using different methods according to which version of AM/OpenAM you are using:

##### Note

In AM 6 and later, it is not possible to use the exported agent configuration as the basis for creating a local configuration file for an agent. In earlier versions, you could use the exported configuration from the console as it was in the correct format.

#### Console (pre-AM 6)

You can export the agent configuration using the console as follows in pre-AM 6:

• AM 5.x console: navigate to: Realms > [Realm Name] > Applications > Agents > [Web or J2EE] > [Agent Name] and click Export Configuration.
• OpenAM 13.x console: navigate to: Realms > [Realm Name] > Agents > [Web or J2EE] > [Agent Name] and click Export Configuration.
• Pre-OpenAM 13 console: navigate to: Access Control > [Realm Name] > Agents > [Web or J2EE] > [Agent Name] and click Export Configuration.

It is not possible to export the agent configuration using the console in AM 6 and later.

#### REST

You can use REST to obtain the agent's profile as follows:

1. Authenticate in order to obtain a session token. The URL changes depending on which version you are using; you must use the actual AM/OpenAM server URL (not lb). For example:
• AM 5 and later:
$curl -X POST -H "X-OpenAM-Username: amadmin" -H "X-OpenAM-Password: cangetinam" -H "Content-Type: application/json" -H "Accept-API-Version: resource=2.0, protocol=1.0" http://host1.example.com:8080/openam/json/realms/root/authenticate  • Pre-AM 5: $ curl -X POST -H "X-OpenAM-Username: amadmin" -H "X-OpenAM-Password: cangetinam" -H "Content-Type: application/json" http://host1.example.com:8080/openam/json/authenticate
Example response:
{ "tokenId": "AQIC5wM2LY4SfcxsuvGEjcsppDSFR8H8DYBSouTtz3m64PI.*AAJTSQACMDIAAlNLABQtNTQwMTU3NzgxODI0NzE3OTIwNAEwNDU2NjE0*", "successUrl": "/openam/console", "realm": "/" }

2. Return the agent's profile using one of the following curl commands against the actual AM/OpenAM server URL (not lb), where the iPlanetDirectoryPro header (default AM/OpenAM session cookie name) is set to the token returned when you authenticated and myAgent in the URL is replaced with the name of your agent:
• AM 5 and later:
$curl -X GET -H "iPlanetDirectoryPro: AQIC5wM2LY4Sfcxs...EwNDU2NjE0*" -H "Content-Type: application/json" http://host1.example.com:8080/openam/json/realms/root/agents/myAgent  • Pre-AM 5: $ curl -X GET -H "iPlanetDirectoryPro: AQIC5wM2LY4Sfcxs...EwNDU2NjE0*" -H "Content-Type: application/json" http://host1.example.com:8080/openam/json/agents/myAgent

#### Amster (AM 5 and later)

You can export the agent configuration using the export-config command in Amster. For example:

• Export configuration for a Web agent called myAgent in the top level realm:
am> read WebAgents --realm / --id myAgent
• Export configuration for a Java agent called Java in the employees realm:
am> read J2eeAgents --realm 'employees' --id Java

You can use ssoadm to return the agent's configuration. Enter the following command:

$./ssoadm show-agent -e [realmname] -b [agentname] -u [adminID] -f [passwordfile] -o [outputfile] replacing [realmname], [agentname], [adminID], [passwordfile] and [outputfile] with appropriate values. #### See Also #### Related Training #### Related Issue Tracker IDs N/A ## How do I diagnose a hung AM/OpenAM (All versions) server? The purpose of this article is to provide information on diagnosing a hung AM/OpenAM server, including the data you should collect to send to ForgeRock Support for troubleshooting. #### Diagnosing a hung AM/OpenAM server There are a number of different types of data that you should collect to help us diagnose a hung AM/OpenAM server. If you suspect you are experiencing a hung AM/OpenAM server, you should collect the following data and submit it to us when you raise the ticket to enable us to help you more quickly: • Debug logs. • JVM data. • Garbage Collector (GC) logs. • HTTP container logs. • Native stack traces. ##### Note It is important that you collect all the data immediately after reproducing the issue so that the timestamps of the data collected and the suspected hung AM/OpenAM server correspond. Please notify us of the exact timestamp when the issue occurred so we can focus our attention on the relevant information. Debug logs Message level debug logs provide more context for diagnosing errors but can generate large files. You should enable message level debugging when you are experiencing an issue as described in How do I enable Message level debugging in AM/OpenAM (All versions) debug files? JVM data JVM data must be collected before killing the affected process or restarting the server; otherwise the information relating to the affected process is lost forever, which may make it impossible to identify the root cause. There are three types of data that you should collect (stack traces, heap histograms and heap dumps) as well as providing details of the current JVM settings as described in How do I collect JVM data for troubleshooting AM/OpenAM (All versions)? See How do I use the msnapshots script to capture information for troubleshooting AM/OpenAM (All versions)? for further information on using a script to collect this data if you are using a Linux®, Unix® or Solaris® system. GC logs GC logs are not always crucial at this stage, but can provide useful information about potential tuning issues that may be contributing to your problems. Enabling GC logging is described in How do I enable Garbage Collector (GC) Logging for AM/OpenAM (All versions)? HTTP container logs HTTP container server logs (Access and Error) are very useful for troubleshooting as between them they show all requests to the server, whether they were successful or not and any corresponding error messages. The actual name and location of the server logs are specific to your web container. For example, for Apache Tomcat™, the logs are called localhost_access_log.YYYY-MM-DD.log and catalina.out, and are located in the /path/to/tomcat/logs/ directory by default. Native stack traces Native stack traces are not always necessary but can provide useful information when troubleshooting, especially if you can isolate the the affected process. You can use the pstack command for Solaris® and Linux® (although pstack is not installed by default on Linux): $ pstack -F [pid]

replacing [pid] with the process ID of the affected process.

N/A

## How do I troubleshoot WDSSO and Kerberos issues in AM/OpenAM (All versions)?

The purpose of this article is to provide information on troubleshooting Windows Desktop SSO (WDSSO) and Kerberos™ issues in AM/OpenAM. You can use the collected data to troubleshoot issues yourself or include it when you raise a ticket to enable us to help you more effectively.

#### Troubleshooting WDSSO and Kerberos issues

If you are experiencing issues with your WDSSO module and/or Kerberos setup in AM/OpenAM, you can use the following troubleshooting steps to debug your issue:

1. Generate a full set of message level debug logs and capture the HTTP trace while reproducing the issue. Required debug logs include the AM/OpenAM ones and the debug logs for the How do I enable debug logging for troubleshooting WDSSO and Kerberos issues in AM/OpenAM (All versions)? of the JVM (this module is called by AM/OpenAM for WDSSO purposes).
2. Verify a Kerberos token has been sent (rather than a NTLM token).
3. Enable Kerberos event logging on the Microsoft® Windows® / Active Directory® server as detailed in: Microsoft - How to enable Kerberos event logging.
4. Export the configuration for your WDSSO module - it is essential to check if a configuration issue is contributing to your issues.
5. Check your keytab file details are correct.
6. Check your browser settings are correct.
7. Check other configuration details.
8. Check connection to Kerberos Key Distribution Center (KDC).

Additionally, information is included about the expected access flows for WDSSO authentication (with a policy agent protected resource) and a federated environment. These flows can help you track down where the breakdown in WDSSO is occurring, which can help focus your troubleshooting efforts.

If you need to raise a ticket to troubleshoot an issue, you should include relevant information to help us investigate your issues more effectively. Refer to Best practice for raising a ticket with ForgeRock Support for further information on creating a ticket.

##### Note

If none of these resolve the issue, another problem must exist with the Kerberos protocol between the browser and Active Directory; you can attempt to track this down using utilities such as the Microsoft Network Monitor or the Microsoft Message Analyzer.

#### Understanding access flows for WDSSO

Windows Desktop SSO Authentication​

The expected access flow for WDSSO single sign-on (SSO) with a policy agent protected resource is as follows:

1. The user attempts to access the protected resource with a SPNEGO-enabled browser.
2. The policy agent intercepts the request and redirects to the AM/OpenAM server.
3. The AM/OpenAM server returns a HTTP 401 Authenticate/Negotiate response.
4. The browser sends the following to the Active Directory server:
• a request for the Ticket Granting Ticket (TGT) if a TGT does not already exist.
• a request (accompanied by the TGT) for a Kerberos Service Ticket.
5. The Kerberos Domain Controller sends a Service Ticket to the browser.
6. The browser sends HTTP, POST, GET, Web-Service and SPNEGO tokens to the AM/OpenAM server.
7. The WDSSO module validates the SPNEGO token and retrieves the user profile from the user data store.
8. The AM/OpenAM server creates an SSO Token.
9. The AM/OpenAM server returns a HTTP 200 OK response and the SSO Token to the browser.
10. The browser sends the SSO token to the policy agent and the policy agent grants access to the protected resource.

Federated environment

The expected access flow for WDSSO in a federated environment is as follows:

1. The user attempts to access the protected resource at the service provider (SP).
2. The SP sends the authorization request to the AM/OpenAM identity provider (IdP).
3. The AM/OpenAM IdP returns a HTTP 401 Authenticate/Negotiate response.
4. The browser sends the following to the Active Directory server:
• a request for the Ticket Granting Ticket (TGT) if a TGT does not already exist.
• a request (accompanied by the TGT) for a Kerberos Service Ticket.
5. The Kerberos Domain Controller sends a Service Ticket to the browser.
6. The browser sends HTTP, POST, GET, Web-Service and SPNEGO tokens to the AM/OpenAM IdP.
7. The WDSSO module validates the SPNEGO token and retrieves the user profile from the user data store.
8. The AM/OpenAM IdP creates an SSO Token.
9. The AM/OpenAM IdP returns a HTTP 200 OK response and the SSO Token to the browser.
10. The AM/OpenAM IdP sends the authorization response of Success to the SP and the SP grants access.

#### Generating debug logs and capturing the HTTP trace

It is important to ensure your debug logs and HTTP trace have corresponding date and timestamps. You should follow this process:

1. Enable message level debugging as described in How do I enable Message level debugging in AM/OpenAM (All versions) debug files?
2. Clear the AM/OpenAM debug logs as described in How do I clear debug logs in AM/OpenAM (All versions)?
3. Enable debug logging for the Krb5Login module of the JVM; this adds additional logging to the container's standard logs. You can enable debug logging as detailed in How do I enable debug logging for troubleshooting WDSSO and Kerberos issues in AM/OpenAM (All versions)?
4. Start a HTTP trace in your browser. There are free third-party tools you can use, for example, in Firefox®, you can use Live HTTP Headers.
5. Reproduce the issue using your browser.
6. Stop the HTTP trace.

#### Verifying a Kerberos token has been sent

Microsoft Windows will first try Kerberos; unless all the requirements are met, it will fallback to NTLM authentication.

You can use the output from the HTTP trace captured in the above section to check that a Kerberos token has been sent as follows:

1. Examine the output from the HTTP trace tool. You are looking for the Authorization: Negotiate section:
• If this begins YII, then Kerberos is being used, for example:
Authorization: Negotiate YIIVDAYGKwYBE...
• If this begins TlR, then Kerberos is not being used,  for example:
Authorization: Negotiate TlRMTVNTUA...

You can also check the Authentication log to see if it has fallen back to NTLM authentication. If it has, you will see the following error:

ERROR: kerberos token is not valid

See Kerberos token is not valid error when authenticating with Windows Desktop SSO in AM/OpenAM (All versions) using Internet Explorer for further information on this error.

#### Exporting the WDSSO module configuration

You can export this configuration using the following ssoadm command:

$./ssoadm get-auth-instance -e [realmname] -m [modulename] -u [adminID] -f [passwordfile]  Replacing [realmname], [modulename], [adminID] and [passwordfile] with appropriate values. For example: $ ./ssoadm get-auth-instance -e / -m wdsso -u amadmin -f pwd.txt


Example response:

Authentication Instance profile:
iplanet-am-auth-windowsdesktopsso-returnRealm=false
iplanet-am-auth-windowsdesktopsso-auth-level=0
iplanet-am-auth-windowsdesktopsso-kerberos-realm=forgerock.com
iplanet-am-auth-windowsdesktopsso-lookupUserInRealm=false
iplanet-am-auth-windowsdesktopsso-kdc=host1.example.com
iplanet-am-auth-windowsdesktopsso-principal-name=HTTP/openam.forgerock.com@forgerock.com
iplanet-am-auth-windowsdesktopsso-keytab-file=/tmp/keytab

#### Checking keytab file details

First you should check the keytab file exists in the location detailed in the WDSSO module export; if it does not, you should update the WDSSO module details. You should also ensure that the user that AM/OpenAM is running as has read access permissions to the keytab file.

You can then output details for your keytab file using either the Kerbtray or Klist utilities from Microsoft (depending on Microsoft Windows version). For example:

$klist An example output from the Klist utility is: Server: HTTP/host1.example.com @ forgerock.com KerbTicket Encryption Type: RSADSI RC4-HMAC(NT) Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize Start Time: 3/17/2016 11:33:06 (local) End Time: 3/17/2016 21:31:54 (local) Renew Time: 3/24/2016 11:31:54 (local) Session Key Type: RSADSI RC4-HMAC(NT) Cache Flags: 0 Kdc Called: SVR1 You can then check the following and resolve as applicable: • Check the Kerberos service ticket was retrieved for AM/OpenAM: • If the Kerberos service ticket does exist for the AM/OpenAM FQDN - check your browser settings to ensure the browser is correctly set up for Windows authentication and update where necessary. See the Checking your browser settings are correct section below for further information. • If the Kerberos service ticket does not exist for the AM/OpenAM FQDN - continue troubleshooting. • Check you have used an appropriate encryption type (KerbTicket Encryption Type in example Klist output) when generating the keytab file and that you have updated the user account properties in Active Directory to match. The corresponding encryption type should be selected in the user account properties. If you want to use a 256-bit AES encryption type, you must have the Oracle® Java JCE unlimited strength jars installed in the$JAVA_HOME/jre/lib/security/ directory and your Microsoft Windows machine must also support this encryption.
• Check the service principal defined in AM/OpenAM (iplanet-am-auth-windowsdesktopsso-principal-name in WDSSO module export) matches the one in the keytab file (Server in Klist output). This can be determined by looking at the Kerberos ticket with klist to see what Principal was actually authenticated in Kerberos.
• The key version number (kvno) in the keytab file must equal the value of the msDS-KeyVersionNumber attribute for the AM/OpenAM principal in Active Directory +1. This is because Active Directory increases the value of kvno by 1 when you use the keypass command to create the keytab file and these values must match.
##### Note

You can query the value of msDS-KeyVersionNumber in Active Directory using the ldapsearch command.

#### Checking your browser settings are correct

You should check the following specific browser settings:

• Check the AM/OpenAM FQDN is listed as a trusted host in the browser and add it if it's not listed. You can check as follows according to which browser you are using:
• Internet Explorer® and Microsoft Edge - the AM/OpenAM FQDN should be added to the trusted Intranet site list (Security tab > Local Intranet > Sites > Advanced).
• Chrome™ - since Chrome uses the Internet Explorer or Microsoft Edgesettings, you should check the AM/OpenAM FQDN is added to the trusted Intranet site list per the previous bullet.
• Firefox - the AM/OpenAM FQDN should be added to the Integrated Authentication Sites list (Tools > Integrated Authentication Sites) if you're using the Integrated Authentication plugin. The equivalent Firefox setting is network.negotiate-auth.trusted-uris (accessed via about:config).
• If you use Internet Explorer or Microsoft Edge, there are two more settings you need to check and configure in Internet Options:
• Ensure the Enable Integrated Windows Authentication option is selected. This option is found on the Advanced tab under Security.
• Ensure the Automatic logon only in Intranet zone option is selected. This option can be accessed from the Security tab. Select Local Intranet and then click the Custom Level button. This option can then be found under User Authentication > Logon.
You must restart Internet Explorer or Microsoft Edge for these settings to take effect.
##### Note

WDSSO only works with Internet Explorer and Microsoft Edge when the server uses HTTP persistent connection.

• If you use Chrome, Chrome inherits its settings from Internet Explorer or Microsoft Edge when you are using Microsoft Windows so will work providing you have configured Internet Explorer or Microsoft Edge as detailed above.
• If you use Firefox, you can use a plugin called Integrated Authentication. You should ensure the AM/OpenAM FQDN is added to the Integrated Authentication Sites list (Tools > Integrated Authentication Sites) if you're using the Integrated Authentication plugin. Alternatively, you can change the equivalent Firefox setting (network.negotiate-auth.trusted-uris) via about:config.
• If you use Safari®, Safari has built-in support for Kerberos SSO and no additional configuration is required.

#### Checking other configuration details

You should also check the following configuration details:

• The krb5.conf file has been set up; this file contains the Kerberos configuration information (see krb5.conf for details).
• An Active Directory datastore is configured for the realm you are using and you can see the Active Directory users on the Identities page (previously the Subjects tab) in that realm.
• Your credentials are correct. You can check them by logging into the realm for the authentication chain that includes the WDSSO module using your Active Directory user credentials. For example, the URL would be similar to this, where the authentication chain is ldapService and the realm is top level:
http://host1.example.com:8080/openam/XUI/#login/&service=ldapService
• The Service Principal Name (SPN) for the AM/OpenAM service is correct in Active Directory; ensure the FQDN, which matches the SPN in Active Directory is used and the specified authcontext matches the authentication module/chain for WDSSO. You can use the following command to check the SPN:
$setspn -l [account] replacing [account] with the name of the account created for AM/OpenAM. An example command looks like this: $ setspn -l openam
and gives an output as follows:
Registered ServicePrincipalNames for CN=OpenAM,OU=employees,DC=example,DC=com:

HTTP/host1.example.com
• The userPrincipalName must be unique for all users.
• If you are using Oracle® JDK 7, you must ensure both TCP port 88 and UDP port 88 are open in your firewall to allow communication between AM/OpenAM and Kerberos.
• The Kerberos Server Name is an A record and not a CNAME, and that the reverse (PTR) record for that IP also resolves back to that Server Name.
• If you are using the Windows Desktop SSO module as part of an authentication chain and Windows Desktop SSO fails, you may no longer be able to POST data to non-NTLM-authenticated web sites. For information on a possible workaround, see KB251404: You cannot post data to a non-NTLM-authenticated Web site.

#### Checking connection to Kerberos Key Distribution Center (KDC)

Your firewall must be configured to allow necessary communications with Kerberos; see Kerberos and Firewalls for the required ports.

You can test the connection to the Kerberos KDC using a third-party tool called HelloKDC.java. This tool will determine if there is a connection issue and give you guidance on what the issue might be. If it shows there is a connection issue, it means the issue is outside of AM/OpenAM and will need resolving before you attempt any further configuration / testing within AM/OpenAM.

##### Note

This is a third-party tool that we suggest can be used for troubleshooting but is not supported by ForgeRock.

Example

The following example shows a connection issue returned from HelloKDC.java, which correlates to an exception in the Authentication logs:

• HelloKDC.java output:
>>>KRBError:
sTime is Thu Jul 20 14:21:25 GMT 2017 1498776265000
suSec is 154839
error code is 25
error Message is Additional pre-authentication required
...
KRBError received: Need to use PA-ENC-TIMESTAMP/PA-PK-AS-REQ

• Authentication logs:
amAuth:20/07/2017 20 14:21:25 GMT: Thread[http-nio-8443-exec-1,5,main]: TransactionId[daffa5fe-cca9-4a16-8cc3-dec02f21a81f-57]
Exception :
Additional pre-authentication required (25) - Need to use PA-ENC-TIMESTAMP/PA-PK-AS-REQ


As you can see, both are showing an error code 25: Additional pre-authentication required (Need to use PA-ENC-TIMESTAMP/PA-PK-AS-REQ). Seeing the same error returned from HelloKDC.java confirms this issue is independent of AM/OpenAM and will need to be resolved outside of ForgeRock support.

N/A

## How do I troubleshoot issues with the CORS filter in AM/OpenAM (All versions)?

#### Troubleshooting techniques

You should use the following approach to troubleshoot issues with the CORS filter:

1. Set debug logging level to message or warning, and grep for the word "WARNING" in the CoreSystem log. Then find the corresponding section in this article to resolve your issue. See How do I enable Message level debugging in AM/OpenAM (All versions) debug files? for further information enabling message level debugging.
2. Obtain a HTTP Trace of the failing flow in your browser. There are free third-party tools you can use, for example, in Firefox®, you can use Live HTTP Headers.
3. If the issue happens in some browsers (for example, Chrome™), but not in others (for example, Firefox®), obtain an HTTP Trace from the working flow to allow comparison.

#### Issue with the Origin

The following warning is displayed in the CoreSystem log:

WARNING: CORS Fail - Requested origin comes from a location not whitelisted.


Troubleshooting

In the HTTP trace, search for a Request that contains an Origin Header, such as:

Origin:http://other.origin.example.com:8080


The response to such a request should contain the following headers:

Access-Control-Allow-Origin:http://other.origin.example.com:8080


Solution

1. If the Response does not contain this header, check the Accepted Origins section of the CORS filter:
<init-param>
<description>
Accepted Origins (Required):
A comma separated list of origins from which to accept CORS requests.
</description>
<param-name>origins</param-name>
<param-value>
http://host1.example.com:8080,https://lb.example.com
</param-value>
</init-param>

2. Add the missing origin to this section accordingly. You need to add the exact same value as displayed in the Origin header of the request. In our example, you would end up with:
<init-param>
<description>
Accepted Origins (Required):
A comma separated list of origins from which to accept CORS requests.
</description>
<param-name>origins</param-name>
<param-value>
http://openam1.example.com:8080,https://lb.example.com,http://other.origin.example.com:8080
</param-value>
</init-param>

3. Restart the web application container in which AM/OpenAM runs.
4. Re-test the failing flow.

#### Issue with the Method

The following warning is displayed in the CoreSystem log:

WARNING: CORS Fail - Requested HTTP method has not been whitelisted.


Troubleshooting

In the HTTP trace, search for a Request that contains an Origin Header, such as:

Origin:http://other.origin.example.com:8080


Find the corresponding method. In the following example, the method is POST:

>Connected to host1.example.com (192.168.57.6) port 8080 (#0)
> POST /openam/json/authenticate HTTP/1.1
> Host: host1.example.com:8080
> User-Agent: curl/7.43.0
> Accept: */*
> cache-control: no-cache
> content-type: application/json
> Origin: http://other.origin.example.com:8080


Solution

1. Check the Accepted Methods section of the CORS filter for each method highlighted above:
<init-param>
<description>
Accepted Methods (Required):
A comma separated list of HTTP methods for which to accept CORS requests.
</description>
<param-name>methods</param-name>
<param-value>PUT</param-value>
</init-param>

2. Add the missing methods to this section accordingly:
<init-param>
<description>
Accepted Methods (Required):
A comma separated list of HTTP methods for which to accept CORS requests.
</description>
<param-name>methods</param-name>
<param-value>POST,PUT</param-value>
</init-param>

3. Restart the web application container in which AM/OpenAM runs.
4. Re-test the failing flow.

The following error can happen with Preflight Requests. Such requests use the OPTIONS method:

WARNING: CORS Fail - Preflight request contained the Access-Control-Request-Headers headers with an invalid value.


##### Note

The OPTIONS method does not need to be added as an Accepted Method in the filter as it is accepted by default.

Troubleshooting

In the HTTP trace, search for a Request that contains an Access Control Request Headers, such as:

'Access-Control-Request-Headers: anyHeader'


The response to such a request should contain the following anyHeader, for example:

Access-Control-Allow-Headers: anyHeader,otherHeader


Solution

1. Check the Allowed Headers section of the CORS filter if the Response does not contain the requested header:
<init-param>
<description>
A comma separated list of HTTP headers which can be included in the requests.
</description>
</init-param>

<init-param>
<description>
A comma separated list of HTTP headers which can be included in the requests.
</description>
</init-param>

3. Restart the web application container in which AM/OpenAM runs.
4. Re-test the failing flow.

#### Issue with the hostname

The following warning is displayed in the CoreSystem log:

WARNING: CORS Fail - Expected hostname does not equal actual hostname.


Troubleshooting

In the HTTP trace, search for a Request that contains an Origin Header, such as:

Origin:http://other.origin.example.com:8080



In the same request, check the Host header:

Host: host1.example.com:8080


Solution

1. The exact hostname must be present in the CORS filter, in the Expected Hostname section:
<init-param>
<description>
Expected Hostname (Optional):
The name of the host expected in the request Host header.
</description>
<param-name>expectedHostname</param-name>
<param-value>http://host1.example.com:8080</param-value>
</init-param>

2. Remove or comment out the Expected Hostname section in the CORS filter if you need more than one hostname (for example, if you use a load balancer) as AM/OpenAM does not currently accept lists of Expected Hostnames. An RFE exists for this: OPENAM-9890 (Allow list in expected Hostname section of CORS Filter).
3. Restart the web application container in which AM/OpenAM runs.
4. Re-test the failing flow.

N/A

## How do I troubleshoot high CPU utilization on AM/OpenAM (All versions) servers?

The purpose of this article is to provide troubleshooting guidance if you are experiencing consistently high CPU utilization on the AM/OpenAM server. If CPU utilization is in the range 70-80% under load, you should definitely investigate what is causing it as performance is likely to be impacted.

#### Troubleshooting high CPU usage

High CPU utilization is usually experienced in conjunction with other symptoms such as servers becoming unresponsive and poor performance.

##### Note

On Linux®, Unix® and Solaris® systems, there is a script you can run to capture the stack traces and process information as detailed in How do I use the msnapshots script to capture information for troubleshooting AM/OpenAM (All versions)?

There are a number of different types of data that you should collect to help us troubleshoot high CPU utilization. If you are seeing high CPU usage, you should collect the following data and submit it to us when you raise the ticket to enable us to help you more quickly:

• Debug and audit logs.
• JVM stack traces and current options.
• Garbage Collector (GC) logs.
##### Note

It is important that you collect all the data at the point you are experiencing high CPU usage so that the timestamps of the data collected and the high CPU usage correspond. Please notify us of the exact timestamp when high CPU was noticed so we can focus our attention on the relevant information.

Debug and audit logs

Message level debug logs provide more context for diagnosing errors but can generate large files. You should enable message level debugging when you are experiencing an issue as described in How do I enable Message level debugging in AM/OpenAM (All versions) debug files?

Audit log files (Access and Error) record all the AM/OpenAM action attempts, including successful results, and any service errors. They are located in the $HOME/[am_instance]/openam/log directory by default. JVM stack traces and current options JVM stack traces must be collected before killing the affected process or restarting the server; otherwise the information relating to the affected process is lost forever, which may make it impossible to identify the root cause. In addition to collecting stack traces, you should also provide details of the current JVM settings as they can provide an opportunity for tuning, which can reduce CPU usage. Collecting stack traces and finding current JVM settings is described in How do I collect JVM data for troubleshooting AM/OpenAM (All versions)? Process / thread information Detailed process / thread information is useful for identifying the exact thread within a process that is experiencing the high CPU. You should collect this data at the same time as the stack trace so we can look for the identified thread in the stack trace for more information. • On Unix® and Linux® systems, you can use the top command to output process information for all threads to a file: $ top -b -n 1 -H -p [pid] > top-output.txt
replacing [pid] with the process ID of the affected process.
• On Solaris®, you can use the prstat command to capture this information:
$prstat -L -p [pid] replacing [pid] with the process ID of the affected process. • On Microsoft® Windows®, you will need to use the Process Monitor. See How do I find which thread is consuming CPU in a Java process in AM/OpenAM (All versions)? for further information on identifying the thread that is causing high CPU usage. GC logs GC logs also provide useful information about tuning issues that may be contributing to the high CPU usage. Enabling GC logging is described in How do I enable Garbage Collector (GC) Logging for AM/OpenAM (All versions)? In addition, please include any information about other symptoms you are seeing at the same time as high CPU usage and any changes you have made to your system that may have contributed to the problem. #### See Also #### Related Training #### Related Issue Tracker IDs N/A ## How do I find which thread is consuming CPU in a Java process in AM/OpenAM (All versions)? The purpose of this article is to provide assistance on identifying which thread is consuming CPU in a Java® process in AM/OpenAM. This is a useful troubleshooting technique when you are experiencing high CPU under load. #### Identifying the thread You can identify which thread is consuming CPU in a Java process as follows: 1. Determine the process ID (PID) of the web container instance in which the affected server is running using the following command: $ jps -v
2. Determine which thread in the PID identified in step 1 is consuming the CPU:
• On Unix® and Linux® systems, you can use the top command:
$top -n 1 -H -p [pid] replacing [pid] with the process ID of the affected process. • On Solaris®, you can use the prstat command: $ prstat -L -p [pid]
replacing [pid] with the process ID of the affected process.
• On Microsoft® Windows®, you can use the Process Monitor.
These commands all give an output listing all the threads in the selected process along with various details, including the CPU usage percentage. For example, the top command gives an output similar to this:
Tasks: 152 total,   0 running, 152 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.0%us,  1.4%sy,  0.0%ni, 94.7%id,  0.8%wa,  0.0%hi,  1.1%si,  0.0%st
Mem:   5008484k total,  4854428k used,   154056k free,    85624k buffers
Swap:  5013496k total,     6516k used,  5006980k free,  3393948k cached
PID  USER     PR  NI  VIRT  RES  SHR S  %CPU  %MEM   TIME+  COMMAND
8706 user.1   20   0 2433m 561m  10m R  32.0  11.5  0:35.45 java
8624 user.1   20   0 2433m 561m  10m S   0.0  11.5  0:00.01 java
8625 user.1   20   0 2433m 561m  10m S   0.0  11.5  0:10.12 java
8626 user.1   20   0 2433m 561m  10m S   0.0  11.5  0:35.60 java
8627 user.1   20   0 2433m 561m  10m S   0.0  11.5  0:00.01 java
8628 user.1   20   0 2433m 561m  10m S   0.0  11.5  0:00.02 java              
In this example, we can see that thread ID 8706 is consuming 32% of the CPU.
3. Produce a stack trace for the PID identified in step 1 using the following command:
$jstack [PID] > jstack-output.txt 4. Convert the thread ID identified as problematic in step 2 to a hex value. In our example output, the thread ID decimal value 8706 converts to hex value 2202. 5. Search the stack trace output for this hex value using grep. You are looking for a thread nid that matches this hex value: $ cat jstack-output.txt | grep -i 2202
This command gives an output similar to this:
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f1448029800 nid=0x2202 runnable
In this example, we can see that the problem is a Garbage Collector (GC) thread, which suggests that GC is not properly tuned.

N/A

N/A

## Where can I find useful logs for troubleshooting ForgeRock products?

#### AM/OpenAM logs

The following logs are available:

Log type Location
AM/OpenAM debug logs $HOME/[am_instance]/openam/debug AM/OpenAM audit logs$HOME/[am_instance]/openam/log
Fedlet debug logs $HOME/fedlet/debug Container logs if deployed on Apache Tomcat™ (localhost_access_log.YYYY-MM-DD.log and catalina.out) /path/to/tomcat/logs SSL debug logs if deployed on Apache Tomcat (catalina.out) /path/to/tomcat/logs #### Policy Agents logs The following logs are available: Log type Location Web policy agents debug.log /path/to/policy_agent/instances/agent_n/logs/debug directory where the policy agent is installed. Web policy agents audit.log /path/to/policy_agent/instances/agent_n/logs/audit directory where the policy agent is installed. Web policy agents install log (installYYYYMMDDHHMMSS.log) /path/to/policy_agent/log directory where the policy agent is installed. Web policy agents system information: • system_n.log (Agents 5.5 and later) • agent.log (Pre-Agents 5.5) /path/to/policy_agent/log directory where the policy agent is installed. Java agents debug log (amAgent) /path/to/policy_agent/agent_n/logs/debug directory where the policy agent is installed. Java agents audit logs /path/to/policy_agent/agent_n/logs/audit directory where the policy agent is installed. Java agents install logs /path/to/policy_agent/installer-logs directory where the policy agent is installed. #### DS/OpenDJ logs The following logs are available: Log type Location Embedded DS/OpenDJ logs (access, errors and replication)$HOME/[am_instance]/opends/logs
External DS/OpenDJ logs (access, errors and replication) /path/to/ds/logs directory where DS/OpenDJ is installed.
SSL debug logs (server.out) /path/to/ds/logs directory where DS/OpenDJ is installed

#### IDM/OpenIDM logs

The following logs are available:

Log type Location
IDM/OpenIDM log (openidm0.log.n) /path/to/idm/logs
IDM/OpenIDM audit logs /path/to/idm/audit
Jetty® Request logs /path/to/idm/logs
DS/OpenDJ password sync plugin /path/to/ds/logs directory where DS/OpenDJ is installed.
Active Directory® password sync plugin (idm.log) The location is specified in the logPath registry entry under the HKEY_LOCAL_MACHINE\SOFTWARE\ForgeRock\OpenIDM\PasswordSync registry key.

#### IG/OpenIG logs

The following logs are available:

Log type Location
IG 5 and later $HOME/.openig/logs OpenIG 3.x and 4.x stdout (catalina.out if you are using Apache Tomcat™) or /tmp/gateway.log depending on how you have configured logging. SAML logs$HOME/.openig/SAML/debug

N/A

N/A

## How do I trace transactions through the audit logs for troubleshooting across ForgeRock products?

The purpose of this article is to provide information on tracing transactions through the audit logs to help with troubleshooting across ForgeRock products. This article covers an IG/OpenIG --> AM/OpenAM --> DS/OpenDJ flow to demonstrate this usage and applies to AM/OpenAM (All versions) and OpenAM 13.x, DS/OpenDJ (All versions) and IG/OpenIG (All versions).

#### Overview

A Trust Transaction Header system property (org.forgerock.http.TrustTransactionHeader) exists to allow the propagation of transaction IDs across ForgeRock products using the HTTP header X-ForgeRock-TransactionId. This header can also be passed to ForgeRock products from client applications so that the same transaction ID is used in client applications and ForgeRock products. For example, in flows such as Client Application --> IG/OpenIG --> AM/OpenAM --> DS/OpenDJ or Client Application --> AM/OpenAM --> DS/OpenDJ.

By default, this system property is set to false so that a malicious actor cannot flood the system with requests using the same transaction ID header to hide their tracks. Once enabled, you can trace a single transaction across multiple products in order to troubleshoot their interactions. See the following links for further information:

This article demonstrates using this HTTP header to trace a single transaction through the audit logs for an IG/OpenIG --> AM/OpenAM --> DS/OpenDJ flow and includes the same transaction in the Apache Tomcat™ access logs for a complete view of the transaction's journey.

##### Note

A related article exists for tracing transaction IDs through IDM/OpenIDM logs using the thread ID: How do I add Thread IDs to log statements in IDM/OpenIDM (All versions)?

#### Tracing transaction IDs

The following example demonstrates configuring the products to propagate transaction IDs and then verifying that they appear in the logs, including the Tomcat log:

1. Configure amHandler in IG/OpenIG to use the ForgeRockClientHandler in place of the ClientHandler. This change ensures the X-ForgeRock-TransactionId header values are passed to AM/OpenAM. For example, this demonstration is based on making this change to the amHandler in PolicyEnforcementFilter. See Configuration Reference › PolicyEnforcementFilter for further details.
2. Enable audit logging in IG/OpenIG as detailed in Gateway Guide › Auditing.
3. Configure DS/OpenDJ to trust transaction IDs sent with client application requests using dsconfig, for example:
$./dsconfig set-global-configuration-prop --advanced --port 4444 --hostname ds1.example.com --bindDN "cn=Directory Manager" --bindPassword password --set trust-transaction-ids:true --trustAll --no-prompt  4. Configure AM/OpenAM to trust any incoming X-ForgeRock-TransactionId headers as detailed in Setup and Maintenance Guide › Configuring the Trust Transaction Header System Property. 5. Update Tomcat's server.xml file to log the X-ForgeRock-Transactionid. You should modify the pattern to match the following example for the org.apache.catalina.valves.AccessLogValve className: <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log" suffix=".txt" pattern="%h %l %u %t &quot;%r&quot; %s %b %{X-ForgeRock-Transactionid}i" />  6. Invoke the IG/OpenIG route. 7. Note the transaction ID and then examine the following logs to trace the progress and corresponding operations of the request - In this example, we search the logs for transaction ID "7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27" (shown in bold): • IG/OpenIG access.audit.json log: {"realm":"/","transactionId":"2f34280f-c061-423c-8e64-adc9f8e12b95-508","userId":"id=jdoe,ou=user,o=employees,ou=services,dc=openam,dc=forgerock,dc=org","client":{"ip":"127.0.0.1","port":37796},"server":{"ip":"127.0.0.1","port":8989},"http":{"request":{"secure":false,"method":"POST","queryParameters":{"_action":["validate"]},"headers":{"accept-api-version":["protocol=1.0,resource=1.1"],"host":["host1.example.com:8989"],"user-agent":["Apache-HttpAsyncClient/4.1.2 (Java/1.8.0_151)"],"x-forgerock-transactionid":["7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/0"],"x-forwarded-for":["192.0.2.0"]},"cookies":{},"path":"http://host1.example.com:8989/openam/json/sessions"}},"request":{"protocol":"CREST","operation":"ACTION","detail":{"action":"validate"}},"timestamp":"2018-05-18T14:32:38.987Z","eventName":"AM-ACCESS-OUTCOME","component":"Session","response":{"status":"SUCCESSFUL","statusCode":"","elapsedTime":11,"elapsedTimeUnits":"MILLISECONDS"},"trackingIds":["2f34280f-c061-423c-8e64-adc9f8e12b95-469"],"_id":"2f34280f-c061-423c-8e64-adc9f8e12b95-510"} {"realm":"/","transactionId":"2f34280f-c061-423c-8e64-adc9f8e12b95-514","userId":"id=jdoe,ou=user,o=employees,ou=services,dc=openam,dc=forgerock,dc=org","client":{"ip":"192.0.2.0","port":55437},"server":{"ip":"192.0.2.24","port":8989},"http":{"request":{"secure":false,"method":"POST","queryParameters":{"_action":["getSessionInfo"]},"headers":{"accept":["application/json, text/javascript, */*; q=0.01"],"accept-api-version":["protocol=1.0,resource=2.0"],"dnt":["1"],"host":["host1.example.com:8989"],"referer":["http://host1.example.com:8989/openam/XUI/?service=LDAP&realm=/employees&goto=http://ig1.example.com:9292/home/pep"],"user-agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0"],"x-requested-with":["XMLHttpRequest"]},"cookies":{"i18next":"en-US"},"path":"http://host1.example.com:8989/openam/json/sessions"}},"request":{"protocol":"CREST","operation":"ACTION","detail":{"action":"getSessionInfo"}},"timestamp":"2018-05-18T14:32:39.009Z","eventName":"AM-ACCESS-OUTCOME","component":"Session","response":{"status":"SUCCESSFUL","statusCode":"","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS"},"trackingIds":["2f34280f-c061-423c-8e64-adc9f8e12b95-469"],"_id":"2f34280f-c061-423c-8e64-adc9f8e12b95-518"} {"realm":"/employees","transactionId":"7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/1","userId":"id=policyAdmin,ou=user,o=employees,ou=services,dc=openam,dc=forgerock,dc=org","timestamp":"2018-05-18T14:32:39.057Z","eventName":"AM-ACCESS-OUTCOME","component":"Authentication","response":{"status":"SUCCESSFUL","statusCode":"","elapsedTime":54,"elapsedTimeUnits":"MILLISECONDS"},"client":{"ip":"127.0.0.1","port":37797},"http":{"request":{"secure":false,"method":"POST","path":"http://host1.example.com:8989/openam/json/employees/authenticate","queryParameters":{},"headers":{"Accept-API-Version":["protocol=2.1"],"host":["host1.example.com:8989"],"user-agent":["Apache-HttpAsyncClient/4.1.2 (Java/1.8.0_151)"],"x-forgerock-transactionid":["7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/1"],"x-forwarded-for":["192.0.2.0"],"x-openam-username":["policyAdmin"]},"cookies":{}}},"trackingIds":["2f34280f-c061-423c-8e64-adc9f8e12b95-516"],"_id":"2f34280f-c061-423c-8e64-adc9f8e12b95-527"} {"realm":"/employees","transactionId":"2f34280f-c061-423c-8e64-adc9f8e12b95-529","userId":"id=policyAdmin,ou=user,o=employees,ou=services,dc=openam,dc=forgerock,dc=org","client":{"ip":"127.0.0.1","port":37796},"server":{"ip":"127.0.0.1","port":8989},"http":{"request":{"secure":false,"method":"POST","path":"http://host1.example.com:8989/openam/json/employees/policies","queryParameters":{"_action":["evaluate"]},"headers":{"accept-api-version":["protocol=1.0,resource=2.0"],"host":["host1.example.com:8989"],"user-agent":["Apache-HttpAsyncClient/4.1.2 (Java/1.8.0_151)"],"x-forgerock-transactionid":["7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/2"],"x-forwarded-for":["192.0.2.0"]},"cookies":{}}},"request":{"protocol":"CREST","operation":"ACTION","detail":{"action":"evaluate"}},"timestamp":"2018-05-18T14:32:39.083Z","eventName":"AM-ACCESS-OUTCOME","component":"Policy","response":{"status":"SUCCESSFUL","statusCode":"","elapsedTime":20,"elapsedTimeUnits":"MILLISECONDS","detail":[{"advices":{},"ttl":9223372036854775807,"resource":"http://igApp.example.com:8081/home/pep","actions":{"POST":true,"GET":true},"attributes":{"givenName":["Jane"],"dn":["uid=jdoe,ou=People,dc=example,dc=com"],"isMemberOf":[],"cn":["Jane Doe"]}}]},"trackingIds":["2f34280f-c061-423c-8e64-adc9f8e12b95-516"],"_id":"2f34280f-c061-423c-8e64-adc9f8e12b95-533"}  • Tomcat localhost_access_log file: 127.0.0.1 - - [18/May/2018:10:32:38 -0400] "POST /openam/json/sessions?_action=validate HTTP/1.1" 200 61 7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/0 192.0.2.0 - - [18/May/2018:10:32:38 -0400] "GET /openam/XUI/org/forgerock/openam/ui/common/models/JSONSchema.js?v=5.5.1 HTTP/1.1" 200 12619 - 192.0.2.0 - - [18/May/2018:10:32:38 -0400] "GET /openam/XUI/org/forgerock/openam/ui/common/models/JSONValues.js?v=5.5.1 HTTP/1.1" 200 16418 - 192.0.2.0 - - [18/May/2018:10:32:38 -0400] "GET /openam/XUI/org/forgerock/openam/ui/admin/services/SMSServiceUtils.js?v=5.5.1 HTTP/1.1" 200 6112 - 192.0.2.0 - - [18/May/2018:10:32:38 -0400] "GET /openam/XUI/org/forgerock/openam/ui/common/util/Promise.js?v=5.5.1 HTTP/1.1" 200 1827 - 192.0.2.0 - - [18/May/2018:10:32:39 -0400] "GET /openam/XUI/org/forgerock/openam/ui/common/models/schemaTransforms/transformEnumTypeToString.js?v=5.5.1 HTTP/1.1" 200 838 - 192.0.2.0 - - [18/May/2018:10:32:39 -0400] "GET /openam/XUI/org/forgerock/openam/ui/common/models/schemaTransforms/transformBooleanTypeToCheckboxFormat.js?v=5.5.1 HTTP/1.1" 200 643 - 192.0.2.0 - - [18/May/2018:10:32:39 -0400] "GET /openam/XUI/org/forgerock/openam/ui/common/models/schemaTransforms/warnOnInferredPasswordWithoutFormat.js?v=5.5.1 HTTP/1.1" 200 999 - 192.0.2.0 - - [18/May/2018:10:32:39 -0400] "POST /openam/json/sessions?_action=getSessionInfo HTTP/1.1" 200 297 - 127.0.0.1 - - [18/May/2018:10:32:39 -0400] "POST /openam/json/employees/authenticate HTTP/1.1" 200 164 7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/1 127.0.0.1 - - [18/May/2018:10:32:39 -0400] "POST /openam/json/employees/policies?_action=evaluate HTTP/1.1" 200 266 7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/2​  • AM/OpenAM access.audit.json file: {"realm":"/","transactionId":"2f34280f-c061-423c-8e64-adc9f8e12b95-508","userId":"id=jdoe,ou=user,o=employees,ou=services,dc=openam,dc=forgerock,dc=org","client":{"ip":"127.0.0.1","port":37796},"server":{"ip":"127.0.0.1","port":8989},"http":{"request":{"secure":false,"method":"POST","queryParameters":{"_action":["validate"]},"headers":{"accept-api-version":["protocol=1.0,resource=1.1"],"host":["host1.example.com:8989"],"user-agent":["Apache-HttpAsyncClient/4.1.2 (Java/1.8.0_151)"],"x-forgerock-transactionid":["7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/0"],"x-forwarded-for":["192.0.2.0"]},"cookies":{},"path":"http://host1.example.com:8989/openam/json/sessions"}},"request":{"protocol":"CREST","operation":"ACTION","detail":{"action":"validate"}},"timestamp":"2018-05-18T14:32:38.987Z","eventName":"AM-ACCESS-OUTCOME","component":"Session","response":{"status":"SUCCESSFUL","statusCode":"","elapsedTime":11,"elapsedTimeUnits":"MILLISECONDS"},"trackingIds":["2f34280f-c061-423c-8e64-adc9f8e12b95-469"],"_id":"2f34280f-c061-423c-8e64-adc9f8e12b95-510"} {"realm":"/","transactionId":"2f34280f-c061-423c-8e64-adc9f8e12b95-514","userId":"id=jdoe,ou=user,o=employees,ou=services,dc=openam,dc=forgerock,dc=org","client":{"ip":"192.0.2.0","port":55437},"server":{"ip":"192.0.2.24","port":8989},"http":{"request":{"secure":false,"method":"POST","queryParameters":{"_action":["getSessionInfo"]},"headers":{"accept":["application/json, text/javascript, */*; q=0.01"],"accept-api-version":["protocol=1.0,resource=2.0"],"dnt":["1"],"host":["host1.example.com:8989"],"referer":["http://host1.example.com:8989/openam/XUI/?service=LDAP&realm=/employees&goto=http://ig1.example.com:9292/home/pep"],"user-agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0"],"x-requested-with":["XMLHttpRequest"]},"cookies":{"i18next":"en-US"},"path":"http://host1.example.com:8989/openam/json/sessions"}},"request":{"protocol":"CREST","operation":"ACTION","detail":{"action":"getSessionInfo"}},"timestamp":"2018-05-18T14:32:39.009Z","eventName":"AM-ACCESS-OUTCOME","component":"Session","response":{"status":"SUCCESSFUL","statusCode":"","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS"},"trackingIds":["2f34280f-c061-423c-8e64-adc9f8e12b95-469"],"_id":"2f34280f-c061-423c-8e64-adc9f8e12b95-518"} {"realm":"/employees","transactionId":"7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/1","userId":"id=policyAdmin,ou=user,o=employees,ou=services,dc=openam,dc=forgerock,dc=org","timestamp":"2018-05-18T14:32:39.057Z","eventName":"AM-ACCESS-OUTCOME","component":"Authentication","response":{"status":"SUCCESSFUL","statusCode":"","elapsedTime":54,"elapsedTimeUnits":"MILLISECONDS"},"client":{"ip":"127.0.0.1","port":37797},"http":{"request":{"secure":false,"method":"POST","path":"http://host1.example.com:8989/openam/json/employees/authenticate","queryParameters":{},"headers":{"Accept-API-Version":["protocol=2.1"],"host":["host1.example.com:8989"],"user-agent":["Apache-HttpAsyncClient/4.1.2 (Java/1.8.0_151)"],"x-forgerock-transactionid":["7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/1"],"x-forwarded-for":["192.0.2.0"],"x-openam-username":["policyAdmin"]},"cookies":{}}},"trackingIds":["2f34280f-c061-423c-8e64-adc9f8e12b95-516"],"_id":"2f34280f-c061-423c-8e64-adc9f8e12b95-527"} {"realm":"/employees","transactionId":"2f34280f-c061-423c-8e64-adc9f8e12b95-529","userId":"id=policyAdmin,ou=user,o=employees,ou=services,dc=openam,dc=forgerock,dc=org","client":{"ip":"127.0.0.1","port":37796},"server":{"ip":"127.0.0.1","port":8989},"http":{"request":{"secure":false,"method":"POST","path":"http://host1.example.com:8989/openam/json/employees/policies","queryParameters":{"_action":["evaluate"]},"headers":{"accept-api-version":["protocol=1.0,resource=2.0"],"host":["host1.example.com:8989"],"user-agent":["Apache-HttpAsyncClient/4.1.2 (Java/1.8.0_151)"],"x-forgerock-transactionid":["7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/2"],"x-forwarded-for":["192.0.2.0"]},"cookies":{}}},"request":{"protocol":"CREST","operation":"ACTION","detail":{"action":"evaluate"}},"timestamp":"2018-05-18T14:32:39.083Z","eventName":"AM-ACCESS-OUTCOME","component":"Policy","response":{"status":"SUCCESSFUL","statusCode":"","elapsedTime":20,"elapsedTimeUnits":"MILLISECONDS","detail":[{"advices":{},"ttl":9223372036854775807,"resource":"http://igApp.example.com:8081/home/pep","actions":{"POST":true,"GET":true},"attributes":{"givenName":["Jane"],"dn":["uid=jdoe,ou=People,dc=example,dc=com"],"isMemberOf":[],"cn":["Jane Doe"]}}]},"trackingIds":["2f34280f-c061-423c-8e64-adc9f8e12b95-516"],"_id":"2f34280f-c061-423c-8e64-adc9f8e12b95-533"}  • DS/OpenDJ ldap-access.audit.json file: {"eventName":"DJ-LDAP","client":{"ip":"127.0.0.1","port":40459},"server":{"ip":"127.0.0.1","port":3389},"request":{"protocol":"LDAP","operation":"SEARCH","connId":20,"msgId":24,"dn":"dc=example,dc=com","scope":"sub","filter":"(uid=policyAdmin)","attrs":["dn","uid"]},"transactionId":"7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/1/0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"timestamp":"2018-05-18T14:32:39.018Z","_id":"7d2d88aa-26f4-4ca6-a248-0e1486292159-124403"} {"eventName":"DJ-LDAP","client":{"ip":"127.0.0.1","port":40460},"server":{"ip":"127.0.0.1","port":3389},"request":{"protocol":"LDAP","operation":"BIND","connId":21,"msgId":24,"version":"3","dn":"uid=policyAdmin,ou=people,dc=example,dc=com","authType":"SIMPLE"},"transactionId":"7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/1/1","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS"},"userId":"uid=policyAdmin,ou=people,dc=example,dc=com","timestamp":"2018-05-18T14:32:39.020Z","_id":"7d2d88aa-26f4-4ca6-a248-0e1486292159-124405"} {"eventName":"DJ-LDAP","client":{"ip":"127.0.0.1","port":40453},"server":{"ip":"127.0.0.1","port":3389},"request":{"protocol":"LDAP","operation":"SEARCH","connId":18,"msgId":43,"dn":"uid=policyAdmin,ou=people,dc=example,dc=com","scope":"base","filter":"(objectClass=*)","attrs":["inetuserstatus"]},"transactionId":"7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/1/2","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"timestamp":"2018-05-18T14:32:39.023Z","_id":"7d2d88aa-26f4-4ca6-a248-0e1486292159-124407"} {"eventName":"DJ-LDAP","client":{"ip":"127.0.0.1","port":40453},"server":{"ip":"127.0.0.1","port":3389},"request":{"protocol":"LDAP","operation":"SEARCH","connId":18,"msgId":44,"dn":"uid=policyAdmin,ou=people,dc=example,dc=com","scope":"base","filter":"(objectClass=*)","attrs":["inetuserstatus"]},"transactionId":"7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/1/3","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"timestamp":"2018-05-18T14:32:39.025Z","_id":"7d2d88aa-26f4-4ca6-a248-0e1486292159-124409"} {"eventName":"DJ-LDAP","client":{"ip":"127.0.0.1","port":40453},"server":{"ip":"127.0.0.1","port":3389},"request":{"protocol":"LDAP","operation":"SEARCH","connId":18,"msgId":45,"dn":"ou=groups,dc=example,dc=com","scope":"sub","filter":"(&(objectclass=groupOfUniqueNames)(uniqueMember=uid=policyAdmin,ou=people,dc=example,dc=com))","attrs":["dn"]},"transactionId":"7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/2/0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"timestamp":"2018-05-18T14:32:39.070Z","_id":"7d2d88aa-26f4-4ca6-a248-0e1486292159-124411"} {"eventName":"DJ-LDAP","client":{"ip":"127.0.0.1","port":40453},"server":{"ip":"127.0.0.1","port":3389},"request":{"protocol":"LDAP","operation":"SEARCH","connId":18,"msgId":46,"dn":"ou=groups,dc=example,dc=com","scope":"sub","filter":"(&(objectclass=groupOfUniqueNames)(uniqueMember=uid=policyAdmin,ou=people,dc=example,dc=com))","attrs":["dn"]},"transactionId":"7db465af-c0fc-4713-b1ac-0d2c889ea1b7-27/2/3","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"timestamp":"2018-05-18T14:32:39.079Z","_id":"7d2d88aa-26f4-4ca6-a248-0e1486292159-124413"} {"eventName":"DJ-LDAP","client":{"ip":"127.0.0.1","port":40272},"server":{"ip":"127.0.0.1","port":3389},"request":{"protocol":"LDAP","operation":"SEARCH","connId":16,"msgId":56,"dn":"","scope":"base","filter":"(objectClass=*)","attrs":["1.1"]},"transactionId":"0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"timestamp":"2018-05-18T14:32:41.824Z","_id":"7d2d88aa-26f4-4ca6-a248-0e1486292159-124415"} {"eventName":"DJ-LDAP","client":{"ip":"127.0.0.1","port":40273},"server":{"ip":"127.0.0.1","port":3389},"request":{"protocol":"LDAP","operation":"SEARCH","connId":17,"msgId":57,"dn":"","scope":"base","filter":"(objectClass=*)","attrs":["1.1"]},"transactionId":"0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"timestamp":"2018-05-18T14:32:41.879Z","_id":"7d2d88aa-26f4-4ca6-a248-0e1486292159-124417"}  ##### Note Tracing the journey of a single transaction can be very effective in locating root cause when it's not really possible to go by timestamp alone. For example, if you are experiencing delays with response times, you can now see if a specific user or group of users has any missing attributes that are expected or needs special consideration during AuthN processing of a request. #### See Also #### Related Training N/A #### Related Issue Tracker IDs N/A ## How do I enable Message level debugging in AM/OpenAM (All versions) debug files? The purpose of this article is to provide information on enabling Message level debugging in AM/OpenAM debug files. Message level debug files are frequently used for troubleshooting issues but are not recommended generally due to their size. #### Background information ##### Note In OpenAM 13 onwards, you can use the Recording facility to capture debug logs. See How do I record troubleshooting information in AM (All versions) and OpenAM 13.x? and Best practice for recording troubleshooting information in AM (All versions) and OpenAM 13.x for further information. You can enable Message level debugging for all debug files or for individual debug files using Debug.jsp. You must have top level admin privileges to do this and be logged into AM/OpenAM as the admin user (called amadmin by default). There are four different debug levels available: • Off - disables debug logging. • Error - only logs error messages (default). • Warning - logs both warning and error messages. • Message - logs detailed diagnostic messages and debugging information. Debug location Debug files are located in the$HOME/[am_instance]/openam/debug directory by default. You can check which directory is being used or change it using either the console or ssoadm:

• AM / OpenAM 13.5 console: navigate to Deployment > Servers > [Server Name] > General > Debugging > Debug Directory.
• Pre-OpenAM 13.5 console: navigate to Configuration > Servers and Site > [Server Name] > General > Debugging > Debug Directory.
• ssoadm: enter the following command to view current settings and look for the com.iplanet.services.debug.directory property:
$./ssoadm list-server-cfg -s [servername] -u [adminID] -f [passwordfile] replacing [servername], [adminID] and [passwordfile] with appropriate values. • ssoadm: enter the following command to update the directory: $ ./ssoadm update-server-cfg -s [servername] -u [adminID] -f [passwordfile] -a com.iplanet.services.debug.directory=[directory]

Debug files

The files in this folder reflect the debug logs for different components; a standard installation includes the following debug files:

• Authentication
• Configuration
• CoreSystem
• Entitlement
• Federation
• IdRepo
• Policy
• Session

#### Enabling Message level debugging for all debug files

You can enable Message level debugging for all debug files using either the console or ssoadm:

• AM / OpenAM 13.5 console: navigate to Deployment > Servers > [Server Name] > General > Debugging > Debug Level and select Message.
• Pre-OpenAM 13.5 console: navigate to: Configuration > Servers and Site > [Server Name] > General > Debugging > Debug Level and select Message.
• ssoadm: enter the following command:
$./ssoadm update-server-cfg -s [servername] -u [adminID] -f [passwordfile] -a com.iplanet.services.debug.level=message replacing [servername], [adminID] and [passwordfile] with appropriate values. Once you have reproduced the problem and captured the debug logs, you should revert the debug level to Error to avoid filling up the disks where the debug logs are stored. ##### Caution Increasing the debug level to message will generate large debug logs, so you are advised to look at log rotation to stop disks filling up: How do I rotate AM/OpenAM (All versions) debug logs? In OpenAM 13, there is a known issue that causes the EmbeddedDJ debug log to grow excessively: OPENAM-8696 (OpenAM 13 EmbeddedDJ log spam ). This is fixed in OpenAM 13.5. #### Enabling Message level debugging for individual files You can enable Message level debugging for individual debug files providing you are logged in as follows: 1. Navigate to: <protocol>://host.fqdn:port/openam/Debug.jsp, for example: http://host1.example.com:8080/openam/Debug.jsp. An example Debug.jsp page looks like this: 1. Select the required debug file from the Category field, select Message from the Level field and then click Submit to change the debug level. Once you have reproduced the problem and captured the debug logs, you should revert the debug level to Error to avoid filling up the disks where the debug logs are stored. ##### Note The Debug.jsp page always shows the debug level as Error, regardless of its actual setting. #### See Also #### Related Training #### Related Issue Tracker IDs ## How do I enable debug logging for troubleshooting Policy Agents (All versions)? The purpose of this article is to provide information on raising the debug logging level for Web and Java Policy Agents. Support frequently use these logging levels for troubleshooting policy agent issues but they are not generally recommended for production due to increase in log size. #### Background information There are different debug levels available to policy agents (varying according to type): Level Description Available to Web policy agents? Available to Java policy agents? Off Disables debug logging. -- Yes Error Logs only error messages. Yes (default) Yes (default) Warning Logs both warning and error messages. Yes Yes Info Logs info, warnings and errors messages. Yes -- Message * Logs debug, info, warning and error messages. Yes Yes All * Max Debug logging. Logs detailed diagnostic messages and debugging information. Yes -- * In Web policy agents 4.2, level All is the same as Message and provides the same output to better match AM/OpenAM's message levels. By default, the policy agent debug log is named and located as follows: • Web policy agents - debug.log, which is located in the /path/to/policy_agent/instances/agent_n/logs/debug directory. • Java policy agents - amAgent, which is located in the /path/to/policy_agent/agent_n/logs/debug directory. #### Web policy agents Depending on what you are trying to troubleshoot for web policy agents, and whether you have a local or centralized configuration will affect where you need to set the log levels: The log levels in AM/OpenAM correspond to log levels in the agent.conf file as follows: AM/OpenAM (console, ssoadm and Amster) log levels agent.conf file log levels Error error Warning warning Info info Message warning All debug Once you have reproduced the problem and captured the debug.log, you should revert the debug level to 'Error' to avoid filling up the disk where the debug log is stored. Troubleshooting startup issues and local configurations To troubleshoot startup issues and local configurations once they are up and running, you should set the log level in the agent.conf file. Edit the com.sun.identity.agents.config.debug.level property and set it as follows (you must use lowercase): com.sun.identity.agents.config.debug.level=debug This local level setting will be overwritten once the policy agent has fully started up but is useful for troubleshooting startup issues. Troubleshooting issues once the policy agent is up and running (centralized configurations) To troubleshoot centralized configurations once the policy agent is up and running, you should set the log level using either the console, ssoadm or Amster (AM 5 and later). These examples show the log level being raised to 'All': • AM 5 and later console: navigate to: Realms > [Realm Name] > Applications > Agents > Web > [Agent Name] > Global > Agent Debug Level and select 'All'. • OpenAM 13.x console: navigate to: Realms > [Realm Name] > Agents > Web > [Agent Name] > Global > Agent Debug Level and select 'All'. • Pre-OpenAM 13 console: navigate to: Access Control > [Realm Name] > Agents > Web > [Agent Name] > Global > Agent Debug Level and select 'All'. • ssoadm: enter the following command: $ ./ssoadm update-agent -e [realmname] -b [agentname] -u [adminID] -f [passwordfile] -a com.sun.identity.agents.config.debug.level=all
• Amster: follow the steps in How do I update property values in AM (All versions) using Amster?with these values:
• Entity: WebAgents
• Property: agentDebugLevel
• Value: All

#### Java policy agents

Depending on what you are trying to troubleshoot for Java policy agents, and whether you have a local or centralized configuration will affect where you need to set the log levels:

Once you have reproduced the problem and captured the amAgent debug log , you should revert the debug level to 'Error' to avoid filling up the disk where the debug log is stored.

Troubleshooting startup issues and local configurations

To troubleshoot startup issues and local configurations once they are up and running, you should set the log level in the OpenSSOAgentBootstrap.properties file. Edit the com.iplanet.services.debug.level property and set it as follows:

com.iplanet.services.debug.level=message

This local level setting will be overwritten once the policy agent has fully started up but is useful for troubleshooting startup issues.

Troubleshooting issues once the policy agent is up and running (centralized configurations)

For centralized configurations, you can enable Message level debugging for Java policy agents using either the console, ssoadm or Amster (AM 5 and later):

• AM 6 and later console: navigate to: Realms > [Realm Name] > Applications > Agents > Java > [Agent ID] > Global > Agent Debug Level and select 'message'.
• AM 5.x console: navigate to: Realms > [Realm Name] > Applications > Agents > J2EE > [Agent Name] > Global > Agent Debug Level and select 'message'.
• OpenAM 13.x console: navigate to: Realms > [Realm Name] > Agents > J2EE > [Agent Name] > Global > Agent Debug Level and select 'message'.
• Pre-OpenAM 13 console: navigate to: Access Control > [Realm Name] > Agents > J2EE > [Agent Name] > Global > Agent Debug Level and select 'message'.
• ssoadm: enter the following command:
$./amster -d  This option outputs debug information to the command line while Amster starts and also when you run any Amster commands within the current session (that is, up until you exit Amster). This output can be quite verbose, so this option should only be used if you are experiencing issues. #### See Also #### Related Training N/A #### Related Issue Tracker IDs N/A ## How do I enable debug logging for troubleshooting WDSSO and Kerberos issues in AM/OpenAM (All versions)? The purpose of this article is to provide information on enabling debug logging for troubleshooting Windows Desktop SSO (WDSSO) and Kerberos™ issues in AM/OpenAM. Debug logging applies to the Krb5Login module of the JVM used by the web application container; this module is called by AM/OpenAM for WDSSO purposes. #### Enabling debug logging You can enable debug logging for the Krb5Login module of the JVM by setting the following JVM options to true in the application web container in which AM/OpenAM runs: -Dsun.security.krb5.debug -Dsun.security.jgss.debug  This adds additional debug output from the Krb5Login module to the stdout. Additionally, you can set the following JVM option to true to enable debug logging for the SPNEGO token: -Dsun.security.spnego.debug ##### Note You should also ensure you have enabled Message level debugging in the AM/OpenAM debug logs as this provides much more information in the Authentication log. See How do I enable Message level debugging in AM/OpenAM (All versions) debug files? for further information. Example using Apache Tomcat™ web container With AM/OpenAM running in the Tomcat web container, you would enable debug logging by specifying CATALINA_OPTS settings in the setenv.sh file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.sh file (located in the /tomcat/bin/ directory). To enable debug logging: 1. Add the following line to the setenv.sh file: export CATALINA_OPTS="-Dsun.security.krb5.debug=true -Dsun.security.jgss.debug=true -Dsun.security.spnego.debug=true" 2. Restart the web container. The additional debug output will be sent to the Tomcat catalina.out log file by default. ##### Note If you can't find an issue on the AM/OpenAM side or instead believe it to be an issue on the Microsoft® Windows® side, you can enable Kerberos event logging on the Windows / Active Directory® server as detailed in: Microsoft - How to enable Kerberos event logging. #### See Also #### Related Training N/A #### Related Issue Tracker IDs ## How do I enable message level debugging for the RADIUS server in AM (All versions) and OpenAM 13.x? The purpose of this article is to provide information on enabling message level debugging for the Remote Authentication Dial-In User Service (RADIUS) server service in AM/OpenAM. #### Enabling message level debugging The Radius debug log shows all messages related to the RADIUS server service, specifically for the amRadiusCommon and amRadiusServer instances. You can enable message level debugging for all logs as described in How do I enable Message level debugging in AM/OpenAM (All versions) debug files? or you can just set it for the RADIUS server as follows: 1. Log into AM/OpenAM as the admin user (called amadmin by default). 2. Navigate to: <protocol>://host.fqdn:port/openam/Debug.jsp, for example: http://host1.example.com:58080/openam/Debug.jsp. 3. Select Radius from the Category field. 4. Select Message from the Level field and click Submit to change the debug level. ##### Note The Debug.jsp page always shows the debug level as Error, regardless of its actual setting. The Radius debug log is located in the$HOME/[am_instance]/openam/debug directory by default.

Once you have reproduced the problem and captured the debug logs, you should revert the debug level to Error to avoid filling up the disks where the debug logs are stored.

#### Debug logs for a working RADIUS server

The following log shows a successful configuration of the RADIUS Server Service for comparison purposes:  See RADIUS Server Guide › Implementing the RADIUS Server Service › Configuring the RADIUS Server Service

amRadiusServer:08/23/2016 10:06:17:581 AM PDT: Thread[RADIUS-1812-Listener,5,main]: TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657] RadiusServerEventRegistrar.packetReceived() called by EventBus amRadiusServer:08/23/2016 10:06:17:583 AM PDT: Thread[RADIUS-1812-Listener,5,main]: TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657] RadiusServerEventRegistrar.packetReceived() - total now 1
Entering globalConfigChange()
Leaving globalConfigChange()
Entering globalConfigChange()
Leaving globalConfigChange()
Entering globalConfigChange()
Leaving globalConfigChange()
Entering ConfigChangeListener.waitForConfigChange()
Entering ConfigChangeListener.waitForConfigChange()
amRadiusServer:08/23/2016 10:16:28:844 AM PDT: Thread[RADIUS-RadiusServerManager,5,main]: TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-33] Entering ConfigChangeListener.waitForConfigChange()


The following log shows a successful authentication attempt by the RADIUS client:

Log Packet Contents for this Client is enabled so that the contents of the packets are logged to the Radius debug log. See Reference › Configuration Reference › RADIUS Server

amRadiusServer:08/23/2016 10:19:34:958 AM PDT: Thread[pool-11-thread-1,5,main]:
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
Entering OpenAMAuthHandler.handle
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
Entering OpenAMAuthHandler.startAuthProcess
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
Leaving OpenAMAuthHandler.startAuthProcess
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
Entering gatherUserInput();
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
--- callbacks == null in gatherUserInput
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
--- NextCallbackSet not-available in gatherUserInput - move to finalization
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
Entering OpenAMAuthHandler.finalizeAuthProcess()
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
Entering ContextHolder.getUniversalId()
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
Leaving ContextHolder.getUniversalId()
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
Leaving OpenAMAuthHandler.handle();
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
Constructed AuthRequestAcceptedEvent.AuthRequestAcceptedEvent()
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
accepted 1
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
WARNING:
ACCESS_ACCEPT [1]
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
Sending response of type ACCESS_ACCEPT to RadiusTestServer
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
Constructing PacketProcessedEvent.PacketProcessedEvent()
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
TransactionId[4dcc2bd6-f055-4ae9-a68d-ea037341fc3e-657]
RadiusServerEventRegistrar.packetProcessed() - total now 1


N/A

N/A

## How do I enable message level debugging for the Java SDK in OpenAM 12.x and 13.x?

The purpose of this article is to provide information on enabling message level debugging for the Java® SDK in OpenAM. The Java SDK is also referred to as the Client SDK. The Java SDK was deprecated in AM 5 and removed in AM 5.5.

#### Enabling message level debugging

You can enable message level debugging for the Java SDK by setting the following property in the AMConfig.properties file (located in the /path/to/tomcat/webapps/openam/WEB-INF/classes directory):

com.iplanet.services.debug.level=message


You should also check the setting for the com.iplanet.services.debug.directory property in this file as this is where the Java SDK logs, including the amRemotePolicy log, are kept.

Once you have reproduced the problem and captured the debug logs, you should revert the debug level to error to avoid filling up the disks where the debug logs are stored.

N/A

N/A

## How do I find network errors in the logs for the configuration store in AM/OpenAM (All versions)?

The purpose of this article is to provide information on finding network errors in the logs for the configuration store in AM/OpenAM. This is useful if you suspect network errors are causing connection issues with the configuration store.

#### Finding network errors

The Configuration debug log shows network errors when the debug level is set to Warning or Message (default is Error). You can set this debug level for all logs as described in How do I enable Message level debugging in AM/OpenAM (All versions) debug files? or you can just set it for the amEventService instance as follows:

1. Log into AM/OpenAM as the admin user (called amadmin by default).
2. Navigate to: <protocol>://host.fqdn:port/openam/Debug.jsp, for example: http://host1.example.com:8080/openam/Debug.jsp.
3. Select amEventService from the Debug Instances field.
4. Select Warning or Message from the Level field and click Submit to change the debug level.
##### Note

The Debug.jsp page always shows the debug level as Error, regardless of its actual setting.

If you have network based errors, you will see errors such as the following in your Configuration log (located in the $HOME/[am_instance]/openam/debug directory by default): amEventService WARNING: EventService.run() LDAPException received: com.sun.identity.ldap.LDAPException: Server or Network error (81) at com.sun.identity.shared.ldap.LDAPConnThread.networkError(LDAPConnThread.java:910) at com sun.identity.shared.ldap.LDAPConnThread.networkError(LDAPConnThread.java:895) at com.sun.identity.shared.ldap.LDAPConnThread.run(LDAPConnThread.java:676) at java.lang.Thread.run(Thread.java:745) ...  Once you have reproduced the problem and captured the debug logs, you should revert the debug level to Error to avoid filling up the disks where the debug logs are stored. #### See Also #### Related Training N/A #### Related Issue Tracker IDs N/A ## How do I collect all the data required for troubleshooting AM/OpenAM and Policy Agents (All versions)? The purpose of this article is to provide assistance with collecting troubleshooting data and diagnostics for debugging AM/OpenAM and Policy Agents on a variety of issues; includes performance, upgrade, configuration, policies, SSL, SAML federation and WDSSO / Kerberos™ issues. You can capture the data to troubleshoot issues yourself or include it when you raise a ticket to enable us to help you more effectively. #### Collecting Troubleshooting Data If you are experiencing issues with AM/OpenAM and/or your policy agents, you can collect data to help troubleshoot your issues. If you need to raise a ticket to troubleshoot an issue, you should include relevant information to help us investigate your issues more effectively. Refer to Best practice for raising a ticket with ForgeRock Support for further information on creating a ticket. Follow this guide for submitting technical information needed to help us resolve your issue as soon as possible. ##### Note You can attach log files to tickets in the BackStage support portal. See Sending troubleshooting data to ForgeRock Support for analysis for further information. ##### Caution It is very important that timestamps match across the entire set of logs and diagnostic data for all servers and components in the request flow. Failing to do this may mean capturing all the data again so timestamps correspond when the issue is seen. The troubleshooting data needed for the following types of issues are covered in this article: Finally, there are some other logs available that can be helpful as detailed in the Other logs section. #### General issues with AM/OpenAM and/or Policy Agents In OpenAM 13 onwards, you can use the Recording facility to capture debug logs, configuration details and thread dumps. See How do I record troubleshooting information in AM (All versions) and OpenAM 13.x? and Best practice for recording troubleshooting information in AM (All versions) and OpenAM 13.x for further information. 1. Export configuration details for AM/OpenAM and the Policy Agent - it is essential to check if a configuration issue is contributing to your issues. You can export these as described in How do I export and import Service configurations for AM/OpenAM (All versions)?and How do I export configuration details for the Policy Agent (All versions)? 2. Generate a full set of debug logs and capture the HTTP trace while reproducing the issue. You should follow this process: 1. Enable message level debugging as described in How do I enable Message level debugging in AM/OpenAM (All versions) debug files? and How do I enable debug logging for troubleshooting Policy Agents (All versions)? 2. Clear the AM/OpenAM and Policy Agent debug logs as described in How do I clear debug logs in AM/OpenAM (All versions)? 3. Reproduce the issue with the REST API by running the relevant curl command. This helps us to isolate the issue. Example curl commands are available in: Development Guide › Developing with the REST API. 4. Start a HTTP trace in your browser. There are free third-party tools you can use, for example, in Firefox®, you can use Live HTTP Headers. 5. Reproduce the issue using your browser. 6. Stop the HTTP trace. 3. Capture HTTP container logs (Access and Error) for both AM/OpenAM and policy agents with date and timestamps that correspond to the policy agent and AM/OpenAM debug logs. 4. Capture LDAP server logs for your data store and configuration store with date and timestamps that correspond to the policy agent and AM/OpenAM debug logs. There are some use cases that require analysis of the LDAP server logs from AM/OpenAM data store (user profile) and Configuration store. 5. Revert the debug level in your AM/OpenAM and Policy Agent debug logs to Error to avoid filling up the disks where the debug logs are stored. 6. Attach the following diagnostic logs and outputs to your support request as detailed in Sending troubleshooting data to ForgeRock Support for analysis • AM/OpenAM and policy configuration exports. • Log files from: • AM/OpenAM -$HOME/[am_instance]/openam/debug and $HOME/[am_instance]/openam/logs • Policy Agent /path/to/policy_agent/instances/agent_n/logs/debug directory or /path/to/policy_agent/agent_n/logs/debug directory where the policy agent is installed (depending on policy agent version). • HTTP trace - from the HTTP trace browser tool. • HTTP container logs - the actual name and location of the server logs are specific to your web container. For example, for AM/OpenAM deployed on Apache Tomcat, the Tomcat logs are called localhost_access_log.YYYY-MM-DD.log and catalina.out, and are located in the /path/to/tomcat/logs/ directory by default. • LDAP server logs - for embedded DS/OpenDJ, these log files are located in the$HOME/[am_instance]/opends/logs directory by default and for external DS/OpenDJ, they are located in the /path/to/ds/logs directory.
• Output from the curl command and response
7. Update your ticket with the following information:
• AM/OpenAM and Policy Agent versions; some issues are version specific so knowing your version will help us focus on relevant issues only.
• Web application container / Web server type and version.
• Detailed problem description with steps to reproduce the issue.
• Date and time at which issue was first seen.
• Details of any changes that have been made to your environment prior to the issue being seen, such as, new install, upgrade, configuration, applications, introduced load balancer, replication, increased user load, new SP or IdP etc.
• Updated deployment diagram if applicable. Ensure to include any load balancers, proxies and firewalls. You can use Projects to create and maintain your deployment diagram to ensure we always have the latest details about your environment.
• Timestamp(s) of when the issue was reproduced so we can correlate this with the logs.

#### High CPU

Logs specific to installation / upgrade process. You can enable these as described in How do I enable message level debugging for install and upgrade issues with AM/OpenAM (All versions) ?

Message level ssoadm logs. You can enable these as described in How do I enable message level debugging for ssoadm in AM/OpenAM (All versions)?

#### Amster issues

Debug mode in Amster. You can enable this as described in How do I enable debug mode for troubleshooting Amster (All versions)?

#### LDAP connection issues (connecting to a LDAP server from AM/OpenAM)

• Data per General issues with AM/OpenAM and/or Policy Agents.
• Run the following command from the AM/OpenAM server to check the basic connection to the LDAP server:
$telnet [host] [port] • Perform a simple ldapsearch using the same credentials, server details and base DN to check if you can connect that way. • Capture a network trace using the following process: 1. Stop the AM/OpenAM server. 2. Start the network trace, for example, you could use the tcpdump command as follows: tcpdump -i any -s 0 -w /tmp/openamldap.pcap port 389  3. Run the ldapsearch command you ran in the previous step. 4. Start the AM/OpenAM server and wait for the errors to appear in the IdRepo log. 5. Stop the network trace. #### SSL connection issues #### LDAPS connection issues (connecting to DS/OpenDJ from AM/OpenAM) #### Policy evaluation issues • Data per General issues with AM/OpenAM and/or Policy Agents. • Export the policy configuration - it is essential to check if a configuration issue is contributing to your issues. You can export this as described in How do I export and import policies in AM/OpenAM (All versions)? • Identify the stage at which the policy evaluation failed if possible. See Best practice for creating and testing policies in AM/OpenAM (All versions) for further information on identifying which stage failed. • Reproduce the issue with the REST API - send us the curl command you used and the response. For example: $ curl -X POST -H "Content-Type: application/json" -H "Accept-API-Version: resource=1.0, protocol=1.0" -H "iPlanetDirectoryPro: AQIC5...DU3*" -d '{
"resources": [
"http://www.example.com/index.html",
"http://www.example.com/do?action=run"
],
"application": "iPlanetAMWebAgentService"
}' https://host1.example.com:8443/openam/json/realms/root/policies?_action=evaluate

[ {
"resource" : "http://www.example.com/do?action=run",
"actions" : {
},
"attributes" : {
},
}
}, {
"resource" : "http://www.example.com/index.html",
"actions" : {
"POST" : false,
"GET" : true
},
"attributes" : {
"cn" : [ "demo" ]
},
}
} ]

#### SAML issues

• Data per General issues with AM/OpenAM and/or Policy Agents - the Federation debug log is particularly useful.
• Export the metadata (standard and extended) - it is essential to check if a configuration issue is contributing to your issues. You can export this as described in How do I export and import SAML metadata in AM/OpenAM (All versions)?
• Capture a SAML trace while reproducing the issue. You should follow this process:
• Start a SAML trace in your browser. There are free third-party tools you can use, for example, in Firefox®, you can use SAML Tracer.
• Reproduce the issue using your browser.
• Stop the SAML trace.
If you cannot capture a SAML trace, a HTTP trace will suffice.

#### Other logs

There are other logs that may be called for depending on the issue you are experiencing. These are:

• Stats logs - these files (amMasterSessionTableStats, amPolicyStats, Entitlements and idRepoCacheStat; you may also see amArtifactMap and amAssertionMap) provide statistics for the associated service. They are located in the $HOME/[am_instance]/openam/stats directory by default. You can check or change stats settings: • AM / OpenAM 13.5 console: navigate to: Configure > Server Defaults > Session > Statistics. • Pre-OpenAM 13.5 console: navigate to: Configuration > Servers and Sites > Default Server Settings > Session > Statistics. amMasterSessionTableStats is the mostly commonly used stats file, and contains both user and application session data. See How do I monitor session statistics in AM/OpenAM (All versions)? for further information. • Policy agent audit logs - these logs provide details about URL access attempts and whether they are successful or unsuccessful. Audit logs are logged remotely (default), locally or both: You can check or change log settings by navigating to: Realms > [Realm Name] > Agents > [Web or J2EE] > [Agent Name] > Global > Audit in the AM / OpenAM 13.x console or Access Control > [Realm Name] > Agents > [Web or J2EE] > [Agent Name] > Global > Audit in the Pre-OpenAM 13 console. • Remotely - located in the$HOME/[am_instance]/openam/log directory.
• Locally - located in the /path/to/policy_agent/instances/agent_n/logs/audit directory or the /path/to/policy_agent/agent_n/logs/audit directory where the policy agent is installed (depending on policy agent version).
##### Note

For Apache Tomcat™, you also need to ensure the Apache user has group write permissions to the /audit directory and that the directory structure permissions are set to 666 for local audit logging to work correctly.

N/A

## How do I clear debug logs in AM/OpenAM (All versions)?

The purpose of this article is to provide information on clearing your debug logs in AM/OpenAM when they get too large (for example, you previously set the debug level to message) and you no longer need them. You can also use this information to clear other logs, such as amAgent. Clearing debug logs as opposed to deleting them negates the need to stop and start your AM/OpenAM server. The information in this article applies if you are using a UNIX® or Linux® system.

#### Clearing an individual debug log

You can clear an individual debug log as follows from your terminal window:

1. Change directory to your debug directory (which is typically located in the $HOME/[am_instance]/openam/ directory). 2. Copy any logs you may want to keep. Once you run the next command, the logs will be permanently gone. 3. Enter the following command: $ cat /dev/null >[filename]
replacing [filename] with the name of the debug log file you want to clear.

#### Clearing all debug logs in the debug directory

You can clear all debug logs in the debug directory as follows:

1. Copy any logs you may want to keep. Once you run the next command, the logs will be permanently gone.
2. Enter the following command in your terminal window:
$for file in /path/to/debug/*; do >$file; done
For example:
$for file in$HOME/openam1/openam/debug/*; do > $file; done to empty all the debug log files in the debug directory. Alias You could also create an alias to run this command to make it easy to re-use. For example, enter the following command: $ alias flushamdebug='for file in $HOME/openam1/openam/debug/*; do >$file; done'

and then just enter the following command to run it at any time:

#### Rotating debug logs

AM/OpenAM does not rotate debug logs by default, but you can configure it to do so using either time interval or size:

• Time interval (in minutes) - for example, every 1440 minutes for once a day. The time interval specified starts when the first log message is logged to the log file.
• Size (in MB) - for example, when the debug log reaches 2MB. The size option is available as of OpenAM 13. In OpenAM 12, you should consider tools such as logadm (Solaris®) or logrotate (Linux®) if you want to rotate by size. These are third-party tools that we suggest can be used for log rotation but are not supported by ForgeRock.

​You can rotate debug log files by updating the debugconfig.properties file, which is located in the /path/to/tomcat/webapps/openam/WEB-INF/classes directory where AM/OpenAM is deployed.

##### Note

It is recommended that you copy the debug logs prior to clearing the contents, rather than deleting the logs as this can cause issues if a process is still holding a filehandle. See How do I clear debug logs in AM/OpenAM (All versions)? for further information. Additionally, you should keep debug logging to a minimum (level: error) and only increase it when troubleshooting an issue.

Time Interval

1. Edit the debugconfig.properties file and set the org.forgerock.openam.debug.rotation property and optionally the org.forgerock.openam.debug.prefix and/or org.forgerock.openam.debug.suffix properties. For example, if you want your debug logs to rotate every 1440 minutes with a suffix of the date and time (timestamp), you would set the rotation and suffix properties in this file as follows:
org.forgerock.openam.debug.rotation=1440
org.forgerock.openam.debug.suffix=-MM.dd.yyyy-HH.mm

2. Restart the web application container in which AM/OpenAM runs to apply these configuration changes.

You would then see debug logs rotated every 1440 minutes with the timestamp indicating when they were rotated. For example:

...
CoreSystem-07.22.2016-16.23
CoreSystem-07.21.2016-16.23
CoreSystem-07.20.2016-16.23
CoreSystem-07.19.2016-16.23
...


Size

1. Edit the debugconfig.properties file and set the org.forgerock.openam.debug.rotation.maxsize property and optionally the org.forgerock.openam.debug.prefix and/or org.forgerock.openam.debug.suffix properties. For example, if you want your debug logs to rotate each time they reach 2MB with a suffix of the date and time (timestamp), you would set the maxsize and suffix properties in this file as follows:
org.forgerock.openam.debug.rotation.maxsize=2
org.forgerock.openam.debug.suffix=-MM.dd.yyyy-HH.mm_ss.SSS
2. Restart the web application container in which AM/OpenAM runs to apply these configuration changes.

N/A

## How do I add logging to server-side scripts in AM (All versions) and OpenAM 13.x?

#### Background information

The Scripting API in AM/OpenAM uses the logger object to log debug messages from scripts to the AM/OpenAM debug logs. Debug logging in scripts is not enabled by default; you must enable it by setting the debug log level for the amScript service.

Debug log messages from scripts are logged to the following AM/OpenAM debug logs depending on the type of script:

Script type Log file
Authentication Authentication
Policy Condition Entitlement
OIDC claims OAuth2Provider

The available logger methods that can be used are detailed in: Development Guide › Developing with Scripts › Debug Logging.

#### Logging in scripts

This example shows how you would output an error message in the Authentication log for an authentication related script. The log level selected and the log level specified in the script should match, else a message will not be output.

1. Set the required debug level for the amScript service by navigating to the Debug.jsp page, for example: http://host1.example.com:8080/openam/Debug.jsp and setting the following fields:
• Select the Authentication debug file from the Category field.
• Select Scripting from the Debug instances field.
• Select Error from the Level field.
2. Click Submit to change the debug level.
3. Update your script to include the required logger code. You can either do this via the Console or the REST API depending on how you manage your scripts. For example, you could add the following to the script you want to debug:
logger.error("Test error message logged for Authentication");

This message will output to the Authentication log when the script is run.

N/A

N/A

## How do I log audit events to a database in OpenAM 12.x and 13.x?

#### Overview

OpenAM 13.0 introduces the ForgeRock Common Audit Framework, which allows OpenAM to log all user and administrative activity in a consistent format across the ForgeRock platform. Logs can be written to file, database or syslog as required. See OpenAM Administration Guide › Configuring Audit Logging for further information on configuring audit logs to be written to a database in OpenAM 13.x.

In releases prior to OpenAM 13.0, you can configure OpenAM to log to a database instead of the default log file if required. The configuration required for this is detailed below. This functionality is deprecated as of OpenAM 13.0.

#### Configuring OpenAM to log to a database

You can configure OpenAM to log to a database using either the OpenAM console or ssoadm:

• OpenAM console: navigate to: Configuration > System > Logging > Logging Type and select the DB option. Complete the following fields to configure logging to a database:
• Log File Location: enter the full directory path using a valid database URL, for example: jdbc:mysql://host:port/dbname.
• Database User Name
• Database Driver Name
• ssoadm: enter the following command:
$./ssoadm set-attr-defs -s iPlanetAMLoggingService -t global -u [adminID] -f [passwordfile] -a iplanet-am-logging-type=DB iplanet-am-logging-location=[JDBCURL] iplanet-am-logging-db-user=[DBuserName] iplanet-am-logging-db-password=[DBpassword] iplanet-am-logging-db-driver=[JDBCdriver] replacing [adminID], [passwordfile], [JDBCURL], [DBuserName], [DBpassword] and [JDBCdriver] with appropriate values. OpenAM automatically creates the necessary table in the database, so there is no need to run a script to create the table; events and errors are logged as unique database file attributes or columns and are labeled as follows: • Oracle database: • *_ACCESS • *_ERROR • MySQL database: • *_access • *_error ##### Note You should ensure the database user has the CREATE TABLE privilege and the JDBC driver is deployed in the web container. #### See Also #### Related Training N/A #### Related Issue Tracker IDs ## How do I create a HAR file for troubleshooting AM/OpenAM (All versions)? The purpose of this article is to provide information on creating a HAR file (HTTP ARchive) for troubleshooting AM/OpenAM. #### Overview A HAR file is output by browser developer tools and other browser extensions. It is essentially a recording of interactions between the browser and application in JSON format; it can be very useful for troubleshooting as it can help identify where the issue is occurring. ##### Note The equivalent file in Internet Explorer® 9, 10 and 11 is an XML file based on the HTTP Archive. #### Creating a HAR file You can create a HAR file as follows: Chrome™ browser 1. Launch the Developer Tools in Chrome and select the Network tab. 2. Enter the URL where you have been experiencing issues and re-create the issue. Do not open a new browser tab as Chrome records details on the tab initially shown when the Developer Tools were launched. 3. Right-click within the Network window and select the Save as HAR with content option. Firefox browser 1. Launch the Developer Tools in Firefox and select the Network option. 2. Select the Persist Logs option on the Network tab to ensure the entire request is saved, even if you are redirected. 3. Enter the URL where you have been experiencing issues and re-create the issue. 4. Right-click within the Network window and select the Save All As HAR option. Microsoft® Edge 1. Launch the Developer Tools in Internet Explorer and select the Network tab. 2. Click the green play button to start recording. 3. Enter the URL where you have been experiencing issues and re-create the issue. Do not open a new browser tab as Internet Explorer records details on the tab initially shown when the Developer Tools were launched. 4. Click the Save button to save the HAR file. #### Creating an XML file (Internet Explorer 9, 10 and 11) You can create an XML file based on the HTTP Archive format as follows: 1. Launch the Developer Tools in Internet Explorer and select the Network tab. 2. Click the Start Capturing button. 3. Enter the URL where you have been experiencing issues and re-create the issue. 4. Click the Save button and choose to save the file as XML. #### See Also #### Related Training N/A #### Related Issue Tracker IDs N/A ## How do I enable Garbage Collector (GC) Logging for AM/OpenAM (All versions)? The purpose of this article is to provide information on enabling GC Logging for AM/OpenAM. It assumes that you already have a working AM/OpenAM server that is installed. #### Enabling GC Logging The information given here is specific to the Apache Tomcat™ web container; you should make similar changes in the configuration file specific to your web container if you use a different one. ##### Note You should ensure there is enough disk space for the GC logs; if not, you should enable GC log rotation as well by including the following options when you enable GC logging: -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=n, -XX:GCLogFileSize=n On Unix® and Linux® systems: ​You should enable GC logging by specifying CATALINA_OPTS settings in the setenv.sh file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.sh file (also typically located in the /tomcat/bin/ directory). To enable GC logging: 1. Add the following line to the setenv.sh file: export CATALINA_OPTS="$CATALINA_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]"
replacing [filename] with the path to the file that you would like to create to store the log file.
2. Restart the web container.

Once the web container has successfully restarted, there should be a GC log file located in the directory specified in the -Xloggc: option. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

On Microsoft® Windows® systems:

​You should enable GC logging by specifying CATALINA_OPTS settings in the setenv.bat file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.bat file (also typically located in the /tomcat/bin/ directory).

To enable GC logging:

1. Add the following line to the setenv.bat file:
set CATALINA_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]
replacing [filename] with the path to the file that you would like to create to store the log file.
2. Restart the web container.

Once the web container has successfully restarted, there should be a GC log file located in the directory specified in the -Xloggc: option. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

N/A

N/A

## How do I collect JVM data for troubleshooting AM/OpenAM (All versions)?

The purpose of this article is to provide information on collecting JVM data for troubleshooting AM/OpenAM. JVM data (such as stack traces, heap histograms and heap dumps) can be very useful for troubleshooting issues such as a hung AM/OpenAM server, Out of Memory errors and suspected memory leaks. This data must be collected before killing the affected process or restarting the server; otherwise the information relating to the affected process is lost forever, which may make it impossible to identify the root cause.

#### Collecting JVM data

##### Note

The following utilities are all JDK utilities and are not included with the JRE; you must have the JDK installed in your production environments in order to collect this data. Additionally, you must be the owner of the identified process to be able to collect this data.

To collect JVM data, you must first determine the process ID of the hung web container instance using the following command:

$jps -v You can then collect the following types of data: #### Stack Traces Stack traces allow us to see what is happening with JVM threads and are sometimes referred to as thread dumps. You should collect several different stack traces (minimum of three) at roughly one minute intervals to enable us to compare them. It is recommended that you also identify the threads associated with the process ID for troubleshooting high CPU as detailed in How do I find which thread is consuming CPU in a Java process in AM/OpenAM (All versions)? You can use the following commands to collect stack traces: • You can collect long listing stack traces using the jstack command: $ jstack -F -l [pid] > [outputfile.txt]
replacing [pid] with the process ID of the hung container instance and [outputfile.txt] with the output file for the stack trace.
• You can use the kill -3 command (Unix® and Linux® systems only), which consistently produces better results:
$kill -3 [pid] replacing [pid] with the process ID of the hung container instance. The kill -3 command outputs the stack trace to the log file where the standard output is directed (typically the web container log file). The kill -3 command does not stop the running process; it will continue to run after outputting the stack trace. You can use the Java Thread Dump Analyzer to analyze your stack traces. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock. #### Heap Histograms Heap histograms enable us to visualize the objects in the JVM heap - we are only interested in live objects. You can collect heap histograms using the jmap command: $ jmap -histo:live [pid]

replacing [pid] with the process ID of the hung container instance.

##### Caution

The jmap utility must come from the same version of JVM that the container is running.

#### Heap Dumps

Heap dumps allow us to see what is happening with memory at the point the heap dump is taken, which can help us to identify high memory usage issues or memory leaks. You should collect heap dumps as close in time to the stack traces as possible to enable us to compare the contents. You can collect heap dumps using the jmap command:

$jmap -F -dump:file=[target file] [pid] replacing [target file] with the output file for the heap dump and [pid] with the process ID of the hung container instance. #### Current JVM settings You can find the current JVM settings for the server using one of the following commands (the jinfo command is the preferred way): • jinfo: $ jinfo [pid]
replacing [pid] with the required process ID.
• jps:
$jps -v For Unix® and Linux® systems where AM/OpenAM is running on an Apache Tomcat™ web container, you can also use: $ ps -ef | grep tomcat


N/A

N/A

## How do I monitor session statistics in AM/OpenAM (All versions)?

The purpose of this article is to provide information on monitoring session statistics in AM/OpenAM. This can provide useful troubleshooting information if you are experiencing unexpectedly high session numbers.

#### Overview

There are a number of ways you can monitor session statistics in AM/OpenAM depending on your version. These include:

AM 5 and later

• Session page in the console - navigate to: Realms > [Realm Name] > Sessions to access the Sessions page, which allows you to view and invalidate active CTS-based user sessions per realm.
• REST API - you can query the /json/sessions endpoint (see the Using the /json/sessions endpoint section below for further information).
• Amster - you can use Amster to query sessions (see the Using Amster section below for further information).
##### Note

Changes in AM 5 that made all servers autonomous mean it is the CTS store that holds the session data rather than the AM server itself. As such, the options listed for OpenAM 13.x and earlier cannot be used with AM 5 since they query individual servers. This is a known issue: OPENAM-9817 (JMX CTS session count is broken).

OpenAM 13.x and earlier

Session details

Session management information, including attribute values such as login time, logout time, time out limits, session creations and terminations, are logged in the amSSO.access log file (typically located in the $HOME/[am_instance]/openam/log directory). You will also see session information in the CoreSystems, Authentication and Session debug files ($HOME/[am_instance]/openam/debug directory).

You may notice a difference in session counts if you have just restarted the server; the monitoring API uses beans to store the values, for example, the session count. It is currently implemented such that each counter has an increment and a decrement method. When you log in, the counter gets incremented and when the session is destroyed, it is decremented. When AM/OpenAM starts up, all counters on the local instance are at 0. On the other hand, the counter used to count sessions for the session quotas and the Sessions tab is different; it actually counts the sessions, depending on "server mode" (single instance, multi instance or SFO). If you're using SFO, it counts all sessions from the CTS session store.

If, for example, you have just restarted AM/OpenAM, you may still have all 10 sessions in the session store (depending on how many of them have expired since). So when you start authenticating, two things happen at the same time (let's assume that you already have 10 sessions in the session store):

• User session quotas are checked and sessions are counted. This will find the existing sessions in LDAP. If the count exceeds the limit, it executes the quota action, for example, "destroy next expiring". This will result in decrementing the session counter in the monitoring bean. At this point it's at 0, so it stays at 0. The number of sessions in the CTS is now 9.
• Authentication succeeds and the session gets created. 10 sessions in CTS, SNMP counter now reads 1.

#### Using the /json/sessions endpoint (AM 5 and later)

You can query the /json/sessions endpoint to find session details. For example, to find sessions in the top level realm, you would use a call such as:

$curl -X GET -H 'Accept: application/json' 'http://host1.example.com:8080/openam/json/sessions?_queryFilter=realm%20eq%20%22%2F%22' The easiest way to find the relevant command is by using the API Explorer: 1. Access the API Explorer. You can either access it from the Help icon in the console or from the following URL: http://host1.example.com:8080/openam/XUI/#api/explorer/applications 2. Navigate to /sessions > Sessions V2.0 > sessions#2.0_query_filter. 3. Populate the query fields as required; _queryFilter is a required field. For example, to query the top level realm, enter realm eq "/". 4. Click Try it Out! This returns session details and also provides the curl command you can use in future. #### Using Amster (AM 5 and later) You can use Amster to query session details using the query Sessions command. For example: am> query Sessions --realm / --filter 'realm eq "/"' Example response: ===> [ { "username": "amadmin", "universalId": "id=amadmin,ou=user,dc=example,dc=com", "realm": "/", "sessionHandle": "shandle:4r8SsX6XJj0oAbLBmexqyUsbC7Y.*AAJTSQACMDEAAlNLABxJNEhkVlRlMnNHRzVKUTlOa1hMQ3BiRzZad0E9AAJTMQAA*", "latestAccessTime": "2018-05-01T12:36:54.487Z", "maxIdleExpirationTime": "2018-05-01T13:06:54Z", "maxSessionExpirationTime": "2018-05-01T14:31:23Z", "_rev": "746064345" }, { "username": "demo", "universalId": "id=demo,ou=user,dc=example,dc=com", "realm": "/", "sessionHandle": "shandle:rn3PS1zCIBxmY5qnMtbbqJOLgkQ.*AAJTSQACMDEAAlNLABxNR2JvL0tUenQxc2N1YnU4MkN2YjNkeGY2UTQ9AAJTMQAA*", "latestAccessTime": "2018-05-01T12:36:50.448Z", "maxIdleExpirationTime": "2018-05-01T13:06:50Z", "maxSessionExpirationTime": "2018-05-01T14:36:50Z", "_rev": "856832111" } ]  See Entity Reference › Sessions › query for further information. #### Using the amMasterSessionTableStats file (OpenAM 13.x and earlier) The amMasterSessionTableStats file contains both user and application session data, and is located in the$HOME/[openam_instance]/openam/stats directory. This file shows current and peak values for the following items:

• Maximum number of sessions in the session table, including both active and timed out sessions.
• Maximum number of active sessions.
• Number of Session Notifications in the queue.
##### Note

There is a known issue with stats logging not working in OpenAM 13.0: OPENAM-6998 (Session "stats" logging doesn't seem to work). This is fixed in OpenAM 13.5.

#### Using ssoadm (OpenAM 13.x and earlier)

The ssoadm list-sessions command lists all the current sessions on a specified server with session details, for example:

$./ssoadm list-sessions -u amadmin -f pwd.txt -t http://host1.example.com:8080/openam  Example response: [Current Session] User Id: amAdmin Time Remain: 1199999 Max Session Time: 1200000 Idle Time: 0 Max Idle Time: 300000 Index: 0 User Id: user1 Time Remain: 1199991 Max Session Time: 1200000 Idle Time: 8 Max Idle Time: 300000 Index: 1 User Id: user7 Time Remain: 1199850 Max Session Time: 1200000 Idle Time: 139 Max Idle Time: 300000 Index: 2 User Id: user12 Time Remain: 1199993 Max Session Time: 1200000 Idle Time: 6 Max Idle Time: 300000 Index: 3 User Id: user3 Time Remain: 1199862 Max Session Time: 1200000 Idle Time: 137 Max Idle Time: 300000 Index: 4 User Id: user19 Time Remain: 1199997 Max Session Time: 1200000 Idle Time: 2 Max Idle Time: 300000 Index: 5 User Id: user28 Time Remain: 1199992 Max Session Time: 1200000 Idle Time: 7 Max Idle Time: 300000 Index: 6 User Id: user17 Time Remain: 1199998 Max Session Time: 1200000 Idle Time: 1 Max Idle Time: 300000 To invalidate sessions, enter the index numbers [CR without a number to exit]: You can run the command with the -q option, which does not prompt you to invalidate sessions and can add a grep command after to just return a total, for example: $ ./ssoadm list-sessions -u amadmin -f pwd.txt -t http://host1.example.com:8080/openam -q | grep "User Id" | wc -l


Example response (which matches the number of sessions listed above):

8


#### Using SNMP monitoring (OpenAM 13.x and earlier)

First you must enable SNMP monitoring; you can do this using either the console or ssoadm:

• OpenAM 13.5 console: navigate to: Configure > Global Services > System > Monitoring and enable Monitoring SNMP interface status.
• Pre-OpenAM 13.5 console: navigate to: Configuration > System > Monitoring > Monitoring SNMP interface status and select the Enabled option.
• ssoadm: enter the following command:
$./ssoadm set-attr-defs -s iPlanetAMMonitoringService -t Global -u [adminID] -f [passwordfile] -a iplanet-am-monitoring-snmp-enabled=true  replacing [adminID] and [passwordfile] with appropriate values. By default, this allows you to listen on port 8085 for SNMP monitoring. You can change the port via the console or ssoadm (iplanet-am-monitoring-snmp-port attribute) using the same navigation / service as above. ##### Note You must restart the web application container in which OpenAM runs to apply these configuration changes. Once enabled, you can use the snmpwalk command to provide specific session details. For example, the following command (assuming you are using port 8085) gives the number of active sessions on the current OpenAM server instance: snmpwalk -c public -v 2c :8085 enterprises.42.2.230.3.1.1.2.1.11.1.0 ##### Note This command gives the same output as the SessionActiveCount JMX attribute; where both are reset to 0 when the OpenAM instance is restarted. You can check which SNMP definitions are supported in the mib/FORGEROCK-OPENAM-SESSION.mib file (enterprises 36733), which can be found in the openam.war/WEB-INF/lib/openam-mib-schema-1.x.x.jar file. See Administration Guide › Monitoring OpenAM Services › SNMP Monitoring for other session related OIDs you can use. #### See Also #### Related Training #### Related Issue Tracker IDs ## How do I clear stats logs in AM/OpenAM (All versions)? The purpose of this article is to provide information on clearing your stats logs in AM/OpenAM when they get too large and you no longer need them. It assumes you are using a UNIX® or Linux® system. Clearing stats logs as opposed to deleting them negates the need to stop and start your AM/OpenAM server. #### Clearing an individual stats log You can clear an individual stats log as follows from your terminal window: 1. Change directory to your stats directory (which is typically located in the$HOME/[am_instance]/openam/ directory).
2. Enter the following command:
$cat /dev/null >[filename] replacing [filename] with the name of the stats log file you want to clear. #### Clearing all stats logs in the stats directory You can clear all stats logs in the stats directory as follows: 1. Enter the following command in your terminal window: $ for file in /path/to/stats/*; do > $file; done For example: $ for file in $HOME/openam1/openam/stats/*; do >$file; done
to empty all the stats log files in the stats directory.

Alias

You could also create an alias to run this command to make it easy to re-use. For example, enter the following command:

$alias flushamstats='for file in$HOME/openam1/openam/stats/*; do > $file; done' and then just enter the following command to run it at any time: $ flushamstats

N/A

## How do I stop stats logging in AM/OpenAM (All versions)?

The purpose of this article is to provide information on stopping stats logging in AM/OpenAM. Stats logs are located in the $HOME/[am_instance]/openam/stats directory by default. #### Stopping stats logging You may want to consider clearing stats logs rather than stopping logging altogether as described in How do I clear stats logs in AM/OpenAM (All versions)? You can stop stats logging using either the console or ssoadm: • AM / OpenAM 13.5 console: navigate to: Configure > Server Defaults > Session > Statistics > State and select the Off option. • Pre-OpenAM 13.5 console: navigate to: Configuration > Servers and Sites > Default Server Settings > Session > Statistics > State and select the Off option. • ssoadm: enter the following command: $ ./ssoadm update-server-cfg -u [adminID] -f [passwordfile] -s default -a com.iplanet.services.stats.state=off

To re-enable stats logging, perform the same steps as above and select the File option:

• AM / OpenAM 13.5 console: navigate to: Configure > Server Defaults > Session > Statistics > State and select the File option.
• Pre-OpenAM 13.5 console: navigate to: Configuration > Servers and Sites > Default Server Settings > Session > Statistics >State and select the File option.
• ssoadm: enter the following command:
$./ssoadm update-server-cfg -u [adminID] -f [passwordfile] -s default -a com.iplanet.services.stats.state=file replacing [adminID] and [passwordfile] with appropriate values. ##### Note You must restart the web application container in which AM/OpenAM runs to apply these configuration changes. #### See Also #### Related Training N/A #### Related Issue Tracker IDs ## How do I avoid common issues with REST calls in AM/OpenAM (All versions)? The purpose of this article is to provide information on some common issues encountered with REST calls in AM/OpenAM. It also provides some tips to help you troubleshoot issues and use REST more effectively. #### Common issues The following common issues (with solutions) are seen when using REST calls: • Used the wrong session cookie name in the REST call. • Made the REST call to the load balancer URL. • Used the wrong authenticate endpoint in the REST call. Used the wrong session cookie name in the REST call If you have changed your AM/OpenAM session cookie name as recommended for securing your installation, you will need to use the new cookie name in REST calls. For example, if your new cookie is called exampleCookie, you would change the following header in your REST calls: • From: -H "iPlanetDirectoryPro: AQIC5..." • To: -H "exampleCookie: AQIC5..." If you are unsure what your session cookie name is, you can check it with the following REST call: $ curl -s 'http://host1.example.com:8080/openam/json/serverinfo/*' | tr ',' '\n' | grep cookieName
"cookieName":"iPlanetDirectoryPro"

If you make a REST call to the load balancer URL, the call can go to any AM/OpenAM instance, which may not be the intended server. You can avoid this issue as follows:

• Always use the actual AM/OpenAM server (FQDN) URL in your REST call to ensure the call goes to the correct instance (recommended).
-H "Cookie: amlbcookie=<serverid>"

Used the wrong authenticate endpoint in the REST call

If you have changed the Organization Authentication Configuration setting from ldapService to a non-DataStore module based chain, the amadmin user will not be able to authenticate using the /authenticate endpoint as they must authenticate via the DataStore module. See Authentication and Single Sign-On Guide › To Set the Default Authentication Tree or Chain for further information. This issue may also affect you if you use a DNS name to map to a different realm.

To ensure the amadmin user can authenticate in this scenario, you must append the /authenticate endpoint with ?authIndexType=service&authIndexValue=adminconsoleservice. For example:

• AM 5 and later:
http://host1.example.com:8080/openam/json/realms/root/authenticate?authIndexType=service&authIndexValue=adminconsoleservice
• Pre-AM 5:
http://host1.example.com:8080/openam/json/authenticate?authIndexType=service&authIndexValue=adminconsoleservice


#### Troubleshooting tips

When you use curl for REST calls, there are a couple of useful options you can include for troubleshooting:

• -v, --verbose - the verbose option provides useful debugging information to help you understand what is happening with your curl command.
• -k, --insecure - the insecure option is useful when you have SSL/TLS enabled (HTTPS) and want to eliminate connection issues as a result of certificates etc. By using this option, curl will connect to your server using an insecure connection. In this way, you can isolate whether you are experiencing issues with SSL connectivity or more general issues with your curl command.

See curl.1 man page for further information on all the available curl options.

Example verbose output when authenticating over REST

* Connected to host1.example.com (192.0.2.255) port 8080 (#0)
> POST /openam/json/authenticate HTTP/1.1
> User-Agent: curl/7.35.0
> Host: host1.example.com:8080
> Accept: */*
> Cache-Control: no-cache
> Content-Type: application/json
>
< HTTP/1.1 200 OK
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< X-Frame-Options: SAMEORIGIN
< Cache-Control: no-cache, no-store, must-revalidate
< Content-API-Version: resource=2.1
< Expires: 0
< Pragma: no-cache
< Content-Type: application/json;charset=UTF-8
< Content-Length: 159
< Date: Wed, 25 Apr 2018 11:20:29 GMT
<
* Connection #0 to host host1.example.com left intact
{"tokenId":"AQIC5wM2LY4SfcwGDl0FeGy0XdsRtoRSuPndpzVIxxrFUrM.*AAJTSQACMDEAAlNLABM2MTk5NjgwMzk2MTQxOTA2OTIxAAJTMQAA*","successUrl":"/openam/console","realm":"/"}


#### Usage tips

The following suggestions will help you identify REST calls and improve usability:

• API Explorer - as of AM 5, you can use the API Explorer to locate REST API endpoints as detailed in Development Guide › Introducing the API Explorer.
• Postman - you can use Postman to make REST calls, which allows you swap parameters easily and save commonly used calls. ForgeRock provides some Postman collections for its customers to use to perform tasks as detailed in How do I use the Postman collections that are provided for AM/OpenAM (All versions)? (this article is only available to customers/partners).
• jq - you can use jq on the command line in conjunction with curl commands to parse the JSON response. One of the most common uses is to prettify the response for readability.
##### Note

These are third-party tools that we suggest can be used with REST calls but are not supported by ForgeRock.

jq

The simplest jq option is to prettify a response on the command line by adding | jq . at the end of the command. Consider the following examples:

• Standard curl command:
$curl -X POST -H "X-OpenAM-Username: amadmin" -H "X-OpenAM-Password: cangetinam" -H "Content-Type: application/json" -H "Accept-API-Version: resource=2.0, protocol=1.0" http://host1.example.com:8080/openam/json/realms/root/authenticate  Example response: {"tokenId":"AQIC5wM2LY4Sfcz1vWJU7s6oqwudOhVVJN4P6P1dFbrBH_s.*AAJTSQACMDEAAlNLABM0MDk5ODM5MDk5MTEzNzUxMzg4AAJTMQAA*","successUrl":"/openam/console","realm":"/"}  • Curl command with jq pretty-print option: $ curl -X POST -H "X-OpenAM-Username: amadmin" -H "X-OpenAM-Password: cangetinam" -H "Content-Type: application/json" -H "Accept-API-Version: resource=2.0, protocol=1.0" http://host1.example.com:8080/openam/json/realms/root/authenticate | jq .

Example response:
{
"realm": "/",
"successUrl": "/openam/console",
"tokenId": "AQIC5wM2LY4Sfcz1vWJU7s6oqwudOhVVJN4P6P1dFbrBH_s.*AAJTSQACMDEAAlNLABM0MDk5ODM5MDk5MTEzNzUxMzg4AAJTMQAA*"
}


N/A

N/A

## FAQ: AM/OpenAM performance and tuning

The purpose of this FAQ is to provide answers to commonly asked questions regarding performance and tuning for AM/OpenAM and Policy Agents.

#### Q. Do I need to performance test AM/OpenAM?

A. Yes, performance testing is a very important step to ensure that the system you have configured will cope with the expected load once it is in production. By performance testing your system, you can tweak any performance issues you encounter before going live.

Before running any performance tests, you should ensure AM/OpenAM is running and configured in a test environment that closely resembles your planned production environment. You should also be aware of the type of load expected in your production system and the types of identity data involved to make your tests as realistic as possible. For example, you may want to test that your system can handle 20 authentications per second with a concurrency of 2,000 live sessions.

Generating test identity data in DS/OpenDJ is described in How do I generate sample user data for performance testing in DS/OpenDJ (All versions)?

Performing a performance test is described in idmdude - It's OK to Get Stressed Out with OpenAM

#### Q. How do I tune AM/OpenAM to improve performance?

A. Tuning AM/OpenAM is described in Setup and Maintenance Guide › Tuning an Instance.

#### Q. How can I improve the performance of ssoadm?

A. There are several approaches you can take to improving the performance of ssoadm as detailed in How do I improve the performance of ssoadm in AM/OpenAM (All versions)?

#### Q. What is the recommended Java Virtual Machines (JVM) heap size for AM/OpenAM?

A. There are no definitive rules for the size of JVM heap size required as it will vary across individual environments and applications, but you should refer to Best practice for JVM Tuning for best practice advice. There are also a number of things you should be aware of when making a decision for AM/OpenAM:

• AM/OpenAM response time is independent of heap size.
• The number of concurrent Single Sign On (SSO) sessions is dependent on heap size. As a rough guide, an AM/OpenAM server in production with a 3 GB heap can handle 100,000 sessions. As heap size increases, so does the number of possible concurrent sessions, however, smaller heap sizes are easier to manage.
• You must ensure you configure JVM garbage collection appropriately as garbage collection runs can get quite long if you have large heap sizes.
##### Note

For a 32-bit JVM or a 32-bit operating system, the limit for the process size is 4GB, that is, 2^32; this cannot be exceeded regardless of the amount of heap space allocated.

#### Q. Can I change caching in AM/OpenAM to improve performance?

A. Yes, you can improve performance in AM/OpenAM by configuring caching on the server side; you can configure caching for configuration data and global user data. This is described in Setup and Maintenance Guide › Tuning Caching. You need to balance performance against memory usage when configuring caching; larger caches lead to less requests to AM/OpenAM but increase memory usage.

#### Q. How do client-based sessions affect performance?

A. Client-based sessions were introduced in OpenAM 13 per: OpenAM 13 Release Notes › What's New in OpenAM › Elasticity and Scaling Features.

Client-based sessions use more CPU than CTS-based sessions; however CTS-based sessions have higher RAM consumption.

See How do I enable Client-based sessions in AM (All versions) and OpenAM 13.x? for further information.

#### Q. How does the Session Purge Delay setting affect performance?

A. The Session Purge Delay setting (com.iplanet.am.session.purgedelay) is set to 0 by default, which removes the session from memory immediately. If you increase this to a value above 0 (not recommended), the session is then held in memory for that number of minutes before being removed. This setting has been removed in AM 5.

If you have a particular need to increase the session purge delay, you should increase your heap size to compensate for the increase in sessions as described in How do I change the JVM heap size for AM/OpenAM (All versions)?

See Setup and Maintenance Guide › Session Settings​ for further information about these settings.

#### Q. What else might affect performance on a Linux system or a Virtual Machine?

A. Low entropy values can cause general system slowness.

##### Note

Low entropy values is a OS/Java/web container issue, which is independent of AM/OpenAM; as such, the suggested resolutions are outside the scope of ForgeRock support. If you want more tailored advice, consider engaging Professional Services.

On Linux systems you can check if a low entropy value is affecting you by running the following command:

$cat /proc/sys/kernel/random/entropy_avail A response of 200 or less is considered low and should be addressed, but you may find that even higher values can cause system slowness. Resolution You can increase the amount of entropy available in a variety of ways; two possible approaches are: • Use an external tool called RNGD (Random Number Generator Deamon). This can either use the RDRAND instruction set or /dev/urandom if RDRAND is not available, although many Virtual Machines (such as VirtualBox and OpenStack) passthrough the RDRAND instruction set to the VM. You will need to install the rng-tools package. • Set the following JVM option in the application web container in which AM/OpenAM runs: -Djava.security.egd=file:/dev/./urandom #### Q. Which Apache httpd Multi-Processing Module (MPM) should I use with web policy agent? A. It is recommended that you always use Worker MPM rather than Prefork MPM. You should ensure there are enough processes and threads available to service the expected number of client requests. You can then tune the Worker mode performance in the conf/extra/http-mpm.conf file as detailed in Web Agents User Guide › Tuning Apache Multi-Processing Modules. You can check which MPM Apache is using with the following command: $ apachectl -V

The MPM in use is shown against Server MPM.

##### Note

Turning off the Apache™ KeepAlive feature can potentially improve performance as well since the KeepAlive feature increases memory usage. You should test your performance with KeepAlive enabled and then again with it off to check what impacts it has on your setup.

#### Q. Are there things I should monitor on an ongoing basis in terms of AM/OpenAM performance?

A. Yes, it is useful to monitor performance on an ongoing basis to allow you to react to changes quickly. Useful things to monitor include:

• Heap size
• Number of open sessions and size of sessions
• CPU utilization - utilization of 70-80% is worth investigating.

See How do I monitor session statistics in AM/OpenAM (All versions)? and Setup and Maintenance Guide › Monitoring Services for further information.

## How do I change the JVM heap size for AM/OpenAM (All versions)?

#### Changing the JVM heap size

Changing the JVM heap size can improve performance and reduce the time it takes to perform authentications. You should try different heap sizes to see what impact it has to determine the best heap size for your setup. The information given here is specific to the Apache Tomcat™ web container; you should make similar changes in the configuration file specific to your web container if you use a different one.

##### Note

It is recommended that you set the minimum and maximum heap sizes to the same value for best performance. Otherwise, the JVM runs a full Garbage Collector (GC) cycle when increasing its heap, during which time it can pause ongoing operations for up to a few seconds. Generally, a smaller heap will increase the frequency at which the JVM GC executes but reduce the duration; similarly, a larger heap will reduce the frequency and increase the duration. When tuning the JVM heap, the goal is to strike a balance between frequency and duration so as to reduce the impact of the GC on the application.

On Unix® and Linux® systems:

You can set the JVM heap size by specifying CATALINA_OPTS settings in the setenv.sh file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.sh file (also typically located in the /tomcat/bin/ directory).

For example, to set the minimum and maximum heap sizes to 2GB, you would add the following line to the setenv.sh file and restart the web container:

export CATALINA_OPTS="$CATALINA_OPTS -Xms2g -Xmx2g" On Microsoft® Windows® systems: Providing you haven't installed Tomcat as a service, you can set the JVM heap size by specifying CATALINA_OPTS settings in the setenv.bat file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.bat file (also typically located in the /tomcat/bin/ directory). For example, to set the minimum and maximum heap sizes to 2GB, you would add the following line to the setenv.bat file and restart the web container: set CATALINA_OPTS=-Xms2g -Xmx2g If you have installed Tomcat as a service on Windows, then you must use the GUI application to configure Tomcat services. This process refers to Tomcat 7, but will work for other versions by changing tomcat7w.exe in the command to match your version. 1. Stop the Tomcat service. 2. Navigate to \path\to\tomcat\bin\ from the command prompt: 3. Enter the following command to display Tomcat Properties: tomcat7w.exe //ES//serviceName Where serviceName is the name of your Tomcat service. 4. Navigate to the Java tab and complete the memory pool fields as follows: Initial memory pool: 2048 Maximum memory pool: 2048 5. Restart the Tomcat service. Alternatively, Windows system administrators may prefer to configure these options in the registry so that they may be configured via group policy. The initial memory pool and maximum memory pool values can be configured in the JvmMS and JvmMX properties under the following registry key for Tomcat 7: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\Tomcat7\Parameters\Java #### See Also #### Related Training N/A #### Related Issue Tracker IDs N/A ## Best practice for JVM Tuning The purpose of this article is to provide best practice advice on JVM tuning ; including key symptoms, understanding heap space and GC logs. This best practice advice applies to AM/OpenAM, DS/OpenDJ, IDM/OpenIDM and IG/OpenIG. JVM tuning is not an exact science and will vary across individual environments and applications. Consistent load and stress testing of your environment will be the only true way to gauge the effect of changes made. #### JVM tuning considerations Before you begin JVM tuning, you should consider the following points: • Costs - depending on your environments, it may be more cost effective to add more hardware to improve performance rather than spending the time to tune. • Desired outcome - tuning for stability is more important in the long run than tuning for performance, although there is an overlap. • Ongoing problems - tuning may only delay or mask a problem temporarily if there is an underlying issue you need to resolve. • Memory leaks - these will always cause Garbage Collection (GC) issues regardless of tuning. #### What is Java® heap and how do I know it needs tuning? The Java heap is the amount of memory allocated to applications running in the JVM. Objects in heap memory can be shared between threads. Garbage collection (GC) refers to the automatic process of managing the runtime memory. The JVM is subject to GC pauses that vary in frequency and duration; if these pauses become more frequent and/or last longer, you will see issues (such as application pauses) which indicate that you may need to tune your Java heap. The heap is initially created when the JVM starts and is split into different spaces or generations, the key ones being Young (Eden or Nursery) and Tenured (Old): • The Young generation is used for new objects. A GC process (ParNew) automatically runs when it becomes full which removes unused objects and moves all objects that have lived long enough to the Tenured generation, thereby freeing up space in the Young generation for more new objects. Typically objects in the Young generation are short lived and temporary. The Young generation is small but active and GC happens frequently but with limited impact on performance. • The Tenured generation is used for the longer lived objects. Another GC process (CMS) runs when it becomes full to remove any unused objects. The Tenured generation is larger and less active than the Young generation but GC tends to have more impact on performance. The following diagram illustrates the heap and also the corresponding GC options that can be set in (Java 7 and earlier): Key symptoms that indicate you need to tune your Java heap are: • High CPU usage. When GC threads consume a lot of CPU, you will see spikes in overall CPU usage. • Application hangs. When your application appears to hang and stops responding, there are gaps in your log files or you experience general slowness when performing routine tasks, poorly tuned GC can be the cause. The first thing to check if you experience these symptoms is your GC logs; this are easy to check and may help you identify your issue quickly. GC Logging There are various different GCs available; ForgeRock recommends the use of CMS (ConcurrentMMarkSweep), which is a throughput collector and tends to give the best performance as it runs concurrently with the application, meaning less pauses. You should specify this collector by setting the following JVM option: -XX:+UseConcMarkSweepGC  You can enable GC logging as described in the following articles: ##### Note Typically you can leave GC logging on without impacting your system, however, you can just enable it when you are experiencing issues. #### Understanding GC Logs These example log files come from a CMS collector, where ParNew refers to the Young generation and CMS refers to the Tenured generation. These log file examples also show typical GC logs for a system where tuning is appropriate. ParNew section An example GC log output for the ParNew section looks like this: 24.245: [GC24.245: [ParNew: 545344K->39439K(613440K), 0.0424360 secs] 594671K->88767K(4101632K), 0.0426850 secs] [Times: user=0.17 sys=0.00, real=0.04 secs] 29.747: [GC29.747: [ParNew: 584783K->27488K(613440K), 0.0889130 secs] 634111K->104723K(4101632K), 0.0892180 secs] [Times: user=0.24 sys=0.01, real=0.09 secs] Each line in this log file represents one GC and shows the following types of information for ParNew: • Timestamp information - if you did not set the PrintGCTimeStamps and PrintGCDateStamps options, this just shows as the number of seconds from JVM startup: 24.245: [GC24.245: 29.747: [GC29.747: This information is useful as it shows the frequency of GCs. You should be aiming for GCs to occur once every 1 to 5 seconds; if they are occurring more than once a second, you need to tune your JVM and it is likely that you will be seeing high CPU usage. • Young generation information: [ParNew: 545344K->39439K(613440K), 0.0424360 secs] [ParNew: 584783K->27488K(613440K), 0.0889130 secs] This information shows the initial size of the Young space before doing the GC, the size after doing the GC, the total size available and the time it took to do the GC. If the time taken to do the GC is large, that is more than 0.1 seconds, then it is possible that your Young generation heap size is too big. The total size available will not grow if you set NewSize and MaxNewSize to the same value and this is what you want to see in your logs. • Overall heap information: 594671K->88767K(4101632K), 0.0426850 secs] 634111K->104723K(4101632K), 0.0892180 secs] This information shows initial size of the overall heap before doing the GC, the size after doing the GC and the total size available. With the second lot of figures (size after doing the GC), you can see how this incrementally grows up to the point when the GC is done and then reduces back down to a baseline figure. If you have a memory leak, you will see the baseline figure growing incrementally as it becomes increasingly full even after doing GC. • System information: [Times: user=0.17 sys=0.00, real=0.04 secs] [Times: user=0.24 sys=0.01, real=0.09 secs] This information shows how much time was spent in user space, how much time spent in kernel or system space and what was the real impact to the user. If the user time is high, it indicates there isn't enough CPU for the number of user threads. If the system time is high, it indicates your system is swapping memory to disk which means that there isn't enough physical memory for your heap size. CMS section The CMS section of the log file is different to the ParNew section as it shows multiple lines for the same GC. An example GC log output for the CMS section looks like this: 40.146: [GC [1 CMS-initial-mark: 26386K(786432K)] 26404K(1048384K), 0.0074495 secs] 40.154: [CMS-concurrent-mark-start] 40.683: [CMS-concurrent-mark: 0.521/0.529 secs] 40.683: [CMS-concurrent-preclean-start] 40.701: [CMS-concurrent-preclean: 0.017/0.018 secs] 40.704: [GC40.704: [Rescan (parallel) , 0.1790103 secs]40.883: [weak refs processing, 0.0100966 secs] [1 CMS-remark: 26386K(786432K)] 52644K(1048384K), 0.1897792 secs] 40.894: [CMS-concurrent-sweep-start] 41.020: [CMS-concurrent-sweep: 0.126/0.126 secs] 41.020: [CMS-concurrent-reset-start] 41.147: [CMS-concurrent-reset: 0.127/0.127 secs] This log shows the different phases the GC goes through. All the ones marked concurrent are happening while your application is running and therefore have little impact. The ones that don't say concurrent are stopping your application from running temporarily and therefore can have an impact. The CMS-initial-mark and CMS-remark show heap sizing details. ##### Note Typically, you will see these sections intermingled, for example, some ParNews then a CMS, some more ParNews, another CMS and so on as the GC processes automatically occur to manage the heap. #### Common issues shown in the GC logs and solutions This section identifies common issues you might find in your GC logs and how you can resolve them. You can also use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock. Young generation - too small The following log snippet shows an issue where the Young generation heap size is too small: 1.813: [GC1.813: [ParNew: 1152K­>128K(1152K), 0.0008100 secs] 16620K­->15756K(26936K), 0.0008350 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 1.816: [GC1.816: [ParNew: 1152K­>128K(1152K), 0.0006430 secs] 16780K­->15913K(26936K), 0.0006640 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 1.819: [GC1.819: [ParNew: 1152K­>128K(1152K), 0.0005370 secs] 16937K-­>16038K(26936K), 0.0005570 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] As you can see, there are multiple GCs occurring within a second. This means you need to increase the size of the Young generation (NewSize and MaxNewSize) and may also need to increase total heap size to compensate for this change (Xms and Xmx). Alternatively you could change the NewRatio option (this is set to 2 by default, which means the Tenured generation heap is twice as big as the Young generation heap or that the Young generation heap is a 1/3 of the size of the Tenured generation heap). Young generation - too large / Tenured generation - too small The following log snippet shows an issue where the Young generation heap size is too large and consequently the Tenured generation heap size is too small: 276.716: [GC (CMS Initial Mark) [1 CMS-initial-mark: 104176K(135168K)] 2758985K(6741248K), 0.8762860 secs] [Times: user=0.88 sys=0.00, real=0.88 secs] 277.592: [CMS-concurrent-mark-start] 277.657: [CMS-concurrent-mark: 0.063/0.065 secs] [Times: user=0.12 sys=0.00, real=0.06 secs] 277.657: [CMS-concurrent-preclean-start] 277.658: [CMS-concurrent-preclean: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 277.658: [CMS-concurrent-abortable-preclean-start] 277.658: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 277.658: [GC (CMS Final Remark)[YG occupancy: 2657419 K (6606080 K)]277.658: [Rescan (parallel) , 0.9815460 secs]278.639: [weak refs processing, 0.0000320 secs]278.640: [scrub string table, 0.0011700 secs] [1 CMS-remark: 104176K(135168K)] 2761595K(6741248K), 0.9828250 secs] [Times: user=7.18 sys=0.09, real=0.99 secs] 278.641: [CMS-concurrent-sweep-start] 278.668: [CMS-concurrent-sweep: 0.026/0.027 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 278.668: [CMS-concurrent-reset-start] 278.668: [CMS-concurrent-reset: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 280.673: [GC (CMS Initial Mark) [1 CMS-initial-mark: 104079K(135168K)] 2774091K(6741248K), 0.9033730 secs] [Times: user=0.90 sys=0.00, real=0.90 secs] 281.577: [CMS-concurrent-mark-start] 281.640: [CMS-concurrent-mark: 0.063/0.063 secs] [Times: user=0.13 sys=0.00, real=0.07 secs] 281.640: [CMS-concurrent-preclean-start] 281.641: [CMS-concurrent-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 281.641: [CMS-concurrent-abortable-preclean-start] 281.641: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 281.641: [GC (CMS Final Remark)[YG occupancy: 2670011 K (6606080 K)]281.641: [Rescan (parallel) , 0.9914290 secs]282.633: [weak refs processing, 0.0000110 secs]282.633: [scrub string table, 0.0008100 secs] [1 CMS-remark: 104079K(135168K)] 2774091K(6741248K), 0.9923100 secs] [Times: user=7.14 sys=0.11, real=0.99 secs] 282.634: [CMS-concurrent-sweep-start] 282.659: [CMS-concurrent-sweep: 0.024/0.025 secs] [Times: user=0.06 sys=0.01, real=0.02 secs] 282.659: [CMS-concurrent-reset-start] 282.659: [CMS-concurrent-reset: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]  As you can see, the CMS collections are occurring too frequently, without any ParNew collections in between. This is a result of the Young generation heap size being too large so that each time a ParNew collection happens, the Tenured heap size instantly becomes full and a CMS collection runs. This could happen if you increased your Young generation heap size without increasing your total heap size. This log snippet could also happen if you have memory leaks. CMS failure The following log snippet shows a CMS failure (concurrent mode failure): (concurrent mode failure): 1551948K->1548050K(1572864K), 15.1154540 secs] 1986510K->1548050K(2044736K), [CMS Perm : 78597K->78588K(524288K)], 15.4406700 secs] [Times: user=15.52 sys=0.00, real=15.44 secs] 703240.983: [GC [1 CMS-initial-mark: 1548050K(1572864K)] 1561545K(2044736K), 0.0059690 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]  This error means that the CMS collector cannot keep up with the amount of garbage being generated (the Tenured generation heap space is about 90% full) and GCs are taking longer meaning your application stops for longer periods of time. This example shows the application stopping for over 15 seconds, which will have a significant impact on performance. You can add the following GC options to try to avoid the CMS failure: • -XX:CMSInitiatingOccupancyFraction=<percentage-of-tenured> • -XX:+UseCMSInitiatingOccupancyOnly You should set -XX:CMSInitiatingOccupancyFraction to 70 or 80% (instead of default ~90%) which may enable the CMS collector to keep up with the amount of garbage being generated. Full GC The following log snippet shows a Full GC: 25.466: [Full GC )25.466: [CMS: 68756K­>75361K(1987392K), 0.2548820 secs] 96571K­->75361K(2064064K), [CMS Perm : 61069K­>60722K(61376K)], 0.2551850 secs] [Times: user=0.25 sys=0.01, real=0.25 secs] 26.185: [Full GC (System.gc())26.185: [CMS: 75361K­>73283K(1987392K), 0.2398450 secs] 76136K-­>73283K(2064064K), [CMS Perm : 60723K­>60723K(101204K)], 0.2400120 secs] [Times: user=0.24 sys=0.00, real=0.24 secs] 27.293: [Full GC (System.gc())27.293: [CMS: 73283K­>73291K(1987392K), 0.2111960 secs] 74096K­>73291K(2064064K), [CMS Perm : 60723K­>60723K(101396K)], 0.2112730 secs] [Times: user=0.21 sys=0.00, real=0.22 secs] 68.081: [Full GC (Heap Inspection Initiated GC)68.081: [CMS: 73291K­->73187K(1987392K), 0.2172670 secs] 77634K­>73187K(2064064K), [CMS Perm : 60742K­>60742K(101396K)], 0.2174520 secs] [Times: user=0.22 sys=0.00, real=0.21 secs] These log snippets refer to Full GCs occurring, meaning that the entire heap is being cleared (Young and Tenured); this often happens due to system processes such as RMI which performs a Full GC every hour on the hour. The PrintGCCause option is very useful to help you find out why the GC is happening (the last three lines in the log above show the additional information output by this option, making it clear which ones are system related). You can also set the DisableExplicitGC option to prevent system related Full GCs from happening. Permanent generation ##### Note The following information does not apply to Java 8 since the Permanent generation space has been removed. The following log snippet shows a Full GC as a result of the Permanent generation space growing (the Permanent generation space contains metadata and other data that should be permanently stored and does not typically grow): 1.241: [Full GC (Ergonomics) [PSYoungGen: 6125K­>0K(45056K)] [ParOldGen: 17659K­->20350K(20480K)] 23785K­>20350K(65536K) [PSPermGen: 13550K­->13546K(27136K)], 0.0531680 secs] [Times: user=0.25 sys=0.00, real=0.05 secs] 1.513: [Full GC (Ergonomics) [PSYoungGen: 38912K­>2783K(45056K)] [ParOldGen: 20350K­>20424K(20480K)] 59262K­>23208K(65536K) [PSPermGen: 15188K­->15188K(31744K)], 0.0992090 secs] [Times: user=0.65 sys=0.02, real=0.10 secs] These log snippets show the Permanent generation space growing (figure in brackets after PSPermGen). You can increase the MaxPermSize option to resolve this issue. #### Best practice advice on tuning Initially, you should configure your JVM as per the recommended settings for your product (if available): You can then tune your JVM as and when required. You should always follow this basic process when tuning: 1. Establish a baseline using a valid load testing process. This step is essential as it is the only way you will know if your tuning is having the desired effect. 2. Change one thing at a time and document each change. This allows you to determine the impact of each change and revert if necessary. Heap tuning guidelines Although we cannot give recommended heap settings because they vary considerably between environments and applications, you should follow these guidelines when tuning: • You should always use the latest supported software versions. • You should use the CMS GC as recommended by ForgeRock. • Your total heap size must not exceed the amount of physical RAM available. • You must leave some physical RAM for other applications to run on your machine. • You should set Xms and Xmx to the same value for best performance. These options determine your total heap size. • You should set NewSize and MaxNewSize to the same value. These options determine how the heap is allocated between generations. • You should set the DisableExplicitGC option to prevent system related full GCs from running. • You must not add lots of options at once or add options you don't understand. • You must not copy and paste options from other applications. • You should use GC logs to help identify issues with your tuning. Getting your total heap size and allocation between generations right is key; once you have these set correctly, you likely won't need to do anymore tuning. Summary In summary, the most important GC knobs to get right are: 1. The collector, which should be CMS. 2. The total heap size, where Xmx = Xms. 3. The split between generations, using NewSize or NewRatio, where NewSize = MaxNewSize. #### See Also AM/OpenAM DS/OpenDJ IDM/OpenIDM IG/OpenIG #### Related Training #### Related Issue Tracker IDs ## How do I improve the performance of ssoadm in AM/OpenAM (All versions)? The purpose of this article is to provide information on improving the performance of ssoadm in AM/OpenAM. #### Improving the performance of ssoadm There are five things you can try to improve the performance of ssoadm: Establishing a baseline You can add time before the ssoadm command to output the execution time of the ssoadm command, which will help you assess the effect of any changes you make. It is recommended you run a simple command, such as the following, prior to making any changes to give you a baseline: $ time ./ssoadm list-servers -u amadmin -f pwd.txt

You can then repeat this command after making changes to assess the impact.

#### Using the do-batch command

You can use the ssoadm do-batch command to run multiple ssoadm commands; using this command means multiple ssoadm commands are executed together in a single JVM call, rather than as individual JVM calls, which speeds up processing time.

See How do I make batch changes using ssoadm in AM/OpenAM (All versions)? for further information.

#### Increasing the ssoadm heap size

The default ssoadm heap size settings are:

-Xms256m -Xmx512m

You can increase these values (providing you have enough physical memory) to improve ssoadm performance. Initially, you should try setting both Xms and Xmx to 1024m; Xms and Xmx should be the same for the best performance.

You can increase the ssoadm heap size values as follows:

1. Edit the ssoadm or ssoadm.bat script and update the following line:
$JAVA_HOME/bin/java -Xms1024m -Xmx1024m -cp "$CLASSPATH" \
##### Note

You can update the ssoadm.template or ssoadm.bat.template file (located in the /path/to/ssoadm/tools/template/unix/bin or /path/to/ssoadm/tools/template/windows/bin directory respectively) prior to installation instead; this means ssoadm will be installed with these default values. This approach is useful if you have multiple installations or do automated installs.

#### Use a client VM

The JDK offers both a client VM and a server VM; the client VM is tuned for reduced startup times and memory footprint, but typically the server VM is used by default. See Java Virtual Machine Technology for further information.

You can force ssoadm to use the client VM as follows:

1. Edit the ssoadm or ssoadm.bat script and update the following line:
$JAVA_HOME/bin/java -Xms1024m -Xmx1024m -client -cp "$CLASSPATH" \
##### Note

You can update the ssoadm.template or ssoadm.bat.template file (located in the /path/to/ssoadm/tools/template/unix/bin or /path/to/ssoadm/tools/template/windows/bin directory respectively) prior to installation instead; this means ssoadm will be installed with these default values. This approach is useful if you have multiple installations or do automated installs.

There is a known issue in OpenAM 13.0 and 13.5: OPENAM-9685 (SSOAdmin is slow with a site configured), where ssoadm is slow in a site configuration. This issue is resolved in OpenAM 13.5.1.

There is also a known issue in OpenAM 13.5 (resolved in OpenAM 13.5.1): OPENAM-9749 (Resource leak in ssoadm's audit logging), where batch ssoadm commands fail with the following error in the ssoadm IdRepo debug log:

ERROR: IdRemoteServicesImpl.processException(): caught remote/un-known exception -
java.net.SocketException: Too many open files

You can mitigate these issues by adding the --nolog option to your ssoadm command to disable audit logging for that command, for example:

$./ssoadm list-servers -u amadmin -f pwd.txt --nolog #### Adding a Java option to change how random bits are generated In some situations, running ssoadm on Linux systems can result in performance issues; this is particularly true if you are using a virtual machine. When this happens, you will see a stack trace similar to the following when generating the SSOToken ID: "main" prio=10 tid=0x00007f806400c000 nid=0x7e3f runnable [0x00007f8068918000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:272) at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:526)
at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:139)
at sun.security.provider.SecureRandom$SeederHolder.(SecureRandom.java:186) at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:203) - locked <0x00000000ed1684f0> (a sun.security.provider.SecureRandom) at java.security.SecureRandom.nextBytes(SecureRandom.java:455) - locked <0x00000000ed168790> (a java.security.SecureRandom) at java.security.SecureRandom.next(SecureRandom.java:477) at java.util.Random.nextLong(Random.java:334) at com.sun.identity.authentication.internal.AuthContext.getSSOToken(AuthContext.java:842) at com.sun.identity.setup.Bootstrap.getSSOToken(Bootstrap.java:289) at com.sun.identity.setup.Bootstrap.getConfiguration(Bootstrap.java:203) at com.sun.identity.setup.Bootstrap.load(Bootstrap.java:135) at com.sun.identity.setup.Bootstrap.load(Bootstrap.java:92) - locked <0x00000000eb0536e0> (a java.lang.Class for com.sun.identity.setup.Bootstrap) at com.sun.identity.cli.CommandManager.main(CommandManager.java:113)  To improve the performance in this situation, you can add the following JVM option to the ssoadm script: -D"java.security.egd=file:/dev/./urandom" \ #### See Also #### Related Training N/A #### Related Issue Tracker IDs ## How do I improve OAuth 2.0 performance in OpenAM 13.0? The purpose of this article is to provide information on improving OAuth 2.0 performance in OpenAM 13.0. #### Improving OAuth 2.0 performance In OpenAM 13.0, there is a known performance issue: OPENAM-8023 (OAuth2 load: contention at com.sun.identity.sm.ServiceConfigImpl.getInstance level (perf regression compared to AM12.0.2)). This issue is resolved in OpenAM 13.5. You can mitigate this issue in OpenAM 13.0 as follows: 1. Create a completely empty file. For example, you can use the following command if you are using a Linux® or Unix® operating system: $ touch [filename]
2. Run the following ssoadm commands against all realms where you are using OAuth2:
$./ssoadm add-svc-realm -e [realmname] -s ScriptingService -u [adminID] -f [passwordfile] -D [emptyfile]$ ./ssoadm create-sub-cfg -e [realmname] -s ScriptingService -g scriptConfigurations -u [adminID] -f [passwordfile] -D [emptyfile]


N/A

N/A