How do I collect all the data required for troubleshooting AM and Agents (All versions)?
The purpose of this article is to provide assistance with collecting troubleshooting data and diagnostics for debugging AM and Agents on a variety of issues; including performance, upgrade, configuration, policies, SSL, SAML federation and WDSSO / Kerberos™ issues. You can capture the data to troubleshoot issues yourself or include it when you raise a ticket to enable us to help you more effectively.
6 readers recommend this article
Collecting Troubleshooting Data
If you are experiencing issues with AM and/or your Agents, you can collect data to help troubleshoot your issues. If you need to raise a ticket to troubleshoot an issue, you should include relevant information to help us investigate your issues more effectively. Refer to Best practice for raising an Identity Platform ticket with ForgeRock Support for further information on creating a ticket. Follow this guide for submitting technical information needed to help us resolve your issue as soon as possible.
Note
You can attach log files to tickets in the Backstage support portal. See Sending troubleshooting data to ForgeRock Support for analysis for further information.
Caution
It is very important that timestamps match across the entire set of logs and diagnostic data for all servers and components in the request flow. Failing to do this may mean capturing all the data again so timestamps correspond when the issue is seen.
The troubleshooting data needed for the following types of issues are covered in this article:
- General issues with AM and/or Agents
- High session numbers
- Hung AM server
- Out of Memory errors or suspected memory leaks
- High CPU
- Install / upgrade issues
- ssoadm issues
- Amster issues
- LDAP connection issues (connecting to a LDAP server from AM)
- SSL connection issues
- LDAPS connection issues (connecting to DS from AM)
- Policy evaluation issues
- SAML issues
- WDSSO / Kerberos issues
Finally, there are some other logs available that can be helpful as detailed in the Other logs section.
General issues with AM and/or Agents
You can use the Recording facility to capture debug logs, configuration details and thread dumps. See How do I record troubleshooting information in AM (All versions)? for further information.
- Export configuration details for AM and the Agent - it is essential to check if a configuration issue is contributing to your issues. You can export these as described in How do I export and import Service configurations for AM (All versions) using Amster or ssoadm? and How do I export configuration details for the Agent (All versions)?
- Generate a full set of debug logs and capture the HTTP trace while reproducing the issue. You should follow this process:
- Enable message level debugging as described in Debug Logging (AM 7 and later) or How do I enable Message level debugging in AM 6.x debug files? and How do I enable debug logging for troubleshooting Agents (All versions)?
- Clear the AM and Agent debug logs as described in How do I clear debug logs in AM (All versions)?
- Reproduce the issue with the REST API by running the relevant curl command. This helps us to isolate the issue. Example curl commands are available in: Getting Started with REST.
- Start a HTTP trace in your browser. You can do this by capturing a HAR file as described in How do I create a HAR file for troubleshooting AM?
- Reproduce the issue using your browser.
- Stop the HTTP trace.
- Capture HTTP container logs (Access and Error) for both AM and Agents with date and timestamps that correspond to the Agent and AM debug logs.
- Capture LDAP server logs for your data store and configuration store with date and timestamps that correspond to the Agent and AM debug logs. There are some use cases that require analysis of the LDAP server logs from AM data store (user profile) and Configuration store.
- Revert the debug level in your AM and Agent debug logs to Error to avoid filling up the disks where the debug logs are stored.
- Attach the following diagnostic logs and outputs to your support request as detailed in Sending troubleshooting data to ForgeRock Support for analysis
- AM and policy configuration exports.
- Log files from:
-
AM
- AM 7 and later: /path/to/am/var/debug and /path/to/am/var/audit
- AM 6.x: /path/to/am/debug and /path/to/am/log
- Agent - /path/to/agent/instances/agent_n/logs/debug directory where the agent is installed.
- HTTP trace - from the HTTP trace browser tool.
- HTTP container logs - the actual name and location of the server logs are specific to your web container. For example, for AM deployed on Apache Tomcat, the Tomcat logs are called localhost_access_log.YYYY-MM-DD.log and catalina.out, and are located in the /path/to/tomcat/logs/ directory by default.
- LDAP server logs - for embedded DS, these log files are located in the /path/to/am/opends/logs directory by default and for external DS, they are located in the /path/to/ds/logs directory.
-
AM
- Output from the curl command and response.
- Update your ticket with the following information:
- AM and Agent versions; some issues are version-specific so knowing your version will help us focus on relevant issues only.
- Web application container / Web server type and version.
- Detailed problem description with steps to reproduce the issue.
- Date and time at which issue was first seen.
- Details of any changes that have been made to your environment prior to the issue being seen, such as, new install, upgrade, configuration, applications, introduced load balancer, replication, increased user load, new SP or IdP and so on.
- Updated deployment diagram if applicable. Ensure to include any load balancers, proxies and firewalls.
- Timestamp(s) of when the issue was reproduced so we can correlate this with the logs.
High session numbers
- Data per General issues with AM and/or Agents.
- Capture LDAP server logs for your CTS data store with date and timestamps that correspond to the set of logs you collected per General issues with AM and/or Agents.
- Session data as described in How do I monitor session statistics in AM (All versions)?
Hung AM server
- Data per General issues with AM and/or Agents.
- Data specific to this issue as detailed in How do I diagnose a hung AM (All versions) server?
Out of Memory errors or suspected memory leaks
- Data per General issues with AM and/or Agents.
- JVM data as described in How do I collect JVM data for troubleshooting AM?
High CPU
- Data per General issues with AM and/or Agents.
- Data specific to this issue as detailed in How do I collect data for troubleshooting high CPU utilization on AM (All versions) servers?
Install / upgrade issues
Logs specific to installation or upgrade process. You can enable these as described in How do I enable message level debugging for install and upgrade issues with AM (All versions)?
ssoadm issues
Message level ssoadm logs. You can enable these as described in How do I enable message level debugging for ssoadm in AM (All versions)?
Amster issues
Debug mode in Amster. You can enable this as described in How do I enable debug mode for troubleshooting Amster (All versions)?
LDAP connection issues (connecting to a LDAP server from AM)
- Data per General issues with AM and/or Agents.
- Run the following command from the AM server to check the basic connection to the LDAP server: $ telnet [host] [port]
- Perform a simple ldapsearch using the same credentials, server details and base DN to check if you can connect that way.
- Capture a network trace using the following process:
- Stop the AM server.
- Start the network trace, for example, you could use the tcpdump command as follows: tcpdump -i any -s 0 -w /tmp/openamldap.pcap port 389
- Run the ldapsearch command you ran in the previous step.
- Start the AM server and wait for the errors to appear in the IdRepo log.
- Stop the network trace.
SSL connection issues
- Data per General issues with AM and/or Agents.
- SSL debug logs. You can enable these as described in FAQ: SSL/TLS secured connections in AM and Agents (Q. How can I debug SSL connection issues?)
LDAPS connection issues (connecting to DS from AM)
- Data per General issues with AM and/or Agents.
- Data specific to this issue as described in How do I troubleshoot connection via LDAPS issues in DS (All versions)?
Policy evaluation issues
- Data per General issues with AM and/or Agents.
- Export the policy configuration - it is essential to check if a configuration issue is contributing to your issues. You can export this as described in How do I export and import policies in AM (All versions)?
- Identify the stage at which the policy evaluation failed if possible. See Best practice for creating and testing policies in AM (All versions) for further information on identifying which stage failed.
- Reproduce the issue with the REST API - send us the curl command you used and the response. For example: $ curl -X POST -H "Content-Type: application/json" -H "Accept-API-Version: resource=1.0" -H "iPlanetDirectoryPro: AQIC5...DU3*" -d '{ "resources": [ "http://www.example.com/index.html", "http://www.example.com/do?action=run" ], "application": "iPlanetAMWebAgentService" }' https://am.example.com:8443/am/json/realms/root/policies?_action=evaluate [ { "resource" : "http://www.example.com/do?action=run", "actions" : { }, "attributes" : { }, "advices" : { "AuthLevelConditionAdvice" : [ "3" ] } }, { "resource" : "http://www.example.com/index.html", "actions" : { "POST" : false, "GET" : true }, "attributes" : { "cn" : [ "demo" ] }, "advices" : { } } ]
SAML issues
- Data per General issues with AM and/or Agents - the Federation debug log is particularly useful.
- Export the metadata (standard and extended) - it is essential to check if a configuration issue is contributing to your issues. You can export this as described in How do I export and import SAML2 metadata in AM (All versions)?
- Capture a SAML trace while reproducing the issue. You should follow this process:
- Start a SAML trace in your browser. There are free third-party tools you can use, for example, in Firefox®, you can use SAML Tracer.
- Reproduce the issue using your browser.
- Stop the SAML trace.
WDSSO / Kerberos issues
- Data per General issues with AM and/or Agents.
- Debug logs for the Krb5Login module of the JVM; this module is called by AM for WDSSO purposes. You can enable debug logging as detailed in How do I enable debug logging for troubleshooting Kerberos and WDSSO issues in AM (All versions)?
- Data specific to this issue as detailed in How do I troubleshoot Kerberos and WDSSO issues in AM (All versions)?
Other logs
There are other logs that may be called for depending on the issue you are experiencing. These are:
- Stats logs - these files (amMasterSessionTableStats, amPolicyStats, Entitlements and idRepoCacheStat; you may also see amArtifactMap and amAssertionMap) provide statistics for the associated service. They are located in the /path/to/am/var/stats directory (AM 7 and later) or the /path/to/am/stats directory (AM 6.x) by default. You can check or change stats settings by navigating to: Configure > Server Defaults > Session > Statistics. amMasterSessionTableStats is the mostly commonly used stats file, and contains both user and application session data. See How do I monitor session statistics in AM (All versions)? for further information.
-
Agent audit logs - these logs provide details about URL access attempts and whether they are successful or unsuccessful. Audit logs are logged remotely (default), locally or both: You can check or change log settings by navigating to: Realms > [Realm Name] > Applications > Agents > [Web or Java] > [Agent Name] > Global > Audit in the AM admin UI.
- Remotely - located in the /path/to/am/var/audit directory (AM 7 and later) or the /path/to/am/log directory (AM 6.x).
- Locally - located in the /path/to/agent/instances/agent_n/logs/audit directory where the agent is installed.
Note
For Apache Tomcat™, you also need to ensure the Apache user has group write permissions to the /audit directory and that the directory structure permissions are set to 666 for local audit logging to work correctly.
See Also
How do I record troubleshooting information in AM (All versions)?
Where can I find useful logs for troubleshooting ForgeRock products?
Troubleshooting (Web Agents)
Troubleshooting (Java Agents)
Related Training
ForgeRock Access Management Deep Dive (AM-410)
Related Issue Tracker IDs
N/A