How To
ForgeRock Identity Platform
Does not apply to Identity Cloud

How do I collect all the data required for troubleshooting AM and Agents (All versions)?

Last updated Jan 11, 2023

The purpose of this article is to provide assistance with collecting troubleshooting data and diagnostics for debugging AM and Agents on a variety of issues; including performance, upgrade, configuration, policies, SSL, SAML federation and WDSSO / Kerberos™ issues. You can capture the data to troubleshoot issues yourself or include it when you raise a ticket to enable us to help you more effectively.

6 readers recommend this article

Collecting Troubleshooting Data

If you are experiencing issues with AM and/or your Agents, you can collect data to help troubleshoot your issues. If you need to raise a ticket to troubleshoot an issue, you should include relevant information to help us investigate your issues more effectively. Refer to Best practice for raising an Identity Platform ticket with ForgeRock Support for further information on creating a ticket. Follow this guide for submitting technical information needed to help us resolve your issue as soon as possible.

Note

You can attach log files to tickets in the Backstage support portal. See Sending troubleshooting data to ForgeRock Support for analysis for further information.

Caution

It is very important that timestamps match across the entire set of logs and diagnostic data for all servers and components in the request flow. Failing to do this may mean capturing all the data again so timestamps correspond when the issue is seen.

The troubleshooting data needed for the following types of issues are covered in this article:

Finally, there are some other logs available that can be helpful as detailed in the Other logs section.

General issues with AM and/or Agents

You can use the Recording facility to capture debug logs, configuration details and thread dumps. See How do I record troubleshooting information in AM (All versions)? for further information.

  1. Export configuration details for AM and the Agent - it is essential to check if a configuration issue is contributing to your issues. You can export these as described in How do I export and import Service configurations for AM (All versions) using Amster or ssoadm? and How do I export configuration details for the Agent (All versions)?
  2. Generate a full set of debug logs and capture the HTTP trace while reproducing the issue. You should follow this process:
    1. Enable message level debugging as described in Debug Logging (AM 7 and later) or How do I enable Message level debugging in AM 6.x debug files? and How do I enable debug logging for troubleshooting Agents (All versions)?
    2. Clear the AM and Agent debug logs as described in How do I clear debug logs in AM (All versions)?
    3. Reproduce the issue with the REST API by running the relevant curl command. This helps us to isolate the issue. Example curl commands are available in: Getting Started with REST.
    4. Start a HTTP trace in your browser. You can do this by capturing a HAR file as described in How do I create a HAR file for troubleshooting AM?
    5. Reproduce the issue using your browser.
    6. Stop the HTTP trace.
  3. Capture HTTP container logs (Access and Error) for both AM and Agents with date and timestamps that correspond to the Agent and AM debug logs.
  4. Capture LDAP server logs for your data store and configuration store with date and timestamps that correspond to the Agent and AM debug logs. There are some use cases that require analysis of the LDAP server logs from AM data store (user profile) and Configuration store.
  5. Revert the debug level in your AM and Agent debug logs to Error to avoid filling up the disks where the debug logs are stored.
  6. Attach the following diagnostic logs and outputs to your support request as detailed in Sending troubleshooting data to ForgeRock Support for analysis
    • AM and policy configuration exports.
    • Log files from:
      • AM
        • AM 7 and later: /path/to/am/var/debug and /path/to/am/var/audit
        • AM 6.x: /path/to/am/debug and /path/to/am/log
      • Agent - /path/to/agent/instances/agent_n/logs/debug directory where the agent is installed.
      • HTTP trace - from the HTTP trace browser tool.
      • HTTP container logs - the actual name and location of the server logs are specific to your web container. For example, for AM deployed on Apache Tomcat, the Tomcat logs are called localhost_access_log.YYYY-MM-DD.log and catalina.out, and are located in the /path/to/tomcat/logs/ directory by default.
      • LDAP server logs - for embedded DS, these log files are located in the /path/to/am/opends/logs directory by default and for external DS, they are located in the /path/to/ds/logs directory.
    • Output from the curl command and response.
  7. Update your ticket with the following information:
    • AM and Agent versions; some issues are version-specific so knowing your version will help us focus on relevant issues only.
    • Web application container / Web server type and version.
    • Detailed problem description with steps to reproduce the issue.
    • Date and time at which issue was first seen.
    • Details of any changes that have been made to your environment prior to the issue being seen, such as, new install, upgrade, configuration, applications, introduced load balancer, replication, increased user load, new SP or IdP and so on.
    • Updated deployment diagram if applicable. Ensure to include any load balancers, proxies and firewalls.
    • Timestamp(s) of when the issue was reproduced so we can correlate this with the logs.

High session numbers

 

Hung AM server

 

Out of Memory errors or suspected memory leaks

 

High CPU

 

Install / upgrade issues

Logs specific to installation or upgrade process. You can enable these as described in How do I enable message level debugging for install and upgrade issues with AM (All versions)?

ssoadm issues

Message level ssoadm logs. You can enable these as described in How do I enable message level debugging for ssoadm in AM (All versions)?

Amster issues

Debug mode in Amster. You can enable this as described in How do I enable debug mode for troubleshooting Amster (All versions)?

LDAP connection issues (connecting to a LDAP server from AM)

 

  • Data per General issues with AM and/or Agents.
  • Run the following command from the AM server to check the basic connection to the LDAP server: $ telnet [host] [port]
  • Perform a simple ldapsearch using the same credentials, server details and base DN to check if you can connect that way.
  • Capture a network trace using the following process:
    1. Stop the AM server.
    2. Start the network trace, for example, you could use the tcpdump command as follows: tcpdump -i any -s 0 -w /tmp/openamldap.pcap port 389
    3. Run the ldapsearch command you ran in the previous step.
    4. Start the AM server and wait for the errors to appear in the IdRepo log.
    5. Stop the network trace.

SSL connection issues

 

LDAPS connection issues (connecting to DS from AM)

 

Policy evaluation issues

 

  • Data per General issues with AM and/or Agents.
  • Export the policy configuration - it is essential to check if a configuration issue is contributing to your issues. You can export this as described in How do I export and import policies in AM (All versions)?
  • Identify the stage at which the policy evaluation failed if possible. See Best practice for creating and testing policies in AM (All versions) for further information on identifying which stage failed.
  • Reproduce the issue with the REST API - send us the curl command you used and the response. For example: $ curl -X POST -H "Content-Type: application/json" -H "Accept-API-Version: resource=1.0" -H "iPlanetDirectoryPro: AQIC5...DU3*" -d '{    "resources": [         "http://www.example.com/index.html",         "http://www.example.com/do?action=run"     ],     "application": "iPlanetAMWebAgentService"  }' https://am.example.com:8443/am/json/realms/root/policies?_action=evaluate [ {   "resource" : "http://www.example.com/do?action=run",   "actions" : {   },   "attributes" : {   },   "advices" : {     "AuthLevelConditionAdvice" : [ "3" ]   } }, {   "resource" : "http://www.example.com/index.html",   "actions" : {     "POST" : false,     "GET" : true   },   "attributes" : {     "cn" : [ "demo" ]   },   "advices" : {   } } ]

SAML issues

 

  • Data per General issues with AM and/or Agents - the Federation debug log is particularly useful.
  • Export the metadata (standard and extended) - it is essential to check if a configuration issue is contributing to your issues. You can export this as described in How do I export and import SAML2 metadata in AM (All versions)?
  • Capture a SAML trace while reproducing the issue. You should follow this process:
    • Start a SAML trace in your browser. There are free third-party tools you can use, for example, in Firefox®, you can use SAML Tracer.
    • Reproduce the issue using your browser.
    • Stop the SAML trace.

If you cannot capture a SAML trace, a HTTP trace will suffice. 

WDSSO / Kerberos issues

 

Other logs

There are other logs that may be called for depending on the issue you are experiencing. These are:

  • Stats logs - these files (amMasterSessionTableStats, amPolicyStats, Entitlements and idRepoCacheStat; you may also see amArtifactMap and amAssertionMap) provide statistics for the associated service. They are located in the /path/to/am/var/stats directory (AM 7 and later) or the /path/to/am/stats directory (AM 6.x) by default. You can check or change stats settings by navigating to: Configure > Server Defaults > Session > Statistics. amMasterSessionTableStats is the mostly commonly used stats file, and contains both user and application session data. See How do I monitor session statistics in AM (All versions)? for further information.
  • Agent audit logs - these logs provide details about URL access attempts and whether they are successful or unsuccessful. Audit logs are logged remotely (default), locally or both: You can check or change log settings by navigating to: Realms > [Realm Name] > Applications > Agents > [Web or Java] > [Agent Name] > Global > Audit in the AM admin UI.
    • Remotely - located in the /path/to/am/var/audit directory (AM 7 and later) or the /path/to/am/log directory (AM 6.x).
    • Locally - located in the /path/to/agent/instances/agent_n/logs/audit directory where the agent is installed.
Note

For Apache Tomcat™, you also need to ensure the Apache user has group write permissions to the /audit directory and that the directory structure permissions are set to 666 for local audit logging to work correctly.

See Also

Troubleshooting AM and Agents

How do I record troubleshooting information in AM (All versions)?

Where can I find useful logs for troubleshooting ForgeRock products?

Audit logging

Monitor AM instances

Troubleshooting (Web Agents)

Troubleshooting (Java Agents)

Log files and messages

Related Training

ForgeRock Access Management Deep Dive (AM-410)

Related Issue Tracker IDs

N/A
Copyright and Trademarks Copyright © 2023 ForgeRock, all rights reserved.