Get audit and debug logs
PingOne Advanced Identity Cloud provides audit and debug logs to help you manage your tenant:
-
Use audit logs to investigate user and system behavior.
-
Use debug logs to investigate any issues that can arise in production.
Advanced Identity Cloud stores logs for 30 days. Use the /monitoring/logs endpoint to access the stored data.
| You need to get an API key and secret before you can authenticate to the endpoints. |
Sources
Advanced Identity Cloud makes browsing the logs easier by storing them in various sources.
View sources
To view a list of the available sources, use the /monitoring/logs/sources endpoint.
Example request:
$ curl \
--request GET 'https://<tenant-env-fqdn>/monitoring/logs/sources' \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>'Example response:
{
"result": [
"am-access",
"am-activity",
"am-authentication",
"am-config",
"am-core",
"am-everything",
"idm-access",
"idm-activity",
"idm-authentication",
"idm-config",
"idm-core",
"idm-everything",
"idm-recon",
"idm-sync"
],
"resultCount": 14,
"pagedResultsCookie": null,
"totalPagedResultsPolicy": "NONE",
"totalPagedResults": 1,
"remainingPagedResults": 0
}Advanced Identity Cloud returns the available sources in the result array.
Source descriptions
The following table lists the available sources and describes their purpose:
| Source | Type | Description |
|---|---|---|
am-access |
Audit |
Captures all incoming Advanced Identity Cloud access calls as audit events. This includes who, what, when, and the output for every access request. Audit events:
Show exampleAccess log format
|
am-activity |
Audit |
Captures state changes to objects that were created, updated, or deleted by Advanced Identity Cloud end users. This includes session, user profile, and device profile changes. Audit events:
Show exampleActivity log format
|
am-authentication |
Audit |
Captures when and how a user authenticated and related audit events. Advanced Identity Cloud records an authentication audit event for each authentication node and the journey outcome. A node can provide extra data in the standard audit event, which is logged when an authentication node completes. Audit events:
Learn more about Authentication log format
|
am-config |
Audit |
Captures access management configuration changes for Advanced Identity Cloud with a timestamp and by whom. Configuration changes can only be performed in development environments, so these logs are empty in staging and production environments. Audit events:
Show exampleConfig log format
|
am-core |
Debug |
Captures access management debug logs for Advanced Identity Cloud. Use am-core when debugging anything in access management without capturing audit events. am-core also captures logging in authentication scripts. Development and sandbox environments provide DEBUG level logs, with logs in several areas tuned to INFO or WARNING. To reduce log volumes, staging and production environments only provide WARNING level logs and above. To troubleshoot and view the latest entries in the stored logs, you can tail am-core source. Learn more in Tail logs. |
am-everything |
Audit, Debug |
Captures all access management audit and debug logs for Advanced Identity Cloud. This includes all the logs captured in |
idm-access |
Audit |
Captures messages for the identity management REST endpoints and the invocation of scheduled tasks. This is the who, what, and output for every identity management access request in Advanced Identity Cloud. Audit events:
Show exampleLearn more about |
idm-activity |
Audit |
Captures operations on internal (managed) and external (system) objects in Advanced Identity Cloud. idm-activity logs the changes to identity content, such as adding or updating users and changing passwords. Audit events:
Show exampleLearn more about |
idm-authentication |
Audit |
Captures the results when authenticating to an If an authentication session already exists in access management, authentication to identity management is not required. In this instance, the authentication logs would appear for am-authentication, with identity management logs in idm-access and idm-activity. Audit events:
Learn more about |
idm-config |
Audit |
Captures identity management configuration changes for Advanced Identity Cloud with a timestamp and by whom. Configuration changes can only be performed in development environments, so these logs are empty in staging and production environments. Audit events:
Show exampleLearn more about |
idm-core |
Debug |
Captures identity management debug logs for Advanced Identity Cloud. Use idm-core when debugging anything in identity management without capturing audit events. Development and sandbox environments provide FINE level logs, with logs in several areas tuned to INFO, WARNING and SEVERE. To reduce log volumes, staging and production environments only provide INFO and WARNING level logs and above. To troubleshoot and view the latest entries in the stored logs, you can tail idm-core source. Learn more in Tail logs. |
idm-everything |
Audit, Debug |
Captures identity management audit and debug logs for Advanced Identity Cloud. This includes all the logs captured in |
idm-recon |
Audit |
Captures reconciliation events for Advanced Identity Cloud. The corresponding audit topic for idm-recon is disabled by default in Advanced Identity Cloud. For reconciliation events to appear in the audit logs, you must enable the recon event handler. Learn more about |
idm-sync |
Audit |
Captures any changes to an object resulting in automatic sync (live sync and implicit sync) when a repository is mapped to Advanced Identity Cloud. This includes situations and the actions taken on each object, by account. The idm-activity log contains additional details about each action. Learn more about |
Retrieve log entries
To retrieve the stored log entries for a source, use the /monitoring/logs endpoint, specifying the source as a parameter.
Example request:
$ curl -G \
--request GET 'https://<tenant-env-fqdn>/monitoring/logs' \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-authentication'Example response:
{
"result": [{
"payload": "<payload>",
"timestamp": "<dateTime>",
"type": "application/json",
"source": "am-authentication"
}, {
"...": "..."
}],
"resultCount": "1000",
"pagedResultsCookie": "<pagedResultsCookie>",
"totalPagedResultsPolicy": "NONE",
"totalPagedResults": -1,
"remainingPagedResults": -1
}Advanced Identity Cloud returns the available log entries in the result array. Results are in JSON format or
plaintext, depending on the source you request.
To reduce the size of the output, log query results are by default restricted to the last 24 hours, unless you add beginTime and/or endTime query parameters. Learn more in Get log results for a time period.
Get log results for a time period
Use the beginTime and endTime query parameters to return entries created between two ISO 8601 formatted times.
Example request:
$ curl -G \
--request GET 'https://<tenant-env-fqdn>/monitoring/logs' \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-authentication' \
--data 'beginTime=2023-03-01T12:45:00Z' \
--data 'endTime=2023-03-01T12:50:00Z'The beginTime and endTime query parameters are subject to these rules:
-
If
endTimeis not specified, it defaults to the current time. -
If
beginTimeis not specified, it defaults to 24 hours beforeendTime. -
If
beginTimeis specified, it must be 24 hours or less beforeendTime.
Tail logs
To tail, or get the latest entries in the stored logs for a source, use the
/monitoring/logs/tail endpoint with the source as a parameter.
The first call to the tail endpoint returns log entries from the last 15 seconds. Subsequent calls return log entries in a range that starts from the last returned log entry in the previous result (inclusive) and ends with the latest log entry but one. If calls to the tail endpoint are not frequent enough to match the rate at which the log entries are produced, the result may not include all available log entries.
The format of the log results depends on the source or sources specified in your request. Some
sources return only JSON formatted log entries and some sources return only plaintext log entries.
Some sources, such as am-everything, can return log entries in both formats.
Example request:
$ curl -G \
--request GET 'https://<tenant-env-fqdn>/monitoring/logs/tail' \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-everything'Example response:
{
"result": [{
"payload": "<payload>",
"timestamp": "<dateTime>",
"type": "<type>",
"source": "am-core"
}, {
"...": "..."
}],
"resultCount": "100",
"pagedResultsCookie": "<pagedResultsCookie>",
"totalPagedResultsPolicy": "NONE",
"totalPagedResults": -1,
"remainingPagedResults": -1
}You can specify multiple sources in a single call. Example request:
$ curl -G \
--request GET 'https://<tenant-env-fqdn>/monitoring/logs/tail' \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-access,idm-access,idm-sync,idm-activity'|
To keep tailing, pass the Example request: |
View logs for a specific request
All log events for an external request into Advanced Identity Cloud are assigned the same unique transaction ID.
The x-forgerock-transactionid response header holds the transaction ID:
$ curl \
--request POST 'https://<tenant-env-fqdn>/am/json/realms/root/realms/alpha/authenticate' \
--include \
--header 'Content-Type: application/json' \
--header 'X-OpenAM-Username: bjensen' \
--header 'X-OpenAM-Password: Passw0rd!' \
--header 'Accept-API-Version: resource=2.0, protocol=1.0' \
...
x-forgerock-transactionid: <transaction-id>
...To filter the logs for a specific transaction ID, add the transactionId parameter to your API request; for example:
$ curl -G \
--request GET 'https://<tenant-env-fqdn>/monitoring/logs' \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-authentication' \
--data 'transactionId=<transaction-id>'Example response:
{
"result": [{
"payload": "<payload>",
"timestamp": "<dateTime>",
"type": "application/json",
"source": "am-authentication"
}, {
"...": "..."
}],
"resultCount": "8",
"pagedResultsCookie": null,
"totalPagedResultsPolicy": "NONE",
"totalPagedResults": -1,
"remainingPagedResults": -1
}Filter log results
Use the _queryFilter parameter to filter log results on any field or combination of
fields in a payload. You can add the parameter to the /monitoring/logs and
/monitoring/logs/tail endpoints.
The benefits of the _queryFilter parameter are:
-
Lets you iteratively refine queries to remove extraneous results and find the specific log entries you are interested in. This is useful when searching logs to debug a production issue.
Use the /monitoring/logsendpoint for iterative searching as the/monitoring/logs/tailendpoint only returns results from the last 15 seconds. -
Lets you tune queries to reduce Advanced Identity Cloud log volume, making integration with external log tools such as Splunk or Elastic Stack more efficient and potentially reducing storage costs.
The _queryFilter parameter takes a URL-encoded filter expression:
$ curl -G \
--request GET 'https://<tenant-env-fqdn>/monitoring/logs' \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=<source>' \
--data-urlencode '_queryFilter=<filter-expression>'Learn more about constructing a filter expression in the
filter expression rules for _queryFilter.
Here are some basic examples:
| Example filter expression | Description |
|---|---|
|
Search plaintext results for a particular string. |
|
Search for JSON results containing a particular client IP address. |
|
Search for JSON results with a particular log level. |
|
Search for JSON results with a particular event name. |
|
Search for JSON results containing a particular event name. |
|
Search for JSON results with a particular timestamp. |
|
Search for JSON results with a timestamp that starts with a particular datetime. |
|
Search for JSON results containing a particular client IP address and also containing a particular log level. |
|
Search for JSON results where an authentication node type is present. |
Filter array items in log results
To filter on array items, do not include an array index in your filter expression.
For example, to search for JSON results where the authentication node type is ScriptedDecisionNode:
-
Wrong:
/payload/entries/0/info/nodeType eq "ScriptedDecisionNode" -
Right:
/payload/entries/info/nodeType eq "ScriptedDecisionNode"
where a log entry for an authentication node looks like this:
{
"payload": {
"_id": "7ae37a4b-f22b-4c5e-8621-2130d5bc603c-9310858",
"component": "Authentication",
"entries": [
{
"info": {
"authLevel": "0",
"displayName": "Using Invite?",
"nodeId": "15edd2f7-22f1-4f32-bf0a-8ca3f98af850",
"nodeOutcome": "False",
"nodeType": "ScriptedDecisionNode",
"treeName": "Login"
}
}
],
"eventName": "AM-NODE-LOGIN-COMPLETED",
...
}Filter log results between two dates
To filter log results between two dates, use the beginTime and endTime query parameters with
ISO 8601 datetime values:
$ curl -G \
--request GET 'https://<tenant-env-fqdn>/monitoring/logs' \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=<source>' \
--data 'beginTime=<begin-datetime>' \
--data 'endTime=<end-datetime>' \
--data-urlencode '_queryFilter=<filter-expression>'For example, to filter log results between two specific dates for a specific user :
$ curl -G \
--request GET 'https://<tenant-env-fqdn>/monitoring/logs' \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-authentication' \
--data 'beginTime=2023-05-24T12:40:00.00Z' \
--data 'endTime=2023-05-24T12:45:00.00Z' \
--data-urlencode '_queryFilter=/payload/principal eq "user.name@example.com"'Add response fields
Authentication events
You can use a script to add extra information to log entries for authentication events. Learn more in Add information to authentication audit log entries.
Identity object events
You can configure audit log results to include additional identity object events. For example, you may want to log the before and after values of specific activities, such as changes to a user’s last name or email address.
To include additional identity object event fields, add them to the includeIf property in the audit configuration. Make these changes in your development environment and then promote them.
| By default, Advanced Identity Cloud audits identity object event fields that are safe to log. When adding audit event fields, be mindful of the type of information that you intend to expose in the logs. For example, you may need to keep personally identifiable information (PII) out of the logs. |
Add identity object event fields to audit logging
-
Get the current audit configuration.
Example request:
$ curl \ --request GET \ --header 'Authorization: Bearer <access-token>' \ --header 'Content-Type: application/json' \ 'https://<tenant-env-fqdn>/openidm/config/audit'Learn more in the IDM REST API reference.
Show response
{ "_id": "audit", "auditServiceConfig": { "availableAuditEventHandlers": [ "org.forgerock.audit.handlers.csv.CsvAuditEventHandler", "org.forgerock.audit.handlers.elasticsearch.ElasticsearchAuditEventHandler", "org.forgerock.audit.handlers.jms.JmsAuditEventHandler", "org.forgerock.audit.handlers.json.JsonAuditEventHandler", "org.forgerock.audit.handlers.json.stdout.JsonStdoutAuditEventHandler", "org.forgerock.openidm.audit.impl.RouterAuditEventHandler", "org.forgerock.audit.handlers.splunk.SplunkAuditEventHandler", "org.forgerock.audit.handlers.syslog.SyslogAuditEventHandler" ], "caseInsensitiveFields": [ "/access/http/request/headers", "/access/http/response/headers" ], "filterPolicies": { "value": { "excludeIf": [ "/access/http/request/cookies/&{com.iplanet.am.cookie.name}", "/access/http/request/cookies/session-jwt", "/access/http/request/headers/&{com.sun.identity.auth.cookieName}", "/access/http/request/headers/&{com.iplanet.am.cookie.name}", "/access/http/request/headers/accept-encoding", "/access/http/request/headers/accept-language", "/access/http/request/headers/Authorization", "/access/http/request/headers/cache-control", "/access/http/request/headers/connection", "/access/http/request/headers/content-length", "/access/http/request/headers/content-type", "/access/http/request/headers/proxy-authorization", "/access/http/request/headers/X-OpenAM-Password", "/access/http/request/headers/X-OpenIDM-Password", "/access/http/request/queryParameters/access_token", "/access/http/request/queryParameters/IDToken1", "/access/http/request/queryParameters/id_token_hint", "/access/http/request/queryParameters/Login.Token1", "/access/http/request/queryParameters/redirect_uri", "/access/http/request/queryParameters/requester", "/access/http/request/queryParameters/sessionUpgradeSSOTokenId", "/access/http/request/queryParameters/tokenId", "/access/http/response/headers/Authorization", "/access/http/response/headers/Set-Cookie", "/access/http/response/headers/X-OpenIDM-Password" ], "includeIf": [] } }, "handlerForQueries": "json" }, "eventHandlers": [ { "class": "org.forgerock.audit.handlers.json.stdout.JsonStdoutAuditEventHandler", "config": { "name": "json", "topics": [ "access", "activity", "sync", "authentication", "config" ] } } ], "eventTopics": { "activity": { "filter": { "actions": [ "create", "update", "delete", "patch", "action" ] }, "passwordFields": [ "password" ], "watchedFields": [] }, "config": { "filter": { "actions": [ "create", "update", "delete", "patch", "action" ] } } }, "exceptionFormatter": { "file": "bin/defaults/script/audit/stacktraceFormatter.js", "type": "text/javascript" } } -
Make a backup of the audit configuration.
-
Update the
includeIfproperty (underfilterPolicies) in the audit configuration to include the fields you want to add.The following example updates the audit configuration to include before and after values of a user’s last name and email address:
$ curl \ --request PUT \ --header 'Authorization: Bearer <access-token>' \ --header 'Content-Type: application/json' \ --data-raw ' { "_id": "audit", ... "filterPolicies": { "value": { "excludeIf": [ "/access/http/request/cookies/&{com.iplanet.am.cookie.name}", "/access/http/request/cookies/session-jwt", ... ], "includeIf": [ "/activity/before/sn", (1) "/activity/after/sn", (2) "/activity/before/mail", (3) "/activity/after/mail" (4) ] } }, ... }' \ 'https://<tenant-env-fqdn>/openidm/config/audit'Fields added:
1 Logs the user’s last name before change. 2 Logs the user’s last name after change. 3 Logs the user’s email address before change. 4 Logs user’s email address after change.
Once updated, idm-activity and idm-everything audit logs will include the fields you have added.
For example, the following entry in a sample idm-activity log shows before and after values for changes to a user’s last name and email address from "Brown" to "Granger":
{
"payload": {
"message": "",
"runAs": "bd220328-9762-458b-b05a-982ac3c7fc54",
"transactionId": "1630683558570-abec9e9304c84ad368ba-28676/0",
"before": {
"sn": "Brown",
"mail": "jbrown@example.com"
},
"operation": "PATCH",
"passwordChanged": false,
"_id": "52f7cea0-285d-4ef6-bda3-83256dda71c5-1300250",
"revision": "00000000412cae36",
"eventName": "activity",
"userId": "bd220328-9762-458b-b05a-982ac3c7fc54",
"status": "SUCCESS",
"objectId": "managed/alpha_user/ce7492dc-8759-47b3-b4ee-eda8d4de4ab1",
"timestamp": "2023-09-03T15:39:42.862Z",
"changedFields": [],
"after": {
"sn": "Granger",
"mail": "jgranger@example.com"
}
"type": "application/json",
"timestamp": "<date-time>"
}|
You can also exclude fields from audit logging by adding them to the For example, to prevent audit logs from showing target object attributes for synchronization and reconciliation events, add the following entries to the |
Rate limiting
Logs endpoint
To reduce unwanted stresses on the system, Advanced Identity Cloud limits the number of requests you can make
to the /monitoring/logs endpoint in a certain timeframe:
-
The page-size limit is 1000 logs per request.
Ping Identity recommends you do not override the page-size limit with a greater value as it could increase request throttling and reduce the overall number of logs you can request per minute. -
The request limit is 60 requests per minute.
-
The theoretical upper rate limit is therefore 60,000 logs per minute.
These limits apply per environment, so your development, staging, and production environments each have their own quota.
The following rate limit notification response headers are sent for each request to the
/monitoring/logs endpoint:
- X-RateLimit-Limit
-
The maximum number of requests allowed in the current rate limit window.
- X-RateLimit-Remaining
-
The number of requests remaining in the current rate limit window.
- X-RateLimit-Reset
-
The time in seconds since Jan. 1, 1970, UTC when the rate limit window resets.
A 429 HTTP status code on the /monitoring/logs endpoint indicates that the rate limit has been exceeded, and no results are returned.
Logs tail endpoint
The /monitoring/logs/tail endpoint has the same limits and response headers as the
/monitoring/logs endpoint described above. However, the endpoint also has a limit of 15,000
lines per request, which supersedes the page-size limit of 1000 logs per request.
Because calls to the /monitoring/logs/tail endpoint do not always fetch all logs, use this
endpoint for debugging only. Use the /monitoring/logs endpoint when you need to fetch all logs.
Troubleshooting
Update audit configuration
Sometimes a log source is shown in the available sources in Advanced Identity Cloud but returns no results when you query the Advanced Identity Cloud logging endpoints. In this case, check the underlying IDM audit configuration to ensure that the corresponding audit topic for the source is enabled.
The following example shows how to enable the recon event handler so that reconciliation events
appear in the audit logs:
-
Get the current audit configuration.
Example request:
$ curl \ --request GET \ --header 'Authorization: Bearer <access-token>' \ --header 'Content-Type: application/json' \ 'https://<tenant-env-fqdn>/openidm/config/audit'Learn more in IDM REST API reference.
-
Update the audit configuration as needed. The following example enables the reconciliation audit event handler.
Example update:
$ curl \ --request PUT 'https://<tenant-env-fqdn>/openidm/config/audit' \ --header 'Authorization: Bearer <access-token>' \ --header 'Content-Type: application/json' \ --data-raw ' { "_id": "audit", ... "eventHandlers": [ { "class": "org.forgerock.audit.handlers.json.stdout.JsonStdoutAuditEventHandler", "config": { "elasticsearchCompatible": false, "enabled": true, "name": "json", "topics": [ "access", "activity", "sync", "authentication", "config", "recon" ] } } ], ... }' \ 'https://<tenant-env-fqdn>/openidm/config/audit'
Include large log entries in filter log results
Some Advanced Identity Cloud log output is too large to be stored as a single log entry, so is stored across two log entries instead. When this happens, any log output in JSON format is stored as two plaintext log entries rather than a single JSON log entry. Consequently, any filter expression that filters on a specific JSON field will not find any of these plaintext log entries.
To work around this, you can combine a specific field filter with a plaintext filter. For example, if you were searching for log results containing a particular transaction ID using the filter expression:
/payload/transactionId co "<transaction-id>"
you could add a plaintext filter as follows:
/payload/transactionId co "<transaction-id>" or /payload co "<transaction-id>"
to include both JSON and plaintext log entries in the log results.
Import cURL commands into Postman
If you use Postman’s import cURL commands feature to convert cURL
commands into Postman requests, you might have problems with the example commands on this page that
use a GET request and include a --data parameter, such as this one:
$ curl -G \
--request GET 'https://<tenant-env-fqdn>/monitoring/logs'
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-authentication'When you import the above command into Postman and run the request, it produces this error:
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>400 Bad Request</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Bad Request</h1>
<h2>Your client has issued a malformed or illegal request.</h2>
<h2></h2>
</body></html>This occurs because Postman adds the value of --data parameter twice—as a query string but also
incorrectly as a message body. The equivalent incorrect cURL query of the import would be:
curl \
--request GET 'https://<tenant-env-fqdn>/monitoring/logs?source=am-authentication' \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'source=am-authentication'To resolve, convert any --data parameters into a query string before importing into Postman:
$ curl \
--request 'https://<tenant-env-fqdn>/monitoring/logs?source=am-authentication'
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>'