Get audit and debug logs
Identity Cloud provides audit and debug logs to help you manage your tenant:
-
Use audit logs to investigate user and system behavior.
-
Use debug logs to investigate any issues that can arise in production.
Identity Cloud stores logs for 30 days. Use the /monitoring/logs
endpoint to access the stored data.
You need to get an API key and secret before you can authenticate to the endpoints. |
Sources
Identity Cloud makes browsing the logs easier by storing them in various sources.
The following knowledge base article lists the sources available and describes their purpose: What logging sources are available in Identity Cloud?
View sources
To view a list of the available sources, use the /monitoring/logs/sources
endpoint.
Example request:
$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
'https://<tenant-env-fqdn>/monitoring/logs/sources'
Example response:
{
"result": [
"am-access",
"am-activity",
"am-authentication",
"am-config",
"am-core",
"am-everything",
"idm-access",
"idm-activity",
"idm-authentication",
"idm-config",
"idm-core",
"idm-everything",
"idm-recon",
"idm-sync"
],
"resultCount": 14,
"pagedResultsCookie": null,
"totalPagedResultsPolicy": "NONE",
"totalPagedResults": 1,
"remainingPagedResults": 0
}
Identity Cloud returns the available sources in the result
array.
Retrieve log entries
To retrieve the stored log entries for a source, use the /monitoring/logs
endpoint, specifying the source as a parameter.
Example request:
$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-authentication' \
'https://<tenant-env-fqdn>/monitoring/logs'
Example response:
{
"result": [{
"payload": "<payload>",
"timestamp": "<dateTime>",
"type": "application/json",
"source": "am-authentication"
}, {
"...": "..."
}],
"resultCount": "1000",
"pagedResultsCookie": "<pagedResultsCookie>",
"totalPagedResultsPolicy": "NONE",
"totalPagedResults": -1,
"remainingPagedResults": -1
}
Identity Cloud returns the available log entries in the result
array.
Results are in JSON format or plaintext, depending on the source you request.
Use the Specify UTC times in ISO 8601 format. For example:
|
Tail logs
To tail, or get the latest entries in the stored logs for a source, use the
/monitoring/logs/tail
endpoint with the source as a parameter.
The first call to the tail endpoint returns log entries from the last 15 seconds. Subsequent calls return log entries in a range that starts from the last returned log entry in the previous result (inclusive) and ends with the latest log entry but one. If calls to the tail endpoint are not frequent enough to match the rate at which the log entries are produced, the result may not include all available log entries.
The format of the log results depends on the source or sources specified in your request. Some
sources return only JSON formatted log entries and some sources return only plaintext log entries.
Some sources, such as am-everything
, can return log entries in both formats.
Example request:
$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-everything' \
'https://<tenant-env-fqdn>/monitoring/logs/tail'
Example response:
{
"result": [{
"payload": "<payload>",
"timestamp": "<dateTime>",
"type": "<type>",
"source": "am-core"
}, {
"...": "..."
}],
"resultCount": "100",
"pagedResultsCookie": "<pagedResultsCookie>",
"totalPagedResultsPolicy": "NONE",
"totalPagedResults": -1,
"remainingPagedResults": -1
}
You can specify multiple sources in a single call. Example request:
$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-access,idm-access,idm-sync,idm-activity' \
'https://<tenant-env-fqdn>/monitoring/logs/tail'
To keep tailing, pass the Example request:
|
View logs for a specific request
All log events for an external request into Identity Cloud are assigned the same unique transaction ID.
The x-forgerock-transactionid
response header holds the transaction ID:
$ curl \
--request POST \
--include \
--header 'Content-Type: application/json' \
--header 'X-OpenAM-Username: bjensen' \
--header 'X-OpenAM-Password: Passw0rd!' \
--header 'Accept-API-Version: resource=2.0, protocol=1.0' \
'https://<tenant-env-fqdn>/am/json/realms/root/realms/alpha/authenticate'
...
x-forgerock-transactionid: <transaction-id>
...
To filter the logs for a specific transaction ID, add the transactionId
parameter to your API request; for example:
$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-authentication' \
--data 'transactionId=<transaction-id>' \
'https://<tenant-env-fqdn>/monitoring/logs'
Example response:
{
"result": [{
"payload": "<payload>",
"timestamp": "<dateTime>",
"type": "application/json",
"source": "am-authentication"
}, {
"...": "..."
}],
"resultCount": "8",
"pagedResultsCookie": null,
"totalPagedResultsPolicy": "NONE",
"totalPagedResults": -1,
"remainingPagedResults": -1
}
Filter log results
Use the _queryFilter
parameter to filter log results on any field or combination of
fields in a payload. You can add the parameter to the /monitoring/logs
and
/monitoring/logs/tail
endpoints.
The benefits of the _queryFilter
parameter are:
-
Lets you iteratively refine queries to remove extraneous results and find the specific log entries you are interested in. This is useful when searching logs to debug a production issue.
Use the /monitoring/logs
endpoint for iterative searching as the/monitoring/logs/tail
endpoint only returns results from the last 15 seconds. -
Lets you tune queries to reduce Identity Cloud log volume, making integration with external log tools such as Splunk or Elastic Stack more efficient and potentially reducing storage costs.
The _queryFilter
parameter takes a URL-encoded filter expression:
$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=<source>' \
--data-urlencode '_queryFilter=<filter-expression>' \
'https://<tenant-env-fqdn>/monitoring/logs'
To understand how to construct a filter expression, refer to the
filter expression rules for _queryFilter
.
Here are some basic examples:
Example filter expression | Description |
---|---|
|
Search plaintext results for a particular string. |
|
Search for JSON results containing a particular client IP address |
|
Search for JSON results containing a particular event name |
|
Search for JSON results with a particular timestamp |
|
Search for JSON results with a timestamp that starts with a particular datetime |
|
Search for JSON results containing a particular client IP address and also containing a particular debug level |
|
Search for JSON results where an authentication node type is present |
Filter array items in log results
To filter on array items, do not include an array index in your filter expression.
For example, to search for JSON results where the authentication node type is ScriptedDecisionNode
:
-
Wrong:
/payload/entries/0/info/nodeType eq "ScriptedDecisionNode"
-
Right:
/payload/entries/info/nodeType eq "ScriptedDecisionNode"
where a log entry for an authentication node looks like this:
{
"payload": {
"_id": "7ae37a4b-f22b-4c5e-8621-2130d5bc603c-9310858",
"component": "Authentication",
"entries": [
{
"info": {
"authLevel": "0",
"displayName": "Using Invite?",
"nodeId": "15edd2f7-22f1-4f32-bf0a-8ca3f98af850",
"nodeOutcome": "False",
"nodeType": "ScriptedDecisionNode",
"treeName": "Login"
}
}
],
"eventName": "AM-NODE-LOGIN-COMPLETED",
...
}
Filter log results between two dates
To filter log results between two dates, use the beginTime
and endTime
query parameters with
ISO 8601 datetime values:
$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=<source>' \
--data 'beginTime=<begin-datetime>' \
--data 'endTime=<end-datetime>' \
--data-urlencode '_queryFilter=<filter-expression>' \
'https://<tenant-env-fqdn>/monitoring/logs'
For example, to filter log results between two specific dates for a specific user :
$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-authentication' \
--data 'beginTime=2023-05-24T12:40:00.00Z' \
--data 'endTime=2023-05-24T12:45:00.00Z' \
--data-urlencode '_queryFilter=/payload/principal eq "user.name@example.com"' \
'https://<tenant-env-fqdn>/monitoring/logs'
Add response fields
The following knowledge base article describes how to add additional fields to an audit request: How do I extend auditing in Identity Cloud to include additional fields?
Rate limiting
Logs endpoint
To reduce unwanted stresses on the system, Identity Cloud limits the number of requests you can make
to the /monitoring/logs
endpoint in a certain timeframe:
-
The page-size limit is 1000 logs per request.
-
The request limit is 60 requests per minute.
-
The theoretical upper rate limit is therefore 60,000 logs per minute.
These limits apply per environment, so your development, staging, and production environments each have their own quota.
The following rate limit notification response headers are sent for each request to the
/monitoring/logs
endpoint:
- X-RateLimit-Limit
-
The maximum number of requests allowed in the current rate limit window.
- X-RateLimit-Remaining
-
The number of requests remaining in the current rate limit window.
- X-RateLimit-Reset
-
The time in seconds since Jan. 1, 1970, UTC when the rate limit window resets.
Logs tail endpoint
The /monitoring/logs/tail
endpoint has the same limits and response headers as the
/monitoring/logs
endpoint described above. However, the endpoint also has a limit of 20,000
lines per request, which supersedes the page-size limit of 1000 logs per request.
Because calls to the /monitoring/logs/tail
endpoint do not always fetch all logs, use this
endpoint for debugging only. Use the /monitoring/logs
endpoint when you need to fetch all logs.
Troubleshooting
Update audit configuration
Sometimes a log source is shown in the available sources in Identity Cloud but returns no results when you query the Identity Cloud logging endpoints. In this case, check the underlying IDM audit configuration to ensure that the corresponding audit topic for the source is enabled.
The following example shows how to enable the recon
event handler so that reconciliation events
appear in the audit logs:
-
Get the current the audit configuration.
Example request:
$ curl --GET \ --header 'Authorization: Bearer <access-token>' \ --header 'Content-Type: application/json' \ 'https://<tenant-env-fqdn>/openidm/config/audit' | jq
For more information, refer to IDM REST API reference.
-
Update the audit configuration as needed. The following example enables the reconciliation audit event handler.
Example update:
$ curl \ --request PUT \ --header 'Authorization: Bearer <access-token>' \ --header 'Content-Type: application/json' \ --data-raw ' { "_id": "audit", ... "eventHandlers": [ { "class": "org.forgerock.audit.handlers.json.stdout.JsonStdoutAuditEventHandler", "config": { "elasticsearchCompatible": false, "enabled": true, "name": "json", "topics": [ "access", "activity", "sync", "authentication", "config", "recon" ] } }, { "class": "org.forgerock.openidm.audit.impl.RepositoryAuditEventHandler", "config": { "enabled": false, "name": "repo", "topics": [ "access", "activity", "sync", "authentication", "config", "recon" ] } } ], ... }' \ 'https://<tenant-env-fqdn>/openidm/config/audit'
Include large log entries in filter log results
Some Identity Cloud log output is too large to be stored as a single log entry, so is stored across two log entries instead. When this happens, any log output in JSON format is stored as two plaintext log entries rather than a single JSON log entry. Consequently, any filter expression that filters on a specific JSON field will not find any of these plaintext log entries.
To work around this, you can combine a specific field filter with a plaintext filter. For example, if you were searching for log results containing a particular transaction ID using the filter expression:
/payload/transactionId co "<transaction-id>"
you could add a plaintext filter as follows:
/payload/transactionId co "<transaction-id>" or /payload co "<transaction-id>"
to include both JSON and plaintext log entries in the log results.