Identity Cloud

Get audit and debug logs

Identity Cloud provides audit and debug logs to help you manage your tenant:

  • Use audit logs to investigate user and system behavior.

  • Use debug logs to investigate any issues that can arise in production.

Identity Cloud stores logs for 30 days. Use the /monitoring/logs endpoint to access the stored data.

You need to get an API key and secret before you can authenticate to the endpoints.

Sources

Identity Cloud makes browsing the logs easier by storing them in various sources.

The following knowledge base article lists the sources available and describes their purpose: What logging sources are available in Identity Cloud?

View sources

To view a list of the available sources, use the /monitoring/logs/sources endpoint.

Example request:

$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
'https://<tenant-env-fqdn>/monitoring/logs/sources'

Example response:

{
  "result": [
    "am-access",
    "am-activity",
    "am-authentication",
    "am-config",
    "am-core",
    "am-everything",
    "idm-access",
    "idm-activity",
    "idm-authentication",
    "idm-config",
    "idm-core",
    "idm-everything",
    "idm-recon",
    "idm-sync"
  ],
  "resultCount": 14,
  "pagedResultsCookie": null,
  "totalPagedResultsPolicy": "NONE",
  "totalPagedResults": 1,
  "remainingPagedResults": 0
}

Identity Cloud returns the available sources in the result array.

Retrieve log entries

To retrieve the stored log entries for a source, use the /monitoring/logs endpoint, specifying the source as a parameter.

Example request:

$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-authentication' \
'https://<tenant-env-fqdn>/monitoring/logs'

Example response:

{
  "result": [{
    "payload": "<payload>",
    "timestamp": "<dateTime>",
    "type": "application/json",
    "source": "am-authentication"
  }, {
    "...": "..."
  }],
  "resultCount": "1000",
  "pagedResultsCookie": "<pagedResultsCookie>",
  "totalPagedResultsPolicy": "NONE",
  "totalPagedResults": -1,
  "remainingPagedResults": -1
}

Identity Cloud returns the available log entries in the result array.

Results are in JSON format or plaintext, depending on the source you request.

Use the beginTime and endTime query parameters to return records created between the two times.

Specify UTC times in ISO 8601 format.

For example:

$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-authentication' \
--data 'beginTime=2023-03-01T12:45:00Z' \
--data 'endTime=2023-03-01T12:50:00Z' \
'https://<tenant-env-fqdn>/monitoring/logs'

Tail logs

To tail, or get the latest entries in the stored logs for a source, use the /monitoring/logs/tail endpoint with the source as a parameter.

The first call to the tail endpoint returns log entries from the last 15 seconds. Subsequent calls return log entries in a range that starts from the last returned log entry in the previous result (inclusive) and ends with the latest log entry but one. If calls to the tail endpoint are not frequent enough to match the rate at which the log entries are produced, the result may not include all available log entries.

The format of the log results depends on the source or sources specified in your request. Some sources return only JSON formatted log entries and some sources return only plaintext log entries. Some sources, such as am-everything, can return log entries in both formats.

Example request:

$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-everything' \
'https://<tenant-env-fqdn>/monitoring/logs/tail'

Example response:

{
  "result": [{
    "payload": "<payload>",
    "timestamp": "<dateTime>",
    "type": "<type>",
    "source": "am-core"
  }, {
    "...": "..."
  }],
  "resultCount": "100",
  "pagedResultsCookie": "<pagedResultsCookie>",
  "totalPagedResultsPolicy": "NONE",
  "totalPagedResults": -1,
  "remainingPagedResults": -1
}

You can specify multiple sources in a single call. Example request:

$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-access,idm-access,idm-sync,idm-activity' \
'https://<tenant-env-fqdn>/monitoring/logs/tail'

To keep tailing, pass the <pagedResultsCookie> from the previous response in your request. This retrieves new records since your request.

Example request:

$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-access,idm-access,idm-sync,idm-activity' \
--data '_pagedResultsCookie=<pagedResultsCookie>' \
'https://<tenant-env-fqdn>/monitoring/logs/tail'

View logs for a specific request

All log events for an external request into Identity Cloud are assigned the same unique transaction ID. The x-forgerock-transactionid response header holds the transaction ID:

$ curl \
--request POST \
--include \
--header 'Content-Type: application/json' \
--header 'X-OpenAM-Username: bjensen' \
--header 'X-OpenAM-Password: Passw0rd!' \
--header 'Accept-API-Version: resource=2.0, protocol=1.0' \
'https://<tenant-env-fqdn>/am/json/realms/root/realms/alpha/authenticate'
...
x-forgerock-transactionid: <transaction-id>
...

To filter the logs for a specific transaction ID, add the transactionId parameter to your API request; for example:

$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-authentication' \
--data 'transactionId=<transaction-id>' \
'https://<tenant-env-fqdn>/monitoring/logs'

Example response:

{
  "result": [{
    "payload": "<payload>",
    "timestamp": "<dateTime>",
    "type": "application/json",
    "source": "am-authentication"
  }, {
    "...": "..."
  }],
  "resultCount": "8",
  "pagedResultsCookie": null,
  "totalPagedResultsPolicy": "NONE",
  "totalPagedResults": -1,
  "remainingPagedResults": -1
}

Filter log results

Use the _queryFilter parameter to filter log results on any field or combination of fields in a payload. You can add the parameter to the /monitoring/logs and /monitoring/logs/tail endpoints.

The benefits of the _queryFilter parameter are:

  • Lets you iteratively refine queries to remove extraneous results and find the specific log entries you are interested in. This is useful when searching logs to debug a production issue.

    Use the /monitoring/logs endpoint for iterative searching as the /monitoring/logs/tail endpoint only returns results from the last 15 seconds.

  • Lets you tune queries to reduce Identity Cloud log volume, making integration with external log tools such as Splunk or Elastic Stack more efficient and potentially reducing storage costs.

The _queryFilter parameter takes a URL-encoded filter expression:

$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=<source>' \
--data-urlencode '_queryFilter=<filter-expression>' \
'https://<tenant-env-fqdn>/monitoring/logs'

To understand how to construct a filter expression, refer to the filter expression rules for _queryFilter. Here are some basic examples:

Example filter expression Description

/payload co "WARNING"

Search plaintext results for a particular string.

/payload/client/ip co "10.104.1.5"

Search for JSON results containing a particular client IP address

/payload/eventName co "AM-ACCESS-ATTEMPT"

Search for JSON results containing a particular event name

/payload/timestamp eq "2023-05-16T08:21:34.632Z"

Search for JSON results with a particular timestamp

/payload/timestamp sw "2023-05-14T16:34:34"

Search for JSON results with a timestamp that starts with a particular datetime

/payload/client/ip co "10.104.1.5" and /payload/level eq "ERROR"

Search for JSON results containing a particular client IP address and also containing a particular debug level

/payload/entries/info/nodeType pr

Search for JSON results where an authentication node type is present

Filter array items in log results

To filter on array items, do not include an array index in your filter expression.

For example, to search for JSON results where the authentication node type is ScriptedDecisionNode:

  • Wrong: /payload/entries/0/info/nodeType eq "ScriptedDecisionNode"

  • Right: /payload/entries/info/nodeType eq "ScriptedDecisionNode"

where a log entry for an authentication node looks like this:

    {
      "payload": {
        "_id": "7ae37a4b-f22b-4c5e-8621-2130d5bc603c-9310858",
        "component": "Authentication",
        "entries": [
          {
            "info": {
              "authLevel": "0",
              "displayName": "Using Invite?",
              "nodeId": "15edd2f7-22f1-4f32-bf0a-8ca3f98af850",
              "nodeOutcome": "False",
              "nodeType": "ScriptedDecisionNode",
              "treeName": "Login"
            }
          }
        ],
        "eventName": "AM-NODE-LOGIN-COMPLETED",
        ...
    }

Filter log results between two dates

To filter log results between two dates, use the beginTime and endTime query parameters with ISO 8601 datetime values:

$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=<source>' \
--data 'beginTime=<begin-datetime>' \
--data 'endTime=<end-datetime>' \
--data-urlencode '_queryFilter=<filter-expression>' \
'https://<tenant-env-fqdn>/monitoring/logs'

For example, to filter log results between two specific dates for a specific user :

$ curl --get \
--header 'x-api-key: <api-key>' \
--header 'x-api-secret: <api-secret>' \
--data 'source=am-authentication' \
--data 'beginTime=2023-05-24T12:40:00.00Z' \
--data 'endTime=2023-05-24T12:45:00.00Z' \
--data-urlencode '_queryFilter=/payload/principal eq "user.name@example.com"' \
'https://<tenant-env-fqdn>/monitoring/logs'

Add response fields

The following knowledge base article describes how to add additional fields to an audit request: How do I extend auditing in Identity Cloud to include additional fields?

Rate limiting

Logs endpoint

To reduce unwanted stresses on the system, Identity Cloud limits the number of requests you can make to the /monitoring/logs endpoint in a certain timeframe:

  • The page-size limit is 1000 logs per request.

  • The request limit is 60 requests per minute.

  • The theoretical upper rate limit is therefore 60,000 logs per minute.

These limits apply per environment, so your development, staging, and production environments each have their own quota.

The following rate limit notification response headers are sent for each request to the /monitoring/logs endpoint:

X-RateLimit-Limit

The maximum number of requests allowed in the current rate limit window.

X-RateLimit-Remaining

The number of requests remaining in the current rate limit window.

X-RateLimit-Reset

The time in seconds since Jan. 1, 1970, UTC when the rate limit window resets.

Logs tail endpoint

The /monitoring/logs/tail endpoint has the same limits and response headers as the /monitoring/logs endpoint described above. However, the endpoint also has a limit of 20,000 lines per request, which supersedes the page-size limit of 1000 logs per request.

Because calls to the /monitoring/logs/tail endpoint do not always fetch all logs, use this endpoint for debugging only. Use the /monitoring/logs endpoint when you need to fetch all logs.

Troubleshooting

Update audit configuration

Sometimes a log source is shown in the available sources in Identity Cloud but returns no results when you query the Identity Cloud logging endpoints. In this case, check the underlying IDM audit configuration to ensure that the corresponding audit topic for the source is enabled.

The following example shows how to enable the recon event handler so that reconciliation events appear in the audit logs:

  1. Get the current the audit configuration.

    Example request:

    $ curl --GET \
--header 'Authorization: Bearer <access-token>' \
--header 'Content-Type: application/json' \
'https://<tenant-env-fqdn>/openidm/config/audit' | jq

    For more information, refer to IDM REST API reference.

  2. Update the audit configuration as needed. The following example enables the reconciliation audit event handler.

    Example update:

    $ curl \
--request PUT \
--header 'Authorization: Bearer <access-token>' \
--header 'Content-Type: application/json' \
--data-raw '
{
  "_id": "audit",
...
  "eventHandlers": [
    {
      "class": "org.forgerock.audit.handlers.json.stdout.JsonStdoutAuditEventHandler",
      "config": {
        "elasticsearchCompatible": false,
        "enabled": true,
        "name": "json",
        "topics": [
          "access",
          "activity",
          "sync",
          "authentication",
          "config",
          "recon"
        ]
      }
    },
    {
      "class": "org.forgerock.openidm.audit.impl.RepositoryAuditEventHandler",
      "config": {
        "enabled": false,
        "name": "repo",
        "topics": [
          "access",
          "activity",
          "sync",
          "authentication",
          "config",
          "recon"
        ]
      }
    }
  ],
...
}' \
'https://<tenant-env-fqdn>/openidm/config/audit'

Include large log entries in filter log results

Some Identity Cloud log output is too large to be stored as a single log entry, so is stored across two log entries instead. When this happens, any log output in JSON format is stored as two plaintext log entries rather than a single JSON log entry. Consequently, any filter expression that filters on a specific JSON field will not find any of these plaintext log entries.

To work around this, you can combine a specific field filter with a plaintext filter. For example, if you were searching for log results containing a particular transaction ID using the filter expression:

/payload/transactionId co "<transaction-id>"

you could add a plaintext filter as follows:

/payload/transactionId co "<transaction-id>" or /payload co "<transaction-id>"

to include both JSON and plaintext log entries in the log results.

Copyright © 2010-2023 ForgeRock, all rights reserved.