Manage logs

Configure a custom access log

This procedure applies only to Common Audit logs.

An access logger with a JSON configuration lets you use any Common Audit event handler, including customer handlers. The content of the configuration file depends on the audit event handler:

Decide whether to trust transaction IDs sent by client applications, used to correlate requests as they traverse multiple servers.

Client applications using the common audit event framework send transaction IDs with their requests. The transaction IDs correlate audit events, tracing the request through multiple applications.

Transaction IDs are sent over LDAP using an internal DS request control. They are sent over HTTP in an HTTP header.

By default, DS servers do not trust transaction IDs sent with client application requests.

When a server trusts transaction IDs from client application requests, outgoing requests reuse the incoming ID. For each outgoing request in the transaction, the request’s transaction ID has the form original-transaction-id/sequence-number, where sequence-number reflects the position of the request in the series of requests for this transaction. For example, if the original-transaction-id is abc123, the first outgoing request has the transaction ID abc123/0, the second abc123/1, the third abc123/2, and so on. This lets you distinguish specific requests within a transaction when correlating audit events from multiple services.

To trust transactions, set the advanced global server property, trust-transaction-ids:true:
```
$ dsconfig \
 set-global-configuration-prop \
 --advanced \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --set trust-transaction-ids:true \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt
```
Create the external JSON configuration file for the handler.

Base your work on the appropriate template in the config/audit-handlers directory.
If this is a custom access logger provided separately, copy the custom handler .jar file to opendj/lib/extensions.

Create a log publisher configuration for the access log.

The type defines whether the log contains messages about LDAP or HTTP requests:

For LDAP access logging, create an external access log publisher:

$ dsconfig \
 create-log-publisher \
 --publisher-name "Custom LDAP Access Logger" \
 --type external-access \
 --set enabled:true \
 --set config-file:config/audit-handlers/handler-conf.json \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

For HTTP access logging, create an external HTTP access log publisher:

$ dsconfig \
 create-log-publisher \
 --publisher-name "Custom HTTP Access Logger" \
 --type external-http-access \
 --set enabled:true \
 --set config-file:config/audit-handlers/handler-conf.json \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

Log access to standard output

This procedure applies only to Common Audit file-based logs, and when you install DS using procedures from Installation.

The sample DS Docker image creates these console access loggers by default:

Console LDAP Access Logger
Console HTTP Access Logger

A JSON stdout handler sends messages to standard output.

Only use this logger when running the server with start-ds --noDetach.

When running as a daemon without the --noDetach option, the server also logs the messages to the file, /path/to/logs/server.out. The server has no mechanism for rotating or removing the server.out log file, which is only cleared when the server starts.

As a result, using the JSON stdout handler when running the server without the --noDetach option can cause the server to eventually run out of disk space.

Enable the JSON stdout handler.

For details, refer to Configure a custom access log.

The JSON configuration file for the JSON stdout handler has the following format:

{
  "class": "org.forgerock.audit.handlers.json.stdout.JsonStdoutAuditEventHandler",
  "config": {
    "enabled": boolean,                   // Is the handler enabled?
    "name": string,                       // Handler name, such as "json.stdout".
    "elasticsearchCompatible" : boolean,  // If true, the message ID field is named _eventId.
                                          // (Default: _id)
    "topics": array,                      // LDAP: "ldap-access"; HTTP: "http-access".
  }
}

For a sample configuration, refer to opendj/config/audit-handlers/json-stdout-config.json-example.

Log errors to standard output

A console-error logger sends messages to standard output.

This procedure applies only when you install DS using procedures from Installation. The sample DS Docker image creates a Console Error Logger by default.

Only use this logger when running the server with start-ds --noDetach.

As a result, using the JSON stdout handler when running the server without the --noDetach option can cause the server to eventually run out of disk space.

Switch to a console-error logger while the server is offline:

$ stop-ds

$ dsconfig \
 delete-log-publisher \
 --publisher-name "File-Based Error Logger" \
 --offline \
 --configFile /path/to/opendj/config/config.ldif \
 --no-prompt

$ dsconfig \
 create-log-publisher \
 --type console-error \
 --publisher-name "Console Error Logger" \
 --set enabled:true \
 --set default-severity:notice \
 --set override-severity:SYNC=INFO \
 --offline \
 --configFile /path/to/opendj/config/config.ldif \
 --no-prompt

$ start-ds --noDetach

Log error messages as JSON

By default, error log messages start with a timestamp, followed by space-separated field=value pairs, where the last msg field contains free-form text. The format is human-readable, but can be less convenient to parse than standard JSON.

You can set the error log publisher property json-output:true to cause the publisher to generate JSON messages:

Stop DS.
Move existing, free-form text error log files.

Change to JSON output.

The following command changes the default error log publisher configuration while DS is stopped:

$ dsconfig \
 set-log-publisher-prop \
 --publisher-name "File-Based Error Logger" \
 --set json-output:true \
 --offline \
 --configFile /path/to/opendj/config/config.ldif \
 --no-prompt

Start DS.

Rotate and retain logs

Each file-based log has a rotation policy, and a retention policy.

The rotation policy specifies when to rotate a log file based on a time, log file age, or log file size. Rotated logs have a rotation timestamp appended to their name.

The retention policy specifies whether to retain logs based on the number of logs, their size, or how much free space should be left on the disk.

List log rotation policies:

$ dsconfig \
 list-log-rotation-policies \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

Log Rotation Policy                 : Type       : file-size-limit : rotation-interval : time-of-day
------------------------------------:------------:-----------------:-------------------:------------
24 Hours Time Limit Rotation Policy : time-limit : -               : 1 d               : -
7 Days Time Limit Rotation Policy   : time-limit : -               : 1 w               : -
Fixed Time Rotation Policy          : fixed-time : -               : -                 : 2359
Size Limit Rotation Policy          : size-limit : 100 mb          : -                 : -

List log retention policies:

$ dsconfig \
 list-log-retention-policies \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

Log Retention Policy             : Type            : disk-space-used : free-disk-space : number-of-files
---------------------------------:-----------------:-----------------:-----------------:----------------
File Count Retention Policy      : file-count      : -               : -               : 10
Free Disk Space Retention Policy : free-disk-space : -               : 500 mb          : -
Size Limit Retention Policy      : size-limit      : 500 mb          : -               : -

View the policies that apply to a given log with the dsconfig get-log-publisher-prop command.

The following example shows that the server keeps 10 access log files, rotating either each day or when the log size reaches 100 MB:

$ dsconfig \
 get-log-publisher-prop \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --publisher-name "Json File-Based Access Logger" \
 --property retention-policy \
 --property rotation-policy \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

Property         : Value(s)
-----------------:-------------------------------------------------------------
retention-policy : File Count Retention Policy
rotation-policy  : 24 Hours Time Limit Rotation Policy, Size Limit Rotation
                 : Policy

Use the dsconfig command to create, update, delete, and assign log rotation and retention policies. Set the policy that applies to a logger with the dsconfig set-log-publisher-prop command.

When using access logs based on the common audit event framework, you can only configure one of each type of retention or rotation policy.

This means you can configure only one file count, free disk space, and size limit log retention policy. You can configure only one fixed time, size limit, and time limit log rotation policy.

Enable an audit log

Enable a file-based audit logger:

$ dsconfig \
 set-log-publisher-prop \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --publisher-name "File-Based Audit Logger" \
 --set enabled:true \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

Wait for, or make a change to directory data.

The following example changes a description:

$ ldapmodify \
 --hostname localhost \
 --port 1636 \
 --useSsl \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --bindDN "uid=bjensen,ou=People,dc=example,dc=com" \
 --bindPassword hifalutin << EOF
dn: uid=bjensen,ou=People,dc=example,dc=com
changetype: modify
replace: description
description: New description
EOF

The audit log records the changes as shown in the following excerpt:

# <datestamp>; conn=<number>; op=<number>
dn: cn=File-Based Audit Logger,cn=Loggers,cn=config
changetype: modify
replace: ds-cfg-enabled
ds-cfg-enabled: true
-

# <datestamp>; conn=<number>; op=<number>
dn: uid=bjensen,ou=people,dc=example,dc=com
changetype: modify
add: description
description: New description
-

Audit logs record changes in LDIF format. This means that when an LDAP entry is deleted, the audit log records only its DN.

Filter out administrative messages

A common development troubleshooting technique consists of sending client requests while tailing the access log:

$ tail -f /path/to/opendj/logs/ldap-access.audit.json

When the dsconfig command accesses the configuration, the access log records this. Such messages can prevent you from noticing the messages of interest from client applications.

You can filter access log messages for administrative connections to the administration port:

Configure access log filtering criteria:

$ dsconfig \
 create-access-log-filtering-criteria \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --publisher-name "Json File-Based Access Logger" \
 --criteria-name "Exclude LDAPS on 4444" \
 --type generic \
 --set connection-port-equal-to:4444 \
 --set connection-protocol-equal-to:ldaps \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

Activate filtering to exclude administrative messages:

$ dsconfig \
 set-log-publisher-prop \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --publisher-name "Json File-Based Access Logger" \
 --set filtering-policy:exclusive \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

The publisher filters messages about administrative requests to the administration port.

Audit configuration changes

This example demonstrates how to set up an audit log file to track changes to the server configuration.

Audit log change records have timestamped comments with connection and operation IDs. You can use these to correlate the changes with messages in access logs:

Create an audit log publisher:

$ dsconfig \
 create-log-publisher \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --publisher-name "File-Based Server Configuration Audit Log" \
 --type file-based-audit \
 --set enabled:true \
 --set filtering-policy:inclusive \
 --set log-file:logs/config-audit \
 --set rotation-policy:"24 Hours Time Limit Rotation Policy" \
 --set rotation-policy:"Size Limit Rotation Policy" \
 --set retention-policy:"File Count Retention Policy" \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

Create log filtering criteria for the logger that matches operations targeting cn=config:

$ dsconfig \
 create-access-log-filtering-criteria \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --publisher-name "File-Based Server Configuration Audit Log" \
 --criteria-name "Record changes to cn=config" \
 --set request-target-dn-equal-to:"**,cn=config" \
 --set request-target-dn-equal-to:"cn=config" \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

The server now writes to the audit log file, /path/to/opendj/logs/config-audit, whenever an administrator changes the server configuration. The following example output shows the resulting LDIF that defines the log filtering criteria:

# <datestamp>; conn=<id>; op=<id>
dn: cn=Record changes to cn=config,cn=Filtering Criteria,cn=File-Based Server Configuration Audit Log,cn=Loggers,cn=config
changetype: add
objectClass: top
objectClass: ds-cfg-access-log-filtering-criteria
cn: Record changes to cn=config
ds-cfg-request-target-dn-equal-to: **,cn=config
ds-cfg-request-target-dn-equal-to: cn=config
createTimestamp: <timestamp>
creatorsName: editable:dsAdminDN["uid=admin"]
entryUUID: <uuid>

Allow log message fields

When an object is passed in a Common Audit event, it might contain information that should not be logged. By default, the Common Audit implementation uses a whitelist to specify which fields of the event appear:
1. For Common Audit HTTP access log publishers, edit the log-field-whitelist property.
  
  The following fields appear by default, with each field listed by its JSON path. You cannot change the default whitelist.
  
  If a whitelisted field contains an object, then listing the field means the whole object is whitelisted:
  - /_id
  - /timestamp
  - /eventName
  - /transactionId
  - /trackingIds
  - /userId
  - /client
  - /server
  - /http/request/secure
  - /http/request/method
  - /http/request/path
  - /http/request/headers/accept
  - /http/request/headers/accept-api-version
  - /http/request/headers/content-type
  - /http/request/headers/host
  - /http/request/headers/user-agent
  - /http/request/headers/x-forwarded-for
  - /http/request/headers/x-forwarded-host
  - /http/request/headers/x-forwarded-port
  - /http/request/headers/x-forwarded-proto
  - /http/request/headers/x-original-uri
  - /http/request/headers/x-real-ip
  - /http/request/headers/x-request-id
  - /http/request/headers/x-requested-with
  - /http/request/headers/x-scheme
  - /request
  - /response
For CSV logs, the values map to the column headers. The terms are separated by dots (.) rather than by slashes (/).
1. LDAP access loggers do not support whitelisting.
  
  By default, all fields are whitelisted.

Deny log message fields

When an object is passed in a Common Audit event, it might contain information that should not be logged. Loggers allow all fields that are safe to log by default. The whitelist is processed before the blacklist, so blacklist settings overwrite the whitelist defaults:

Blacklist individual fields in common audit access logs to prevent the fields from appearing in messages.

The following example prevents all request headers from appearing in JSON HTTP access logs:
```
$ dsconfig \
 set-log-publisher-prop \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --publisher-name "Json File-Based HTTP Access Logger" \
 --set log-field-blacklist:/http/response/headers \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt
```
The blacklist values are JSON paths to the fields in log messages.

For CSV logs, the blacklist values map to the column headers. The terms are separated by dots (.) rather than by slashes (/).

Make tampering evident

This procedure applies only to Common Audit-based logs.

Tamper-evident logging depends on a public key/private key pair and a secret key. The Common Audit framework accesses the keys in a JCEKS-type keystore. Follow these steps to prepare the keystore:

Create a password for the keystore.

The examples below use an AUDIT_KEYSTORE_PIN environment variable that contains the password.
Generate a key pair in the keystore.

The keystore holds a signing key with the alias Signature. Generate the key with the RSA key algorithm, and the SHA256withRSA signature algorithm.

The following example uses the default file name:
```
$ keytool \
 -genkeypair \
 -keyalg RSA \
 -sigalg SHA256withRSA \
 -alias "Signature" \
 -dname "CN=ds.example.com,O=Example Corp,C=FR" \
 -keystore /path/to/opendj/config/audit-keystore \
 -storetype JCEKS \
 -storepass:env AUDIT_KEYSTORE_PIN \
 -keypass:env AUDIT_KEYSTORE_PIN
```
You can configure the file name with the log publisher key-store-file property.
Generate a secret key in the keystore.

The keystore holds a symmetric key with the alias Password. Generate the key with the HmacSHA256 key algorithm, and 256-bit key size.

The following example uses the default file name:
```
$ keytool \
 -genseckey \
 -keyalg HmacSHA256 \
 -keysize 256 \
 -alias "Password" \
 -keystore /path/to/opendj/config/audit-keystore \
 -storetype JCEKS \
 -storepass:env AUDIT_KEYSTORE_PIN \
 -keypass:env AUDIT_KEYSTORE_PIN
```
You can configure the file name with the log publisher key-store-file property.

Verify that the keystore contains signature and password keys:

$ keytool \
 -list \
 -keystore /path/to/opendj/config/audit-keystore \
 -storetype JCEKS \
 -storepass:env AUDIT_KEYSTORE_PIN

password, <date>, SecretKeyEntry,
signature, <date>, PrivateKeyEntry,
Certificate fingerprint (SHA-256): <fingerprint>

PingDS 7.5.1