Troubleshooting

Define the Problem

To solve your problem, save time by clearly defining it first. A problem statement compares the difference between observed behavior and expected behavior:

  • What exactly is the problem?

    What is the behavior you expected?

    What is the behavior you observed?

  • How do you reproduce the problem?

  • When did the problem begin?

    Under similar circumstances, when does the problem not occur?

  • Is the problem permanent?

    Intermittent?

    Is it getting worse? Getting better? Staying the same?

Installation Problems

Use the logs

Installation and upgrade procedures result in a log file tracing the operation. Look for this in the command output:

See file for a detailed log of this operation.
Antivirus interference

Prevent antivirus and intrusion detection systems from interfering with DS software.

Before using DS software with antivirus or intrusion detection software, consider the following potential problems:

Interference with normal file access

Antivirus and intrusion detection systems that perform virus scanning, sweep scanning, or deep file inspection are not compatible with DS file access, particularly database file access.

Antivirus and intrusion detection software can interfere with the normal process of opening and closing database working files. They may incorrectly mark such files as suspect to infection due to normal database processing, which involves opening and closing files in line with the database's internal logic.

Prevent antivirus and intrusion detection systems from scanning database and changelog database files.

At minimum, configure antivirus software to whitelist the DS server database files. By default, exclude the following file system directories from virus scanning:

  • /path/to/opendj/changelogDb/ (if replication is enabled)

    Prevent the antivirus software from scanning these changelog database files.

  • /path/to/opendj/db/

    Prevent the antivirus software from scanning database files, especially *.jdb files.

Port blocking

Antivirus and intrusion detection software can block ports that DS uses to provide directory services.

Make sure that your software does not block the ports that DS software uses. For details, see "Administrative Access".

Negative performance impact

Antivirus software consumes system resources, reducing resources available to other services including DS servers.

Running antivirus software can therefore have a significant negative impact on DS server performance. Make sure that you test and account for the performance impact of running antivirus software before deploying DS software on the same systems.

JE initialization

When starting a directory server on a Linux system, make sure the server user can watch enough files. If the server user cannot watch enough files, you might see an error message in the server log such as this:

InitializationException: The database environment could not be opened:
com.sleepycat.je.EnvironmentFailureException: (JE version) /path/to/opendj/db/userData
or its sub-directories to WatchService.
UNEXPECTED_EXCEPTION: Unexpected internal Exception, may have side effects.
Environment is invalid and must be closed.

A directory server backend database monitors file events. On Linux systems, backend databases use the inotify API for this purpose. The kernel tunable fs.inotify.max_user_watches indicates the maximum number of files a user can watch with the inotify API. Make sure this tunable is set to at least 512K:

$ sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 524288

If this tunable is set lower than that, update the /etc/sysctl.conf file to change the setting permanently, and use the sysctl -p command to reload the settings:

$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
[sudo] password for admin: 
$ sudo sysctl -p
fs.inotify.max_user_watches = 524288

Forgotten Superuser Password

By default, DS servers store the entry for the directory superuser in an LDIF backend. Edit the file to reset the password:

  1. Generate the encoded version of the new password:

    $ encode-password --storageScheme PBKDF2-HMAC-SHA256 --clearPassword password
    encode-password --storageScheme PBKDF2-HMAC-SHA256 --clearPassword password
  2. Stop the server while you edit the LDIF file for the backend:

    $ stop-ds
  3. Replace the existing password with the encoded version.

    In the db/rootUser/rootUser.ldif file, carefully replace the userPassword value with the new, encoded password:

    dn: uid=admin
    ...
    uid: admin
    userPassword: <encoded-password>

    Trailing whitespace is significant in LDIF. Take care not to add any trailing whitespace at the end of the line.

  4. Restart the server:

    $ start-ds
  5. Verify that you can use the directory superuser account with the new password:

    $ status \
     --bindDn uid=admin \
     --bindPassword password \
     --hostname localhost \
     --port 4444 \
     --usePkcs12TrustStore /path/to/opendj/config/keystore \
     --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
     --script-friendly
    ...
    "isRunning" : true,

Debug Logging

Caution

DS debug logging can generate a high volume of debug messages. Use debug logging very sparingly on production systems.

  1. Create one or more debug targets.

    No debug targets are enabled by default:

    $ dsconfig \
     list-debug-targets \
     --hostname localhost \
     --port 4444 \
     --bindDN uid=admin \
     --bindPassword password \
     --publisher-name "File-Based Debug Logger" \
     --usePkcs12TrustStore /path/to/opendj/config/keystore \
     --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
     --no-prompt
    Debug Target : enabled : debug-exceptions-only
    -------------:---------:----------------------

    A debug target specifies a fully qualified DS Java package, class, or method:

    $ dsconfig \
     create-debug-target \
     --hostname localhost \
     --port 4444 \
     --bindDN uid=admin \
     --bindPassword password \
     --publisher-name "File-Based Debug Logger" \
     --type generic \
     --target-name org.opends.server.api \
     --set enabled:true \
     --usePkcs12TrustStore /path/to/opendj/config/keystore \
     --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
     --no-prompt
  2. Enable the debug log, opendj/logs/debug:

    $ dsconfig \
     set-log-publisher-prop \
     --hostname localhost \
     --port 4444 \
     --bindDN uid=admin \
     --bindPassword password \
     --publisher-name "File-Based Debug Logger" \
     --set enabled:true \
     --usePkcs12TrustStore /path/to/opendj/config/keystore \
     --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
     --no-prompt

    The server immediately begins to write debug messages to the log file.

  3. Read messages in the debug log file:

    $ tail -f /path/to/opendj/logs/debug
  4. Disable the debug log as soon as it is no longer required.

Lockdown Mode

Misconfiguration can put the DS server in a state where you must prevent users and applications from accessing the directory until you have fixed the problem.

DS servers support lockdown mode. Lockdown mode permits connections only on the loopback address, and permits only operations requested by superusers, such as uid=admin.

To put the DS server into lockdown mode, the server must be running. You cause the server to enter lockdown mode by starting a task. Notice that the modify operation is performed over the loopback address (accessing the DS server on the local host):

$ ldapmodify \
 --hostname localhost \
 --port 1636 \
 --useSsl \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
 --bindDN uid=admin \
 --bindPassword password << EOF
dn: ds-task-id=Enter Lockdown Mode,cn=Scheduled Tasks,cn=tasks
objectClass: top
objectClass: ds-task
ds-task-id: Enter Lockdown Mode
ds-task-class-name: org.opends.server.tasks.EnterLockdownModeTask
EOF

The DS server logs a notice message in logs/errors when lockdown mode takes effect:

...msg=Lockdown task Enter Lockdown Mode finished execution

Client applications that request operations get a message concerning lockdown mode:

$ ldapsearch \
 --hostname localhost \
 --port 1636 \
 --useSsl \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
 --baseDN "" \
 --searchScope base \
 "(objectclass=*)" \
 +
# The LDAP search request failed: 53 (Unwilling to Perform)
# Additional Information:  Rejecting the requested operation because the server is in lockdown mode and will only accept requests from root users over loopback connections

Leave lockdown mode by starting a task:

$ ldapmodify \
 --hostname localhost \
 --port 1636 \
 --useSsl \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
 --bindDN uid=admin \
 --bindPassword password << EOF
dn: ds-task-id=Leave Lockdown Mode,cn=Scheduled Tasks,cn=tasks
objectClass: top
objectClass: ds-task
ds-task-id: Leave Lockdown Mode
ds-task-class-name: org.opends.server.tasks.LeaveLockdownModeTask
EOF

The DS server logs a notice message when leaving lockdown mode:

...msg=Leave Lockdown task Leave Lockdown Mode finished execution

LDIF Import

  • By default, DS directory servers check that entries you import match the LDAP schema.

    You can temporarily bypass this check with the import-ldif --skipSchemaValidation option.

  • By default, DS servers ensure that entries have only one structural object class.

    You can relax this behavior with the advanced global configuration property, single-structural-objectclass-behavior.

    This can be useful when importing data exported from Sun Directory Server.

    For example, warn when entries have more than one structural object class, rather than rejecting them:

    $ dsconfig \
     set-global-configuration-prop \
     --hostname localhost \
     --port 4444 \
     --bindDN uid=admin \
     --bindPassword password \
     --set single-structural-objectclass-behavior:warn \
     --usePkcs12TrustStore /path/to/opendj/config/keystore \
     --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
     --no-prompt
  • By default, DS servers check syntax for several attribute types. Relax this behavior using the advanced global configuration property, invalid-attribute-syntax-behavior.

  • Use the import-ldif -R rejectFile --countRejects options to log rejected entries and to return the number of rejected entries as the command's exit code.

Once you resolve the issues, reinstate the default behavior to avoid importing bad data.

Security Problems

Certificate-Based Authentication

Replication uses TLS to protect directory data on the network. Misconfiguration can cause replicas to fail to connect due to handshake errors. This leads to repeated error log messages in the replication log file such as the following:

...msg=Replication server accepted a connection from address
 to local address address but the SSL handshake failed.
 This is probably benign, but may indicate a transient network outage
 or a misconfigured client application connecting to this replication server.
 The error was: Received fatal alert: certificate_unknown

You can collect debug trace messages to help determine the problem. To see the TLS debug messages, start the server with javax.net.debug set:

$ OPENDJ_JAVA_ARGS="-Djavax.net.debug=all" start-ds

The debug trace settings result in a large number of messages. To resolve the problem, review the output of starting the server, looking in particular for handshake errors.

If the chain of trust for your PKI is broken somehow, consider renewing or replacing keys, as described in "Key Management". Make sure that trusted CA certificates are configured as expected.

Compromised Keys

How you handle the problem depends on which key was compromised:

  • For keys generated by the server, or with a deployment key, see "Retire Secret Keys".

  • For a private key whose certificate was signed by a CA, contact the CA for help. The CA might choose to publish a certificate revocation list (CRL) that identifies the certificate of the compromised key.

    Replace the key pair that has the compromised private key.

  • For a private key whose certificate was self-signed, replace the key pair that has the compromised private key.

    Make sure the clients remove the compromised certificate from their truststores. They must replace the certificate of the compromised key with the new certificate.

Client Problems

Use the logs

By default, DS servers record messages for LDAP client operations in the logs/ldap-access.audit.json log file.

{
  "eventName": "DJ-LDAP",
  "client": {
    "ip": "<clientIp>",
    "port": <clientPort>
  },
  "server": {
    "ip": "<clientIp>",
    "port": 1389
  },
  "request": {
    "protocol": "LDAP",
    "operation": "CONNECT",
    "connId": 0
  },
  "transactionId": "0",
  "response": {
    "status": "SUCCESSFUL",
    "statusCode": "0",
    "elapsedTime": 0,
    "elapsedTimeUnits": "MILLISECONDS"
  },
  "timestamp": "<timestamp>",
  "_id": "<uuid>"
}
{
  "eventName": "DJ-LDAP",
  "client": {
    "ip": "<clientIp>",
    "port": <clientPort>
  },
  "server": {
    "ip": "<clientIp>",
    "port": 1389
  },
  "request": {
    "protocol": "LDAP",
    "operation": "SEARCH",
    "connId": 0,
    "msgId": 1,
    "dn": "dc=example,dc=com",
    "scope": "sub",
    "filter": "(uid=bjensen)",
    "attrs": ["ALL"]
  },
  "transactionId": "0",
  "response": {
    "status": "SUCCESSFUL",
    "statusCode": "0",
    "elapsedTime": 9,
    "elapsedTimeUnits": "MILLISECONDS",
    "nentries": 1
  },
  "timestamp": "<timestamp>",
  "_id": "<uuid>"
}
{
  "eventName": "DJ-LDAP",
  "client": {
    "ip": "<clientIp>",
    "port": <clientPort>
  },
  "server": {
    "ip": "<clientIp>",
    "port": 1389
  },
  "request": {
    "protocol": "LDAP",
    "operation": "UNBIND",
    "connId": 0,
    "msgId": 2
  },
  "transactionId": "0",
  "timestamp": "<timestamp>",
  "_id": "<uuid>"
}
{
  "eventName": "DJ-LDAP",
  "client": {
    "ip": "<clientIp>",
    "port": <clientPort>
  },
  "server": {
    "ip": "<clientIp>",
    "port": 1389
  },
  "request": {
    "protocol": "LDAP",
    "operation": "DISCONNECT",
    "connId": 0
  },
  "transactionId": "0",
  "response": {
    "status": "SUCCESSFUL",
    "statusCode": "0",
    "elapsedTime": 0,
    "elapsedTimeUnits": "MILLISECONDS",
    "reason": "Client Unbind"
  },
  "timestamp": "<timestamp>",
  "_id": "<uuid>"
}

Each message specifies the operation performed, the client that requested the operation, and when it completed.

By default, the server does not log internal LDAP operations corresponding to HTTP requests. To match HTTP client operations to internal LDAP operations:

  1. Prevent the server from suppressing log messages for internal operations.

    Set suppress-internal-operations:false on the LDAP access log publisher.

  2. Match the request/connId field in the HTTP access log with the same field in the LDAP access log.

Client access

To help diagnose client errors due to access permissions, see "Effective Rights".

Simple paged results

For some versions of Linux, you see a message in the DS access logs such as the following:

The request control with Object Identifier (OID) "1.2.840.113556.1.4.319"
cannot be used due to insufficient access rights

This message means clients are trying to use the simple paged results control without authenticating. By default, a global ACI allows only authenticated users to use the control.

To grant anonymous (unauthenticated) user access to the control, add a global ACI for anonymous use of the simple paged results control:

$ dsconfig \
 set-access-control-handler-prop \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword "password" \
 --add global-aci:"(targetcontrol=\"SimplePagedResults\") \
 (version 3.0; acl \"Anonymous simple paged results access\"; allow(read) \
 userdn=\"ldap:///anyone\";)" \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
 --no-prompt

Replication Problems

Replicas do not connect

If you set up servers with different deployment keys, they cannot trust each others' connections. You may see messages like the following in the logs/replication log file:

msg=Replication server accepted a connection from /address:port to local address /address:port but the SSL handshake failed.

Unless the servers use your own CA, make sure their keys are generated with the same deployment key/password. Either set up the servers again with the same deployment key, or see "Replace Deployment Keys".

Temporary delays

Replication can generally recover from conflicts and transient issues. Temporary delays are normal and expected while replicas converge, especially when the write load is heavy. This is a feature of eventual convergence, not a bug.

For more information, see "Replication Delay (LDAP)".

Use the logs

Replication uses its own error log file, logs/replication. Error messages in the log file have category=SYNC. The messages have the following form. The following example message is folded for readability:

...msg=Replication server accepted a connection from 10.10.0.10/10.10.0.10:52859
 to local address 0.0.0.0/0.0.0.0:8989 but the SSL handshake failed.
 This is probably benign, but may indicate a transient network outage
 or a misconfigured client application connecting to this replication server.
 The error was: Remote host closed connection during handshake
Stale data

DS servers maintain historical information to bring replicas up to date, and to resolve conflicts. To prevent historical information from growing without limit, servers purge historical information after a configurable delay (replication-purge-delay, default: 3 days). A replica can become irrevocably out of sync if you restore it from a backup that is older than the purge delay, or if you stop it for longer than the purge delay. If this happens, reinitialize the replica from a recent backup or from a server that is up to date.

Support

Sometimes you cannot resolve a problem yourself, and must ask for help or technical support. In such cases, identify the problem and how you reproduce it, and the version where you see the problem:

$ status --offline --version
ForgeRock Directory Services 7.0.1
Build <datestamp>

Be prepared to provide the following additional information:

  • The Java home set in config/java.properties.

  • Access and error logs showing what the server was doing when the problem started occurring.

  • A copy of the server configuration file, config/config.ldif, in use when the problem started occurring.

  • Other relevant logs or output, such as those from client applications experiencing the problem.

  • A description of the environment where the server is running, including system characteristics, hostnames, IP addresses, Java versions, storage characteristics, and network characteristics. This helps to understand the logs, and other information.

  • The .zip file generated using the supportextract command.

    For an example showing how to use the command, see "Examples".

Read a different version of :