How To
ForgeRock Identity Platform
Does not apply to Identity Cloud

How do I troubleshoot issues with my indexes in DS (All versions)?

Last updated Jun 14, 2021

The purpose of this article is to provide assistance on troubleshooting issues with your indexes in DS. You may be aware of issues with your indexes due to symptoms such as slow searches, poor performance, high CPU usage when searching and even long login times in AM when your user store is on DS.


2 readers recommend this article

Troubleshooting indexes

Issues with indexes can be caused by one of the following:

General health check

You can use the following backendstat command to provide a general health check on all indexes to help you evaluate how well attributes are indexed; you must stop the server prior to running this command:

  • DS 7 and later: $ ./backendstat show-index-status --backendID appData --baseDN dc=example,dc=com
  • Pre-DS 7: $ ./backendstat show-index-status --backendID userRoot --baseDN dc=example,dc=com

These commands return a list of all indexes and include information about the type of index, the associated database name, whether the index is valid and whether any keys in the index have exceeded the index's entry limit.

Caution

These commands can be expensive and take a long time to complete, as they read all indexes for all backends.

Degraded indexes

Degraded indexes occur when you have created or made changes to indexes, or the index becomes corrupt due to missing data. You can check for degraded indexes using the verify-index command. This is described in How do I verify indexes in DS (All versions) are correct?

Any degraded indexes must be rebuilt. You can rebuild specific indexes, all indexes or only degraded indexes as described in How do I rebuild indexes in DS (All versions)?

Missing Indexes 

Missing indexes (unindexed) result in long search times (high elapsedTimes values in the access log). This can even affect logging into AM if your user store is on DS and an attribute involved in the login process is not indexed, for example, the LDAP SAML attribute.

Missing indexes need to be created and built.

See Unindexed searches causing slow searches and poor performance on DS (All versions) server for further information on the symptoms and solution.

Incomplete indexing

Incomplete indexing can also cause problems with slow searches, but sometimes it can be difficult to pinpoint which attribute is causing the issue if there are multiple attributes in the search filter. 

You can use the debugsearchindex attribute with the ldapsearch command to identify issues such as unindexed attributes and exceeded entry limits. The debugsearchindex attribute tells DS to return an entry containing debug information that describes how the search operation would have been processed (rather than actually performing the search). See the debugsearchindex example below for further information.

Unnecessary indexes

Unused or rarely used indexes can also impact performance as DS has to maintain entries for all indexes.

You can run a script (topfilters) to find out which indexes are used most often to help you identify underused indexes. You can then remove these indexes using the dsconfig delete-backend-index command.

Caution

You must not delete the following indexes:

The source for the topfilters script can be found here: Githutb: OpenDJ Utilities - topfilters.

High index entry limits

Index entry limits are set to 4000 by default, which means that once the number of entries matching a given index exceeds this number, DS stops maintaining the entries for that index as it is an inefficient use of resources. This default value should be sufficient for most indexes other than objectClass indexes.

Caution

Do not set the index-entry-limit to the number of entries in your backend.

If you have set your index's entry limits much higher than this for any of your indexes, you may find your performance is impacted as DS has to maintain entries for very large indexes. You should try reducing the index entry limit back to the default value to see if this improves performance. See Index Entry Limits for further information.

Poorly defined search filters

Typically you should aim to have specific search filters in your searches to produce more accurate matches and also reduce the time taken to search.

For example, a search filter such as (objectClass=person) is very generalized and will produce a lot of matches. Instead, you should focus your search by adding other attributes to your search filter that would produce the matches you require. For example, you could use a filter similar to this to reduce the number of matches:

(&(mail=username@maildomain.net)(objectClass=person))

debugsearchindex example

You can use the information contained in the access logs to determine what you should include in your debugsearchindex. For example, the following sample access log snippets show high elapsedTimes for two different searches:

{"eventName":"DJ-LDAP","client":{"ip":"198.51.100.0","port":8443},"server":{"ip":"198.51.100.0","port":1636},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":107918,"msgId":2,"dn":"ou=People,dc=example,dc=org","scope":"one","filter":"(&(modifytimestamp>=20200501120001Z)(objectclass=person))","attrs":["uid email"]},"transactionId":"0","response":{"status":"FAILED","statusCode":"118","elapsedTime":16761,"elapsedTimeUnits":"MILLISECONDS","detail":"Client Disconnect","additionalItems":"unindexed","nentries":5},"timestamp":"2020-05-11T03:11:46.114Z","_id":"e3a44b25-4800-8cfa-1bad-6e2bb6461607-12"} {"eventName":"DJ-LDAP","client":{"ip":"198.51.100.0","port":8443},"server":{"ip":"198.51.100.0","port":1636},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":108320,"msgId":2,"dn":"ou=People,dc=example,dc=org","scope":"one","filter":"(&(iscurrent=1)(objectclass=person))","attrs":["uid email"]},"transactionId":"0","response":{"status":"FAILED","statusCode":"118","elapsedTime":46961,"elapsedTimeUnits":"MILLISECONDS","detail":"Client Disconnect","additionalItems":"unindexed","nentries":9},"timestamp":"2020-05-11T05:22:19.128Z","_id":"e3a44b25-4800-8cfa-1bad-6e2bb6461607-12"}

You can then construct searches using this information to show which indexes are being used to help you identify incomplete or missing indexes. For example:

  • DS 7.1 and later: $ ./ldapsearch --hostname ds1.example.com --port 1636 --useSsl --usePkcs12TrustStore /path/to/ds/config/keystore --trustStorePassword:file /path/to/ds/config/keystore.pin --bindDN uid=admin --bindPassword password --baseDN ou=People,dc=example,dc=org --searchScope one "(&(modifytimestamp>=20161019161001Z)(objectclass=person))" debugsearchindex dn: cn=debugsearch debugsearchindex: filter=(&(objectClass=person)[INDEX:objectClass.equality][LIMIT-EXCEEDED](modifyTimestamp>=20161019161001Z)[NOT-INDEXED])[NOT-INDEXED] scope=one[LIMIT-EXCEEDED:24482] final=[NOT-INDEXED] $ ./ldapsearch --hostname ds1.example.com --port 1636 --useSsl --usePkcs12TrustStore /path/to/ds/config/keystore --trustStorePassword:file /path/to/ds/config/keystore.pin --bindDN uid=admin --bindPassword password --baseDN ou=People,dc=example,dc=org --searchScope one "(&(iscurrent=1)(objectclass=person))" debugsearchindex dn: cn=debugsearch debugsearchindex: filter=(&(objectClass=person)[INDEX:objectClass.equality][LIMIT-EXCEEDED](iscurrent=1)[NOT-INDEXED])[NOT-INDEXED] scope=one[LIMIT-EXCEEDED:26021] final=[NOT-INDEXED]
  • DS 7: $ ./ldapsearch --hostname ds1.example.com --port 1636 --useSsl --usePkcs12TrustStore /path/to/ds/config/keystore --trustStorePasswordFile /path/to/ds/config/keystore.pin --bindDN uid=admin --bindPassword password --baseDN ou=People,dc=example,dc=org --searchScope one "(&(modifytimestamp>=20161019161001Z)(objectclass=person))" debugsearchindex dn: cn=debugsearch debugsearchindex: filter=(&(objectClass=person)[INDEX:objectClass.equality][LIMIT-EXCEEDED](modifyTimestamp>=20161019161001Z)[NOT-INDEXED])[NOT-INDEXED] scope=one[LIMIT-EXCEEDED:24482] final=[NOT-INDEXED] $ ./ldapsearch --hostname ds1.example.com --port 1636 --useSsl --usePkcs12TrustStore /path/to/ds/config/keystore --trustStorePasswordFile /path/to/ds/config/keystore.pin --bindDN uid=admin --bindPassword password --baseDN ou=People,dc=example,dc=org --searchScope one "(&(iscurrent=1)(objectclass=person))" debugsearchindex dn: cn=debugsearch debugsearchindex: filter=(&(objectClass=person)[INDEX:objectClass.equality][LIMIT-EXCEEDED](iscurrent=1)[NOT-INDEXED])[NOT-INDEXED] scope=one[LIMIT-EXCEEDED:26021] final=[NOT-INDEXED]
  • Pre-DS 7: $ ./ldapsearch --hostname ds1.example.com --port 1389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN ou=People,dc=example,dc=org --searchScope one "(&(modifytimestamp>=20161019161001Z)(objectclass=person))" debugsearchindex dn: cn=debugsearch debugsearchindex: filter=(&(objectClass=person)[INDEX:objectClass.equality][LIMIT-EXCEEDED](modifyTimestamp>=20161019161001Z)[NOT-INDEXED])[NOT-INDEXED] scope=one[LIMIT-EXCEEDED:24482] final=[NOT-INDEXED] $ ./ldapsearch --hostname ds1.example.com --port 1389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN ou=People,dc=example,dc=org --searchScope one "(&(iscurrent=1)(objectclass=person))" debugsearchindex dn: cn=debugsearch debugsearchindex: filter=(&(objectClass=person)[INDEX:objectClass.equality][LIMIT-EXCEEDED](iscurrent=1)[NOT-INDEXED])[NOT-INDEXED] scope=one[LIMIT-EXCEEDED:26021] final=[NOT-INDEXED]

See Debug Search Indexes for further information on the results obtained from the debugsearchindex option.

See Also

Unindexed searches causing slow searches and poor performance on DS (All versions) server

How do I verify indexes in DS (All versions) are correct?

How do I rebuild indexes in DS (All versions)?

How do I collect data for troubleshooting high CPU utilization on DS (All versions) servers?

Indexes

Configuration Reference

Related Training

ForgeRock Directory Services Core Concepts (DS-400)

Related Issue Tracker IDs

N/A


Copyright and Trademarks Copyright © 2021 ForgeRock, all rights reserved.