Guide to configuring and using ForgeRock® Directory Services features.

Phone companies have been managing directories for many decades. The Internet itself has relied on distributed directory services like DNS since the mid 1980s.

It was not until the late 1980s, however, that experts from what is now the International Telecommunications Union published the X.500 set of international standards, including Directory Access Protocol. The X.500 standards specify Open Systems Interconnect (OSI) protocols and data definitions for general purpose directory services. The X.500 standards were designed to meet the needs of systems built according to the X.400 standards, covering electronic mail services.

Lightweight Directory Access Protocol has been around since the early 1990s. LDAP was originally developed as an alternative protocol that would allow directory access over Internet protocols rather than OSI protocols, and be lightweight enough for desktop implementations. By the mid-1990s, LDAP directory servers became generally available and widely used.

Until the late 1990s, LDAP directory servers were designed primarily with quick lookups and high availability for lookups in mind. LDAP directory servers replicate data, so when an update is made, that update is applied to other peer directory servers. Thus, if one directory server goes down, lookups can continue on other servers. Furthermore, if a directory service needs to support more lookups, the administrator can simply add another directory server to replicate with its peers.

As organizations rolled out larger and larger directories serving more and more applications, they discovered that they needed high availability not only for lookups, but also for updates. Around the year 2000, directories began to support multi-master replication; that is, replication with multiple read-write servers. Soon thereafter, the organizations with the very largest directories started to need higher update performance as well as availability.

The DS code base began in the mid-2000s, when engineers solving the update performance issue decided that the cost of adapting the existing C-based directory technology for high-performance updates would be higher than the cost of building new, high-performance directory using Java technology.

LDAP directory data is organized into entries, similar to the entries for words in the dictionary, or for subscriber names in the phone book. A sample entry follows:

dn: uid=bjensen,ou=People,dc=example,dc=com
uid: bjensen
cn: Babs Jensen
cn: Barbara Jensen
facsimileTelephoneNumber: +1 408 555 1992
gidNumber: 1000
givenName: Barbara
homeDirectory: /home/bjensen
l: San Francisco
mail: bjensen@example.com
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: posixAccount
objectClass: top
ou: People
ou: Product Development
roomNumber: 0209
sn: Jensen
telephoneNumber: +1 408 555 1862
uidNumber: 1076
  

Barbara Jensen's entry has a number of attributes, such as uid: bjensen, telephoneNumber: +1 408 555 1862, and objectClass: posixAccount. (The objectClass attribute type indicates which types of attributes are required and allowed for the entry. As the entries object classes can be updated online, and even the definitions of object classes and attributes are expressed as entries that can be updated online, directory data is extensible on the fly.) When you look up her entry in the directory, you specify one or more attributes and values to match. The directory server then returns entries with attribute values that match what you specified.

The attributes you search for are indexed in the directory, so the directory server can retrieve them more quickly. Attribute values do not have to be strings. Some attribute values, like certificates and photos, are binary.

The entry also has a unique identifier, shown at the top of the entry, dn: uid=bjensen,ou=People,dc=example,dc=com. DN is an acronym for distinguished name. No two entries in the directory have the same distinguished name. Yet, DNs are typically composed of case-insensitive attributes.

Sometimes distinguished names include characters that you must escape. The following example shows an entry that includes escaped characters in the DN:

$ ldapsearch --port 1389 --baseDN dc=example,dc=com "(uid=escape)"
dn: cn=DN Escape Characters \" # \+ \, \; \< = \> \\,dc=example,dc=com
objectClass: person
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: top
givenName: DN Escape Characters
uid: escape
cn: DN Escape Characters " # + , ; < = > \
sn: " # + , ; < = > \
mail: escape@example.com

  

LDAP entries are arranged hierarchically in the directory. The hierarchical organization resembles a file system on a PC or a web server, often imagined as an upside down tree structure, or a pyramid. The distinguished name consists of components separated by commas, uid=bjensen,ou=People,dc=example,dc=com. The names are little-endian. The components reflect the hierarchy of directory entries.

"Directory Data" shows the hierarchy.

Directory Data

Barbara Jensen's entry is located under an entry with DN ou=People,dc=example,dc=com, an organization unit and parent entry for the people at Example.com. The ou=People entry is located under the entry with DN dc=example,dc=com, the base entry for Example.com. DC is an acronym for domain component. The directory has other base entries, such as cn=config, under which the configuration is accessible through LDAP. A directory can serve multiple organizations, too. You might find dc=example,dc=com, dc=mycompany,dc=com, and o=myOrganization in the same LDAP directory. Therefore, when you look up entries, you specify the base DN to look under in the same way you need to know whether to look in the New York, Paris, or Tokyo phone book to find a telephone number. The root entry for the directory, technically the entry with DN "" (the empty string), is called the root DSE. It contains information about what the server supports, including the other base DNs it serves.

A directory server stores two kinds of attributes in a directory entry: user attributes and operational attributes. User attributes hold the information for users of the directory. All of the attributes shown in the entry at the outset of this section are user attributes. Operational attributes hold information used by the directory itself. Examples of operational attributes include entryUUID, modifyTimestamp, and subschemaSubentry. When an LDAP search operation finds an entry in the directory, the directory server returns all the visible user attributes unless the search request restricts the list of attributes by specifying those attributes explicitly. The directory server does not, however, return any operational attributes unless the search request specifically asks for them. Generally speaking, applications should change only user attributes, and leave updates of operational attributes to the server, relying on public directory server interfaces to change server behavior. An exception is access control instruction (aci) attributes, which are operational attributes used to control access to directory data.

In some client server communication, like web browsing, a connection is set up and then torn down for each client request to the server. LDAP has a different model. In LDAP the client application connects to the server and authenticates, then requests any number of operations, perhaps processing results in between requests, and finally disconnects when done.

For practical examples showing how to perform the key operations using the command-line tools delivered with DS servers, read "Performing LDAP Operations" in the Developer's Guide.

LDAP has standardized two mechanisms for extending the operations directory servers can perform beyond the basic operations listed above. One mechanism involves using LDAP controls. The other mechanism involves using LDAP extended operations.

LDAP controls are information added to an LDAP message to further specify how an LDAP operation should be processed. For example, the Server-Side Sort request control modifies a search to request that the directory server return entries to the client in sorted order. The Subtree Delete request control modifies a delete to request that the server also remove child entries of the entry targeted for deletion.

One special search operation that DS servers support is Persistent Search. The client application sets up a Persistent Search to continue receiving new results whenever changes are made to data that is in the scope of the search, thus using the search as a form of change notification. Persistent Searches are intended to remain connected permanently, though they can be idle for long periods of time.

The directory server can also send response controls in some cases to indicate that the response contains special information. Examples include responses for entry change notification, password policy, and paged results.

For the list of supported LDAP controls, see "LDAP Controls" in the Reference.

LDAP extended operations are additional LDAP operations not included in the original standard list. For example, the Cancel Extended Operation works like an abandon operation, but finishes with a response from the server after the cancel is complete. The StartTLS Extended Operation allows a client to connect to a server on an unsecure port, but then starts Transport Layer Security negotiations to protect communications.

For the list of supported LDAP extended operations, see "LDAP Extended Operations" in the Reference.

As mentioned early in this chapter, directories have indexes for multiple attributes. By default, DS does not let normal users perform searches that are not indexed, because such searches mean DS servers have to scan an entire directory database when looking for matches.

As directory administrator, part of your responsibility is making sure directory data is properly indexed. DS software provides tools for building and rebuilding indexes, for verifying indexes, and for evaluating how well indexes are working.

For help better understanding and managing indexes, read "Indexing Attribute Values".

Some databases are designed to hold huge amounts of data for a particular application. Although such databases might support multiple applications, how their data is organized depends a lot on the particular applications served.

In contrast, directories are designed for shared, centralized services. Although the first guides to deploying directory services suggested taking inventory of all the applications that would access the directory, many current directory administrators do not even know how many applications use their services. The shared, centralized nature of directory services fosters interoperability in practice, and has helped directory services be successful in the long term.

Part of what makes this possible is the shared model of directory user information, and in particular the LDAP schema. LDAP schema defines what the directory can contain. This means that directory entries are not arbitrary data, but instead tightly codified objects whose attributes are completely predictable from publicly readable definitions. Many schema definitions are in fact standard. They are the same not just across a directory service but across different directory services.

At the same time, unlike some databases, LDAP schema and the data it defines can be extended on the fly while the service is running. LDAP schema is also accessible over LDAP. One attribute of every entry is its set of objectClass values. This gives you as administrator great flexibility in adapting your directory service to store new data without losing or changing the structure of existing data, and also without ever stopping your directory service.

For a closer look, see "Managing Schema".

In addition to directory schema, another feature of directory services that enables sharing is fine-grained access control.

As directory administrator, you can control who has access to what data when, how, where and under what conditions by using access control instructions (ACI). You can allow some directory operations and not others. You can scope access control from the whole directory service down to individual attributes on directory entries. You can specify when, from what host or IP address, and what strength of encryption is needed in order to perform a particular operation.

As ACIs are stored on entries in the directory, you can furthermore update access controls while the service is running, and even delegate that control to client applications. DS software combines the strengths of ACIs with separate administrative privileges to help you secure access to directory data.

For more information, read "Configuring Privileges and Access Control".

DS replication consists of copying each update to the directory service to multiple directory servers. This brings both redundancy, in the case of network partitions or of crashes, and scalability for read operations. Most directory deployments involve multiple servers replicating together.

When you have replicated servers, all of which are writable, you can have replication conflicts. What if, for example, there is a network outage between two replicas, and meanwhile two different values are written to the same attribute on the same entry on the two replicas? In nearly all cases, DS replication can resolve these situations automatically without involving you, the directory administrator. This makes your directory service resilient and safe even in the unpredictable real world.

One perhaps counterintuitive aspect of replication is that although you do add directory read capacity by adding replicas to your deployment, you do not add directory write capacity by adding replicas. As each write operation must be replayed everywhere, the result is that if you have N servers, you have N write operations to replay.

Another aspect of replication to keep in mind is that it is "loosely consistent." Loosely consistent means that directory data will eventually converge to be the same everywhere, but it will not necessarily be the same everywhere right away. Client applications sometimes get this wrong when they write to a pool of load balanced directory servers, immediately read back what they wrote, and are surprised that it is not the same. If your users are complaining about this, either make sure their application always gets sent to the same server, or else ask that they adapt their application to work in a more realistic manner.

To get started with replication, see "Managing Data Replication".

Directory Services Markup Language (DSMLv2) v2.0 became a standard in 2001. DSMLv2 describes directory data and basic directory operations in XML format, so they can be carried in Simple Object Access Protocol (SOAP) messages. DSMLv2 further allows clients to batch multiple operations together in a single request, to be processed either in sequential order or in parallel.

DS software provides support for DSMLv2 as a DSML gateway, which is a Servlet that connects to any standard LDAPv3 directory. DSMLv2 opens basic directory services to SOAP-based web services and service oriented architectures.

To set up DSMLv2 access, see "DSML Client Access".

DS software can expose directory data as JSON resources over HTTP to REST clients, providing easy access to directory data for developers who are not familiar with LDAP. RESTful access depends on a configuration that describes how the JSON representation maps to LDAP entries.

Although client applications have no need to understand LDAP, the underlying implementation still uses the LDAP model for its operations. The mapping adds some overhead. Furthermore, depending on the configuration, individual JSON resources can require multiple LDAP operations. For example, an LDAP user entry represents manager as a DN (of the manager's entry). The same manager might be represented in JSON as an object holding the manager's user ID and full name, in which case the software must look up the manager's entry to resolve the mapping for the manager portion of the JSON resource, in addition to looking up the user's entry. As another example, suppose a large group is represented in LDAP as a set of 100,000 DNs. If the JSON resource is configured so that a member is represented by its name, then listing that resource would involve 100,000 LDAP searches to translate DNs to names.

A primary distinction between LDAP entries and JSON resources is that LDAP entries hold sets of attributes and their values, whereas JSON resources are documents containing arbitrarily nested objects. As LDAP data is governed by schema, almost no LDAP objects are arbitrary collections of data. (LDAP has the object class extensibleObject, but its use should be the exception rather than the rule.) Furthermore, JSON resources can hold arrays, ordered collections that can contain duplicates, whereas LDAP attributes are sets, unordered collections without duplicates. For most directory and identity data, these distinctions do not matter. You are likely to run into them, however, if you try to turn your directory into a document store for arbitrary JSON resources.

Despite some extra cost in terms of system resources, exposing directory data over HTTP can unlock directory services for a new generation of applications. The configuration provides flexible mapping, so that you can configure views that correspond to how client applications need to see directory data.

DS software also give you a deployment choice for HTTP access. You can deploy the REST to LDAP gateway, which is a Servlet that connects to any standard LDAPv3 directory, or you can activate the HTTP connection handler on a server to allow direct and more efficient HTTP and HTTPS access.

For examples showing how to use RESTful access, see "Performing RESTful Operations" in the Developer's Guide.

This chapter is meant to serve as an introduction, and so does not even cover everything in this guide, let alone everything you might want to know about directory services.

When you have understood enough of the concepts to build the directory services that you want to deploy, you must still build a prototype and test it before you roll out shared, centralized services for your organization. Read "Tuning Servers For Performance" for a look at how to meet the service levels that directory clients expect.

This section covers the tools installed with server software.

Before you try the examples in this guide, set your PATH to include the DS server tools. The location of the tools depends on the operating environment and on the packages used to install server software. "Paths To Administration Tools" indicates where to find the tools.

You find the installation and upgrade tools, setup, and upgrade, in the parent directory of the other tools, as these tools are not used for everyday administration. For example, if the path to most tools is /path/to/opendj/bin you can find these tools in /path/to/opendj. For instructions on how to use the installation and upgrade tools, see the Installation Guide.

All DS command-line tools take the --help option.

All commands call Java programs and therefore involve starting a JVM.

"Tools and Server Constraints" indicates the constraints, if any, that apply when using a command-line tool with a server.

This section describes how DS command-line tools determine whether to trust server certificates.

When DS command-line tools connect securely to a server, the server presents its digital certificate. The tool must then determine whether to trust the server certificate and continue negotiating the secure connection, or not to trust the server certificate and drop the connection.

An important part of trusting a server certificate is trusting the signing certificate of the party who signed the server certificate. The process is described in more detail in "About Certificates, Private Keys, and Secret Keys" in the Security Guide.

Put simply, a tool can automatically trust the server certificate if the tool's truststore contains the signing certificate. "Command-Line Tools and Truststores" indicates where to find the truststore. The signing certificate could be a CA certificate, or the server certificate itself if the certificate was self-signed.

When run in interactive mode, DS command-line tools can prompt you to decide whether to trust a server certificate not found in the truststore. If you have not specified a truststore and you choose to trust the certificate permanently, the tools store the certificate in a file. The file is user.home/.opendj/keystore, where user.home is the Java system property. user.home is $HOME on Linux and UNIX, and %USERPROFILE% on Windows. The keystore password is OpenDJ. Neither the file name nor the password can be changed.

When run in non-interactive mode, the tools either rely on the specified truststore, or use this default truststore if none is specified.

Property value substitution uses configuration expressions to introduce variables into the server configuration. You set configuration expressions as the values of configuration properties. The effective property values can be evaluated in a number of ways. For more information about property evaluation, see "Expression Evaluation and Order of Precedence".

$ export SSL_CIPHER_SUITES=\
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,\
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,\
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,\
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,\
TLS_EMPTY_RENEGOTIATION_INFO_SCSV
$ stop-ds --quiet
$ dsconfig \
 set-connection-handler-prop \
 --offline \
 --handler-name LDAP \
 --add ssl-protocol:TLSv1.2 \
 --add ssl-cipher-suite:'&{ssl.cipher.suites}' \
 --no-prompt
$ start-ds --quiet
  

ssl-cipher-suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
ssl-cipher-suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
ssl-cipher-suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
ssl-cipher-suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
ssl-cipher-suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV

$ dsconfig \
 create-debug-target \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --publisher-name "File-Based Debug Logger" \
 --type generic \
 --target-name org.forgerock.config.resolvers \
 --set enabled:true \
 --trustAll \
 --no-prompt
$ dsconfig \
 set-log-publisher-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --publisher-name "File-Based Debug Logger" \
 --set enabled:true \
 --trustAll \
 --no-prompt
$ stop-ds --restart --quiet
  

By default, DS servers save a compressed version of the server configuration used on successful startup. This ensures that the server provides a last known good configuration, which can be used as a reference or copied into the active configuration if the server fails to start with the current active configuration. It is possible, though not usually recommended, to turn this behavior off by changing the global server setting save-config-on-successful-startup to false.

Although DS servers are designed to recover from failure and disorderly shutdown, it is safer to shut the server down cleanly, because a clean shutdown reduces startup delays during which the server attempts to recover database backend state, and prevents situations where the server cannot recover automatically.

Follow these steps to shut down the DS server cleanly:

The following server administration commands can be run in online and offline mode. They invoke data-intensive operations, and so potentially take a long time to complete. The links below are to the reference documentation for each command:

When you run these commands in online mode, they run as tasks on the server. Server tasks are scheduled operations that can run one or more times as long as the server is up. For example, you can schedule the backup and export-ldif commands to run recurrently in order to back up server data on a regular basis. See the command's reference documentation for details.

You schedule a task as a directory administrator, sending the request to the administration port. You can therefore schedule a task on a remote server if you choose. When you schedule a task on a server, the command returns immediately, yet the task can start later, and might run for a long time before it completes. You can access tasks by using the "manage-tasks — manage server administration tasks" in the Reference command.

Although you can schedule a server task on a remote server, the data for the task must be accessible to the server locally. For example, when you schedule a backup task on a remote server, that server writes backup files to a file system on the remote server. Similarly, when you schedule a restore task on a remote server, that server restores backup files from a file system on the remote server.

DS servers tend to show resilience when restarting after a crash or after the server process is killed abruptly. After disorderly shutdown, the DS server might have to replay the last few entries in a transaction log. Generally, DS servers return to service quickly.

Database recovery messages are found in the database log file, such as /path/to/opendj/db/userRoot/dj.log.

The following example shows two example messages from the recovery log. The first message is written at the beginning of the recovery process. The second message is written at the end of the process:

[/path/to/opendj/db/userRoot]Recovery underway, found end of log
...
[/path/to/opendj/db/userRoot]Recovery finished: Recovery Info ...

Directory data cached in memory is lost during a crash. Loading directory data into memory when the server starts can take time. DS servers start accepting client requests before this process is complete.

When you install DS directory servers, you have the option of importing sample data that is generated during the installation. This procedure demonstrates how to generate LDIF by using the makeldif command, described in "makeldif — generate test LDIF" in the Reference.

The following procedures demonstrate how to use the import-ldif and export-ldif commands, described in "import-ldif — import directory data from LDIF" in the Reference and "export-ldif — export directory data in LDIF" in the Reference.

DS directory server stores data in a backend. A backend is a private server repository that can be implemented in memory, as a file, or as an embedded database.

Database backends are designed to hold large amounts of user data. DS servers have tools for backing up and restoring database backends, as described in "Backing Up and Restoring Data". By default, a directory server stores your data in a database backend named userRoot, unless you use a setup profile.

When installing the server and importing user data, and when creating a database backend, you choose the backend type. DS directory servers use JE backends for local data.

The JE backend type is implemented using B-tree data structures. It stores data as key-value pairs, which is different from the relational model exposed to clients of relational databases.

A JE backend stores data on disk using append-only log files with names like number.jdb. The JE backend writes updates to the highest-numbered log file. The log files grow until they reach a specified size (default: 1 GB). When the current log file reaches the specified size, the JE backend creates a new log file.

To avoid an endless increase in database size on disk, JE backends clean their log files in the background. A cleaner thread copies active records to new log files. Log files that no longer contain active records are deleted.

By default, JE backends let the operating system potentially cache data for a period of time before flushing the data to disk. This setting trades full durability with higher disk I/O for good performance with lower disk I/O. With this setting, it is possible to lose the most recent updates that were not yet written to disk in the event of an underlying OS or hardware failure. You can modify this behavior by changing the advanced configuration settings for the JE backend. If necessary, you can change the advanced setting for durability, db-durability, using the dsconfig set-backend-prop command.

When a JE backend is opened, it recovers by recreating its B-tree structure from its log files. This is a normal process, one that allows the backend to recover after an orderly shutdown or after a crash.

Due to the cleanup processes, JE backends can be actively writing to disk even when there are no pending client or replication operations. To back up a server using a file system snapshot, you must stop the server before taking the snapshot.

In addition to the cleanup process, JE backends run checksum verification periodically on the database logs. If the verification process detects backend database corruption, then the server logs an error message and the backend is taken offline. In this case, restore the corrupted backend from backup so that it can be used again. By default, the verification runs every night at midnight local time. If necessary, you can change this behavior by adjusting the advanced settings, db-run-log-verifier, and db-log-verifier-schedule, using the dsconfig set-backend-prop command.

You can create new backends using the dsconfig create-backend command, described in "create-backend" in the Configuration Reference. When you create a backend, choose the type of backend that fits your purpose.

The following example creates a database backend named exampleOrgBackend. The backend is of type je, which relies on a JE database for data storage and indexing:

$ dsconfig \
 create-backend \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name exampleOrgBackend \
 --type je \
 --set enabled:true \
 --set base-dn:dc=example,dc=org \
 --trustAll \
 --no-prompt
  

After creating the backend, you can view the settings as in the following example:

$ dsconfig \
 get-backend-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name exampleOrgBackend \
 --trustAll \
 --no-prompt
Property                : Value(s)
------------------------:---------------------
backend-id              : exampleOrgBackend
base-dn                 : "dc=example,dc=org"
cipher-key-length       : 128
cipher-transformation   : AES/CBC/PKCS5Padding
compact-encoding        : true
confidentiality-enabled : false
db-cache-percent        : 50
db-cache-size           : 0 b
db-directory            : db
enabled                 : true
writability-mode        : enabled
  

In some cases, such as directory services with subtree replication, you might choose to split directory data across multiple backends.

In the example in this section, the exampleData backend holds all data under dc=example,dc=com except ou=People,dc=example,dc=com. A separate peopleData backend holds data under ou=People,dc=example,dc=com. Replication for these backends is configured separately as described in "Subtree Replication".

The following example assumes you perform the steps when initially setting up the directory data. It uses the sample data from "Generating Test Data":

# Create a backend for the data not in the sub-suffix:
$ dsconfig \
 create-backend \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name exampleData \
 --type je \
 --set enabled:true \
 --set base-dn:dc=example,dc=com \
 --trustAll \
 --no-prompt

# Create a backend for the data in the sub-suffix:
$ dsconfig \
 create-backend \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name peopleData \
 --type je \
 --set enabled:true \
 --set base-dn:ou=People,dc=example,dc=com \
 --trustAll \
 --no-prompt

# Import data not in the sub-suffix:
$ import-ldif \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backendID exampleData \
 --excludeBranch ou=People,dc=example,dc=com \
 --ldifFile generated.ldif \
 --trustAll

# Import data in the sub-suffix:
$ import-ldif \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backendID peopleData \
 --includeBranch ou=People,dc=example,dc=com \
 --ldifFile generated.ldif \
 --trustAll
  

When you split an existing backend, you must handle the service interruption that results when you move the sub-suffix data out of the original backend. If there is not a maintenance window where you can bring the service down in order to move the data, consider alternatives for replaying changes applied between the time that you exported the data and the time that you retired the old sub-suffix.

DS directory servers can encrypt directory data before storing it in a database backend on disk, keeping the data confidential until it is accessed by a directory client.

As explained in "Protect DS Server Files", DS directory servers do not encrypt directory data by default. This means that any user with system access to read directory files can potentially access directory data in cleartext.

You can verify what a system user could see with read access to backend database files by using the backendstat dump-raw-db command. The backendstat subcommands list-raw-dbs and dump-raw-db help you list and view the low-level databases within a backend. Unlike the output of other subcommands, the output of the dump-raw-db subcommand is neither decrypted nor formatted for readability. Instead, you can see values as they are stored in the backend file.

In a backend database, the id2entry index holds LDIF representations of directory entries. For a database that is not encrypted, the corresponding low-level database shows the cleartext strings, as is evident in the following example:

$ stop-ds --quiet
$ backendstat list-raw-dbs --backendId dsEvaluation
...
/dc=com,dc=example/id2entry...
...
$ backendstat \
 dump-raw-db \
 --backendId dsEvaluation \
 --dbName /dc=com,dc=example/id2entry
...
Key (len 8):
 00 00 00 00 00 00 00 1E                          ........
 Value (len 437):
 02 00 81 B1 03 01 06 27 75 69 64 3D 62 6A 65 6E  .......'uid=bjen
 73 65 6E 2C 6F 75 3D 50 65 6F 70 6C 65 2C 64 63  sen,ou=People,dc
 3D 65 78 61 6D 70 6C 65 2C 64 63 3D 63 6F 6D 01  =example,dc=com.
 06 11 01 08 01 13 62 6A 65 6E 73 65 6E 40 65 78  ......bjensen@ex
 61 6D 70 6C 65 2E 63 6F 6D 01 09 01 04 30 32 30  ample.com....020
 39 01 16 01 0C 65 6E 2C 20 6B 6F 3B 71 3D 30 2E  9....en, ko;q=0.
 38 01 10 01 29 75 69 64 3D 74 72 69 67 64 65 6E  8...)uid=trigden
 2C 20 6F 75 3D 50 65 6F 70 6C 65 2C 20 64 63 3D  , ou=People, dc=
 65 78 61 6D 70 6C 65 2C 64 63 3D 63 6F 6D 01 04  example,dc=com..
 02 13 50 72 6F 64 75 63 74 20 44 65 76 65 6C 6F  ..Product Develo
 70 6D 65 6E 74 06 50 65 6F 70 6C 65 01 0B 01 07  pment.People....
 42 61 72 62 61 72 61 01 0C 01 0F 2B 31 20 34 30  Barbara....+1 40
 38 20 35 35 35 20 31 38 36 32 01 0D 01 06 4A 65  8 555 1862....Je
 6E 73 65 6E 01 07 02 0E 42 61 72 62 61 72 61 20  nsen....Barbara
 4A 65 6E 73 65 6E 0B 42 61 62 73 20 4A 65 6E 73  Jensen.Babs Jens
 65 6E 01 0E 01 0D 2F 68 6F 6D 65 2F 62 6A 65 6E  en..../home/bjen
 73 65 6E 01 0F 01 0F 2B 31 20 34 30 38 20 35 35  sen....+1 408 55
 35 20 31 39 39 32 01 11 01 04 31 30 30 30 01 12  5 1992....1000..
 01 2E 7B 53 53 48 41 7D 33 45 66 54 62 33 70 37  ..{SSHA}3EfTb3p7
 71 75 6F 75 73 4B 35 34 2B 41 4F 34 71 44 57 6C  quousK54+AO4qDWl
 56 33 4F 39 54 58 48 57 49 4A 49 32 4E 41 3D 3D  V3O9TXHWIJI2NA==
 01 13 01 04 31 30 37 36 01 05 01 14 4F 72 69 67  ....1076....Orig
 69 6E 61 6C 20 64 65 73 63 72 69 70 74 69 6F 6E  inal description
 01 14 01 07 62 6A 65 6E 73 65 6E 01 15 01 0D 53  ....bjensen....S
 61 6E 20 46 72 61 6E 63 69 73 63 6F 01 01 02 01  an Francisco....
 24 38 38 37 37 33 32 65 38 2D 33 64 62 32 2D 33  $887732e8-3db2-3
 31 62 62 2D 62 33 32 39 2D 32 30 63 64 36 66 63  1bb-b329-20cd6fc
 65 63 63 30 35                                   ecc05
...
 

To maintain data confidentiality on disk, you must configure it explicitly. In addition to preventing read access by other users as described in "Setting Up a System Account for a Server", you can configure confidentiality for database backends. When confidentiality is enabled for a backend, the directory server encrypts entries before storing them in the backend.

Enable backend confidentiality with the default encryption settings as shown in the following example:

$ dsconfig \
 set-backend-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name dsEvaluation \
 --set confidentiality-enabled:true \
 --no-prompt \
 --trustAll
 

After confidentiality is enabled, entries are encrypted when next written. That is, the DS directory server does not automatically rewrite all entries in encrypted form. Instead, it encrypts each entry on update, for example, when a user updates their entry or when you import data.

If you import the data again after enabling confidentiality, you can see with the backendstat dump-raw-db command that the low-level database for id2index no longer contains cleartext:

$ stop-ds --quiet
$ [../resources/tests/admin-guide/import_export.bash:#export-backup-ldif]
$ [../resources/tests/admin-guide/import_export.bash:#import-backup-ldif]
$ backendstat \
 dump-raw-db \
 --backendId dsEvaluation \
 --dbName /dc=com,dc=example/id2entry
...
Key (len 8):
 00 00 00 00 00 00 00 C2                          ........
 Value (len 437):
 02 02 81 82 01 C4 95 87 5B A5 2E 47 97 80 23 F4  ........[..G..#.
 CE 5D 93 25 97 D4 13 F9 0A A3 A8 31 9A D9 7A 70  .].%.......1..zp
 FE 3E AC 9D 64 41 EB 7B D5 7F 7E B8 B7 74 52 B8  .>..dA.{.^?~..tR.
 C7 7F C8 79 19 46 7D C5 5D 5B 83 9C 5B 9F 85 28  .^?.y.F}.][..[..(
 83 A2 5F A0 C1 B1 09 FC 2F E3 D8 82 4A AA 8B D9  .._...../...J...
 78 43 34 50 AE A1 52 88 5B 70 97 D2 E1 EA 87 CA  xC4P..R.[p......
 3B 4D 07 DC F9 F8 30 BB D2 76 51 C8 75 FF FA 80  ;M....0..vQ.u...
 77 E1 6A 8B 5B 8F DA A4 F4 0B B5 20 56 B3 19 19  w.j.[...... V...
 22 D8 9D 38 04 E3 4D 94 A7 99 4B 81 16 AD 88 46  "..8..M...K....F
 FC 3F 7E 78 66 B8 D1 E9 86 A0 F3 AC B6 68 0D A9  .?~xf........h..
 9A A7 3C 30 40 37 97 4E 90 DD 63 16 8E 11 0F 5E  ..<0@7.N..c....^
 9D 5B 86 90 AF 4E E2 1F 9E 70 73 14 0A 11 5C DB  .[...N...ps...\.
 B7 BC B8 A9 31 3F 74 8D 0A 9F F4 6C E1 B0 36 78  ....1?t....l..6x
 F0 5A 5E CD 7C B3 A2 36 66 8E 88 86 A0 8B 9A 77  .Z^.|..6f......w
 D5 CD 7E 9C 4E 62 20 0E D0 DB AD E7 7E 99 46 4F  ..~.Nb .....~.FO
 67 C7 A6 7E 2C 24 82 50 51 9F A7 B2 02 44 5B 30  g..~,$.PQ....D[0
 74 41 99 D9 83 69 EF AE 2E C0 FF C4 E6 4F F2 2F  tA...i.......O./
 95 FB 93 65 30 2A 2D 8D 20 88 83 B5 DE 35 B6 20  ...e0*-. ....5.
 47 17 30 25 60 FD E3 43 B9 D6 A4 F7 47 B6 6C 9F  G.0%`..C....G.l.
 47 FD 63 8E 7F A5 00 CE 6C 3E BC 95 23 69 ED D0  G.c.^?...l>..#i..
 69 4F BE 61 BD 30 C2 40 66 F6 F9 C3 3E C1 D7 8C  iO.a.0.@f...>...
 B0 C8 4A 2E 27 BE 13 6C 40 88 B0 13 A3 12 F4 50  ..J.'..l@......P
 CA 92 D8 EB 4A E5 3F E2 64 A3 76 C7 5C 2B D8 89  ....J.?.d.v.\+..
 A3 6E C1 F7 0A C2 37 7A BD AF 14 4B 52 04 6B F2  .n....7z...KR.k.
 8F 4F C3 F8 00 90 BA 0F EC 6D B1 2D A8 18 0C A6  .O.......m.-....
 29 96 82 3B 5C BC D0 F4 2B BE 9C C5 8B 18 7A DE  )..;\...+.....z.
 C7 B5 10 2D 45 50 4F 77 ED F7 23 34 95 AF C3 2E  ...-EPOw..#4....
 B0 9B FA E9 DF                                   .....
...
  

Similar checks can be run on other low-level databases if you enable confidentiality by backend indexes as described below.

The default settings for confidentiality are cipher-transformation: AES/CBC/PKCS5Padding and cipher-key-length: 128. This means the algorithm is the Advanced Encryption Standard (AES), the cipher mode is Cipher Block Chaining (CBC), and the padding is PKCS#5 padding as described in RFC 2898: PKCS #5: Password-Based Cryptography Specification. The syntax for the cipher-transformation is algorithm/mode/padding, and all three must be specified. When the algorithm does not require a mode, use NONE. When the algorithm does not require padding, use NoPadding. Use of larger cipher-key-length values can require that you install JCE policy files such as those for unlimited strength, as described in "Using Unlimited Strength Cryptography" in the Security Guide.

DS servers encrypt data using a symmetric key that is stored with the server configuration. The symmetric key is encrypted in turn with the server's public key that is also stored with the server configuration. When multiple servers are configured to replicate data as described in "Configuring Replication Settings", the servers replicate the keys as well, allowing any server replica to decrypt the data.

In addition to entry encryption, you can enable confidentiality by backend index, as long as confidentiality is enabled for the backend itself. Confidentiality hashes keys for equality type indexes using SHA-1, and encrypts the list of entries matching a substring key for substring indexes. The following example shows how to enable confidentiality for the mail index:

$ dsconfig \
 set-backend-index-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name dsEvaluation \
 --index-name mail \
 --set confidentiality-enabled:true \
 --no-prompt \
 --trustAll
  

After changing the index configuration, you can rebuild the index to enforce confidentiality immediately. For details, see "Configuring and Rebuilding Indexes".

Avoid using sensitive attributes in VLV indexes. Confidentiality cannot be enabled for VLV indexes.

Encrypting and decrypting data comes with costs in terms of cryptographic processing that reduces throughput and of extra space for larger encrypted values. In general, tests with default settings show that the cost of enabling confidentiality can be quite modest, but your results can vary based on your systems and on the settings used for cipher-transformation and cipher-key-length. Make sure you test your deployment to qualify the impact of confidentiality before enabling it in production.

Directory data growth depends on applications that use the directory. As a result, when directory applications add more data than they delete, the database backend grows until it fills the available disk space. The system can end up in an unrecoverable state if no disk space is available.

Database backends therefore have advanced properties, disk-low-threshold and disk-full-threshold. When available disk space falls below disk-low-threshold, the directory server only allows updates from users and applications that have the bypass-lockdown privilege, as described in "About Privileges". When available space falls below disk-full-threshold, the directory server stops allowing updates, instead returning an UNWILLING_TO_PERFORM error to each update request.

A directory server continues to apply replication updates without regard to the thresholds. The server can therefore fill available disk space despite the thresholds, by accepting replication updates made on other servers. You can give yourself more time to react to the situation both by monitoring directory data growth and also by increasing the thresholds.

If growth across the directory service tends to happen quickly, set the thresholds higher than the defaults to allow more time to react when growth threatens to fill the disk. The following example sets disk-low-threshold to 10 GB disk-full-threshold to 5 GB for the dsEvaluation backend:

$ dsconfig \
 set-backend-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name dsEvaluation \
 --set "disk-low-threshold:10 GB" \
 --set "disk-full-threshold:5 GB" \
 --trustAll \
 --no-prompt
  

The properties disk-low-threshold and disk-full-threshold are listed as advanced properties. To examine their values with the dsconfig command, use the --advanced option as shown in the following example:

$ dsconfig \
 get-backend-prop \
 --advanced \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name dsEvaluation \
 --property disk-low-threshold \
 --property disk-full-threshold \
 --trustAll \
 --no-prompt
Property            : Value(s)
--------------------:---------
disk-full-threshold : 5 gb
disk-low-threshold  : 10 gb
  

If the directory service creates many entries that expire and should be deleted, it is possible to delete the entries by finding them with a time-based search and then deleting them individually. That approach uses replication to delete the entries in all replicas, and works with older versions of the directory service. It has the disadvantage of generating potentially large amounts of replication traffic. With upgraded replicas, you can instead configure the backend database to delete the entries as they expire. This new approach deletes expired entries at the backend database level, without generating replication traffic.

Backend indexes for generalized time (timestamp) attributes have these properties to configure automated, optimized entry expiration and removal:

Once you configure and build the index, the backend can delete expired entries. At intervals of 10 seconds, the backend automatically deletes entries whose timestamps on the attribute are older than the specified lifetime. Entries that expire in the interval between deletions are removed on the next round. Client applications should therefore check that entries have not expired, as it is possible for expired entries to remain available until the next round of deletions.

In addition to letting you create new backends as described in "Creating a New Database Backend", DS directory servers let you add a new base DN to an existing backend.

The following example adds the suffix o=example to the existing backend dsEvaluation:

$ dsconfig \
 set-backend-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name dsEvaluation \
 --add base-dn:o=example \
 --trustAll \
 --no-prompt
$ dsconfig \
 get-backend-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name dsEvaluation \
 --property base-dn \
 --trustAll \
 --no-prompt
Property : Value(s)
---------:-------------------------------
base-dn  : "dc=example,dc=com", o=example
  

You delete a database backend by using the dsconfig delete-backend command, described in "delete-backend" in the Configuration Reference.

When you delete a database backend by using the dsconfig delete-backend command, the directory server does not actually remove the database files for two reasons:

Instead, after you run the dsconfig delete-backend command you must also manually remove the database backend files.

If you do run the dsconfig delete-backend command by mistake and have not yet deleted the actual files, then you can recover from the mistake by creating the backend again, reconfiguring the indexes that were removed, and rebuilding the indexes as described in "Configuring and Rebuilding Indexes".

You configure LDAP client access using the dsconfig command.

The reserved port number for LDAP is 389. Most examples in the documentation use 1389, which is accessible to non-privileged users.

This section shows how to prepare the server to use a file-based keystore to manage the keys essential to secure communications. For more information about the keys, see "About Certificates, Private Keys, and Secret Keys" in the Security Guide.

For instructions on importing trusted certificates on PKCS#11 devices, see the documentation for the device.

StartTLS negotiations start on the unsecure LDAP port, and then protect communication with the client. You can configure StartTLS when setting up a server, or later using the dsconfig command.

An LDAP connection handler configured to use LDAPS (LDAP/SSL) allows only secure connections from client applications. You can configure LDAPS when setting up a server, or later using the dsconfig command.

The reserved port number for LDAPS is 636. Most examples in the documentation use 1636, which is accessible to non-privileged users.

Using a server's global configuration properties, you can add global restrictions on how clients access the server. These settings are per server, and so must be set independently on each server.

These global settings are fairly coarse-grained. For a full discussion of the rich set of administrative privileges and fine-grained access control instructions that DS servers support, see "Configuring Privileges and Access Control".

When a server and client negotiate a secure connection, they negotiate use of a common protocol and cipher suite. If they cannot negotiate a common protocol and cipher suite, they will fail to set up a secure connection.

Furthermore, researchers continue to find vulnerabilities in protocols and cipher suites. If the server continues to support those protocols and cipher suites, then clients can use them when negotiating connections. Attackers can then exploit the vulnerabilities. It is therefore important to understand how to read and to restrict the list of supported protocols and cipher suites.

DS servers depend on the underlying JVM to support security protocols and cipher suites. By default, all supported protocols and cipher suites are available. For details, see the documentation for the JVM. For Oracle Java, see the Java Cryptography Architecture Oracle Providers Documentation describing the The SunJSSE Provider.

Bear in mind that support for protocols and cipher suites can be added and removed in Java update releases. The protocols and cipher suites available are also determined by the security policy. For example, see "Using Unlimited Strength Cryptography" in the Security Guide.

This section clarifies the roles that client applications' X.509 digital certificates play in establishing secure connections and in authenticating the client as a directory user. Be aware that establishing a secure connection happens before the server handles the LDAP or HTTP requests that the client sends over the secure connection. Establishing a secure connection is handled separately from authenticating a client as a directory user, even though both processes can involve the client's certificate.

When a client and a server negotiate a secure connection over LDAPS or HTTPS, or over LDAP using the StartTLS operation, they can use public key cryptography to authenticate each other. The server, client, or both present certificates to each other. By default, DS LDAPS and HTTPS connection handlers are configured to present the server certificate, and to consider the client certificate optional. The connection handler property ssl-client-auth-policy makes the latter behavior configurable. For the DSML and REST to LDAP gateways, HTTPS negotiation is handled by the web application container where the gateway runs. See the web application container documentation for details on configuring how the container handles the client certificate.

One step toward establishing a secure connection involves validating the certificate that was presented by the other party. Part of this is trusting the certificate. The certificate identifies the client or server and the CA certificate used to sign the client or server certificate. The validating party checks that the other party corresponds to the one identified by the certificate, and checks that the signature can be trusted. If the signature is valid, and the CA certificate used to sign the certificate can be trusted, then the certificate can be trusted. This part of the validation process is also described briefly in "How Keys are Used" in the Security Guide.

Certificates can be revoked after they are signed. Therefore, the validation process can involve checking whether the certificate is still valid. Two different methods for performing this validation use the Online Certificate Status Protocol (OCSP) or Certificate Revocation Lists (CRLs). OCSP is a newer solution that provides an online service to handle the revocation check for a specific certificate. CRLs are potentially large lists of user certificates that are no longer valid or that are on hold. A CRL is signed by the CA. The validating party obtains the CRL and checks that the certificate being validated is not listed. For a brief comparison, see OCSP: Comparison to CRLs. A certificate can include links to contact the OCSP responder or to the CRL distribution point. The validating party can use these links to check whether the the certificate is still valid.

In both cases, the CA who signed the certificate acts as the OCSP responder or publishes the CRLs. When establishing a secure connection with a client application, the server relies on the CA for OCSP and CRLs. This is the case even when the DS server is the repository for the CRLs.

DS directory services are logical repositories for certificates and CRLs. For example, your DS directory service can store CRLs in a certificateRevocationList attribute as in the following example entry:

dn: cn=My CA,dc=example,dc=com
objectClass: top
objectClass: applicationProcess
objectClass: certificationAuthority
cn: My CA
authorityRevocationList;binary: Base64-encoded ARL
cACertificate;binary:: Base64-encoded CA certificate
certificateRevocationList;binary:: Base64-encoded CRL
 

The CRL could then be replicated to other DS directory servers for high availability. (Notice the ARL in this entry. An ARL is like a CRL, but for CA certificates.)

Again, despite being a repository for CRLs, the DS directory service does not use the CRLs directly when checking a client certificate. Instead, when negotiating a secure connection, the server depends on the JVM security configuration. The JVM configuration governs whether validation uses OCSP, CRLs, or both. As described in the Java PKI Programmer's Guide under Support for the CRL Distribution Points Extension, and Appendix C: On-Line Certificate Status Protocol (OCSP) Support, the JVM relies on system properties that define whether to use the CRL distribution points defined in certificates, and how to handle OCSP requests. These system properties can be set system-wide in $JAVA_HOME/lib/security/java.security ($JAVA_HOME/jre/lib/security/java.security for JDK 8). The JVM handles revocation checking without the DS server's involvement.

After a connection is negotiated, the server can authenticate a client application at the LDAP level based on the certificate. For details, see "Authenticating Client Applications With a Certificate" in the Security Guide.

OCSP and obtaining CRLs depend on network access to the CA. If DS servers or the DSML or REST to LDAP gateways run on a network where the CA is not accessible, and the deployment nevertheless requires OSCP or checking CRLs for client application certificates, then you must provide some alternative means to handle OCSP or CRL requests. The JVM can be configured to use a locally available OCSP responder, for example, and that OCSP responder might depend on DS servers. If the solution depends on CRLs, you could regularly update the CRLs in the directory with copies of the CA CRLs obtained by other means.

DS servers also expose administrative data over HTTP. For details, see "To Set Up REST Access to Administrative Data".

Directory Services Markup Language (DSML) client access is implemented as a servlet that runs in a web application container.

You configure DSML client access by editing the WEB-INF/web.xml after you deploy the web application. In particular, you must at least set the ldap.host and ldap.port parameters if they differ from the default values, which are localhost and 389.

The DSML servlet translates between DSML and LDAP, and passes requests to the directory service. For initial testing purposes, you might try JXplorer, where DSML Service: /webapp-dir/DSMLServlet. Here, webapp-dir refers to the name of the directory in which you unpacked the DSML .war. "JXplorer Accessing the Directory Service" shows the result.

JXplorer Accessing the Directory Service

You configure Java Management Extensions (JMX) client access by using the command-line tool, dsconfig.

The LDIF connection handler lets you make changes to directory data by placing LDIF files in a file system directory that the DS server regularly polls for changes. The LDIF, once consumed, is deleted.

For instructions on setting up the SNMP connection handler, see "SNMP-Based Monitoring".

Privileges provide access control for server administration independently from ACIs.

DS server ACIs exist as operational aci attribute values on directory entries, and as global ACI attributes stored in the configuration. ACIs apply to a scope defined in the instruction. They set permissions that depend on what operation is requested, who requested the operation, and how the client connected to the server.

For example, the ACIs on the following entry allow anonymous read access to all attributes except passwords, and allow read-write access for directory administrators under dc=example,dc=com:

dn: dc=example,dc=com
objectClass: domain
objectClass: top
dc: example
aci: (target ="ldap:///dc=example,dc=com")
 (targetattr != "userPassword")(version 3.0;acl "Anonymous read-search access";
 allow(read, search, compare)(userdn = "ldap:///anyone");)
aci: (target="ldap:///dc=example,dc=com")
 (targetattr = "*")(version 3.0; acl "allow all Admin group";
 allow(all) groupdn = "ldap:///cn=Directory Administrators,ou=Groups,dc=example,dc=com";)
 

The DS server default behavior is that no access is allowed unless it is specifically granted by an ACI. (The bypass-acl privilege, assigned to certain users such as cn=Directory Manager, can allow users to bypass access control checks.)

Global ACIs are defined on the access control handler, and apply to the entire server. You must adjust the default global ACIs to match the security policies for your organization, for example, to restrict anonymous access.

ACI attribute values use a specific language described in this section. Although ACI attribute values can become difficult to read in LDIF, the basic syntax is simple:

targets(version 3.0;acl "name";permissions subjects;)

Separate multiple pairs of permissions-subjects definitions with semicolons, ;. When you specify multiple permissions-subjects pairs, at least one must match (OR).

allow(action[, action …])

deny(action[, action …])

ACIs are defined in the data on aci attributes. They can be imported in LDIF and modified over LDAP. In order to make changes to ACIs, however, users first need the modify-acl privilege. By default, only the directory superuser has the modify-acl privilege.

Global ACIs on cn=Access Control Handler,cn=config can be set using the dsconfig command. Global ACIs have attribute type ds-cfg-global-aci.

For details, see "Default Global ACIs".

Users with write access to add ACIs and with the modify-acl privilege can use the ldapmodify command to change ACIs located in user data.

This section focuses on ACI examples, rather than demonstrating how to update the directory for each example. To update ACIs, use the ldapmodify command.

For hints on updating directory entries with the ldapmodify command, see "Modifying Entry Attributes" in the Developer's Guide. Keep in mind that the name of the ACI attribute is aci as shown in the examples that follow.

aci: (target ="ldap:///dc=example,dc=com")
 (targetattr != "authPassword || userPassword")
 (version 3.0;acl "Anonymous read-search access";
 allow (read, search, compare)(userdn = "ldap:///anyone");)

$ dsconfig \
 get-access-control-handler-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --property global-aci \
 --trustAll
  

#
# Edit Access Control Handler global-aci attributes, replacing
# userdn="ldap:///anyone" (anonymous) with userdn="ldap:///all" (authenticated)
# in "Anonymous read access" and "User-Visible Operational Attributes" ACIs.
#
# To make this change, you remove existing values, and add edited values:
#
$ dsconfig \
 set-access-control-handler-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --remove global-aci:\(targetattr!=\"userPassword\|\|authPassword\|\|debugsearchindex\|\|changes\|\|changeNumber\|\|changeType\|\|changeTime\|\|targetDN\|\|newRDN\|\|newSuperior\|\|deleteOldRDN\"\)\(version\ 3.0\;\ acl\ \"Anonymous\ read\ access\"\;\ allow\ \(read,search,compare\)\ userdn=\"ldap:///anyone\"\;\) \
 --remove global-aci:\(targetattr=\"createTimestamp\|\|creatorsName\|\|modifiersName\|\|modifyTimestamp\|\|entryDN\|\|entryUUID\|\|subschemaSubentry\|\|etag\|\|governingStructureRule\|\|structuralObjectClass\|\|hasSubordinates\|\|numSubordinates\|\|isMemberOf\|\|alive\|\|healthy\"\)\(version\ 3.0\;\ acl\ \"User-Visible\ Operational\ Attributes\"\;\ allow\ \(read,search,compare\)\ userdn=\"ldap:///anyone\"\;\) \
 --add global-aci:\(targetattr!=\"userPassword\|\|authPassword\|\|debugsearchindex\|\|changes\|\|changeNumber\|\|changeType\|\|changeTime\|\|targetDN\|\|newRDN\|\|newSuperior\|\|deleteOldRDN\"\)\(version\ 3.0\;\ acl\ \"Anonymous\ read\ access\"\;\ allow\ \(read,search,compare\)\ userdn=\"ldap:///all\"\;\) \
 --add global-aci:\(targetattr=\"createTimestamp\|\|creatorsName\|\|modifiersName\|\|modifyTimestamp\|\|entryDN\|\|entryUUID\|\|subschemaSubentry\|\|etag\|\|governingStructureRule\|\|structuralObjectClass\|\|hasSubordinates\|\|numSubordinates\|\|isMemberOf\|\|alive\|\|healthy\"\)\(version\ 3.0\;\ acl\ \"User-Visible\ Operational\ Attributes\"\;\ allow\ \(read,search,compare\)\ userdn=\"ldap:///all\"\;\) \
 --trustAll \
 --no-prompt
  

$ cat remove-anon.ldif
dn: dc=example,dc=com
changetype: modify
delete: aci
aci: (target ="ldap:///dc=example,dc=com")(targetattr != "userPassword")(version 3.0;acl "Anonymous read-search access";allow (read, search, compare)(userdn = "ldap:///anyone");)
-
add: aci
aci: (target ="ldap:///dc=example,dc=com")(targetattr != "userPassword")(version 3.0;acl "Anonymous read-search access";allow (read, search, compare)(userdn = "ldap:///all");)

$ ldapmodify \
 --port 1389 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 remove-anon.ldif
  

$ ldapsearch --port 1389 --baseDN dc=example,dc=com "(uid=bjensen)"
$ ldapsearch  \
 --port 1389 \
 --bindDN uid=bjensen,ou=people,dc=example,dc=com \
 --bindPassword hifalutin \
 --baseDN dc=example,dc=com \
 "(uid=bjensen)" cn uid
dn: uid=bjensen,ou=People,dc=example,dc=com
cn: Barbara Jensen
cn: Babs Jensen
uid: bjensen
  

$ dsconfig \
 set-global-configuration-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --set unauthenticated-requests-policy:allow-discovery \
 --trustAll \
 --no-prompt
  

$ ldapsearch --port 1389 --baseDN dc=example,dc=com "(uid=bjensen)"
# The LDAP search request failed: 53 (Unwilling to Perform)
# Additional Information:  Rejecting the requested operation because the connection has not been authenticated
  

aci: (target="ldap:///dc=example,dc=com")
 (targetattr = "* || +")
 (version 3.0;acl "Admins can run amok";
  allow(all, proxy, import, export)
  groupdn = "ldap:///cn=Directory Administrators,ou=Groups,dc=example,dc=com";)

aci: (target ="ldap:///ou=People,dc=example,dc=com")
 (targetattr = "authPassword || userPassword")
 (version 3.0;acl "Allow users to change pass words";
   allow (write)(userdn = "ldap:///self");)
  

aci: (target ="ldap:///ou=Self Service,ou=Groups,dc=example,dc=com")
 (targetattr = "member")
 (version 3.0;acl "Self registration"; allow(selfwrite)
 (userdn = "ldap:///uid=*,ou=People,dc=example,dc=com");)
  

aci: (target ="ldap:///ou=Self Service,ou=Groups,dc=example,dc=com")
 (targattrfilters="add=objectClass:(objectClass=groupOfNames)")
 (version 3.0; acl "All can create self service groups";
   allow (add)(userdn= "ldap:///uid=*,ou=People,dc=example,dc=com");)
aci: (target ="ldap:///ou=Self Service,ou=Groups,dc=example,dc=com")
 (version 3.0; acl "Owner can delete self service groups";
   allow (delete)(userattr= "owner#USERDN");)
  

aci: (target = "ldap:///dc=example,dc=com")
 (targetattr = "*")
 (version 3.0;acl "Use loopback only for LDAP in the clear";
   deny (all)(ip != "127.0.0.1" and ssf <= "1");)
  

aci: (target = "ldap:///dc=example,dc=com")
 (targetattr = "aci") (version 3.0;acl "Modify ACIs"; allow (all)
 (groupdn = "ldap:///cn=Directory Administrators,ou=Groups,dc=example,dc=com";)
  

Global access control policies are similar to ACIs, but are implemented as entries in the server configuration, rather than attributes on entries. They are managed using the dsconfig command.

Unlike ACIs, policies can only allow access, not deny it. This constraint makes them easier to read and to change in isolation. Policies are, however, applied in addition to ACIs and privileges. It is still possible that an ACI could deny something allowed by a policy.

By default, no access is allowed until permitted by a policy or other access control. By default, a policy matches all entries, all types of connection, and all users. You set the properties of the policy to restrict its scope of application. Policies can have the settings to allow the following:

For details, see "Global Access Control Policy" in the Configuration Reference.

You manage global access control policies with the dsconfig command. As the policies are part of the server configuration, they are not replicated, and must be configured on each server.

$ dsconfig \
 delete-global-access-control-policy \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --policy-name "Anonymous access all entries" \
 --trustAll \
 --no-prompt
$ dsconfig \
 delete-global-access-control-policy \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --policy-name "Authenticated access all entries" \
 --trustAll \
 --no-prompt
$ dsconfig \
 create-global-access-control-policy \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --policy-name "Authenticated access all entries" \
 --set authentication-required:true \
 --set request-target-dn-not-equal-to:"**,cn=changelog" \
 --set permission:read \
 --set allowed-attribute:"*" \
 --set allowed-attribute:createTimestamp \
 --set allowed-attribute:creatorsName \
 --set allowed-attribute:entryDN \
 --set allowed-attribute:entryUUID \
 --set allowed-attribute:etag \
 --set allowed-attribute:governingStructureRule \
 --set allowed-attribute:hasSubordinates \
 --set allowed-attribute:isMemberOf \
 --set allowed-attribute:modifiersName \
 --set allowed-attribute:modifyTimestamp \
 --set allowed-attribute:numSubordinates \
 --set allowed-attribute:structuralObjectClass \
 --set allowed-attribute:subschemaSubentry \
 --set allowed-attribute-exception:authPassword \
 --set allowed-attribute-exception:userPassword \
 --set allowed-attribute-exception:debugSearchIndex \
 --set allowed-attribute-exception:@changeLogEntry \
 --set allowed-control:Assertion \
 --set allowed-control:AuthorizationIdentity \
 --set allowed-control:Csn \
 --set allowed-control:ManageDsaIt \
 --set allowed-control:MatchedValues \
 --set allowed-control:Noop \
 --set allowed-control:PasswordPolicy \
 --set allowed-control:PermissiveModify \
 --set allowed-control:PostRead \
 --set allowed-control:PreRead \
 --set allowed-control:ProxiedAuthV2 \
 --set allowed-control:RealAttributesOnly \
 --set allowed-control:ServerSideSort \
 --set allowed-control:SimplePagedResults \
 --set allowed-control:TransactionId \
 --set allowed-control:VirtualAttributesOnly \
 --set allowed-control:Vlv \
 --set allowed-extended-operation:GetSymmetricKey \
 --set allowed-extended-operation:PasswordModify \
 --set allowed-extended-operation:PasswordPolicyState \
 --set allowed-extended-operation:StartTls \
 --set allowed-extended-operation:WhoAmI \
 --trustAll \
 --no-prompt
$ dsconfig \
 create-global-access-control-policy \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --policy-name "Anonymous extended operation access" \
 --set authentication-required:false \
 --set allowed-extended-operation:GetSymmetricKey \
 --set allowed-extended-operation:StartTls \
 --trustAll \
 --no-prompt
  

$ dsconfig \
 create-global-access-control-policy \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --policy-name "Require secure connections for example.com data" \
 --set request-target-dn-equal-to:"**,dc=example,dc=com" \
 --set request-target-dn-equal-to:dc=example,dc=com \
 --set connection-minimum-ssf:128 \
 --trustAll \
 --no-prompt
  

$ dsconfig \
 set-global-access-control-policy-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --policy-name "Anonymous extended operation access" \
 --set connection-client-address-equal-to:.example.com \
 --trustAll \
 --no-prompt
$ dsconfig \
 set-global-access-control-policy-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --policy-name "Root DSE access" \
 --set connection-client-address-equal-to:.example.com \
 --trustAll \
 --no-prompt
  

For additional details, see "Global Access Control Policy" in the Configuration Reference.

As the number of ACIs increases, it can be difficult to understand by inspection what rights a user actually has to a given entry. The following example shows how the Get Effective Rights control can help.

By default, only users such as Directory Manager who can bypass ACIs can use the Get Effective Rights control (OID 1.3.6.1.4.1.42.2.27.9.5.2), and related operational attributes, aclRights and aclRightsInfo. For this example, explicitly grant access to My App using global ACIs:

$ dsconfig \
 set-access-control-handler-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --add global-aci:\(targetcontrol=\"1.3.6.1.4.1.42.2.27.9.5.2\"\)\ \(version\ 3.0\;acl\ \"Allow\ My\ App\ to\ get\ effective\ rights\"\;\ allow\(read\)\ userdn=\"ldap:///cn=My\ App,ou=Apps,dc=example,dc=com\"\;\) \
 --add global-aci:\(targetattr=\"aclRights\|\|aclRightsInfo\"\)\(version\ 3.0\;\ acl\ \"Allow\ My\ App\ to\ read\ effective\ rights\ attributes\"\;\ allow\ \(read,search,compare\)\ userdn=\"ldap:///cn=My\ App,ou=Apps,dc=example,dc=com\"\;\) \
 --trustAll \
 --no-prompt
  

In this example, Babs Jensen is the owner of a small group of people who are willing to carpool:

$ ldapsearch \
 --port 1389 \
 --bindDN "uid=bjensen,ou=people,dc=example,dc=com" \
 --bindPassword hifalutin \
 --baseDN "ou=Self Service,ou=Groups,dc=example,dc=com" \
 "(cn=Carpoolers)"
dn: cn=Carpoolers,ou=Self Service,ou=Groups,dc=example,dc=com
objectClass: top
objectClass: groupOfNames
description: People who are willing to carpool
cn: Carpoolers
owner: uid=bjensen,ou=People,dc=example,dc=com
member: uid=bjensen,ou=People,dc=example,dc=com
  

When My App performs the same search with the get effective rights control, and requests the aclRights attribute, it sees the rights that it has on the entry:

$ ldapsearch \
 --control effectiverights \
 --port 1389 \
 --bindDN "cn=My App,ou=Apps,dc=example,dc=com" \
 --bindPassword password \
 --baseDN "ou=Self Service,ou=Groups,dc=example,dc=com" \
 "(cn=Carpoolers)" \
 aclRights
dn: cn=Carpoolers,ou=Self Service,ou=Groups,dc=example,dc=com
aclRights;entryLevel: add:1,delete:1,read:1,write:1,proxy:1
  

When you request the aclRightsInfo attribute, the server responds with information about the ACIs applied:

$ ldapsearch \
 --control effectiverights \
 --port 1389 \
 --bindDN "cn=My App,ou=Apps,dc=example,dc=com" \
 --bindPassword password \
 --baseDN "ou=Self Service,ou=Groups,dc=example,dc=com" \
 "(cn=Carpoolers)" \
 aclRights \
 aclRightsInfo
dn: cn=Carpoolers,ou=Self Service,ou=Groups,dc=example,dc=com
aclRightsInfo;logs;entryLevel;add: acl_summary(main): access allowed(add) on ent
 ry/attr(cn=Carpoolers,ou=Self Service,ou=Groups,dc=example,dc=com, NULL) to (cn
 =My App,ou=Apps,dc=example,dc=com) (not proxied) ( reason: evaluated allow , de
 ciding_aci: Allow apps proxied auth)
aclRightsInfo;logs;entryLevel;delete: acl_summary(main): access allowed(delete)
 on entry/attr(cn=Carpoolers,ou=Self Service,ou=Groups,dc=example,dc=com, NULL)
 to (cn=My App,ou=Apps,dc=example,dc=com) (not proxied) ( reason: evaluated allo
 w , deciding_aci: Allow apps proxied auth)
aclRightsInfo;logs;entryLevel;read: acl_summary(main): access allowed(read) on e
 ntry/attr(cn=Carpoolers,ou=Self Service,ou=Groups,dc=example,dc=com, objectClas
 s) to (cn=My App,ou=Apps,dc=example,dc=com) (not proxied) ( reason: evaluated a
 llow , deciding_aci: Anonymous read-search access)
aclRightsInfo;logs;entryLevel;write: acl_summary(main): access allowed(write) on
  entry/attr(cn=Carpoolers,ou=Self Service,ou=Groups,dc=example,dc=com, NULL) to
  (cn=My App,ou=Apps,dc=example,dc=com) (not proxied) ( reason: evaluated allow
 , deciding_aci: Allow apps proxied auth)
aclRightsInfo;logs;entryLevel;proxy: acl_summary(main): access allowed(proxy) on
  entry/attr(cn=Carpoolers,ou=Self Service,ou=Groups,dc=example,dc=com, NULL) to
  (cn=My App,ou=Apps,dc=example,dc=com) (not proxied) ( reason: evaluated allow
 , deciding_aci: Allow apps proxied auth)
aclRights;entryLevel: add:1,delete:1,read:1,write:1,proxy:1
  

You can also request the effective rights for another user by using the --getEffectiveRightsAuthzid option (short form: -g). This option takes the authorization identity of the user as an argument. The following example shows My App checking Babs's rights to the same entry:

$ ldapsearch \
 --getEffectiveRightsAuthzid "dn:uid=bjensen,ou=People,dc=example,dc=com" \
 --port 1389 \
 --bindDN "cn=My App,ou=Apps,dc=example,dc=com" \
 --bindPassword password \
 --baseDN "ou=Self Service,ou=groups,dc=example,dc=com" \
 "(cn=Carpoolers)" \
 aclRights
dn: cn=Carpoolers,ou=Self Service,ou=Groups,dc=example,dc=com
aclRights;entryLevel: add:0,delete:1,read:1,write:0,proxy:0
  

The following example checks anonymous user rights to the same entry. Notice that the authorization identity for an anonymous user is expressed as dn::

$ ldapsearch \
 --getEffectiveRightsAuthzid "dn:" \
 --port 1389 \
 --bindDN "cn=My App,ou=Apps,dc=example,dc=com" \
 --bindPassword password \
 --baseDN "ou=Self Service,ou=groups,dc=example,dc=com" \
 "(cn=Carpoolers)" \
 aclRights
dn: cn=Carpoolers,ou=Self Service,ou=Groups,dc=example,dc=com
aclRights;entryLevel: add:0,delete:0,read:1,write:0,proxy:0
  

To check access to an attribute that might not yet exist on the entry, use the --getEffectiveRightsAttribute option (short form: -e). This option takes a comma-separated attribute list as an argument. The following example checks Andy Hall's access to the member attribute for the Carpooler's group entry:

$ ldapsearch \
 --getEffectiveRightsAuthzid "dn:uid=ahall,ou=People,dc=example,dc=com" \
 --getEffectiveRightsAttribute member \
 --port 1389 \
 --bindDN "cn=My App,ou=Apps,dc=example,dc=com" \
 --bindPassword password \
 --baseDN "ou=Self Service,ou=groups,dc=example,dc=com" \
 "cn=Carpoolers" \
 aclRights
dn: cn=Carpoolers,ou=Self Service,ou=Groups,dc=example,dc=com
aclRights;attributeLevel;member: search:1,read:1,compare:1,write:0,selfwrite_add:1,selfwrite_delete:1,proxy:0
aclRights;entryLevel: add:0,delete:0,read:1,write:0,proxy:0
  

A basic, standard directory feature is the ability to respond quickly to searches.

LDAP directory servers like DS directory servers even go so far as to disallow searches that cannot be handled expediently using indexes. Maintaining appropriate indexes is a key aspect of directory administration.

The role of an index is to answer the question, "Which entries have an attribute with this corresponding value?" Each index is therefore specific to an attribute. Each index is also specific to the comparison implied in the search filter. For example, a directory server maintains distinct indexes for exact (equality) matching and for substring matching. The types of indexes are explained in "Index Types and Their Functions". Furthermore, indexes are configured in specific directory backends.

An index is implemented as a tree of key-value pairs. The key is a form of the value to match, such as babs jensen. The value is a list of IDs for entries that match the key. "An Equality Index" shows an equality (case ignore exact match) index with five keys from a total of four entries. If the data set were large, there could be more than one entry ID per key.

An Equality Index

This is how DS directory servers use indexes. When the search filter is (cn=Babs Jensen), the directory server retrieves the IDs for entries with a CN matching Babs Jensen from the equality index of the CN attribute. (For a complex filter, the directory server might optimize the search by changing the order in which it uses the indexes.) A successful result is zero or more entry IDs. These are the candidate result entries.

For each candidate, the DS directory server retrieves the entry by ID from a special system index called id2entry, which, as its name suggests, returns an entry for an entry ID. If there is a match, and the client application has the right to access to the data, the directory server returns the search result. It continues this process until no candidates are left.

If there are no indexes that correspond to a search request, then the DS directory server must potentially check for a match against every entry in the scope of the search. Evaluating every entry for a match is referred to as an unindexed search. An unindexed search is an expensive operation, particularly for large directories. For this reason, a directory server refuses unindexed searches unless the user making the request has specific permission to make such requests. Permission to perform an unindexed search is granted with the unindexed-search privilege. This privilege is reserved for the directory root user by default, and should not be granted lightly.

If the number of user data entries is smaller than the default resource limits, you can still perform what appear to be unindexed searches, meaning searches with filters for which no index appears to exist. That is because the dn2id index returns all user data entries without hitting a resource limit that would make the search unindexed.

DS directory server search performance depends on indexes as described in "About Indexes".

DS directory servers maintain generally useful indexes for data imported into the default backend. When you create a new backend, the directory server only maintains the necessary system indexes unless you configure additional indexes. For details, see "Default Indexes".

The default settings are fine for evaluating DS software, and they work well with sample data. The default settings might not, however, fit your directory data and the searches performed on your directory service.

You can manage indexes using the command-line tools demonstrated in "Configuring and Rebuilding Indexes".

$ grep unindexed /path/to/opendj/logs/ldap-access.audit.json
{...,"request":{"protocol":"LDAP","operation":"SEARCH",..."detail":"You do not have sufficient privileges to perform an unindexed search","additionalItems":"unindexed"...}
{...,"request":{"protocol":"LDAP","operation":"SEARCH",..."detail":"You do not have sufficient privileges to perform an unindexed search","additionalItems":"unindexed"...}
{...,"request":{"protocol":"LDAP","operation":"SEARCH",..."detail":"You do not have sufficient privileges to perform an unindexed search","additionalItems":"unindexed"...}
   

$ dsconfig \
 set-backend-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name userRoot \
 --set index-filter-analyzer-enabled:true \
 --no-prompt \
 --trustAll
   

$ ldapsearch \
 --port 1389 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --baseDN ds-cfg-backend-id=userRoot,cn=Backends,cn=monitor \
 "(objectclass=*)" \
 filter-use
dn: ds-cfg-backend-id=userRoot,cn=Backends,cn=monitor
filter-use: (sn=Ishee) hits:1 maxmatches:8 message:
filter-use: (mail=*.com) hits:1 maxmatches:-1 message:The filter value exceeded the index entry limit for the /dc=com,dc=example/mail.caseIgnoreIA5SubstringsMatch:6 index
filter-use: (uid=user.86182) hits:1 maxmatches:1 message:
filter-use: (objectclass=person) hits:2 maxmatches:-1 message:The filter value exceeded the index entry limit for the /dc=com,dc=example/objectClass.objectIdentifierMatch index
filter-use: (employeenumber=86182) hits:1 maxmatches:-1 message:equality index type is disabled for the employeeNumber attribute
filter-use: (mail=*@example.com) hits:1 maxmatches:-1 message:The filter value exceeded the index entry limit for the /dc=com,dc=example/mail.caseIgnoreIA5SubstringsMatch:6 indexThe filter value exceeded the index entry limit for the /dc=com,dc=example/mail.caseIgnoreIA5SubstringsMatch:6 indexThe filter value exceeded the index entry limit for the /dc=com,dc=example/mail.caseIgnoreIA5SubstringsMatch:6 indexThe filter value exceeded the index entry limit for the /dc=com,dc=example/mail.caseIgnoreIA5SubstringsMatch:6 indexThe filter value exceeded the index entry limit for the /dc=com,dc=example/mail.caseIgnoreIA5SubstringsMatch:6 indexThe filter value exceeded the index entry limit for the /dc=com,dc=example/mail.caseIgnoreIA5SubstringsMatch:6 indexThe filter value exceeded the index entry limit for the /dc=com,dc=example/mail.caseIgnoreIA5SubstringsMatch:6 indexThe filter value exceeded the index entry limit for the /dc=com,dc=example/mail.caseIgnoreIA5SubstringsMatch:6 indexThe filter value exceeded the index entry limit for the /dc=com,dc=example/mail.caseIgnoreIA5SubstringsMatch:6 indexThe filter value exceeded the index entry limit for the /dc=com,dc=example/mail.caseIgnoreIA5SubstringsMatch:6 index
   

$ dsconfig \
 set-backend-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name userRoot \
 --set index-filter-analyzer-enabled:false \
 --no-prompt \
 --trustAll
   

$ ldapsearch \
 --port 1389 \
 --baseDN ou=people,dc=example,dc=com \
 "(&(mail=*.com)(objectclass=person))"
# The LDAP search request failed: 50 (Insufficient Access Rights)
# Additional Information:  You do not have sufficient privileges to perform an unindexed search
   

The server is performing an unindexed internal search request
with base DN '%s', scope '%s', and filter '%s'.
Unindexed internal searches are usually unexpected and could impact performance.
Please verify that that backend's indexes are configured correctly
for these search parameters.

$ dsconfig \
 set-access-control-handler-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --add global-aci:"(targetattr=\"debugsearchindex\")(version 3.0; acl \"Debug search indexes\"; \
  allow (read,search,compare) userdn=\"ldap:///uid=user.0,ou=people,dc=example,dc=com\";)" \
 --trustAll \
 --no-prompt
   

$ ldapsearch \
 --port 1389 \
 --bindDN uid=user.0,ou=people,dc=example,dc=com \
 --bindPassword password \
 --baseDN dc=example,dc=com \
 "(&(objectclass=person)(givenName=aa*))" \
 debugsearchindex | sed -n -e "s/^debugsearchindex: //p"
{
  "filter": {
    "intersection": [
      {
        "index": "givenName.caseIgnoreSubstringsMatch:6",
        "range": "[aa,ab[",
        "candidates": 171,
        "retained": 171
      },
      {
        "index": "givenName.caseIgnoreMatch",
        "range": "[aa,ab[",
        "candidates": 50,
        "retained": 50
      },
      {
        "filter": "(objectclass=person)",
        "union": [
          {
            "index": "objectClass.objectIdentifierMatch",
            "exact": "person",
            "candidates": "[LIMIT-EXCEEDED]"
          }
        ],
        "candidates": "[LIMIT-EXCEEDED]",
        "retained": 50
      }
    ],
    "candidates": 50
  },
  "scope": {
    "type": "sub",
    "candidates": "[NOT-INDEXED]",
    "retained": 50
  },
  "final": 50
}
   

$ ldapsearch \
 --port 1389 \
 --baseDN dc=example,dc=com \
 --bindDN uid=user.0,ou=people,dc=example,dc=com \
 --bindPassword password \
 "(|(initials=aa*)(initials=zz*))" \
 debugsearchindex | sed -n -e "s/^debugsearchindex: //p"
{
  "filter": {
    "union": [
      {
        "filter": "(initials=aa*)",
        "index": "initials.presence",
        "diagnostic": "not indexed",
        "candidates": "[NOT-INDEXED]"
      }
    ],
    "candidates": "[NOT-INDEXED]"
  },
  "scope": {
    "type": "sub",
    "candidates": "[NOT-INDEXED]",
    "retained": "[NOT-INDEXED]"
  },
  "final": "[NOT-INDEXED]"
}

$ dsconfig \
 create-backend-index \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name userRoot \
 --index-name initials \
 --set index-type:equality \
 --trustAll \
 --no-prompt
$ rebuild-index \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDn "cn=Directory Manager" \
 --bindPassword password \
 --baseDn dc=example,dc=com \
 --index initials \
 --trustAll

$ ldapsearch \
 --port 1389 \
 --baseDN dc=example,dc=com \
 --bindDN uid=user.0,ou=people,dc=example,dc=com \
 --bindPassword password \
 "(|(initials=aa*)(initials=zz*))" \
 debugsearchindex | sed -n -e "s/^debugsearchindex: //p"
{
  "filter": {
    "union": [
      {
        "filter": "(initials=aa*)",
        "intersection": [
          {
            "index": "initials.caseIgnoreSubstringsMatch:6",
            "range": "[aa,ab[",
            "diagnostic": "not indexed",
            "tryPresenceIndex": {
              "index": "initials.presence",
              "diagnostic": "not indexed"
            },
            "candidates": "[NOT-INDEXED]",
            "retained": "[NOT-INDEXED]"
          },
          {
            "index": "initials.caseIgnoreMatch",
            "range": "[aa,ab[",
            "candidates": 378,
            "retained": 378
          }
        ],
        "candidates": 378
      },
      {
        "filter": "(initials=zz*)",
        "intersection": [
          {
            "index": "initials.caseIgnoreSubstringsMatch:6",
            "range": "[zz,z{[",
            "diagnostic": "not indexed",
            "tryPresenceIndex": {
              "index": "initials.presence",
              "diagnostic": "not indexed"
            },
            "candidates": "[NOT-INDEXED]",
            "retained": "[NOT-INDEXED]"
          },
          {
            "index": "initials.caseIgnoreMatch",
            "range": "[zz,z{[",
            "candidates": 26,
            "retained": 26
          }
        ],
        "candidates": 26
      }
    ],
    "candidates": 404
  },
  "scope": {
    "type": "sub",
    "candidates": "[NOT-INDEXED]",
    "retained": 404
  },
  "final": 404
}

DS directory servers support multiple index types, each corresponding to a different type of search. This section describes the index types and what they are used for.

View what is indexed by using the backendstat list-indexes command. For details about a particular index, you can use the backendstat dump-index command.

$ ldapsearch \
 --port 1389 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --baseDN dc=example,dc=com \
 "(aci=*)" \
 aci
   

$ stop-ds --quiet
$ backendstat \
 dump-index \
 --backendId dsEvaluation \
 --baseDn dc=example,dc=com \
 --indexName aci.presence
Key (len 1): PRESENCE
Value (len 4): [COUNT:3] 1 9 11

Total Records: 1
Total / Average Key Size: 1 bytes / 1 bytes
Total / Average Data Size: 4 bytes / 4 bytes
   

$ ldapsearch --port 1389 --baseDN dc=example,dc=com "(uid=bjensen)" mail
dn: uid=bjensen,ou=People,dc=example,dc=com
mail: bjensen@example.com
   

$ stop-ds --quiet
$ backendstat \
 dump-index \
 --backendID dsEvaluation \
 --baseDN dc=example,dc=com \
 --indexName sn.caseIgnoreMatch
...
Key (len 6): jensen
Value (len 10): [COUNT:9] 18 31 32 66 79 94 133 134 150
...
   

$ ldapsearch --port 1389 --baseDN dc=example,dc=com "(cn~=Babs Jansen)" cn
dn: uid=bjensen,ou=People,dc=example,dc=com
cn: Barbara Jensen
cn: Babs Jensen
   

$ stop-ds --quiet
$ backendstat \
 dump-index \
 --backendID dsEvaluation \
 --baseDN dc=example,dc=com \
 --indexName sn.ds-mr-double-metaphone-approx
...
Key (len 4): JNSN
Value (len 11): [COUNT:10] 18 31 32 59 66 79 94 133 134 150
...
   

$ ldapsearch --port 1389 --baseDN dc=example,dc=com "(cn=Barb*)" cn
dn: uid=bfrancis,ou=People,dc=example,dc=com
cn: Barbara Francis

dn: uid=bhal2,ou=People,dc=example,dc=com
cn: Barbara Hall

dn: uid=bjablons,ou=People,dc=example,dc=com
cn: Barbara Jablonski

dn: uid=bjensen,ou=People,dc=example,dc=com
cn: Barbara Jensen
cn: Babs Jensen

dn: uid=bmaddox,ou=People,dc=example,dc=com
cn: Barbara Maddox
   

$ stop-ds --quiet
$ backendstat \
dump-index \
 --backendID dsEvaluation \
 --baseDN dc=example,dc=com \
 --indexName sn.caseIgnoreSubstringsMatch:6
...
Key (len 1): e
Value (len 25): [COUNT:22] ...
...
Key (len 2): en
Value (len 15): [COUNT:12] ...
...
Key (len 3): ens
Value (len 3): [COUNT:1] 147
Key (len 5): ensen
Value (len 10): [COUNT:9] 18 31 32 66 79 94 133 134 150
...
Key (len 6): jensen
Value (len 10): [COUNT:9] 18 31 32 66 79 94 133 134 150
...
Key (len 1): n
Value (len 35): [COUNT:32] ...
...
Key (len 2): ns
Value (len 3): [COUNT:1] 147
Key (len 4): nsen
Value (len 10): [COUNT:9] 18 31 32 66 79 94 133 134 150
...
Key (len 1): s
Value (len 13): [COUNT:12] 12 26 47 64 95 98 108 131 135 147 149 154
...
Key (len 2): se
Value (len 7): [COUNT:6] 52 58 75 117 123 148
Key (len 3): sen
Value (len 10): [COUNT:9] 18 31 32 66 79 94 133 134 150
...
   

$ ldapsearch --port 1389 --baseDN dc=example,dc=com "(sn>=winter)" sn
dn: uid=aworrell,ou=People,dc=example,dc=com
sn: Worrell

dn: uid=kwinters,ou=People,dc=example,dc=com
sn: Winters

dn: uid=pworrell,ou=People,dc=example,dc=com
sn: Worrell
   

You modify index configurations by using the dsconfig command. The subcommands to use depend on the backend type, as shown in the examples that follow. The configuration changes then take effect after you rebuild the index according to the new configuration, using the rebuild-index command. The dsconfig --help-database command lists subcommands for creating, reading, updating, and deleting index configuration.

$ dsconfig \
 create-backend-index \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name userRoot \
 --index-name description \
 --set index-type:equality \
 --trustAll \
 --no-prompt
    

$ dsconfig \
 set-backend-index-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name dsEvaluation \
 --index-name sn \
 --add index-type:approximate \
 --trustAll \
 --no-prompt
    

$ dsconfig \
 create-backend-index \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name userRoot \
 --set index-type:extensible \
 --set index-extensible-matching-rule:1.3.6.1.4.1.26027.1.4.5 \
 --set index-extensible-matching-rule:1.3.6.1.4.1.26027.1.4.6 \
 --index-name lastLoginTime \
 --trustAll \
 --no-prompt
    

attributeTypes: ( json-attribute-oid NAME 'json'
  SYNTAX 1.3.6.1.4.1.36733.2.1.3.1 EQUALITY caseIgnoreJsonQueryMatch
  SINGLE-VALUE X-ORIGIN 'DS Documentation Examples' )

$ dsconfig \
 create-schema-provider \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --provider-name "Custom JSON Query Matching Rule" \
 --type json-query-equality-matching-rule \
 --set enabled:true \
 --set case-sensitive-strings:false \
 --set ignore-white-space:true \
 --set matching-rule-name:caseIgnoreJsonQueryMatch \
 --set matching-rule-oid:1.3.6.1.4.1.36733.2.1.4.1 \
 --set indexed-field:access_token \
 --set indexed-field:refresh_token \
 --trustAll \
 --no-prompt
    

$ cat json-schema.ldif
dn: cn=schema
changetype: modify
add: attributeTypes
attributeTypes: ( json-attribute-oid NAME 'json'
  SYNTAX 1.3.6.1.4.1.36733.2.1.3.1 EQUALITY caseIgnoreJsonQueryMatch
  SINGLE-VALUE X-ORIGIN 'DS Documentation Examples' )
-
add: objectClasses
objectClasses: ( json-object-class-oid NAME 'jsonObject' SUP top
  AUXILIARY MAY ( json ) X-ORIGIN 'DS Documentation Examples' )

$ ldapmodify \
 --hostname opendj.example.com \
 --port 1389 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 json-schema.ldif
    

$ dsconfig \
 create-backend-index \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name userRoot \
 --index-name json \
 --set index-type:equality \
 --trustAll \
 --no-prompt
    

$ dsconfig \
 create-schema-provider \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --provider-name "Custom JSON Token ID Matching Rule" \
 --type json-equality-matching-rule \
 --set enabled:true \
 --set case-sensitive-strings:false \
 --set ignore-white-space:true \
 --set matching-rule-name:caseIgnoreJsonTokenIDMatch \
 --set matching-rule-oid:1.3.6.1.4.1.36733.2.1.4.4.1 \
 --set json-keys:id \
 --trustAll \
 --no-prompt
    

$ cat json-token-schema.ldif
dn: cn=schema
changetype: modify
add: attributeTypes
attributeTypes: ( jsonToken-attribute-oid NAME 'jsonToken'
  SYNTAX 1.3.6.1.4.1.36733.2.1.3.1 EQUALITY caseIgnoreJsonTokenIDMatch
  SINGLE-VALUE X-ORIGIN 'DS Documentation Examples' )
-
add: objectClasses
objectClasses: ( json-token-object-class-oid NAME 'JsonTokenObject' SUP top
  AUXILIARY MAY ( jsonToken ) X-ORIGIN 'DS Documentation Examples' )

$ ldapmodify \
 --hostname opendj.example.com \
 --port 1389 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 json-token-schema.ldif
    

$ dsconfig \
 create-backend-index \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name userRoot \
 --index-name jsonToken \
 --set index-type:equality \
 --trustAll \
 --no-prompt
    

$ dsconfig \
 create-backend-vlv-index \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDn "cn=Directory Manager" \
 --bindPassword password \
 --backend-name dsEvaluation \
 --index-name people-by-last-name \
 --set base-dn:ou=People,dc=example,dc=com \
 --set filter:"(|(givenName=*)(sn=*))" \
 --set scope:single-level \
 --set sort-order:"+sn +givenName" \
 --trustAll \
 --no-prompt
   

$ dsconfig \
 create-backend-vlv-index \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDn "cn=Directory Manager" \
 --bindPassword password \
 --backend-name dsEvaluation \
 --index-name by-last-name \
 --set base-dn:ou=People,dc=example,dc=com \
 --set filter:"(&)" \
 --set scope:subordinate-subtree \
 --set sort-order:"+l +roomNumber +sn +cn" \
 --trustAll \
 --no-prompt
$ rebuild-index \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDn "cn=Directory Manager" \
 --bindPassword password \
 --baseDn dc=example,dc=com \
 --index vlv.by-last-name \
 --trustAll
$ cat allow-unindexed-search.ldif
dn: uid=kvaughan,ou=People,dc=example,dc=com
changetype: modify
add: ds-privilege-name
ds-privilege-name: unindexed-search

$ ldapmodify \
 --hostname opendj.example.com \
 --port 1389 \
 --bindDn "cn=Directory Manager" \
 --bindPassword password \
 allow-unindexed-search.ldif
$ cat allow-server-side-sort.ldif
dn: dc=example,dc=com
changetype: modify
add: aci
aci: (targetcontrol = "2.16.840.1.113730.3.4.3")
 (version 3.0;acl "Allow Server-Side Sort for Kirsten Vaughan";
 allow (read)(userdn = "ldap:///uid=kvaughan,ou=People,dc=example,dc=com");)

$ ldapmodify \
 --hostname opendj.example.com \
 --port 1389 \
 --bindDn "cn=Directory Manager" \
 --bindPassword password \
 allow-server-side-sort.ldif
$ ldapsearch \
 --hostname opendj.example.com \
 --port 1389 \
 --bindDn uid=kvaughan,ou=people,dc=example,dc=com \
 --bindPassword bribery \
 --baseDn dc=example,dc=com \
 --sortOrder +l,+roomNumber,+sn,+cn \
 --simplePageSize 5 \
 "(&)" \
 cn l roomNumber
dn: uid=cnewport,ou=People,dc=example,dc=com
cn: Christoph Newport
l: Bristol
roomNumber: 0056

dn: uid=awhite,ou=People,dc=example,dc=com
cn: Alan White
l: Bristol
roomNumber: 0142

dn: uid=rdaugherty,ou=People,dc=example,dc=com
cn: Robert Daugherty
l: Bristol
roomNumber: 0194

dn: uid=mlott,ou=People,dc=example,dc=com
cn: Mike Lott
l: Bristol
roomNumber: 0498

dn: uid=brentz,ou=People,dc=example,dc=com
cn: Bertram Rentz
l: Bristol
roomNumber: 0617

Press RETURN to continue ^C
    

$ rebuild-index \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --baseDN dc=example,dc=com \
 --index cn \
 --trustAll
    

$ rebuild-index \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --baseDN dc=example,dc=com \
 --rebuildDegraded \
 --trustAll
    

$ stop-ds --quiet
    

$ backendstat \
 show-index-status \
 --backendID userRoot \
 --baseDN dc=example,dc=com \
 | grep newUnusedAttribute
newUnusedAttribute.presence                    ... false ...
newUnusedAttribute.caseIgnoreMatch             ... false ...
newUnusedAttribute.caseIgnoreSubstringsMatch:6 ... false ...
    

$ rebuild-index \
 --offline \
 --baseDN dc=example,dc=com \
 --clearDegradedState \
 --index newUnusedAttribute
    

$ backendstat \
 show-index-status \
 --backendID userRoot \
 --baseDN dc=example,dc=com \
 | grep newUnusedAttribute
newUnusedAttribute.presence                    ... true ...
newUnusedAttribute.caseIgnoreMatch             ... true ...
newUnusedAttribute.caseIgnoreSubstringsMatch:6 ... true ...
    

$ start-ds --quiet
    

7.4.5. Understanding Index Entry Limits

As described in "About Indexes", an index is implemented as a tree of key-value pairs. The key is what the search is trying to match. The value is a list of entry IDs.

As the number of entries in the directory grows, the list of entry IDs for some keys can become very large. For example, every entry in the directory has the value top for the objectClass attribute. If the directory maintains a substring index for mail, the number of entries ending in .com could be huge.

A directory server therefore defines an index entry limit. When the number of entry IDs for a key exceeds the limit, the server stops maintaining a list of IDs for that key. The limit effectively makes a search using that key unindexed. Searches using other keys in the same index are not affected.

"Index Entry Limit Exceeded For a Single Key" shows a fragment from a substring index for the mail attribute. The number of email addresses ending in .com has exceeded the index entry limit. For the other substring keys, the entry ID lists are still maintained, but to save space the entry IDs are not shown in the diagram.

Index Entry Limit Exceeded For a Single Key

Ideally, the limit is set at the point where it becomes more expensive to maintain the entry ID list for a key and to perform an indexed search than to perform an unindexed search. In practice, the limit is a trade off, with a default index entry limit value of 4000.

To Check Index Entry Limits

The following steps show how to get information about indexes where the index entry limit is exceeded for some keys. In this case, the directory server holds 100,000 user entries. The settings for this directory server are reasonable.

Use the backendstat show-index-status command, described in "backendstat — gather OpenDJ backend debugging information" in the Reference.

1. Stop DS servers before you use the backendstat command:

   $ stop-ds --quiet
        

2. Non-zero values in the Over Entry Limit column of the output table indicate the number of keys for which the limit has been reached. The keys that are over the limit are then listed below the table:

   $ backendstat show-index-status --backendID userRoot --baseDN dc=example,dc=com
   Index Name                                       ... Valid  ... Over Entry Limit ...
   ...
   mail.caseIgnoreIA5SubstringsMatch:6              ... true   ... 34 ...
   ...
   givenName.caseIgnoreSubstringsMatch:6            ... true   ... 9  ...
   telephoneNumber.telephoneNumberSubstringsMatch:6 ... true   ... 10 ...
   ...
   sn.caseIgnoreSubstringsMatch:6                   ... true   ... 14 ...
   ...
   cn.caseIgnoreSubstringsMatch:6                   ... true   ... 14 ...
   objectClass.objectIdentifierMatch                ... true   ... 4  ...
   ...
   
   Index: /dc=com,dc=example/mail.caseIgnoreIA5SubstringsMatch:6
   Over index-entry-limit keys: [.com] [0@mail] ...
   
   Index: /dc=com,dc=example/givenName.caseIgnoreSubstringsMatch:6
   Over index-entry-limit keys: [a] [e] [i] [ie] [l] [n] [na] [ne] [y]
   
   Index: /dc=com,dc=example/telephoneNumber.telephoneNumberSubstringsMatch:6
   Over index-entry-limit keys: [0] [1] [2] [3] [4] [5] [6] [7] [8] [9]
   
   Index: /dc=com,dc=example/cn.caseIgnoreSubstringsMatch:6
   Over index-entry-limit keys: [a] [an] [e] [er] [i] [k] [l] [n] [o] [on] [r] [s] [t] [y]
   
   Index: /dc=com,dc=example/objectClass.objectIdentifierMatch
   Over index-entry-limit keys: [inetorgperson] [organizationalperson] [person] [top]
   
   Index: /dc=com,dc=example/sn.caseIgnoreSubstringsMatch:6
   Over index-entry-limit keys: [a] [an] [e] [er] [i] [k] [l] [n] [o] [on] [r] [s] [t] [y]
         

   For example, every user entry has the object classes listed, and every user entry has an email address ending in .com, so those values are not specific enough to be used in search filters.

3. Start the server:

   $ start-ds --quiet
        

Index Entry Limit Changes

In rare cases, the index entry limit might be too low for a certain key. This could manifest itself as a frequent, useful search becoming unindexed, with no reasonable way to narrow the search.

You can change the index entry limit on a per-index basis. Do not do this in production unless you can explain and show why the benefits outweigh the costs.

Important

Changing the index entry limit significantly can result in serious performance degradation. Be prepared to test performance thoroughly before you roll out an index entry limit change in production.

Consider a directory with more than 4000 groups in a backend. When the backend is brought online, a directory server searches for the groups with a search filter of (|(objectClass=groupOfNames)(objectClass=groupOfEntries)(objectClass=groupOfUniqueNames)), which is an unindexed search due to the default index entry limit setting. The following example raises the index entry limit for the objectClass index to 10000, and then rebuilds the index for the configuration change to take effect. The steps are the same for any other index:

$ dsconfig \
 set-backend-index-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name userRoot \
 --index-name objectClass \
 --set index-entry-limit:10000 \
 --no-prompt \
 --trustAll
$ stop-ds --quiet
$ rebuild-index \
 --baseDN dc=example,dc=com \
 --index objectClass \
 --offline
$ start-ds --quiet
    

It is also possible, but not recommended, to configure the global index-entry-limit for a backend. This changes the default for all indexes in the backend. Use the dsconfig set-backend-prop command as shown in the following example:

# Not recommended
$ dsconfig \
 set-backend-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --backend-name userRoot \
 --set index-entry-limit:10000 \
 --trustAll \
 --no-prompt
   

You can verify that indexes correspond to current directory data, and that indexes do not contain errors by using the verify-index command, described in "verify-index — check index for consistency or errors" in the Reference.

$ stop-ds --quiet
$ verify-index \
 --baseDN dc=example,dc=com \
 --index cn \
 --clean \
 --countErrors
   

When you first install a directory server and import your data from LDIF, the indexes are configured as shown in "Default Indexes".

Read this section to learn more about how DS replication works.

8.1.2. Replication Per Suffix

The primary unit of replication is the suffix, specified by a base DN such as dc=example,dc=com. (When you configure partial and fractional replication, however, you can replicate only part of a suffix, or only certain attributes on entries. Also, if you split your suffix across multiple backends, then you need to set up replication separately for each part of suffix in a different backend.) Replication also depends on the directory schema, defined on cn=schema, and the cn=admin data suffix with administrative identities and certificates for protecting communications. That content gets replicated as well.

The set of DS servers replicating data for a given suffix is called a replication topology. You can have more than one replication topology. For example, one topology could be devoted to dc=example,dc=com, and another to dc=example,dc=org. DS servers serve more than one suffix, and participate in more than one replication topology as shown in "Correct Replication Topology Configuration".

Correct Replication Topology Configuration

Within a replication topology, the suffixes being replicated are identified to the replication servers by their DNs. All the replication servers are fully connected in a topology. Consequently it is impossible to have multiple separate, independent topologies for data under the same DN within the overall set of servers. This is illustrated in "Incorrect Replication Topology Configuration".

Incorrect Replication Topology Configuration

8.1.3. Replication Network Use and Operations

DS servers listen on dedicated ports for administrative requests and for replication requests. These dedicated ports must remain open to remote requests from configuration tools and from other servers.

DS server configuration tools securely connect to administration ports. Administrative connections are short-lived. When configuring replication or reading replication status, a single dsreplication command can connect to multiple servers.

DS directory server replicas connect to DS replication ports for replication requests. A server listening on a replication port is called a replication server, whether it is running inside the same process as a directory server, or on a separate host system. Replication connections are long-lived. Each DS replica connects to a replication server upon initialization and then at startup time. The replica keeps the connection open to push and receive updates, although it can connect to another replication server as detailed in "Replication Connection Selection".

The following sequence diagrams show how key operations involve the network. These operations are the following:

• Setting up replication (configuration, then initialization)

• Data replication (normal operation on update involving no administrative intervention)

• Checking replication status

The commands to configure and to initialize replication use the administrative and replication ports, respectively, as shown in the following sequence diagram:

Replication Configuration and Initialization

DS replicas push updates to and receive updates from replication servers over replication port connections. When processing an update, a directory server (DS) replica pushes it to the replication server (RS) it is connected to. The replication server pushes the update to connected directory server replicas and to other replication servers. Notice that directory servers do not connect directly to each other, but through replication servers. This is shown in the following sequence diagram:

Data Replication

The command to monitor replication status uses the administration ports on multiple servers, as shown in the following sequence diagram:

Replication Status

"Limiting System and Administrative Access" in the Security Guide provides detailed descriptions of the ports used.

This section shows how to configure replication with command-line tools, such as the dsreplication command, described in "dsreplication — manage directory data replication" in the Reference.

$ dsconfig \
 set-global-configuration-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --set server-id:1 \
 --trustAll \
 --no-prompt

$ dsreplication \
 configure \
 --adminUID admin \
 --adminPassword password \
 --baseDN dc=example,dc=com \
 --host1 opendj.example.com \
 --port1 4444 \
 --bindDN1 "cn=Directory Manager" \
 --bindPassword1 password \
 --replicationPort1 8989 \
 --host2 replica.example.com \
 --port2 4444 \
 --bindDN2 "cn=Directory Manager" \
 --bindPassword2 password \
 --replicationPort2 8989 \
 --trustAll \
 --no-prompt
...
Replication has been successfully configured.
...
   

There is an error with the certificate presented by the server.

Unwilling to Perform: ... Cannot change the configuration while a total update is in progress

$ dsconfig \
 set-replication-server-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --provider-name "Multimaster Synchronization" \
 --set "disk-low-threshold:10 GB" \
 --set "disk-full-threshold:5 GB" \
 --trustAll \
 --no-prompt

8.2.5. Standalone Replication Servers

DS replication is designed to be easy to implement in environments with a few servers, and scalable in environments with many servers. You can configure the replication service on each DS directory server in your deployment, for example, to limit the number of servers you deploy.

In a large deployment, you can use standalone replication servers—servers that do nothing but relay replication messages—to configure (and troubleshoot) the replication service separately from the directory service. You need only a few standalone replication servers publishing changes to serve many directory servers subscribed to the changes. Furthermore, replication is designed so a directory server needs to connect only to the nearest replication server in order to to replicate with all servers in the topology. Only the standalone replication servers participate in fully meshed replication.

All replication servers in a topology are connected to all other replication servers. Directory servers are connected only to one replication server at a time, and their connections should be to replication servers on the same LAN. Therefore, the total number of replication connections, Total_conn, is expressed as follows:

Total_conn = (N_RS * (N_RS-1))/2 + N_DS

Here, N_RS is the number of replication servers, and N_DS is the number of standalone directory servers.

In other words, if you have only three servers, then Total_conn is three with no standalone servers. However, if you have two data centers, and need 12 directory servers, then with no standalone directory servers Total_conn is (12 * 11)/2 or 66. Yet, with four standalone replication servers, and 12 standalone directory servers, Total_conn is (4 * 3)/2 + 12, or 18. Only four of those connections extend over the WAN. (By running four directory servers that also run replication servers and eight standalone directory servers, you reduce the number of replication connections to 14 for 12 replicas.)

This is shown in "Deployment For Multiple Data Centers".

Deployment For Multiple Data Centers

To Use a Standalone Replication Server

This procedure uses a standalone replication server to handle the replication traffic between two standalone directory servers.

In a production deployment, set up additional standalone replication servers to avoid a single point of failure.

Follow these steps:

1. Set up a standalone replication server, for example rs-only.example.com, as described in "Installing a Replication Server" in the Installation Guide.

2. Set up two standalone directory servers, for example opendj.example.com and replica.example.com, as described in "Installing a Directory Server" in the Installation Guide.

3. Configure replication with the --noReplicationServer or --onlyReplicationServer option:

   $ dsreplication \
    configure \
    --adminUID admin \
    --adminPassword password \
    --baseDN dc=example,dc=com \
    --host1 opendj.example.com \
    --port1 4444 \
    --bindDN1 "cn=Directory Manager" \
    --bindPassword1 password \
    --noReplicationServer1 \
    --host2 rs-only.example.com \
    --port2 4444 \
    --bindDN2 "cn=Directory Manager" \
    --bindPassword2 password \
    --replicationPort2 8989 \
    --onlyReplicationServer2 \
    --trustAll \
    --no-prompt
   
   $ dsreplication \
    configure \
    --adminUID admin \
    --adminPassword password \
    --baseDN dc=example,dc=com \
    --host1 replica.example.com \
    --port1 4444 \
    --bindDN1 "cn=Directory Manager" \
    --bindPassword1 password \
    --noReplicationServer1 \
    --host2 rs-only.example.com \
    --port2 4444 \
    --bindDN2 "cn=Directory Manager" \
    --bindPassword2 password \
    --replicationPort2 8989 \
    --onlyReplicationServer2 \
    --trustAll \
    --no-prompt
        

4. Initialize replication from one of the directory servers, for example opendj.example.com:

   $ dsreplication \
    initialize-all \
    --adminUID admin \
    --adminPassword password \
    --baseDN dc=example,dc=com \
    --hostname opendj.example.com \
    --port 4444 \
    --trustAll \
    --no-prompt
        

$ dsconfig \
 set-global-configuration-prop \
 --hostname replica.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --set writability-mode:internal-only \
 --trustAll \
 --no-prompt
   

$ dsconfig \
 set-replication-domain-prop \
 --hostname replica.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --provider-name "Multimaster Synchronization" \
 --domain-name "dc=example,dc=com" \
 --set fractional-include:inetorgperson:cn,givenname,mail,mobile,sn,telephonenumber \
 --trustAll \
 --no-prompt
   

$ dsconfig \
 set-replication-domain-prop \
 --hostname replica.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --provider-name "Multimaster Synchronization" \
 --domain-name "dc=example,dc=com" \
 --set fractional-exclude:*:sessionToken \
 --trustAll \
 --no-prompt
   

$ dsconfig \
 set-replication-server-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --provider-name "Multimaster Synchronization" \
 --set listen-address:192.168.0.10 \
 --trustAll \
 --no-prompt

Some applications require notification when directory data updates occur. For example, an application might need to sync directory data with another database, or the application might need to kick off other processing when certain updates occur.

In addition to supporting persistent search operations, DS replication servers provide an external change log mechanism to allow applications to be notified of changes to directory data.

This section includes the following procedures:

$ ldapsearch \
 --hostname existing.example.com \
 --port 1389 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --baseDN "" \
 --searchScope base \
 "(&)" lastChangeNumber
dn:
lastChangeNumber: 285924


Result Code:  0 (Success)

$ ldapsearch \
 --hostname new.example.com \
 --port 1389 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --baseDN "" \
 --searchScope base \
 "(&)" lastChangeNumber
dn:
lastChangeNumber: 198643


Result Code:  0 (Success)
   

This section covers how to restore accidentally deleted or changed data.

Changes to a replicated DS directory service are similar to those made with the Unix rm command, but with a twist. With the rm command, if you make a mistake you can restore your files from backup, and lose only the work done since the last backup. If you make a mistake with a update to the directory service however, then after you restore a server from backup, replication efficiently replays your mistake to the server you restored.

There is more than one way to recover from user error. None of the ways involve simply changing DS settings. All of the ways instead involve manually fixing mistakes.

Replication is eventually consistent by design. Rather than dealing with some sort of distributed transaction mechanism that would make write operations impossible during a network partition, replication supports basic write availability. Changes are applied locally and then replayed to remote replicas. This means it is possible to have conflicts, however. A replication conflict arises when incompatible changes are applied concurrently to multiple read-write replicas.

Two types of conflicts can occur: modify conflicts and naming conflicts. Modify conflicts involve concurrent modifications to the same entry. Naming conflicts involve other operations that affect the DN of the entry.

Replication can resolve modify conflicts and many naming conflicts automatically by replaying the changes in the correct order. To determine the relative order in which changes occurred, replicas retain historical information for each update. This information is stored in the target entry's ds-sync-hist operational attribute.

When replication cannot resolve naming conflicts automatically, the server renames the conflicting entry using its entryUUID operational attribute. The resulting conflicting entry has a DN with the following form:

entryuuid=entryUUID-value+original-RDN,original-parent-DN

When you install a directory server, the setup process creates a bak/ directory as the default location for binary backups. When you create a backup, the bak/backup.info file contains information about the archive. This is acceptable if you have only one backend to back up. Each backup.info file only contains information about one backend, however. If you have more than one backend, then use a separate backup directory for each backend in order to have separate backup.info files for each backend ID.

Archives produced by the backup command only contain backups of the directory data. Snapshots of server configuration are found under the var/ directory.

This section includes the following procedures:

When you restore data, the procedure to follow depends on whether the directory server is replicated.

When you back up directory data using the DS tools, you do not back up server configuration files. Server configuration files are found in the /path/to/opendj/config/ directory.

Snapshots of the main server configuration file, config.ldif, are found in the var directory in the server instance directory. You can use these snapshots to recover from misconfiguration performed with the dsconfig command. The snapshots do not, however, include any other files in the config/ directory.

Stop the server during backup and restore of configuration files as described in the following procedures:

DS password policies govern not only passwords, but also account lockout, and how DS servers provide notification about account status.

DS servers support password policies as part of the server configuration, and subentry password policies as part of the (replicated) user data.

$ dsconfig \
 get-password-policy-prop \
 --hostname opendj.example.com \
 --port 4444 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --policy-name "Default Password Policy" \
 --advanced \
 --trustAll \
 --no-prompt
Property                                  : Value(s)
------------------------------------------:------------------------------------
account-status-notification-handler       : -
allow-expired-password-changes            : false
allow-multiple-password-values            : false
allow-pre-encoded-passwords               : false
allow-user-password-changes               : true
default-password-storage-scheme           : Salted SHA-512
deprecated-password-storage-scheme        : -
expire-passwords-without-warning          : false
force-change-on-add                       : false
force-change-on-reset                     : false
grace-login-count                         : 0
idle-lockout-interval                     : 0 s
java-class                                : org.opends.server.core.PasswordPoli
                                          : cyFactory
last-login-time-attribute                 : -
last-login-time-format                    : -
lockout-duration                          : 0 s
lockout-failure-count                     : 0
lockout-failure-expiration-interval       : 0 s
max-password-age                          : 0 s
max-password-reset-age                    : 0 s
min-password-age                          : 0 s
password-attribute                        : userPassword
password-change-requires-current-password : false
password-expiration-warning-interval      : 5 d
password-generator                        : Random Password Generator
password-history-count                    : 0
password-history-duration                 : 0 s
password-validator                        : -
previous-last-login-time-format           : -
require-change-by-time                    : -
require-secure-authentication             : false
require-secure-password-changes           : false
skip-validation-for-administrators        : false
state-update-failure-policy               : reactive
  

$ ldapsearch \
 --port 1389 \
 --bindDN "cn=Directory Manager" \
 --bindPassword password \
 --baseDN dc=example,dc=com \
 "(uid=bjensen)" \
 userpassword
dn: uid=bjensen,ou=People,dc=example,dc=com
userpassword: {SSHA512}<hash>