Guide to deploying ForgeRock® Directory Services.

In deployments with multiple replicas, directory servers replay replicated operations. Expect each replica to replay every successful update to any replica.

When deciding how to deploy a directory server, think of it as a copy of the database. A large, high-performance, distributed database serving lots of applications obviously requires more system resources than a small database embedded in an application.

In all deployments using replication, the replication service provides the foundation of directory service availability. This is as important to the directory service as a naming service is for a network. When deploying replicated directory services, start by installing the replication service.

To avoid a single point of failure, install two or more replication servers in each location.

When deploying a replication server, keep its foundational role in mind. Directory servers communicate with other replicas through replication servers. Directory proxy servers rely on replication servers to find directory servers.

A directory proxy server can hide the underlying directory service architecture from client applications, enabling you to build a single point of directory service access.

A directory proxy server can discover directory servers through a replication server. This capability relies, however, on the replication server configuration. If you use the proxy server with third-party directory service components, then you must manually maintain the network locations for directory servers.

A directory proxy server provides LDAP access to remote LDAP directory servers. If you want to provide HTTP access to remote LDAP directory servers, use the REST to LDAP gateway instead. For details, see "REST to LDAP Gateway".

In order to route requests appropriately, a directory proxy server must decode incoming requests and encode ongoing requests. It must also decode and encode incoming and outgoing responses. When deploying a directory proxy server, keep this decoding and encoding in mind, because it explains why you might need as many proxy servers as directory servers.

A DSML gateway runs in a Java web application server. It is limited to one host:port combination for the LDAP directory service.

A DSML gateway maintains only its own service configuration, recorded in the web application WEB-INF/web.xml file. It depends on the host web application server for other services, such as logging.

A REST to LDAP gateway runs in a Java web application server. It can be configured to contact multiple LDAP directory servers.

One RESTful HTTP request can generate multiple LDAP requests. This is particularly true if the REST to LDAP mapping configuration includes references to resolve before returning response entries. For example, an LDAP user entry can have a manager attribute that holds the DN of the user's manager's entry. Rather than return an LDAP-specific DN in the REST response, the REST to LDAP mapping is configured to return the manager's name in the response. As a result, every time a user's manager is returned in the response, the gateway must make a request for the user's LDAP information and another request for the user's manager's name.

A REST to LDAP gateway maintains only its own service configuration, recorded in files as described in "REST to LDAP Configuration" in the Reference. It depends on the host web application server for other services, such as logging.

The project initiation phase begins by defining the overall scope and requirements of the deployment. Plan the following items:

The design phase involves defining the deployment architecture. Plan the following items:

The implementation phase involves deploying directory services. Plan the following items:

The automation and continuous integration phase involves using tools for testing. Plan the following items:

The supportability phase involves creating the runbook for system administrators and operators. This includes procedures for backup and restore operations, debugging, change control, and other processes.

If you have a ForgeRock Support contract, it ensures everything is in place prior to your deployment.

As described in "Managing Schema" in the Administration Guide, DS servers ship with many standard LDAP schema definitions. In addition, you can update LDAP schema definitions while the server is online.

This does not mean, however, that you can avoid schema updates for your deployment. Instead, unless the data for your deployment requires only standard definitions, you must add LDAP schema definitions before importing your data.

Once your schema modifications are ready, use comments to document your choices in the source LDIF. Keep the file under source control. Apply a change control process to avoid breaking schema definitions in the future.

Perhaps you can request object identifiers (OIDs) for new schema definitions from an OID manager in your organization. If not, either take charge of OID assignment, or else find an owner who takes charge. OIDs must remain globally unique, and must not be reused.

When defining the base DNs and hierarchical structure of the DIT, keep the following points in mind:

Once you have defined your DIT, arrange the directory data you import to follow its structure.

If client applications only use LDAP to access the directory service, you have few choices to make when configuring connection handlers. The choices involve how to secure connections, and whether to limit access by client host name or address mask. The client applications all have the same LDAP view of the directory data.

This is true for DSML applications as well if you use the DSML gateway.

If client applications use RESTful HTTP APIs to access the directory service, then you have the same choices as for LDAP. In addition, you must define how the HTTP JSON resources map to LDAP entries.

Mapping JSON Resources to LDAP Entries

As shown above, you can define multiple versioned APIs providing alternative views of the underlying LDAP data if required for your deployment. The basic sample API that ships with DS servers is likely not sufficient. For details, see "REST to LDAP Configuration" in the Reference.

In a shared or high-scale directory service, service management—installation and configuration, backup, and recovery—may be the responsibility of only a few specialists. These tasks may be carefully scripted. "Directory Service Administration" in the Security Guide lists service management tasks.

Directory data management is, however, often a task shared by multiple users. "Directory Data Administration" in the Security Guide lists tasks that these user administrators perform. Many of these tasks may be performed manually. In addition, users may be responsible for profile data in their own entry, including passwords, for example. As described in "Planning the DIT", you can arrange the DIT hierarchically to make it easier to scope control of administrative access.

Your plan must define who should have what access to which data, and list the privileges and access controls to grant such access. Read "Securing Directory Administration" in the Security Guide to review the alternatives.

As described in "Working With Groups of Entries" in the Developer's Guide, DS directory servers offer static, dynamic, and virtual static group implementations:

Consider your data and client applications. Use dynamic or virtual static groups whenever possible. When you cannot use dynamic or virtual static groups, use static groups and consider caching them in memory as described in "Caching Large, Frequently Used Entries" in the Administration Guide.

As described in "Working With Virtual and Collective Attributes" in the Developer's Guide, DS servers support virtual and collective attributes that let entries share attribute values. Sharing attribute values where it makes sense can significantly reduce data duplication, saving space and avoiding maintenance updates.

Consider your directory data. You can use virtual or collective attributes to replace attributes that repeat on many entries and can remain read-only on those entries. Familiar use cases include postal addresses that are the same for everyone in a given location, and class of service properties that depend on a service level attribute.

A directory server is an object-oriented database. It will therefore exhibit best performance when its data is cached in memory. This is as true for large static groups mentioned in "Using Groups" as it is for all directory data.

A disadvantage of caching all data is that systems with enough RAM are more expensive. Furthermore, the JVM can grow so large that garbage collection and heap dumps become unwieldy. Consider the suggestions in "Database Cache Settings" in the Administration Guide, testing the results for your data when planning your deployment. Decide how to configure cache sizes based on your observations.

DS servers have the following features that make them well-suited to serve identity data:

The fewer restrictions you place on an administrative account, the greater the danger the account will be misused.

As described in "About the Roles Directory Administrators Play" in the Security Guide, you can avoid using directory superuser accounts for most operations. Instead, limit administrator privileges and access to grant only what their roles require. The first high-level distinction to make is between operational staff who manage the service, and those users who manage directory data. Read the section cited for fine-grained distinctions.

When your deployment involves delegated administration, it is particularly important to grant only required access to the delegates. This is easier if your DIT supports appropriate access scopes by default, as described in "Planning the DIT".

An immutable configuration does not change at runtime. A mutable configuration does change at runtime.

With an immutable configuration, you maintain the server configuration as an artifact under source control, and manage changes by applying the same process you use for other source code. This approach helps prevent surprises in production configurations. If properly applied, there is little risk of rolling out an untested change.

With a mutable configuration, operational staff have more flexibility to make changes. This approach requires even more careful change management for test and production systems.

DS server configurations can be immutable, except for the portion devoted to replication, which evolves as peer servers come and go.

DS directory data, however, must remain mutable to support write operations. As long as you separate directory data from the configuration, this does not prevent you from replacing directory server replicas. As described in "Initializing Replicas" in the Administration Guide, new replicas can start with existing data sets.

DS servers provide both HTTP and LDAP access to directory data. As detailed in "Request Handling" in the Reference, HTTP access to directory data eventually translates to LDAP access internally. At the LDAP level, DS servers provide powerful, fine-grained access control.

The default server behavior is to refuse all access. All DS servers therefore grant some level of access through privileges as described above, and through access controls. For details, see "Securing Access to Directory Data" in the Security Guide.

Access control instructions (ACIs) in diretory data take the form of aci LDAP attributes, or global-aci properties in the server configuration. You write ACIs in a domain-specific language, described in "About ACIs" in the Administration Guide. The language lets you describe concisely who has access to what under what conditions. When configuring access control, notice that access controls apply beneath their location in the directory information tree. As a result, some ACIs, such as those granting access to LDAP controls and extended operations, must be configured for the entire server rather than a particular location in the data.

Administrative privileges provide a mechanism that is separate from access control to restrict what administrators can do.

You assign privileges to users as values of the ds-privilege-name LDAP attribute. You can assign privileges to multiple users with collective attribute subentries. For more information, see "About Privileges" in the Administration Guide.

Take care when granting privileges, especially the following privileges:

DS servers support a variety of authentication mechanisms.

When planning your service, use the following guidelines:

For additional suggesions and details, see "Securing Authentication" in the Security Guide.

DS directory proxy servers, and DS DSML and REST to LDAP gateway applications encapsulate DS directory services, and offer access to other directory services.

Unlike directory servers, directory proxy servers do not hold directory data, and so use global access policies rather than ACIs. You define global access policies as server configuration objects. For details, see "About Global Access Control Policies" in the Administration Guide.

As mentioned in "Directory Proxy System Resources", be aware that for high-performance services you may need to deploy as many proxy servers or gateways as directory servers.

For details about DS LDAP proxy services, see "Configuring LDAP Proxy Services" in the Administration Guide.

As described in "RESTful Client Access Over HTTP" in the Administration Guide, you can configure DS servers to provide RESTful HTTP access.

If you deploy this capability, you must configure how HTTP resources map to LDAP entries. This is explained in "Preparing Data Views". For details, see "REST to LDAP Configuration" in the Reference.

Although LDAP and RESTful HTTP access ensure high performance, your deployment may require a higher level of abstraction than LDAP or HTTP can provide.

Other ForgeRock Identity Platform™ components offer such higher-level abstractions. For example, ForgeRock Access Management software lets you plug into directory services for authentication and account profiles, and then orchestrate powerful authentication and authorization scenarios. ForgeRock Identity Management software can plug into directory services to store configuration and account profiles, to provide user self-services, and to synchronize data with a wide variety of third-party systems.

For an introduction to the alternatives, read the Identity Platform Guide.

Replication is designed to provide high availability with tolerance for network partitions. In other words, the service continues to allow both read and write operations when the network is down. Replication provides eventual consistency, not immediate consistency.

When the network connection is down between two replicas, replication is temporarily interrupted. Client applications continue to receive responses to their requests^[1], but clients making requests to different servers will not have the same view of the latest updates. The discrepancy in data on different replicas also arises temporarily when a high update load takes time to fully process across all servers.

Eventual consistency can be a trap for the unwary. The client developer who tests software only with a single directory server might not notice problems that become apparent when a load balancer spreads requests evenly across multiple servers. A single server is immediately consistent for its own data. Implicit assumptions about consistency therefore go untested.

For example, a client application that implicitly assumes immediate consistency might perform a write quickly followed by a read of the same data. Tests are all successful when only one server is involved. In deployment, however, a load balancer distributes requests across multiple servers. When the load balancer sends the read to a replica that has not yet processed the write, the client application appears to perform a successful write, followed by a successful read that is inconsistent with the write that succeeded!

When deploying replicated DS servers, keep this eventual consistency trap in mind. Educate developers about the trade off, review patches, and test and fix client applications under your control. In deployments with client applications that cannot be fixed, use affinity load balancing in DS directory proxy servers to work around broken clients. For details, see "Choosing Load Balancing Settings" in the Administration Guide.

In DS software, the role of a replication server is to transmit messages about updates. Directory servers receive replication messages from replication servers, and apply updates accordingly, meanwhile serving client applications.

Deploy at least two replication servers per local network in case one fails, and deploy more if you have many directory servers per replication server. For LAN-based deployments, each directory server can double as a replication server. For large, WAN-based deployments, consider using standalone replication servers and directory servers.

In a widely distributed deployment, be aware that replication servers all communicate with each other. Directory servers always communicate through replication servers, even if the replication service runs in the same server process as the directory server. By assigning servers to replication groups, you can ensure that directory servers only connect to local replication servers until they need to fail over to remote replication servers. This limits the WAN replication traffic to messages between replication servers, except when all replication servers on the LAN are down. For details, see "Standalone Replication Servers" in the Administration Guide and "Replication Groups" in the Administration Guide.

Deploy the replication servers first. You can think of them as providing a network service (replication) in the same way DNS provides a network service (name resolution). You therefore install and start replication servers before you add directory servers.

After you install a directory server and configure it as a replica, you must initialize it to the current replication state. There are a number of choices for this, as described in "Initializing Replicas" in the Administration Guide. Once a replica has been initialized, replication eventually brings its data into a consistent state with the other replicas. As described in "Trading Consistency for Availability", give a heavy update load or significant network latency, temporary inconsistency is expected. You can monitor the replication status to estimate when replicas will converge on the same data set.

Client applications can adopt best practices that work with eventual consistency as described in "Best Practices For Application Developers" in the Developer's Guide, "Modify: Using Optimistic Concurrency" in the Developer's Guide, and "Updating Resources" in the Developer's Guide. To work around broken client applications that assume immediate consistency, use affinity load balancing in DS directory proxy servers. For details, see "Choosing Load Balancing Settings" in the Administration Guide.

Some client applications need notifications when directory data changes. Client applications cannot participate in replication itself, but can get change notifications. For details, see "Change Notification For Your Applications" in the Administration Guide.

When scaling replicated directory services, keep the following rules in mind:

When a replica writes an update to its directory data set, it transmits the change information to its replication server for replay elsewhere. The replication server transmits the information to connected directory servers, and to other replication servers replicating the same data. Those replication servers transmit the message to others until all directory servers have received the change information. Each directory server must process the change, reconciling it with other change information.

As a result, you cannot scale up write capacity by adding servers. Each server must replay all the writes.

If necessary, you can scale up write capacity by increasing the capacity of each server (faster disks, more powerful servers) or by splitting the data into separate sets that you replicate independently (data distribution).

In shared directory service deployments, the directory must continue serving client requests during maintenance operations, including service upgrades, during network outage recovery, and in spite of system failures.

DS replication lets you build a directory service that is always online. DS directory proxy capabilities enable you to hide maintenance operations from client applications. You must still plan appropriate use of these features, however.

As described above, replication lets you use redundant servers and systems that tolerate network partitions. Directory server replicas continue to serve requests when peer servers fail or become unavailable. Directory proxy servers route around directory servers that are down for maintenance or down due to failure. When you upgrade the service, you roll out one upgraded DS server at a time. New servers continue to interoperate with older servers, so the whole service never goes down. All of this depends on deploying redundant systems, including network links, to eliminate single points of failure. For more, see "Building a Highly Available Directory".

As shown in that section, your deployment may involve multiple locations, with servers communicating locally over LANs and remotely over WANs. Your deployment can also use separate replication topologies, for example, in order to sustain very high write loads, or to separate volatile data from more static data. Carefully plan your load balancing strategy to offer good service at a reasonable cost. By using replication groups, you can limit most replication traffic over WAN links to communications between replication servers. Directory proxy servers can direct client traffic to local servers until it becomes necessary to failover to remote servers.

Sound operational procedures play as important a role in availability as good design. Operational staff maintaining the directory service must be well-trained and organized so that someone is always available to respond if necessary. They must have appropriate tools to monitor the service in order to detect situations that need attention. When maintenance, debugging, or recovery is required, they should have a planned response in most cases. Your deployment plans should therefore cover the requirements and risks that affect your service, as described in the other sections of this chapter.

Before finalizing deployment plans, make sure that you understand key availability features in detail. For details about replication, read "Managing Data Replication" in the Administration Guide. For details about proxy features, read "Configuring LDAP Proxy Services" in the Administration Guide.

Functional testing validates that specified test cases work with the software considered as a black box.

As ForgeRock already tests DS servers and gateways functionally, focus your functional testing on customizations and service level functions. For each key capability, devise automated functional tests. Automated tests make it easier to integrate new deliveries to take advantage of recent bug fixes and to check that fixes and new features do not cause regressions.

As part of the overall plan, include not only tasks to develop and maintain your functional tests, but also to provision and to maintain a test environment in which you run the functional tests before you significantly change anything in your deployment. For example, run functional tests whenever you upgrade any server or custom component, and analyze the output to understand the effect on your deployment.

With written SLOs, even if your first version consists of guesses, you turn performance plans from an open-ended project to a clear set of measurable goals for a manageable project with a definite outcome. Therefore, start your testing with service level objectives clear definitions of success.

Also, start your testing with a system for load generation that can reproduce the traffic you expect in production, and underlying systems that behave as you expect in production. To run your tests, you must therefore generate representative load data and test clients based on what you expect in production. You can then use the load generation system to perform iterative performance testing.

Iterative performance testing consists of identifying underperformance and the bottlenecks that cause it, and discovering ways to eliminate or work around those bottlenecks. Underperformance means that the system under load does not meet service level objectives. Sometimes resizing or tuning the system can help remove bottlenecks that cause underperformance.

Based on SLOs and availability requirements, define acceptance criteria for performance testing, and iterate until you have eliminated underperformance.

Tools for running performance testing include the tools listed in "Testing Performance" in the Administration Guide, and Gatling, which uses a domain specific language for load testing. To mimic the production load, examine both the access patterns and also the data that DS servers store. The representative load should reflect the distribution of client access expected in production.

Although you cannot use actual production data for testing, you can generate similar test data using tools, such as the makeldif command.

As part of the overall plan, include not only tasks to develop and maintain performance tests, but also to provision and to maintain a pre-production test environment that mimics your production environment. Security measures in your test environment must also mimic your production environment, as security measures can impact performance.

Once you are satisfied that the baseline performance is acceptable, run performance tests again when something in your deployment changes significantly with respect to performance. For example, if the load or number of clients changes significantly, it could cause the system to underperform. Also, consider the thresholds that you can monitor in the production system to estimate when your system might start to underperform.

Deployment testing is a description rather than a term in the context of this guide. It refers to the testing implemented within the deployment window after the system is deployed to the production environment, but before client applications and users access the system.

Plan for minimal changes between the pre-production test environment and the actual production environment. Then test that those changes have not cause any issues, and that the system generally behaves as expected.

Take the time to agree upfront with stakeholders regarding the acceptance criteria for deployment tests. When the production deployment window is small, and you have only a short time to deploy and test the deployment, you must trade off thorough testing for adequate testing. Make sure to plan enough time in the deployment window for performing the necessary tests and checks.

Include preparation for this exercise in your overall plan, as well as time to check the plans close to the deployment date.

DS servers store configuration in files under the server's config directory. When you set up a server, the setup process creates the initial configuration files based on templates in the server's template directory. "File Layout" in the Reference describes the files.

When a server starts, it reads its configuration files to build an object view of the configuration in memory. This view holds the configuration objects, and the constraints and relationships between objects. This view of the configuration is accessible over client-side and server-side APIs. Configuration files provide a persistent, static representation of the configuration objects.

Configuration tools use the client-side API to discover the server configuration and to check for constraint violations and missing relationships. The tools prevent you from breaking the server configuration structurally by validating structural changes before applying them. The server-side API allows the server to validate configuration changes, and to synchronize the view of the configuration in memory with the file representation on disk. If you make changes to the configuration files on disk while the server is running, the server can neither validate the changes beforehand, nor guarantee that they are in sync with the view of the configuration in memory.

"DS Server Configuration" summarizes the configuration options.

Once a server begins to replicate data with other servers, the part of the configuration pertaining to replication is specific to that server. As a result, a server effectively cannot be cloned once it has begun to participate in data replication. When deploying servers, do not initialize replication until you have deployed the server.

You edit files to configure DS DSML and REST to LDAP gateway web applications.

The gateways do not have external configuration APIs, and must be restarted after you edit configuration files for the changes to take effect.

Before building a test directory service, start sizing systems by considering service level objectives (SLOs) and directory data.

Define SLOs as described in "Defining Performance Requirements and Constraints" in the Administration Guide. Once you have defined the SLOs, model directory client traffic to test them using your own tools or the tools described in "Testing Performance" in the Administration Guide.

For a directory server, estimate the memory required to cache the database, as described in "Database Cache Settings" in the Administration Guide.

When scaling up on appropriately powerful server systems, each system must have the resources to run a high-scale DS server. As described in "Scaling Replication", a directory server replica is only required to absorb its share of the full read load. But each replica must be able to absorb the full write load for the service.

Make sure that the estimates you arrived at in "Planning for High Scale" remain within the capabilities of each server and system.

When scaling out onto multiple server systems, you must find a usable way to distribute or shard the data into separate replication domains. In some cases, each replication domain holds a branch of the DIT with a similar amount of traffic and an equivalent amount of data. Entries could then be distributed based on location or network or some other attribute. Branches could join at a base DN that brings all the entries together in the same logical view.

Separate at least the directory server replicas in each replication domain, so that they share only minimal and top-level entries. To achieve this, use subtree replication, which is briefly described in "Subtree Replication" in the Administration Guide. Each replica can hold minimal and top-level entries in one database backend, but its primary database backend holds only the branch it shares with others in the domain.

If the data to scale out is all under a single DN, consider using a DS proxy server layer to perform the data distribution as described in "Scaling Out Using Data Distribution".

DS directory proxy servers can distribute data across multiple shards. Proxy settings are described in "Configuring LDAP Proxy Services" in the Administration Guide. This section shows you how to try data distribution for AM Core Token Service (CTS) tokens.

The example that follows is intended for evaluation on a single computer. It involves installing five DS servers and one AM server:

Sample CTS Store With Data Distribution

Data replication is critical to a high-scale, highly available directory service. For deployments where data protection is also critical, you must, however, make sure that you do not replicate data outside locations where you can guarantee compliance with local regulations.

As described in "Deploying Replication", replication messages flow from directory servers through replication servers to other directory servers. Replication messages contain data that has changed, including data governed by privacy regulations:

As described in "Replication Per Suffix" in the Administration Guide, the primary unit of replication is the suffix, specified by a base DN. Subtree replication refers to putting different subtrees (branches) in separate backends, and then replicating those subtrees only to specified servers. For example, you can ensure that the data replicates only to locations where you can guarantee compliance with the regulations in force.

For subtree replication, the RDN of the subtree base DN identifies the subtree. This leads to a hierarchical directory layout. The directory service retains the logical view of a flatter layout, because the branches all join at a top-level base DN.

The example uses nested groups to avoid referencing local accounts at the top level, but still allowing users to belong to top-level groups:

# %<--- Start of LDIF for userRoot --->%
# Base entries are stored in the userRoot backend:
dn: dc=example,dc=com                         # Base DN of userRoot backend
...

dn: ou=groups,dc=example,dc=com               # Stored in userRoot backend
...

dn: ou=Top-level Group,ou=groups,dc=example,dc=com
...
member: ou=R1 Group,ou=groups,ou=Region 1,dc=example,dc=com
member: ou=R2 Group,ou=groups,ou=Region 2,dc=example,dc=com

dn: ou=people,dc=example,dc=com               # Stored in userRoot backend
...

# %<--- End of LDIF for userRoot --->%
# %<--- Start of LDIF for Region 1 --->%
# Subtree entries are stored in a country or region-specific backend.
dn: ou=Region 1,dc=example,dc=com             # Base DN of region1 backend
...

dn: ou=groups,ou=Region 1,dc=example,dc=com   # Stored in region1 backend
...

dn: ou=R1 Group,ou=groups,ou=Region 1,dc=example,dc=com
...
member: uid=aqeprfEUXIEuMa7M,ou=people,ou=Region 1,dc=example,dc=com
...

dn: ou=people,ou=Region 1,dc=example,dc=com   # Stored in region1 backend
...

dn: uid=aqeprfEUXIEuMa7M,ou=people,ou=Region 1,dc=example,dc=com
uid: aqeprfEUXIEuMa7M
...

# %<--- End of LDIF for Region 1 --->%
# %<--- Start of LDIF for Region 2 --->%
dn: ou=Region 2,dc=example,dc=com             # Base DN of region2 backend
...

dn: ou=groups,ou=Region 2,dc=example,dc=com   # Stored in region2 backend
...

dn: ou=groups,ou=R2 Group,ou=Region 2,dc=example,dc=com
...
member: uid=8EvlfE0rRa3rgbX0,ou=people,ou=Region 2,dc=example,dc=com
...

dn: ou=people,ou=Region 2,dc=example,dc=com   # Stored in region2 backend
...

dn: uid=8EvlfE0rRa3rgbX0,ou=people,ou=Region 2,dc=example,dc=com
uid: 8EvlfE0rRa3rgbX0
...

# %<--- End of LDIF for Region 2 --->%

Separate Replication Domains for Data Sovereignty

Mixed Replication Domains for Data Sovereignty

When you use subtree replication in this way, client applications can continue to read and update directory data as they normally would. Directory servers only return data that is locally available.

For additional information, see "Subtree Replication" in the Administration Guide, and "Splitting Data Across Multiple Backends" in the Administration Guide.

In some deployments, regulations allow you to replicate some user attributes. For example, consider a deployment where data sovereignty regulations in one region allow UIDs and class of service levels to be present everywhere, but do not allow users' personal information to leave their location.

Consider the following entry where the attribute values are preceded by the comment, # Can be replicated everywhere, are the values that you can replicate outside the user's region:

dn: uid=aqeprfEUXIEuMa7M,ou=people,ou=Region 1,dc=example,dc=com
objectClass: top
objectClass: cos
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
# Can be replicated everywhere:
classOfService: bronze
cn: Babs Jensen
cn: Barbara Jensen
facsimiletelephonenumber: +1 408 555 1992
gidNumber: 1000
givenname: Barbara
homeDirectory: /home/bjensen
l: Region 1
mail: bjensen@example.com
manager: uid=2jD5NanzOZGjMmcz,ou=people,ou=Region 1,dc=example,dc=com
ou: People
ou: Product Development
preferredLanguage: en, ko;q=0.8
roomnumber: 0209
sn: Jensen
telephonenumber: +1 408 555 1862
# Can be replicated everywhere:
uid: aqeprfEUXIEuMa7M
uidNumber: 1076
userpassword: {SSHA512}2olpp6P5dWDZyYmOQwyk...9fOSlN4ZrpJcrUz2qVMiRpplkI

To allow you to replicate only a portion of each entry, DS servers implement fractional replication. You configure fractional replication by updating the directory server configuration to specify which attributes to include or exclude in change messages from replication servers to the directory server replica.

The replication server must remain located with the directory server replicas that hold full entries which include all attributes. The replication server can receive updates from these replicas, and also from replicas that hold fractional entries. Each replication server must therefore remain within the location where the full entries are processed. Otherwise, replication messages describing changes to protected attributes would be sent outside the location where the full entries are processed.

Fractional Replication for Protected Data

To leave schema checking enabled on the replicas that receive fractional updates, portions of entries that are replicated must themselves be complete entries. In other words, in the example above, the entry's structural object class would have to allow classOfService and uid. This would require editing the schema and the objectClass values of the entries. For details, see "Managing Schema" in the Administration Guide.

For additional information, see "Fractional Replication" in the Administration Guide.

Adding a directory proxy layer can help you deploy alongside an existing directory service. The proxy layer lets you provide a single entry point to both new and old directory services.

You configure a directory proxy server to connect to servers in each directory. DS proxy servers can discover DS directory servers by connecting to DS replication servers. For other directories, you must statically enumerate the directory server to contact. DS proxy servers work with any LDAP directory server that supports the standard proxied authorization control defined in RFC 4370.

Each DS proxy server forwards client requests to the directory service based on the target DN of the operation. As long as the base DNs for each directory service differ, the proxy layer can provide a single entry point to multiple directory services.

For details, see "Configuring LDAP Proxy Services" in the Administration Guide, and in particular "Deploying a Single Point of Directory Access" in the Administration Guide.

For cases where an existing directory service holds authentication credentials, DS servers provide a feature called pass-through authentication.

With pass-through authentication, the DS server effectively redirects LDAP bind operations to a remote LDAP directory service. If the DS and remote user accounts do not have the same DN, you configure the DS server to automatically map local entries to the remote entries. Pass-through authentication can cache passwords if necessary for higher performance with frequent authentication.

For details, see "Configuring Pass-Through Authentication" in the Administration Guide.

You may need to continually synchronize changes across multiple services, or to migrate data from an existing directory service.

For ongoing data synchronization across multiple services, consider ForgeRock Identity Management software or a similar solution. ForgeRock Identity Management software supports configurable data reconciliation and synchronization at high scale and with multiple data sources, including directory services.

Not all directory clients expect the same directory data. Clients might even expect completely different identity objects.

DS servers expose the same LDAP data view to all directory clients. (You can adjust this behavior somewhat for update operations as described in "Filtering Add and Modify Operations" in the Developer's Guide.)

The RESTful views of directory data for HTTP clients are fully configurable, however. By developing alternative REST to LDAP mappings and exposing multiple APIs, or different versions of the same API, you can present directory data in different ways to different applications. For details, see "To Set Up REST Access to User Data" in the Administration Guide, and "REST to LDAP Configuration" in the Reference.

Directory servers consume significant CPU resources when processing username-password authentications where the password storage scheme is computationally intensive (Bcrypt, PBKDF2, PKCS5S2).

DS servers also use CPU resources to decode requests and encode responses, and to set up secure connections. LDAP is a connection-oriented protocol, so the cost of setting up a connection may be small compared to the lifetime of the connection. HTTP, however, requires a new connection for each operation. If you have a significant volume of HTTPS traffic, provision enough CPU resources to set up secure connections.

Directory server memory requirements depend primarily on how you cache directory data. Be aware that default settings cause the directory server to request half the available memory for each directory database backend. At minimum, therefore, you must revisit this setting if you have more than one directory database backend. If your directory data set can fit entirely into system memory and you want to optimize for fast reads and searches, provision enough RAM to cache everything.

If caching the entire directory data set is not an option, provision enough RAM to cache at least all internal database nodes. For details, see "Database Cache Settings" in the Administration Guide.

DS servers also use memory to maintain active connections and processes. As indicated in "Fulfilling Memory Requirements", provision an additional minimum of 2 GB RAM or more depending on the volume of traffic to your service.

When sizing network connections, account for all requests and responses, including replication traffic. When calculating request and response traffic, base your estimates on your key client applications. When calculating replication traffic, be aware that all write operations must be communicated over the network, and replayed on each directory server. Each write operation results in at least N-1 replication messages, where N is the total number of servers. Also, keep in mind that all DS servers running a replication service are fully connected, including those servers that are separated by WAN links.

For deployments in multiple regions, account especially for traffic over WAN links, as this is much more likely to be an issue than traffic over LAN links.

Make sure to size enough bandwidth for peak throughput, and do not forget redundancy for availability.

The largest disk I/O loads for DS servers arise from logging and writing directory data. You can also expect high disk I/O when performing a backup operation or exporting data to LDIF.

I/O rates depend on the service levels that the deployment provides. When you size disk I/O and disk space, you must account for peak rates and leave a safety margin when you must briefly enable debug logging to troubleshoot any issues that arise.

Also, keep in mind the possible sudden I/O increases that can arise in a highly available service when one server fails and other servers must take over for the failed server temporarily.

DS server access log files grow more quickly than other logs. Default settings prevent each access logger's files from growing larger than 2 GB before removing the oldest. If you configure multiple access loggers at once, multiply 2 GB by their number.

Directory server database backend size grows as client applications modify directory data. Even if data set's size remains constant, the size of the backend grows. Historical data on modified directory entries increases until purged by the directory server when it reaches the replication purge delay (default: 3 days). In order to get an accurate disk space estimate, follow the process described in "Planning for High Scale".

Replication server changelog backend size is subject to the same growth pattern as historical data. Run the service under load until it reaches the replication purge delay to estimate disk use.

For highest performance, use fast SSD disk and separate disk subsystems logging, backup, and database backends.

The DS DSML and REST to LDAP gateway web applications run on any supported application server, regardless of the underlying operating system.

The only persistent state for gateway applications is in their configuration files. The gateway configuration files are portable across application servers and operating systems.

When installing a directory server for evaluation, you need 256 MB memory (32-bit) or 1 GB memory (64-bit) available.

For installation in production, read the rest of this section. You need at least 2 GB memory for a directory server and four times the disk space needed for initial production data in LDIF format. A replicated directory server stores data, indexes for the data, operational attribute data, and historical information for replication. The server configuration trades disk space for performance and resilience, compacting and purging data for good performance and for protection against temporary outages. In addition, leave space for growth in database size as client applications modify and add entries over time.

For a more accurate estimate of the disk space needed, import a known fraction of the initial LDIF with the server configured for production. Run tests to estimate change and growth in directory data, and extrapolate from the actual space occupied in testing to estimate the disk space required in production.

Directory servers almost always benefit from caching all directory database files in system memory. Reading from and writing to memory is much faster than reading from and writing to disk storage.

For large directories with millions of user directory entries, there might not be room to install enough memory to cache everything. To improve performance in such cases, use quality solid state drives either for all directory data, or as an intermediate cache between memory and disk storage.

To evaluate DS software, make sure you have 10 GB free disk space for the software and for sample data.

The more data you have, the more disk space you need. Before deploying production systems, make sure you have enough space. For details, see "Planning for High Scale".

Processor architectures that provide fast single thread execution tend to help Directory Services software deliver the lowest response times. For top-end performance in terms of sub-millisecond response times and of throughput ranging from tens of thousands to hundreds of thousands of operations per second, the latest x86/x64 architecture chips tend to perform better than others.

When deploying DS servers with replication enabled, allow at minimum two CPU cores per server. Allow more CPU cores per server, especially in high-volume deployments or when using CPU-intensive features such as encryption. Single CPU systems seriously limit server performance.

Chip multi-threading (CMT) processors can work well for directory servers providing pure search throughput, though response times are higher. However, CMT processors are slow to absorb hundreds or thousands of write operations per second. Their slower threads get blocked waiting on resources, and thus are not optimal for deployments with high write throughput requirements.

On systems with fast processors and enough memory to cache directory data completely, the network can become a bottleneck. Even if a single 1 Gb Ethernet interface offers plenty of bandwidth to handle your average traffic load, it can be too small for peak traffic loads. Consider using separate interfaces for administrative traffic and for application traffic.

To estimate the network hardware required, calculate the size of the data returned to applications during peak load. For example, if you expect to have a peak load of 100,000 searches per second, each returning a full 8 KB entry, you require a network that can handle 800 MB/sec (3.2 Gb/sec) throughput, not counting other operations, such as replication traffic.

For a directory server, storage hardware must house both directory data, including historical data for replication, and server logs. On a heavily used server, you might improve performance by putting access logs on dedicated storage.

Storage must keep pace with throughput for write operations. Write throughput can arise from modify, modify DN, add, and delete operations, and from bind operations when a login timestamp is recorded, and when account lockout is configured, for example.

In a replicated topology, a directory server writes entries to disk when they are changed, and a replication server writes changelog entries. The server also records historical information to resolve potential replication conflicts.

As for network throughput, base storage throughput required on peak loads rather than average loads.

DS servers need to be able to open many file descriptors, especially when handling thousands of client connections. Linux systems in particular often set a limit of 1024 per user, which is too low to handle many client connections to the DS server.

When setting up your DS server for production use, make sure the server can use at least 64K (65536) file descriptors. For example, when running the server as user opendj on a Linux system that uses /etc/security/limits.conf to set user level limits, you can set soft and hard limits by adding these lines to the file:

opendj soft nofile 65536
opendj hard nofile 131072

The example above assumes the system has enough file descriptors available overall. You can check the Linux system overall maximum as follows:

$ cat /proc/sys/fs/file-max
204252

A directory server backend database monitors file events. On Linux systems, backend databases use the inotify API for this purpose. The kernel tunable fs.inotify.max_user_watches indicates the maximum number of files a user can watch with the inotify API. Make sure this tunable is set to at least 512K:

$ sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 524288

If this tunable is set lower than that, change it as shown in the following example:

$ sudo sysctl --write fs.inotify.max_user_watches=524288
[sudo] password for admin:
fs.inotify.max_user_watches = 524288

The data in different DevOps environments will not be the same. For example, in the development environment, you are likely to use only a few generated entries in each backend. In the production environment, you might have millions of entries subject to live updates at any time in backends that you maintain for years.

When installing directory replicas, create a separate backend for each base DN you will use in production. Also, create at least a base entry at setup time for each backend.

The setup process requires a keystore holding public key certificates and private keys for secure communications with other applications. In a DevOps deployment, the default is to generate a file-based keystore that contains public and private keys. You should instead install the server with your own keystore or other method of accessing keys.

The setup process also creates a file-based keystore, db/ads-truststore, to hold the keys for replication and symmetric key distribution. Let the server manage this keystore, and make it part of your template replica image. Do not replace it with settings from the environment. The mechanism for allowing you to manage these keys as part of the DevOps environment is evolving.

While preparing DS replicas in your development environment, configure the servers as replicas of each other as described in "Configuring Replication Settings" in the Administration Guide.

Some subcommands of the dsreplication command, such as status, are not compatible with configuration expressions. Do not use the dsreplication command after initial setup. After you use expressions in configuration settings, monitor replication delay over HTTP or LDAP instead.

$ dsconfig \
 set-global-configuration-prop \
 --set server-id:\&{ds.server.id} \
 --offline \
 --configFile /path/to/opendj/config/config.ldif \
 --no-prompt

$ dsconfig \
 set-sasl-mechanism-handler-prop \
 --handler-name DIGEST-MD5 \
 --set server-fqdn:\&{ds.fqdn} \
 --offline \
 --configFile /path/to/opendj/config/config.ldif \
 --no-prompt

$ dsconfig \
 set-backend-prop \
 --backend-name userRoot \
 --set enabled:\&{ds.userRoot.enabled\|true} \
 --offline \
 --configFile /path/to/opendj/config/config.ldif \
 --no-prompt

$ dsconfig \
 set-replication-domain-prop \
 --provider-name Multimaster\ Synchronization \
 --domain-name dc=example,dc=com \
 --set enabled:\&{ds.userRoot.enabled\|true} \
 --offline \
 --configFile /path/to/opendj/config/config.ldif \
 --no-prompt

$ dsconfig \
 set-replication-domain-prop \
 --provider-name Multimaster\ Synchronization \
 --domain-name cn=admin\ data \
 --set replication-server:\&{ds.replication.servers} \
 --offline \
 --configFile /path/to/opendj/config/config.ldif \
 --no-prompt
$ dsconfig \
 set-replication-domain-prop \
 --provider-name Multimaster\ Synchronization \
 --domain-name cn=schema \
 --set enabled:\&{ds.schema.replication.enabled\|false} \
 --set replication-server:\&{ds.replication.servers} \
 --offline \
 --configFile /path/to/opendj/config/config.ldif \
 --no-prompt
$ dsconfig \
 set-replication-domain-prop \
 --provider-name Multimaster\ Synchronization \
 --domain-name dc=example,dc=com \
 --set replication-server:\&{ds.replication.servers} \
 --offline \
 --configFile /path/to/opendj/config/config.ldif \
 --no-prompt

$ dsconfig \
 set-replication-server-prop \
 --provider-name Multimaster\ Synchronization \
 --set replication-server:\&{ds.replication.servers} \
 --offline \
 --configFile /path/to/opendj/config/config.ldif \
 --no-prompt

At this point, your server is ready to be used as a template. For additional details on setting expression values before starting the server, see "Using Configuration Property Value Substitution" in the Administration Guide.

The recommended way to apply further configuration changes in your DevOps development environment is to run the dsconfig command interactively, and store the changed configuration file config/config.ldif under source control.

When run in interactive mode, the dsconfig command displays the equivalent command for each update you make. You can store the commands displayed in your source control commit messages that explain each change.

Editing the configuration manually is not recommended, as config/config.ldif is not a supported public interface. It can change without notice between releases.

The very first time that you build a new service in an environment, you might already have existing application data. If you do not, you can skip this section.

Backup and restore operations are the same in DevOps deployments as in static deployments. DevOps deployments will automate the restore process as much as possible to limit administrator interaction.

When you bring up a new replica in an existing environment to replace an existing replica, mount the replica's data from persistent storage at the proper mount points and start the server. Only initialize from backup if you suspect the persisted data is corrupt.

If the new replica is an upgrade, perform a data-only upgrade before starting the server as described in "Upgrading".

When adding a replica based on your template, you add a server that is both a directory server and a replication server. You must initialize the data and update the runtime configuration of other replicas to make them aware of the new replication server.