How To
ForgeRock Identity Platform
Does not apply to Identity Cloud

How do I identify reconciliation performance issues in IDM (All versions)?

Last updated Apr 8, 2021

The purpose of this article is to provide guidance on isolating and diagnosing IDM reconciliation performance issues. This article also includes tips on tuning reconciliation performance based on your findings.


1 reader recommends this article

Overview

Isolating the cause(s) of performance bottlenecks within a large IDM topology comes down to a process of elimination. Identifying bottlenecks requires the system to be broken down to its individual components; each component must then be tested and tuned to achieve the necessary throughput.

For example, given the following basic topology there are multiple components which might limit the throughput of the system:

Although far from exhaustive, the following are some of the possible bottlenecks which might be encountered within the above topology:

  • Source LDAP Read ops/s
  • Target LDAP Read/Write ops/s
  • PostgreSQL Read/Write ops/s
  • JVM Heap limitations
  • Host Disk I/O limitations
  • Host CPU limitations
  • Undersized / incorrectly configured virtualization
  • IDM configuration issues
  • IDM triggers/custom code
  • IDM product defect
Note

One configuration change that can help with reconciliation performance is to increase the number of connector instances in the connection pool as detailed in How do I configure pooled connections for a connector in IDM (All versions)?

Reconciliation performance

Measuring the performance of individual components within the topology can be done outside the scope of IDM by using external benchmarking tools and processes or via IDM itself. Benchmarking components via IDM consists of using targeted operations and configuration changes to isolate a single component within the system and measure its performance as it relates to reconciliation.

You should consider the following components:

External source system

During reconciliation, IDM queries the source system in order to obtain the complete list of source object IDs to be reconciled. It may also perform multiple queries to retrieve the complete source object during the reconciliation process or rely on a cache of the source objects returned by the configured sourceQuery.

To begin to understand the performance of the source system, the following can be performed:

  • LDAP searchrate command to measure raw read ops capability. This command is provided in the OpenDJ LDAP Toolkit.
  • Measure execution time via targeted queries against the source system using IDM REST APIs:
    • Execute the generic ‘query-all-ids’ query against the source system.
    • Execute any ‘sourceQuery’ queries which have been configured.
    • Execute individual source object queries via Query FIlter, specifying the individual source object attributes use within the configured ‘sourceQuery’ queries.

Based on the results of the above actions, tune the source system to resolve performance bottlenecks. Examples of things which might reduce read performance on a source LDAP system are:

  • Lack of necessary indexes, resulting in un-indexed (full DB scans) searches.
  • Insufficient JVM Heap.
  • Insufficient CPU resources to handle the load.
  • Insufficient Disk I/O performance (non-SSD drives) to handle the load.

External target system

During reconciliation, IDM queries the target system in order to obtain the complete list of target object IDs to be reconciled. It may also perform multiple queries to retrieve the complete target object during the reconciliation process or rely on a cache of the target objects returned by the configured ‘targetQuery’. Target objects calculated based on the configured mappings are then written to the target system.

To begin to understand the performance of the target system, the following can be performed:

  • LDAP searchrate and modrate commands to measure raw read/write ops capability. These commands are provided in the OpenDJ LDAP Toolkit.
  • Measure execution time via targeted queries against the target system using IDM REST APIs:
    • Execute the generic ‘query-all-ids’ query against the target system.
    • Execute any ‘targetQuery’ queries which have been configured.
    • Execute any ‘correlationQuery’ queries which have been configured.
    • Execute individual target object queries via Query FIlter, specifying the individual target object attributes use within both the configured ‘targetQuery’ and ‘correlationQuery’ queries.

Based on the results of the above actions, tune the target system to resolve performance bottlenecks. Examples of things which might reduce write performance on a target LDAP system are:

  • Excessive indexes, resulting in increased overhead when writing entries. Specifically take note that substring indexes are extremely costly and may greatly reduce performance.
  • Insufficient JVM Heap.
  • Insufficient CPU resources to handle the load.
  • Insufficient Disk I/O performance (non-SSD drives) to handle the load.

JDBC repository

In order to isolate the JDBC repository from the performance of the reconciliation engine or the target system, the following can be performed:

Read/Write Test

Mapping: LDAP Source -> Managed Object

Situation Action
ABSENT CREATE (Default)

Read Test

Mapping: Managed Object -> LDAP Target

Situation Action
ALL ASYNC (Read Only)

Reconciliation engine

Note

Measuring the performance of the Reconciliation Engine should be done only after having resolved any performance issues observed during testing of the source and target systems.

The IDM reconciliation engine is responsible for the bidirectional synchronization of objects between different data stores. In the case of our topology above, this is the synchronization of the objects between the source and target LDAP systems.

During reconciliation IDM determines the state of the source and target objects (situation assessment), identifies the necessary actions to be performed and performs the actions against the source and target systems. Included within each phase of the reconciliation process are various JavaScript triggers which may or may not impact the performance of the reconciliation process depending on their efficiency.

In order to isolate the reconciliation engine from the performance of the IDM repository or the target system, the following can be performed:

Read-Only Test

Mapping: LDAP Source -> LDAP Target

Situation Action
ALL ASYNC (Read Only)
Note

The following should be performed with a source system which has been populated with production-like data and an EMPTY target system.

  1. Set the actions associated with all the situation policies to ‘Read-Only’ via the IDM Admin UI. This ensures that during the reconciliation process no actions are performed and that nothing is written to the IDM repository, or the target system.
  2. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM instance and JVM heap usage via verbose GC logging.

Read-Write Test

Mapping: LDAP Source -> LDAP Target

Situation Action
ABSENT CREATE (Default)
  1. Set the action associated with the ABSENT situation policy to CREATE. Set all other situation policies to ‘Read Only’. This ensures that during the reconciliation process objects are created and links are established between the source and target objects. No other actions are performed.
  2. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM instance and JVM heap usage via verbose GC logging.

Link Test

Mapping: LDAP Source -> LDAP Target

Situation Action
FOUND LINK (Custom)
  1. Ensure you have defined your desired ‘correlationQuery’ within your mapping.
  2. Delete the contents of the ‘links’ table within the IDM repository.
  3. Set the action associated with the FOUND situation policy to LINK. Set all other situation policies to ‘Read Only’. This ensures that during the reconciliation process the ‘correlationQuery’ will be executed to correlate the existing target objects (created during the read-write test) to source objects. For each correlated object a new link will be written to the IDM repository.
  4. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM instance and JVM heap usage via verbose GC logging.

Source Phase

Mapping: LDAP Source -> LDAP Target

Situation Action
CONFIRMED Update (Default)
  1. Set the ‘runTargetPhase’ property within the mapping to ‘false’. This ensures that ONLY the source phase of the reconciliation process is executed.
  2. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM instance and JVM heap usage via verbose GC logging.

Target Phase

Mapping: LDAP Source -> LDAP Target

Situation Action
CONFIRMED Update (Default)
  1. Delete the contents of the ‘links’ table within the IDM repository.
  2. Set the ‘sourceQuery’ within the mapping to point to a single specific source object. The goal is to reduce the scope of the source objects processed by the reconciliation engine down to a single object.
  3. Set the ‘runTargetPhase’ property within the mapping to ‘true’. Ensure that either you do NOT define a ‘targetQuery’ or that your ‘targetQuery’ is set to your desired value within production. Do not set the ‘targetQuery’ to point to a single object as is done with the ‘sourceQuery’. This ensures that all the target objects will be processed by the target phase of the reconciliation.
  4. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM instance and JVM heap usage via verbose GC logging.

See Also

Synchronization Guide › Tuning Reconciliation Performance

How do I collect data for troubleshooting high CPU utilization on IDM (All versions) servers?

How do I collect JVM data for troubleshooting IDM (All versions)?

How do I enable Garbage Collector (GC) Logging for IDM?

How do I change the JVM heap size for IDM (All versions)?

Best practice for JVM Tuning with G1 GC

Best practice for JVM Tuning with CMS GC

How do I troubleshoot issues with my indexes in DS (All versions)?

Maintenance Guide › Performance Tuning

Related Training

N/A

Related Issue Tracker IDs

N/A


Copyright and Trademarks Copyright © 2021 ForgeRock, all rights reserved.