Performance tuning and monitoring ForgeRock products

This book provides information on performance tuning and monitoring ForgeRock products (AM/OpenAM, DS/OpenDJ, IDM/OpenIDM and IG/OpenIG).


Best practice for JVM Tuning

The purpose of this article is to provide best practice advice on JVM tuning ; including key symptoms, understanding heap space and GC logs. This best practice advice applies to AM/OpenAM, DS/OpenDJ, IDM/OpenIDM and IG/OpenIG. JVM tuning is not an exact science and will vary across individual environments and applications. Consistent load and stress testing of your environment will be the only true way to gauge the effect of changes made.

JVM tuning considerations

Before you begin JVM tuning, you should consider the following points:

  • Costs - depending on your environments, it may be more cost effective to add more hardware to improve performance rather than spending the time to tune.
  • Desired outcome - tuning for stability is more important in the long run than tuning for performance, although there is an overlap.
  • Ongoing problems - tuning may only delay or mask a problem temporarily if there is an underlying issue you need to resolve.
  • Memory leaks - these will always cause Garbage Collection (GC) issues regardless of tuning.

What is Java® heap and how do I know it needs tuning?

The Java heap is the amount of memory allocated to applications running in the JVM. Objects in heap memory can be shared between threads. Garbage collection (GC) refers to the automatic process of managing the runtime memory. The JVM is subject to GC pauses that vary in frequency and duration; if these pauses become more frequent and/or last longer, you will see issues (such as application pauses) which indicate that you may need to tune your Java heap. The heap is initially created when the JVM starts and is split into different spaces or generations, the key ones being Young (Eden or Nursery) and Tenured (Old):

  • The Young generation is used for new objects. A GC process (ParNew) automatically runs when it becomes full which removes unused objects and moves all objects that have lived long enough to the Tenured generation, thereby freeing up space in the Young generation for more new objects. Typically objects in the Young generation are short lived and temporary. The Young generation is small but active and GC happens frequently but with limited impact on performance.
  • The Tenured generation is used for the longer lived objects. Another GC process (CMS) runs when it becomes full to remove any unused objects. The Tenured generation is larger and less active than the Young generation but GC tends to have more impact on performance.

The following diagram illustrates the heap and also the corresponding GC options that can be set in (Java 7 and earlier):

Key symptoms that indicate you need to tune your Java heap are:

  • High CPU usage. When GC threads consume a lot of CPU, you will see spikes in overall CPU usage.
  • Application hangs. When your application appears to hang and stops responding, there are gaps in your log files or you experience general slowness when performing routine tasks, poorly tuned GC can be the cause.

The first thing to check if you experience these symptoms is your GC logs; this are easy to check and may help you identify your issue quickly.

GC Logging

There are various different GCs available; ForgeRock recommends the use of CMS (ConcurrentMMarkSweep), which is a throughput collector and tends to give the best performance as it runs concurrently with the application, meaning less pauses. You should specify this collector by setting the following JVM option:

-XX:+UseConcMarkSweepGC

You can enable GC logging as described in the following articles:

Note

Typically you can leave GC logging on without impacting your system, however, you can just enable it when you are experiencing issues.

Understanding GC Logs

These example log files come from a CMS collector, where ParNew refers to the Young generation and CMS refers to the Tenured generation. These log file examples also show typical GC logs for a system where tuning is appropriate.

ParNew section

An example GC log output for the ParNew section looks like this:

24.245: [GC24.245: [ParNew: 545344K->39439K(613440K), 0.0424360 secs] 594671K->88767K(4101632K), 0.0426850 secs] [Times: user=0.17 sys=0.00, real=0.04 secs] 
29.747: [GC29.747: [ParNew: 584783K->27488K(613440K), 0.0889130 secs] 634111K->104723K(4101632K), 0.0892180 secs] [Times: user=0.24 sys=0.01, real=0.09 secs]

Each line in this log file represents one GC and shows the following types of information for ParNew:

  • Timestamp information - if you did not set the PrintGCTimeStamps and PrintGCDateStamps options, this just shows as the number of seconds from JVM startup:
    24.245: [GC24.245:
    29.747: [GC29.747:
    This information is useful as it shows the frequency of GCs. You should be aiming for GCs to occur once every 1 to 5 seconds; if they are occurring more than once a second, you need to tune your JVM and it is likely that you will be seeing high CPU usage.
  • Young generation information:
    [ParNew: 545344K->39439K(613440K), 0.0424360 secs] 
    [ParNew: 584783K->27488K(613440K), 0.0889130 secs]
    This information shows the initial size of the Young space before doing the GC, the size after doing the GC, the total size available and the time it took to do the GC. If the time taken to do the GC is large, that is more than 0.1 seconds, then it is possible that your Young generation heap size is too big. The total size available will not grow if you set NewSize and MaxNewSize to the same value and this is what you want to see in your logs.
  • Overall heap information:
    594671K->88767K(4101632K), 0.0426850 secs]
    634111K->104723K(4101632K), 0.0892180 secs]
    This information shows initial size of the overall heap before doing the GC, the size after doing the GC and the total size available. With the second lot of figures (size after doing the GC), you can see how this incrementally grows up to the point when the GC is done and then reduces back down to a baseline figure. If you have a memory leak, you will see the baseline figure growing incrementally as it becomes increasingly full even after doing GC.
  • System information:
    [Times: user=0.17 sys=0.00, real=0.04 secs] 
    [Times: user=0.24 sys=0.01, real=0.09 secs]
    This information shows how much time was spent in user space, how much time spent in kernel or system space and what was the real impact to the user. If the user time is high, it indicates there isn't enough CPU for the number of user threads. If the system time is high, it indicates your system is swapping memory to disk which means that there isn't enough physical memory for your heap size.

CMS section

The CMS section of the log file is different to the ParNew section as it shows multiple lines for the same GC. An example GC log output for the CMS section looks like this:

40.146: [GC [1 CMS-initial-mark: 26386K(786432K)] 26404K(1048384K), 0.0074495 secs]
40.154: [CMS-concurrent-mark-start]
40.683: [CMS-concurrent-mark: 0.521/0.529 secs]
40.683: [CMS-concurrent-preclean-start]
40.701: [CMS-concurrent-preclean: 0.017/0.018 secs]
40.704: [GC40.704: [Rescan (parallel) , 0.1790103 secs]40.883: [weak refs processing, 0.0100966 secs] [1 CMS-remark: 26386K(786432K)] 52644K(1048384K), 0.1897792 secs]
40.894: [CMS-concurrent-sweep-start]
41.020: [CMS-concurrent-sweep: 0.126/0.126 secs]
41.020: [CMS-concurrent-reset-start]
41.147: [CMS-concurrent-reset: 0.127/0.127 secs]

This log shows the different phases the GC goes through. All the ones marked concurrent are happening while your application is running and therefore have little impact. The ones that don't say concurrent are stopping your application from running temporarily and therefore can have an impact. The CMS-initial-mark and CMS-remark show heap sizing details.

Note

Typically, you will see these sections intermingled, for example, some ParNews then a CMS, some more ParNews, another CMS and so on as the GC processes automatically occur to manage the heap.

Common issues shown in the GC logs and solutions

This section identifies common issues you might find in your GC logs and how you can resolve them. You can also use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

Young generation - too small

The following log snippet shows an issue where the Young generation heap size is too small:

1.813: [GC1.813: [ParNew: 1152K­>128K(1152K), 0.0008100 secs] 16620K­->15756K(26936K), 0.0008350 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
1.816: [GC1.816: [ParNew: 1152K­>128K(1152K), 0.0006430 secs] 16780K­->15913K(26936K), 0.0006640 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
1.819: [GC1.819: [ParNew: 1152K­>128K(1152K), 0.0005370 secs] 16937K-­>16038K(26936K), 0.0005570 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]

As you can see, there are multiple GCs occurring within a second. This means you need to increase the size of the Young generation (NewSize and MaxNewSize) and may also need to increase total heap size to compensate for this change (Xms and Xmx). Alternatively you could change the NewRatio option (this is set to 2 by default, which means the Tenured generation heap is twice as big as the Young generation heap or that the Young generation heap is a 1/3 of the size of the Tenured generation heap).

Young generation - too large / Tenured generation - too small

The following log snippet shows an issue where the Young generation heap size is too large and consequently the Tenured generation heap size is too small:

276.716: [GC (CMS Initial Mark) [1 CMS-initial-mark: 104176K(135168K)] 2758985K(6741248K), 0.8762860 secs] [Times: user=0.88 sys=0.00, real=0.88 secs] 
277.592: [CMS-concurrent-mark-start]
277.657: [CMS-concurrent-mark: 0.063/0.065 secs] [Times: user=0.12 sys=0.00, real=0.06 secs] 
277.657: [CMS-concurrent-preclean-start]
277.658: [CMS-concurrent-preclean: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
277.658: [CMS-concurrent-abortable-preclean-start]
277.658: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
277.658: [GC (CMS Final Remark)[YG occupancy: 2657419 K (6606080 K)]277.658: [Rescan (parallel) , 0.9815460 secs]278.639: [weak refs processing, 0.0000320 secs]278.640: [scrub string table, 0.0011700 secs] [1 CMS-remark: 104176K(135168K)] 2761595K(6741248K), 0.9828250 secs] [Times: user=7.18 sys=0.09, real=0.99 secs] 
278.641: [CMS-concurrent-sweep-start]
278.668: [CMS-concurrent-sweep: 0.026/0.027 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
278.668: [CMS-concurrent-reset-start]
278.668: [CMS-concurrent-reset: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
280.673: [GC (CMS Initial Mark) [1 CMS-initial-mark: 104079K(135168K)] 2774091K(6741248K), 0.9033730 secs] [Times: user=0.90 sys=0.00, real=0.90 secs] 
281.577: [CMS-concurrent-mark-start]
281.640: [CMS-concurrent-mark: 0.063/0.063 secs] [Times: user=0.13 sys=0.00, real=0.07 secs] 
281.640: [CMS-concurrent-preclean-start]
281.641: [CMS-concurrent-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
281.641: [CMS-concurrent-abortable-preclean-start]
281.641: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
281.641: [GC (CMS Final Remark)[YG occupancy: 2670011 K (6606080 K)]281.641: [Rescan (parallel) , 0.9914290 secs]282.633: [weak refs processing, 0.0000110 secs]282.633: [scrub string table, 0.0008100 secs] [1 CMS-remark: 104079K(135168K)] 2774091K(6741248K), 0.9923100 secs] [Times: user=7.14 sys=0.11, real=0.99 secs] 
282.634: [CMS-concurrent-sweep-start]
282.659: [CMS-concurrent-sweep: 0.024/0.025 secs] [Times: user=0.06 sys=0.01, real=0.02 secs] 
282.659: [CMS-concurrent-reset-start]
282.659: [CMS-concurrent-reset: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 

As you can see, the CMS collections are occurring too frequently, without any ParNew collections in between. This is a result of the Young generation heap size being too large so that each time a ParNew collection happens, the Tenured heap size instantly becomes full and a CMS collection runs. This could happen if you increased your Young generation heap size without increasing your total heap size. This log snippet could also happen if you have memory leaks.

CMS failure

The following log snippet shows a CMS failure (concurrent mode failure):

(concurrent mode failure): 1551948K->1548050K(1572864K), 15.1154540 secs] 1986510K->1548050K(2044736K), [CMS Perm : 78597K->78588K(524288K)], 15.4406700 secs] [Times: user=15.52 sys=0.00, real=15.44 secs] 
703240.983: [GC [1 CMS-initial-mark: 1548050K(1572864K)] 1561545K(2044736K), 0.0059690 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 

This error means that the CMS collector cannot keep up with the amount of garbage being generated (the Tenured generation heap space is about 90% full) and GCs are taking longer meaning your application stops for longer periods of time. This example shows the application stopping for over 15 seconds, which will have a significant impact on performance.

You can add the following GC options to try to avoid the CMS failure:

  • -XX:CMSInitiatingOccupancyFraction=<percentage-of-tenured> 
  • -XX:+UseCMSInitiatingOccupancyOnly

You should set -XX:CMSInitiatingOccupancyFraction to 70 or 80% (instead of default ~90%) which may enable the CMS collector to keep up with the amount of garbage being generated.

Full GC

The following log snippet shows a Full GC:

25.466: [Full GC )25.466: [CMS: 68756K­>75361K(1987392K), 0.2548820 secs] 96571K­->75361K(2064064K), [CMS Perm : 61069K­>60722K(61376K)], 0.2551850 secs] [Times: user=0.25 sys=0.01, real=0.25 secs]
26.185: [Full GC (System.gc())26.185: [CMS: 75361K­>73283K(1987392K), 0.2398450 secs] 76136K-­>73283K(2064064K), [CMS Perm : 60723K­>60723K(101204K)], 0.2400120 secs] [Times: user=0.24 sys=0.00, real=0.24 secs]
27.293: [Full GC (System.gc())27.293: [CMS: 73283K­>73291K(1987392K), 0.2111960 secs] 74096K­>73291K(2064064K), [CMS Perm : 60723K­>60723K(101396K)], 0.2112730 secs] [Times: user=0.21 sys=0.00, real=0.22 secs]
68.081: [Full GC (Heap Inspection Initiated GC)68.081: [CMS: 73291K­->73187K(1987392K), 0.2172670 secs] 77634K­>73187K(2064064K), [CMS Perm : 60742K­>60742K(101396K)], 0.2174520 secs] [Times: user=0.22 sys=0.00, real=0.21 secs]

These log snippets refer to Full GCs occurring, meaning that the entire heap is being cleared (Young and Tenured); this often happens due to system processes such as RMI which performs a Full GC every hour on the hour. The PrintGCCause option is very useful to help you find out why the GC is happening (the last three lines in the log above show the additional information output by this option, making it clear which ones are system related). You can also set the DisableExplicitGC option to prevent system related Full GCs from happening.

Permanent generation 

Note

The following information does not apply to Java 8 since the Permanent generation space has been removed.

The following log snippet shows a Full GC as a result of the Permanent generation space growing (the Permanent generation space contains metadata and other data that should be permanently stored and does not typically grow):

1.241: [Full GC (Ergonomics) [PSYoungGen: 6125K­>0K(45056K)] [ParOldGen: 17659K­->20350K(20480K)] 23785K­>20350K(65536K) [PSPermGen: 13550K­->13546K(27136K)], 0.0531680 secs] [Times: user=0.25 sys=0.00, real=0.05 secs]
1.513: [Full GC (Ergonomics) [PSYoungGen: 38912K­>2783K(45056K)] [ParOldGen: 20350K­>20424K(20480K)] 59262K­>23208K(65536K) [PSPermGen: 15188K­->15188K(31744K)], 0.0992090 secs] [Times: user=0.65 sys=0.02, real=0.10 secs]

These log snippets show the Permanent generation space growing (figure in brackets after PSPermGen). You can increase the MaxPermSize option to resolve this issue.

Best practice advice on tuning

Initially, you should configure your JVM as per the recommended settings for your product (if available):

You can then tune your JVM as and when required.

You should always follow this basic process when tuning:

  1. Establish a baseline using a valid load testing process. This step is essential as it is the only way you will know if your tuning is having the desired effect.
  2. Change one thing at a time and document each change. This allows you to determine the impact of each change and revert if necessary.

Heap tuning guidelines

Although we cannot give recommended heap settings because they vary considerably between environments and applications, you should follow these guidelines when tuning:

  • You should always use the latest supported software versions.
  • You should use the CMS GC as recommended by ForgeRock.
  • Your total heap size must not exceed the amount of physical RAM available.
  • You must leave some physical RAM for other applications to run on your machine.
  • You should set Xms and Xmx to the same value for best performance. These options determine your total heap size.
  • You should set NewSize and MaxNewSize to the same value. These options determine how the heap is allocated between generations.
  • You should set the DisableExplicitGC option to prevent system related full GCs from running.
  • You must not add lots of options at once or add options you don't understand.
  • You must not copy and paste options from other applications.
  • You should use GC logs to help identify issues with your tuning.

Getting your total heap size and allocation between generations right is key; once you have these set correctly, you likely won't need to do anymore tuning.

Summary

In summary, the most important GC knobs to get right are:

  1. The collector, which should be CMS.
  2. The total heap size, where Xmx = Xms.
  3. The split between generations, using NewSize or NewRatio, where NewSize = MaxNewSize.

See Also

Java Platform, Standard Edition HotSpot Virtual Machine Garbage Collection Tuning Guide

AM/OpenAM

How do I change the JVM heap size for AM/OpenAM (All versions)?

FAQ: AM/OpenAM performance and tuning

How do I collect JVM data for troubleshooting AM/OpenAM (All versions)?

How do I diagnose a hung AM/OpenAM (All versions) server?

How do I collect data for troubleshooting high CPU utilization on AM/OpenAM (All versions) servers?

How do I find which thread is consuming CPU in a Java process in AM/OpenAM (All versions)?

How do I use the msnapshots script to capture information for troubleshooting AM/OpenAM (All versions)?

DS/OpenDJ

How do I tune DS/OpenDJ (All versions) process sizes: JVM heap and database cache?

FAQ: DS/OpenDJ performance and tuning

How do I collect JVM data for troubleshooting DS/OpenDJ (All versions)?

How do I collect data for troubleshooting high CPU utilization on DS/OpenDJ (All versions) servers?

How do I find which thread is consuming CPU in a Java process in DS/OpenDJ (All versions)?

How do I use the msnapshots script to capture information for troubleshooting DS/OpenDJ (All versions)?

IDM/OpenIDM

How do I change the JVM heap size for IDM/OpenIDM (All versions)?

FAQ: IDM/OpenIDM performance and tuning

How do I collect JVM data for troubleshooting IDM/OpenIDM (All versions)?

How do I collect data for troubleshooting high CPU utilization on IDM/OpenIDM (All versions) servers?

How do I find which thread is consuming CPU in a Java process in IDM/OpenIDM (All versions)?

How do I use the msnapshots script to capture information for troubleshooting IDM/OpenIDM (All versions)?

IG/OpenIG

How do I change the JVM heap size for IG/OpenIG (All versions)?

How do I collect JVM data for troubleshooting IG/OpenIG (All versions)?

How do I collect data for troubleshooting high CPU utilization or Out of Memory errors on IG/OpenIG (All versions) servers?

How do I find which thread is consuming CPU in a Java process in IG/OpenIG (All versions)?

How do I use the msnapshots script to capture information for troubleshooting IG/OpenIG (All versions)?

Related Training

ForgeRock Directory Services Core Concepts

Related Issue Tracker IDs

OPENAM-4414 (Apache Policy Agent does not complete cleanup / logout)

OPENIDM-1563 (Task scanner creates a new thread pool for each execution resulting in a thread leak.)


OpenAM/AM


FAQ: AM/OpenAM performance and tuning

The purpose of this FAQ is to provide answers to commonly asked questions regarding performance and tuning for AM/OpenAM and Policy Agents.

Frequently asked questions

Q. Do I need to performance test AM/OpenAM?

A. Yes, performance testing is a very important step to ensure that the system you have configured will cope with the expected load once it is in production. By performance testing your system, you can tweak any performance issues you encounter before going live.

Before running any performance tests, you should ensure AM/OpenAM is running and configured in a test environment that closely resembles your planned production environment. You should also be aware of the type of load expected in your production system and the types of identity data involved to make your tests as realistic as possible. For example, you may want to test that your system can handle 20 authentications per second with a concurrency of 2,000 live sessions.

Generating test identity data in DS/OpenDJ is described in How do I generate sample user data for performance testing in DS/OpenDJ (All versions)?

Performing a performance test is described in idmdude - It's OK to Get Stressed Out with OpenAM

Q. How do I tune AM/OpenAM to improve performance?

A. Tuning AM/OpenAM is described in Setup and Maintenance Guide › Tuning an Instance.

Q. How can I improve the performance of ssoadm?

A. There are several approaches you can take to improving the performance of ssoadm as detailed in How do I improve the performance of ssoadm in AM/OpenAM (All versions)?

Q. What is the recommended Java Virtual Machines (JVM) heap size for AM/OpenAM?

A. There are no definitive rules for the size of JVM heap size required as it will vary across individual environments and applications, but you should refer to Best practice for JVM Tuning for best practice advice. There are also a number of things you should be aware of when making a decision for AM/OpenAM:

  • AM/OpenAM response time is independent of heap size.
  • The number of concurrent Single Sign On (SSO) sessions is dependent on heap size. As a rough guide, an AM/OpenAM server in production with a 3 GB heap can handle 100,000 sessions. As heap size increases, so does the number of possible concurrent sessions, however, smaller heap sizes are easier to manage.
  • You must ensure you configure JVM garbage collection appropriately as garbage collection runs can get quite long if you have large heap sizes.

See How do I change the JVM heap size for AM/OpenAM (All versions)? and Setup and Maintenance Guide › Tuning Java Virtual Machine Settings for further information.

Note

For a 32-bit JVM or a 32-bit operating system, the limit for the process size is 4GB, that is, 2^32; this cannot be exceeded regardless of the amount of heap space allocated.

Q. Can I change caching in AM/OpenAM to improve performance?

A. Yes, you can improve performance in AM/OpenAM by configuring caching on the server side; you can configure caching for configuration data and global user data. This is described in Setup and Maintenance Guide › Tuning Caching. You need to balance performance against memory usage when configuring caching; larger caches lead to less requests to AM/OpenAM but increase memory usage.

Q. How do client-based sessions affect performance?

A. Client-based sessions were introduced in OpenAM 13 per: OpenAM 13 Release Notes › What's New in OpenAM › Elasticity and Scaling Features.

Client-based sessions use more CPU than CTS-based sessions; however CTS-based sessions have higher RAM consumption.

See How do I enable Client-based sessions in AM (All versions) and OpenAM 13.x? for further information.

Q. How does the Session Purge Delay setting affect performance?

A. The Session Purge Delay setting (com.iplanet.am.session.purgedelay) is set to 0 by default, which removes the session from memory immediately. If you increase this to a value above 0 (not recommended), the session is then held in memory for that number of minutes before being removed. This setting has been removed in AM 5.

If you have a particular need to increase the session purge delay, you should increase your heap size to compensate for the increase in sessions as described in How do I change the JVM heap size for AM/OpenAM (All versions)?

See Setup and Maintenance Guide › Session Settings​ for further information about these settings. 

Q. What else might affect performance on a Linux system or a Virtual Machine?

A. Low entropy values can cause general system slowness.

Note

Low entropy values is a OS/Java/web container issue, which is independent of AM/OpenAM; as such, the suggested resolutions are outside the scope of ForgeRock support. If you want more tailored advice, consider engaging Deployment Support Services.

On Linux systems you can check if a low entropy value is affecting you by running the following command:

$ cat /proc/sys/kernel/random/entropy_avail

A response of 200 or less is considered low and should be addressed, but you may find that even higher values can cause system slowness.

Resolution

You can increase the amount of entropy available in a variety of ways; two possible approaches are:

  • Use an external tool called RNGD (Random Number Generator Deamon). This can either use the RDRAND instruction set or /dev/urandom if RDRAND is not available, although many Virtual Machines (such as VirtualBox and OpenStack) passthrough the RDRAND instruction set to the VM. You will need to install the rng-tools package.
  • Set the following JVM option in the application web container in which AM/OpenAM runs:
    -Djava.security.egd=file:/dev/./urandom

Q. Which Apache httpd Multi-Processing Module (MPM) should I use with web policy agent?

A. It is recommended that you always use Worker MPM rather than Prefork MPM. You should ensure there are enough processes and threads available to service the expected number of client requests. You can then tune the Worker mode performance in the conf/extra/http-mpm.conf file as detailed in Web Agents User Guide › Tuning Apache Multi-Processing Modules.

You can check which MPM Apache is using with the following command:

$ apachectl -V

The MPM in use is shown against Server MPM.

Note

Turning off the Apache™ KeepAlive feature can potentially improve performance as well since the KeepAlive feature increases memory usage. You should test your performance with KeepAlive enabled and then again with it off to check what impacts it has on your setup.

Q. Are there things I should monitor on an ongoing basis in terms of AM/OpenAM performance?

A. Yes, it is useful to monitor performance on an ongoing basis to allow you to react to changes quickly. Useful things to monitor include:

  • Heap size 
  • Number of open sessions and size of sessions 
  • CPU utilization - utilization of 70-80% is worth investigating.

See How do I monitor session statistics in AM/OpenAM (All versions)? and Setup and Maintenance Guide › Monitoring Services for further information.

See Also

FAQ: Upgrading AM/OpenAM

FAQ: Installing AM/OpenAM

FAQ: Configuring AM/OpenAM

FAQ: General AM/OpenAM

Performance tuning and monitoring ForgeRock products

Related Training

ForgeRock Access Management Core Concepts (AM-400)


FAQ: Caching in AM/OpenAM

The purpose of this FAQ is to provide answers to commonly asked questions regarding caching in AM/OpenAM.

Frequently asked questions

Note

Tuning recommendations are environment specific and are outside the scope of ForgeRock support; if you want more tailored advice, consider engaging Deployment Support Services

Q. How can I control caching for configuration and user data using ssoadm?

A. There are three properties available for enabling / disabling caching for configuration and user data stores:

  • com.iplanet.am.sdk.caching.enabled - enables caching for both the configuration and user data stores. The following properties are ignored when this is set to true.
  • com.sun.identity.idm.cache.enabled - enables caching for the user data store (IdRepo cache).
  • com.sun.identity.sm.cache.enabled - enables caching for the configuration data store (service configuration caching).

See Setup and Maintenance Guide › Cache Settings for further information about these properties and other cache related properties, including time based expiration. This guide also provides steps for setting these advanced properties in console. 

You can set these properties to true or false using the update-server-cfg command. For example, the following command enables caching for the user data store only:

$ ./ssoadm update-server-cfg -s default -u amadmin -f pwd.txt -a com.iplanet.am.sdk.caching.enabled=false com.sun.identity.idm.cache.enabled=true com.sun.identity.sm.cache.enabled=false

Time-to-Live

If caching is enabled for either of these stores, you can also set the following properties using the above ssoadm command: 

  • com.sun.identity.sm.cache.ttl.enable - determines if entries in the configuration store cache are expired based on time-to-live (TTL). This is set to false by default.
  • com.sun.identity.sm.cache.ttl - specifies the time to live in minutes when the above property is set to true. This is set to 30 by default; when TTL is enabled, entries will remain in the cache for 30 minutes after their last modification. After 30 minutes, the data in these entries will expire and further requests for the data will be retrieved from the configuration store.
  • com.sun.identity.idm.cache.entry.expire.enabled - determines if entries in the user data store cache are expired based on time-to-live (TTL). This is set to false by default.
  • com.sun.identity.idm.cache.entry.default.expire.time - specifies the time to live in minutes when the above property is set to true. This is set to 30 by default; when TTL is enabled, entries will remain in the cache for 30 minutes after their last modification. After 30 minutes, the data in these entries will expire and further requests for the data will be retrieved from the user data store.

These properties can also be set using the update-server-cfg command.

Note

There is a known issue with disabling caching in OpenAM 13: OPENAM-8269 ("AuthId JWT Signature not valid" error in multi-instance deployments on 13). This is fixed in OpenAM 13.5. See Authentication fails in OpenAM 13.0 with an AuthId JWT Signature not valid error for further information.

Q. How does LDAP persistent search affect caching?

A. AM/OpenAM uses LDAP persistent search to receive notifications of changes to cached data. However, operational attributes such as the isMemberOf attribute (DS/OpenDJ data store) or the memberOf attribute (Active Directory data store) are not updated by the persistent search mechanism as they are computed dynamically. For use cases where you need to be notified of user membership changes, configuring your user data store to expire caches based on TTL is a valid option. See the Solution in isMemberOf attribute does not return current group membership details for a user in AM/OpenAM (All versions) for other options.

Unless all your user data stores support LDAP persistent search, you should disable the global cache for user data stores. When caching is enabled for user data stores that do not support LDAP persistent search, user data caches do not stay in sync with changes to user data. See Setup and Maintenance Guide › To Turn Off Global User Data Caching for further information on disabling the global cache.

Additionally, you should also set Persistent Search Base DN (sun-idrepo-ldapv3-config-psearchbase) to blank in the user data stores that do not support persistent search to disable persistent search.

See Setup and Maintenance Guide › Setting Up Identity Data Stores​ for further information on making changes to individual data stores.

Note

OpenLDAP does not support persistent search.

Q. Is there a recommended Maximum Session Cache Size?

A. No, this will depend on your setup. This setting (applicable to AM 5 and later) specifies the maximum number of sessions to cache in each AM server’s internal session cache. The cache is an LRU (Least Recently Used) cache, which evicts the least used session (usually the first session to be added to the cache) when this maximum session cache size value is reached. For example, with the default size value of 5000: if there are 5000 active sessions within a single AM, when session 5001 arrives, session 1 is evicted. Until then, all sessions remain in the cache until modified and/or deleted. 

This setting applies to each server and assuming sticky load balancing has been applied, a guide to setting this appropriately is to take the maximum number of active sessions at any one time divided by the number of available AM instances.

Caution

It is critical that the value chosen is thoroughly tested as this cache does consume JVM heap space, more so with additional session properties. If the value is set too high, there is a risk of instability or Out Of Memory exceptions. 

Q. Can I disable session caching in AM/OpenAM?

A. You can disable session caching in pre-AM 5 but not later versions due to changes in session handling when moving to Autonomous Session Management (the removal of session crosstalk and making the CTS token store the authoritative source for sessions rather than the AM server itself). However, disabling caching is likely to degrade performance since clients will need to query OpenAM more frequently for session details.

You can disable session caching by setting Maximum Caching Time to 0 as detailed in How do I change the Maximum Caching Time in AM 5.x, 6 and OpenAM 12.x, 13.x?

Q. Can I set the Maximum Caching Time to a fraction of a minute?

A. No, the Maximum Caching Time must be whole integers (minutes); settings such as 0.5 are not permitted.

Q. Is there a recommended Maximum Caching Time?

A. No, the most appropriate Maximum Caching Time for your setup depends on many different factors and also which version you are using. See How do I change the Maximum Caching Time in AM 5.x, 6 and OpenAM 12.x, 13.x? for an explanation of how this setting is used.

When setting it in OpenAM 12.x or 13.x, it should always be less than the maximum idle time. You need to balance performance against memory usage when configuring caching; larger caches lead to less requests to OpenAM but increase memory usage.

See How do I configure session timeouts in AM/OpenAM (All versions)? for further information on changing the maximum idle time.

Q. How long is the Certificate Revocation List (CRL) held in the cache?

A. Once the CRL is read by AM/OpenAM, it is held in the cache until AM/OpenAM is restarted or until it is updated in the LDAP directory.

It can only be updated in the LDAP directory if the Update CA CRLs from CRLDistributionPoint option (openam-am-auth-cert-update-crl) is enabled. When this option is enabled, AM/OpenAM checks the nextUpdate date field in the CRL each time it uses it. Once this date has passed, AM/OpenAM fetches new CA CRLs from the CRLDistributionPoint and updates the CRL held in the LDAP directory.

See How do I configure certificate-based authentication in AM/OpenAM (All versions)? and Authentication and Single Sign-On Guide › Certificate Authentication Module Properties for further information on these settings.

Note

The Certificate Authentication module must be configured to cache the CRL by enabling the Cache CRLs in memory setting (openam-am-auth-cert-attr-cache-crl).

See Also

FAQ: AM/OpenAM performance and tuning

Performance tuning and monitoring ForgeRock products

Setup and Maintenance Guide › Tuning Caching


How do I change the Maximum Caching Time in AM 5.x, 6 and OpenAM 12.x, 13.x?

The purpose of this article is to provide information on changing the Maximum Caching Time in AM/OpenAM, which affects session caching. You can change this globally, per realm or per user as required, where realm level overrides global setting and user level overrides both realm and global settings.

Background information

The maximum caching time setting has evolved over time and is used as follows depending on your version:

AM 5.x and 6

AM 5 introduced changes to session handling when moving to Autonomous Session Management (the removal of session crosstalk and making the CTS token store the authoritative source for sessions rather than the AM server itself ), which means this setting is no longer used internally by AM sessions and is only used by older policy agents. Since these policy agents are not supported in AM 6.5, this setting is obsolete. See What versions of Policy Agents are compatible with AM/OpenAM? for further details.

OpenAM 12.x and 13.x

The maximum caching time specifies the number of minutes that an OpenAM session is held in memory before it is refreshed. It is also used to recommend the number of minutes that the policy agents (Web 4.x and JEE 3.5.x) should cache the session for. This setting is simply a suggestion though and can be overridden in the policy agent profile.

Changing the global setting

You can configure the global maximum caching time using either the console or ssoadm:

  • AM / OpenAM 13.5 console: navigate to: Configure > Global Services > Session > Dynamic Attributes and enter the required number of minutes for the maximum caching time.
  • Pre-OpenAM 13.5 console: navigate to: Configuration > Global > Session > Dynamic Attributes and enter the required number of minutes for the maximum caching time.
  • ssoadm: enter the following command:
    $ ./ssoadm set-attr-defs -s iPlanetAMSessionService -t dynamic -u [adminID] -f [passwordfile] -a iplanet-am-session-max-caching-time=[minutes]
    replacing [adminID], [passwordfile] and [minutes] with appropriate values.

Changing the realm level setting

Note

You may need to add the Session service if it is not listed under Services by clicking Add a Service or Add and then selecting Session. If you are using ssoadm, you can replace set-realm-svc-attrs in the ssoadm command with add-svc-realm to add this service and set the attributes with the same command.

You can configure the realm level maximum caching time using either the console or ssoadm:

  • AM / OpenAM 13.x console: navigate to: Realms > [Realm Name] > Services > Session and enter the required number of minutes for the maximum caching time.
  • Pre-OpenAM 13 console: navigate to: Access Control > [Realm Name] > Services > Session and enter the required number of minutes for the maximum caching time.
  • ssoadm: enter the following command:
    $ ./ssoadm set-realm-svc-attrs -s iPlanetAMSessionService -e [realmname] -u [adminID] -f [passwordfile] -a iplanet-am-session-max-caching-time=[minutes]
    replacing [realmname], [adminID], [passwordfile] and [minutes] with appropriate values.

Changing the user level setting

Warning

You cannot set the maximum caching time at a user level if you use Active Directory® for your data store as the Session service attributes cannot currently be mapped to this data store.

You must select the Load Schema when saved option (enable Load Schema in AM 6) for your data store prior to specifying user level settings. In AM / OpenAM 13.x, navigate to Realms > [Realm Name] > Data Stores > [Data Store Name]. In Pre-OpenAM 13, navigate to Access Control > [Realm Name] > Data Stores > [Data Store Name]. 

Note

You may need to add the Session service if it is not listed under Services by clicking Add Service or Add and then selecting Session. If you are using ssoadm, you can replace set-identity-svc-attrs in the ssoadm command with add-svc-identity to add this service and set the attributes with the same command.

You can configure the user level maximum caching time using either the console or ssoadm:

  • AM 6 and later console: navigate to: Realms > [Realm Name] > Identities > [User Name] > Services > Session and enter the required number of minutes for the maximum caching time.
  • AM 5.x / OpenAM 13.x console: navigate to: Realms > [Realm Name] > Subjects > [User Name] > Services > Session and enter the required number of minutes for the maximum caching time.
  • Pre-OpenAM 13 console: navigate to: Access Control > [Realm Name] > Subjects > [User Name] > Services > Session and enter the required number of minutes for the maximum caching time. 
  • ssoadm: enter the following command:
    $ ./ssoadm set-identity-svc-attrs -s iPlanetAMSessionService -e [realmname] -t User -i [username] -u [adminID] -f [passwordfile] -a iplanet-am-session-max-caching-time=[minutes]
    replacing [realmname], [username], [adminID], [passwordfile] and [minutes] with appropriate values.

See Also

FAQ: Caching in AM/OpenAM

How do I configure session timeouts in AM/OpenAM (All versions)?

Reference › Configuration Reference › Session

Web Agents 5.5 › Release Notes › Access Management Requirements

Web Agents 5 › Release Notes › Access Management Requirements

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I change the JVM heap size for AM/OpenAM (All versions)?

The purpose of this article is to provide information on changing the JVM heap size for AM/OpenAM. This article provides details for changing the JVM heap size on Linux®, Unix® and Windows® systems.

Changing the JVM heap size

Changing the JVM heap size can improve performance and reduce the time it takes to perform authentications. You should try different heap sizes to see what impact it has to determine the best heap size for your setup. The information given here is specific to the Apache Tomcat™ web container; you should make similar changes in the configuration file specific to your web container if you use a different one.

Note

It is recommended that you set the minimum and maximum heap sizes to the same value for best performance. Otherwise, the JVM runs a full Garbage Collector (GC) cycle when increasing its heap, during which time it can pause ongoing operations for up to a few seconds. Generally, a smaller heap will increase the frequency at which the JVM GC executes but reduce the duration; similarly, a larger heap will reduce the frequency and increase the duration. When tuning the JVM heap, the goal is to strike a balance between frequency and duration so as to reduce the impact of the GC on the application.

On Unix® and Linux® systems:

You can set the JVM heap size by specifying CATALINA_OPTS settings in the setenv.sh file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.sh file (also typically located in the /tomcat/bin/ directory).

For example, to set the minimum and maximum heap sizes to 2GB, you would add the following line to the setenv.sh file and restart the web container:

export CATALINA_OPTS="$CATALINA_OPTS -Xms2g -Xmx2g"

On Microsoft® Windows® systems:

Providing you haven't installed Tomcat as a service, you can set the JVM heap size by specifying CATALINA_OPTS settings in the setenv.bat file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.bat file (also typically located in the /tomcat/bin/ directory).

For example, to set the minimum and maximum heap sizes to 2GB, you would add the following line to the setenv.bat file and restart the web container:

set CATALINA_OPTS=-Xms2g -Xmx2g

If you have installed Tomcat as a service on Windows, then you must use the GUI application to configure Tomcat services. This process refers to Tomcat 7, but will work for other versions by changing tomcat7w.exe in the command to match your version.

  1. Stop the Tomcat service.
  2. Navigate to \path\to\tomcat\bin\ from the command prompt:
  3. Enter the following command to display Tomcat Properties:
    tomcat7w.exe //ES//serviceName
    Where serviceName is the name of your Tomcat service.
  4. Navigate to the Java tab and complete the memory pool fields as follows:
    Initial memory pool: 2048
    Maximum memory pool: 2048
  5. Restart the Tomcat service.

Alternatively, Windows system administrators may prefer to configure these options in the registry so that they may be configured via group policy. The initial memory pool and maximum memory pool values can be configured in the JvmMS and JvmMX properties under the following registry key for Tomcat 7:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\Tomcat7\Parameters\Java

See Also

Best practice for JVM Tuning

FAQ: AM/OpenAM performance and tuning

How do I diagnose a hung AM/OpenAM (All versions) server?

How do I enable Garbage Collector (GC) Logging for AM/OpenAM (All versions)?

Setup and Maintenance Guide › Maintaining an Instance › Tuning Java Virtual Machine Settings

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I configure the heartbeat timeout in AM/OpenAM (All versions)?

The purpose of this article is to provide information on configuring the heartbeat timeout in AM/OpenAM. This allows you to tune the heartbeat timeout if you experience issues with heartbeat timeouts.

Overview

Heartbeat timeout is the time spent on AM/OpenAM sending and receiving the heartbeat request and is not pure processing time on DS/OpenDJ.

The default heartbeat timeout of 3 seconds is configurable (since OpenAM 12.0.3), which is useful if you are experiencing heartbeat timeouts. You will see errors such as the following in your logs if you are experiencing heartbeat timeouts:

  • Session debug log (when debug level is set to Message):
    Caused by: org.forgerock.opendj.ldap.ConnectionException: Server Connection Closed: Heartbeat timed out after 500 ms
    
  • IdRepo debug log (when debug level is set to Message):
    org.forgerock.opendj.ldap.ConnectionException: Server Connection Closed: Heartbeat timed out after 500 ms
    
  • Web application container log (for example, catalina.out for Apache Tomcat™):
    WARNING: No heartbeat detected for connection 'LDAPConnection(/192.10.210.10:8080,host1.example.com/192.10.210.15:18080)'
    

If you do see errors such as the above, you should increase the heartbeat timeout; it is recommended that you initially increase it to 10 seconds and then slowly increase it further if you continue to see errors.

Configuring the heartbeat timeout

You can configure the heartbeat timeout using either the console or ssoadm:

  • AM / OpenAM 13.5 console: navigate to: Deployment > Servers > [Server Name] > Advanced and add the org.forgerock.openam.ldap.heartbeat.timeout property and enter the heartbeat timeout in seconds for the value. To add the property as a server default, navigate to: Configure > Server Defaults > Advanced instead. Once you have entered the property and value, click + to add followed by Save Changes.
  • Pre-OpenAM 13.5 console: navigate to: Configuration > Servers and Sites > [Server Name] > Advanced and add the org.forgerock.openam.ldap.heartbeat.timeout property and enter the heartbeat timeout in seconds for the value. You can configure this for all servers using Default Server Settings instead of [Server Name]. 
  • ssoadm: enter the following command:
    $ ./ssoadm update-server-cfg -s [servername] -u [adminID] -f [passwordfile] -a org.forgerock.openam.ldap.heartbeat.timeout=[seconds]
    
    replacing [servername], [adminID], [passwordfile] and [seconds] with appropriate values, where [servername] can be default if you want to change the default server settings instead. For example:
    $ ./ssoadm update-server-cfg -s default -u amadmin -f pwd.txt -a org.forgerock.openam.ldap.heartbeat.timeout=10
Note

You must restart the web application container in which AM/OpenAM runs to apply these configuration changes. 

See Also

How do I perform a heartbeat check against DS/OpenDJ (All versions)?

Related Training

N/A

Related Issue Tracker IDs

OPENAM-3266 (Heartbeat timeouts can occur during operation)


How do I improve OAuth 2.0 performance in OpenAM 13.0?

The purpose of this article is to provide information on improving OAuth 2.0 performance in OpenAM 13.0.

Improving OAuth 2.0 performance

In OpenAM 13.0, there is a known performance issue: OPENAM-8023 (OAuth2 load: contention at com.sun.identity.sm.ServiceConfigImpl.getInstance level (perf regression compared to AM12.0.2)). This issue is resolved in OpenAM 13.5. 

You can mitigate this issue in OpenAM 13.0 as follows:

  1. Create a completely empty file. For example, you can use the following command if you are using a Linux® or Unix® operating system:
    $ touch [filename]
  2. Run the following ssoadm commands against all realms where you are using OAuth2:
    $ ./ssoadm add-svc-realm -e [realmname] -s ScriptingService -u [adminID] -f [passwordfile] -D [emptyfile]
    $ ./ssoadm create-sub-cfg -e [realmname] -s ScriptingService -g scriptConfigurations -u [adminID] -f [passwordfile] -D [emptyfile] 
    
    replacing [realmname], [adminID], [passwordfile] and [emptyfile] with appropriate values.

See Also

N/A

Related Training

N/A

Related Issue Tracker IDs

OPENAM-8023 (OAuth2 load: contention at com.sun.identity.sm.ServiceConfigImpl.getInstance level (perf regression compared to AM12.0.2))


How do I tune LDAP connection pool settings in AM/OpenAM (All versions) using ssoadm?

The purpose of this article is to provide assistance with tuning LDAP connection pool settings in AM/OpenAM using ssoadm. These connection pool settings apply to your LDAP data stores and LDAP authentication modules.

Tuning LDAP connection pool settings

LDAP data store settings

You can configure the LDAP data store connection pool sizes as follows using ssoadm:

  • LDAP Connection Pool Minimum Size: enter the following command:
    $ ./ssoadm update-datastore -e [realmname] -m [datastorename] -u [adminID] -f [passwordfile] -a sun-idrepo-ldapv3-config-connection_pool_min_size=[size]
    replacing [realmname], [datastorename], [adminID], [passwordfile] and [size] with appropriate values.
  • LDAP Connection Pool Maximum Size: enter the following command:
    $ ./ssoadm update-datastore -e [realmname] -m [datastorename] -u [adminID] -f [passwordfile] -a sun-idrepo-ldapv3-config-connection_pool_max_size=[size]
    replacing [realmname], [datastorename], [adminID], [passwordfile] and [size] with appropriate values.

LDAP authentication modules 

You can configure the LDAP authentication module connection pool sizes (Default LDAP Connection Pool Size) as follows using ssoadm:

$ ./ssoadm set-attr-defs -s iPlanetAMAuthService -t global -u [adminID] -f [passwordfile] -a iplanet-am-auth-ldap-connection-pool-default-size=[minSize:maxSize]

replacing [adminID], [passwordfile] and [minSize:maxSize] with appropriate values.

For example, if you wanted to tune this to the recommended production settings: 10 (minimum connection pool size) and 65 (maximum connection pool size), you would use an ssoadm command such as:

$ ./ssoadm set-attr-defs -s iPlanetAMAuthService -t global -u amadmin -f pwd.txt -a iplanet-am-auth-ldap-connection-pool-default-size=10:65

See Also

How do I understand what the user data store is used for in AM/OpenAM (All versions)?

FAQ: Installing and using ssoadm in AM/OpenAM

Setup and Maintenance Guide › Maintaining an Instance › Tuning LDAP Data Store Settings

Setup and Maintenance Guide › Maintaining an Instance › Tuning LDAP Authentication Module Settings

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I improve the performance of ssoadm in AM/OpenAM (All versions)?

The purpose of this article is to provide information on improving the performance of ssoadm in AM/OpenAM.

Improving the performance of ssoadm

There are five things you can try to improve the performance of ssoadm:

Establishing a baseline

You can add time before the ssoadm command to output the execution time of the ssoadm command, which will help you assess the effect of any changes you make. It is recommended you run a simple command, such as the following, prior to making any changes to give you a baseline:

$ time ./ssoadm list-servers -u amadmin -f pwd.txt

You can then repeat this command after making changes to assess the impact.

Using the do-batch command

You can use the ssoadm do-batch command to run multiple ssoadm commands; using this command means multiple ssoadm commands are executed together in a single JVM call, rather than as individual JVM calls, which speeds up processing time.

See How do I make batch changes using ssoadm in AM/OpenAM (All versions)? for further information.

Increasing the ssoadm heap size

The default ssoadm heap size settings are:

-Xms256m -Xmx512m

You can increase these values (providing you have enough physical memory) to improve ssoadm performance. Initially, you should try setting both Xms and Xmx to 1024m; Xms and Xmx should be the same for the best performance.

You can increase the ssoadm heap size values as follows:

  1. Edit the ssoadm or ssoadm.bat script and update the following line:
    $JAVA_HOME/bin/java -Xms1024m -Xmx1024m -cp "$CLASSPATH" \
  2. Save your changes.
Note

You can update the ssoadm.template or ssoadm.bat.template file (located in the /path/to/ssoadm/tools/template/unix/bin or /path/to/ssoadm/tools/template/windows/bin directory respectively) prior to installation instead; this means ssoadm will be installed with these default values. This approach is useful if you have multiple installations or do automated installs.

Use a client VM

The JDK offers both a client VM and a server VM; the client VM is tuned for reduced startup times and memory footprint, but typically the server VM is used by default. See Java Virtual Machine Technology for further information.

You can force ssoadm to use the client VM as follows:

  1. Edit the ssoadm or ssoadm.bat script and update the following line:
    $JAVA_HOME/bin/java -Xms1024m -Xmx1024m -client -cp "$CLASSPATH" \
  2. Save your changes.
Note

You can update the ssoadm.template or ssoadm.bat.template file (located in the /path/to/ssoadm/tools/template/unix/bin or /path/to/ssoadm/tools/template/windows/bin directory respectively) prior to installation instead; this means ssoadm will be installed with these default values. This approach is useful if you have multiple installations or do automated installs.

Adding the --nolog option to the ssoadm command

The following known issues can be mitigated using the --nolog option:

--nolog option

You can mitigate these issues by adding the --nolog option to your ssoadm command to disable audit logging for that command, for example:

$ ./ssoadm list-servers -u amadmin -f pwd.txt --nolog

Adding a Java option to change how random bits are generated

In some situations, running ssoadm on Linux systems can result in performance issues; this is particularly true if you are using a virtual machine. When this happens, you will see a stack trace similar to the following when generating the SSOToken ID:

"main" prio=10 tid=0x00007f806400c000 nid=0x7e3f runnable [0x00007f8068918000]
   java.lang.Thread.State: RUNNABLE
      at java.io.FileInputStream.readBytes(Native Method)
      at java.io.FileInputStream.read(FileInputStream.java:272)
      at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:526)
      at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:139) 
      at sun.security.provider.SecureRandom$SeederHolder.(SecureRandom.java:186) 
      at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:203) 
      - locked <0x00000000ed1684f0> (a sun.security.provider.SecureRandom) 
      at java.security.SecureRandom.nextBytes(SecureRandom.java:455) 
      - locked <0x00000000ed168790> (a java.security.SecureRandom) 
      at java.security.SecureRandom.next(SecureRandom.java:477) 
      at java.util.Random.nextLong(Random.java:334) 
      at com.sun.identity.authentication.internal.AuthContext.getSSOToken(AuthContext.java:842) 
      at com.sun.identity.setup.Bootstrap.getSSOToken(Bootstrap.java:289) 
      at com.sun.identity.setup.Bootstrap.getConfiguration(Bootstrap.java:203) 
      at com.sun.identity.setup.Bootstrap.load(Bootstrap.java:135) 
      at com.sun.identity.setup.Bootstrap.load(Bootstrap.java:92) 
      - locked <0x00000000eb0536e0> (a java.lang.Class for com.sun.identity.setup.Bootstrap) 
      at com.sun.identity.cli.CommandManager.main(CommandManager.java:113)

To improve the performance in this situation, you can add the following JVM option to the ssoadm script:

-D"java.security.egd=file:/dev/./urandom" \

See Also

How do I make batch changes using ssoadm in AM/OpenAM (All versions)?

How do I add multiple attributes with a single ssoadm command in AM/OpenAM (All versions)?

FAQ: Installing and using ssoadm in AM/OpenAM

Using ssoadm in AM/OpenAM

Best practice for JVM Tuning

Related Training

N/A

Related Issue Tracker IDs

OPENAM-10514 (update-entity-keyinfo throws java.lang.IndexOutOfBoundsException)

OPENAM-9749 (Resource leak in ssoadm's audit logging)

OPENAM-9685 (SSOAdmin is slow with a site configured)

OPENAM-8441 (Deleting an audit service using the REST SMS takes a long time)

OPENAM-4549 (ssoadm should use 'com.iplanet.am.sdk.ldap' instead of 'com.iplanet.am.sdk.remote' for com.iplanet.am.sdk.package)

OPENAM-2283 (Document a workaround for ssoadm slowness on virtual machines)


How do I enable Garbage Collector (GC) Logging for AM/OpenAM (All versions)?

The purpose of this article is to provide information on enabling GC Logging for AM/OpenAM. It assumes that you already have a working AM/OpenAM server that is installed.

Enabling GC Logging

The information given here is specific to the Apache Tomcat™ web container; you should make similar changes in the configuration file specific to your web container if you use a different one.

Note

You should ensure there is enough disk space for the GC logs; if not, you should enable GC log rotation as well by including the following options when you enable GC logging: -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=n, -XX:GCLogFileSize=n

On Unix® and Linux® systems:

​You should enable GC logging by specifying CATALINA_OPTS settings in the setenv.sh file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.sh file (also typically located in the /tomcat/bin/ directory).

To enable GC logging:

  1. Add the following line to the setenv.sh file:
    export CATALINA_OPTS="$CATALINA_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]"
    replacing [filename] with the path to the file that you would like to create to store the log file.
  2. Restart the web container.

Once the web container has successfully restarted, there should be a GC log file located in the directory specified in the -Xloggc: option. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

On Microsoft® Windows® systems:

​You should enable GC logging by specifying CATALINA_OPTS settings in the setenv.bat file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.bat file (also typically located in the /tomcat/bin/ directory).

To enable GC logging:

  1. Add the following line to the setenv.bat file:
    set CATALINA_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]
    replacing [filename] with the path to the file that you would like to create to store the log file.
  2. Restart the web container.

Once the web container has successfully restarted, there should be a GC log file located in the directory specified in the -Xloggc: option. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

See Also

Best practice for JVM Tuning

How do I change the JVM heap size for AM/OpenAM (All versions)?

How do I collect JVM data for troubleshooting AM/OpenAM (All versions)?

FAQ: AM/OpenAM performance and tuning

How do I diagnose a hung AM/OpenAM (All versions) server?

Setup and Maintenance Guide › Maintaining an Instance › Tuning Java Virtual Machine Settings

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I diagnose a hung AM/OpenAM (All versions) server?

The purpose of this article is to provide information on diagnosing a hung AM/OpenAM server, including the data you should collect to send to ForgeRock Support for troubleshooting.

Diagnosing a hung AM/OpenAM server

There are a number of different types of data that you should collect to help us diagnose a hung AM/OpenAM server. If you suspect you are experiencing a hung AM/OpenAM server, you should collect the following data and submit it to us when you raise the ticket to enable us to help you more quickly:

  • Debug logs.
  • JVM data.
  • Garbage Collector (GC) logs.
  • HTTP container logs.
  • Native stack traces.
Note

It is important that you collect all the data immediately after reproducing the issue so that the timestamps of the data collected and the suspected hung AM/OpenAM server correspond. Please notify us of the exact timestamp when the issue occurred so we can focus our attention on the relevant information. 

Debug logs

Message level debug logs provide more context for diagnosing errors but can generate large files. You should enable message level debugging when you are experiencing an issue as described in How do I enable Message level debugging in AM/OpenAM (All versions) debug files?

JVM data

JVM data must be collected before killing the affected process or restarting the server; otherwise the information relating to the affected process is lost forever, which may make it impossible to identify the root cause. There are three types of data that you should collect (stack traces, heap histograms and heap dumps) as well as providing details of the current JVM settings as described in How do I collect JVM data for troubleshooting AM/OpenAM (All versions)?

See How do I use the msnapshots script to capture information for troubleshooting AM/OpenAM (All versions)? for further information on using a script to collect this data if you are using a Linux®, Unix® or Solaris® system.

GC logs

GC logs are not always crucial at this stage, but can provide useful information about potential tuning issues that may be contributing to your problems. Enabling GC logging is described in How do I enable Garbage Collector (GC) Logging for AM/OpenAM (All versions)?

HTTP container logs

HTTP container server logs (Access and Error) are very useful for troubleshooting as between them they show all requests to the server, whether they were successful or not and any corresponding error messages. The actual name and location of the server logs are specific to your web container. For example, for Apache Tomcat™, the logs are called localhost_access_log.YYYY-MM-DD.log and catalina.out, and are located in the /path/to/tomcat/logs/ directory by default.

Native stack traces

Native stack traces are not always necessary but can provide useful information when troubleshooting, especially if you can isolate the the affected process. You can use the pstack command for Solaris® and Linux® (although pstack is not installed by default on Linux):

$ pstack -F [pid]

replacing [pid] with the process ID of the affected process.

See Also

How do I collect data for troubleshooting high CPU utilization on AM/OpenAM (All versions) servers?

How do I collect all the data required for troubleshooting AM/OpenAM and Policy Agents (All versions)?

Troubleshooting AM/OpenAM and Policy Agents

Related Training

ForgeRock Access Management Core Concepts (AM-400)

Related Issue Tracker IDs

N/A


Monitoring


How do I monitor session statistics in AM/OpenAM (All versions)?

The purpose of this article is to provide information on monitoring session statistics in AM/OpenAM. This can provide useful troubleshooting information if you are experiencing unexpectedly high session numbers.

Overview

There are a number of ways you can monitor session statistics in AM/OpenAM depending on your version. These include:

AM 5 and later

You can monitor sessions in AM 5 and later as follows:

  • Session page - navigate to: Realms > [Realm Name] > Sessions to access the Sessions page, which allows you to view and invalidate active CTS-based user sessions per realm.
  • REST API - you can query the /json/sessions endpoint (see the Using the /json/sessions endpoint section below for further information).
  • Amster - you can use Amster to query sessions (see the Using Amster section below for further information).

By default, only 120 sessions are returned using these methods. You can change this default using the Maximum Number of Search Results setting. See Authentication and Single Sign-On Guide › Session Search for further details.

Note

Changes in AM 5 that made all servers autonomous mean it is the CTS store that holds the session data rather than the AM server itself. As such, the options listed for OpenAM 13.x and earlier cannot be used with AM 5 since they query individual servers. This is a known issue: OPENAM-9817 (JMX CTS session count is broken).

OpenAM 13.x and earlier

Session details

Session management information, including attribute values such as login time, logout time, time out limits, session creations and terminations, are logged in the amSSO.access log file (typically located in the $HOME/[am_instance]/openam/log directory). You will also see session information in the CoreSystems, Authentication and Session debug files ($HOME/[am_instance]/openam/debug directory).

You may notice a difference in session counts if you have just restarted the server; the monitoring API uses beans to store the values, for example, the session count. It is currently implemented such that each counter has an increment and a decrement method. When you log in, the counter gets incremented and when the session is destroyed, it is decremented. When AM/OpenAM starts up, all counters on the local instance are at 0. On the other hand, the counter used to count sessions for the session quotas and the Sessions tab is different; it actually counts the sessions, depending on "server mode" (single instance, multi instance or SFO). If you're using SFO, it counts all sessions from the CTS session store.

If, for example, you have just restarted AM/OpenAM, you may still have all 10 sessions in the session store (depending on how many of them have expired since). So when you start authenticating, two things happen at the same time (let's assume that you already have 10 sessions in the session store):

  • User session quotas are checked and sessions are counted. This will find the existing sessions in LDAP. If the count exceeds the limit, it executes the quota action, for example, "destroy next expiring". This will result in decrementing the session counter in the monitoring bean. At this point it's at 0, so it stays at 0. The number of sessions in the CTS is now 9.
  • Authentication succeeds and the session gets created. 10 sessions in CTS, SNMP counter now reads 1.

Using the /json/sessions endpoint (AM 5 and later)

You can query the /json/sessions endpoint to find session details. For example, to find sessions in the top level realm, you would use a call such as:

$ curl -X GET -H 'Accept: application/json' 'http://host1.example.com:8080/openam/json/sessions?_queryFilter=realm%20eq%20%22%2F%22'

The easiest way to find the relevant command is by using the API Explorer:

  1. Access the API Explorer. You can either access it from the Help icon in the console or from the following URL: 
    http://host1.example.com:8080/openam/XUI/#api/explorer/applications
  2. Navigate to /sessions > Sessions V2.0 > sessions#2.0_query_filter.
  3. Populate the query fields as required; _queryFilter is a required field. For example, to query the top level realm, enter realm eq "/".
  4. Click Try it Out! This returns session details and also provides the curl command you can use in future.

See Development Guide › Developing with the REST API › Introducing the API Explorer and Development Guide › Developing with the REST API › Query for further information.

Using Amster (AM 5 and later)

You can use Amster to query session details using the query Sessions command. For example:

am> query Sessions --realm / --filter 'realm eq "/"'

Example response:

===> [ { "username": "amadmin", "universalId": "id=amadmin,ou=user,dc=example,dc=com", "realm": "/", "sessionHandle": "shandle:4r8SsX6XJj0oAbLBmexqyUsbC7Y.*AAJTSQACMDEAAlNLABxJNEhkVlRlMnNHRzVKUTlOa1hMQ3BiRzZad0E9AAJTMQAA*", "latestAccessTime": "2018-05-01T12:36:54.487Z", "maxIdleExpirationTime": "2018-05-01T13:06:54Z", "maxSessionExpirationTime": "2018-05-01T14:31:23Z", "_rev": "746064345" }, { "username": "demo", "universalId": "id=demo,ou=user,dc=example,dc=com", "realm": "/", "sessionHandle": "shandle:rn3PS1zCIBxmY5qnMtbbqJOLgkQ.*AAJTSQACMDEAAlNLABxNR2JvL0tUenQxc2N1YnU4MkN2YjNkeGY2UTQ9AAJTMQAA*", "latestAccessTime": "2018-05-01T12:36:50.448Z", "maxIdleExpirationTime": "2018-05-01T13:06:50Z", "maxSessionExpirationTime": "2018-05-01T14:36:50Z", "_rev": "856832111" } ]

See Entity Reference › Sessions › query for further information.

Using the amMasterSessionTableStats file (OpenAM 13.x and earlier)

The amMasterSessionTableStats file contains both user and application session data, and is located in the $HOME/[openam_instance]/openam/stats directory. This file shows current and peak values for the following items:

  • Maximum number of sessions in the session table, including both active and timed out sessions.
  • Maximum number of active sessions.
  • Number of Session Notifications in the queue.
Note

There is a known issue with stats logging not working in OpenAM 13.0: OPENAM-6998 (Session "stats" logging doesn't seem to work). This is fixed in OpenAM 13.5.

Using ssoadm (OpenAM 13.x and earlier)

The ssoadm list-sessions command lists all the current sessions on a specified server with session details, for example:

$ ./ssoadm list-sessions -u amadmin -f pwd.txt -t http://host1.example.com:8080/openam

Example response:

[Current Session] User Id: amAdmin Time Remain: 1199999 Max Session Time: 1200000 Idle Time: 0 Max Idle Time: 300000
Index: 0 User Id: user1 Time Remain: 1199991 Max Session Time: 1200000 Idle Time: 8 Max Idle Time: 300000
Index: 1 User Id: user7 Time Remain: 1199850 Max Session Time: 1200000 Idle Time: 139 Max Idle Time: 300000
Index: 2 User Id: user12 Time Remain: 1199993 Max Session Time: 1200000 Idle Time: 6 Max Idle Time: 300000
Index: 3 User Id: user3 Time Remain: 1199862 Max Session Time: 1200000 Idle Time: 137 Max Idle Time: 300000
Index: 4 User Id: user19 Time Remain: 1199997 Max Session Time: 1200000 Idle Time: 2 Max Idle Time: 300000
Index: 5 User Id: user28 Time Remain: 1199992 Max Session Time: 1200000 Idle Time: 7 Max Idle Time: 300000
Index: 6 User Id: user17 Time Remain: 1199998 Max Session Time: 1200000 Idle Time: 1 Max Idle Time: 300000
To invalidate sessions, enter the index numbers
[CR without a number to exit]:

You can run the command with the -q option, which does not prompt you to invalidate sessions and can add a grep command after to just return a total, for example:

$ ./ssoadm list-sessions -u amadmin -f pwd.txt -t http://host1.example.com:8080/openam -q | grep "User Id" | wc -l

Example response (which matches the number of sessions listed above):

8

Using SNMP monitoring (OpenAM 13.x and earlier)

First you must enable SNMP monitoring; you can do this using either the console or ssoadm:

  • OpenAM 13.5 console: navigate to: Configure > Global Services > System > Monitoring and enable Monitoring SNMP interface status.
  • Pre-OpenAM 13.5 console: navigate to: Configuration > System > Monitoring > Monitoring SNMP interface status and select the Enabled option.
  • ssoadm: enter the following command:
    $ ./ssoadm set-attr-defs -s iPlanetAMMonitoringService -t Global -u [adminID] -f [passwordfile] -a iplanet-am-monitoring-snmp-enabled=true
    
    replacing [adminID] and [passwordfile] with appropriate values.

By default, this allows you to listen on port 8085 for SNMP monitoring. You can change the port via the console or ssoadm (iplanet-am-monitoring-snmp-port attribute) using the same navigation / service as above. 

Note

You must restart the web application container in which OpenAM runs to apply these configuration changes. 

Once enabled, you can use the snmpwalk command to provide specific session details. For example, the following command (assuming you are using port 8085) gives the number of active sessions on the current OpenAM server instance:

snmpwalk -c public -v 2c :8085 enterprises.42.2.230.3.1.1.2.1.11.1.0
Note

This command gives the same output as the SessionActiveCount JMX attribute; where both are reset to 0 when the OpenAM instance is restarted.

You can check which SNMP definitions are supported in the mib/FORGEROCK-OPENAM-SESSION.mib file (enterprises 36733), which can be found in the openam.war/WEB-INF/lib/openam-mib-schema-1.x.x.jar file. See Administration Guide › Monitoring OpenAM Services › SNMP Monitoring for other session related OIDs you can use.

See Also

Out of Memory exception causes AM/OpenAM (All versions) to hang due to increasing number of open policy agent sessions

Setup and Maintenance Guide › Maintaining an Instance › Managing Sessions

Reference › Configuration Reference › Session Properties

Setup and Maintenance Guide › Maintaining an Instance › Monitoring CTS Tokens

Installation Guide › Reference › Core Token Service (CTS) Object Identifiers

Related Training

ForgeRock Access Management Core Concepts (AM-400)

Related Issue Tracker IDs

OPENAM-11976 (XUI Session query session by username does not work with +)

OPENAM-11700 (ssoadm list-sessions fails with Failed to get the valid sessions from the specified server)

OPENAM-11178 (RFE: XUI Sessions List - provide an overview of all sessions)

OPENAM-9817 (JMX CTS session count is broken)

OPENAM-9738 (Enable CTS segregation to allow each token type to write to a different CTS instance)

OPENAM-9728 (Provide ability for all monitoring ports per OpenAM instance and not globally. )

OPENAM-6998 (Session "stats" logging doesn't seem to work)

OPENAM-6002 ("stats" logs directory and contents are not documented)


How do I set up a monitoring page for the load balancer in front of Web Policy Agents (All versions) for health checks?

The purpose of this article is to provide information on setting up a monitoring page for the load balancer in front of Web policy agents. Creating a monitoring page is best practice for health checking when you have a load balancer in front of your Web policy agents. This article assumes you have already configured your load balancer and policy agents.

Overview

Creating a monitoring page on the server being protected by the Web policy agent means you can eliminate the policy agent being involved in the load balancer's health check.

This monitoring page should be an unprotected resource and exist on the Not Enforced URL list. Even though the URL is on the Not Enforced URL list, the policy agent is still invoked each time the load balancer checks the monitoring page to determine whether the resource needs protecting or not; this means you can use this configuration to check if the policy agent is responding without the need for policy evaluation.

Creating a monitoring page

You can create a monitoring page as follows:

  1. Create a monitor.html file on one of the servers being protected by the Web policy agent. This file can simply contain the HTML tags; for example, you can use the printf command to create is as follows:
    $ printf '<HTML>\n</HTML>' > monitor.html
  2. Add the URL for this monitoring page to the Not Enforced URL list for this server:
    • AM 5 and later console: navigate to: Realms > [Realm Name] > Applications > Agents > Web > [Agent Name] > Application > Not Enforced URLs and add the URL for the monitoring page, for example:
      http://www.host1.example.com:8080/monitor.html 
    • OpenAM 13.x console: navigate to: Realms > [Realm Name] > Agents > Web > [Agent Name] > Application > Not Enforced URLs and add the URL for the monitoring page.
    • Pre-OpenAM 13 console: navigate to: Access Control > [Realm Name] > Agents > Web > [Agent Name] > Application > Not Enforced URLs and add the URL for the monitoring page.
    • ssoadm: enter the following command:
      $ ./ssoadm update-agent -e [realmname] -b [agentname] -u [adminID] -f [passwordfile] -a com.sun.identity.agents.config.notenforced.url[0]=[URL]
      replacing [realmname], [agentname], [adminID], [passwordfile] and [URL] with appropriate values.
  3. Repeat steps 1 and 2 on each server being protected by the Web policy agent.

The load balancer will now check the monitoring page URL, for example, http://www.host1.example.com:8080/monitor.html on each server to check if the server is up.

You can check this by navigating to the monitoring page URL and observing that you do not need to log in; if you check the agent debug log (when the debug level is set to All) you will see that the policy agent has been invoked to determine if the resource needs protecting and matches the URL on the Not Enforced URL list. For example:

2016-08-16 20:33:53.504    Debug 65819:7f22ec000950 all: in_not_enforced_list(http://host1.example.com:8080/monitor.html): matched 'http://host1.example.com:8080/monitor.html' entry in not-enforced list
2016-08-16 20:33:53.504    Debug 65819:7f22ec000950 all: in_not_enforced_list: Allowing access to http://host1.example.com:8080/monitor.html

See Also

How do I define a list of Not Enforce URLs that Web Policy Agents can ignore for authentication purposes in AM/OpenAM (All versions)?

How do I configure a Web Policy Agent (All versions) for SSL offloading?

FAQ: Configuring Policy Agents in AM/OpenAM

Agents and policies in AM/OpenAM

User Guide › Introducing Web Agents › Not-Enforced URL and Client IP Lists

User Guide › Introducing Web Agents › FQDN Checking

User Guide › Reference › Configuring Application Properties

User Guide › Reference › Configuring Global Properties

Related Training

N/A

Related Issue Tracker IDs

OPENAM-5693 (RFE: Web Agent to provide means of monitoring health and active state of Agent to AM communication )


OpenDJ/DS


FAQ: DS/OpenDJ performance and tuning

The purpose of this FAQ is to provide answers to commonly asked questions regarding DS/OpenDJ performance and tuning.

Frequently asked questions

Q. Do I need to performance test DS/OpenDJ?

A. Yes, performance testing is a very important step to ensure that the system you have configured will cope with the expected load once it is in production. By performance testing your system, you can tweak any performance issues you encounter before going live.

Before running any performance tests, you should ensure DS/OpenDJ is running and configured in a test environment that closely resembles your planned production environment. You should also be aware of the type of load expected in your production system and the types of user data involved to make your tests as realistic as possible.

Generating sample data in DS/OpenDJ is described in: How do I generate sample user data for performance testing in DS/OpenDJ (All versions)?

Q. How do I tune DS/OpenDJ to improve performance?

A. Tuning DS/OpenDJ is described in Administration Guide › Tuning Servers For Performance.

Q. What is the recommended way to load balance client connections?

A. In most scenarios, the way to load balance client connections is specific to your topology so we don't have definitive best practice recommendations. You may want to have pooled connections so that clients only open a set number of connections or your clients may not be able to pool connections and open individual connections for each request.

In either case, you may or may not want to load balance based on:

  • Number of opened connections.
  • System load of the DS/OpenDJ server.
  • Round robin.
  • Failover only
  • Etc...

You should consider setting an idle time limit to ensure idle connections are closed; in particular, you must set this lower than the idle time limit for the load balancer as detailed in Administration Guide › Limiting Idle Time.

if you want more tailored advice, consider engaging Deployment Support Services.

Q. What is the recommended Java® Virtual Machines (JVM) heap size for DS/OpenDJ?

A. There are no definitive rules for the size of JVM heap size required as it will vary across individual environments and applications, but you should refer to Best practice for JVM Tuning for best practice advice. You can use the size of a backend after doing an initial import-ldif of all the expected data to give you an indication of the size required and base your initial heap size on this.

See How do I tune DS/OpenDJ (All versions) process sizes: JVM heap and database cache?and Administration Guide › Java Settings for further information.

Note

You must ensure you configure JVM garbage collection appropriately as garbage collection runs can get quite long if you have large heap sizes.

Q. Can I change caching in DS/OpenDJ to improve performance?

A. Yes, you can improve performance in DS/OpenDJ by configuring the database cache and entry cache, although entry caches are generally not used in DS/OpenDJ (unlike DSEE). This is described in How do I tune DS/OpenDJ (All versions) process sizes: JVM heap and database cache?Administration Guide › Database Cache Settings and Administration Guide › Caching Large, Frequently Used Entries. You need to balance performance against memory usage when configuring caching as larger caches lead to improved performance but increase memory usage.

See Cache strategy for OpenDJ LDAP directory server for further information on devising a caching strategy.

Q. Are there things I should monitor on an ongoing basis in terms of DS/OpenDJ performance?

A. Yes, it is useful to monitor performance on an ongoing basis to allow you to react to changes quickly. Useful things to monitor include:

  • Heap size.
  • Number of open sessions / size of sessions.
  • CPU utilization - utilization of 70-80% is worth investigating.

See FAQ: Monitoring DS/OpenDJHow do I use cn=monitor entry in DS/OpenDJ (All versions) for monitoring?Administration Guide › Monitoring, Logging, and Alerts and How do I collect data for troubleshooting high CPU utilization on DS/OpenDJ (All versions) servers? for further information.

Note

You should also ensure the access log file is enabled; this log file contains information about operations the server process. This is described Administration Guide › Server Logs.

See Also

Unindexed searches causing slow searches and poor performance on DS/OpenDJ (All versions) server

Performance tuning and monitoring ForgeRock products

Administration Guide › Tuning Servers For Performance › Testing Performance

Administration Guide › Tuning Servers For Performance › Tweaking OpenDJ Performance

Related Training

ForgeRock Directory Services Core Concepts


How do I generate sample user data for performance testing in DS/OpenDJ (All versions)?

The purpose of this article is to provide information on generating sample user data for performance testing in DS/OpenDJ. It is generally recommended that you use sample data for performance testing rather than actual data. This ensures that you do not expose any sensitive data, such as passwords, and also gives you more consistent data for testing purposes.

Generating sample user data

You can generate sample user data using makeldif in DS or make-ldif in OpenDJ; this command uses a template file and data files to create sample data that you can then import into DS/OpenDJ. A sample template file (example.template located in the /path/to/ds/config/MakeLDIF/ directory) is included in DS/OpenDJ, which generates 10,001 LDAP entries. This sample template file points to other data files (located in the same directory) from which makeldif or make-ldif derives the data to populate the LDAP entries. 

You should create your own template and data files based on these sample files to ensure the test data you create is similar to your actual data. The template file is detailed below to help you create your own template and data files.

The process for generating the test data is described in: Administration Guide › Managing Directory Data › Generating Test Data

Caution

Before generating a very large LDIF file, make sure you have enough space on disk.

Template file

The example.template file looks like this:

define suffix=dc=example,dc=com
define maildomain=example.com
define numusers=10001

branch: [suffix]

branch: ou=People,[suffix]
subordinateTemplate: person:[numusers]

template: person
rdnAttr: uid
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
givenName: <first>
sn: <last>
cn: {givenName} {sn}
initials: {givenName:1}<random:chars:ABCDEFGHIJKLMNOPQRSTUVWXYZ:1>{sn:1}
employeeNumber: <sequential:0>
uid: user.{employeeNumber}
mail: {uid}@[maildomain]
userPassword: password
telephoneNumber: <random:telephone>
homePhone: <random:telephone>
pager: <random:telephone>
mobile: <random:telephone>
street: <random:numeric:5> <file:streets> Street
l: <file:cities>
st: <file:states>
postalCode: <random:numeric:5>
postalAddress: {cn}${street}${l}, {st}  {postalCode}
description: This is the description for {cn}.

This file breaks down into the following components with descriptions:

Component Description
define suffix=dc=example,dc=com 
define maildomain=example.com 
define numusers=10001
Defines global variables that are used throughout the makeldif or make-ldif process to build up LDAP entries. The suffix and maildomain are self explanatory; the numusers is the total number of LDAP entries that should be created.
branch: [suffix]
Creates a domain branch under dc=example,dc=com (the suffix variable). This branch should not already exist in your directory server, otherwise you will get an error when you attempt the import. The branch created in the ldif output file looks like this:
dn: dc=example,dc=com objectClass: top objectClass: domain dc: example {
[various aci settings]
}
branch: ou=People,[suffix]
Creates another branch under dc=example,dc=com. This branch is an organizational unit called People. The branch created in the ldif output file looks like this:
dn: ou=People,dc=example,dc=com objectClass: top objectClass: organizationalunit ou: People {
[various aci settings]
}
subordinateTemplate: person:[numusers]
Tells the makeldif or make-ldif process to run the person subordinate template 10001 times (the numusers variable).
template: person
Defines the type of subordinate template to use as makeldif or make-ldif can create multiple object class types. In this case it is person.
rdnAttr: uid 
objectClass: top 
objectClass: person 
objectClass: organizationalPerson 
objectClass: inetOrgPerson
Gives the object class definition section for a person; this is repeated for each LDAP entry. The first entry created in the ldif output file looks like this:
dn: uid=user.0,ou=People,dc=example,dc=com objectClass: top objectClass: person objectClass: organizationalperson objectClass: inetorgperson
givenName: <first>
Populates the givenName value from a randomly selected line in the sample first.names file.
sn: <last>
Populates the sn value from a randomly selected line in the sample last.names file.
cn: {givenName} {sn}
Assigns the cn value based on the populated givenName and sn values; makeldif or make-ldif is sequential so values must be assigned before being used to assign another attribute.
initials: {givenName:1}<random:chars:ABCDEFGHIJKLMNOPQRSTUVWXYZ:1>{sn:1}
Derives the three letter initials value. The initials are derived as follows in this template: the first letter of the givenName value is taken, followed by a randomly selected character from the string, followed by the first letter of the sn value.
employeeNumber: <sequential:0>
Assigns a sequential integer to the employeeNumber value starting with 0.
uid: user.{employeeNumber}
Creates a uid value by appending a sequential integer to user. starting with 0, for example, user.0, user.1 etc.
mail: {uid}@[maildomain]
Creates an email address from the derived uid value and the maildomain variable, for example, user.0@example.com
userPassword: password
Creates the userPassword value, which in this case equals password for all users. You could alternatively set this to something like P@ssw0rd! to capture a variety of characters if you plan on enforcing stronger passwords or you could use something like <random:alphanumeric:8> to generate a random 8 character password for each user.
telephoneNumber: <random:telephone> 
homePhone: <random:telephone> 
pager: <random:telephone> 
mobile: <random:telephone>
Generates random numbers for the telephoneNumber, homePhone, pager and mobile values using USA style telephone numbers. You could change this to suit other countries, for example, you could use +44 <random:numeric:4> <random:numeric:6> for UK based numbers.
street: <random:numeric:5> <file:streets> Street
Populates the street value by generating a 5 digit number followed by a random line from the sample streets file. 
l: <file:cities> 
st: <file:states>
Populates the l and st values from randomly selected lines in the sample cities and states files respectively. 
postalCode: <random:numeric:5> 
postalAddress: {cn}${street}${l}, {st}  {postalCode} 
description: This is the description for {cn}.
Populates the postalCode value by generating a 5 digit number. The postalAddress value is derived by concatenating the derived cn value, followed by $, the derived street value, followed by $ and then the derived l value. This is followed by the derived state value and the derived postalCode value. Finally, plain text (This is a description for) is displayed followed by the cn value.

See Also

FAQ: AM/OpenAM performance and tuning

Administration Guide › Managing Directory Data › Generating Test Data

DS Reference › Tools Reference › makeldif.template

OpenDJ Reference › Tools Reference › make-ldif.template

Administration Guide › Tuning Servers For Performance

Administration Guide › Tuning Servers For Performance › Testing Performance

Administration Guide › Tuning Servers For Performance › Tweaking OpenDJ Performance

Related Training

ForgeRock Directory Services Core Concepts 

Related Issue Tracker IDs

N/A


How do I tune DS/OpenDJ (All versions) process sizes: JVM heap and database cache?

The purpose of this article is to provide information on tuning DS/OpenDJ process sizes, including setting the JVM heap size and the database cache size.

Tuning heap size

Setting a larger heap size for the DS/OpenDJ servers is generally a good thing that improves performance. You should also ensure all DS/OpenDJ servers in a cluster have the same heap sizes.

The correct way to set the heap size is to edit the start-ds.java-args line in the java.properties file in the /path/to/ds/config directory. For example, to set the minimum and maximum heap sizes to 2GB, you would change this line as follows:

start-ds.java-args=-server -Xms2g -Xmx2g

In DS 5 and later, you must restart the DS server after you have edited the java.properties file.

In pre-DS 5, you must execute the /path/to/ds/bin/dsjavaproperties program (or the /path/to/ds/bat/dsjavaproperties.bat program if you are using Microsoft® Windows®) and restart the OpenDJ server after you have edited the java.properties file. This command has been removed in DS 5.

Recommendations 

  • Set the minimum and maximum heap sizes to the same value for best performance. Otherwise, the JVM runs a full Garbage Collector (GC) cycle when increasing its heap, during which time it can pause ongoing operations for up to a few seconds. Generally, a smaller heap will increase the frequency at which the JVM GC executes but reduce the duration; similarly, a larger heap will reduce the frequency and increase the duration. When tuning the JVM heap, the goal is to strike a balance between frequency and duration so as to reduce the impact of the GC on the application.
  • If you are using a 64-bit JVM and your heap size is less than 32GB, you should also specify CompressedOops by adding -XX:+UseCompressedOops to this line. For 64-bit JVMs, explicitly adding CompressedOops is necessary so that the DS/OpenDJ database can calculate its available memory and caches, and accordingly size its cache properly. For example:
    start-ds.java-args=-server -Xms3g -Xmx3g -XX:+UseCompressedOops
  • Typically, a 64-bit JVM needs a larger heap size than an equivalent 32-bit JVM as a 64-bit JVM has a larger memory footprint due to its object headers and object references being greater in size. An approximate guide is the heap size should be 1.5 times bigger in a 64-bit JVM compared to a similar 32-bit JVM. 

Tuning database cache size

As of DS 6.5, new DS servers use a shared cache by default for all JE database backends. See Release Notes › New Features for further information.

In earlier versions, you have to configure this cache per JE backend.

Cache size for individual JE database backends (pre-DS 6.5)

For optimal performances, database (DB) cache size should be larger than the sum of the DB file sizes.

First check the size of /path/to/ds/db/userRoot. If the DB has just been imported or hasn't been updated much since import, double that size. This will give you a maximum size the DB can grow to with the current data. You can then budget for the growth of the user data.

Once you have an estimated overall size of the DB, you should make sure that it can fit in the DB cache, which is by default estimated as a percentage of the overall heap size. The default cache size percentage (db-cache-percent) is 50%. You should adjust the database cache settings to avoid allocating all memory available to the JVM to database cache when creating additional database backends. Over-allocating memory to database cache leads to out of memory errors.

You can change the db-cache-percent using the dsconfig command. For example, to increase it to 60%:

$ ./dsconfig set-backend-prop --port 4444 --bindPassword password --backend-name userRoot --set db-cache-percent:60 --trustAll --no-prompt
Note

We've run tests with many heap sizes, including some very large ones, and most of the large deployments use between 2GB and 32GB of JVM heap size.

See Also

Best practice for JVM Tuning

FAQ: DS/OpenDJ performance and tuning

How do I tune the DS/OpenDJ (All versions) database file cache?

How do I tune Background Database Verification in DS (All versions) and OpenDJ 3.5.2, 3.5.3?

How do I enable Garbage Collector (GC) Logging for DS/OpenDJ (All versions)?

Administration Guide › Tuning Servers For Performance

Configuration Reference › JE Backend

Configuration Reference › Entry Cache

Related Training

ForgeRock Directory Services Core Concepts (DS-400) 

Related Issue Tracker IDs

N/A


How do I tune the DS/OpenDJ (All versions) database file cache?

The purpose of this article is to provide information on tuning the DS/OpenDJ database file cache.

Tuning database file cache

The je.log.fileCacheSize property represents the number of file handles that are cached, that is, file handles that remain open by the underlying database (DB). The default value is 200 (100 in pre-DS 6), which is probably far less than the number of .jdb files you have in the /path/to/ds/db/userRoot directory.

Note

You should also be aware of the maximum number of file handlers defined in DS/OpenDJ as discussed in the release notes applicable to your version: Release Notes › Setting Maximum Open Files.

When the DB cannot be fully cached, the DB needs to access the record from the log files. Since only a portion of the files can be opened at one time, the DB keeps closing them and opening the ones it needs. Each close does a disk sync, which has a large penalty on performance.

The number of .jdb files depends on the DB content and also on the occupancy ratio. Occupancy ratio has an impact on the number of files. Berkeley DB JE uses a rolling log algorithm in which writes to the DB are always appended to the current log file. When the log reaches a certain size, JE creates a new one and writes to it. When records are deleted or even updated, the old record is marked as deleted and the new one is appended to the log file. This creates unused records in the previous log files. The occupancy ratio is the ratio of used records / total number of records per log file. When the ratio for one log file hits a certain limit (50% by default), the remaining valid records are copied to the current log file (still appending) and the old log file is deleted. So in theory and in the worse case, the number of log files cannot exceed twice the initial number of files after an import (with the same amount of data).

To set a proper value for the je.log.fileCacheSize, therefore, count the current number of .jdb files, double it and round down.

If it happens that the number of .jdb files exceeds the value set, then the DB starts closing files and reopening other ones, slowly degrading performance.

This property can be maintained using the db-log-filecache-size configuration attribute, for example:

$ ./dsconfig set-backend-prop --port 4444 --bindPassword password --backend-name userRoot --advanced --set db-log-filecache-size:[cachesize] --trustAll --no-prompt

See Also

FAQ: DS/OpenDJ performance and tuning

How do I tune DS/OpenDJ (All versions) process sizes: JVM heap and database cache?

Performance tuning and monitoring ForgeRock products

Configuration Reference › JE Backend

Related Training

ForgeRock Directory Services Core Concepts (DS-400)

Related Issue Tracker IDs

OPENDJ-383 (Expose JE fileCacheSize property through OpenDJ configuration and admin)


How do I tune Background Database Verification in DS (All versions) and OpenDJ 3.5.2, 3.5.3?

The purpose of this article is to provide information on tuning Background Database Verification in DS/OpenDJ. The JE 7 database periodically performs automatic database verification, which can impact service availability.

Overview

Version 7 of JE introduced a feature that verifies the entire database at midnight every night to check for some common types of database corruption. The extra disk I/O can result in problems for performance-critical deployments. All the servers in a single timezone will run the verification at exactly the same time, which can also cause more widespread issues. The JE 7 embedded database has been used in backends since OpenDJ 3.5.2.

Note

Recovering from any database corruption would typically involve reinitializing the replica directory server or restoring it from a backup taken before the corruption occurred. See FAQ: Backup and restore in DS/OpenDJ for further information.

See the following sections for information on configuring the JE verifier depending on version:

In OpenDJ 3.5.2 and OpenDJ 3.5.3, you could also use the PDB database for backends; the functionality described here does not apply if you use the PDB backend. The PDB backend type was deprecated in DS 5 and removed in DS 6. 

DS 6 and later

DS 6 and later provides advanced properties for configuring the JE verifier (db-run-log-verifier and db-log-verifier-schedule), which allow you to:

  • Configure the JE verifier schedule
  • Disable the JE verifier

See Configuration Reference › Advanced Properties for further information.

Configuring the JE verifier schedule

By default, the JE verifier runs automatically at midnight local time. It is best practice to alter the verification schedule on each server to avoid all servers simultaneously using extra disk I/O and potentially impacting the entire service.

The verifier schedule is a cron-style field and can be set using the dsconfig command. The following example sets the verifier for the userRoot backend to run at 1am (local time):

$ ./dsconfig set-backend-prop --backend-name userRoot --set db-run-log-verifier:true --set 'db-log-verifier-schedule:0 1 * * *' --hostname ds1.example.com --port 4444 --bindDn "cn=Directory Manager" --bindPassword password --trustAll --no-prompt

You must restart the DS server to apply these changes.

Disabling the JE verifier

The JE verifier can be completely disabled, which means that some types of database corruption will not be reported.

You can use the dsconfig command to disable verification on a single backend. For example, to disable it on the userRoot backend:

$ ./dsconfig set-backend-prop --backend-name userRoot --set db-run-log-verifier:false --hostname ds1.example.com --port 4444 --bindDn "cn=Directory Manager" --bindPassword password --trustAll --no-prompt

You must restart the DS server to apply these changes.

Pre-DS 6

In earlier versions, you can configure the JE verifier in one of two ways:

  • Configure the JE verifier schedule
  • Disable the JE verifier

Configuring the JE verifier schedule

By default, the JE verifier runs automatically at midnight local time. It is best practice to alter the verification schedule on each server to avoid all servers simultaneously using extra disk I/O and potentially impacting the entire service.

The verifier schedule is a cron-style field and can be set using the dsconfig command. The following example sets the verifier for the userRoot backend to run at 1am (local time):

$ ./dsconfig set-backend-prop --backend-name userRoot --set 'je-property:je.env.verifySchedule:0 1 * * *' --hostname ds1.example.com --port 4444 --bindDn "cn=Directory Manager" --bindPassword password --trustAll --no-prompt

You must restart the DS/OpenDJ server to apply these changes.

Disabling the JE verifier

The JE verifier can be completely disabled, which means that some types of database corruption will not be reported.

You can use the dsconfig command to disable verification on a single backend. For example, to disable it on the userRoot backend:

$ ./dsconfig set-backend-prop --backend-name userRoot --set je-property:je.env.runVerifier=false --hostname ds1.example.com --port 4444 --bindDn "cn=Directory Manager" --bindPassword password --trustAll --no-prompt

You must restart the DS/OpenDJ server to apply these changes.

See Also

Installing and Administering DS/OpenDJ

Performance tuning and monitoring ForgeRock products

Related Training

N/A

Related Issue Tracker IDs

OPENDJ-4465 (Facilitate configuration of JE backend's DB verification feature)

OPENDJ-4418 (Investigate how to disable automatic JE database verification using JE properties)


How do I improve performance when using the PBKDF2 storage scheme in DS/OpenDJ (All versions)?

The purpose of this article is to provide information on improving performance when using the PBKDF2 storage scheme in DS/OpenDJ, including how and why you might change the number of iterations performed. The default number of iterations is 10,000.

Overview

PBKDF2 is designed to be flexible in the number of iterations used for computing the hash. This is intended so that it can be extended over time to keep up to date with current CPU speeds, and so that an implementation can be tailored to specific hardware or security requirements. Requirements are also likely to be different depending on whether hashes are calculated client-side or server-side for authentication attempts.

The higher the number of iterations, the longer it takes to compute a hash from a plain text password, and therefore the harder it becomes to run brute force attacks against a known hash.

You can improve performance as follows:

  • Upgrade to OpenDJ 2.6.4 or later / DS, which improves the concurrency of the PBKDF2 password storage scheme.
  • Tune the number of iterations used for the storage scheme; this is described in detail below.
  • Use the Bouncy Castle JCE provider, which improves the PBKDF2 authentication rate; in tests, we have found this to improve authentication rates by about 30%. This is a third-party library for Java® which contains additional algorithms and also faster versions of the algorithms already in Java. An overview of the setup required is provided below.
Note

Bouncy Castle is a third-party open source library that is not supported by ForgeRock.

Our performance tests

We've made some comparisons between Java 7 and Bouncy Castle. Bouncy Castle's performance over Java 7 showed a marked improvement.

BIND etimes with 100000 PBKDF2 iterations with the Java 7 JCE:

300ms

BIND etimes with 100000 PBKDF2 iterations with Bouncy Castle v1.52:

200ms

BIND etimes for fewer iterations are proportionally faster with Bouncy Castle.

Tuning PBKDF2 performance

In a scenario where DS/OpenDJ (the server) receives passwords in plain text and needs to run the algorithm itself, the benefits of having a large number of iterations should be balanced with not overwhelming the server.

To tune this trade-off of security versus performance, you can alter the number of iterations used for the storage scheme using dsconfig in interactive mode, see:

dsconfig: Password Storage Scheme > View and edit an existing... > PBKDF2 > pbkdf2-iterations

The change won't affect currently stored passwords. The new iteration value will be used when passwords are next changed. At this time, they'll be rehashed with the updated number of iterations.

The number of iterations is prepended to the final stored password hash so different users with different levels of PBKDF2 hashed passwords can co-exist simultaneously.

Setting up Bouncy Castle

The following process provides an overview of installing Bouncy Castle: 

  1. Download the latest bcprov-ext-jdk15on-xxx.jar and bcprov-jdk15on-xxx.jar files from Bouncy Castle; they are listed in the SIGNED JAR FILES section.
  2. Copy these two jar files to a directory that the JVM searches; the JVM searches the $JAVA_HOME/jre/lib/ext/ directory by default, so this is a good place to put them.
  3. Ensure the file permissions for these two jar files are set to allow them to be read.
  4. Update the list of security providers in the JVM to put Bouncy Castle first and then renumber the other security providers to follow. This list is set in the java.security text file (located in the $JAVA_HOME/jre/lib/security/ directory). The security provider list should now look similar to this:
    security.provider.1=org.bouncycastle.jce.provider.BouncyCastleProvider
    security.provider.2=sun.security.provider.Sun
    [...]
    This step is recommended by Bouncy Castle and you can read more about it here: The Legion of the Bouncy Castle - Specifications.
  5. Save this file and restart DS/OpenDJ.

You can check the installation has been successful using a command such pfiles or lsof on DS/OpenDJ and look for these two bcprov jar files.

See Also

How does DS/OpenDJ (All versions) store password values?

Administration Guide › Configuring Password Policy

Configuration Reference › PBKDF2 Password Storage Scheme

Administration Guide › Tuning Servers For Performance

Administration Guide › Testing Performance

Related Training

N/A

Related Issue Tracker IDs

OPENDJ-2151 (High contention with the PBKDF2 password storage scheme.)


How do I know what index types are needed for search filters in DS/OpenDJ (All versions)?

The purpose of this article is to provide information on what index types are needed for search filters in DS/OpenDJ.

Index types needed for search filters

A search with single-level or subtree scope will need indexes. The index types required depend on all the terms in the search filter and are illustrated in the following table (the uid attribute is used as an example):

Filter Term Index Required
(uid=*) uid presence
(uid=m*) uid substring
(uid=*m) uid substring
(uid=*m*) uid substring
(uid=m*m) uid substring
(uid>=10) uid ordering
(uid<=10) uid ordering
(uid=m) uid equality
(uid~=doe) uid approximate

Further information about the different index types, with examples, is provided in Administration Guide › Indexing Attribute Values › Index Types and Their Functions.

See Also

Unindexed searches causing slow searches and poor performance on DS/OpenDJ (All versions) server

How do I troubleshoot issues with my indexes in DS/OpenDJ (All versions)?

Administration Guide › Indexing Attribute Values › Index Types and Their Functions

Related Training

ForgeRock Directory Services Core Concepts

Related Issue Tracker IDs

N/A


How do I enable Garbage Collector (GC) Logging for DS/OpenDJ (All versions)?

The purpose of this article is to provide information on enabling GC Logging for DS/OpenDJ. It assumes that you already have a working DS/OpenDJ server that is installed.

Enabling GC Logging

Note

You should ensure there is enough disk space for the GC logs; if not, you should enable GC log rotation as well by including the following options when you enable GC logging: -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=n, -XX:GCLogFileSize=n

 You can enable GC logging as follows:

  1. Locate and edit your /path/to/ds/config/java.properties file for the DS/OpenDJ instance for which you want to enable GC Logging.
  2. Append the following options to the entry start-ds.java-args=
    -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]
    replacing [filename] with the path to the file that you would like to create to store the log file. For example:
    start-ds.java-args=-server -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:/tmp/gc.log
  3. Pre-DS 5 only: Execute the /path/to/ds/bin/dsjavaproperties program (or the /path/to/ds/bat/dsjavaproperties.bat program if you are using Microsoft® Windows®). You should get the following output:
    The operation was successful. The server commands will use the java arguments and java home specified in the properties file located in /path/to/ds/config/java.properties. 
    The dsjavaproperties command has been removed in DS 5.
  4. Restart the DS/OpenDJ server.

Once the server has successfully restarted, there should be a GC log file located in the directory specified in the -Xloggc: option. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

See Also

Best practice for JVM Tuning

How do I tune DS/OpenDJ (All versions) process sizes: JVM heap and database cache?

How do I collect JVM data for troubleshooting DS/OpenDJ (All versions)?

FAQ: DS/OpenDJ performance and tuning

Administration Guide › Tuning Servers For Performance › Java Settings

Related Training

ForgeRock Directory Services Core Concepts 

Related Issue Tracker IDs

N/A


Monitoring


How do I create a dedicated user for monitoring in DS 6.x?

The purpose of this article is to provide assistance if you need to create a user for DS monitoring.

Overview

When installing DS 6 and later, you are given the option to create a default user for querying monitoring information (see Installation Guide › To Set Up a Directory Server for further information). If you did not create a user at this stage, you can use one of the following options to create one post install:

Once you have created your user/account, you can then enable the appropriate connection handler to expose the required monitoring endpoints:

After completing these steps, you will be ready to start monitoring using your user/account per Administration Guide › Monitoring, Logging, and Alerts

Granting monitoring rights to an existing account

To grant access to an existing user:

  1. Create an LDIF file to apply the ds-privilege-name: monitor-read to the user's entry, for example:
    $ cat monitor-read.ldif
    dn: uid=jdoe,ou=People,dc=example,dc=com
    changetype: modify
    add: ds-privilege-name
    ds-privilege-name: monitor-read
  2. Apply the changes using the following ldapmodify command:
    $ ./ldapmodify --port 1389 --bindDN "cn=Directory Manager" --bindPassword password monitor-read.ldif
  3. Verify the changes are applied, for example:
    $ ./ldapsearch --port 1389 --bindDN "uid=jdoe,ou=People,dc=example,dc=com" --bindPassword password --baseDN cn=monitor "(ds-cfg-backend-id=userRoot)"
    
    objectClass: top
    objectClass: ds-monitor
    objectClass: ds-monitor-backend
    objectClass: ds-monitor-backend-pluggable
    objectClass: ds-monitor-backend-db
    ds-mon-backend-is-private: false
    ds-mon-backend-entry-count: 2002
    ds-mon-backend-writability-mode: enabled
    ds-mon-backend-degraded-index-count: 0
    ds-mon-backend-ttl-is-running: false
    ds-mon-backend-ttl-last-run-time: 20180809151924.210Z
    ds-mon-backend-ttl-thread-count: 0
    ds-mon-backend-ttl-queue-size: 0
    ds-mon-backend-ttl-entries-deleted: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000}
    ds-mon-backend-filter-use-start-time: 19700101000000Z
    ds-mon-backend-filter-use-indexed: 0
    ds-mon-backend-filter-use-unindexed: 0
    ds-mon-db-version: 7.5.11
    ds-mon-db-cache-evict-internal-nodes-count: 0
    ds-mon-db-cache-evict-leaf-nodes-count: 0
    ds-mon-db-cache-total-tries-internal-nodes: 1045
    ds-mon-db-cache-total-tries-leaf-nodes: 882
    ds-mon-db-cache-misses-internal-nodes: 12
    ds-mon-db-cache-misses-leaf-nodes: 70
    ds-mon-db-cache-size-active: 3230029
    ds-mon-db-log-size-active: 4604407
    ds-mon-db-log-cleaner-file-deletion-count: 0
    ds-mon-db-log-utilization-min: 56
    ds-mon-db-log-utilization-max: 56
    ds-mon-db-log-size-total: 4604407
    ds-mon-db-log-files-open: 1
    ds-mon-db-log-files-opened: 5
    ds-mon-db-checkpoint-count: 0
    ds-cfg-backend-id: userRoot
    

Creating a dedicated monitoring account

To create a dedicated user, you can set up an LDIF backend and apply the privileges (which is what DS does during install if this option is selected): 

  1. Create a directory for the user in the /path/to/ds/db directory (this is where the special users are created during install), for example:
    $ mkdir /path/to/ds/db/monitorUser
    
  2. Create an encoded password for the monitoring user, for example:
    $ ./encode-password -c password -s SSHA512
    {SSHA512}NUhN2CulFulVLDDJo+6uZ2NhjSpkl2iFn2sRNgFvjnZM/0LddL3hXPAecLTALCYnKfE+64lXiWwBfPvgYJR+0fL1ojGvsruE
  3. Create an LDIF file that contains the monitoring user entry including the encoded password from step 2, for example:
    $ cat /path/to/ds/db/monitorUser/monitorUser.ldif
    dn: uid=Monitor
    objectClass: top
    objectClass: person
    objectClass: organizationalPerson
    objectClass: inetOrgPerson
    sn: User
    cn: Monitor
    ds-privilege-name: monitor-read
    userPassword: {SSHA512}NUhN2CulFulVLDDJo+6uZ2NhjSpkl2iFn2sRNgFvjnZM/0LddL3hXPAecLTALCYnKfE+64lXiWwBfPvgYJR+0fL1ojGvsruE
    
  4. Create an LDIF backend:
    $ ./dsconfig create-backend --set base-dn:uid=Monitor --set enabled:true --set ldif-file:/path/to/ds/db/monitorUser/monitorUser.ldif --type ldif --backend-name monitorUser --port 4444 --bindDn "cn=Directory Manager" --bindPassword password --trustAll --no-prompt
    
  5. Verify the monitoring user has read access to cn=monitor, for example:
    $ ./ldapsearch --port 1389 --bindDN "uid=Monitor" --bindPassword password --baseDN cn=monitor "(ds-cfg-backend-id=userRoot)"
    
    objectClass: top
    objectClass: ds-monitor
    objectClass: ds-monitor-backend
    objectClass: ds-monitor-backend-pluggable
    objectClass: ds-monitor-backend-db
    ds-mon-backend-is-private: false
    ds-mon-backend-entry-count: 2002
    ds-mon-backend-writability-mode: enabled
    ds-mon-backend-degraded-index-count: 0
    ds-mon-backend-ttl-is-running: false
    ds-mon-backend-ttl-last-run-time: 20180809151924.210Z
    ds-mon-backend-ttl-thread-count: 0
    ds-mon-backend-ttl-queue-size: 0
    ds-mon-backend-ttl-entries-deleted: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000}
    ds-mon-backend-filter-use-start-time: 19700101000000Z
    ds-mon-backend-filter-use-indexed: 0
    ds-mon-backend-filter-use-unindexed: 0
    ds-mon-db-version: 7.5.11
    ds-mon-db-cache-evict-internal-nodes-count: 0
    ds-mon-db-cache-evict-leaf-nodes-count: 0
    ds-mon-db-cache-total-tries-internal-nodes: 1045
    ds-mon-db-cache-total-tries-leaf-nodes: 882
    ds-mon-db-cache-misses-internal-nodes: 12
    ds-mon-db-cache-misses-leaf-nodes: 70
    ds-mon-db-cache-size-active: 3230029
    ds-mon-db-log-size-active: 4604407
    ds-mon-db-log-cleaner-file-deletion-count: 0
    ds-mon-db-log-utilization-min: 56
    ds-mon-db-log-utilization-max: 56
    ds-mon-db-log-size-total: 4604407
    ds-mon-db-log-files-open: 1
    ds-mon-db-log-files-opened: 5
    ds-mon-db-checkpoint-count: 0
    ds-cfg-backend-id: userRoot

See Also

How do I use cn=monitor entry in DS/OpenDJ (All versions) for monitoring?

FAQ: Monitoring DS/OpenDJ

Performance tuning and monitoring ForgeRock products

Release Notes › New Features

Related Training

N/A

Related Issue Tracker IDs

N/A


FAQ: Monitoring DS/OpenDJ

The purpose of this FAQ is to provide answers to commonly asked questions regarding monitoring in DS/OpenDJ.

Frequently asked questions

Q. How can I check that DS/OpenDJ is up and running?

A. You can perform a heartbeat check against DS/OpenDJ to verify that the server is up and running.

See How do I perform a heartbeat check against DS/OpenDJ (All versions)?for further information.

Q. How can I check if a backend is online?

A. You can check if a backend is online by performing a ldapsearch against the specific backend.

See How do I check if a backend is online in DS/OpenDJ (All versions)? for further information.

Q. How can I monitor replication?

A. You can use the dsreplication status command to give you an overall view of the replication topology, including whether the servers are synchronized. For example:

$ ./dsreplication status --adminUID admin --adminPassword password --hostname ds1.example.com --port 4444 --trustAll

See How do I troubleshoot replication issues in DS/OpenDJ (All versions)? and Reference › dsreplication for further information.

Note

The M.C.and A.O.M.C. metrics returned from the dsreplication status command are deprecated in DS 6 and replaced with replication delay in DS 6.5. You should monitor replication delay instead; you can also monitor this over LDAP and HTTP as detailed in Administration Guide › Monitoring Replication Delay Over LDAP and Administration Guide › Monitoring Replication Delay Over HTTP.

Q. How can I monitor performance?

A. You can monitor performance by performing a ldapsearch against cn=monitor to return operation statistics.

See How do I use cn=monitor entry in DS/OpenDJ (All versions) for monitoring? for further information on monitoring operation statistics.

Q. What can I monitor with the cn=monitor entry?

A. DS/OpenDJ exposes a lot of different monitoring information over LDAP under this entry.

See How do I use cn=monitor entry in DS/OpenDJ (All versions) for monitoring? for further information, including specific examples you may want to monitor.

Q. Do the statistics under the cn=monitor entry persist when the server is restarted?

A. No, the statistics under cn=monitor are reset each time the server is restarted.

Q. Do I have to use the Directory Manager user for JMX monitoring?

A. No, you can use any user for JMX monitoring; you just need to add the JMX privileges (jmx-notify, jmx-read, and jmx-write) to the user you want to access JMX monitoring. No users have access to JMX monitoring by default.

See Administration Guide › JMX-Based Monitoring for an example of how to add these privileges to a user (Directory Manager, but you can substitute any user you want).

Q. How can I connect to JMX to ensure mbeans are returned?

A. You must be authenticated to the DS/OpenDJ server via a JMX client to see the mbeans with associated attributes and operations. Authenticating to the server is the only way to expose the sensitive elements within the mbeans; connecting to the process directly will not show them. Additionally, you must ensure the user who authenticates has JMX privileges.

See JMX Client Access and JMX-Based Monitoring for more information.

Q. Can I change the default listen-address for the JMX Connection Handler?

A. Yes, as of OpenDJ 2.6.2 you can change the default listen-address. The default listen-address for the JMX Connection Handler is 0.0.0.0.

Q. What URL should I use in JConsole to log in?

A. You must always use a remote URL, even if you are using a local connection to the JVM. For example:

service:jmx:rmi:///jndi/rmi://localhost:1689/org.opends.server.protocols.jmx.client-unknown

You can substitute a hostname alias (such as ds1.example.com) or an IP address for the localhost part of this URL. 

Note

If SSL is enabled, you must set up the truststore on the system where JConsole is running and configure SSL properly to monitor a remote application. See Using JConsole for further information.

See Also

FAQ: DS/OpenDJ performance and tuning

How do I generate sample user data for performance testing in DS/OpenDJ (All versions)?

Unindexed searches causing slow searches and poor performance on DS/OpenDJ (All versions) server

Administration Guide › Monitoring, Logging, and Alerts

Related Training

ForgeRock Directory Services Core Concepts


How do I use cn=monitor entry in DS/OpenDJ (All versions) for monitoring?

The purpose of this article is to provide information on using the cn=monitor entry in DS/OpenDJ for monitoring purposes. DS/OpenDJ exposes monitoring information over LDAP under this entry.

Overview

You can perform a ldapsearch against the cn=monitor entry and sub-entries to provide a variety of statistics that are useful for monitoring DS/OpenDJ. 

Some key baseDNs to search are:

Version baseDN Details
All cn=monitor Provides general server information (an example of the type of information returned is shown below).
All cn=Disk Space Monitor,cn=monitor Provides information about disks, including disk location and space available.
All cn=Work Queue,cn=monitor Provides information about the work queue, including its backlog, average and max backlog. 
DS 6 and later cn=jvm,cn=monitor

Provides information about the system and the JVM, including memory usage.

Replaces cn=System Information,cn=monitor and cn=JVM Memory Usage,cn=monitor

DS 6 and later cn=LDAP,cn=connection handlers,cn=monitor

Provides information about all open client connections.

Replaces cn=Client Connections,cn=monitor

DS 6 and later

ds-cfg-backend-id=userRoot,cn=Backends,cn=monitor

Provides monitoring information about the Berkeley DB Java Edition backend.

Replaces cn=userRoot Database Environment,cn=monitor

Pre-DS 6 cn=System Information,cn=monitor Provides information about the system and the JVM.
Pre-DS 6 cn=JVM Memory Usage,cn=monitor Provides information about memory usage in the JVM.
Pre-DS 6 cn=Client Connections,cn=monitor Provides information about all open client connections.
Pre-DS 6 cn=userRoot Database Environment,cn=monitor Provides monitoring information about the Berkeley DB Java Edition backend.

Examples

See the following sections for useful monitoring examples:

Note

All the examples use the standard non-secure port (389). You should adjust this according to your environment; in particular, if you are using production mode or have the LDAPS Connection Handler enabled, you should use --port 1636 --useSsl --trustAll instead.

Using the cn=monitor entry

Note

Returning all monitoring information using the below command is a great way to initially find the LDAP metrics and corresponding LDAP attributes that you’re looking to measure; you can then construct specific searches once you know the baseDN and attributes you're interested in.

You can return all monitoring information using a command such as:

$ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=monitor" --searchScope sub "(objectClass=*)" \*

An example output in DS 6 looks like this:

dn: cn=monitor
objectClass: top
objectClass: ds-monitor
objectClass: ds-monitor-server
objectClass: extensibleObject
ds-mon-product-name: ForgeRock Directory Services
ds-mon-short-name: OpenDJ
ds-mon-vendor-name: ForgeRock AS.
ds-mon-full-version: ForgeRock Directory Services 6.0.0
ds-mon-compact-version: OpenDJ-6.0.0
ds-mon-current-connections: 3
ds-mon-max-connections: 3
ds-mon-total-connections: 17

...

dn: cn=Administration Connector,cn=monitor
objectClass: top
objectClass: ds-monitor
objectClass: ds-monitor-connection-handler
objectClass: ds-monitor-ldap-connection-handler
ds-mon-config-dn: cn=Administration Connector,cn=config
ds-mon-protocol: LDAPS
ds-mon-listen-address: 0.0.0.0:6444
ds-mon-active-connections-count: 0
ds-mon-connections: {"count":8,"total":8.000,"mean_rate":0.001,"m1_rate":0.000,"m5_r
ate":0.000,"m15_rate":0.000}
ds-mon-bytes-read: {"count":474,"total":100877.000,"mean_rate":18.642,"m1_rate":0.00
0,"m5_rate":0.000,"m15_rate":0.676}
ds-mon-bytes-written: {"count":1181,"total":444547.000,"mean_rate":82.151,"m1_rate":
0.000,"m5_rate":0.001,"m15_rate":3.112}
ds-mon-active-persistent-searches: 0
ds-mon-abandoned-requests: 0
ds-mon-requests-abandon: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000,
"m5_rate":0.000,"m15_rate":0.000,"mean":0.000,"min":0.000,"max":0.000,"stddev":0.000
,"p50":0.000,"p75":0.000,"p95":0.000,"p98":0.000,"p99":0.000,"p999":0.000,"p9999":0.
000,"p99999":0.000}
ds-mon-requests-add: {"count":3,"total":50.000,"mean_rate":0.001,"m1_rate":0.000,"m5
_rate":0.000,"m15_rate":0.000,"mean":16.707,"min":0.999,"max":37.224,"stddev":15.102
,"p50":12.059,"p75":12.059,"p95":37.224,"p98":37.224,"p99":37.224,"p999":37.224,"p99
99":37.224,"p99999":37.224}
ds-mon-requests-bind: {"count":7,"total":773.000,"mean_rate":0.001,"m1_rate":0.000,"
m5_rate":0.000,"m15_rate":0.000,"mean":110.414,"min":10.945,"max":272.630,"stddev":7
8.111,"p50":76.022,"p75":126.353,"p95":272.630,"p98":272.630,"p99":272.630,"p999":27
2.630,"p9999":272.630,"p99999":272.630}

...

Alternatively, you can perform more specific searches against individual baseDNs and include particular objectClass parameters and/or attributes to filter the information returned.

Monitoring replication

You can monitor replication for each Directory server and Replication server that the server searched knows about using a command similar to the following:

$ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=Replication,cn=monitor" --searchScope sub "(objectClass=*)" \*

Specifically, you would want to check the following attributes (depending on version), which can signify issues with replication if they do not equal to 0:

  • DS 6 and later: ds-mon-current-delay
  • Pre-DS 6:
    • missing-changes
    • approximate-delay

Monitoring operation statistics

You can monitor operation statistics using a command similar to the following:

  • DS 6 and later:
    $ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=monitor" --searchScope sub "(objectClass=ds-monitor-ldap-connection-handler)" \*
    
  • Pre-DS 6:
    $ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=monitor" --searchScope sub "(objectClass=ds-connectionhandler-statistics-monitor-entry)" \*

This command returns three sets of statistics:

  • Overall read/write statistics (count of messages, bytes etc).
  • Number of requests and responses per operation.
  • Performance metrics - total counts of each operation finished and the total execution time of these operations.

This final set of statistics can be really useful for monitoring performance. You can also determine average operation times by querying this on a regular basis and calculating the differences from the previous query. Both the average operations per second and etime per operation can be derived.

Monitoring the work queue

You can monitor the work queue using a command similar to the following:

$ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=Work Queue,cn=monitor" --searchScope sub "(objectClass=*)" \*

This command will give you an understanding of how busy your worker threads are as the queue will typically stay empty if there are free worker threads. This can be seen by checking:

  • ds-mon-requests-rejected-queue-full (requestsRejectedDueToQueueFull in pre-DS 6), which only increments when the queue is full.
  • ds-mon-requests-in-queue (maxRequestBacklog, currentRequestBacklog and averageRequestBacklog in pre-DS 6), which indicates queue size. 

Monitoring database size

You can monitor the database size using a command similar to the following:

  • DS 6 and later:
    $ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=monitor" --searchScope sub "(|(ds-mon-backend-entry-count=*)(ds-mon-base-dn-entry-count=*))" \*
    
  • Pre-DS 6:
    $ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=monitor" --searchScope sub "(objectClass=ds-backend-monitor-entry)" \*

This command will return database size details for each backend, where:

  • ds-mon-backend-entry-count (ds-backend-entry-count in pre-DS 6) shows the total number of entries in the the backends.
  • ds-mon-base-dn-entry-count (ds-base-dn-entry-count in pre-DS 6) shows the total number of entries per base DN.

Monitoring active users

You can monitor users who are currently connected to the DS/OpenDJ server using a command similar to the following:

  • DS 6 and later:
    $ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=monitor" --searchScope sub "(objectClass=ds-monitor-connection*)" \*
    
  • Pre-DS 6:
    $ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=monitor" --searchScope sub "(&(objectClass=ds-monitor-entry)(connection=*))" \*

This command searches the connection handlers and returns the connection attribute. An example output in DS 6 looks like this:

dn: cn=Administration Connector,cn=monitor
objectClass: top
objectClass: ds-monitor
objectClass: ds-monitor-connection-handler
objectClass: ds-monitor-ldap-connection-handler
ds-mon-config-dn: cn=Administration Connector,cn=config
ds-mon-protocol: LDAPS
ds-mon-listen-address: 0.0.0.0:4444
ds-mon-active-connections-count: 0
ds-mon-connections: {"count":6,"total":6.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000}
ds-mon-bytes-read: {"count":117,"total":25811.000,"mean_rate":0.084,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000}
ds-mon-bytes-written: {"count":241,"total":155863.000,"mean_rate":0.508,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000}
ds-mon-active-persistent-searches: 0
ds-mon-abandoned-requests: 0

...

dn: cn=LDAP,cn=connection handlers,cn=monitor
objectClass: top
objectClass: ds-monitor
objectClass: ds-monitor-connection-handler
objectClass: ds-monitor-ldap-connection-handler
ds-mon-config-dn: cn=LDAP,cn=connection handlers,cn=config
ds-mon-protocol: LDAP
ds-mon-listen-address: 0.0.0.0:389
ds-mon-connection: {"connID":14,"connectTime":"20190117102734Z","source":"127.0.0.1:36914","destination":"127.0.0.1:389","ldapVersion":3,"authDN":"cn=Directory Manager","ssf":0,"opsInProgress":0,"persistentSearches":0}
ds-mon-connection: {"connID":16,"connectTime":"20190117102934Z","source":"127.0.0.1:36920","destination":"127.0.0.1:389","ldapVersion":3,"authDN":"cn=Directory Manager","ssf":0,"opsInProgress":0,"persistentSearches":0}

...

See Also

Reference › Monitoring Metrics

Administration Guide › Monitoring, Logging, and Alerts

FAQ: Monitoring DS/OpenDJ

How do I perform a heartbeat check against DS/OpenDJ (All versions)?

How do I check if a backend is online in DS/OpenDJ (All versions)?

FAQ: DS/OpenDJ performance and tuning

Related Training

ForgeRock Directory Services Core Concepts (DS-400)

Related Issue Tracker IDs

OPENDJ-1479 (monitoring data / attributes shown for 'cn=monitor' are not documented)

OPENDJ-1882 (currentConnections from cn=monitor is not decremented when JMX connections close)


How do I perform a heartbeat check against DS/OpenDJ (All versions)?

The purpose of this article is to provide information on performing a heartbeat check against DS/OpenDJ. A heartbeat check allows you to perform a simple health check on the server to ensure it is up and running.

Overview

DS 6.5 and later servers provide health status checks for anonymous requests over HTTP and LDAP. This allows a remote application to check that a server is "alive" and "healthy". See Release Notes › What's New in 6.5 (Monitoring) for further information. In all versions, you can perform a simple health check using heartbeat connections.

Heartbeat connections

If your load balancer or application is capable of using a heartbeat connection to check if DS/OpenDJ is online and responding, use of a proper LDAP connection is key.

A proper heartbeat check should progress as follows to fully verify the status of the DS/OpenDJ server:

CONNECT -> BIND -> SEARCH -> UNBIND -> DISCONNECT

or

CONNECT -> BIND -> UNBIND -> DISCONNECT
Caution

It is important to disconnect again, else the heartbeat check can remain open and consume server resources

Performing a heartbeat check

You can use a command such as the following to connect, bind, issue a base (scope) level search on the baseDN of "", unbind and finally disconnect:

$ ./ldapsearch --port 4444 --bindDN "cn=Directory Manager" --useSsl --trustAll --baseDN "" --searchScope base "(objectClass=*)" 1.1
dn:

Replacing "cn=Directory Manager" with the bind DN of the account used to bind to the Configuration store in AM/OpenAM if different.

The results of the search operation show "dn:" (only). This is expected as the LDAP request attribute "1.1." returns the distinguished name only. In this case, you are searching the "rootDSE" and the expected dn: is "null". 

An example entry seen in the access logs for successful heartbeat check is shown below depending on your DS/OpenDJ version:

  • DS 5 and later (JSON log format):
    {"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"CONNECT","connId":2},"transactionId":"0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS"},"timestamp":"2018-03-15T16:44:49.400Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-1"}
    {"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"BIND","connId":2,"msgId":1,"version":"3","dn":"cn=Directory Manager","authType":"SIMPLE"},"transactionId":"5009191b-a09a-4c7f-84cd-e240b2810a67-2","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":3,"elapsedTimeUnits":"MILLISECONDS"},"userId":"cn=Directory Manager","timestamp":"2018-0315T16:44:49.687Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-4"}
    {"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"SEARCH","connId":2,"msgId":2,"dn":"","scope":"base","filter":"(objectClass=*)","attrs":["1.1"]},"transactionId":"5009191b-a09a-4c7f-84cd-e240b2810a67-5","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"timestamp":"2018-03-15T16:44:49.717Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-7"}
    {"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"UNBIND","connId":2,"msgId":3},"transactionId":"5009191b-a09a-4c7f-84cd-e240b2810a67-8","timestamp":"2018-03-15T16:44:49.729Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-10"}
    {"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"DISCONNECT","connId":2},"transactionId":"0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","reason":"Client Unbind"},"timestamp":"2018-03-15T16:44:49.734Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-12"}
    
  • Pre-DS 5 (file-based log format):
    [15/Mar/2018:16:44:49 -0600] CONNECT conn=2 from=203.0.113.0:52597 to=203.0.113.0:1389 protocol=LDAP 
    [15/Mar/2018:16:44:49 -0600] BIND REQ conn=2 op=0 msgID=1 version=3 type=SIMPLE dn="cn=Directory Manager" 
    [15/Mar/2018:16:44:49 -0600] BIND RES conn=2 op=0 msgID=1 result=0 authDN="cn=Directory Manager,cn=Root DNs,cn=config" etime=3 
    [15/Mar/2018:16:44:49 -0600] SEARCH REQ conn=2 op=1 msgID=2 base="" scope=baseObject filter="(objectClass=*)" attrs="1.1" 
    [15/Mar/2018:16:44:49 -0600] SEARCH RES conn=2 op=1 msgID=2 result=0 nentries=1 etime=1 
    [15/Mar/2018:16:44:49 -0600] UNBIND REQ conn=2 op=2 msgID=3 
    [15/Mar/2018:16:44:49 -0600] DISCONNECT conn=2 reason="Client Unbind"
    

Alternately, if your application can perform a Connect with a Bind only, this is a viable alternative:

  • DS 5 and later (JSON log format):
    {"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"CONNECT","connId":2},"transactionId":"0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS"},"timestamp":"2018-03-15T16:44:49.400Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-1"}
    {"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"BIND","connId":2,"msgId":1,"version":"3","dn":"cn=Heartbeat Admins":"SUCCESSFUL","statusCode":"0","elapsedTime":3,"elapsedTimeUnits":"MILLISECONDS"},"userId":"cn=Heartbeat Admin,cn=Root DNs,cn=config","timestamp":"2018-0315T16:44:49.687Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-4"}
    {"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"UNBIND","connId":2,"msgId":3},"transactionId":"5009191b-a09a-4c7f-84cd-e240b2810a67-8","timestamp":"2018-03-15T16:44:49.729Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-10"}
    {"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"DISCONNECT","connId":2},"transactionId":"0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","reason":"Client Unbind"},"timestamp":"2018-03-15T16:44:49.734Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-12"}
    
  • Pre-DS 5 (file-based log format):
    [15/Mar/2018:16:44:49 -0600] CONNECT conn=2 from=203.0.113.0:52597 to=203.0.113.0:1389 protocol=LDAP 
    [15/Mar/2018:16:44:49 -0600] BIND REQ conn=2 op=0 msgID=1 version=3 type=SIMPLE dn="cn=Heartbeat Admin" 
    [15/Mar/2018:16:44:49 -0600] BIND RES conn=2 op=0 msgID=1 result=0 authDN="cn=Heartbeat Admin,cn=Root DNs,cn=config" etime=3 
    [15/Mar/2018:16:44:49 -0600] UNBIND REQ conn=2 op=2 msgID=3 
    [15/Mar/2018:16:44:49 -0600] DISCONNECT conn=2 reason="Client Unbind"
Note

You can also perform a ldapsearch against cn=monitor for a complete status of the DS/OpenDJ server as detailed in How do I use cn=monitor entry in DS/OpenDJ (All versions) for monitoring? or you could use the Connections.newHeartBeatConnectionFactory() method as detailed in Developer's Guide › Health Check Connections if your load balancer can use a Java based application.

See Also

How do I use cn=monitor entry in DS/OpenDJ (All versions) for monitoring?

How do I check if a backend is online in DS/OpenDJ (All versions)?

How do I verify that a DS/OpenDJ (All versions) server is responding to LDAP requests without providing a password?

How do I use the Access log to troubleshoot DS/OpenDJ (All versions)?

Performance tuning and monitoring ForgeRock products

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I check if a backend is online in DS/OpenDJ (All versions)?

The purpose of this article is to provide information on checking if a backend is online in DS/OpenDJ. This can be a useful check as part of your DS/OpenDJ monitoring.

Checking if a backend is online (DS 6 and later)

You can perform a ldapsearch against the backend to check it is online. This example performs a check against the userRoot backend:

$ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "ds-cfg-backend-id=userRoot,cn=backends,cn=monitor" --searchScope base "(objectClass=*)"

Online backend

If the backend is online, you would get a response such as the following:

dn: ds-cfg-backend-id=userRoot,cn=backends,cn=monitor
objectClass: top
objectClass: ds-monitor
objectClass: ds-monitor-backend
objectClass: ds-monitor-backend-pluggable
objectClass: ds-monitor-backend-db
ds-mon-backend-is-private: false
ds-mon-backend-entry-count: 1511
ds-mon-backend-writability-mode: enabled
ds-mon-backend-degraded-index-count: 0
ds-mon-backend-ttl-is-running: false
ds-mon-backend-ttl-last-run-time: 20190131162956.795Z
ds-mon-backend-ttl-thread-count: 0
ds-mon-backend-ttl-queue-size: 0
ds-mon-backend-ttl-entries-deleted: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000}
ds-mon-backend-filter-use-start-time: 19700101000000Z
ds-mon-backend-filter-use-indexed: 0
ds-mon-backend-filter-use-unindexed: 0
ds-mon-db-version: 7.5.11
ds-mon-db-cache-evict-internal-nodes-count: 0
ds-mon-db-cache-evict-leaf-nodes-count: 0
ds-mon-db-cache-total-tries-internal-nodes: 1709
ds-mon-db-cache-total-tries-leaf-nodes: 1382
ds-mon-db-cache-misses-internal-nodes: 31
ds-mon-db-cache-misses-leaf-nodes: 438
ds-mon-db-cache-size-active: 5042101
ds-mon-db-log-size-active: 7550013
ds-mon-db-log-cleaner-file-deletion-count: 0
ds-mon-db-log-utilization-min: 34
ds-mon-db-log-utilization-max: 34
ds-mon-db-log-size-total: 7550013
ds-mon-db-log-files-open: 1
ds-mon-db-log-files-opened: 3
ds-mon-db-checkpoint-count: 0
ds-cfg-backend-id: userRoot

Offline backend

If the backend is offline, you would get a response such as the following:

# The LDAP search request failed: 32 (No Such Entry)
# Additional Information:  Entry ds-cfg-backend-id=userRoot,cn=backends,cn=monitor does not exist in the "monitor" backend
# Matched DN:  cn=backends,cn=monitor

Checking if a backend is online (Pre-DS 6)

You can perform a ldapsearch against the backend to check it is online. This example performs a check against the userRoot backend:

$ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=userRoot Backend,cn=monitor" --searchScope sub "(objectClass=*)"

Online backend

If the backend is online, you would get a response such as the following:

dn: cn=userRoot Backend,cn=monitor
objectClass: top
objectClass: ds-monitor-entry
objectClass: ds-backend-monitor-entry
ds-backend-is-private: FALSE
ds-backend-writability-mode: enabled
cn: userRoot Backend
ds-backend-entry-count: 200002
ds-base-dn-entry-count: 200002 dc=forgerock,dc=com
ds-backend-id: userRoot
ds-backend-base-dn: dc=forgerock,dc=com

Offline backend

If the backend is offline, you would get a response such as the following:

SEARCH operation failed
Result Code:  32 (No Such Entry)
Additional Information:  Entry cn=userRoot Backend,cn=monitor does not exist in the memory-based backend
Matched DN:  cn=monitor

You would also see a corresponding error in the DS/OpenDJ log, for example:

[24/Mar/2015:14:41:59 -0600] category=CONFIG severity=SEVERE_ERROR msgID=3407988 msg=An error occurred while trying to initialize a backend loaded from class org.opends.server.backends.jeb.BackendImpl with the information in configuration entry ds-cfg-backend-id=userRoot,cn=Backends,cn=config:  The database environment could not be opened: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /Users/jdoe/Software/db/userRoot com.sleepycat.je.log.ChecksumException: Incomplete log entry header, size=0 lsn=0x10/0x67b292 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x10/0x67b292 parent IN=3 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x1a/0x9136a5 lastLoggedVersion=0x1f/0xc74a1 parent.getDirty()=false state=0 (BackendImpl.java:1754 BackendImpl.java:319 BackendConfigManager.java:1298 BackendConfigManager.java:279 DirectoryServer.java:2210 DirectoryServer.java:1397 DirectoryServer.java:9651).  This backend will be disabled

See Also

How do I perform a heartbeat check against DS/OpenDJ (All versions)?

FAQ: Monitoring DS/OpenDJ

How do I use cn=monitor entry in DS/OpenDJ (All versions) for monitoring?

How do I verify that a DS/OpenDJ (All versions) server is responding to LDAP requests without providing a password?

Administration Guide › Monitoring, Logging, and Alerts

Related Training

ForgeRock Directory Services Core Concepts

Related Issue Tracker IDs

N/A


OpenIDM/IDM


FAQ: IDM/OpenIDM performance and tuning

The purpose of this FAQ is to provide answers to commonly asked questions regarding performance and tuning for IDM/OpenIDM.

Frequently asked questions

Q. Are there any recommendations for sizing IDM/OpenIDM servers?

A. No, the performance of IDM/OpenIDM depends entirely on your specific environment and the exact nature of scripted customizations. To establish appropriate sizing, you will need to perform adequate benchmark testing. 

See Release Notes › Fulfilling Memory Requirements for further information.

Note

Sizing and/or tuning recommendations are outside the scope of ForgeRock support; if you want more tailored advice, consider engaging Deployment Support Services.

Q. Is there any best practice advice for benchmark testing?

A. ForgeRock recommends the following:

  • Maintain a staging environment that matches production where you can simulate the load you will have in production. This allows you to resolve any issues you identify in a non-production environment.
  • Establish a simple benchmark prior to adding external resources; you can then incrementally add resources and processes to establish what impact each one has.

Q. What is the recommended Java Virtual Machines (JVM) heap size for IDM/OpenIDM?

A. There are no definitive rules for the size of JVM heap size required as it will vary across individual environments and applications, but you should refer to Best practice for JVM Tuning for best practice advice. Additionally, ensure you configure JVM garbage collection appropriately as GC times can increase with large heap sizes causing significant impacts to application performance and CPU utilization. See How do I change the JVM heap size for IDM/OpenIDM (All versions)? for further information.

You can monitor JVM memory usage using one of the health endpoints as detailed in Integrator's Guide › Memory Health Check.

Note

For a 32-bit JVM or a 32-bit operating system, the limit for the process size is 4GB, that is, 2^32; this cannot be exceeded regardless of the amount of heap space allocated.

Q. How can I troubleshoot performance issues?

A. If you have no benchmark tests for comparison and encounter performance issues, the recommendation is to reduce your system to the bare minimum and then incrementally add back resources and processes in order to identify which one is causing the bottleneck.

The following tips should also help:

Q. How can I improve reconciliation performance?

A. Firstly, you should identify reconciliation performance issues per the advice in How do I identify reconciliation performance issues in IDM/OpenIDM (All versions)?; this article also offers tuning advice. In addition to this, you can:

Q. Are there any recommendations for sizing the database needed for the IDM/OpenIDM repository?

A. Database sizing for the IDM/OpenIDM repository depends on various factors such as the number of managed objects (users, groups, roles etc), relationships, links, audit logs, configurations, workflow, reconciliations etc. Since all these factors vary from one deployment to another, it is not possible to give specific recommendations. However, the following guidelines should help when estimating the size of database required: 

Managed objects

You can calculate the size of your IDM/OpenIDM deployment by first calculating the size of your database for sample data (such as 20 users) and then multiplying that by the expected number of users. The following tables typically grow as more managed objects are added: 

  • managedobjects - one entry per managed object. 
  • managedobjectproperties - N entries per managed object, where N is the number of indexed / searchable properties: Integrator's Guide › Creating and Modifying Managed Object Types.
  • links - the total number of links is less than or equal to the number of managed objects multiplied by the number of unique mappings. For example, if you have 20 managed users and 3 mappings (systemLdapAccounts_managedUser, managedUser_systemLdapAccounts and systemADAccount_managedUser), the total number of links would be less than or equal to 40 (20 * 2). The systemLdapAccounts_managedUser and managedUser_systemLdapAccounts mappings are bidirectional syncs and may use the same links: Integrator's Guide › Mapping Source Objects to Target Objects.
  • relationships, relationshipproperties - relationships are especially sensitive to growth when searchablebydefault is set to true (the various repo.jdbc.json files provided in the /path/to/idm/db directory have different defaults so you may end up generating more of this data that you need or expect). Additionally, roles utilize the relationships table as of OpenIDM 4. 
  • Activiti - these tables can grow over time as running workflows result in persisting data in these tables. 

Most of the other tables such as configobjects, internaluser etc are static once you have a working IDM/OpenIDM configuration and should not have much impact on database sizing.

Audit logs

Audit logs can have a huge impact on database sizing and disk space. By default, IDM/OpenIDM stores audit logs in both CSV files and in the database. The size of audit logs depends on IDM/OpenIDM usage. There are different types of audit logs and their corresponding events: Integrator's Guide › Audit Event Topics. The following tables in particular grow with each reconciliation:

  • recon/audit - this table reports the status for each user and grows by one entry for each source record. Additionally, a further entry is created for each target record that is not linked or correlated (CONFIRMED, FOUND). The size of a single reconciliation report depends on the data size of source and target; the overall data requirement depends on how many reports you keep. You can reduce the size of the recon/audit table for actions you're not interested in by setting the action to NOREPORT.
  • recon/activity - this table reports all activity and grows by one entry for each modifying action regardless of the source of the action, for example, reconciliation, synchronization etc. The overall size depends on how many changes are processed and how long these records are kept.

The following best practices should be heeded:

Q. How can I manage the QueuedThreadPool in Jetty?

A. You can change the Jetty thread pool settings by updating the config.properties file (located in the /path/to/idm/conf directory) or by setting the OPENIDM_OPTS environment variable when you start IDM/OpenIDM. See Integrator's Guide › Adjusting Jetty Thread Settings for further information.

See Also

How do I enable Garbage Collector (GC) Logging for IDM/OpenIDM (All versions)?

How do I collect data for troubleshooting high CPU utilization on IDM/OpenIDM (All versions) servers?

Performance tuning and monitoring ForgeRock products

Integrator's Guide › Monitoring Basic Server Health

Integrator's Guide › Optimizing Reconciliation Performance

Related Training

N/A


How do I change the JVM heap size for IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on changing the JVM heap size for IDM/OpenIDM.

Changing the JVM heap size

Changing the JVM heap size can improve performance and reduce the time it takes to run reconciliations. You should try different heap sizes to see what impact it has to determine the best heap size for your setup.

You can set the JVM heap size via the OPENIDM_OPTS environment variable. If OPENIDM_OPTS is undefined, the JVM maximum heap size defaults to 1GB. For example, to set the minimum and maximum heap sizes to 2GB, you would enter the following prior to starting IDM/OpenIDM:

On Unix® and Linux® systems:

$ cd /path/to/idm/
$ export OPENIDM_OPTS="-Xms2048m -Xmx2048m"
$ ./startup.sh

On Microsoft® Windows® systems:

C:\> cd \path\to\idm
C:\path\to\idm> set OPENIDM_OPTS=-Xms2048m -Xmx2048m
C:\path\to\idm> startup.bat
Note

You can also edit the startup.sh or startup.bat files to update the default OPENIDM_OPTS values.

Note

It is recommended that you set the minimum and maximum heap sizes to the same value for best performance. Otherwise, the JVM runs a full Garbage Collector (GC) cycle when increasing its heap, during which time it can pause ongoing operations for up to a few seconds. Generally, a smaller heap will increase the frequency at which the JVM GC executes but reduce the duration; similarly, a larger heap will reduce the frequency and increase the duration. When tuning the JVM heap, the goal is to strike a balance between frequency and duration so as to reduce the impact of the GC on the application.

See Also

Best practice for JVM Tuning

How do I enable Garbage Collector (GC) Logging for IDM/OpenIDM (All versions)?

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I identify reconciliation performance issues in IDM/OpenIDM (All versions)?

The purpose of this article is to provide guidance on isolating and diagnosing IDM/OpenIDM reconciliation performance issues. This article also includes tips on tuning reconciliation performance based on your findings.

Overview

Isolating the cause(s) of performance bottlenecks within a large IDM/OpenIDM topology comes down to a process of elimination. Identifying bottlenecks requires the system to be broken down to its individual components; each component must then be tested and tuned to achieve the necessary throughput.

For example, given the following basic topology there are multiple components which might limit the throughput of the system:

Although far from exhaustive, the following are some of the possible bottlenecks which might be encountered within the above topology:

  • Source LDAP Read ops/s
  • Target LDAP Read/Write ops/s
  • PostgreSQL Read/Write ops/s
  • JVM Heap limitations
  • Host Disk I/O limitations
  • Host CPU limitations
  • Undersized / incorrectly configured virtualization
  • IDM/OpenIDM configuration issues
  • IDM/OpenIDM triggers/custom code
  • IDM/OpenIDM product defect
Note

One configuration change that can help with reconciliation performance is to increase the number of connector instances in the connection pool as detailed in How do I configure pooled connections for a connector in IDM/OpenIDM (All versions)?

Reconciliation performance

Measuring the performance of individual components within the topology can be done outside the scope of IDM/OpenIDM by using external benchmarking tools and processes or via IDM/OpenIDM itself. Benchmarking components via IDM/OpenIDM consists of using targeted operations and configuration changes to isolate a single component within the system and measure its performance as it relates to reconciliation.

You should consider the following components:

External source system

During reconciliation, IDM/OpenIDM queries the source system in order to obtain the complete list of source object IDs to be reconciled. It may also perform multiple queries to retrieve the complete source object during the reconciliation process or rely on a cache of the source objects returned by the configured sourceQuery.

To begin to understand the performance of the source system, the following can be performed:

  • LDAP searchrate command to measure raw read ops capability. This command is provided in the OpenDJ LDAP Toolkit.
  • Measure execution time via targeted queries against the source system using IDM/OpenIDM REST APIs:
    • Execute the generic ‘query-all-ids’ query against the source system.
    • Execute any ‘sourceQuery’ queries which have been configured.
    • Execute individual source object queries via Query FIlter, specifying the individual source object attributes use within the configured ‘sourceQuery’ queries.

Based on the results of the above actions, tune the source system to resolve performance bottlenecks.  Examples of things which might reduce read performance on a source LDAP system are:

  • Lack of necessary indexes, resulting in un-indexed (full DB scans) searches.
  • Insufficient JVM Heap.
  • Insufficient CPU resources to handle the load.
  • Insufficient Disk I/O performance (non-SSD drives) to handle the load.

External target system

During reconciliation, IDM/OpenIDM queries the target system in order to obtain the complete list of target object IDs to be reconciled. It may also perform multiple queries to retrieve the complete target object during the reconciliation process or rely on a cache of the target objects returned by the configured ‘targetQuery’. Target objects calculated based on the configured mappings are then written to the target system.

To begin to understand the performance of the target system, the following can be performed:

  • LDAP searchrate and modrate commands to measure raw read/write ops capability. These commands are provided in the OpenDJ LDAP Toolkit.
  • Measure execution time via targeted queries against the target system using IDM/OpenIDM REST APIs:
    • Execute the generic ‘query-all-ids’ query against the target system.
    • Execute any ‘targetQuery’ queries which have been configured.
    • Execute any ‘correlationQuery’ queries which have been configured.
    • Execute individual target object queries via Query FIlter, specifying the individual target object attributes use within both the configured ‘targetQuery’ and ‘correlationQuery’ queries.

Based on the results of the above actions, tune the target system to resolve performance bottlenecks. Examples of things which might reduce write performance on a target LDAP system are:

  • Excessive indexes, resulting in increased overhead when writing entries. Specifically take note that substring indexes are extremely costly and may greatly reduce performance.
  • Insufficient JVM Heap.
  • Insufficient CPU resources to handle the load.
  • Insufficient Disk I/O performance (non-SSD drives) to handle the load.

JDBC repository

In order to isolate the JDBC repository from the performance of the reconciliation engine or the target system, the following can be performed:

Read/Write Test

Mapping: LDAP Source -> Managed Object

Situation Action
ABSENT CREATE (Default)

Read Test

Mapping: Managed Object -> LDAP Target

Situation Action
ALL ASYNC (Read Only)

Reconciliation engine

Note

Measuring the performance of the Reconciliation Engine should be done only after having resolved any performance issues observed during testing of the source  and target systems.

The IDM/OpenIDM reconciliation engine is responsible for the bidirectional synchronization of objects between different data stores. In the case of our topology above, this is the synchronization of the objects between the source and target LDAP systems.

During reconciliation IDM/OpenIDM determines the state of the source and target objects (situation assessment), identifies the necessary actions to be performed and performs the actions against the source and target systems. Included within each phase of the reconciliation process are various JavaScript triggers which may or may not impact the performance of the reconciliation process depending on their efficiency.

In order to isolate the reconciliation engine from the performance of the IDM/OpenIDM repository or the target system, the following can be performed:

Read-Only Test

Mapping: LDAP Source -> LDAP Target

Situation Action
ALL ASYNC (Read Only)
Note

The following should be performed with a source system which has been populated with production-like data and an EMPTY target system.

  1. Set the actions associated with all of the situation policies to ‘Read-Only’ via the IDM/OpenIDM Admin UI. This ensures that during the reconciliation process no actions are performed and that nothing is written to the IDM/OpenIDM repository, or the target system.
  2. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging.

Read-Write Test

Mapping: LDAP Source -> LDAP Target

Situation Action
ABSENT CREATE (Default)
  1. Set the action associated with the ABSENT situation policy to CREATE. Set all other situation policies to ‘Read Only’. This ensures that during the reconciliation process objects are created and links are established between the source and target objects. No other actions are performed.
  2. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging.

Link Test

Mapping: LDAP Source -> LDAP Target

Situation Action
FOUND LINK (Custom)
  1. Ensure you have defined your desired ‘correlationQuery’ within your mapping.
  2. Delete the contents of the ‘links’ table within the IDM/OpenIDM repository.
  3. Set the action associated with the FOUND situation policy to LINK. Set all other situation policies to ‘Read Only’. This ensures that during the reconciliation process the ‘correlationQuery’ will be executed to correlate the existing target objects (created during the read-write test) to source objects.  For each correlated object a new link will be written to the IDM repository.
  4. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging.

Source Phase

Mapping: LDAP Source -> LDAP Target

Situation Action
CONFIRMED Update (Default)
  1. Set the ‘runTargetPhase’ property within the mapping to ‘false’. This ensures that ONLY the source phase of the reconciliation process is executed.
  2. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging.

Target Phase

Mapping: LDAP Source -> LDAP Target

Situation Action
CONFIRMED Update (Default)
  1. Delete the contents of the ‘links’ table within the IDM/OpenIDM repository.
  2. Set the ‘sourceQuery’ within the mapping to point to a single specific source object. The goal is to reduce the scope of the source objects processed by the reconciliation engine down to a single object.
  3. Set the ‘runTargetPhase’ property within the mapping to ‘true’. Ensure that either you do NOT define a ‘targetQuery’ or that your ‘targetQuery’ is set to your desired value within production.  Do not set the ‘targetQuery’ to point to a single object as is done with the ‘sourceQuery’. This ensures that all of the target objects will be processed by the target phase of the reconciliation.
  4. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging.

See Also

Integrator's Guide › Synchronizing Data Between Resources › Optimizing Reconciliation Performance

How do I collect data for troubleshooting high CPU utilization on IDM/OpenIDM (All versions) servers?

How do I monitor IDM 5 and OpenIDM 3.x, 4.x using SmartEvent?

How do I collect JVM data for troubleshooting IDM/OpenIDM (All versions)?

How do I enable Garbage Collector (GC) Logging for IDM/OpenIDM (All versions)?

How do I change the JVM heap size for IDM/OpenIDM (All versions)?

Best practice for JVM Tuning

Administration Guide › Tuning Servers For Performance

How do I troubleshoot issues with my indexes in DS/OpenDJ (All versions)?

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I configure the connection pool for the managed repository in IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on configuring the JDBC connection pool for the managed repository in IDM/OpenIDM. These configurations apply when you are using a JDBC database.

Overview

IDM 5 and later

IDM 5 and later uses the Hikari connection pool by default. See Integrator's Guide › Managing the Repository › Understanding the JDBC Connection Configuration File and Hikari Project Page for further information on configuring the connection pool settings.

Pre-IDM 5

Pre-IDM 5 uses the BoneCP connection pool to manage connections to the managed repository. The 0.7.1 version of the BoneCP connection pool is bundled with OpenIDM. 

You can configure the connection pool settings by changing the values of the default settings in this connection section as required and by adding additional BoneCP properties. All the properties you can configure and/or add are described in BoneCP 0.7.1 - BoneCPConfig.java.

Partition properties are particularly useful; these can be used to group connections in the connection pool to minimize connection locks and improve performance. For example, you can add the following 3 partition properties:

  • partitionCount - this property defines the number of connection groups. By default, this property is set to 1 but you can increase it. The BoneCP configuration documents suggest a setting of 2-4, depending on your application. Other people suggest setting this to the number of cores, since this is a good indicator of how many threads can concurrently access the connection pool. In some cases where the threads are very short-lived, a higher partitionCount may yield better performance. You should try different settings to see what setting is most appropriate to your environment.
  • minConnectionsPerPartition - this property defines the number of connections immediately allocated per partition when the connection pool is created.
  • maxConnectionsPerPartition - this property defines the maximum number of connections per partition. The total number of connections available is therefore maxConnectionsPerPartition x partitionCount, although the number of connections initially created is determined by the minConnectionsPerPartition setting.

See the following sections for examples according to which release of OpenIDM you are using; the connection pool settings are managed in different files.

Configuring the connection pool in OpenIDM 4.x

The connection pool is always enabled in OpenIDM 4.x.

Example

The following example shows the datasource.jdbc-default.json file (located in the /path/to/idm/conf directory) for a MySQL™ database with the additional partition properties added:

{
    "driverClass" : "com.mysql.jdbc.Driver",
    "jdbcUrl" : "jdbc:mysql://localhost:3306/openidm?allowMultiQueries=true&characterEncoding=utf8",
    "databaseName" : "openidm",
    "username" : "openidm",
    "password" : "openidm",
    "connectionTimeout" : 30000,
    "connectionPool" : {
        "type" : "bonecp",
        "partitionCount" : 3,
        "minConnectionsPerPartition" : 4,
        "maxConnectionsPerPartition" : 7
    }
}

These partition settings would give you a total of 21 connections with 12 being created initially.

Configuring the connection pool in OpenIDM 3.x

You can check the connection pool is enabled by looking for (or adding if it's missing), the following line in the connection section of the repo.jdbc.json file (located in the /path/to/idm/conf directory):

"enableConnectionPool" : true,​

Example

The following example shows the repo.jdbc.json file for an Oracle® database with the additional partition properties added:

{
    "connection" : {
        "dbType" : "ORACLE",
        "jndiName" : "",
        "driverClass" : "oracle.jdbc.OracleDriver",
        "jdbcUrl" : "jdbc:oracle:thin:@//HOSTNAME:PORT/DEFAULTCATALOG",
        "username" : "openidm",
        "password" : "password",
        "defaultCatalog" : "openidm",
        "maxBatchSize" : 100,
        "maxTxRetry" : 5,
        "enableConnectionPool" : true,
        "connectionTimeoutInMs" : 30000,
        "partitionCount" : 3,
        "minConnectionsPerPartition" : 4,
        "maxConnectionsPerPartition" : 7
    },

These partition settings would give you a total of 21 connections with 12 being created initially.

See Also

Hikari Project Page

BoneCP Configuration

BoneCP 0.7.1 - BoneCPConfig.java

Related Training

N/A

Related Issue Tracker IDs

OPENIDM-2995 (JDBCRepoService doesn't close the BoneCP database connection pool when it is deactivated.)

OPENIDM-3613 (BoneCP: unexplained connections getting created)


How do I configure pooled connections for a connector in IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on configuring pooled connections for a connector in IDM/OpenIDM. This information applies to all poolable connectors (such as the LDAP connector). Poolable connectors create another instance of the connection if one is not available when it needs to do an operation or if it needs to do multiple operations in parallel. Increasing the number of connector instances in the connection pool can improve reconciliation performance.

Configuring pooled connections

You can configure the settings used for pooled connections in your provisioner configuration file (for example, provisioner.openicf-ldap.json), which is located in the /path/to/idm/conf directory.

This file has a poolConfigOption section, with default settings as follows:

 "poolConfigOption" : {
        "maxObjects" : 10,
        "maxIdle" : 10,
        "maxWait" : 150000,
        "minEvictableIdleTimeMillis" : 120000,
        "minIdle" : 1
    },

You can amend these settings as needed to configure how the pooled connections are used, where:

Setting Description
maxObjects

The maximum number of instances permitted in the pool (total active and idle connections).

maxIdle The maximum number of idle instances allowed in the pool.
maxWait The maximum time (in milliseconds) that the pool waits for an instance to become available before timing out.
minEvictableIdleTimeMillis The minimum time (in milliseconds) that an instance must be idle for before it is removed. The default 120000ms equals a 2 minutes idle timeout period.
minIdle The minimum number of idle instances allowed in the pool. Once the idle timeout is reached, idle instances will be removed from the pool to bring the number of instances available down to the minIdle value. For example, if you set minIdle to 0, the pool would be emptied of idle instances when the idle timeout is reached.

You should bear the following in mind when adjusting these settings:

  • You should ensure maxObjects and maxIdle are always set to the same value to allow pooling to work efficiently and prevent excessive CPU usage. Failure to use the same value will cause excessive churn within the connector instance pool and cause an excessive number of new connections to be established.
  • You can increase maxObjects and maxIdle to increase the number of connector instances available. You should experiment to determine the best settings for your setup; a good starting point is to double these values to 20.
  • You should also increase the number of recontask threads as the number of connector instances increase. You can do this by increasing the value of the taskThreads property in the sync.json file to a value other than the default of 10. See Integrator's Guide › Synchronizing Data Between Resources › Parallel Reconciliation Threads for further information.

See Also

How do I identify reconciliation performance issues in IDM/OpenIDM (All versions)?

How do I configure the connection pool for the managed repository in IDM/OpenIDM (All versions)?

Integrator's Guide › Connecting to External Resources › Configuring Connectors

Integrator's Guide › Connecting to External Resources › Setting the Pool Configuration

Integrator's Guide › Synchronizing Data Between Resources › Parallel Reconciliation Threads

Connector Reference › Connection Pooling Configuration

Related Training

ForgeRock Identity Management Core Concepts (IDM-400)

Related Issue Tracker IDs

N/A


How do I enable Garbage Collector (GC) Logging for IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on enabling GC Logging for IDM/OpenIDM. It assumes that you already have a working IDM/OpenIDM server that is installed.

Enabling GC Logging

You should enable GC logging via the OPENIDM_OPTS environment variable. You can either add them prior to starting IDM/OpenIDM as demonstrated below or you can edit the startup.sh or startup.bat files to update the default OPENIDM_OPTS values.

Note

You should ensure there is enough disk space for the GC logs; if not, you should enable GC log rotation as well by including the following options when you enable GC logging: -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=n, -XX:GCLogFileSize=n.

On Unix® and Linux® systems:

Enter the following prior to starting IDM/OpenIDM to enable GC logging:

$ cd /path/to/idm/
$ export OPENIDM_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]"
$ ./startup.sh

replacing [filename] with the path to the file that you would like to create to store the log file. The Xloggc property is optional; if omitted, the GC data is written to the openidm.log file (located in the /path/to/idm/logs directory). This can be useful when trying to correlate GC events with events occurring within the server instance.

Once IDM/OpenIDM has started, there should be a GC log file located in the directory specified in the -Xloggc: option or GC data included in the openidm.log file. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

On Microsoft® Windows® systems:

Enter the following prior to starting IDM/OpenIDM to enable GC logging:

C:\> cd \path\to\idm
C:\path\to\idm> set OPENIDM_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]
C:\path\to\idm> startup.bat

replacing [filename] with the path to the file that you would like to create to store the log file. The Xloggc property is optional; if omitted, the GC data is written to the openidm.log file (located in the \path\to\idm\logs directory). This can be useful when trying to correlate GC events with events occurring within the server instance.

Once IDM/OpenIDM has started, there should be a GC log file located in the directory specified in the -Xloggc: option or GC data included in the openidm.log file. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

See Also

Best practice for JVM Tuning

How do I change the JVM heap size for IDM/OpenIDM (All versions)?

How do I collect JVM data for troubleshooting IDM/OpenIDM (All versions)?

FAQ: IDM/OpenIDM performance and tuning

Related Training

N/A

Related Issue Tracker IDs

N/A


Monitoring


How do I perform an anonymous health check on the IDM/OpenIDM server (All versions)?

The purpose of this article is to provide information on performing an anonymous health check on the IDM/OpenIDM server. Anonymous access may be required if you have an application or load balancer that needs to check the availability of the IDM/OpenIDM instance but you don't want to provide any user credentials.

Performing anonymous health checks

Basic health checks can be performed against the info/ping endpoint to check that IDM/OpenIDM is available.

To perform an anonymous health check, you must specifically include the following headers in your rest call to identify yourself as anonymous, else the call will fail:

X-OpenIDM-Password: anonymous 
X-OpenIDM-Username: anonymous

The anonymous user (internal/role/openidm-reg role) has access to the info/ping endpoint by default.

Example

To perform an anonymous health check, you would use a REST call such as the following:

$ curl -H "X-OpenIDM-Username: anonymous" -H "X-OpenIDM-Password: anonymous" -X GET "http://idm.example.com:8080/openidm/info/ping"

Example response:

{
  "_id" : "",
  "state" : "ACTIVE_READY",
  "shortDesc" : "OpenIDM ready"
}

See Also

How does IDM/OpenIDM (All versions) use anonymous access?

Integrator's Guide › Basic Health Checks

Related Training

N/A

Related Issue Tracker IDs

OPENIDM-5118 (Need ability to perform unauthenticated health checks )


How do I monitor IDM 5 and OpenIDM 3.x, 4.x using SmartEvent?

The purpose of this article is to provide guidance on monitoring IDM/OpenIDM using SmartEvent. SmartEvent is replaced in IDM 5.5 with DropWizard Metrics.

Enabling SmartEvent

IDM/OpenIDM includes an event monitoring facility called SmartEvent. You can enable SmartEvent by using the openidm.smartevent.enabled system property. If you want to view a summary or statistics in the openidm log, you can enable openidm.smartevent.summarylogging.

To get started with SmartEvent, enable it via JAVA_OPTS during startup as follows:

$ env JAVA_OPTS="-Dopenidm.smartevent.enabled=true -Dopenidm.smartevent.summarylogging=true" ./startup.sh

The SmartEvent summary information is then output to the openidm.log.0 file, for example:

Jun 16, 2016 14:22:51 PM org.forgerock.openidm.smartevent.core.StatisticsHandler$1 run
INFO: Summary: {openidm/internal/repo/orientdb/raw/query/query-all-ids=Invocations: 1 total time: 18.448 ms mean: 18.448 ms, openidm/internal/repo/orientdb/raw/query/query-cluster-events=Invocations: 119 total time: 31.958 ms mean: 0.269 ms, openidm/internal/repo/orientdb/raw/query/query-cluster-failed-instances=Invocations: 119 total time: 54.032 ms mean: 0.454 ms, openidm/internal/router/repo/scheduler/read=Invocations: 26 total time: 98.177 ms mean: 3.776 ms}

IDM/OpenIDM SmartEvent MBeans are also available locally by attaching a monitoring tool such as JConsole or VisualVM. These tools allow you to attach to a local process. If you wish to attach remotely, see the following section.

Accessing SmartEvent remotely using JMX

To access SmartEvent remotely using JMX, you will need to specify the following JAVA_OPTS when starting IDM/OpenIDM:

​$ env JAVA_OPTS="-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=9010 
-Dcom.sun.management.jmxremote.local.only=true 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false 
-Dopenidm.smartevent.enabled=true 
-Dopenidm.smartevent.summarylogging=false" ./startup.sh 

Attach a monitoring tool to a remote process in the form <hostname>:<port>.

See Also

How do I monitor IDM 5.5 using DropWizard Metrics?

How do I configure Jetty Request logging for IDM/OpenIDM (All versions)?

Troubleshooting IDM/OpenIDM

Using JConsole

Getting Started with VisualVM

Related Issue Tracker IDs

OPENIDM-571 (Initial monitoring support)

OPENIDM-2312 (SmartEvent framework maintains a unbounded event name cache which consumes the entire heap)


OpenIG/IG


How do I change the JVM heap size for IG/OpenIG (All versions)?

The purpose of this article is to provide information on changing the JVM heap size for IG/OpenIG. This article provides details for changing the JVM heap size on Linux®, Unix® and Windows® systems.

Changing the JVM heap size

Changing the JVM heap size can improve performance and reduce the time it takes to process HTTP requests and responses. You should try different heap sizes to see what impact it has to determine the best heap size for your setup. The information given here is specific to the Apache Tomcat™ web container; you should make similar changes in the configuration file specific to your web container if you use a different one.

Note

It is recommended that you set the minimum and maximum heap sizes to the same value for best performance. Otherwise, the JVM runs a full Garbage Collector (GC) cycle when increasing its heap, during which time it can pause ongoing operations for up to a few seconds. Generally, a smaller heap will increase the frequency at which the JVM GC executes but reduce the duration; similarly, a larger heap will reduce the frequency and increase the duration. When tuning the JVM heap, the goal is to strike a balance between frequency and duration so as to reduce the impact of the GC on the application.

On Unix® and Linux® systems:

You can set the JVM heap size by specifying CATALINA_OPTS settings in the setenv.sh file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.sh file (also typically located in the /tomcat/bin/ directory).

For example, to set the minimum and maximum heap sizes to 2GB, you would add the following line to the setenv.sh file and restart the web container:

export CATALINA_OPTS="$CATALINA_OPTS -Xms2048m -Xmx2048m"

On Microsoft® Windows® systems:

Providing you haven't installed Tomcat as a service, you can set the JVM heap size by specifying CATALINA_OPTS settings in the setenv.bat file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.bat file (also typically located in the /tomcat/bin/ directory).

For example, to set the minimum and maximum heap sizes to 2GB, you would add the following line to the setenv.bat file and restart the web container:

set CATALINA_OPTS=-Xms2048m -Xmx2048m

If you have installed Tomcat as a service on Windows, then you must use the GUI application to configure Tomcat services. This process refers to Tomcat 7, but will work for other versions by changing tomcat7w.exe in the command to match your version.

  1. Stop the Tomcat service.
  2. Navigate to \path\to\tomcat\bin\ from the command prompt:
  3. Enter the following command to display Tomcat Properties:
    tomcat7w.exe //ES//serviceName
    Where serviceName is the name of your Tomcat service.
  4. Navigate to the Java tab and complete the memory pool fields as follows:
    Initial memory pool: 2048
    Maximum memory pool: 2048
  5. Restart the Tomcat service.

Alternatively, Windows system administrators may prefer to configure these options in the registry so that they may be configured via group policy. The initial memory pool and maximum memory pool values can be configured in the JvmMS and JvmMX properties under the following registry key for Tomcat 7:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\Tomcat7\Parameters\Java

See Also

Best practice for JVM Tuning

How do I enable Garbage Collector (GC) Logging for IG/OpenIG (All versions)?

How do I collect JVM data for troubleshooting IG/OpenIG (All versions)?

How do I use the msnapshots script to capture information for troubleshooting IG/OpenIG (All versions)?

Setup and Maintenance Guide › Tuning Java Virtual Machine Settings

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I enable Garbage Collector (GC) Logging for IG/OpenIG (All versions)?

The purpose of this article is to provide information on enabling GC Logging for IG/OpenIG. It assumes that you already have a working IG/OpenIG server installed.

Enabling GC Logging

The information given here is specific to the Apache Tomcat™ web container; you should make similar changes in the configuration file specific to your web container if you use a different one.

Note

You should ensure there is enough disk space for the GC logs; if not, you should enable GC log rotation as well by including the following options when you enable GC logging: -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=n, -XX:GCLogFileSize=n

On Unix® and Linux® systems:

​You should enable GC logging by specifying CATALINA_OPTS settings in the setenv.sh file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.sh file (also typically located in the /tomcat/bin/ directory).

To enable GC logging:

  1. Add the following line to the setenv.sh file:
    export CATALINA_OPTS="$CATALINA_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]"
    replacing [filename] with the path to the file that you would like to create to store the log file.
  2. Restart the web container.

Once the web container has successfully restarted, there should be a GC log file located in the directory specified in the -Xloggc: option. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

On Microsoft® Windows® systems:

​You should enable GC logging by specifying CATALINA_OPTS settings in the setenv.bat file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.bat file (also typically located in the /tomcat/bin/ directory).

To enable GC logging:

  1. Add the following line to the setenv.bat file:
    set CATALINA_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]
    replacing [filename] with the path to the file that you would like to create to store the log file.
  2. Restart the web container.

Once the web container has successfully restarted, there should be a GC log file located in the directory specified in the -Xloggc: option. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

See Also

Best practice for JVM Tuning

How do I change the JVM heap size for IG/OpenIG (All versions)?

How do I collect JVM data for troubleshooting IG/OpenIG (All versions)?

How do I use the msnapshots script to capture information for troubleshooting IG/OpenIG (All versions)?

Setup and Maintenance Guide › Maintaining an Instance › Tuning Java Virtual Machine Settings

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I migrate OpenIG 4 scripts from using blocking APIs to non-blocking APIs?

The purpose of this article is to provide information on migrating existing OpenIG 4 scripts from using blocking APIs to non-blocking APIs. Moving to non-blocking APIs guarantees an efficient use of resources and can prevent deadlock situations. You should also migrate your scripts if you have upgraded to OpenIG 4.5 or later, or are planning to upgrade in the future; otherwise you will likely encounter a "Cannot execute script" error when trying to run an existing script.

Overview

There is a known issue where existing scriptable filters and handlers stop working after upgrading to OpenIG 4.5 or later. This issue is caused by a class version conflict in the CatalogManager class, which is used by the xml-resolver-1.2.jar and is required by the HTTPBuilder API. This issue affects all scripts that use the CatalogManager class or libraries that depend on it, for example, the Groovy http-builder library. You can resolve this issue by migrating to CHF supported APIs.

You will see an error similar to the following in your logs when you encounter this issue:

TUE MAY 02 17:02:59 CET 2017 WARNING {ScriptableFilter}/handler/config/filters/0 --- Cannot execute script
TUE MAY 02 17:02:59 CET 2017 WARNING {ScriptableFilter}/handler/config/filters/0 --- java.lang.Exception: java.lang.NoSuchMethodError: org.apache.xml.resolver.CatalogManager.setIgnoreMissingProperties(Z)V 
javax.script.ScriptException: java.lang.Exception: java.lang.NoSuchMethodError: org.apache.xml.resolver.CatalogManager.setIgnoreMissingProperties(Z)V
   at org.forgerock.openig.script.Script$GroovyImpl.run(Script.java:62)
   at org.forgerock.openig.script.Script.run(Script.java:245) 
Caused by: java.lang.Exception: java.lang.NoSuchMethodError: org.apache.xml.resolver.CatalogManager.setIgnoreMissingProperties(Z)V
   ... 42 more
Caused by: java.lang.NoSuchMethodError: org.apache.xml.resolver.CatalogManager.setIgnoreMissingProperties(Z)V
   at groovyx.net.http.ParserRegistry.<clinit>(ParserRegistry.java:111)
   at groovyx.net.http.HTTPBuilder.<init>(HTTPBuilder.java:194)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
   ... 41 more

It is strongly recommended that you migrate to CHF supported APIs and use documented variables even if you are not experiencing issues with your scripts. CHF non-blocking APIs guarantee an efficient use of resources and can prevent deadlock situations; they also ensure you can upgrade to OpenIG 4.5 and later without encountering this issue.

This article details the following two step migration process:

  1. Migrate http-builder APIs to CHF blocking APIs (they are simple to understand and starting using).
  2. Migrate to CHF asynchronous APIs.

Example script

The following example script to log a user out of AM/OpenAM uses the Groovy http-builder library; it fails in OpenIG 4.5 and later because there is a class version conflict on CatalogManager, which is provided by both the AM/OpenAM SAML Fedlet library and the Groovy http-builder library:

@Grab('org.codehaus.groovy.modules.http-builder:http-builder:0.7.1')
import groovyx.net.http.RESTClient

def openAMRESTClient = new RESTClient(openamUrl)

// Check if OpenAM session cookie is present
if (null != request.cookies['iPlanetDirectoryPro']) {
   String openAMCookieValue = request.cookies['iPlanetDirectoryPro'][0].value

   // Perform logout
   logger.info("iPlanetDirectoryPro cookie found, performing logout")
   def response = openAMRESTClient.post(path: 'sessions/',
                                        query: ['_action': 'logout'],
                                        headers: ['iplanetDirectoryPro': openAMCookieValue])
   def result = response.getData().get("result")
   logger.info("OpenAM logout response: " + result)
}

return next.handle(context, request)

The logic in this script is quite simple:

  1. If there is a session token in the request, get it.
  2. Call a dedicated AM/OpenAM REST endpoint to revoke the token (which logs the user out).
  3. Continue the execution of the chain (after the response has been received).

Migrating to CHF blocking APIs

This step details removing the dependency on the http-builder library and using the built-in CHF APIs instead. The migrated example script below demonstrates the basics of the CHF API: request creation, obtaining a response and reading its content as JSON (without forgetting to release resources).

Background

In each script’s execution context there is an HTTP binding (Client interface) that provides you with a way to perform HTTP requests from within your script.

From the Client API:

class Client {
   Promise<Response, NeverThrowsException> send(Request request);
   Promise<Response, NeverThrowsException> send(Context context, Request request);
}

There is only one thing that you can obviously do with this interface, that is, send an HTTP request. You do not get back the response directly, but you have the promise of a response. This is comparable to a Future on which you would attach listeners that get notified when the response has been fully received (or an error has happened).

This part of the migration uses the Promise as a Future (blocking on a .get()). In essence, this migration step is all about creating and populating a CHF Request object.

Migrated example script

The following shows the above example script after it has been migrated to use the built-in CHF APIs:

// Check if OpenAM session cookie is present
if (null != request.cookies[ 'iPlanetDirectoryPro' ]) {
   String openAMCookieValue = request.cookies[ 'iPlanetDirectoryPro' ][ 0 ].value

   logger.info("iPlanetDirectoryPro cookie found, performing logout")

   def logout = new Request()
   logout.method = "POST"
   logout.uri = "${openamUrl}/sessions"
   logout.uri.query = "_action=logout"
   logout.headers['iPlanetDirectoryPro'] = openAMCookieValue

   // Block for at most 20 seconds before using the response
   def logoutResponse = http.send(logout)
                            .get(20, SECONDS)

   def result = logoutResponse.entity.json.result
   logger.info("OpenAM logout response: " + result)

   // Don’t forget to release resources associated with the response
   logoutResponse.close()
}

return next.handle(context, request)

The logic in this script is as follows:

  1. Request initialization: the following snippet of this script creates a POST request to the AM/OpenAM sessions endpoint (using Groovy String substitution). It specifies the logout action in the query URL component and places the current request’s SSO Token (the iPlanetDirectoryPro cookie) in a header (as this is where the sessions endpoint expects it):
    def logout = new Request()
    logout.method = "POST"
    logout.uri = "${openamUrl}/sessions"
    logout.uri.query = "_action=logout"
    logout.headers['iPlanetDirectoryPro'] = openAMCookieValue
    
  2. Perform the request and wait for the response: the following snippet of this script uses the get() method. The returned promise can never throw an exception, which means you do not need to use the getOrThrow() variant of the get() method; if something goes wrong when making the request, you will receive a 502 Bad Gateway response. This example also includes a timeout to prevent deadlocks if there is no response. When the timeout is reached, a RuntimeException is returned; ideally the timeout should be configurable and IG/OpenIG provides the args configuration point in the script configuration, which can be used for this purpose:
    // Block for at most 20 seconds before using the response
    def logoutResponse = http.send(logout)
                             .get(20, SECONDS)
    
  3. Read the response's content using the CHF's native JSON support: the following snippet of this script calls logoutResponse.entity.json, which provides the JSON-parsed content of the message, with a focus on the result JSON attribute. The message is then closed to release resources:
    def result = logoutResponse.entity.json.result
    
    // Don’t forget to release resources associated with the response
    logoutResponse.close()
    
    
  4. Call the next element of the chain and continue the request processing:
    return next.handle(context, request)
    

Migrate to CHF asynchronous APIs

This step details moving away from synchronous programming (where we spend valuable CPU time waiting for things to happen) to asynchronous programming (where we register callbacks to notify us when things are ready and act when that happens rather than waiting for something; this allows our valuable CPU time to do other things). It also introduces the http.send() returned Promise object, where it’s the promise of a response that may or may not have been received yet. Put simply, we want to read the response content after it has been received, log the result and then continue the chain execution.

This step demonstrates adding an asynchronous function that calls the next element in the chain, after the response processing has been done.

Migrated example script

The following shows the example script after it has been migrated to use the CHF asynchronous APIs:

// Check if OpenAM session cookie is present
if (null != request.cookies[ 'iPlanetDirectoryPro' ]) {
   String openAMCookieValue = request.cookies[ 'iPlanetDirectoryPro' ][ 0 ].value

   logger.info("iPlanetDirectoryPro cookie found, performing logout")

   def logout = new Request()
   logout.method = "POST"
   logout.uri = "${openamUrl}/sessions"
   logout.uri.query = "_action=logout"
   logout.headers['iPlanetDirectoryPro'] = openAMCookieValue

   // Return the "promise" of a result => non-blocking
   // Processing will happen when a response from AM will be received
   return http.send(logout)
              .then { response ->
                   def result = response.entity.json.result
                   logger.info("OpenAM logout response: " + result)
                   response.close()
                   return response
              }
              .thenAsync({
                   next.handle(context, request)
              } as AsyncFunction)
}

logger.info("iPlanetDirectoryPro cookie not found, continue and ignore logout action")
return next.handle(context, request)

As you can see, most of the changes have occurred towards the end of the script, with the response being processed using callbacks. Additionally, since we no longer block, we do not need a timeout to prevent deadlocks.

The logic in this script is as follows:

  1. Process the response when it’s ready: the following snippet of this script introduces a then(Function) method to process this response. The callback Function provided in this then() method accepts the result of the invocation as a parameter, which can be processed as required; this example reuses the response processing logic from before to return the response (a result is expected from Function callbacks):
    return http.send(logout)
               .then { response ->
                    def result = response.entity.json.result
                    logger.info("OpenAM logout response: " + result)
                    response.close()
                    return response
               }
    
    
    You must include the first return in this section of the script to ensure the promise that is configured with the response processing function is returned when the logout action is triggered. If you exclude the return, for example, the first line is simply:
    http.send(logout)
    
    The script will execute, do an HTTP call, register a callback on the promise and then just continue. This means the logout action will get missed when there is an iPlanetDirectoryPro cookie, which is not the desired behavior of this script.
  2. Continue the execution of the chain after triggering the logout action: the following snippet of this script calls the next element of the chain by performing the call in a then-able fashion (chaining different then() methods together). You can use this approach because you will just see the response of the logout action when a request with an iPlanetDirectoryPro cookie enters the filter, rather than the usual response sent back from the protected application:
               .thenAsync({
                   next.handle(context, request)
               } as AsyncFunction)
    

Groovy specific points

  •  There is no return statement, because Groovy considers the last evaluated expression of a method to be the returned value.
  • The as AsyncFunction is only required for asynchronous functions (ResultHandler, ExceptionHandler and Function don’t need it). The Groovy compiler doesn’t handle all the generic types of AsyncFunction very well.

Further information on the Promise object

There are a number of then-able methods offered by the Promise API that you can try, which will make your code simpler and avoid 'callback hell'. These methods fall into the following categories:

thenOn****(****Handler) These methods are pure notification functions (on successful result, on error). You can’t return anything from within these methods, they are here as a side-effect-free type of handler. No exceptions should be thrown in theses callbacks.
then(Function), with 1, 2 or 3 Function arguments These methods are designed for when you need to return something different to the next handler in the Promise chain (a transformation). They could be parsing an input string and return a JSON, or modifying the result. Throwing exceptions is accepted here and they will be propagated to the next functions in the Promise chain.
thenAsync(AsyncFunction), with 1, 2 or 3 AsyncFunction arguments These methods are quite similar to those in then(Function). The real difference is that they have to return a promise of their result instead of the actual result. These methods are an ideal fit when you need to call another method that itself returns a promise (like returning the result of the filter’s chain).
then***(), the rest of the then-able methods (thenAlways(), thenFinally(), thenCatch(), ...) These methods are really syntactic sugar to make the promise more fluent to read.

Now you can understand why our first then() can be turned into a thenOnResult() instead, which saves the last return statement as the received parameter is returned:

return http.send(logout)
           .thenOnResult { response ->
               def result = response.entity.json.result
               logger.info("OpenAM logout response: " + result)
               response.close()
           }
           .thenAsync({
               next.handle(context, request)
           } as AsyncFunction)

See Also

N/A

Related Training

N/A

Related Issue Tracker IDs

OPENIG-910 (ScriptableFilter : Get error `Cannot execute script` with groovy scripts previously working)


Monitoring


How do I check if IG/OpenIG (All versions) is up and running?

The purpose of this article is to provide a way to check if IG/OpenIG is up and running when it is behind a load balancer. This article provides a way to call a page that is similar to the AM/OpenAM isAlive.jsp page.

Checking if IG/OpenIG is up and running

Since most of what IG/OpenIG does is proxy, its health mainly depends on the health of upstream applications for which IG/OpenIG is the proxy.

The following method provides a simple way to check that IG/OpenIG is up and running by establishing that IG/OpenIG can read its configuration and return a result without depending on a downstream application. The static text and URI shown are just examples, you should use values that make sense in the environment where IG/OpenIG is deployed:  

  1. Create a simple isAlive route. The result should return a 200 status along with some static text; for example, "Server is ALIVE". These results can be used to check on the health of IG/OpenIG.
  2. Add a new route to the IG/OpenIG configuration. For example, you could create a 00-isAlive.json file in the $HOME/.openig/config/routes directory, similar to the following:
    {
      "handler": {
          "type": "StaticResponseHandler",
          "config": {
            "status": 200,
            "reason": "Found",
            "entity": "Server is ALIVE"
          }
        },
      "condition": "${request.uri.path == '/isAlive.jsp'}"
    }
    
    Notice that /openig/ is not used for the route since this is reserved for admin purposes only; instead the /isAlive.jsp URI is used in this example.
  3. Configure the load balancer to check the results from this URL, for example, by calling the following URL: 
    http://openig.example.com:8080/isAlive.jsp
Note

If you require more detailed checks, you could make use of a scriptable handler instead: Configuration Reference › Handlers › ScriptableHandler.

See Also

How do I collect debug logs in IG/OpenIG (All versions) for troubleshooting?

How do I collect JVM data for troubleshooting IG/OpenIG (All versions)?

How do I check if AM/OpenAM (All versions) is up and running?

Gateway Guide › Configuring Routers and Routes

Gateway Guide › Monitoring

Related Training

N/A

Related Issue Tracker IDs

OPENIG-1370 (OpenIG Healthcheck Template)


Copyright and TrademarksCopyright © 2018 - 2019 ForgeRock, all rights reserved.

This content has been optimized for printing.

Loading...