# Performance tuning and monitoring ForgeRock products

This book provides information on performance tuning and monitoring ForgeRock products (AM/OpenAM, DS/OpenDJ, IDM/OpenIDM and IG/OpenIG).

## Best practice for JVM Tuning

The purpose of this article is to provide best practice advice on JVM tuning ; including key symptoms, understanding heap space and GC logs. This best practice advice applies to AM/OpenAM, DS/OpenDJ, IDM/OpenIDM and IG/OpenIG. JVM tuning is not an exact science and will vary across individual environments and applications. Consistent load and stress testing of your environment will be the only true way to gauge the effect of changes made.

#### JVM tuning considerations

Before you begin JVM tuning, you should consider the following points:

• Costs - depending on your environments, it may be more cost effective to add more hardware to improve performance rather than spending the time to tune.
• Desired outcome - tuning for stability is more important in the long run than tuning for performance, although there is an overlap.
• Ongoing problems - tuning may only delay or mask a problem temporarily if there is an underlying issue you need to resolve.
• Memory leaks - these will always cause Garbage Collection (GC) issues regardless of tuning.

#### What is Java® heap and how do I know it needs tuning?

The Java heap is the amount of memory allocated to applications running in the JVM. Objects in heap memory can be shared between threads. Garbage collection (GC) refers to the automatic process of managing the runtime memory. The JVM is subject to GC pauses that vary in frequency and duration; if these pauses become more frequent and/or last longer, you will see issues (such as application pauses) which indicate that you may need to tune your Java heap. The heap is initially created when the JVM starts and is split into different spaces or generations, the key ones being Young (Eden or Nursery) and Tenured (Old):

• The Young generation is used for new objects. A GC process (ParNew) automatically runs when it becomes full which removes unused objects and moves all objects that have lived long enough to the Tenured generation, thereby freeing up space in the Young generation for more new objects. Typically objects in the Young generation are short lived and temporary. The Young generation is small but active and GC happens frequently but with limited impact on performance.
• The Tenured generation is used for the longer lived objects. Another GC process (CMS) runs when it becomes full to remove any unused objects. The Tenured generation is larger and less active than the Young generation but GC tends to have more impact on performance.

The following diagram illustrates the heap and also the corresponding GC options that can be set in (Java 7 and earlier):

Key symptoms that indicate you need to tune your Java heap are:

• High CPU usage. When GC threads consume a lot of CPU, you will see spikes in overall CPU usage.
• Application hangs. When your application appears to hang and stops responding, there are gaps in your log files or you experience general slowness when performing routine tasks, poorly tuned GC can be the cause.

The first thing to check if you experience these symptoms is your GC logs; this are easy to check and may help you identify your issue quickly.

GC Logging

There are various different GCs available; ForgeRock recommends the use of CMS (ConcurrentMMarkSweep), which is a throughput collector and tends to give the best performance as it runs concurrently with the application, meaning less pauses. You should specify this collector by setting the following JVM option:

-XX:+UseConcMarkSweepGC


You can enable GC logging as described in the following articles:

##### Note

Typically you can leave GC logging on without impacting your system, however, you can just enable it when you are experiencing issues.

#### Understanding GC Logs

These example log files come from a CMS collector, where ParNew refers to the Young generation and CMS refers to the Tenured generation. These log file examples also show typical GC logs for a system where tuning is appropriate.

ParNew section

An example GC log output for the ParNew section looks like this:

24.245: [GC24.245: [ParNew: 545344K->39439K(613440K), 0.0424360 secs] 594671K->88767K(4101632K), 0.0426850 secs] [Times: user=0.17 sys=0.00, real=0.04 secs]
29.747: [GC29.747: [ParNew: 584783K->27488K(613440K), 0.0889130 secs] 634111K->104723K(4101632K), 0.0892180 secs] [Times: user=0.24 sys=0.01, real=0.09 secs]

Each line in this log file represents one GC and shows the following types of information for ParNew:

• Timestamp information - if you did not set the PrintGCTimeStamps and PrintGCDateStamps options, this just shows as the number of seconds from JVM startup:
24.245: [GC24.245:
29.747: [GC29.747:
This information is useful as it shows the frequency of GCs. You should be aiming for GCs to occur once every 1 to 5 seconds; if they are occurring more than once a second, you need to tune your JVM and it is likely that you will be seeing high CPU usage.
• Young generation information:
[ParNew: 545344K->39439K(613440K), 0.0424360 secs]
[ParNew: 584783K->27488K(613440K), 0.0889130 secs]
This information shows the initial size of the Young space before doing the GC, the size after doing the GC, the total size available and the time it took to do the GC. If the time taken to do the GC is large, that is more than 0.1 seconds, then it is possible that your Young generation heap size is too big. The total size available will not grow if you set NewSize and MaxNewSize to the same value and this is what you want to see in your logs.
• Overall heap information:
594671K->88767K(4101632K), 0.0426850 secs]
634111K->104723K(4101632K), 0.0892180 secs]
This information shows initial size of the overall heap before doing the GC, the size after doing the GC and the total size available. With the second lot of figures (size after doing the GC), you can see how this incrementally grows up to the point when the GC is done and then reduces back down to a baseline figure. If you have a memory leak, you will see the baseline figure growing incrementally as it becomes increasingly full even after doing GC.
• System information:
[Times: user=0.17 sys=0.00, real=0.04 secs]
[Times: user=0.24 sys=0.01, real=0.09 secs]
This information shows how much time was spent in user space, how much time spent in kernel or system space and what was the real impact to the user. If the user time is high, it indicates there isn't enough CPU for the number of user threads. If the system time is high, it indicates your system is swapping memory to disk which means that there isn't enough physical memory for your heap size.

CMS section

The CMS section of the log file is different to the ParNew section as it shows multiple lines for the same GC. An example GC log output for the CMS section looks like this:

40.146: [GC [1 CMS-initial-mark: 26386K(786432K)] 26404K(1048384K), 0.0074495 secs]
40.154: [CMS-concurrent-mark-start]
40.683: [CMS-concurrent-mark: 0.521/0.529 secs]
40.683: [CMS-concurrent-preclean-start]
40.701: [CMS-concurrent-preclean: 0.017/0.018 secs]
40.704: [GC40.704: [Rescan (parallel) , 0.1790103 secs]40.883: [weak refs processing, 0.0100966 secs] [1 CMS-remark: 26386K(786432K)] 52644K(1048384K), 0.1897792 secs]
40.894: [CMS-concurrent-sweep-start]
41.020: [CMS-concurrent-sweep: 0.126/0.126 secs]
41.020: [CMS-concurrent-reset-start]
41.147: [CMS-concurrent-reset: 0.127/0.127 secs]

This log shows the different phases the GC goes through. All the ones marked concurrent are happening while your application is running and therefore have little impact. The ones that don't say concurrent are stopping your application from running temporarily and therefore can have an impact. The CMS-initial-mark and CMS-remark show heap sizing details.

##### Note

Typically, you will see these sections intermingled, for example, some ParNews then a CMS, some more ParNews, another CMS and so on as the GC processes automatically occur to manage the heap.

#### Common issues shown in the GC logs and solutions

This section identifies common issues you might find in your GC logs and how you can resolve them. You can also use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

Young generation - too small

The following log snippet shows an issue where the Young generation heap size is too small:

1.813: [GC1.813: [ParNew: 1152K­>128K(1152K), 0.0008100 secs] 16620K­->15756K(26936K), 0.0008350 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
1.816: [GC1.816: [ParNew: 1152K­>128K(1152K), 0.0006430 secs] 16780K­->15913K(26936K), 0.0006640 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
1.819: [GC1.819: [ParNew: 1152K­>128K(1152K), 0.0005370 secs] 16937K-­>16038K(26936K), 0.0005570 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]

As you can see, there are multiple GCs occurring within a second. This means you need to increase the size of the Young generation (NewSize and MaxNewSize) and may also need to increase total heap size to compensate for this change (Xms and Xmx). Alternatively you could change the NewRatio option (this is set to 2 by default, which means the Tenured generation heap is twice as big as the Young generation heap or that the Young generation heap is a 1/3 of the size of the Tenured generation heap).

Young generation - too large / Tenured generation - too small

The following log snippet shows an issue where the Young generation heap size is too large and consequently the Tenured generation heap size is too small:

276.716: [GC (CMS Initial Mark) [1 CMS-initial-mark: 104176K(135168K)] 2758985K(6741248K), 0.8762860 secs] [Times: user=0.88 sys=0.00, real=0.88 secs]
277.592: [CMS-concurrent-mark-start]
277.657: [CMS-concurrent-mark: 0.063/0.065 secs] [Times: user=0.12 sys=0.00, real=0.06 secs]
277.657: [CMS-concurrent-preclean-start]
277.658: [CMS-concurrent-preclean: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
277.658: [CMS-concurrent-abortable-preclean-start]
277.658: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
277.658: [GC (CMS Final Remark)[YG occupancy: 2657419 K (6606080 K)]277.658: [Rescan (parallel) , 0.9815460 secs]278.639: [weak refs processing, 0.0000320 secs]278.640: [scrub string table, 0.0011700 secs] [1 CMS-remark: 104176K(135168K)] 2761595K(6741248K), 0.9828250 secs] [Times: user=7.18 sys=0.09, real=0.99 secs]
278.641: [CMS-concurrent-sweep-start]
278.668: [CMS-concurrent-sweep: 0.026/0.027 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
278.668: [CMS-concurrent-reset-start]
278.668: [CMS-concurrent-reset: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
280.673: [GC (CMS Initial Mark) [1 CMS-initial-mark: 104079K(135168K)] 2774091K(6741248K), 0.9033730 secs] [Times: user=0.90 sys=0.00, real=0.90 secs]
281.577: [CMS-concurrent-mark-start]
281.640: [CMS-concurrent-mark: 0.063/0.063 secs] [Times: user=0.13 sys=0.00, real=0.07 secs]
281.640: [CMS-concurrent-preclean-start]
281.641: [CMS-concurrent-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
281.641: [CMS-concurrent-abortable-preclean-start]
281.641: [CMS-concurrent-abortable-preclean: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
281.641: [GC (CMS Final Remark)[YG occupancy: 2670011 K (6606080 K)]281.641: [Rescan (parallel) , 0.9914290 secs]282.633: [weak refs processing, 0.0000110 secs]282.633: [scrub string table, 0.0008100 secs] [1 CMS-remark: 104079K(135168K)] 2774091K(6741248K), 0.9923100 secs] [Times: user=7.14 sys=0.11, real=0.99 secs]
282.634: [CMS-concurrent-sweep-start]
282.659: [CMS-concurrent-sweep: 0.024/0.025 secs] [Times: user=0.06 sys=0.01, real=0.02 secs]
282.659: [CMS-concurrent-reset-start]
282.659: [CMS-concurrent-reset: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 

As you can see, the CMS collections are occurring too frequently, without any ParNew collections in between. This is a result of the Young generation heap size being too large so that each time a ParNew collection happens, the Tenured heap size instantly becomes full and a CMS collection runs. This could happen if you increased your Young generation heap size without increasing your total heap size. This log snippet could also happen if you have memory leaks.

CMS failure

The following log snippet shows a CMS failure (concurrent mode failure):

(concurrent mode failure): 1551948K->1548050K(1572864K), 15.1154540 secs] 1986510K->1548050K(2044736K), [CMS Perm : 78597K->78588K(524288K)], 15.4406700 secs] [Times: user=15.52 sys=0.00, real=15.44 secs]
703240.983: [GC [1 CMS-initial-mark: 1548050K(1572864K)] 1561545K(2044736K), 0.0059690 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]


This error means that the CMS collector cannot keep up with the amount of garbage being generated (the Tenured generation heap space is about 90% full) and GCs are taking longer meaning your application stops for longer periods of time. This example shows the application stopping for over 15 seconds, which will have a significant impact on performance.

You can add the following GC options to try to avoid the CMS failure:

• -XX:CMSInitiatingOccupancyFraction=<percentage-of-tenured>
• -XX:+UseCMSInitiatingOccupancyOnly

You should set -XX:CMSInitiatingOccupancyFraction to 70 or 80% (instead of default ~90%) which may enable the CMS collector to keep up with the amount of garbage being generated.

Full GC

The following log snippet shows a Full GC:

25.466: [Full GC )25.466: [CMS: 68756K­>75361K(1987392K), 0.2548820 secs] 96571K­->75361K(2064064K), [CMS Perm : 61069K­>60722K(61376K)], 0.2551850 secs] [Times: user=0.25 sys=0.01, real=0.25 secs]
26.185: [Full GC (System.gc())26.185: [CMS: 75361K­>73283K(1987392K), 0.2398450 secs] 76136K-­>73283K(2064064K), [CMS Perm : 60723K­>60723K(101204K)], 0.2400120 secs] [Times: user=0.24 sys=0.00, real=0.24 secs]
27.293: [Full GC (System.gc())27.293: [CMS: 73283K­>73291K(1987392K), 0.2111960 secs] 74096K­>73291K(2064064K), [CMS Perm : 60723K­>60723K(101396K)], 0.2112730 secs] [Times: user=0.21 sys=0.00, real=0.22 secs]
68.081: [Full GC (Heap Inspection Initiated GC)68.081: [CMS: 73291K­->73187K(1987392K), 0.2172670 secs] 77634K­>73187K(2064064K), [CMS Perm : 60742K­>60742K(101396K)], 0.2174520 secs] [Times: user=0.22 sys=0.00, real=0.21 secs]

These log snippets refer to Full GCs occurring, meaning that the entire heap is being cleared (Young and Tenured); this often happens due to system processes such as RMI which performs a Full GC every hour on the hour. The PrintGCCause option is very useful to help you find out why the GC is happening (the last three lines in the log above show the additional information output by this option, making it clear which ones are system related). You can also set the DisableExplicitGC option to prevent system related Full GCs from happening.

Permanent generation

##### Note

The following information does not apply to Java 8 since the Permanent generation space has been removed.

The following log snippet shows a Full GC as a result of the Permanent generation space growing (the Permanent generation space contains metadata and other data that should be permanently stored and does not typically grow):

1.241: [Full GC (Ergonomics) [PSYoungGen: 6125K­>0K(45056K)] [ParOldGen: 17659K­->20350K(20480K)] 23785K­>20350K(65536K) [PSPermGen: 13550K­->13546K(27136K)], 0.0531680 secs] [Times: user=0.25 sys=0.00, real=0.05 secs]
1.513: [Full GC (Ergonomics) [PSYoungGen: 38912K­>2783K(45056K)] [ParOldGen: 20350K­>20424K(20480K)] 59262K­>23208K(65536K) [PSPermGen: 15188K­->15188K(31744K)], 0.0992090 secs] [Times: user=0.65 sys=0.02, real=0.10 secs]

These log snippets show the Permanent generation space growing (figure in brackets after PSPermGen). You can increase the MaxPermSize option to resolve this issue.

#### Best practice advice on tuning

Initially, you should configure your JVM as per the recommended settings for your product (if available):

You can then tune your JVM as and when required.

You should always follow this basic process when tuning:

1. Establish a baseline using a valid load testing process. This step is essential as it is the only way you will know if your tuning is having the desired effect.
2. Change one thing at a time and document each change. This allows you to determine the impact of each change and revert if necessary.

Heap tuning guidelines

Although we cannot give recommended heap settings because they vary considerably between environments and applications, you should follow these guidelines when tuning:

• You should always use the latest supported software versions.
• You should use the CMS GC as recommended by ForgeRock.
• Your total heap size must not exceed the amount of physical RAM available.
• You must leave some physical RAM for other applications to run on your machine.
• You should set Xms and Xmx to the same value for best performance. These options determine your total heap size.
• You should set NewSize and MaxNewSize to the same value. These options determine how the heap is allocated between generations.
• You should set the DisableExplicitGC option to prevent system related full GCs from running.
• You must not add lots of options at once or add options you don't understand.
• You must not copy and paste options from other applications.
• You should use GC logs to help identify issues with your tuning.

Getting your total heap size and allocation between generations right is key; once you have these set correctly, you likely won't need to do anymore tuning.

Summary

In summary, the most important GC knobs to get right are:

1. The collector, which should be CMS.
2. The total heap size, where Xmx = Xms.
3. The split between generations, using NewSize or NewRatio, where NewSize = MaxNewSize.

AM/OpenAM

DS/OpenDJ

IDM/OpenIDM

IG/OpenIG

## FAQ: AM/OpenAM performance and tuning

The purpose of this FAQ is to provide answers to commonly asked questions regarding performance and tuning for AM/OpenAM and Policy Agents.

#### Q. Do I need to performance test AM/OpenAM?

A. Yes, performance testing is a very important step to ensure that the system you have configured will cope with the expected load once it is in production. By performance testing your system, you can tweak any performance issues you encounter before going live.

Before running any performance tests, you should ensure AM/OpenAM is running and configured in a test environment that closely resembles your planned production environment. You should also be aware of the type of load expected in your production system and the types of identity data involved to make your tests as realistic as possible. For example, you may want to test that your system can handle 20 authentications per second with a concurrency of 2,000 live sessions.

Generating test identity data in DS/OpenDJ is described in How do I generate sample user data for performance testing in DS/OpenDJ (All versions)?

Performing a performance test is described in idmdude - It's OK to Get Stressed Out with OpenAM

#### Q. How do I tune AM/OpenAM to improve performance?

A. Tuning AM/OpenAM is described in Setup and Maintenance Guide › Tuning an Instance.

#### Q. How can I improve the performance of ssoadm?

A. There are several approaches you can take to improving the performance of ssoadm as detailed in How do I improve the performance of ssoadm in AM/OpenAM (All versions)?

#### Q. What is the recommended Java Virtual Machines (JVM) heap size for AM/OpenAM?

A. There are no definitive rules for the size of JVM heap size required as it will vary across individual environments and applications, but you should refer to Best practice for JVM Tuning for best practice advice. There are also a number of things you should be aware of when making a decision for AM/OpenAM:

• AM/OpenAM response time is independent of heap size.
• The number of concurrent Single Sign On (SSO) sessions is dependent on heap size. As a rough guide, an AM/OpenAM server in production with a 3 GB heap can handle 100,000 sessions. As heap size increases, so does the number of possible concurrent sessions, however, smaller heap sizes are easier to manage.
• You must ensure you configure JVM garbage collection appropriately as garbage collection runs can get quite long if you have large heap sizes.
##### Note

For a 32-bit JVM or a 32-bit operating system, the limit for the process size is 4GB, that is, 2^32; this cannot be exceeded regardless of the amount of heap space allocated.

#### Q. Can I change caching in AM/OpenAM to improve performance?

A. Yes, you can improve performance in AM/OpenAM by configuring caching on the server side; you can configure caching for configuration data and global user data. This is described in Setup and Maintenance Guide › Tuning Caching. You need to balance performance against memory usage when configuring caching; larger caches lead to less requests to AM/OpenAM but increase memory usage.

#### Q. How do client-based sessions affect performance?

A. Client-based sessions were introduced in OpenAM 13 per: OpenAM 13 Release Notes › What's New in OpenAM › Elasticity and Scaling Features.

Client-based sessions use more CPU than CTS-based sessions; however CTS-based sessions have higher RAM consumption.

See How do I enable Client-based sessions in AM (All versions) and OpenAM 13.x? for further information.

#### Q. How does the Session Purge Delay setting affect performance?

A. The Session Purge Delay setting (com.iplanet.am.session.purgedelay) is set to 0 by default, which removes the session from memory immediately. If you increase this to a value above 0 (not recommended), the session is then held in memory for that number of minutes before being removed. This setting has been removed in AM 5.

If you have a particular need to increase the session purge delay, you should increase your heap size to compensate for the increase in sessions as described in How do I change the JVM heap size for AM/OpenAM (All versions)?

See Setup and Maintenance Guide › Session Settings​ for further information about these settings.

#### Q. What else might affect performance on a Linux system or a Virtual Machine?

A. Low entropy values can cause general system slowness.

##### Note

Low entropy values is a OS/Java/web container issue, which is independent of AM/OpenAM; as such, the suggested resolutions are outside the scope of ForgeRock support. If you want more tailored advice, consider engaging Deployment Support Services.

On Linux systems you can check if a low entropy value is affecting you by running the following command:

$cat /proc/sys/kernel/random/entropy_avail A response of 200 or less is considered low and should be addressed, but you may find that even higher values can cause system slowness. Resolution You can increase the amount of entropy available in a variety of ways; two possible approaches are: • Use an external tool called RNGD (Random Number Generator Deamon). This can either use the RDRAND instruction set or /dev/urandom if RDRAND is not available, although many Virtual Machines (such as VirtualBox and OpenStack) passthrough the RDRAND instruction set to the VM. You will need to install the rng-tools package. • Set the following JVM option in the application web container in which AM/OpenAM runs: -Djava.security.egd=file:/dev/./urandom #### Q. Which Apache httpd Multi-Processing Module (MPM) should I use with web policy agent? A. It is recommended that you always use Worker MPM rather than Prefork MPM. You should ensure there are enough processes and threads available to service the expected number of client requests. You can then tune the Worker mode performance in the conf/extra/http-mpm.conf file as detailed in Web Agents User Guide › Tuning Apache Multi-Processing Modules. You can check which MPM Apache is using with the following command: $ apachectl -V

The MPM in use is shown against Server MPM.

##### Note

Turning off the Apache™ KeepAlive feature can potentially improve performance as well since the KeepAlive feature increases memory usage. You should test your performance with KeepAlive enabled and then again with it off to check what impacts it has on your setup.

#### Q. Are there things I should monitor on an ongoing basis in terms of AM/OpenAM performance?

A. Yes, it is useful to monitor performance on an ongoing basis to allow you to react to changes quickly. Useful things to monitor include:

• Heap size
• Number of open sessions and size of sessions
• CPU utilization - utilization of 70-80% is worth investigating.

See How do I monitor session statistics in AM/OpenAM (All versions)? and Setup and Maintenance Guide › Monitoring Services for further information.

## FAQ: Caching in AM/OpenAM

The purpose of this FAQ is to provide answers to commonly asked questions regarding caching in AM/OpenAM.

##### Note

Tuning recommendations are environment specific and are outside the scope of ForgeRock support; if you want more tailored advice, consider engaging Deployment Support Services

#### Q. How can I control caching for configuration and user data using ssoadm?

A. There are three properties available for enabling / disabling caching for configuration and user data stores:

• com.iplanet.am.sdk.caching.enabled - enables caching for both the configuration and user data stores. The following properties are ignored when this is set to true.
• com.sun.identity.idm.cache.enabled - enables caching for the user data store (IdRepo cache).
• com.sun.identity.sm.cache.enabled - enables caching for the configuration data store (service configuration caching).

See Setup and Maintenance Guide › Cache Settings for further information about these properties and other cache related properties, including time based expiration. This guide also provides steps for setting these advanced properties in console.

You can set these properties to true or false using the update-server-cfg command. For example, the following command enables caching for the user data store only:

#### Changing the realm level setting

##### Note

You may need to add the Session service if it is not listed under Services by clicking Add a Service or Add and then selecting Session. If you are using ssoadm, you can replace set-realm-svc-attrs in the ssoadm command with add-svc-realm to add this service and set the attributes with the same command.

You can configure the realm level maximum caching time using either the console or ssoadm:

• AM / OpenAM 13.x console: navigate to: Realms > [Realm Name] > Services > Session and enter the required number of minutes for the maximum caching time.
• Pre-OpenAM 13 console: navigate to: Access Control > [Realm Name] > Services > Session and enter the required number of minutes for the maximum caching time.
• ssoadm: enter the following command:

N/A

N/A

## How do I change the JVM heap size for AM/OpenAM (All versions)?

#### Changing the JVM heap size

Changing the JVM heap size can improve performance and reduce the time it takes to perform authentications. You should try different heap sizes to see what impact it has to determine the best heap size for your setup. The information given here is specific to the Apache Tomcat™ web container; you should make similar changes in the configuration file specific to your web container if you use a different one.

##### Note

It is recommended that you set the minimum and maximum heap sizes to the same value for best performance. Otherwise, the JVM runs a full Garbage Collector (GC) cycle when increasing its heap, during which time it can pause ongoing operations for up to a few seconds. Generally, a smaller heap will increase the frequency at which the JVM GC executes but reduce the duration; similarly, a larger heap will reduce the frequency and increase the duration. When tuning the JVM heap, the goal is to strike a balance between frequency and duration so as to reduce the impact of the GC on the application.

On Unix® and Linux® systems:

You can set the JVM heap size by specifying CATALINA_OPTS settings in the setenv.sh file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.sh file (also typically located in the /tomcat/bin/ directory).

For example, to set the minimum and maximum heap sizes to 2GB, you would add the following line to the setenv.sh file and restart the web container:

export CATALINA_OPTS="$CATALINA_OPTS -Xms2g -Xmx2g" On Microsoft® Windows® systems: Providing you haven't installed Tomcat as a service, you can set the JVM heap size by specifying CATALINA_OPTS settings in the setenv.bat file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.bat file (also typically located in the /tomcat/bin/ directory). For example, to set the minimum and maximum heap sizes to 2GB, you would add the following line to the setenv.bat file and restart the web container: set CATALINA_OPTS=-Xms2g -Xmx2g If you have installed Tomcat as a service on Windows, then you must use the GUI application to configure Tomcat services. This process refers to Tomcat 7, but will work for other versions by changing tomcat7w.exe in the command to match your version. 1. Stop the Tomcat service. 2. Navigate to \path\to\tomcat\bin\ from the command prompt: 3. Enter the following command to display Tomcat Properties: tomcat7w.exe //ES//serviceName Where serviceName is the name of your Tomcat service. 4. Navigate to the Java tab and complete the memory pool fields as follows: Initial memory pool: 2048 Maximum memory pool: 2048 5. Restart the Tomcat service. Alternatively, Windows system administrators may prefer to configure these options in the registry so that they may be configured via group policy. The initial memory pool and maximum memory pool values can be configured in the JvmMS and JvmMX properties under the following registry key for Tomcat 7: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\Tomcat7\Parameters\Java #### See Also #### Related Training N/A #### Related Issue Tracker IDs N/A ## How do I configure the heartbeat timeout in AM/OpenAM (All versions)? The purpose of this article is to provide information on configuring the heartbeat timeout in AM/OpenAM. This allows you to tune the heartbeat timeout if you experience issues with heartbeat timeouts. #### Overview Heartbeat timeout is the time spent on AM/OpenAM sending and receiving the heartbeat request and is not pure processing time on DS/OpenDJ. The default heartbeat timeout of 3 seconds is configurable (since OpenAM 12.0.3), which is useful if you are experiencing heartbeat timeouts. You will see errors such as the following in your logs if you are experiencing heartbeat timeouts: • Session debug log (when debug level is set to Message): Caused by: org.forgerock.opendj.ldap.ConnectionException: Server Connection Closed: Heartbeat timed out after 500 ms  • IdRepo debug log (when debug level is set to Message): org.forgerock.opendj.ldap.ConnectionException: Server Connection Closed: Heartbeat timed out after 500 ms  • Web application container log (for example, catalina.out for Apache Tomcat™): WARNING: No heartbeat detected for connection 'LDAPConnection(/192.10.210.10:8080,host1.example.com/192.10.210.15:18080)'  If you do see errors such as the above, you should increase the heartbeat timeout; it is recommended that you initially increase it to 10 seconds and then slowly increase it further if you continue to see errors. #### Configuring the heartbeat timeout You can configure the heartbeat timeout using either the console or ssoadm: • AM / OpenAM 13.5 console: navigate to: Deployment > Servers > [Server Name] > Advanced and add the org.forgerock.openam.ldap.heartbeat.timeout property and enter the heartbeat timeout in seconds for the value. To add the property as a server default, navigate to: Configure > Server Defaults > Advanced instead. Once you have entered the property and value, click + to add followed by Save Changes. • Pre-OpenAM 13.5 console: navigate to: Configuration > Servers and Sites > [Server Name] > Advanced and add the org.forgerock.openam.ldap.heartbeat.timeout property and enter the heartbeat timeout in seconds for the value. You can configure this for all servers using Default Server Settings instead of [Server Name]. • ssoadm: enter the following command: $ ./ssoadm update-server-cfg -s [servername] -u [adminID] -f [passwordfile] -a org.forgerock.openam.ldap.heartbeat.timeout=[seconds]

replacing [servername], [adminID], [passwordfile] and [seconds] with appropriate values, where [servername] can be default if you want to change the default server settings instead. For example:
$./ssoadm update-server-cfg -s default -u amadmin -f pwd.txt -a org.forgerock.openam.ldap.heartbeat.timeout=10 ##### Note You must restart the web application container in which AM/OpenAM runs to apply these configuration changes. #### See Also #### Related Training N/A #### Related Issue Tracker IDs ## How do I improve OAuth 2.0 performance in OpenAM 13.0? The purpose of this article is to provide information on improving OAuth 2.0 performance in OpenAM 13.0. #### Improving OAuth 2.0 performance In OpenAM 13.0, there is a known performance issue: OPENAM-8023 (OAuth2 load: contention at com.sun.identity.sm.ServiceConfigImpl.getInstance level (perf regression compared to AM12.0.2)). This issue is resolved in OpenAM 13.5. You can mitigate this issue in OpenAM 13.0 as follows: 1. Create a completely empty file. For example, you can use the following command if you are using a Linux® or Unix® operating system: $ touch [filename]
2. Run the following ssoadm commands against all realms where you are using OAuth2:
$./ssoadm add-svc-realm -e [realmname] -s ScriptingService -u [adminID] -f [passwordfile] -D [emptyfile]$ ./ssoadm create-sub-cfg -e [realmname] -s ScriptingService -g scriptConfigurations -u [adminID] -f [passwordfile] -D [emptyfile]


N/A

N/A

## How do I tune LDAP connection pool settings in AM/OpenAM (All versions) using ssoadm?

#### Tuning LDAP connection pool settings

LDAP data store settings

You can configure the LDAP data store connection pool sizes as follows using ssoadm:

• LDAP Connection Pool Minimum Size: enter the following command:
$./ssoadm update-datastore -e [realmname] -m [datastorename] -u [adminID] -f [passwordfile] -a sun-idrepo-ldapv3-config-connection_pool_min_size=[size] replacing [realmname], [datastorename], [adminID], [passwordfile] and [size] with appropriate values. • LDAP Connection Pool Maximum Size: enter the following command: $ ./ssoadm update-datastore -e [realmname] -m [datastorename] -u [adminID] -f [passwordfile] -a sun-idrepo-ldapv3-config-connection_pool_max_size=[size]

LDAP authentication modules

You can configure the LDAP authentication module connection pool sizes (Default LDAP Connection Pool Size) as follows using ssoadm:

$./ssoadm set-attr-defs -s iPlanetAMAuthService -t global -u [adminID] -f [passwordfile] -a iplanet-am-auth-ldap-connection-pool-default-size=[minSize:maxSize] replacing [adminID], [passwordfile] and [minSize:maxSize] with appropriate values. For example, if you wanted to tune this to the recommended production settings: 10 (minimum connection pool size) and 65 (maximum connection pool size), you would use an ssoadm command such as: $ ./ssoadm set-attr-defs -s iPlanetAMAuthService -t global -u amadmin -f pwd.txt -a iplanet-am-auth-ldap-connection-pool-default-size=10:65

N/A

N/A

## How do I improve the performance of ssoadm in AM/OpenAM (All versions)?

#### Improving the performance of ssoadm

There are five things you can try to improve the performance of ssoadm:

Establishing a baseline

You can add time before the ssoadm command to output the execution time of the ssoadm command, which will help you assess the effect of any changes you make. It is recommended you run a simple command, such as the following, prior to making any changes to give you a baseline:

#### Adding a Java option to change how random bits are generated

In some situations, running ssoadm on Linux systems can result in performance issues; this is particularly true if you are using a virtual machine. When this happens, you will see a stack trace similar to the following when generating the SSOToken ID:

"main" prio=10 tid=0x00007f806400c000 nid=0x7e3f runnable [0x00007f8068918000]
at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:526) at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:139) at sun.security.provider.SecureRandom$SeederHolder.(SecureRandom.java:186)
at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:203)
- locked <0x00000000ed1684f0> (a sun.security.provider.SecureRandom)
at java.security.SecureRandom.nextBytes(SecureRandom.java:455)
- locked <0x00000000ed168790> (a java.security.SecureRandom)
at java.security.SecureRandom.next(SecureRandom.java:477)
at java.util.Random.nextLong(Random.java:334)
at com.sun.identity.authentication.internal.AuthContext.getSSOToken(AuthContext.java:842)
at com.sun.identity.setup.Bootstrap.getSSOToken(Bootstrap.java:289)
at com.sun.identity.setup.Bootstrap.getConfiguration(Bootstrap.java:203)
- locked <0x00000000eb0536e0> (a java.lang.Class for com.sun.identity.setup.Bootstrap)
at com.sun.identity.cli.CommandManager.main(CommandManager.java:113)


To improve the performance in this situation, you can add the following JVM option to the ssoadm script:

-D"java.security.egd=file:/dev/./urandom" \

N/A

## How do I enable Garbage Collector (GC) Logging for AM/OpenAM (All versions)?

#### Enabling GC Logging

The information given here is specific to the Apache Tomcat™ web container; you should make similar changes in the configuration file specific to your web container if you use a different one.

##### Note

You should ensure there is enough disk space for the GC logs; if not, you should enable GC log rotation as well by including the following options when you enable GC logging: -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=n, -XX:GCLogFileSize=n

On Unix® and Linux® systems:

​You should enable GC logging by specifying CATALINA_OPTS settings in the setenv.sh file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.sh file (also typically located in the /tomcat/bin/ directory).

To enable GC logging:

1. Add the following line to the setenv.sh file:
export CATALINA_OPTS="$CATALINA_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]" replacing [filename] with the path to the file that you would like to create to store the log file. 2. Restart the web container. Once the web container has successfully restarted, there should be a GC log file located in the directory specified in the -Xloggc: option. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock. On Microsoft® Windows® systems: ​You should enable GC logging by specifying CATALINA_OPTS settings in the setenv.bat file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.bat file (also typically located in the /tomcat/bin/ directory). To enable GC logging: 1. Add the following line to the setenv.bat file: set CATALINA_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename] replacing [filename] with the path to the file that you would like to create to store the log file. 2. Restart the web container. Once the web container has successfully restarted, there should be a GC log file located in the directory specified in the -Xloggc: option. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock. #### See Also #### Related Training N/A #### Related Issue Tracker IDs N/A ## How do I diagnose a hung AM/OpenAM (All versions) server? The purpose of this article is to provide information on diagnosing a hung AM/OpenAM server, including the data you should collect to send to ForgeRock Support for troubleshooting. #### Diagnosing a hung AM/OpenAM server There are a number of different types of data that you should collect to help us diagnose a hung AM/OpenAM server. If you suspect you are experiencing a hung AM/OpenAM server, you should collect the following data and submit it to us when you raise the ticket to enable us to help you more quickly: • Debug logs. • JVM data. • Garbage Collector (GC) logs. • HTTP container logs. • Native stack traces. ##### Note It is important that you collect all the data immediately after reproducing the issue so that the timestamps of the data collected and the suspected hung AM/OpenAM server correspond. Please notify us of the exact timestamp when the issue occurred so we can focus our attention on the relevant information. Debug logs Message level debug logs provide more context for diagnosing errors but can generate large files. You should enable message level debugging when you are experiencing an issue as described in How do I enable Message level debugging in AM/OpenAM (All versions) debug files? JVM data JVM data must be collected before killing the affected process or restarting the server; otherwise the information relating to the affected process is lost forever, which may make it impossible to identify the root cause. There are three types of data that you should collect (stack traces, heap histograms and heap dumps) as well as providing details of the current JVM settings as described in How do I collect JVM data for troubleshooting AM/OpenAM (All versions)? See How do I use the msnapshots script to capture information for troubleshooting AM/OpenAM (All versions)? for further information on using a script to collect this data if you are using a Linux®, Unix® or Solaris® system. GC logs GC logs are not always crucial at this stage, but can provide useful information about potential tuning issues that may be contributing to your problems. Enabling GC logging is described in How do I enable Garbage Collector (GC) Logging for AM/OpenAM (All versions)? HTTP container logs HTTP container server logs (Access and Error) are very useful for troubleshooting as between them they show all requests to the server, whether they were successful or not and any corresponding error messages. The actual name and location of the server logs are specific to your web container. For example, for Apache Tomcat™, the logs are called localhost_access_log.YYYY-MM-DD.log and catalina.out, and are located in the /path/to/tomcat/logs/ directory by default. Native stack traces Native stack traces are not always necessary but can provide useful information when troubleshooting, especially if you can isolate the the affected process. You can use the pstack command for Solaris® and Linux® (although pstack is not installed by default on Linux): $ pstack -F [pid]

replacing [pid] with the process ID of the affected process.

N/A

## How do I monitor session statistics in AM/OpenAM (All versions)?

The purpose of this article is to provide information on monitoring session statistics in AM/OpenAM. This can provide useful troubleshooting information if you are experiencing unexpectedly high session numbers.

#### Overview

There are a number of ways you can monitor session statistics in AM/OpenAM depending on your version. These include:

AM 5 and later

You can monitor sessions in AM 5 and later as follows:

• Session page - navigate to: Realms > [Realm Name] > Sessions to access the Sessions page, which allows you to view and invalidate active CTS-based user sessions per realm.
• REST API - you can query the /json/sessions endpoint (see the Using the /json/sessions endpoint section below for further information).
• Amster - you can use Amster to query sessions (see the Using Amster section below for further information).

By default, only 120 sessions are returned using these methods. You can change this default using the Maximum Number of Search Results setting. See Authentication and Single Sign-On Guide › Session Search for further details.

##### Note

Changes in AM 5 that made all servers autonomous mean it is the CTS store that holds the session data rather than the AM server itself. As such, the options listed for OpenAM 13.x and earlier cannot be used with AM 5 since they query individual servers. This is a known issue: OPENAM-9817 (JMX CTS session count is broken).

OpenAM 13.x and earlier

Session details

Session management information, including attribute values such as login time, logout time, time out limits, session creations and terminations, are logged in the amSSO.access log file (typically located in the $HOME/[am_instance]/openam/log directory). You will also see session information in the CoreSystems, Authentication and Session debug files ($HOME/[am_instance]/openam/debug directory).

You may notice a difference in session counts if you have just restarted the server; the monitoring API uses beans to store the values, for example, the session count. It is currently implemented such that each counter has an increment and a decrement method. When you log in, the counter gets incremented and when the session is destroyed, it is decremented. When AM/OpenAM starts up, all counters on the local instance are at 0. On the other hand, the counter used to count sessions for the session quotas and the Sessions tab is different; it actually counts the sessions, depending on "server mode" (single instance, multi instance or SFO). If you're using SFO, it counts all sessions from the CTS session store.

If, for example, you have just restarted AM/OpenAM, you may still have all 10 sessions in the session store (depending on how many of them have expired since). So when you start authenticating, two things happen at the same time (let's assume that you already have 10 sessions in the session store):

• User session quotas are checked and sessions are counted. This will find the existing sessions in LDAP. If the count exceeds the limit, it executes the quota action, for example, "destroy next expiring". This will result in decrementing the session counter in the monitoring bean. At this point it's at 0, so it stays at 0. The number of sessions in the CTS is now 9.
• Authentication succeeds and the session gets created. 10 sessions in CTS, SNMP counter now reads 1.

#### Using the /json/sessions endpoint (AM 5 and later)

You can query the /json/sessions endpoint to find session details. For example, to find sessions in the top level realm, you would use a call such as:

$curl -X GET -H 'Accept: application/json' 'http://host1.example.com:8080/openam/json/sessions?_queryFilter=realm%20eq%20%22%2F%22' The easiest way to find the relevant command is by using the API Explorer: 1. Access the API Explorer. You can either access it from the Help icon in the console or from the following URL: http://host1.example.com:8080/openam/XUI/#api/explorer/applications 2. Navigate to /sessions > Sessions V2.0 > sessions#2.0_query_filter. 3. Populate the query fields as required; _queryFilter is a required field. For example, to query the top level realm, enter realm eq "/". 4. Click Try it Out! This returns session details and also provides the curl command you can use in future. #### Using Amster (AM 5 and later) You can use Amster to query session details using the query Sessions command. For example: am> query Sessions --realm / --filter 'realm eq "/"' Example response: ===> [ { "username": "amadmin", "universalId": "id=amadmin,ou=user,dc=example,dc=com", "realm": "/", "sessionHandle": "shandle:4r8SsX6XJj0oAbLBmexqyUsbC7Y.*AAJTSQACMDEAAlNLABxJNEhkVlRlMnNHRzVKUTlOa1hMQ3BiRzZad0E9AAJTMQAA*", "latestAccessTime": "2018-05-01T12:36:54.487Z", "maxIdleExpirationTime": "2018-05-01T13:06:54Z", "maxSessionExpirationTime": "2018-05-01T14:31:23Z", "_rev": "746064345" }, { "username": "demo", "universalId": "id=demo,ou=user,dc=example,dc=com", "realm": "/", "sessionHandle": "shandle:rn3PS1zCIBxmY5qnMtbbqJOLgkQ.*AAJTSQACMDEAAlNLABxNR2JvL0tUenQxc2N1YnU4MkN2YjNkeGY2UTQ9AAJTMQAA*", "latestAccessTime": "2018-05-01T12:36:50.448Z", "maxIdleExpirationTime": "2018-05-01T13:06:50Z", "maxSessionExpirationTime": "2018-05-01T14:36:50Z", "_rev": "856832111" } ]  See Entity Reference › Sessions › query for further information. #### Using the amMasterSessionTableStats file (OpenAM 13.x and earlier) The amMasterSessionTableStats file contains both user and application session data, and is located in the$HOME/[openam_instance]/openam/stats directory. This file shows current and peak values for the following items:

• Maximum number of sessions in the session table, including both active and timed out sessions.
• Maximum number of active sessions.
• Number of Session Notifications in the queue.
##### Note

There is a known issue with stats logging not working in OpenAM 13.0: OPENAM-6998 (Session "stats" logging doesn't seem to work). This is fixed in OpenAM 13.5.

#### Using ssoadm (OpenAM 13.x and earlier)

The ssoadm list-sessions command lists all the current sessions on a specified server with session details, for example:

$./ssoadm list-sessions -u amadmin -f pwd.txt -t http://host1.example.com:8080/openam  Example response: [Current Session] User Id: amAdmin Time Remain: 1199999 Max Session Time: 1200000 Idle Time: 0 Max Idle Time: 300000 Index: 0 User Id: user1 Time Remain: 1199991 Max Session Time: 1200000 Idle Time: 8 Max Idle Time: 300000 Index: 1 User Id: user7 Time Remain: 1199850 Max Session Time: 1200000 Idle Time: 139 Max Idle Time: 300000 Index: 2 User Id: user12 Time Remain: 1199993 Max Session Time: 1200000 Idle Time: 6 Max Idle Time: 300000 Index: 3 User Id: user3 Time Remain: 1199862 Max Session Time: 1200000 Idle Time: 137 Max Idle Time: 300000 Index: 4 User Id: user19 Time Remain: 1199997 Max Session Time: 1200000 Idle Time: 2 Max Idle Time: 300000 Index: 5 User Id: user28 Time Remain: 1199992 Max Session Time: 1200000 Idle Time: 7 Max Idle Time: 300000 Index: 6 User Id: user17 Time Remain: 1199998 Max Session Time: 1200000 Idle Time: 1 Max Idle Time: 300000 To invalidate sessions, enter the index numbers [CR without a number to exit]: You can run the command with the -q option, which does not prompt you to invalidate sessions and can add a grep command after to just return a total, for example: $ ./ssoadm list-sessions -u amadmin -f pwd.txt -t http://host1.example.com:8080/openam -q | grep "User Id" | wc -l


Example response (which matches the number of sessions listed above):

8


#### Using SNMP monitoring (OpenAM 13.x and earlier)

First you must enable SNMP monitoring; you can do this using either the console or ssoadm:

• OpenAM 13.5 console: navigate to: Configure > Global Services > System > Monitoring and enable Monitoring SNMP interface status.
• Pre-OpenAM 13.5 console: navigate to: Configuration > System > Monitoring > Monitoring SNMP interface status and select the Enabled option.
• ssoadm: enter the following command:
$./ssoadm set-attr-defs -s iPlanetAMMonitoringService -t Global -u [adminID] -f [passwordfile] -a iplanet-am-monitoring-snmp-enabled=true  replacing [adminID] and [passwordfile] with appropriate values. By default, this allows you to listen on port 8085 for SNMP monitoring. You can change the port via the console or ssoadm (iplanet-am-monitoring-snmp-port attribute) using the same navigation / service as above. ##### Note You must restart the web application container in which OpenAM runs to apply these configuration changes. Once enabled, you can use the snmpwalk command to provide specific session details. For example, the following command (assuming you are using port 8085) gives the number of active sessions on the current OpenAM server instance: snmpwalk -c public -v 2c :8085 enterprises.42.2.230.3.1.1.2.1.11.1.0 ##### Note This command gives the same output as the SessionActiveCount JMX attribute; where both are reset to 0 when the OpenAM instance is restarted. You can check which SNMP definitions are supported in the mib/FORGEROCK-OPENAM-SESSION.mib file (enterprises 36733), which can be found in the openam.war/WEB-INF/lib/openam-mib-schema-1.x.x.jar file. See Administration Guide › Monitoring OpenAM Services › SNMP Monitoring for other session related OIDs you can use. #### See Also #### Related Training #### Related Issue Tracker IDs ## How do I set up a monitoring page for the load balancer in front of Web Policy Agents (All versions) for health checks? The purpose of this article is to provide information on setting up a monitoring page for the load balancer in front of Web policy agents. Creating a monitoring page is best practice for health checking when you have a load balancer in front of your Web policy agents. This article assumes you have already configured your load balancer and policy agents. #### Overview Creating a monitoring page on the server being protected by the Web policy agent means you can eliminate the policy agent being involved in the load balancer's health check. This monitoring page should be an unprotected resource and exist on the Not Enforced URL list. Even though the URL is on the Not Enforced URL list, the policy agent is still invoked each time the load balancer checks the monitoring page to determine whether the resource needs protecting or not; this means you can use this configuration to check if the policy agent is responding without the need for policy evaluation. #### Creating a monitoring page You can create a monitoring page as follows: 1. Create a monitor.html file on one of the servers being protected by the Web policy agent. This file can simply contain the HTML tags; for example, you can use the printf command to create is as follows: $ printf '<HTML>\n</HTML>' > monitor.html
2. Add the URL for this monitoring page to the Not Enforced URL list for this server:
• AM 5 and later console: navigate to: Realms > [Realm Name] > Applications > Agents > Web > [Agent Name] > Application > Not Enforced URLs and add the URL for the monitoring page, for example:
http://www.host1.example.com:8080/monitor.html
• OpenAM 13.x console: navigate to: Realms > [Realm Name] > Agents > Web > [Agent Name] > Application > Not Enforced URLs and add the URL for the monitoring page.
• Pre-OpenAM 13 console: navigate to: Access Control > [Realm Name] > Agents > Web > [Agent Name] > Application > Not Enforced URLs and add the URL for the monitoring page.
• ssoadm: enter the following command:

##### Note

We've run tests with many heap sizes, including some very large ones, and most of the large deployments use between 2GB and 32GB of JVM heap size.

N/A

## How do I tune the DS/OpenDJ (All versions) database file cache?

#### Tuning database file cache

The je.log.fileCacheSize property represents the number of file handles that are cached, that is, file handles that remain open by the underlying database (DB). The default value is 200 (100 in pre-DS 6), which is probably far less than the number of .jdb files you have in the /path/to/ds/db/userRoot directory.

##### Note

You should also be aware of the maximum number of file handlers defined in DS/OpenDJ as discussed in the release notes applicable to your version: Release Notes › Setting Maximum Open Files.

When the DB cannot be fully cached, the DB needs to access the record from the log files. Since only a portion of the files can be opened at one time, the DB keeps closing them and opening the ones it needs. Each close does a disk sync, which has a large penalty on performance.

The number of .jdb files depends on the DB content and also on the occupancy ratio. Occupancy ratio has an impact on the number of files. Berkeley DB JE uses a rolling log algorithm in which writes to the DB are always appended to the current log file. When the log reaches a certain size, JE creates a new one and writes to it. When records are deleted or even updated, the old record is marked as deleted and the new one is appended to the log file. This creates unused records in the previous log files. The occupancy ratio is the ratio of used records / total number of records per log file. When the ratio for one log file hits a certain limit (50% by default), the remaining valid records are copied to the current log file (still appending) and the old log file is deleted. So in theory and in the worse case, the number of log files cannot exceed twice the initial number of files after an import (with the same amount of data).

To set a proper value for the je.log.fileCacheSize, therefore, count the current number of .jdb files, double it and round down.

If it happens that the number of .jdb files exceeds the value set, then the DB starts closing files and reopening other ones, slowly degrading performance.

This property can be maintained using the db-log-filecache-size configuration attribute, for example:

$./dsconfig set-backend-prop --port 4444 --bindPassword password --backend-name userRoot --advanced --set db-log-filecache-size:[cachesize] --trustAll --no-prompt  #### See Also #### Related Training #### Related Issue Tracker IDs ## How do I tune Background Database Verification in DS (All versions) and OpenDJ 3.5.2, 3.5.3? The purpose of this article is to provide information on tuning Background Database Verification in DS/OpenDJ. The JE 7 database periodically performs automatic database verification, which can impact service availability. #### Overview Version 7 of JE introduced a feature that verifies the entire database at midnight every night to check for some common types of database corruption. The extra disk I/O can result in problems for performance-critical deployments. All the servers in a single timezone will run the verification at exactly the same time, which can also cause more widespread issues. The JE 7 embedded database has been used in backends since OpenDJ 3.5.2. ##### Note Recovering from any database corruption would typically involve reinitializing the replica directory server or restoring it from a backup taken before the corruption occurred. See FAQ: Backup and restore in DS/OpenDJ for further information. See the following sections for information on configuring the JE verifier depending on version: In OpenDJ 3.5.2 and OpenDJ 3.5.3, you could also use the PDB database for backends; the functionality described here does not apply if you use the PDB backend. The PDB backend type was deprecated in DS 5 and removed in DS 6. #### DS 6 and later DS 6 and later provides advanced properties for configuring the JE verifier (db-run-log-verifier and db-log-verifier-schedule), which allow you to: • Configure the JE verifier schedule • Disable the JE verifier See Configuration Reference › Advanced Properties for further information. Configuring the JE verifier schedule By default, the JE verifier runs automatically at midnight local time. It is best practice to alter the verification schedule on each server to avoid all servers simultaneously using extra disk I/O and potentially impacting the entire service. The verifier schedule is a cron-style field and can be set using the dsconfig command. The following example sets the verifier for the userRoot backend to run at 1am (local time): $ ./dsconfig set-backend-prop --backend-name userRoot --set db-run-log-verifier:true --set 'db-log-verifier-schedule:0 1 * * *' --hostname ds1.example.com --port 4444 --bindDn "cn=Directory Manager" --bindPassword password --trustAll --no-prompt


You must restart the DS server to apply these changes.

Disabling the JE verifier

The JE verifier can be completely disabled, which means that some types of database corruption will not be reported.

You can use the dsconfig command to disable verification on a single backend. For example, to disable it on the userRoot backend:

$./dsconfig set-backend-prop --backend-name userRoot --set db-run-log-verifier:false --hostname ds1.example.com --port 4444 --bindDn "cn=Directory Manager" --bindPassword password --trustAll --no-prompt  You must restart the DS server to apply these changes. #### Pre-DS 6 In earlier versions, you can configure the JE verifier in one of two ways: • Configure the JE verifier schedule • Disable the JE verifier Configuring the JE verifier schedule By default, the JE verifier runs automatically at midnight local time. It is best practice to alter the verification schedule on each server to avoid all servers simultaneously using extra disk I/O and potentially impacting the entire service. The verifier schedule is a cron-style field and can be set using the dsconfig command. The following example sets the verifier for the userRoot backend to run at 1am (local time): $ ./dsconfig set-backend-prop --backend-name userRoot --set 'je-property:je.env.verifySchedule:0 1 * * *' --hostname ds1.example.com --port 4444 --bindDn "cn=Directory Manager" --bindPassword password --trustAll --no-prompt


You must restart the DS/OpenDJ server to apply these changes.

Disabling the JE verifier

The JE verifier can be completely disabled, which means that some types of database corruption will not be reported.

You can use the dsconfig command to disable verification on a single backend. For example, to disable it on the userRoot backend:

$./dsconfig set-backend-prop --backend-name userRoot --set je-property:je.env.runVerifier=false --hostname ds1.example.com --port 4444 --bindDn "cn=Directory Manager" --bindPassword password --trustAll --no-prompt You must restart the DS/OpenDJ server to apply these changes. #### See Also #### Related Training N/A #### Related Issue Tracker IDs ## How do I improve performance when using the PBKDF2 storage scheme in DS/OpenDJ (All versions)? The purpose of this article is to provide information on improving performance when using the PBKDF2 storage scheme in DS/OpenDJ, including how and why you might change the number of iterations performed. The default number of iterations is 10,000. #### Overview PBKDF2 is designed to be flexible in the number of iterations used for computing the hash. This is intended so that it can be extended over time to keep up to date with current CPU speeds, and so that an implementation can be tailored to specific hardware or security requirements. Requirements are also likely to be different depending on whether hashes are calculated client-side or server-side for authentication attempts. The higher the number of iterations, the longer it takes to compute a hash from a plain text password, and therefore the harder it becomes to run brute force attacks against a known hash. You can improve performance as follows: • Upgrade to OpenDJ 2.6.4 or later / DS, which improves the concurrency of the PBKDF2 password storage scheme. • Tune the number of iterations used for the storage scheme; this is described in detail below. • Use the Bouncy Castle JCE provider, which improves the PBKDF2 authentication rate; in tests, we have found this to improve authentication rates by about 30%. This is a third-party library for Java® which contains additional algorithms and also faster versions of the algorithms already in Java. An overview of the setup required is provided below. ##### Note Bouncy Castle is a third-party open source library that is not supported by ForgeRock. Our performance tests We've made some comparisons between Java 7 and Bouncy Castle. Bouncy Castle's performance over Java 7 showed a marked improvement. BIND etimes with 100000 PBKDF2 iterations with the Java 7 JCE: 300ms BIND etimes with 100000 PBKDF2 iterations with Bouncy Castle v1.52: 200ms BIND etimes for fewer iterations are proportionally faster with Bouncy Castle. #### Tuning PBKDF2 performance In a scenario where DS/OpenDJ (the server) receives passwords in plain text and needs to run the algorithm itself, the benefits of having a large number of iterations should be balanced with not overwhelming the server. To tune this trade-off of security versus performance, you can alter the number of iterations used for the storage scheme using dsconfig in interactive mode, see: dsconfig: Password Storage Scheme > View and edit an existing... > PBKDF2 > pbkdf2-iterations The change won't affect currently stored passwords. The new iteration value will be used when passwords are next changed. At this time, they'll be rehashed with the updated number of iterations. The number of iterations is prepended to the final stored password hash so different users with different levels of PBKDF2 hashed passwords can co-exist simultaneously. #### Setting up Bouncy Castle The following process provides an overview of installing Bouncy Castle: 1. Download the latest bcprov-ext-jdk15on-xxx.jar and bcprov-jdk15on-xxx.jar files from Bouncy Castle; they are listed in the SIGNED JAR FILES section. 2. Copy these two jar files to a directory that the JVM searches; the JVM searches the$JAVA_HOME/jre/lib/ext/ directory by default, so this is a good place to put them.
3. Ensure the file permissions for these two jar files are set to allow them to be read.
4. Update the list of security providers in the JVM to put Bouncy Castle first and then renumber the other security providers to follow. This list is set in the java.security text file (located in the $JAVA_HOME/jre/lib/security/ directory). The security provider list should now look similar to this: security.provider.1=org.bouncycastle.jce.provider.BouncyCastleProvider security.provider.2=sun.security.provider.Sun [...] This step is recommended by Bouncy Castle and you can read more about it here: The Legion of the Bouncy Castle - Specifications. 5. Save this file and restart DS/OpenDJ. You can check the installation has been successful using a command such pfiles or lsof on DS/OpenDJ and look for these two bcprov jar files. #### See Also #### Related Training N/A #### Related Issue Tracker IDs ## How do I know what index types are needed for search filters in DS/OpenDJ (All versions)? The purpose of this article is to provide information on what index types are needed for search filters in DS/OpenDJ. #### Index types needed for search filters A search with single-level or subtree scope will need indexes. The index types required depend on all the terms in the search filter and are illustrated in the following table (the uid attribute is used as an example): Filter Term Index Required (uid=*) uid presence (uid=m*) uid substring (uid=*m) uid substring (uid=*m*) uid substring (uid=m*m) uid substring (uid>=10) uid ordering (uid<=10) uid ordering (uid=m) uid equality (uid~=doe) uid approximate Further information about the different index types, with examples, is provided in Administration Guide › Indexing Attribute Values › Index Types and Their Functions. #### See Also #### Related Training #### Related Issue Tracker IDs N/A ## How do I enable Garbage Collector (GC) Logging for DS/OpenDJ (All versions)? The purpose of this article is to provide information on enabling GC Logging for DS/OpenDJ. It assumes that you already have a working DS/OpenDJ server that is installed. #### Enabling GC Logging ##### Note You should ensure there is enough disk space for the GC logs; if not, you should enable GC log rotation as well by including the following options when you enable GC logging: -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=n, -XX:GCLogFileSize=n You can enable GC logging as follows: 1. Locate and edit your /path/to/ds/config/java.properties file for the DS/OpenDJ instance for which you want to enable GC Logging. 2. Append the following options to the entry start-ds.java-args= -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename] replacing [filename] with the path to the file that you would like to create to store the log file. For example: start-ds.java-args=-server -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:/tmp/gc.log 3. Pre-DS 5 only: Execute the /path/to/ds/bin/dsjavaproperties program (or the /path/to/ds/bat/dsjavaproperties.bat program if you are using Microsoft® Windows®). You should get the following output: The operation was successful. The server commands will use the java arguments and java home specified in the properties file located in /path/to/ds/config/java.properties.  The dsjavaproperties command has been removed in DS 5. 4. Restart the DS/OpenDJ server. Once the server has successfully restarted, there should be a GC log file located in the directory specified in the -Xloggc: option. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock. #### See Also #### Related Training #### Related Issue Tracker IDs N/A ## Monitoring ## How do I create a dedicated user for monitoring in DS 6.x? The purpose of this article is to provide assistance if you need to create a user for DS monitoring. #### Overview When installing DS 6 and later, you are given the option to create a default user for querying monitoring information (see Installation Guide › To Set Up a Directory Server for further information). If you did not create a user at this stage, you can use one of the following options to create one post install: Once you have created your user/account, you can then enable the appropriate connection handler to expose the required monitoring endpoints: After completing these steps, you will be ready to start monitoring using your user/account per Administration Guide › Monitoring, Logging, and Alerts #### Granting monitoring rights to an existing account To grant access to an existing user: 1. Create an LDIF file to apply the ds-privilege-name: monitor-read to the user's entry, for example: $ cat monitor-read.ldif
dn: uid=jdoe,ou=People,dc=example,dc=com
changetype: modify
ds-privilege-name: monitor-read
2. Apply the changes using the following ldapmodify command:
$./ldapmodify --port 1389 --bindDN "cn=Directory Manager" --bindPassword password monitor-read.ldif 3. Verify the changes are applied, for example: $ ./ldapsearch --port 1389 --bindDN "uid=jdoe,ou=People,dc=example,dc=com" --bindPassword password --baseDN cn=monitor "(ds-cfg-backend-id=userRoot)"

objectClass: top
objectClass: ds-monitor
objectClass: ds-monitor-backend
objectClass: ds-monitor-backend-pluggable
objectClass: ds-monitor-backend-db
ds-mon-backend-is-private: false
ds-mon-backend-entry-count: 2002
ds-mon-backend-writability-mode: enabled
ds-mon-backend-ttl-is-running: false
ds-mon-backend-ttl-last-run-time: 20180809151924.210Z
ds-mon-backend-ttl-queue-size: 0
ds-mon-backend-ttl-entries-deleted: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000}
ds-mon-backend-filter-use-start-time: 19700101000000Z
ds-mon-backend-filter-use-indexed: 0
ds-mon-backend-filter-use-unindexed: 0
ds-mon-db-version: 7.5.11
ds-mon-db-cache-evict-internal-nodes-count: 0
ds-mon-db-cache-evict-leaf-nodes-count: 0
ds-mon-db-cache-total-tries-internal-nodes: 1045
ds-mon-db-cache-total-tries-leaf-nodes: 882
ds-mon-db-cache-misses-internal-nodes: 12
ds-mon-db-cache-misses-leaf-nodes: 70
ds-mon-db-cache-size-active: 3230029
ds-mon-db-log-size-active: 4604407
ds-mon-db-log-cleaner-file-deletion-count: 0
ds-mon-db-log-utilization-min: 56
ds-mon-db-log-utilization-max: 56
ds-mon-db-log-size-total: 4604407
ds-mon-db-log-files-open: 1
ds-mon-db-log-files-opened: 5
ds-mon-db-checkpoint-count: 0
ds-cfg-backend-id: userRoot


#### Creating a dedicated monitoring account

To create a dedicated user, you can set up an LDIF backend and apply the privileges (which is what DS does during install if this option is selected):

1. Create a directory for the user in the /path/to/ds/db directory (this is where the special users are created during install), for example:
$mkdir /path/to/ds/db/monitorUser  2. Create an encoded password for the monitoring user, for example: $ ./encode-password -c password -s SSHA512
{SSHA512}NUhN2CulFulVLDDJo+6uZ2NhjSpkl2iFn2sRNgFvjnZM/0LddL3hXPAecLTALCYnKfE+64lXiWwBfPvgYJR+0fL1ojGvsruE
3. Create an LDIF file that contains the monitoring user entry including the encoded password from step 2, for example:
$cat /path/to/ds/db/monitorUser/monitorUser.ldif dn: uid=Monitor objectClass: top objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson sn: User cn: Monitor ds-privilege-name: monitor-read userPassword: {SSHA512}NUhN2CulFulVLDDJo+6uZ2NhjSpkl2iFn2sRNgFvjnZM/0LddL3hXPAecLTALCYnKfE+64lXiWwBfPvgYJR+0fL1ojGvsruE  4. Create an LDIF backend: $ ./dsconfig create-backend --set base-dn:uid=Monitor --set enabled:true --set ldif-file:/path/to/ds/db/monitorUser/monitorUser.ldif --type ldif --backend-name monitorUser --port 4444 --bindDn "cn=Directory Manager" --bindPassword password --trustAll --no-prompt

$./ldapsearch --port 1389 --bindDN "uid=Monitor" --bindPassword password --baseDN cn=monitor "(ds-cfg-backend-id=userRoot)" objectClass: top objectClass: ds-monitor objectClass: ds-monitor-backend objectClass: ds-monitor-backend-pluggable objectClass: ds-monitor-backend-db ds-mon-backend-is-private: false ds-mon-backend-entry-count: 2002 ds-mon-backend-writability-mode: enabled ds-mon-backend-degraded-index-count: 0 ds-mon-backend-ttl-is-running: false ds-mon-backend-ttl-last-run-time: 20180809151924.210Z ds-mon-backend-ttl-thread-count: 0 ds-mon-backend-ttl-queue-size: 0 ds-mon-backend-ttl-entries-deleted: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000} ds-mon-backend-filter-use-start-time: 19700101000000Z ds-mon-backend-filter-use-indexed: 0 ds-mon-backend-filter-use-unindexed: 0 ds-mon-db-version: 7.5.11 ds-mon-db-cache-evict-internal-nodes-count: 0 ds-mon-db-cache-evict-leaf-nodes-count: 0 ds-mon-db-cache-total-tries-internal-nodes: 1045 ds-mon-db-cache-total-tries-leaf-nodes: 882 ds-mon-db-cache-misses-internal-nodes: 12 ds-mon-db-cache-misses-leaf-nodes: 70 ds-mon-db-cache-size-active: 3230029 ds-mon-db-log-size-active: 4604407 ds-mon-db-log-cleaner-file-deletion-count: 0 ds-mon-db-log-utilization-min: 56 ds-mon-db-log-utilization-max: 56 ds-mon-db-log-size-total: 4604407 ds-mon-db-log-files-open: 1 ds-mon-db-log-files-opened: 5 ds-mon-db-checkpoint-count: 0 ds-cfg-backend-id: userRoot #### See Also #### Related Training N/A #### Related Issue Tracker IDs N/A ## FAQ: Monitoring DS/OpenDJ The purpose of this FAQ is to provide answers to commonly asked questions regarding monitoring in DS/OpenDJ. #### Frequently asked questions #### Q. How can I check that DS/OpenDJ is up and running? A. You can perform a heartbeat check against DS/OpenDJ to verify that the server is up and running. See How do I perform a heartbeat check against DS/OpenDJ (All versions)?for further information. #### Q. How can I check if a backend is online? A. You can check if a backend is online by performing a ldapsearch against the specific backend. See How do I check if a backend is online in DS/OpenDJ (All versions)? for further information. #### Q. How can I monitor replication? A. You can use the dsreplication status command to give you an overall view of the replication topology, including whether the servers are synchronized. For example: $ ./dsreplication status --adminUID admin --adminPassword password --hostname ds1.example.com --port 4444 --trustAll

See How do I troubleshoot replication issues in DS/OpenDJ (All versions)? and Reference › dsreplication for further information.

##### Note

The M.C.and A.O.M.C. metrics returned from the dsreplication status command are deprecated in DS 6 and replaced with replication delay in DS 6.5. You should monitor replication delay instead; you can also monitor this over LDAP and HTTP as detailed in Administration Guide › Monitoring Replication Delay Over LDAP and Administration Guide › Monitoring Replication Delay Over HTTP.

#### Q. How can I monitor performance?

A. You can monitor performance by performing a ldapsearch against cn=monitor to return operation statistics.

See How do I use cn=monitor entry in DS/OpenDJ (All versions) for monitoring? for further information on monitoring operation statistics.

#### Q. What can I monitor with the cn=monitor entry?

A. DS/OpenDJ exposes a lot of different monitoring information over LDAP under this entry.

See How do I use cn=monitor entry in DS/OpenDJ (All versions) for monitoring? for further information, including specific examples you may want to monitor.

#### Q. Do the statistics under the cn=monitor entry persist when the server is restarted?

A. No, the statistics under cn=monitor are reset each time the server is restarted.

#### Q. Do I have to use the Directory Manager user for JMX monitoring?

A. No, you can use any user for JMX monitoring; you just need to add the JMX privileges (jmx-notify, jmx-read, and jmx-write) to the user you want to access JMX monitoring. No users have access to JMX monitoring by default.

See Administration Guide › JMX-Based Monitoring for an example of how to add these privileges to a user (Directory Manager, but you can substitute any user you want).

#### Q. How can I connect to JMX to ensure mbeans are returned?

A. You must be authenticated to the DS/OpenDJ server via a JMX client to see the mbeans with associated attributes and operations. Authenticating to the server is the only way to expose the sensitive elements within the mbeans; connecting to the process directly will not show them. Additionally, you must ensure the user who authenticates has JMX privileges.

#### Q. Can I change the default listen-address for the JMX Connection Handler?

A. Yes, as of OpenDJ 2.6.2 you can change the default listen-address. The default listen-address for the JMX Connection Handler is 0.0.0.0.

#### Q. What URL should I use in JConsole to log in?

A. You must always use a remote URL, even if you are using a local connection to the JVM. For example:

service:jmx:rmi:///jndi/rmi://localhost:1689/org.opends.server.protocols.jmx.client-unknown

You can substitute a hostname alias (such as ds1.example.com) or an IP address for the localhost part of this URL.

##### Note

If SSL is enabled, you must set up the truststore on the system where JConsole is running and configure SSL properly to monitor a remote application. See Using JConsole for further information.

## How do I use cn=monitor entry in DS/OpenDJ (All versions) for monitoring?

The purpose of this article is to provide information on using the cn=monitor entry in DS/OpenDJ for monitoring purposes. DS/OpenDJ exposes monitoring information over LDAP under this entry.

#### Overview

You can perform a ldapsearch against the cn=monitor entry and sub-entries to provide a variety of statistics that are useful for monitoring DS/OpenDJ.

Some key baseDNs to search are:

Version baseDN Details
All cn=monitor Provides general server information (an example of the type of information returned is shown below).
All cn=Disk Space Monitor,cn=monitor Provides information about disks, including disk location and space available.
All cn=Work Queue,cn=monitor Provides information about the work queue, including its backlog, average and max backlog.
DS 6 and later cn=jvm,cn=monitor

Provides information about the system and the JVM, including memory usage.

Replaces cn=System Information,cn=monitor and cn=JVM Memory Usage,cn=monitor

DS 6 and later cn=LDAP,cn=connection handlers,cn=monitor

Provides information about all open client connections.

Replaces cn=Client Connections,cn=monitor

DS 6 and later

ds-cfg-backend-id=userRoot,cn=Backends,cn=monitor

Provides monitoring information about the Berkeley DB Java Edition backend.

Replaces cn=userRoot Database Environment,cn=monitor

Pre-DS 6 cn=System Information,cn=monitor Provides information about the system and the JVM.
Pre-DS 6 cn=JVM Memory Usage,cn=monitor Provides information about memory usage in the JVM.
Pre-DS 6 cn=Client Connections,cn=monitor Provides information about all open client connections.
Pre-DS 6 cn=userRoot Database Environment,cn=monitor Provides monitoring information about the Berkeley DB Java Edition backend.

Examples

See the following sections for useful monitoring examples:

##### Note

All the examples use the standard non-secure port (389). You should adjust this according to your environment; in particular, if you are using production mode or have the LDAPS Connection Handler enabled, you should use --port 1636 --useSsl --trustAll instead.

#### Using the cn=monitor entry

##### Note

Returning all monitoring information using the below command is a great way to initially find the LDAP metrics and corresponding LDAP attributes that you’re looking to measure; you can then construct specific searches once you know the baseDN and attributes you're interested in.

You can return all monitoring information using a command such as:

$./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=monitor" --searchScope sub "(objectClass=*)" \* An example output in DS 6 looks like this: dn: cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-server objectClass: extensibleObject ds-mon-product-name: ForgeRock Directory Services ds-mon-short-name: OpenDJ ds-mon-vendor-name: ForgeRock AS. ds-mon-full-version: ForgeRock Directory Services 6.0.0 ds-mon-compact-version: OpenDJ-6.0.0 ds-mon-current-connections: 3 ds-mon-max-connections: 3 ds-mon-total-connections: 17 ... dn: cn=Administration Connector,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-connection-handler objectClass: ds-monitor-ldap-connection-handler ds-mon-config-dn: cn=Administration Connector,cn=config ds-mon-protocol: LDAPS ds-mon-listen-address: 0.0.0.0:6444 ds-mon-active-connections-count: 0 ds-mon-connections: {"count":8,"total":8.000,"mean_rate":0.001,"m1_rate":0.000,"m5_r ate":0.000,"m15_rate":0.000} ds-mon-bytes-read: {"count":474,"total":100877.000,"mean_rate":18.642,"m1_rate":0.00 0,"m5_rate":0.000,"m15_rate":0.676} ds-mon-bytes-written: {"count":1181,"total":444547.000,"mean_rate":82.151,"m1_rate": 0.000,"m5_rate":0.001,"m15_rate":3.112} ds-mon-active-persistent-searches: 0 ds-mon-abandoned-requests: 0 ds-mon-requests-abandon: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000, "m5_rate":0.000,"m15_rate":0.000,"mean":0.000,"min":0.000,"max":0.000,"stddev":0.000 ,"p50":0.000,"p75":0.000,"p95":0.000,"p98":0.000,"p99":0.000,"p999":0.000,"p9999":0. 000,"p99999":0.000} ds-mon-requests-add: {"count":3,"total":50.000,"mean_rate":0.001,"m1_rate":0.000,"m5 _rate":0.000,"m15_rate":0.000,"mean":16.707,"min":0.999,"max":37.224,"stddev":15.102 ,"p50":12.059,"p75":12.059,"p95":37.224,"p98":37.224,"p99":37.224,"p999":37.224,"p99 99":37.224,"p99999":37.224} ds-mon-requests-bind: {"count":7,"total":773.000,"mean_rate":0.001,"m1_rate":0.000," m5_rate":0.000,"m15_rate":0.000,"mean":110.414,"min":10.945,"max":272.630,"stddev":7 8.111,"p50":76.022,"p75":126.353,"p95":272.630,"p98":272.630,"p99":272.630,"p999":27 2.630,"p9999":272.630,"p99999":272.630} ...  Alternatively, you can perform more specific searches against individual baseDNs and include particular objectClass parameters and/or attributes to filter the information returned. #### Monitoring replication You can monitor replication for each Directory server and Replication server that the server searched knows about using a command similar to the following: $ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=Replication,cn=monitor" --searchScope sub "(objectClass=*)" \*

Specifically, you would want to check the following attributes (depending on version), which can signify issues with replication if they do not equal to 0:

• DS 6 and later: ds-mon-current-delay
• Pre-DS 6:
• missing-changes
• approximate-delay

#### Monitoring operation statistics

You can monitor operation statistics using a command similar to the following:

• DS 6 and later:
$./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=monitor" --searchScope sub "(objectClass=ds-monitor-ldap-connection-handler)" \*  • Pre-DS 6: $ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=monitor" --searchScope sub "(objectClass=ds-connectionhandler-statistics-monitor-entry)" \*

This command returns three sets of statistics:

• Overall read/write statistics (count of messages, bytes etc).
• Number of requests and responses per operation.
• Performance metrics - total counts of each operation finished and the total execution time of these operations.

This final set of statistics can be really useful for monitoring performance. You can also determine average operation times by querying this on a regular basis and calculating the differences from the previous query. Both the average operations per second and etime per operation can be derived.

#### Monitoring the work queue

You can monitor the work queue using a command similar to the following:


• Pre-DS 6:
$./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=monitor" --searchScope sub "(objectClass=ds-backend-monitor-entry)" \* This command will return database size details for each backend, where: • ds-mon-backend-entry-count (ds-backend-entry-count in pre-DS 6) shows the total number of entries in the the backends. • ds-mon-base-dn-entry-count (ds-base-dn-entry-count in pre-DS 6) shows the total number of entries per base DN. #### Monitoring active users You can monitor users who are currently connected to the DS/OpenDJ server using a command similar to the following: • DS 6 and later: $ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=monitor" --searchScope sub "(objectClass=ds-monitor-connection*)" \*

• Pre-DS 6:
$./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=monitor" --searchScope sub "(&(objectClass=ds-monitor-entry)(connection=*))" \* This command searches the connection handlers and returns the connection attribute. An example output in DS 6 looks like this: dn: cn=Administration Connector,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-connection-handler objectClass: ds-monitor-ldap-connection-handler ds-mon-config-dn: cn=Administration Connector,cn=config ds-mon-protocol: LDAPS ds-mon-listen-address: 0.0.0.0:4444 ds-mon-active-connections-count: 0 ds-mon-connections: {"count":6,"total":6.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000} ds-mon-bytes-read: {"count":117,"total":25811.000,"mean_rate":0.084,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000} ds-mon-bytes-written: {"count":241,"total":155863.000,"mean_rate":0.508,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000} ds-mon-active-persistent-searches: 0 ds-mon-abandoned-requests: 0 ... dn: cn=LDAP,cn=connection handlers,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-connection-handler objectClass: ds-monitor-ldap-connection-handler ds-mon-config-dn: cn=LDAP,cn=connection handlers,cn=config ds-mon-protocol: LDAP ds-mon-listen-address: 0.0.0.0:389 ds-mon-connection: {"connID":14,"connectTime":"20190117102734Z","source":"127.0.0.1:36914","destination":"127.0.0.1:389","ldapVersion":3,"authDN":"cn=Directory Manager","ssf":0,"opsInProgress":0,"persistentSearches":0} ds-mon-connection: {"connID":16,"connectTime":"20190117102934Z","source":"127.0.0.1:36920","destination":"127.0.0.1:389","ldapVersion":3,"authDN":"cn=Directory Manager","ssf":0,"opsInProgress":0,"persistentSearches":0} ...  #### See Also #### Related Training #### Related Issue Tracker IDs ## How do I perform a heartbeat check against DS/OpenDJ (All versions)? The purpose of this article is to provide information on performing a heartbeat check against DS/OpenDJ. A heartbeat check allows you to perform a simple health check on the server to ensure it is up and running. #### Overview DS 6.5 and later servers provide health status checks for anonymous requests over HTTP and LDAP. This allows a remote application to check that a server is "alive" and "healthy". See Release Notes › What's New in 6.5 (Monitoring) for further information. In all versions, you can perform a simple health check using heartbeat connections. Heartbeat connections If your load balancer or application is capable of using a heartbeat connection to check if DS/OpenDJ is online and responding, use of a proper LDAP connection is key. A proper heartbeat check should progress as follows to fully verify the status of the DS/OpenDJ server: CONNECT -> BIND -> SEARCH -> UNBIND -> DISCONNECT or CONNECT -> BIND -> UNBIND -> DISCONNECT ##### Caution It is important to disconnect again, else the heartbeat check can remain open and consume server resources #### Performing a heartbeat check You can use a command such as the following to connect, bind, issue a base (scope) level search on the baseDN of "", unbind and finally disconnect: $ ./ldapsearch --port 4444 --bindDN "cn=Directory Manager" --useSsl --trustAll --baseDN "" --searchScope base "(objectClass=*)" 1.1
dn:


Replacing "cn=Directory Manager" with the bind DN of the account used to bind to the Configuration store in AM/OpenAM if different.

The results of the search operation show "dn:" (only). This is expected as the LDAP request attribute "1.1." returns the distinguished name only. In this case, you are searching the "rootDSE" and the expected dn: is "null".

An example entry seen in the access logs for successful heartbeat check is shown below depending on your DS/OpenDJ version:

• DS 5 and later (JSON log format):
{"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"CONNECT","connId":2},"transactionId":"0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS"},"timestamp":"2018-03-15T16:44:49.400Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-1"}
{"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"BIND","connId":2,"msgId":1,"version":"3","dn":"cn=Directory Manager","authType":"SIMPLE"},"transactionId":"5009191b-a09a-4c7f-84cd-e240b2810a67-2","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":3,"elapsedTimeUnits":"MILLISECONDS"},"userId":"cn=Directory Manager","timestamp":"2018-0315T16:44:49.687Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-4"}
{"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"SEARCH","connId":2,"msgId":2,"dn":"","scope":"base","filter":"(objectClass=*)","attrs":["1.1"]},"transactionId":"5009191b-a09a-4c7f-84cd-e240b2810a67-5","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"timestamp":"2018-03-15T16:44:49.717Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-7"}
{"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"UNBIND","connId":2,"msgId":3},"transactionId":"5009191b-a09a-4c7f-84cd-e240b2810a67-8","timestamp":"2018-03-15T16:44:49.729Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-10"}
{"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"DISCONNECT","connId":2},"transactionId":"0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","reason":"Client Unbind"},"timestamp":"2018-03-15T16:44:49.734Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-12"}

• Pre-DS 5 (file-based log format):
[15/Mar/2018:16:44:49 -0600] CONNECT conn=2 from=203.0.113.0:52597 to=203.0.113.0:1389 protocol=LDAP
[15/Mar/2018:16:44:49 -0600] BIND REQ conn=2 op=0 msgID=1 version=3 type=SIMPLE dn="cn=Directory Manager"
[15/Mar/2018:16:44:49 -0600] BIND RES conn=2 op=0 msgID=1 result=0 authDN="cn=Directory Manager,cn=Root DNs,cn=config" etime=3
[15/Mar/2018:16:44:49 -0600] SEARCH REQ conn=2 op=1 msgID=2 base="" scope=baseObject filter="(objectClass=*)" attrs="1.1"
[15/Mar/2018:16:44:49 -0600] SEARCH RES conn=2 op=1 msgID=2 result=0 nentries=1 etime=1
[15/Mar/2018:16:44:49 -0600] UNBIND REQ conn=2 op=2 msgID=3
[15/Mar/2018:16:44:49 -0600] DISCONNECT conn=2 reason="Client Unbind"


Alternately, if your application can perform a Connect with a Bind only, this is a viable alternative:

• DS 5 and later (JSON log format):
{"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"CONNECT","connId":2},"transactionId":"0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS"},"timestamp":"2018-03-15T16:44:49.400Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-1"}
{"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"UNBIND","connId":2,"msgId":3},"transactionId":"5009191b-a09a-4c7f-84cd-e240b2810a67-8","timestamp":"2018-03-15T16:44:49.729Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-10"}
{"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"DISCONNECT","connId":2},"transactionId":"0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","reason":"Client Unbind"},"timestamp":"2018-03-15T16:44:49.734Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-12"}

• Pre-DS 5 (file-based log format):
[15/Mar/2018:16:44:49 -0600] CONNECT conn=2 from=203.0.113.0:52597 to=203.0.113.0:1389 protocol=LDAP
[15/Mar/2018:16:44:49 -0600] BIND REQ conn=2 op=0 msgID=1 version=3 type=SIMPLE dn="cn=Heartbeat Admin"
[15/Mar/2018:16:44:49 -0600] BIND RES conn=2 op=0 msgID=1 result=0 authDN="cn=Heartbeat Admin,cn=Root DNs,cn=config" etime=3
[15/Mar/2018:16:44:49 -0600] UNBIND REQ conn=2 op=2 msgID=3
[15/Mar/2018:16:44:49 -0600] DISCONNECT conn=2 reason="Client Unbind"
##### Note

You can also perform a ldapsearch against cn=monitor for a complete status of the DS/OpenDJ server as detailed in How do I use cn=monitor entry in DS/OpenDJ (All versions) for monitoring? or you could use the Connections.newHeartBeatConnectionFactory() method as detailed in Developer's Guide › Health Check Connections if your load balancer can use a Java based application.

N/A

N/A

## How do I check if a backend is online in DS/OpenDJ (All versions)?

#### Checking if a backend is online (DS 6 and later)

You can perform a ldapsearch against the backend to check it is online. This example performs a check against the userRoot backend:

$./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "ds-cfg-backend-id=userRoot,cn=backends,cn=monitor" --searchScope base "(objectClass=*)"  Online backend If the backend is online, you would get a response such as the following: dn: ds-cfg-backend-id=userRoot,cn=backends,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-backend objectClass: ds-monitor-backend-pluggable objectClass: ds-monitor-backend-db ds-mon-backend-is-private: false ds-mon-backend-entry-count: 1511 ds-mon-backend-writability-mode: enabled ds-mon-backend-degraded-index-count: 0 ds-mon-backend-ttl-is-running: false ds-mon-backend-ttl-last-run-time: 20190131162956.795Z ds-mon-backend-ttl-thread-count: 0 ds-mon-backend-ttl-queue-size: 0 ds-mon-backend-ttl-entries-deleted: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000} ds-mon-backend-filter-use-start-time: 19700101000000Z ds-mon-backend-filter-use-indexed: 0 ds-mon-backend-filter-use-unindexed: 0 ds-mon-db-version: 7.5.11 ds-mon-db-cache-evict-internal-nodes-count: 0 ds-mon-db-cache-evict-leaf-nodes-count: 0 ds-mon-db-cache-total-tries-internal-nodes: 1709 ds-mon-db-cache-total-tries-leaf-nodes: 1382 ds-mon-db-cache-misses-internal-nodes: 31 ds-mon-db-cache-misses-leaf-nodes: 438 ds-mon-db-cache-size-active: 5042101 ds-mon-db-log-size-active: 7550013 ds-mon-db-log-cleaner-file-deletion-count: 0 ds-mon-db-log-utilization-min: 34 ds-mon-db-log-utilization-max: 34 ds-mon-db-log-size-total: 7550013 ds-mon-db-log-files-open: 1 ds-mon-db-log-files-opened: 3 ds-mon-db-checkpoint-count: 0 ds-cfg-backend-id: userRoot  Offline backend If the backend is offline, you would get a response such as the following: # The LDAP search request failed: 32 (No Such Entry) # Additional Information: Entry ds-cfg-backend-id=userRoot,cn=backends,cn=monitor does not exist in the "monitor" backend # Matched DN: cn=backends,cn=monitor  #### Checking if a backend is online (Pre-DS 6) You can perform a ldapsearch against the backend to check it is online. This example performs a check against the userRoot backend: $ ./ldapsearch --port 389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "cn=userRoot Backend,cn=monitor" --searchScope sub "(objectClass=*)"

Online backend

If the backend is online, you would get a response such as the following:

dn: cn=userRoot Backend,cn=monitor
objectClass: top
objectClass: ds-monitor-entry
objectClass: ds-backend-monitor-entry
ds-backend-is-private: FALSE
ds-backend-writability-mode: enabled
cn: userRoot Backend
ds-backend-entry-count: 200002
ds-base-dn-entry-count: 200002 dc=forgerock,dc=com
ds-backend-id: userRoot
ds-backend-base-dn: dc=forgerock,dc=com

Offline backend

If the backend is offline, you would get a response such as the following:

SEARCH operation failed
Result Code:  32 (No Such Entry)
Additional Information:  Entry cn=userRoot Backend,cn=monitor does not exist in the memory-based backend
Matched DN:  cn=monitor

You would also see a corresponding error in the DS/OpenDJ log, for example:

[24/Mar/2015:14:41:59 -0600] category=CONFIG severity=SEVERE_ERROR msgID=3407988 msg=An error occurred while trying to initialize a backend loaded from class org.opends.server.backends.jeb.BackendImpl with the information in configuration entry ds-cfg-backend-id=userRoot,cn=Backends,cn=config:  The database environment could not be opened: (JE 5.0.73) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.73) /Users/jdoe/Software/db/userRoot com.sleepycat.je.log.ChecksumException: Incomplete log entry header, size=0 lsn=0x10/0x67b292 LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x10/0x67b292 parent IN=3 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x1a/0x9136a5 lastLoggedVersion=0x1f/0xc74a1 parent.getDirty()=false state=0 (BackendImpl.java:1754 BackendImpl.java:319 BackendConfigManager.java:1298 BackendConfigManager.java:279 DirectoryServer.java:2210 DirectoryServer.java:1397 DirectoryServer.java:9651).  This backend will be disabled

N/A

## FAQ: IDM/OpenIDM performance and tuning

The purpose of this FAQ is to provide answers to commonly asked questions regarding performance and tuning for IDM/OpenIDM.

#### Q. Are there any recommendations for sizing IDM/OpenIDM servers?

A. No, the performance of IDM/OpenIDM depends entirely on your specific environment and the exact nature of scripted customizations. To establish appropriate sizing, you will need to perform adequate benchmark testing.

See Release Notes › Fulfilling Memory Requirements for further information.

##### Note

Sizing and/or tuning recommendations are outside the scope of ForgeRock support; if you want more tailored advice, consider engaging Deployment Support Services.

#### Q. Is there any best practice advice for benchmark testing?

A. ForgeRock recommends the following:

• Maintain a staging environment that matches production where you can simulate the load you will have in production. This allows you to resolve any issues you identify in a non-production environment.
• Establish a simple benchmark prior to adding external resources; you can then incrementally add resources and processes to establish what impact each one has.

#### Q. What is the recommended Java Virtual Machines (JVM) heap size for IDM/OpenIDM?

A. There are no definitive rules for the size of JVM heap size required as it will vary across individual environments and applications, but you should refer to Best practice for JVM Tuning for best practice advice. Additionally, ensure you configure JVM garbage collection appropriately as GC times can increase with large heap sizes causing significant impacts to application performance and CPU utilization. See How do I change the JVM heap size for IDM/OpenIDM (All versions)? for further information.

You can monitor JVM memory usage using one of the health endpoints as detailed in Integrator's Guide › Memory Health Check.

##### Note

For a 32-bit JVM or a 32-bit operating system, the limit for the process size is 4GB, that is, 2^32; this cannot be exceeded regardless of the amount of heap space allocated.

#### Q. How can I troubleshoot performance issues?

A. If you have no benchmark tests for comparison and encounter performance issues, the recommendation is to reduce your system to the bare minimum and then incrementally add back resources and processes in order to identify which one is causing the bottleneck.

The following tips should also help:

#### Q. How can I improve reconciliation performance?

A. Firstly, you should identify reconciliation performance issues per the advice in How do I identify reconciliation performance issues in IDM/OpenIDM (All versions)?; this article also offers tuning advice. In addition to this, you can:

#### Q. Are there any recommendations for sizing the database needed for the IDM/OpenIDM repository?

A. Database sizing for the IDM/OpenIDM repository depends on various factors such as the number of managed objects (users, groups, roles etc), relationships, links, audit logs, configurations, workflow, reconciliations etc. Since all these factors vary from one deployment to another, it is not possible to give specific recommendations. However, the following guidelines should help when estimating the size of database required:

Managed objects

You can calculate the size of your IDM/OpenIDM deployment by first calculating the size of your database for sample data (such as 20 users) and then multiplying that by the expected number of users. The following tables typically grow as more managed objects are added:

• managedobjects - one entry per managed object.
• managedobjectproperties - N entries per managed object, where N is the number of indexed / searchable properties: Integrator's Guide › Creating and Modifying Managed Object Types.
• links - the total number of links is less than or equal to the number of managed objects multiplied by the number of unique mappings. For example, if you have 20 managed users and 3 mappings (systemLdapAccounts_managedUser, managedUser_systemLdapAccounts and systemADAccount_managedUser), the total number of links would be less than or equal to 40 (20 * 2). The systemLdapAccounts_managedUser and managedUser_systemLdapAccounts mappings are bidirectional syncs and may use the same links: Integrator's Guide › Mapping Source Objects to Target Objects.
• relationships, relationshipproperties - relationships are especially sensitive to growth when searchablebydefault is set to true (the various repo.jdbc.json files provided in the /path/to/idm/db directory have different defaults so you may end up generating more of this data that you need or expect). Additionally, roles utilize the relationships table as of OpenIDM 4.
• Activiti - these tables can grow over time as running workflows result in persisting data in these tables.

Most of the other tables such as configobjects, internaluser etc are static once you have a working IDM/OpenIDM configuration and should not have much impact on database sizing.

Audit logs

Audit logs can have a huge impact on database sizing and disk space. By default, IDM/OpenIDM stores audit logs in both CSV files and in the database. The size of audit logs depends on IDM/OpenIDM usage. There are different types of audit logs and their corresponding events: Integrator's Guide › Audit Event Topics. The following tables in particular grow with each reconciliation:

• recon/audit - this table reports the status for each user and grows by one entry for each source record. Additionally, a further entry is created for each target record that is not linked or correlated (CONFIRMED, FOUND). The size of a single reconciliation report depends on the data size of source and target; the overall data requirement depends on how many reports you keep. You can reduce the size of the recon/audit table for actions you're not interested in by setting the action to NOREPORT.
• recon/activity - this table reports all activity and grows by one entry for each modifying action regardless of the source of the action, for example, reconciliation, synchronization etc. The overall size depends on how many changes are processed and how long these records are kept.

The following best practices should be heeded:

#### Q. How can I manage the QueuedThreadPool in Jetty?

A. You can change the Jetty thread pool settings by updating the config.properties file (located in the /path/to/idm/conf directory) or by setting the OPENIDM_OPTS environment variable when you start IDM/OpenIDM. See Integrator's Guide › Adjusting Jetty Thread Settings for further information.

N/A

## How do I change the JVM heap size for IDM/OpenIDM (All versions)?

#### Changing the JVM heap size

Changing the JVM heap size can improve performance and reduce the time it takes to run reconciliations. You should try different heap sizes to see what impact it has to determine the best heap size for your setup.

You can set the JVM heap size via the OPENIDM_OPTS environment variable. If OPENIDM_OPTS is undefined, the JVM maximum heap size defaults to 1GB. For example, to set the minimum and maximum heap sizes to 2GB, you would enter the following prior to starting IDM/OpenIDM:

On Unix® and Linux® systems:

$cd /path/to/idm/$ export OPENIDM_OPTS="-Xms2048m -Xmx2048m"
$./startup.sh On Microsoft® Windows® systems: C:\> cd \path\to\idm C:\path\to\idm> set OPENIDM_OPTS=-Xms2048m -Xmx2048m C:\path\to\idm> startup.bat  ##### Note You can also edit the startup.sh or startup.bat files to update the default OPENIDM_OPTS values. ##### Note It is recommended that you set the minimum and maximum heap sizes to the same value for best performance. Otherwise, the JVM runs a full Garbage Collector (GC) cycle when increasing its heap, during which time it can pause ongoing operations for up to a few seconds. Generally, a smaller heap will increase the frequency at which the JVM GC executes but reduce the duration; similarly, a larger heap will reduce the frequency and increase the duration. When tuning the JVM heap, the goal is to strike a balance between frequency and duration so as to reduce the impact of the GC on the application. #### See Also #### Related Training N/A #### Related Issue Tracker IDs N/A ## How do I identify reconciliation performance issues in IDM/OpenIDM (All versions)? The purpose of this article is to provide guidance on isolating and diagnosing IDM/OpenIDM reconciliation performance issues. This article also includes tips on tuning reconciliation performance based on your findings. #### Overview Isolating the cause(s) of performance bottlenecks within a large IDM/OpenIDM topology comes down to a process of elimination. Identifying bottlenecks requires the system to be broken down to its individual components; each component must then be tested and tuned to achieve the necessary throughput. For example, given the following basic topology there are multiple components which might limit the throughput of the system: Although far from exhaustive, the following are some of the possible bottlenecks which might be encountered within the above topology: • Source LDAP Read ops/s • Target LDAP Read/Write ops/s • PostgreSQL Read/Write ops/s • JVM Heap limitations • Host Disk I/O limitations • Host CPU limitations • Undersized / incorrectly configured virtualization • IDM/OpenIDM configuration issues • IDM/OpenIDM triggers/custom code • IDM/OpenIDM product defect ##### Note One configuration change that can help with reconciliation performance is to increase the number of connector instances in the connection pool as detailed in How do I configure pooled connections for a connector in IDM/OpenIDM (All versions)? #### Reconciliation performance Measuring the performance of individual components within the topology can be done outside the scope of IDM/OpenIDM by using external benchmarking tools and processes or via IDM/OpenIDM itself. Benchmarking components via IDM/OpenIDM consists of using targeted operations and configuration changes to isolate a single component within the system and measure its performance as it relates to reconciliation. You should consider the following components: #### External source system During reconciliation, IDM/OpenIDM queries the source system in order to obtain the complete list of source object IDs to be reconciled. It may also perform multiple queries to retrieve the complete source object during the reconciliation process or rely on a cache of the source objects returned by the configured sourceQuery. To begin to understand the performance of the source system, the following can be performed: • LDAP searchrate command to measure raw read ops capability. This command is provided in the OpenDJ LDAP Toolkit. • Measure execution time via targeted queries against the source system using IDM/OpenIDM REST APIs: • Execute the generic ‘query-all-ids’ query against the source system. • Execute any ‘sourceQuery’ queries which have been configured. • Execute individual source object queries via Query FIlter, specifying the individual source object attributes use within the configured ‘sourceQuery’ queries. Based on the results of the above actions, tune the source system to resolve performance bottlenecks. Examples of things which might reduce read performance on a source LDAP system are: • Lack of necessary indexes, resulting in un-indexed (full DB scans) searches. • Insufficient JVM Heap. • Insufficient CPU resources to handle the load. • Insufficient Disk I/O performance (non-SSD drives) to handle the load. #### External target system During reconciliation, IDM/OpenIDM queries the target system in order to obtain the complete list of target object IDs to be reconciled. It may also perform multiple queries to retrieve the complete target object during the reconciliation process or rely on a cache of the target objects returned by the configured ‘targetQuery’. Target objects calculated based on the configured mappings are then written to the target system. To begin to understand the performance of the target system, the following can be performed: • LDAP searchrate and modrate commands to measure raw read/write ops capability. These commands are provided in the OpenDJ LDAP Toolkit. • Measure execution time via targeted queries against the target system using IDM/OpenIDM REST APIs: • Execute the generic ‘query-all-ids’ query against the target system. • Execute any ‘targetQuery’ queries which have been configured. • Execute any ‘correlationQuery’ queries which have been configured. • Execute individual target object queries via Query FIlter, specifying the individual target object attributes use within both the configured ‘targetQuery’ and ‘correlationQuery’ queries. Based on the results of the above actions, tune the target system to resolve performance bottlenecks. Examples of things which might reduce write performance on a target LDAP system are: • Excessive indexes, resulting in increased overhead when writing entries. Specifically take note that substring indexes are extremely costly and may greatly reduce performance. • Insufficient JVM Heap. • Insufficient CPU resources to handle the load. • Insufficient Disk I/O performance (non-SSD drives) to handle the load. #### JDBC repository In order to isolate the JDBC repository from the performance of the reconciliation engine or the target system, the following can be performed: Read/Write Test Mapping: LDAP Source -> Managed Object Situation Action ABSENT CREATE (Default) Read Test Mapping: Managed Object -> LDAP Target Situation Action ALL ASYNC (Read Only) #### Reconciliation engine ##### Note Measuring the performance of the Reconciliation Engine should be done only after having resolved any performance issues observed during testing of the source and target systems. The IDM/OpenIDM reconciliation engine is responsible for the bidirectional synchronization of objects between different data stores. In the case of our topology above, this is the synchronization of the objects between the source and target LDAP systems. During reconciliation IDM/OpenIDM determines the state of the source and target objects (situation assessment), identifies the necessary actions to be performed and performs the actions against the source and target systems. Included within each phase of the reconciliation process are various JavaScript triggers which may or may not impact the performance of the reconciliation process depending on their efficiency. In order to isolate the reconciliation engine from the performance of the IDM/OpenIDM repository or the target system, the following can be performed: Read-Only Test Mapping: LDAP Source -> LDAP Target Situation Action ALL ASYNC (Read Only) ##### Note The following should be performed with a source system which has been populated with production-like data and an EMPTY target system. 1. Set the actions associated with all of the situation policies to ‘Read-Only’ via the IDM/OpenIDM Admin UI. This ensures that during the reconciliation process no actions are performed and that nothing is written to the IDM/OpenIDM repository, or the target system. 2. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging. Read-Write Test Mapping: LDAP Source -> LDAP Target Situation Action ABSENT CREATE (Default) 1. Set the action associated with the ABSENT situation policy to CREATE. Set all other situation policies to ‘Read Only’. This ensures that during the reconciliation process objects are created and links are established between the source and target objects. No other actions are performed. 2. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging. Link Test Mapping: LDAP Source -> LDAP Target Situation Action FOUND LINK (Custom) 1. Ensure you have defined your desired ‘correlationQuery’ within your mapping. 2. Delete the contents of the ‘links’ table within the IDM/OpenIDM repository. 3. Set the action associated with the FOUND situation policy to LINK. Set all other situation policies to ‘Read Only’. This ensures that during the reconciliation process the ‘correlationQuery’ will be executed to correlate the existing target objects (created during the read-write test) to source objects. For each correlated object a new link will be written to the IDM repository. 4. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging. Source Phase Mapping: LDAP Source -> LDAP Target Situation Action CONFIRMED Update (Default) 1. Set the ‘runTargetPhase’ property within the mapping to ‘false’. This ensures that ONLY the source phase of the reconciliation process is executed. 2. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging. Target Phase Mapping: LDAP Source -> LDAP Target Situation Action CONFIRMED Update (Default) 1. Delete the contents of the ‘links’ table within the IDM/OpenIDM repository. 2. Set the ‘sourceQuery’ within the mapping to point to a single specific source object. The goal is to reduce the scope of the source objects processed by the reconciliation engine down to a single object. 3. Set the ‘runTargetPhase’ property within the mapping to ‘true’. Ensure that either you do NOT define a ‘targetQuery’ or that your ‘targetQuery’ is set to your desired value within production. Do not set the ‘targetQuery’ to point to a single object as is done with the ‘sourceQuery’. This ensures that all of the target objects will be processed by the target phase of the reconciliation. 4. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging. #### See Also #### Related Training N/A #### Related Issue Tracker IDs N/A ## How do I configure the connection pool for the managed repository in IDM/OpenIDM (All versions)? The purpose of this article is to provide information on configuring the JDBC connection pool for the managed repository in IDM/OpenIDM. These configurations apply when you are using a JDBC database. #### Overview IDM 5 and later IDM 5 and later uses the Hikari connection pool by default. See Integrator's Guide › Managing the Repository › Understanding the JDBC Connection Configuration File and Hikari Project Page for further information on configuring the connection pool settings. Pre-IDM 5 Pre-IDM 5 uses the BoneCP connection pool to manage connections to the managed repository. The 0.7.1 version of the BoneCP connection pool is bundled with OpenIDM. You can configure the connection pool settings by changing the values of the default settings in this connection section as required and by adding additional BoneCP properties. All the properties you can configure and/or add are described in BoneCP 0.7.1 - BoneCPConfig.java. Partition properties are particularly useful; these can be used to group connections in the connection pool to minimize connection locks and improve performance. For example, you can add the following 3 partition properties: • partitionCount - this property defines the number of connection groups. By default, this property is set to 1 but you can increase it. The BoneCP configuration documents suggest a setting of 2-4, depending on your application. Other people suggest setting this to the number of cores, since this is a good indicator of how many threads can concurrently access the connection pool. In some cases where the threads are very short-lived, a higher partitionCount may yield better performance. You should try different settings to see what setting is most appropriate to your environment. • minConnectionsPerPartition - this property defines the number of connections immediately allocated per partition when the connection pool is created. • maxConnectionsPerPartition - this property defines the maximum number of connections per partition. The total number of connections available is therefore maxConnectionsPerPartition x partitionCount, although the number of connections initially created is determined by the minConnectionsPerPartition setting. See the following sections for examples according to which release of OpenIDM you are using; the connection pool settings are managed in different files. #### Configuring the connection pool in OpenIDM 4.x The connection pool is always enabled in OpenIDM 4.x. Example The following example shows the datasource.jdbc-default.json file (located in the /path/to/idm/conf directory) for a MySQL™ database with the additional partition properties added: { "driverClass" : "com.mysql.jdbc.Driver", "jdbcUrl" : "jdbc:mysql://localhost:3306/openidm?allowMultiQueries=true&characterEncoding=utf8", "databaseName" : "openidm", "username" : "openidm", "password" : "openidm", "connectionTimeout" : 30000, "connectionPool" : { "type" : "bonecp", "partitionCount" : 3, "minConnectionsPerPartition" : 4, "maxConnectionsPerPartition" : 7 } }  These partition settings would give you a total of 21 connections with 12 being created initially. #### Configuring the connection pool in OpenIDM 3.x You can check the connection pool is enabled by looking for (or adding if it's missing), the following line in the connection section of the repo.jdbc.json file (located in the /path/to/idm/conf directory): "enableConnectionPool" : true,​ Example The following example shows the repo.jdbc.json file for an Oracle® database with the additional partition properties added: { "connection" : { "dbType" : "ORACLE", "jndiName" : "", "driverClass" : "oracle.jdbc.OracleDriver", "jdbcUrl" : "jdbc:oracle:thin:@//HOSTNAME:PORT/DEFAULTCATALOG", "username" : "openidm", "password" : "password", "defaultCatalog" : "openidm", "maxBatchSize" : 100, "maxTxRetry" : 5, "enableConnectionPool" : true, "connectionTimeoutInMs" : 30000, "partitionCount" : 3, "minConnectionsPerPartition" : 4, "maxConnectionsPerPartition" : 7 },  These partition settings would give you a total of 21 connections with 12 being created initially. #### See Also #### Related Training N/A #### Related Issue Tracker IDs ## How do I configure pooled connections for a connector in IDM/OpenIDM (All versions)? The purpose of this article is to provide information on configuring pooled connections for a connector in IDM/OpenIDM. This information applies to all poolable connectors (such as the LDAP connector). Poolable connectors create another instance of the connection if one is not available when it needs to do an operation or if it needs to do multiple operations in parallel. Increasing the number of connector instances in the connection pool can improve reconciliation performance. #### Configuring pooled connections You can configure the settings used for pooled connections in your provisioner configuration file (for example, provisioner.openicf-ldap.json), which is located in the /path/to/idm/conf directory. This file has a poolConfigOption section, with default settings as follows:  "poolConfigOption" : { "maxObjects" : 10, "maxIdle" : 10, "maxWait" : 150000, "minEvictableIdleTimeMillis" : 120000, "minIdle" : 1 },  You can amend these settings as needed to configure how the pooled connections are used, where: Setting Description maxObjects The maximum number of instances permitted in the pool (total active and idle connections). maxIdle The maximum number of idle instances allowed in the pool. maxWait The maximum time (in milliseconds) that the pool waits for an instance to become available before timing out. minEvictableIdleTimeMillis The minimum time (in milliseconds) that an instance must be idle for before it is removed. The default 120000ms equals a 2 minutes idle timeout period. minIdle The minimum number of idle instances allowed in the pool. Once the idle timeout is reached, idle instances will be removed from the pool to bring the number of instances available down to the minIdle value. For example, if you set minIdle to 0, the pool would be emptied of idle instances when the idle timeout is reached. You should bear the following in mind when adjusting these settings: • You should ensure maxObjects and maxIdle are always set to the same value to allow pooling to work efficiently and prevent excessive CPU usage. Failure to use the same value will cause excessive churn within the connector instance pool and cause an excessive number of new connections to be established. • You can increase maxObjects and maxIdle to increase the number of connector instances available. You should experiment to determine the best settings for your setup; a good starting point is to double these values to 20. • You should also increase the number of recontask threads as the number of connector instances increase. You can do this by increasing the value of the taskThreads property in the sync.json file to a value other than the default of 10. See Integrator's Guide › Synchronizing Data Between Resources › Parallel Reconciliation Threads for further information. #### See Also #### Related Training #### Related Issue Tracker IDs N/A ## How do I enable Garbage Collector (GC) Logging for IDM/OpenIDM (All versions)? The purpose of this article is to provide information on enabling GC Logging for IDM/OpenIDM. It assumes that you already have a working IDM/OpenIDM server that is installed. #### Enabling GC Logging You should enable GC logging via the OPENIDM_OPTS environment variable. You can either add them prior to starting IDM/OpenIDM as demonstrated below or you can edit the startup.sh or startup.bat files to update the default OPENIDM_OPTS values. ##### Note You should ensure there is enough disk space for the GC logs; if not, you should enable GC log rotation as well by including the following options when you enable GC logging: -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=n, -XX:GCLogFileSize=n. On Unix® and Linux® systems: Enter the following prior to starting IDM/OpenIDM to enable GC logging: $ cd /path/to/idm/
$export OPENIDM_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]"$ ./startup.sh

replacing [filename] with the path to the file that you would like to create to store the log file. The Xloggc property is optional; if omitted, the GC data is written to the openidm.log file (located in the /path/to/idm/logs directory). This can be useful when trying to correlate GC events with events occurring within the server instance.

Once IDM/OpenIDM has started, there should be a GC log file located in the directory specified in the -Xloggc: option or GC data included in the openidm.log file. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

On Microsoft® Windows® systems:

Enter the following prior to starting IDM/OpenIDM to enable GC logging:

C:\> cd \path\to\idm
C:\path\to\idm> set OPENIDM_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]
C:\path\to\idm> startup.bat


replacing [filename] with the path to the file that you would like to create to store the log file. The Xloggc property is optional; if omitted, the GC data is written to the openidm.log file (located in the \path\to\idm\logs directory). This can be useful when trying to correlate GC events with events occurring within the server instance.

Once IDM/OpenIDM has started, there should be a GC log file located in the directory specified in the -Xloggc: option or GC data included in the openidm.log file. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

N/A

N/A

## How do I perform an anonymous health check on the IDM/OpenIDM server (All versions)?

The purpose of this article is to provide information on performing an anonymous health check on the IDM/OpenIDM server. Anonymous access may be required if you have an application or load balancer that needs to check the availability of the IDM/OpenIDM instance but you don't want to provide any user credentials.

#### Performing anonymous health checks

Basic health checks can be performed against the info/ping endpoint to check that IDM/OpenIDM is available.

To perform an anonymous health check, you must specifically include the following headers in your rest call to identify yourself as anonymous, else the call will fail:

X-OpenIDM-Password: anonymous
X-OpenIDM-Username: anonymous

The anonymous user (internal/role/openidm-reg role) has access to the info/ping endpoint by default.

Example

To perform an anonymous health check, you would use a REST call such as the following:

$curl -H "X-OpenIDM-Username: anonymous" -H "X-OpenIDM-Password: anonymous" -X GET "http://idm.example.com:8080/openidm/info/ping" Example response: { "_id" : "", "state" : "ACTIVE_READY", "shortDesc" : "OpenIDM ready" } #### See Also #### Related Training N/A #### Related Issue Tracker IDs ## How do I monitor IDM 5 and OpenIDM 3.x, 4.x using SmartEvent? The purpose of this article is to provide guidance on monitoring IDM/OpenIDM using SmartEvent. SmartEvent is replaced in IDM 5.5 with DropWizard Metrics. #### Enabling SmartEvent IDM/OpenIDM includes an event monitoring facility called SmartEvent. You can enable SmartEvent by using the openidm.smartevent.enabled system property. If you want to view a summary or statistics in the openidm log, you can enable openidm.smartevent.summarylogging. To get started with SmartEvent, enable it via JAVA_OPTS during startup as follows: $ env JAVA_OPTS="-Dopenidm.smartevent.enabled=true -Dopenidm.smartevent.summarylogging=true" ./startup.sh

The SmartEvent summary information is then output to the openidm.log.0 file, for example:

Jun 16, 2016 14:22:51 PM org.forgerock.openidm.smartevent.core.StatisticsHandler$1 run INFO: Summary: {openidm/internal/repo/orientdb/raw/query/query-all-ids=Invocations: 1 total time: 18.448 ms mean: 18.448 ms, openidm/internal/repo/orientdb/raw/query/query-cluster-events=Invocations: 119 total time: 31.958 ms mean: 0.269 ms, openidm/internal/repo/orientdb/raw/query/query-cluster-failed-instances=Invocations: 119 total time: 54.032 ms mean: 0.454 ms, openidm/internal/router/repo/scheduler/read=Invocations: 26 total time: 98.177 ms mean: 3.776 ms} IDM/OpenIDM SmartEvent MBeans are also available locally by attaching a monitoring tool such as JConsole or VisualVM. These tools allow you to attach to a local process. If you wish to attach remotely, see the following section. #### Accessing SmartEvent remotely using JMX To access SmartEvent remotely using JMX, you will need to specify the following JAVA_OPTS when starting IDM/OpenIDM: ​$ env JAVA_OPTS="-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dopenidm.smartevent.enabled=true
-Dopenidm.smartevent.summarylogging=false" ./startup.sh 

Attach a monitoring tool to a remote process in the form <hostname>:<port>.

## How do I change the JVM heap size for IG/OpenIG (All versions)?

#### Changing the JVM heap size

Changing the JVM heap size can improve performance and reduce the time it takes to process HTTP requests and responses. You should try different heap sizes to see what impact it has to determine the best heap size for your setup. The information given here is specific to the Apache Tomcat™ web container; you should make similar changes in the configuration file specific to your web container if you use a different one.

##### Note

It is recommended that you set the minimum and maximum heap sizes to the same value for best performance. Otherwise, the JVM runs a full Garbage Collector (GC) cycle when increasing its heap, during which time it can pause ongoing operations for up to a few seconds. Generally, a smaller heap will increase the frequency at which the JVM GC executes but reduce the duration; similarly, a larger heap will reduce the frequency and increase the duration. When tuning the JVM heap, the goal is to strike a balance between frequency and duration so as to reduce the impact of the GC on the application.

On Unix® and Linux® systems:

You can set the JVM heap size by specifying CATALINA_OPTS settings in the setenv.sh file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.sh file (also typically located in the /tomcat/bin/ directory).

For example, to set the minimum and maximum heap sizes to 2GB, you would add the following line to the setenv.sh file and restart the web container:

export CATALINA_OPTS="$CATALINA_OPTS -Xms2048m -Xmx2048m" On Microsoft® Windows® systems: Providing you haven't installed Tomcat as a service, you can set the JVM heap size by specifying CATALINA_OPTS settings in the setenv.bat file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.bat file (also typically located in the /tomcat/bin/ directory). For example, to set the minimum and maximum heap sizes to 2GB, you would add the following line to the setenv.bat file and restart the web container: set CATALINA_OPTS=-Xms2048m -Xmx2048m If you have installed Tomcat as a service on Windows, then you must use the GUI application to configure Tomcat services. This process refers to Tomcat 7, but will work for other versions by changing tomcat7w.exe in the command to match your version. 1. Stop the Tomcat service. 2. Navigate to \path\to\tomcat\bin\ from the command prompt: 3. Enter the following command to display Tomcat Properties: tomcat7w.exe //ES//serviceName Where serviceName is the name of your Tomcat service. 4. Navigate to the Java tab and complete the memory pool fields as follows: Initial memory pool: 2048 Maximum memory pool: 2048 5. Restart the Tomcat service. Alternatively, Windows system administrators may prefer to configure these options in the registry so that they may be configured via group policy. The initial memory pool and maximum memory pool values can be configured in the JvmMS and JvmMX properties under the following registry key for Tomcat 7: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\Tomcat7\Parameters\Java #### See Also #### Related Training N/A #### Related Issue Tracker IDs N/A ## How do I enable Garbage Collector (GC) Logging for IG/OpenIG (All versions)? The purpose of this article is to provide information on enabling GC Logging for IG/OpenIG. It assumes that you already have a working IG/OpenIG server installed. #### Enabling GC Logging The information given here is specific to the Apache Tomcat™ web container; you should make similar changes in the configuration file specific to your web container if you use a different one. ##### Note You should ensure there is enough disk space for the GC logs; if not, you should enable GC log rotation as well by including the following options when you enable GC logging: -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=n, -XX:GCLogFileSize=n On Unix® and Linux® systems: ​You should enable GC logging by specifying CATALINA_OPTS settings in the setenv.sh file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.sh file (also typically located in the /tomcat/bin/ directory). To enable GC logging: 1. Add the following line to the setenv.sh file: export CATALINA_OPTS="$CATALINA_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]"
replacing [filename] with the path to the file that you would like to create to store the log file.
2. Restart the web container.

Once the web container has successfully restarted, there should be a GC log file located in the directory specified in the -Xloggc: option. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

On Microsoft® Windows® systems:

​You should enable GC logging by specifying CATALINA_OPTS settings in the setenv.bat file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.bat file (also typically located in the /tomcat/bin/ directory).

To enable GC logging:

1. Add the following line to the setenv.bat file:
set CATALINA_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCCause -Xloggc:[filename]
replacing [filename] with the path to the file that you would like to create to store the log file.
2. Restart the web container.

Once the web container has successfully restarted, there should be a GC log file located in the directory specified in the -Xloggc: option. You can use the Universal GC Log Analyzer to analyze your GC log. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

N/A

N/A

## How do I migrate OpenIG 4 scripts from using blocking APIs to non-blocking APIs?

The purpose of this article is to provide information on migrating existing OpenIG 4 scripts from using blocking APIs to non-blocking APIs. Moving to non-blocking APIs guarantees an efficient use of resources and can prevent deadlock situations. You should also migrate your scripts if you have upgraded to OpenIG 4.5 or later, or are planning to upgrade in the future; otherwise you will likely encounter a "Cannot execute script" error when trying to run an existing script.

#### Overview

There is a known issue where existing scriptable filters and handlers stop working after upgrading to OpenIG 4.5 or later. This issue is caused by a class version conflict in the CatalogManager class, which is used by the xml-resolver-1.2.jar and is required by the HTTPBuilder API. This issue affects all scripts that use the CatalogManager class or libraries that depend on it, for example, the Groovy http-builder library. You can resolve this issue by migrating to CHF supported APIs.

You will see an error similar to the following in your logs when you encounter this issue:

TUE MAY 02 17:02:59 CET 2017 WARNING {ScriptableFilter}/handler/config/filters/0 --- Cannot execute script
TUE MAY 02 17:02:59 CET 2017 WARNING {ScriptableFilter}/handler/config/filters/0 --- java.lang.Exception: java.lang.NoSuchMethodError: org.apache.xml.resolver.CatalogManager.setIgnoreMissingProperties(Z)V
javax.script.ScriptException: java.lang.Exception: java.lang.NoSuchMethodError: org.apache.xml.resolver.CatalogManager.setIgnoreMissingProperties(Z)V
at org.forgerock.openig.script.Script$GroovyImpl.run(Script.java:62) at org.forgerock.openig.script.Script.run(Script.java:245) Caused by: java.lang.Exception: java.lang.NoSuchMethodError: org.apache.xml.resolver.CatalogManager.setIgnoreMissingProperties(Z)V ... 42 more Caused by: java.lang.NoSuchMethodError: org.apache.xml.resolver.CatalogManager.setIgnoreMissingProperties(Z)V at groovyx.net.http.ParserRegistry.<clinit>(ParserRegistry.java:111) at groovyx.net.http.HTTPBuilder.<init>(HTTPBuilder.java:194) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ... 41 more It is strongly recommended that you migrate to CHF supported APIs and use documented variables even if you are not experiencing issues with your scripts. CHF non-blocking APIs guarantee an efficient use of resources and can prevent deadlock situations; they also ensure you can upgrade to OpenIG 4.5 and later without encountering this issue. This article details the following two step migration process: 1. Migrate http-builder APIs to CHF blocking APIs (they are simple to understand and starting using). 2. Migrate to CHF asynchronous APIs. Example script The following example script to log a user out of AM/OpenAM uses the Groovy http-builder library; it fails in OpenIG 4.5 and later because there is a class version conflict on CatalogManager, which is provided by both the AM/OpenAM SAML Fedlet library and the Groovy http-builder library: @Grab('org.codehaus.groovy.modules.http-builder:http-builder:0.7.1') import groovyx.net.http.RESTClient def openAMRESTClient = new RESTClient(openamUrl) // Check if OpenAM session cookie is present if (null != request.cookies['iPlanetDirectoryPro']) { String openAMCookieValue = request.cookies['iPlanetDirectoryPro'][0].value // Perform logout logger.info("iPlanetDirectoryPro cookie found, performing logout") def response = openAMRESTClient.post(path: 'sessions/', query: ['_action': 'logout'], headers: ['iplanetDirectoryPro': openAMCookieValue]) def result = response.getData().get("result") logger.info("OpenAM logout response: " + result) } return next.handle(context, request)  The logic in this script is quite simple: 1. If there is a session token in the request, get it. 2. Call a dedicated AM/OpenAM REST endpoint to revoke the token (which logs the user out). 3. Continue the execution of the chain (after the response has been received). #### Migrating to CHF blocking APIs This step details removing the dependency on the http-builder library and using the built-in CHF APIs instead. The migrated example script below demonstrates the basics of the CHF API: request creation, obtaining a response and reading its content as JSON (without forgetting to release resources). Background In each script’s execution context there is an HTTP binding (Client interface) that provides you with a way to perform HTTP requests from within your script. From the Client API: class Client { Promise<Response, NeverThrowsException> send(Request request); Promise<Response, NeverThrowsException> send(Context context, Request request); } There is only one thing that you can obviously do with this interface, that is, send an HTTP request. You do not get back the response directly, but you have the promise of a response. This is comparable to a Future on which you would attach listeners that get notified when the response has been fully received (or an error has happened). This part of the migration uses the Promise as a Future (blocking on a .get()). In essence, this migration step is all about creating and populating a CHF Request object. Migrated example script The following shows the above example script after it has been migrated to use the built-in CHF APIs: // Check if OpenAM session cookie is present if (null != request.cookies[ 'iPlanetDirectoryPro' ]) { String openAMCookieValue = request.cookies[ 'iPlanetDirectoryPro' ][ 0 ].value logger.info("iPlanetDirectoryPro cookie found, performing logout") def logout = new Request() logout.method = "POST" logout.uri = "${openamUrl}/sessions"
logout.uri.query = "_action=logout"

// Block for at most 20 seconds before using the response
def logoutResponse = http.send(logout)
.get(20, SECONDS)

def result = logoutResponse.entity.json.result
logger.info("OpenAM logout response: " + result)

// Don’t forget to release resources associated with the response
logoutResponse.close()
}

return next.handle(context, request)



The logic in this script is as follows:

1. Request initialization: the following snippet of this script creates a POST request to the AM/OpenAM sessions endpoint (using Groovy String substitution). It specifies the logout action in the query URL component and places the current request’s SSO Token (the iPlanetDirectoryPro cookie) in a header (as this is where the sessions endpoint expects it):
def logout = new Request()
logout.method = "POST"
logout.uri = "${openamUrl}/sessions" logout.uri.query = "_action=logout" logout.headers['iPlanetDirectoryPro'] = openAMCookieValue  2. Perform the request and wait for the response: the following snippet of this script uses the get() method. The returned promise can never throw an exception, which means you do not need to use the getOrThrow() variant of the get() method; if something goes wrong when making the request, you will receive a 502 Bad Gateway response. This example also includes a timeout to prevent deadlocks if there is no response. When the timeout is reached, a RuntimeException is returned; ideally the timeout should be configurable and IG/OpenIG provides the args configuration point in the script configuration, which can be used for this purpose: // Block for at most 20 seconds before using the response def logoutResponse = http.send(logout) .get(20, SECONDS)  3. Read the response's content using the CHF's native JSON support: the following snippet of this script calls logoutResponse.entity.json, which provides the JSON-parsed content of the message, with a focus on the result JSON attribute. The message is then closed to release resources: def result = logoutResponse.entity.json.result // Don’t forget to release resources associated with the response logoutResponse.close()   4. Call the next element of the chain and continue the request processing: return next.handle(context, request)  #### Migrate to CHF asynchronous APIs This step details moving away from synchronous programming (where we spend valuable CPU time waiting for things to happen) to asynchronous programming (where we register callbacks to notify us when things are ready and act when that happens rather than waiting for something; this allows our valuable CPU time to do other things). It also introduces the http.send() returned Promise object, where it’s the promise of a response that may or may not have been received yet. Put simply, we want to read the response content after it has been received, log the result and then continue the chain execution. This step demonstrates adding an asynchronous function that calls the next element in the chain, after the response processing has been done. Migrated example script The following shows the example script after it has been migrated to use the CHF asynchronous APIs: // Check if OpenAM session cookie is present if (null != request.cookies[ 'iPlanetDirectoryPro' ]) { String openAMCookieValue = request.cookies[ 'iPlanetDirectoryPro' ][ 0 ].value logger.info("iPlanetDirectoryPro cookie found, performing logout") def logout = new Request() logout.method = "POST" logout.uri = "${openamUrl}/sessions"
logout.uri.query = "_action=logout"

// Return the "promise" of a result => non-blocking
// Processing will happen when a response from AM will be received
return http.send(logout)
.then { response ->
def result = response.entity.json.result
logger.info("OpenAM logout response: " + result)
response.close()
return response
}
.thenAsync({
next.handle(context, request)
} as AsyncFunction)
}

return next.handle(context, request)



As you can see, most of the changes have occurred towards the end of the script, with the response being processed using callbacks. Additionally, since we no longer block, we do not need a timeout to prevent deadlocks.

The logic in this script is as follows:

1. Process the response when it’s ready: the following snippet of this script introduces a then(Function) method to process this response. The callback Function provided in this then() method accepts the result of the invocation as a parameter, which can be processed as required; this example reuses the response processing logic from before to return the response (a result is expected from Function callbacks):
return http.send(logout)
.then { response ->
def result = response.entity.json.result
logger.info("OpenAM logout response: " + result)
response.close()
return response
}


You must include the first return in this section of the script to ensure the promise that is configured with the response processing function is returned when the logout action is triggered. If you exclude the return, for example, the first line is simply:
http.send(logout)

The script will execute, do an HTTP call, register a callback on the promise and then just continue. This means the logout action will get missed when there is an iPlanetDirectoryPro cookie, which is not the desired behavior of this script.
2. Continue the execution of the chain after triggering the logout action: the following snippet of this script calls the next element of the chain by performing the call in a then-able fashion (chaining different then() methods together). You can use this approach because you will just see the response of the logout action when a request with an iPlanetDirectoryPro cookie enters the filter, rather than the usual response sent back from the protected application:
           .thenAsync({
next.handle(context, request)
} as AsyncFunction)


Groovy specific points

•  There is no return statement, because Groovy considers the last evaluated expression of a method to be the returned value.
• The as AsyncFunction is only required for asynchronous functions (ResultHandler, ExceptionHandler and Function don’t need it). The Groovy compiler doesn’t handle all the generic types of AsyncFunction very well.

#### Further information on the Promise object

There are a number of then-able methods offered by the Promise API that you can try, which will make your code simpler and avoid 'callback hell'. These methods fall into the following categories:

 thenOn****(****Handler) These methods are pure notification functions (on successful result, on error). You can’t return anything from within these methods, they are here as a side-effect-free type of handler. No exceptions should be thrown in theses callbacks. then(Function), with 1, 2 or 3 Function arguments These methods are designed for when you need to return something different to the next handler in the Promise chain (a transformation). They could be parsing an input string and return a JSON, or modifying the result. Throwing exceptions is accepted here and they will be propagated to the next functions in the Promise chain. thenAsync(AsyncFunction), with 1, 2 or 3 AsyncFunction arguments These methods are quite similar to those in then(Function). The real difference is that they have to return a promise of their result instead of the actual result. These methods are an ideal fit when you need to call another method that itself returns a promise (like returning the result of the filter’s chain). then***(), the rest of the then-able methods (thenAlways(), thenFinally(), thenCatch(), ...) These methods are really syntactic sugar to make the promise more fluent to read.

Now you can understand why our first then() can be turned into a thenOnResult() instead, which saves the last return statement as the received parameter is returned:

return http.send(logout)
.thenOnResult { response ->
def result = response.entity.json.result
logger.info("OpenAM logout response: " + result)
response.close()
}
.thenAsync({
next.handle(context, request)
} as AsyncFunction)



N/A

N/A

## How do I check if IG/OpenIG (All versions) is up and running?

#### Checking if IG/OpenIG is up and running

Since most of what IG/OpenIG does is proxy, its health mainly depends on the health of upstream applications for which IG/OpenIG is the proxy.

The following method provides a simple way to check that IG/OpenIG is up and running by establishing that IG/OpenIG can read its configuration and return a result without depending on a downstream application. The static text and URI shown are just examples, you should use values that make sense in the environment where IG/OpenIG is deployed:

1. Create a simple isAlive route. The result should return a 200 status along with some static text; for example, "Server is ALIVE". These results can be used to check on the health of IG/OpenIG.
2. Add a new route to the IG/OpenIG configuration. For example, you could create a 00-isAlive.json file in the $HOME/.openig/config/routes directory, similar to the following: { "handler": { "type": "StaticResponseHandler", "config": { "status": 200, "reason": "Found", "entity": "Server is ALIVE" } }, "condition": "${request.uri.path == '/isAlive.jsp'}"
}

Notice that /openig/ is not used for the route since this is reserved for admin purposes only; instead the /isAlive.jsp URI is used in this example.
3. Configure the load balancer to check the results from this URL, for example, by calling the following URL:
http://openig.example.com:8080/isAlive.jsp
##### Note

If you require more detailed checks, you could make use of a scriptable handler instead: Configuration Reference › Handlers › ScriptableHandler.

N/A

#### Related Issue Tracker IDs

This content has been optimized for printing.