Synchronization in IDM/OpenIDM

This book provides information on synchronization in IDM/OpenIDM including Reconciliation, LiveSync and Implicit synchronization.


Synchronization


How do I synchronize hashed passwords from IDM/OpenIDM (All versions) to DS/OpenDJ (All versions)?

The purpose of this article is to provide information on how to pass pre-hashed passwords values within IDM/OpenIDM to DS/OpenDJ.

Background information

The synchronization model for IDM/OpenIDM is to take the encrypted password from its own store, decrypt it and pass it in plain text to DS/OpenDJ for it to hash and store. IDM/OpenIDM also has the capability to hash passwords not just encrypt them. The way in which the passwords are synchronized will differ depending on whether they are encrypted or hashed:

  • Encrypted: the password is decrypted on the IDM/OpenIDM side using a transform script (defined in sync.json) and then pushed to DS/OpenDJ as plain text. Upon arrival, DS/OpenDJ encrypts or hashes the password.
  • Hashed: the hashed password is pushed directly to DS/OpenDJ since hashes are one way and cannot be decrypted before synchronization. Hashing is different between systems. If you are hashing within IDM/OpenIDM, the SHA256 algorithm uses a 16 byte Salt value whereas hashing within DS/OpenDJ natively only uses an 8 byte Salt value (How does DS/OpenDJ (All versions) store password values?) which means the same hashing cannot be shared. This is desirable from a security point of view but does mean that users won't be able to authenticate against DS/OpenDJ. The following section details how to identify the Salt value needed for storing and synchronizing hashed passwords.
Note

If you want to synchronize hashed passwords the other way, you must use the DS/OpenDJ Password Sync Plugin as there is no way for IDM/OpenIDM to decrypt a key that has been encrypted by DS/OpenDJ. See Password Synchronization Plugin Guide for further information.

Salt values

IDM/OpenIDM performs hashing using a randomly generated Salt value. You can identify this Salt value, if required, by manipulating the hashed object since the data value consists of the base64 encoded version of the Hashed value concatenated with the Salt value.

This example demonstrates retrieving the Salt value used when performing the hashing function:

  1. The following JavaScript® code hashes the password value (test) and then outputs information to the log file to show the base64 decoded version of the data and the Salt value used:
    testOutput = openidm.hash("test","SHA-256");
    logger.info("Hash Output: {}", testOutput);
    
    var base64 = Packages.org.forgerock.util.encode.Base64url
    b64tO = base64.decode(testOutput.$crypto.value.data);
    logger.info("Decoded: {}", b64tO);
    
    salt = b64tO.slice(32);
    logger.info("Salt: {}", salt);
    
    
  2. Example log output; observe that the decoded value (meaning the Salt and the password itself) returns as a byteArray:
    Aug 10, 2018 2:28:34 PM org.forgerock.script.scope.FunctionFactory$1$3 call
    INFO: Hash Output: {$crypto={value={algorithm=SHA-256, data=sDSamS5Qqn1BD3ctddaKJlMQ1XO3QiE5D4cq2laOaFBJPpx58E8xbsAnleJ14r9F}, type=salted-hash}}
    Aug 10, 2018 2:28:34 PM org.forgerock.script.scope.FunctionFactory$1$3 call
    INFO: Decoded: [-80, 52, -102, -103, 46, 80, -86, 125, 65, 15, 119, 45, 117, -42, -118, 38, 83, 16, -43, 115, -73, 66, 33, 57, 15, -121, 42, -38, 86, -114, 104, 80, 73, 62, -100, 121, -16, 79, 49, 110, -64, 39, -107, -30, 117, -30, -65, 69]
    Aug 10, 2018 2:28:34 PM org.forgerock.script.scope.FunctionFactory$1$3 call
    INFO: Salt: [73, 62, -100, 121, -16, 79, 49, 110, -64, 39, -107, -30, 117, -30, -65, 69]

Synchronizing hashed passwords

You can use passwords in DS/OpenDJ that have been hashed by IDM/OpenIDM as follows:

  1. Configure DS/OpenDJ to allow pre-encoded passwords for the relevant password policy. You can set this using dsconfig, for example:
    $ ./dsconfig set-password-policy-prop --policy-name "Default Password Policy" --port 4444 --bindDN "cn=Directory Manager" --bindPassword password --advanced --set allow-pre-encoded-passwords:true --trustAll --no-prompt
  2. Configure IDM/OpenIDM to hash passwords as detailed in Integrator's Guide › Encoding Attribute Values by Using Salted Hash Algorithms. This change does not update all existing user passwords. Passwords will only be stored as hashes if they are created or updated after this change.
  3. Update the transformation of the password in the IDM/OpenIDM sync.json file (located in the /path/to/idm/conf directory)​ to use the base64 encoded hash and append the appropriate hashing algorithm as a prefix. For example, if you are using the SHA256 algorithm, the updated section would look similar to this where {SSHA256} has been appended:
    {
      "source" : "password",
      "condition" : {
        "type" : "text/javascript",
        "source" : "object.password != null"
      },
      "transform" : {
        "type" : "text/javascript",
        "source" : "var hash = \"{SSHA256}\" + source.$crypto.value.data;hash;"
      },
      "target" : "userPassword"
    },
    
  4. Update the password in IDM/OpenIDM, which will cause the hashed value to be pushed to DS/OpenDJ. You will see something similar to the following in the DS/OpenDJ audit log when this happens:
    -
    replace: userPassword
    userPassword: {SSHA256}zRaha9G3jX+ptpLL79oBDU5wcz48/t+Cd2GOBAlAPbO4PIRR5udD6nGjO6OEgfGh
    -
    
  5. Verify DS/OpenDJ is now storing the password hash by viewing the object for the user whose password you updated using ldapsearch, for example​:
    $ ./ldapsearch --port 1389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "dc=example,dc=com" "(uid=jdoe)" --trustAll 
    
    dn: uid=jdoe,ou=People,dc=example,dc=com
    objectClass: person
    objectClass: inetOrgPerson
    objectClass: organizationalPerson
    objectClass: top
    givenName: Jame
    description: Created for OpenIDM
    uid: jdoe
    cn: Jane Doe
    sn: Doe
    telephoneNumber: 1-360-229-7105
    userPassword: {SSHA256}zRaha9G3jX+ptpLL79oBDU5wcz48/t+Cd2GOBAlAPbO4PIRR5udD6nGjO6OEgfGh
    mail: jdoe@example.com
    

See Also

Integrator's Guide › Synchronizing Data Between Resources

How do I change a password storage scheme and apply a new password policy to users in DS/OpenDJ (All versions)?

How do I improve performance when using the PBKDF2 storage scheme in DS/OpenDJ (All versions)?

FAQ: Passwords in DS/OpenDJ

Synchronization in IDM/OpenIDM

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I configure a managed object property to be required and not empty in IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on configuring a managed object property to be required and not empty in IDM/OpenIDM. This configuration is used to prevent null values being permitted.

Configuring a managed object property

To make a managed object property both required and not empty, you can specify both the required and not-empty functions. These functions are available in the default policy script file (policy.js, located in the /path/to/idm/bin/defaults/script directory).

Example

You can add a policies section to the required property in the managed.json file (located in the /path/to/idm/conf directory) with both these functions defined. For example, it would look like this for the city property:

                    "city" : {
                        "type" : "string",
                        "title" : "City",
                        "description" : "City",
                        "viewable" : true,
                        "userEditable" : true,
                        "usageDescription" : "",
                        "isPersonal" : false,
                        "policies" : [
                            {
                                "policyId" : "required",
                                "params" : { }
                            },
                            {
                                "policyId" : "not-empty",
                                "params" : { }
                            }
                        ]
                    },  

This configuration ensures the city property is always present and contains an actual value when creating or updating objects.

See Also

Policy validation failed error when running reconciliation or using a workflow to set or update attributes to null in IDM (All versions) and OpenIDM 4.5

Administering and configuring IDM/OpenIDM

Integrator's Guide › Using Policies to Validate Data

Integrator's Guide › Understanding the Policy Configuration Element

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I configure the scheduler in IDM/OpenIDM (All versions) to ensure JSON tasks are executed correctly when the time changes?

The purpose of this article is to provide information on configuring the scheduler in IDM/OpenIDM to pick up time changes (such as clocks going back an hour due to daylight saving time) to ensure JSON tasks are executed correctly.

Overview

Scheduler timing is handled by the Quartz library in IDM/OpenIDM using the cronTrigger syntax. The Quartz scheduler adds a trigger to the list for the next scheduled task. If the time goes back before the task is executed, the trigger does not get modified and the next scheduled task gets missed as illustrated by this example:

01:59:59 - Schedule fires, Trigger created for 02:00:00
02:00:00 - DST Occurs, Clock set to 01:00:00
01:00:00 - No trigger, no schedule executed
..
01:59:59 - No trigger, no schedule executed
02:00:00 - Trigger found, schedule executed

This is a known shortcoming as described in the Quartz documentation: Quartz - Daylight Saving Time and Triggers. Clocks going forward an hour are handled correctly.

To avoid missing a scheduled event when the clocks go back, you can add the timeZone field to your scheduler (the schedule-<scheduleName>.json file located in the /path/to/idm/conf directory) and set it to UTC. The UTC setting ensures the scheduler uses Coordinated Universal Time, which does not follow a daylight savings schedule and retains the same time zone all year round.

IDM 6

The timeZone field is deprecated in IDM 6; although you can continue to use it at this time, you should specify a time zone for schedules using the startTime and endTime fields instead. For example:

   "startTime" : "2018-01-09T00:00:00.000Z",
   "endTime" : "2018-12-31T00:00:00.000Z",

Where UTC is specified by the addition of the Z.

Configuring the scheduler to pick up time changes

The following worked example shows the time zone being set to UTC and subsequent testing to prove that it handles time changes as expected. 

Note

When re-testing, the Quartz library exhibits some odd behavior if the first invocation of the schedule is within a 1 hour window; to avoid this you should ensure the schedule fires at least once before this time elapses.

  1. Update your schedule-<scheduleName>.json file with the timeZone field, or the startTime and endTime fields. For example, the sample schedules-script.json file (located in the /path/to/idm/samples/schedules directory) looks like this after setting the timeZone field:
    {
        "enabled" : true,
        "type": "cron",
        "schedule": "0/2 * * * * ?",
        "concurrentExecution" : false,
        "invokeService": "script",
        "timeZone" : "UTC",
        "invokeContext": {
            "script" : {
                  "type" : "text/javascript",
                  "source" : "java.lang.System.out.println('It is working: ' + input.edit);",
                  "input": { "edit": 26}
                }
        }
    }
    
  2. Shutdown IDM/OpenIDM in order to start testing:
    $ cd /path/to/idm
    $ ./shutdown.sh
  3. Stop the ntpd Service (or any other auto-time services). Change the system time to a date and time prior to the clocks going back; this should also be prior to the time they will change to. For example in the US, you could use 00:30:00 AM EDT on 4th November 2018 (where the clocks go back at 02:00 AM on the same day). If you use the ntpd Service to sync your system time, you would use these commands:
    $ service ntpd stop
    $ timedatectl set-time "2018-11-04 00:30:00"​
  4. Restart IDM/OpenIDM:
    $ cd /path/to/idm
    $ ./startup.sh
  5. Confirm that the schedule is executing within the IDM/OpenIDM console:
    Executing ./startup.sh...
    Using OPENIDM_HOME:   /opt/openidm
    Using PROJECT_HOME:   /opt/openidm
    Using OPENIDM_OPTS:   -Xmx1024m -Xms1024m
    Using LOGGING_CONFIG: -Djava.util.logging.config.file=/opt/openidm/conf/logging.properties
    Using boot properties at /opt/openidm/conf/boot/boot.properties
    OpenIDM ready
    It is working: 26
    It is working: 26
    It is working: 26
    It is working: 26
    
  6. Restart the ntpd Service (or any other auto-time services). 
    $ service ntpd restart
    
  7. With IDM/OpenIDM still executing, change the system time to a minute before the clocks are due to change (for example, 01:59:00 AM EDT) and monitor. 
    $ timedatectl set-time "2018-11-04 01:59:00"
    
  8. Observe the schedule continuing to execute within the IDM/OpenIDM console:
    It is working: 26
    
    -> This is approximately when the time was changed to "2018-11-04 01:59:00"
    It is working: 26
    It is working: 26
    It is working: 26
    It is working: 26
    It is working: 26
    
    As you can see, the "It is working: 26" lines continue even after the clocks go back to 01:00 AM EST, proving that the schedule continues to work after the time change.

See Also

FAQ: Task Scanner in IDM/OpenIDM

Synchronization in IDM/OpenIDM

Integrator's Guide › Scheduling Tasks and Events

Integrator's Guide › Configuring Schedules

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I configure IDM (All versions) and OpenIDM 4.x to log synchronization data to the audit log?

The purpose of this article is to provide information on configuring IDM/OpenIDM to log synchronization data to the audit log. Auditing data received from a source system via reconciliation and LiveSync can be useful for troubleshooting and debugging purposes.

Overview

You can use the script hooks invoked during sync operations to create log or audit entries, for example, onSync. If you specifically want to send output to the audit logs, you can use openidm.create with either the audit/sync or audit/recon resource; if you do this, you must include the transactionId and timestamp fields (they cannot be null). For example, a very basic script would be:

var content = {};
content.message = "Got here!";
content.transactionId= "yourTransactionId";
content.timestamp = "yourTimestamp"
openidm.create("audit/sync", null, content);

See Integrator's Guide › Variables Available to Scripts for information on what variables are available for the various scripts.

The exact way in which these scripts are implemented vary by version as demonstrated in the following sections:

Outputting sync data to logs in IDM 5 and later

The following example demonstrates how to output sync data to the sync.audit.json file in IDM 5 and later. You should update the onCreate and onUpdate scripts in the mappings from external systems to managed user (or similar) section in the sync.json file (located in the /path/to/idm/conf directory), for example:

            "onCreate" : {
                "type" : "text/javascript",
                "globals" : { },
                "source" : "var content = {};\ncontent.message = \"Create - got here!\";\ncontent.transactionId= \"yourTransactionId\";\ncontent.timestamp = \"yourTimestamp\";\nopenidm.create(\"audit/sync\", null, content);"
            },
            "onUpdate" : {
                "type" : "text/javascript",
                "globals" : { },
                "source" : "var content = {};\ncontent.message = \"Update - got here!\";\ncontent.transactionId= \"yourTransactionId\";\ncontent.timestamp = \"yourTimestamp\";\nopenidm.create(\"audit/sync\", null, content);"
            }

Example output in the sync.audit.json file:

{"source":{"objectClass":["top","inetOrgPerson","organizationalPerson","person"],"kbaInfo":[],"employeeType":null,"mail":"jdoe@example.com","aliasList":[],"ldapGroups":[],"description":null,"sn":"doe","givenName":"john","disabled":null,"telephoneNumber":"12344512123123123","uid":"jdoe","cn":"john","dn":"uid=jdoe,ou=People,dc=example,dc=com","_id":"uid=jdoe,ou=People,dc=example,dc=com"},"message":"Update - got here!","transactionId":"yourTransactionId","timestamp":"yourTimestamp","target":{"displayName":"john","description":null,"givenName":"john","mail":"jdoe@example.com","telephoneNumber":"12344512123123123","sn":"doe","userName":"jdoe","ldapGroups":[],"accountStatus":"active","effectiveRoles":[],"effectiveAssignments":[],"_id":"1d91475b-d34a-4703-96a0-478bac1850b4","_rev":"7","lastSync":{"managedUser_systemLdapAccounts":{"effectiveAssignments":[],"timestamp":"2017-09-15T14:39:41.102+01:00"}},"preferences":{"updates":false,"marketing":false},"password":{"$crypto":{"type":"x-simple-encryption","value":{"cipher":"AES/CBC/PKCS5Padding","salt":"UCfiBDHiCjEazOJEecJBGA==","data":"BKR+yuBkg6NwMJCAHdfDMw==","iv":"Tuqiskep3wpbJApmO/6Y/g==","key":"openidm-sym-default","mac":"ANBABUHS+b+0/+63M2wIJA=="}}}},"_id":"105054a4-395c-4cbb-b8d3-2610fb51e3c2-381"}

Outputting sync data to logs in OpenIDM 4.x

The following example demonstrates how to output sync data to the sync.csv file in OpenIDM 4.x:

  1. Add a new custom audit event topic and CSV event handler to the audit.json file (located in the /path/to/openidm/conf directory). A custom topic is needed as it is not possible to amend the default sync topic to include the source and target. For example, this shows the additions for a new audit event topic called extendedSync and the corresponding CSV event handler definition:
        "eventHandlers" : [
            {
                "class" : "org.forgerock.audit.handlers.csv.CsvAuditEventHandler",
                "config" : {
                    "name" : "csv",
                    "logDirectory" : "&{launcher.working.location}/audit",
                    "topics" : [
                        "access",
                        "activity",
                        "recon",
                        "sync",
                        "authentication",
                        "config",
                        "extendedSync"
                    ]
                }
            },
    
    ...
    
        "eventTopics" : {
        "extendedSync": {
          "schema": {
            "$schema": "http://json-schema.org/draft-04/schema#",
            "id": "/",
            "type": "object",
            "properties": {
              "_id": {
                "type": "string"
              },
              "transactionId": {
                "type": "string"
              },
              "timestamp": {
                "type": "string"
              },
              "eventName": {
                "type": "string"
              },
              "userId": {
                "type": "string"
              },
              "trackingIds": {
                "type": "array",
                "items": {
                  "id": "0",
                  "type": "string"
                }
              },
              "action": {
                "type": "string"
              },
              "exception": {
                "type": "string"
              },
              "linkQualifier": {
                "type": "string"
              },
              "mapping": {
                "type": "string"
              },
              "message": {
                "type": "string"
              },
              "messageDetail": {
                "type": "object",
                "properties": {}
              },
              "situation": {
                "type": "string"
              },
              "sourceObjectId": {
                "type": "string"
              },
              "status": {
                "type": "string"
              },
              "targetObjectId": {
                "type": "string"
              },
              "sourceObject": {
                "type":"object",
                "properties": {}
              },
              "targetObject": {
                "type":"object",
                "properties": {}
              }
            }
          }
        },
    
  2. Update the onCreate and onUpdate scripts in the mappings from external systems to managed user (or similar) section in the sync.json file (located in the /path/to/openidm/conf directory), for example:
              "onCreate" : {
                    "type" : "text/javascript",
                    "globals" : { },
                    "source" : "var content = {};\ncontent.message = \"Create - got here!\";\ncontent.transactionId= \"yourTransactionId\";\ncontent.timestamp = \"yourTimestamp\";\ncontent.sourceObject = source;\ncontent.targetObject = target;\nopenidm.create(\"audit/extendedSync\", null, content);"
                },
                "onUpdate" : {
                    "type" : "text/javascript",
                    "globals" : { },
                    "source" : "var content = {};\ncontent.message = \"Update - got here!\";\ncontent.transactionId= \"yourTransactionId\";\ncontent.timestamp = \"yourTimestamp\";\ncontent.sourceObject = source;\ncontent.targetObject = target;\nopenidm.create(\"audit/extendedSync\", null, content);"
                }
    
    These additions are similar to the ones needed for IDM 5 and later, but also include changes to take account of the custom audit entry that you defined in step 1.
  3. Test a synchronization event and check for a file named extendedSync.csv and verify the expected log entries are shown in the file, for example:
    "8e7a58c5-e2d5-4b99-a2ba-3ca5ece06373-563","yourTransactionId","yourTimestamp",,,,,,,,"Update - got here!",,,,,,"{""sn"":""doe"",""employeeType"":null,""givenName"":""john"",""mail"":""jdoe@example.com"",""uid"":""jdoe"",""telephoneNumber"":""12344512123123123"",""dn"":""uid=jdoe,ou=People,dc=example,dc=com"",""disabled"":null,""cn"":""john"",""description"":null,""ldapGroups"":[],""_id"":""uid=jdoe,ou=People,dc=example,dc=com""}","{""displayName"":""john"",""description"":null,""givenName"":""john"",""mail"":""jdoe@example.com"",""telephoneNumber"":""12344512123123123"",""sn"":""doe"",""userName"":""jdoe"",""accountStatus"":""active"",""effectiveRoles"":[],""effectiveAssignments"":[],""_id"":""590a30f9-5ff8-428d-bb77-3eddd2679281"",""_rev"":""3"",""lastSync"":{""managedUser_systemLdapAccounts"":{""effectiveAssignments"":[],""timestamp"":""2017-12-18T16:08:18.431Z""}}}"
    

See Also

How do I monitor LiveSync activity using REST in IDM/OpenIDM (All versions)?

How do I query individual reconciliation synchronization failures using REST in IDM/OpenIDM (All versions)?

How do I identify reconciliation performance issues in IDM/OpenIDM (All versions)?

Troubleshooting IDM/OpenIDM

Integrator's Guide › Setting Up Audit Logging

Related Training

N/A

Related Issue Tracker IDs

N/A


Known Issues


NameAlreadyBoundException when attempting to sync in IDM/OpenIDM (All versions)

The purpose of this article is to provide assistance if you receive a NameAlreadyBoundException: [LDAP: error code 68 - The entry cannot be added because an entry with that name already exists] when attempting to sync in IDM/OpenIDM.

Symptoms

An error similar to the following is shown in the IDM/OpenIDM log when attempting to sync and the FOUND situation should be actioned but is not, even though both the target and source exists:

Caused by: org.identityconnectors.framework.common.exceptions.AlreadyExistsException: javax.naming.NameAlreadyBoundException: [LDAP: error code 68 - The entry uid=A123456,ou=people,dc=example,dc=com cannot be added because an entry with that name already exists]; remaining name 'uid=A123456,ou=people,dc=example,dc=com' 
   at org.identityconnectors.ldap.modify.LdapCreate.doCreate(LdapCreate.jav a:185) 
   at org.identityconnectors.ldap.modify.LdapCreate.executeImpl(LdapCreate. java:136) 
   at org.identityconnectors.ldap.modify.LdapCreate.execute(LdapCreate.java :80) 
   at org.identityconnectors.ldap.LdapConnector.create(LdapConnector.java:2 09) 
   at org.identityconnectors.framework.impl.api.local.operations.CreateImpl .create(CreateImpl.java:88) 
...

Recent Changes

N/A

Causes

A correlation query is missing from your mappings in the sync.json file (located in the /path/to/idm/conf directory). A correlation query is needed to query the target system for a user who matches the source account; if this is missing, IDM/OpenIDM will identify the situation as MISSING and perform the associated action (CREATE by default).

Solution

This issue can be resolved by adding a correlation query to your mappings in the sync.json file.

Correlation queries are described in more detail in Integrator's Guide › Configuring Synchronization › Correlating Source Objects With Existing Target Objects.

See Also

Integrator's Guide › Configuring Synchronization › Correlating Source Objects With Existing Target Objects

Related Training

N/A

Related Issue Tracker IDs

N/A


The operation attempted to assign a zero-length value to an attribute with the directory string syntax error in IDM/OpenIDM (All versions)

The purpose of this article is to provide assistance if you receive a "The operation attempted to assign a zero-length value to an attribute with the directory string syntax" error in IDM/OpenIDM when syncing data to DS/OpenDJ. This error is Caused by: "javax.naming.directory.InvalidAttributeValueException: [LDAP: error code 21" and causes the synchronization to fail.

Symptoms

Errors similar to the following are shown when the data synchronization to DS/OpenDJ fails, where the attribute causing issues in this example is called Team:

Caused by: org.identityconnectors.framework.common.exceptions.ConnectorException: javax.naming.directory.InvalidAttributeValueException: [LDAP: error code 21 - Entry "employeeId=user10,ou=Users,dc=example,dc=com" contains a value "" for attribute Team that is invalid according to the syntax for that attribute: The operation attempted to assign a zero-length value to an attribute with the directory string syntax]; remaining name 'employeeId=user10,ou=Users,dc=example,dc=com'
Caused by: javax.naming.directory.InvalidAttributeValueException: [LDAP: error code 21 - When attempting to modify entry cn=John,ou=People,dc=example,dc=com to replace the set of values for attribute Team, value "" was found to be invalid according to the associated syntax: The operation attempted to assign a zero-length value to an attribute with the directory string syntax]; remaining name 'cn=John,ou=People,dc=example,dc=com'

This error is not seen if the attribute has a value in IDM/OpenIDM; it only happens if the attribute has an empty value. 

Recent Changes

Configured the LDAP connector to synchronize data to DS/OpenDJ.

Updated the mapping configuration file (sync.json, located in the /path/to/idm/conf directory).

Causes

By default, DS/OpenDJ does not allow empty string values (zero-length-values) for attributes that have a syntax of Directory String. 

The LDAP result code: 21 is caused by invalid attribute syntax. This error is received when the requested operation failed because it violated the syntax for a specified attribute.

Per RFC 4517 - A zero-length character string is not permitted.

Note

Although it is possible to change the schema for the affected attribute to allow zero-length-values, this is not recommended.  You may face issues at any time with other ldap applications that are adhering to the spec.

Solution

You can resolve this issue by adding a transform script to the sync.json file for all attributes that have a syntax of Directory String. For example, the following simple transform script checks if the source property has an empty string value, and if so, sets the target property value to null. If the value is not empty, then the target property value is updated to match the source property value:

{
      "source" : "Team",
      "target" : "Team",
                    "transform" : {
                        "type" : "text/javascript",
                        "source" : "if(source.TEAM === ""){value = null} else{value = source.TEAM}"
},

This transform script has been included in the sync.json file; alternatively, you could include this transform script in a separate file and call the file from sync.json instead, for example:

{
      "source" : "Team",
      "target" : "Team",
                    "transform" : {
                        "type" : "text/javascript",
                        "file" : "script/DirectoryStringTransform.js"
},

See Also

How do I update attributes stored in arrays in IDM/OpenIDM (All versions) using JavaScript?

How do I provision external accounts in a pre-defined order in IDM/OpenIDM (All versions)?

Integrator's Guide › Synchronizing Data Between Resources › Transforming Attributes in a Mapping

Integrator's Guide › Synchronizing Data Between Resources › Configuring Synchronization Between Two Resources

DS 5 › Developer's Guide › Respecting LDAP Schema › Example 5.5. Invalid Attribute Syntax

RFC 4517 > LDAP: Syntaxes and Matching Rules > 3.3.6 > Directory String

Related Training

N/A

Related Issue Tracker IDs

N/A


Reconciliation


How do I exclude specific users from syncing during reconciliation in IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on excluding specific users from syncing during reconciliation in IDM/OpenIDM using the validSource trigger. The validSource trigger applies to the entire mapping and cannot be used for specific actions such as onCreate.

Specifying users to sync

You can specify users to exclude from syncing during reconciliation by using the validSource trigger within the IDM/OpenIDM mapping to evaluate whether the source account (for example, an LDAP account) is valid based on specific criteria. The criteria should be scripted in a way that returns a boolean value true or false. Only users who meet the conditions specified by the validSource trigger (boolean value returned = true) are synced.

You can implement the validSource trigger as follows:

  1. Add the validSource trigger in the sync.json file (located in the /path/to/idm/conf directory). You can either specify the condition to meet or the script to call. For example:
    • This validSource trigger sets a condition that excludes all employees whose employeeId attribute is set to null or blank:
      "validSource": {
                      "type": "text/javascript",
                      "source": "source.employeeId != null && source.employeeId != ''"
                  }
      
    • This validSource trigger calls a script called filterusers.js:
      "validSource": {
                      "type": "text/javascript",
                      "file": "script/filterusers.js"
                  }
      
      If you have specified a script to call, you need to create the script with the same name in the directory specified.

See Also

How do I exclude specific users from syncing during LiveSync in IDM/OpenIDM (All versions)?

Integrator's Guide › Synchronizing Data Between Resources › Filtering Synchronized Objects

Integrator's Guide › Synchronizing Data Between Resources › Mapping Source Objects to Target Objects

Integrator's Guide › Synchronizing Data Between Resources › Synchronization Situations and Actions

Related Training

ForgeRock Identity Management Core Concepts (IDM-400)

Related Issue Tracker IDs

N/A


How do I invoke a reconciliation using the openidm.action function in IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on invoking a reconciliation using the openidm.action function in IDM/OpenIDM.

Invoking a reconciliation using the openidm.action function

The method signature for this function is:

openidm.action(1,2,3,4)

where:

  1. resource path (string)
  2. action name (string)
  3. request content (full complex object support)
  4. request parameters (flat map of strings)

This maps to REST calls via http as follows:

POST 
/openidm/resourcePath?_action=actionName&requestParameter1=val1&requestParameter2=val2
{
  "requesContent1": "val1"
}

Example

If your request to trigger reconciliation from http is:

POST /openidm/recon?_action=recon&mapping=myMappingName

then your call to openidm.action would look like this:

openidm.action("recon", "recon", null, { "mapping": "myMappingName" })

See Also

FAQ: Scripts in IDM/OpenIDM

Integrator's Guide › Scripting Reference › openidm.action(resource, actionName, content, params, fields)

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I query individual reconciliation synchronization failures using REST in IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on querying individual recon sync failures using REST in IDM/OpenIDM. This can be useful when troubleshooting why a reconciliation synchronization has failed.

Querying individual reconciliation synchronization failures

You can use the following REST request to query the details of individual synchronizations that have failed:

$ curl --cacert self-signed.crt -H "X-OpenIDM-Username: openidm-admin" -H "X-OpenIDM-Password: openidm-admin" -X GET 'https://localhost:8443/openidm/audit/recon' -G --data-urlencode '_queryFilter=status+eq+\"FAILURE\"'

Example response (with data removed to only show response summary with resultCount):

{
  "result": [
    {
        ...
    }
  ],
  "resultCount": 6,
  "pagedResultsCookie": null,
  "totalPagedResultsPolicy": "NONE",
  "totalPagedResults": -1,
  "remainingPagedResults": -1
}

Specific reconciliation IDs

If you want to query failed synchronization operations for a specific reconciliation, you can include the actual reconciliation ID. For example, you can use the following REST request:

$ curl --cacert self-signed.crt -H "X-OpenIDM-Username: openidm-admin" -H "X-OpenIDM-Password: openidm-admin" -X GET 'https://localhost:8443/openidm/audit/recon' -G --data-urlencode '_queryFilter=status+eq+\"FAILURE\"&_reconId+eq+\"208154c0-c97b-41d3-bebb-4cf22112c110-138\"'

Example response (which shows a resultCount of 1):

{
  "result": [
    {
      "_id": "208154c0-c97b-41d3-bebb-4cf22112c110-176",
      "_rev": "1",
      "status": "FAILURE",
      "reconId": "208154c0-c97b-41d3-bebb-4cf22112c110-138",
      "transactionId": "208154c0-c97b-41d3-bebb-4cf22112c110-138",
      "timestamp": "2016-08-17T12:50:54.447Z",
      "eventName": "recon",
      "userId": "openidm-admin",
      "action": "CREATE",
      "exception": "org.forgerock.openidm.sync.impl.SynchronizationException: Create rejected as Object with same ID already exists. Cannot index record managed_user{displayName:Morley Waghorne-2,description:null,givenName:Morley,mail:WaghornM@ns-mail3.com,telephoneNumber:+1 818 885-8439,sn:Waghorne,userName:WaghornM,ldapGroups:[0],accountStatus:active,effectiveRoles:[0],effectiveAssignments:[0],_openidm_id:68a9bac8-2994-4f1d-b388-100fc59e5d12}: found duplicated key 'WaghornM' in index 'managed_user!userName!Idx' previously assigned to the record #12:8\n\tat org.forgerock.openidm.sync.impl.ObjectMapping.createTargetObject(ObjectMapping.java:568)\n\tat org.forgerock.openidm.sync.impl.ObjectMapping.access$1400(ObjectMapping.java:69)\n\tat org.forgerock.openidm.sync.impl.ObjectMapping$SyncOperation.performAction(ObjectMapping.java:1776)\n\tat org.forgerock.openidm.sync.impl.ObjectMapping$SourceSyncOperation.sync(ObjectMapping.java:2164)\n\tat org.forgerock.openidm.sync.impl.ObjectMapping$2.recon(ObjectMapping.java:1174)\n\tat org.forgerock.openidm.sync.impl.ObjectMapping$ReconTask.call(ObjectMapping.java:1302)\n\tat org.forgerock.openidm.sync.impl.ObjectMapping$ReconTask.call(ObjectMapping.java:1275)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: org.forgerock.json.resource.PreconditionFailedException: Create rejected as Object with same ID already exists. Cannot index record managed_user{displayName:Morley Waghorne-2,description:null,givenName:Morley,mail:WaghornM@ns-mail3.com,telephoneNumber:+1 818 885-8439,sn:Waghorne,userName:WaghornM,ldapGroups:[0],accountStatus:active,effectiveRoles:[0],effectiveAssignments:[0],_openidm_id:68a9bac8-2994-4f1d-b388-100fc59e5d12}: found duplicated key 'WaghornM' in index 'managed_user!userName!Idx' previously assigned to the record #12:8\n\tat org.forgerock.openidm.repo.orientdb.impl.OrientDBRepoService.create(OrientDBRepoService.java:295)\n\tat org.forgerock.openidm.repo.orientdb.impl.OrientDBRepoService.handleCreate(OrientDBRepoService.java:180)\n\tat org.forgerock.json.resource.Router.handleCreate(Router.java:255)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:69)\n\tat org.forgerock.json.resource.Filters$ConditionalFilter.filterCreate(Filters.java:62)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:67)\n\tat org.forgerock.json.resource.Filters$ConditionalFilter.filterCreate(Filters.java:62)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:67)\n\tat org.forgerock.json.resource.Filters$ConditionalFilter.filterCreate(Filters.java:62)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:67)\n\tat org.forgerock.openidm.audit.filter.AuditFilter.filterCreate(AuditFilter.java:110)\n\tat org.forgerock.json.resource.Filters$ConditionalFilter.filterCreate(Filters.java:60)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:67)\n\tat org.forgerock.openidm.servlet.internal.ServletConnectionFactory$5.filterCreate(ServletConnectionFactory.java:499)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:67)\n\tat org.forgerock.openidm.maintenance.impl.PassthroughFilter.filterCreate(PassthroughFilter.java:48)\n\tat org.forgerock.openidm.maintenance.impl.MaintenanceService.filterCreate(MaintenanceService.java:229)\n\tat org.forgerock.json.resource.Filters$ConditionalFilter.filterCreate(Filters.java:60)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:67)\n\tat org.forgerock.json.resource.FilterChain.handleCreate(FilterChain.java:213)\n\tat org.forgerock.json.resource.InternalConnection.createAsync(InternalConnection.java:44)\n\tat org.forgerock.json.resource.AbstractAsynchronousConnection.create(AbstractAsynchronousConnection.java:44)\n\tat org.forgerock.json.resource.AbstractConnectionWrapper.create(AbstractConnectionWrapper.java:96)\n\tat org.forgerock.openidm.servlet.internal.ServletConnectionFactory$1$1.create(ServletConnectionFactory.java:177)\n\tat org.forgerock.json.resource.AbstractConnectionWrapper.create(AbstractConnectionWrapper.java:96)\n\tat org.forgerock.openidm.managed.ManagedObjectSet.createInstance(ManagedObjectSet.java:650)\n\tat org.forgerock.json.resource.InterfaceCollectionHandler.handleCreate(InterfaceCollectionHandler.java:40)\n\tat org.forgerock.json.resource.Router.handleCreate(Router.java:255)\n\tat org.forgerock.openidm.managed.ManagedObjectService$ManagedObjectSetRequestHandler.handleCreate(ManagedObjectService.java:180)\n\tat org.forgerock.json.resource.Router.handleCreate(Router.java:255)\n\tat org.forgerock.openidm.managed.ManagedObjectService.handleCreate(ManagedObjectService.java:283)\n\tat org.forgerock.json.resource.Router.handleCreate(Router.java:255)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:69)\n\tat org.forgerock.json.resource.Filters$ConditionalFilter.filterCreate(Filters.java:62)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:67)\n\tat org.forgerock.openidm.servlet.internal.ScriptedFilter$2.apply(ScriptedFilter.java:96)\n\tat org.forgerock.openidm.servlet.internal.ScriptedFilter$2.apply(ScriptedFilter.java:93)\n\tat org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:221)\n\tat org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:205)\n\tat org.forgerock.openidm.servlet.internal.ScriptedFilter.filterResourceResponseRequest(ScriptedFilter.java:167)\n\tat org.forgerock.openidm.servlet.internal.ScriptedFilter.filterCreate(ScriptedFilter.java:92)\n\tat org.forgerock.json.resource.Filters$ConditionalFilter.filterCreate(Filters.java:60)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:67)\n\tat org.forgerock.json.resource.Filters$ConditionalFilter.filterCreate(Filters.java:62)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:67)\n\tat org.forgerock.openidm.audit.filter.AuditFilter.filterCreate(AuditFilter.java:110)\n\tat org.forgerock.json.resource.Filters$ConditionalFilter.filterCreate(Filters.java:60)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:67)\n\tat org.forgerock.openidm.servlet.internal.ServletConnectionFactory$5.filterCreate(ServletConnectionFactory.java:499)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:67)\n\tat org.forgerock.openidm.maintenance.impl.PassthroughFilter.filterCreate(PassthroughFilter.java:48)\n\tat org.forgerock.openidm.maintenance.impl.MaintenanceService.filterCreate(MaintenanceService.java:229)\n\tat org.forgerock.json.resource.Filters$ConditionalFilter.filterCreate(Filters.java:60)\n\tat org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:67)\n\tat org.forgerock.json.resource.FilterChain.handleCreate(FilterChain.java:213)\n\tat org.forgerock.json.resource.InternalConnection.createAsync(InternalConnection.java:44)\n\tat org.forgerock.json.resource.AbstractAsynchronousConnection.create(AbstractAsynchronousConnection.java:44)\n\tat org.forgerock.json.resource.AbstractConnectionWrapper.create(AbstractConnectionWrapper.java:96)\n\tat org.forgerock.openidm.servlet.internal.ServletConnectionFactory$1$1.create(ServletConnectionFactory.java:177)\n\tat org.forgerock.json.resource.AbstractConnectionWrapper.create(AbstractConnectionWrapper.java:96)\n\tat org.forgerock.openidm.sync.impl.ObjectMapping.createTargetObject(ObjectMapping.java:561)\n\t... 12 more\nCaused by: com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record managed_user{displayName:Morley Waghorne-2,description:null,givenName:Morley,mail:WaghornM@ns-mail3.com,telephoneNumber:+1 818 885-8439,sn:Waghorne,userName:WaghornM,ldapGroups:[0],accountStatus:active,effectiveRoles:[0],effectiveAssignments:[0],_openidm_id:68a9bac8-2994-4f1d-b388-100fc59e5d12}: found duplicated key 'WaghornM' in index 'managed_user!userName!Idx' previously assigned to the record #12:8 RID=#12:8\n\tat com.orientechnologies.orient.core.index.OIndexTxAwareOneValue.checkEntry(OIndexTxAwareOneValue.java:52)\n\tat com.orientechnologies.orient.core.index.OClassIndexManager.checkIndexedPropertiesOnCreation(OClassIndexManager.java:517)\n\tat com.orientechnologies.orient.core.index.OClassIndexManager.checkIndexes(OClassIndexManager.java:495)\n\tat com.orientechnologies.orient.core.index.OClassIndexManager.onRecordBeforeCreate(OClassIndexManager.java:61)\n\tat com.orientechnologies.orient.core.hook.ODocumentHookAbstract.onTrigger(ODocumentHookAbstract.java:237)\n\tat com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callbackHooks(ODatabaseRecordAbstract.java:1466)\n\tat com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:1103)\n\tat com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:84)\n\tat com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:322)\n\tat com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:40)\n\tat com.orientechnologies.orient.core.record.ORecordAbstract.save(ORecordAbstract.java:335)\n\tat com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1439)\n\tat com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1428)\n\tat com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1417)\n\tat org.forgerock.openidm.repo.orientdb.impl.OrientDBRepoService.create(OrientDBRepoService.java:287)\n\t... 72 more\n",
      "linkQualifier": "default",
      "mapping": "systemLdapAccounts_managedUser",
      "message": "Create rejected as Object with same ID already exists. Cannot index record managed_user{displayName:Morley Waghorne-2,description:null,givenName:Morley,mail:WaghornM@ns-mail3.com,telephoneNumber:+1 818 885-8439,sn:Waghorne,userName:WaghornM,ldapGroups:[0],accountStatus:active,effectiveRoles:[0],effectiveAssignments:[0],_openidm_id:68a9bac8-2994-4f1d-b388-100fc59e5d12}: found duplicated key 'WaghornM' in index 'managed_user!userName!Idx' previously assigned to the record #12:8. Root cause: Cannot index record managed_user{displayName:Morley Waghorne-2,description:null,givenName:Morley,mail:WaghornM@ns-mail3.com,telephoneNumber:+1 818 885-8439,sn:Waghorne,userName:WaghornM,ldapGroups:[0],accountStatus:active,effectiveRoles:[0],effectiveAssignments:[0],_openidm_id:68a9bac8-2994-4f1d-b388-100fc59e5d12}: found duplicated key 'WaghornM' in index 'managed_user!userName!Idx' previously assigned to the record #12:8",
      "messageDetail": {
        "code": 412,
        "reason": "Precondition Failed",
        "message": "Create rejected as Object with same ID already exists. Cannot index record managed_user{displayName:Morley Waghorne-2,description:null,givenName:Morley,mail:WaghornM@ns-mail3.com,telephoneNumber:+1 818 885-8439,sn:Waghorne,userName:WaghornM,ldapGroups:[0],accountStatus:active,effectiveRoles:[0],effectiveAssignments:[0],_openidm_id:68a9bac8-2994-4f1d-b388-100fc59e5d12}: found duplicated key 'WaghornM' in index 'managed_user!userName!Idx' previously assigned to the record #12:8"
      },
      "situation": "ABSENT",
      "sourceObjectId": "system/ldap/account/cn=Morley Waghorne-2,ou=People,dc=example,dc=com",
      "targetObjectId": "managed/user/null",
      "reconciling": "source",
      "ambiguousTargetObjectIds": "",
      "entryType": "entry"
    }
  ],
  "resultCount": 1,
  "pagedResultsCookie": null,
  "totalPagedResultsPolicy": "NONE",
  "totalPagedResults": -1,
  "remainingPagedResults": -1
} 

See Also

Policy validation failed error when running reconciliation or using a workflow to set or update attributes to null in IDM (All versions) and OpenIDM 4.5

How do I identify reconciliation performance issues in IDM/OpenIDM (All versions)?

Integrator's Guide › REST API Reference › Accessing Log Entries Over REST

Integrator's Guide › Triggering LiveSync Over REST

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I identify reconciliation performance issues in IDM/OpenIDM (All versions)?

The purpose of this article is to provide guidance on isolating and diagnosing IDM/OpenIDM reconciliation performance issues. This article also includes tips on tuning reconciliation performance based on your findings.

Overview

Isolating the cause(s) of performance bottlenecks within a large IDM/OpenIDM topology comes down to a process of elimination. Identifying bottlenecks requires the system to be broken down to its individual components; each component must then be tested and tuned to achieve the necessary throughput.

For example, given the following basic topology there are multiple components which might limit the throughput of the system:

Although far from exhaustive, the following are some of the possible bottlenecks which might be encountered within the above topology:

  • Source LDAP Read ops/s
  • Target LDAP Read/Write ops/s
  • PostgreSQL Read/Write ops/s
  • JVM Heap limitations
  • Host Disk I/O limitations
  • Host CPU limitations
  • Undersized / incorrectly configured virtualization
  • IDM/OpenIDM configuration issues
  • IDM/OpenIDM triggers/custom code
  • IDM/OpenIDM product defect
Note

One configuration change that can help with reconciliation performance is to increase the number of connector instances in the connection pool as detailed in How do I configure pooled connections for a connector in IDM/OpenIDM (All versions)?

Reconciliation performance

Measuring the performance of individual components within the topology can be done outside the scope of IDM/OpenIDM by using external benchmarking tools and processes or via IDM/OpenIDM itself. Benchmarking components via IDM/OpenIDM consists of using targeted operations and configuration changes to isolate a single component within the system and measure its performance as it relates to reconciliation.

You should consider the following components:

External source system

During reconciliation, IDM/OpenIDM queries the source system in order to obtain the complete list of source object IDs to be reconciled. It may also perform multiple queries to retrieve the complete source object during the reconciliation process or rely on a cache of the source objects returned by the configured sourceQuery.

To begin to understand the performance of the source system, the following can be performed:

  • LDAP searchrate command to measure raw read ops capability. This command is provided in the OpenDJ LDAP Toolkit.
  • Measure execution time via targeted queries against the source system using IDM/OpenIDM REST APIs:
    • Execute the generic ‘query-all-ids’ query against the source system.
    • Execute any ‘sourceQuery’ queries which have been configured.
    • Execute individual source object queries via Query FIlter, specifying the individual source object attributes use within the configured ‘sourceQuery’ queries.

Based on the results of the above actions, tune the source system to resolve performance bottlenecks.  Examples of things which might reduce read performance on a source LDAP system are:

  • Lack of necessary indexes, resulting in un-indexed (full DB scans) searches.
  • Insufficient JVM Heap.
  • Insufficient CPU resources to handle the load.
  • Insufficient Disk I/O performance (non-SSD drives) to handle the load.

External target system

During reconciliation, IDM/OpenIDM queries the target system in order to obtain the complete list of target object IDs to be reconciled. It may also perform multiple queries to retrieve the complete target object during the reconciliation process or rely on a cache of the target objects returned by the configured ‘targetQuery’. Target objects calculated based on the configured mappings are then written to the target system.

To begin to understand the performance of the target system, the following can be performed:

  • LDAP searchrate and modrate commands to measure raw read/write ops capability. These commands are provided in the OpenDJ LDAP Toolkit.
  • Measure execution time via targeted queries against the target system using IDM/OpenIDM REST APIs:
    • Execute the generic ‘query-all-ids’ query against the target system.
    • Execute any ‘targetQuery’ queries which have been configured.
    • Execute any ‘correlationQuery’ queries which have been configured.
    • Execute individual target object queries via Query FIlter, specifying the individual target object attributes use within both the configured ‘targetQuery’ and ‘correlationQuery’ queries.

Based on the results of the above actions, tune the target system to resolve performance bottlenecks. Examples of things which might reduce write performance on a target LDAP system are:

  • Excessive indexes, resulting in increased overhead when writing entries. Specifically take note that substring indexes are extremely costly and may greatly reduce performance.
  • Insufficient JVM Heap.
  • Insufficient CPU resources to handle the load.
  • Insufficient Disk I/O performance (non-SSD drives) to handle the load.

JDBC repository

In order to isolate the JDBC repository from the performance of the reconciliation engine or the target system, the following can be performed:

Read/Write Test

Mapping: LDAP Source -> Managed Object

Situation Action
ABSENT CREATE (Default)

Read Test

Mapping: Managed Object -> LDAP Target

Situation Action
ALL ASYNC (Read Only)

Reconciliation engine

Note

Measuring the performance of the Reconciliation Engine should be done only after having resolved any performance issues observed during testing of the source  and target systems.

The IDM/OpenIDM reconciliation engine is responsible for the bidirectional synchronization of objects between different data stores. In the case of our topology above, this is the synchronization of the objects between the source and target LDAP systems.

During reconciliation IDM/OpenIDM determines the state of the source and target objects (situation assessment), identifies the necessary actions to be performed and performs the actions against the source and target systems. Included within each phase of the reconciliation process are various JavaScript triggers which may or may not impact the performance of the reconciliation process depending on their efficiency.

In order to isolate the reconciliation engine from the performance of the IDM/OpenIDM repository or the target system, the following can be performed:

Read-Only Test

Mapping: LDAP Source -> LDAP Target

Situation Action
ALL ASYNC (Read Only)
Note

The following should be performed with a source system which has been populated with production-like data and an EMPTY target system.

  1. Set the actions associated with all of the situation policies to ‘Read-Only’ via the IDM/OpenIDM Admin UI. This ensures that during the reconciliation process no actions are performed and that nothing is written to the IDM/OpenIDM repository, or the target system.
  2. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging.

Read-Write Test

Mapping: LDAP Source -> LDAP Target

Situation Action
ABSENT CREATE (Default)
  1. Set the action associated with the ABSENT situation policy to CREATE. Set all other situation policies to ‘Read Only’. This ensures that during the reconciliation process objects are created and links are established between the source and target objects. No other actions are performed.
  2. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging.

Link Test

Mapping: LDAP Source -> LDAP Target

Situation Action
FOUND LINK (Custom)
  1. Ensure you have defined your desired ‘correlationQuery’ within your mapping.
  2. Delete the contents of the ‘links’ table within the IDM/OpenIDM repository.
  3. Set the action associated with the FOUND situation policy to LINK. Set all other situation policies to ‘Read Only’. This ensures that during the reconciliation process the ‘correlationQuery’ will be executed to correlate the existing target objects (created during the read-write test) to source objects.  For each correlated object a new link will be written to the IDM repository.
  4. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging.

Source Phase

Mapping: LDAP Source -> LDAP Target

Situation Action
CONFIRMED Update (Default)
  1. Set the ‘runTargetPhase’ property within the mapping to ‘false’. This ensures that ONLY the source phase of the reconciliation process is executed.
  2. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging.

Target Phase

Mapping: LDAP Source -> LDAP Target

Situation Action
CONFIRMED Update (Default)
  1. Delete the contents of the ‘links’ table within the IDM/OpenIDM repository.
  2. Set the ‘sourceQuery’ within the mapping to point to a single specific source object. The goal is to reduce the scope of the source objects processed by the reconciliation engine down to a single object.
  3. Set the ‘runTargetPhase’ property within the mapping to ‘true’. Ensure that either you do NOT define a ‘targetQuery’ or that your ‘targetQuery’ is set to your desired value within production.  Do not set the ‘targetQuery’ to point to a single object as is done with the ‘sourceQuery’. This ensures that all of the target objects will be processed by the target phase of the reconciliation.
  4. Perform a full reconciliation between the source and target system. You should monitor both the CPU usage of the IDM/OpenIDM instance as well as JVM heap usage via verbose GC logging.

See Also

Integrator's Guide › Synchronizing Data Between Resources › Optimizing Reconciliation Performance

How do I collect data for troubleshooting high CPU utilization on IDM/OpenIDM (All versions) servers?

How do I monitor IDM 5 and OpenIDM 3.x, 4.x using SmartEvent?

How do I collect JVM data for troubleshooting IDM/OpenIDM (All versions)?

How do I enable Garbage Collector (GC) Logging for IDM/OpenIDM (All versions)?

How do I change the JVM heap size for IDM/OpenIDM (All versions)?

Best practice for JVM Tuning

Administration Guide › Tuning Servers For Performance

How do I troubleshoot issues with my indexes in DS/OpenDJ (All versions)?

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I purge reconciliation audit logs in IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on purging reconciliation audit logs in IDM/OpenIDM. You can use the same concepts and sample files as a template for purging other audit data.

Overview

The autoPurgeAuditRecon.js file (located in the /path/to/idm/bin/defaults/script/audit directory) calls an openidm.action against the repo/audit/recon endpoint; this then executes the "purge-by-..." queries in the repo.jdbc.json file (located in the /path/to/idm/conf directory).

The autoPurgeAuditRecon.js file and the corresponding "purge-by-..." queries in the repo.jdbc.json file can be customized to purge reconciliation audit logs as required. You can add new commands or queries to limit the amount of data that is deleted in one operation, for example:

  • You could set up queries that pass in the reconid as a parameter and then delete only entries from that reconciliation.
  • You could add LIMIT or FETCH FIRST n ROWS ONLY to the defined query to provide a limit.

Limiting the amount of data deleted in one operation is particularly relevant if you have a lot of directories or input sources, and frequent reconciliation runs since this scenario can cause the auditrecon and auditactivity tables to grow rapidly. The default purge process does a simple delete against the auditrecon table, which can consume  large amounts of database memory and log space when this table is large. Additionally, a delete operation with a large number of rows to be deleted can lock the database and prevent IDM/OpenIDM from functioning properly as threads can stall when trying to perform audit logging.

The example provided below using a stored procedure will allow you to purge those tables much quicker, as it purges records in increments with a database commit; this makes it much less intrusive to your database server.

Known issue with the Oracle® repository

If you use the Oracle repository in OpenIDM 4.5, you need to add the  ${_dbSchema} prefix to the table in the get-recons script per  OPENIDM-6862 (Oracle - Explicit Table - get-recons and for-internalcredentials queries do not include ${_dbSchema} prefix for table). For example, the default script should be:

"get-recons" : "SELECT reconId, timestamp, mapping, message FROM ${_dbSchema}.audit_recon WHERE mapping LIKE ${includeMapping} AND mapping NOT LIKE ${excludeMapping} AND entryType = 'summary' ORDER BY timestamp DESC"

Known issues with the IBM® DB2 repository

If you use the IBM DB2 repository, you need to make the following changes to the repo.jdbc.json file for compatibility with DB2:

  • Remove the following "purge-by-recon-number-of" query:
    "purge-by-recon-number-of" : "set @num := 0, @mapping := ''; DELETE r FROM ${_dbSchema}.${_table} r INNER JOIN ( SELECT reconId, mapping, activitydate, @num := if(@mapping = mapping, @num + 1, 1) AS row_number, @mapping := mapping as m FROM ${_dbSchema}.${_table} WHERE mapping LIKE ${includeMapping} AND mapping NOT LIKE ${excludeMapping} AND entryType = 'summary' ORDER BY mapping, activitydate desc ) AS x ON r.reconId = x.reconId WHERE x.row_number > ${numberOf}"
    
  • Add the following get-recons query (the get-recons query should already exist in IDM / OpenIDM 4.5):
    "get-recons" : "SELECT reconid, activitydate, mapping, message FROM ${_dbSchema}.${_table} WHERE mapping LIKE ${includeMapping} AND mapping NOT LIKE ${excludeMapping} AND entrytype = 'summary' ORDER BY activitydate DESC"
    
  • Ensure the following purge-by-recon-expired query exists and add if not:
    "purge-by-recon-expired" : "DELETE r FROM ${_dbSchema}.${_table} r INNER JOIN (SELECT reconId FROM ${_dbSchema}.${_table} WHERE mapping LIKE ${includeMapping} AND mapping NOT LIKE ${excludeMapping} AND activitydate < ${timestamp} AND entryType = 'summary') AS x ON x.reconId = r.reconId"
    

Example

This example process demonstrates deleting records from the auditrecon and auditactivity tables in batches of 10,000 records using a stored procedure. A stored procedure is one possible approach, but you could use a different method depending on your setup. The stored procedure commands given in this example are specific to the Microsoft® SQL Server; if you use a different database for your repository, you will need to adjust these commands accordingly.

To delete records in batches of 10,000:

  1. Create a stored procedure with the following commands.
    USE [openidm]  GO  /****** Object: StoredProcedure [dbo].[sp_audit_purge] Script Date: 12/14/2016 8:02:32 AM ******/  SET ANSI_NULLS ON  GO  SET QUOTED_IDENTIFIER ON  GO
    
    CREATE PROC [dbo].[sp_audit_purge]  (@includeMapping nvarchar(511) = NULL,  @excludeMapping nvarchar(511) = NULL,  @purgeDate datetime,  @batchsize int = 10000,  @maxloops int = 1000)  AS
    
    SET NOCOUNT ON
    
    DECLARE @run_count int = 0  DECLARE @error int = 0  declare @rc int = 1  Declare @purgeDate_char nvarchar(29)  Declare @where varchar(500), @sql varchar(1000)
    
    Set @purgeDate_char = CONVERT(nvarchar(29), @purgeDate, 127)
    
    ---Print @purgeDate_char
    
    WHILE @rc <> 0 and @run_count < @maxloops  BEGIN  BEGIN TRAN
    
    Set @where = 'Where activitydate < ''' + @purgeDate_char + ''''
    
    If @includeMapping is not NULL  Set @where = @where + ' and mapping LIKE ''' + @includeMapping + ''''
    
    If @excludeMapping is not NULL  Set @where = @where + ' and mapping NOT LIKE ''' + @excludeMapping + ''''  Set @sql = 'DELETE TOP (' + convert(varchar(10), @batchsize) + ') FROM openidm.auditrecon ' + @where  --Print @sql  Exec (@sql)
    
    SELECT @rc = @@ROWCOUNT, @error = @@ERROR
    
    IF @error <> 0  GOTO EXIT_PROC
    
    COMMIT TRAN  set @run_count += 1  END
    
    set @run_count = 0  set @rc = 1
    
    WHILE @rc <> 0 and @run_count < @maxloops  BEGIN  BEGIN TRAN
    
    DELETE TOP (@batchsize)  FROM openidm.auditactivity  WHERE activitydate < @purgeDate_char
    
    SELECT @rc = @@ROWCOUNT, @error = @@ERROR
    
    IF @error <> 0  GOTO EXIT_PROC
    
    COMMIT TRAN  set @run_count += 1
    
    END  RETURN
    
    EXIT_PROC:
    
    ROLLBACK TRAN  RETURN
  2. Update the "purge-by-recon-expired" script in the repo.jdbc.json file to invoke this stored procedure. The default "purge-by-recon-expired" script is:
    "purge-by-recon-expired" : "DELETE FROM audit_recon WHERE mapping LIKE ${includeMapping} AND mapping NOT LIKE ${excludeMapping} AND timestamp < ${timestamp}",
    
    You should replace it with the following version:
    "purge-by-recon-expired" : "EXECUTE sp_audit_purge ${includeMapping}, ${excludeMapping}, ${timestamp}"
    
  3. Set up a schedule for the auto purge based on the sample file (schedule-autoPurgeAuditRecon.json in the /path/to/idm/samples/schedules directory). You should use the purgeByExpired purgeType, for example:
    {
        "enabled" : true,
        "type" : "cron",
        "schedule" : "0 0 */12 * * ?",
        "persisted" : true,
        "misfirePolicy" : "doNothing",
        "invokeService" : "script",
        "invokeContext" : {
            "script" : {
                "type" : "text/javascript",
                "file" : "audit/autoPurgeAuditRecon.js",
                "input" : {
                    "mappings" : [ "%" ],
                    "purgeType" : "purgeByExpired",
                    "numOfRecons" : 1,
                    "intervalUnit" : "minutes",
                    "intervalValue" : 1
                }
            }
        }
    }
    

See Also

How do I add logging to JavaScript files in IDM/OpenIDM (All versions)?

How do I add logging to Groovy scripts in IDM/OpenIDM (All versions)?

Integrator's Guide › Purging Obsolete Audit Information

Integrator's Guide › Logging Audit Information

Related Training

N/A

Related Issue Tracker IDs

OPENIDM-7431 ("purge-by-recon-number-of" query missing from default Oracle DB repo file)

OPENIDM-6862 (Oracle - Explicit Table - get-recons and for-internalcredentials queries do not include ${_dbSchema} prefix for table)

OPENIDM-6826 (Include auto prune script for AUDITACTIVITY table)

OPENIDM-6820 (cleanup autoPurgeAuditRecon.js related repo queries and commands)


How do I set up the Active Directory Connector to achieve failover synchronization in OpenIDM 3.x and 4.x when there are multiple Domain Controllers?

The purpose of this article is to provide information about setting up the Active Directory® (AD) Connector to achieve failover synchronization in OpenIDM when there are multiple Domain Controllers (DC). This configuration is only applicable to reconciliation not LiveSync.

Setting up the AD Connector

Note

This configuration is not compatible with LiveSync because LiveSync uses the AD changelog; this means it is bound to the DC that it initially connects to and cannot failover to a different DC as this would cause the sync token to become out of sync.

When using the AD Connector for reconciliation, you can set the AD Connector to use the AD Global Catalog to identify a DC to handle requests.

To acheive this, you need to set the following configuration properties in your AD Connector provisioner config file (for example, provisioner.openicf-ad.json), which is located in the /path/to/idm/conf directory: You should not change any other settings in this section.

"configurationProperties" :
      {
         "LDAPHostName" : null,
         "SearchChildDomains" : true,
      },

Setting the LDAPHostName to null means the AD Connector allows the Active Directory Service Interface (ADSI) to choose a valid DC each time a request is made. Setting the SearchChildDomains property to true means the Global Catalog is used for search and query operations only; the Global Catalog is never used for create, update or delete operations.

See Also

Connector Reference › Active Directory Connector

Related Training

N/A

Related Issue Tracker IDs

OPENICF-120 (AD connector liveSync feature doesn't support AD DC failover)


Known Issues


Timeout on system object message during reconciliation process in IDM/OpenIDM (All versions)

The purpose of this article is to provide information on adjusting operation timeouts for an OpenICF connector in IDM/OpenIDM if you see a "Timeout on system object" message for a single object when running the reconciliation process.

Symptoms

A response similar to the following is shown for a single object when running the reconciliation process:

{ "code":503, "reason":"Service Unavailable", "message":"Operation UPDATE Timeout on system object: <GUID=5f70e361ecf3c952ac2d1acef12c9171>" }

It may show a different operation type, for example, CREATE.

Recent Changes

Implemented operation timeouts.

Causes

By default, the provisioner configuration files are set up without any operational timeouts (all operations are set to -1). This message is shown when the operation takes longer to complete than the timeout specified.

Solution

This issue can be resolved by increasing the operation timeout in your provisioner configuration file (for example, provisioner.openicf-ldap.json), which is located in the /path/to/idm/conf directory. For the example message shown in the Symptoms section, you would need to increase the timeout for the UPDATE operation. 

You can either increase the timeout to a new value to see if the errors go away and adjust as needed, or you can perform a similar update directly using LDAP modify (ldapmodify command) to see how long it takes and adjust accordingly.

Here is a sample configuration where all operations are configured to time out after 1 minute (60,000 miliseconds):

{
  "CREATE"              : 60000,
  "TEST"                : 60000,
  "AUTHENTICATE"        : 60000,
  "SEARCH"              : 60000,
  "VALIDATE"            : 60000,
  "GET"                 : 60000,
  "UPDATE"              : 60000,
  "DELETE"              : 60000,
  "SCRIPT_ON_CONNECTOR" : 60000,
  "SCRIPT_ON_RESOURCE"  : 60000,
  "SYNC"                : 60000,
  "SCHEMA"              : 60000
}

See Integrator's Guide › Connecting to External Resources › Setting the Operation Timeouts for further information.

Note

You should also check the number of connector instances specified in the poolConfigOption is appropriately set for your environment as detailed in  How do I configure pooled connections for a connector in IDM/OpenIDM (All versions)?The poolConfigOption is used to determine how many connector instances are pooled by IDM and made available to service requests.

See Also

How do I configure pooled connections for a connector in IDM/OpenIDM (All versions)?

OpenICF Connector Configuration Reference

Reference › ldapmodify

Related Training

N/A

Related Issue Tracker IDs

N/A


Reconciliation fails in IDM (All versions) and OpenIDM 4.x when an entry has a multi-value attribute that has been defined as a single value attribute

The purpose of this article is to provide assistance when reconciliation fails in IDM/OpenIDM when an entry has an attribute with more than one value but that entry has been defined as a single value attribute in the provisioner configuration.

Symptoms

An error similar to the following is shown when reconciliation fails, where the attribute causing the issue in this example is mail:

2016-10-04 16:22:38:120 WARN Incorrect schema configuration. Expecting mail attribute to be single but it has multi value. [AttributeInfoHelper]Failed to read target object
org.forgerock.json.resource.ConflictException: The mail attribute is not single value attribute.
   at org.forgerock.json.resource.ResourceException.newResourceException(ResourceException.java:230)
   at org.forgerock.openidm.provisioner.openicf.impl.OpenICFProvisionerService$ObjectClassResourceProvider.handleRead(OpenICFProvisionerService.java:1509)
   at org.forgerock.openidm.provisioner.openicf.impl.OpenICFProvisionerService$ObjectClassRequestHandler.handleRead(OpenICFProvisionerService.java:1048)

This error may have been shown in earlier versions of OpenIDM, but it did not cause reconciliation to fail.

Similarly, an attempt to sync using a REST call results in the following response:

{"code":409,"reason":"Conflict","message":"Synchronization failed"}

Recent Changes

Upgraded to, or installed IDM 5 or later.

Upgraded to, or installed OpenIDM 4.x.

Causes

The reconciliation process is expecting a single value attribute per the provisioner configuration. In IDM / OpenIDM 4.x, the presence of an entry with a multi-valued attribute causes the entire reconciliation process to fail with an error (whereas in previous versions, the reconciliation process continued despite the error).

Solution

This issue can be resolved as follows:

  1. Update the provisioner configuration file (for example, provisioner.openicf-ldap.json in the /path/to/idm/conf directory) to define the affected attribute as an array to allow multiple values. For example, you would change the definition of the mail attribute to the following to make it an array:
      "mail" : {
                        "type" : "array",
                        "items" : {
                            "type" : "string",
                            "nativeType" : "string"
                        },
                        "nativeName" : "mail",
                        "nativeType" : "string"
                    },
    
  2. Make changes to the target system to recognize the updated attribute; you can use one of the following approaches:
    • Update the target system to make the attribute an array as well.
    • Add a transformation script to the mapping if the attribute should remain single valued in the target system; there are various ways to do this depending on the desired behavior. For example:
                        {
                          "source" : "mail",
                          "target" : "mail",
                          "transform" : {
                              "type" : "text/javascript",
                              "globals" : { },
                              "source" : "source.length == 1 ? source[0] : source"
                          }
                      }
      
      
      This script checks if the source array contains exactly one item, and if so, it returns this item as source[0]; otherwise the script returns an array. If the attribute is configured as single valued in the target system, it will cause an exception when an array is passed to it and reconciliation of the entry will fail. The rest of the reconciliation will proceed however, resulting in a behavior similar to OpenIDM 3.x. Note that this script example will fail the reconciliation of an item for which the source attribute is an empty array, additional handling for that case may be required. The reverse mapping is rather simple, all it takes is to package the single valued attribute in an array:
                        {
                          "source" : "mail",
                          "target" : "mail",
                          "transform" : {
                              "type" : "text/javascript",
                              "globals" : { },
                              "source" : "[source]"
                          }
                      }
      
      
      See Integrator's Guide › Synchronizing Data Between Resources › Transforming Attributes in a Mapping for further information.

See Also

Integrator's Guide › Synchronizing Data Between Resources › Transforming Attributes in a Mapping

Related Training

N/A

Related Issue Tracker IDs

OPENIDM-6156 (multi-valued mail attribute causes reconciliation to stop in failure with no log information)


Policy validation failed error when running reconciliation or using a workflow to set or update attributes to null in IDM (All versions) and OpenIDM 4.5

The purpose of this article is to provide assistance if you get a "Policy validation failed" error when running reconciliation, or using a workflow to set or update attributes to null in IDM/OpenIDM.

Symptoms

Reconciliation

You will see one of the following errors in the IDM/OpenIDM log when performing a reconciliation where one of the fields is empty (in this example, it is the title field):

Nov 08, 2016 10:21:02 AM org.forgerock.openidm.sync.impl.ObjectMapping createTargetObject
WARNING: Failed to create target object
org.forgerock.json.resource.ForbiddenException: Policy validation failed
   at org.forgerock.json.resource.ResourceException.newResourceException(ResourceException.java:224)
   at org.forgerock.json.resource.ResourceException.getException(ResourceException.java:330)
   at org.forgerock.script.exception.ScriptThrownException.toResourceException(ScriptThrownException.java:125)
   at org.forgerock.script.engine.Utils.adapt(Utils.java:255)
...	
Caused by: org.forgerock.script.exception.ScriptThrownException: [object Object] {code=403, detail={result=false, failedPolicyRequirements=[{policyRequirements=[{params={invalidType=null, validTypes=[string]}, policyRequirement=VALID_TYPE}], property=title}]}, message=Policy validation failed}
Nov 08, 2016 10:21:02 AM org.forgerock.openidm.sync.impl.ObjectMapping$2 recon
WARNING: Unexpected failure during source reconciliation f050a1e6-4060-4907-bfff-43e7174a758e-125
org.forgerock.openidm.sync.impl.SynchronizationException: Policy validation failed
	at org.forgerock.openidm.sync.impl.ObjectMapping$SyncOperation.performAction(ObjectMapping.java:1915)
	at org.forgerock.openidm.sync.impl.ObjectMapping$SourceSyncOperation.sync(ObjectMapping.java:2225)
	at org.forgerock.openidm.sync.impl.ObjectMapping$2.recon(ObjectMapping.java:1198)
	at org.forgerock.openidm.sync.impl.ObjectMapping$ReconTask.call(ObjectMapping.java:1325)
	at org.forgerock.openidm.sync.impl.ObjectMapping$ReconTask.call(ObjectMapping.java:1298)
...
Caused by: org.forgerock.openidm.sync.impl.SynchronizationException: Policy validation failed
	at org.forgerock.openidm.sync.impl.ObjectMapping.createTargetObject(ObjectMapping.java:581)
	at org.forgerock.openidm.sync.impl.ObjectMapping.access$1300(ObjectMapping.java:75)
	at org.forgerock.openidm.sync.impl.ObjectMapping$SyncOperation.performAction(ObjectMapping.java:1808)
	... 10 more
Caused by: org.forgerock.json.resource.ForbiddenException: Policy validation failed

Workflow

The following response is shown when a workflow tries to set a field to null:

{ 
   "code": 403, 
   "reason": "Forbidden", 
   "message": "Policy validation failed", 
   "detail": { 
      "result": false, 
      "failedPolicyRequirements": [ 
         { 
         "policyRequirements": [ 
            { 
            "policyRequirement": "VALID_TYPE", 
            "params": { 
               "validTypes": [ 
               "object" 
               ], 
            "invalidType": "null" 
            } 
            } 
         ], 
      "property": "title" 
      } 
] 
}
}

Recent Changes

Upgraded to, or installed IDM 5.0 or later.

Upgraded to, or installed OpenIDM 4.5.

Causes

The schema in OpenIDM 4.5 changed to enforce the type of values that can be set and null is now considered to be its own type. Null was a permitted value in earlier versions of OpenIDM.

Solution

This issue can be resolved by updating the managed object schema to make null a permitted type for the affected field.

You can configure the managed object schema to accept null values by updating the managed.json file (located in the /path/to/idm/conf directory) and changing the type to null. A property can have more than one type by defining it as an array, for example:

"type" : ["string","null"],
Note

There is a known issue if you set type to ["relationship","null"] that can cause tabs to disappear in the Admin UI in OpenIDM 4.5: OPENIDM-6742 (["relationship","null"] on 'manager' in managed.json causes tabs to disappear in the UI). This is fixed in IDM 5.

See Also

OpenIDM 4.5 Release Notes › OpenIDM Compatibility › Important Changes to Existing Functionality

Integrator's Guide › Using Policies to Validate Data › Validation of Managed Object Data Types

Integrator's Guide › Managing Users, Groups, Roles and Relationships › Creating and Modifying Managed Object Types

Related Training

N/A

Related Issue Tracker IDs

OPENIDM-6742 (["relationship","null"] on 'manager' in managed.json causes tabs to disappear in the UI)


LiveSync


Best practice for LiveSync in IDM/OpenIDM (All versions) with multiple DS/OpenDJ instances

The purpose of this article is to provide best practice advice on using LiveSync in IDM/OpenIDM when there are multiple DS/OpenDJ instances and you are using the LDAP connector.

Warning

Do not compress, tamper with, or otherwise alter changelog database files directly unless specifically instructed to do so by a qualified ForgeRock technical support engineer. External changes to changelog database files can render them unusable by the server. By default, changelog database files are located under the /path/to/ds/changelogDb directory.

LiveSync best practice

Your approach to using LiveSync in IDM/OpenIDM is affected by the version of your DS/OpenDJ instances if you want to use the DS/OpenDJ changelog:

  • OpenDJ 3.x / DS - OpenDJ 3.0 introduced an improved replication changelog implementation, which guarantees consistent changelog numbers across replicated DS/OpenDJ servers. This means IDM/OpenIDM can connect to a load balancer or VIP that is in front of OpenDJ 3.x / DS.
  • OpenDJ 2.6.x - The changelog does not work reliably for LiveSync purposes when there are multiple OpenDJ 2.6.x instances behind a load balancer or VIP since each instance maintains its own changelog and the changelog is not replicated between instances. Since the LDAP connector is a pooled connector, concurrent LiveSync requests sent from IDM/OpenIDM to different OpenDJ instances will each have their own pooled connection. This means that although the data is in sync, the change numbers in the changelogs are not guaranteed to be in sync, which means the changelog cannot work reliably for LiveSync purposes when going via a load balancer or VIP. It is therefore strongly recommended that you do not use a load balancer or VIP if you want to use LiveSync with multiple OpenDJ instances. You should instead point IDM/OpenIDM to a single OpenDJ instance.

Alternatively, you can use the timestamp mechanism for LiveSync rather than the changelog, although this has limitations as discussed below.

Changelog

You can use the changelog with OpenDJ 3.x / DS and have a load balancer or VIP in front of DS/OpenDJ as required. This is the recommended setup.

If you want to use the changelog with OpenDJ 2.6.x, you should have two or more separate IDM/OpenIDM instances in a cluster that share the same configuration, where:

  • one or more IDM/OpenIDM instances are used for search and CRUD operations; these instances can connect to a load balancer or VIP in front of OpenDJ and can be configured for failover as detailed in How do I configure the LDAP connector in IDM/OpenIDM (All versions) for LDAP failover?
  • one IDM/OpenIDM instance is used for LiveSync purposes; this instance must connect directly to a single OpenDJ instance and must not be configured for failover (failover property must be empty). If this OpenDJ instance fails, LiveSync will pick up the correct change number when it is restored.
  • Additionally, you must use property value substitution for the host field in the provisioner configuration file (for example, provisioner.openicf-ldap.json located in the /path/to/idm/conf directory) for the OpenDJ instances. For example, the instance that performs CRUD operations would start up with a pointer to the OpenDJ load balancer host ID while the other instance would start up with a pointer to the host ID of the single OpenDJ node used for LiveSync. This configuration allows multiple IDM/OpenIDM nodes in a cluster to start up pointing to the correct OpenDJ instance. See Integrator's Guide › Configuring the Server › Using Property Value Substitution In the Configuration for further details.

Timestamp mechanism

You can use the timestamp mechanism regardless of which version of DS/OpenDJ you are using and you can use a load balancer or VIP in front of DS/OpenDJ as required.

Timestamps are maintained per entry for create and modify operations; however, delete operations cannot be detected via timestamps. If delete synchronization is a high priority, you should continue to use the changelog for LiveSync.

Note

LDAP connector 1.4.0.1 or later is required if you want to use the timestamp mechanism. Suitable versions are included in IDM / OpenIDM 4.x.

If you want to use timestamps, you should set the following property in in your provisioner configuration file (for example, provisioner.openicf-ldap.json), which is located in the /path/to/idm/conf directory:

"useTimestampsForSync" : "true",
Caution

You should thoroughly test LiveSync with the timestamp mechanism in a development environment first to ensure it meets your needs. Using LiveSync with timestamps can potentially cause performance issues if the DS/OpenDJ instance is managing millions of entries; the timestamp search can have a high cost when there are many entries.

LiveSync retry policy

You should consider configuring the LiveSync retry policy to define how many times a failed modification should be reattempted and what should happen in the event that the modification is unsuccessful after the specified number of attempts. This is discussed in: Integrator's Guide › Synchronizing Data Between Resources › Configuring the LiveSync Retry Policy. If no retry policy is configured, IDM/OpenIDM reattempts the change an infinite number of times, until the change is successful.

See Also

Best practice for LiveSync in IDM/OpenIDM (All versions) with Active Directory

How do I configure the LDAP connector in IDM/OpenIDM (All versions) for LDAP failover?

Connectors Reference › Generic LDAP Connector

Integrator's Guide › Configuring OpenIDM › Using Property Value Substitution In the Configuration

Integrator's Guide › Synchronizing Data Between Resources › Configuring the LiveSync Retry Policy

Integrator's Guide › Connecting to External Resources

Samples Guide › Connecting to a MySQL Database With ScriptedSQL

Related Training

N/A

Related Issue Tracker IDs

OPENICF-406 (ldap liveSync not working using TimeStamp strategy)


Best practice for LiveSync in IDM/OpenIDM (All versions) with Active Directory

The purpose of this article is to provide best practice advice on using LiveSync in IDM/OpenIDM with Active Directory® when you are using the LDAP connector; this information does not apply to Azure® Active Directory. This article also discusses high availability in this scenario.

LiveSync with Active Directory

LiveSync captures the changes that occur in Active Directory and pushes them to IDM/OpenIDM.

The LiveSync mechanism for Active Directory is based on the uSNChanged and highestCommittedUSN attributes. The LiveSync synchronization token is set to the uSNCHanged (USN) number for the last update that was processed and subsequent LiveSync attempts query the Active Directory instance for changes since then.

Caution

The USN numbers are unique per Domain Controller (DC) and therefore the LDAP connector must point to a single DC (no load balancer in between). Failover to an alternate DC may cause LiveSync to break as the USN numbers are not replicated across DCs, meaning the USN numbers will be inconsistent.

You do not need to make any changes to the provisioner configuration file (for example, provisioner.openicf-ldap.json) to specify the attributes used by LiveSync if you are using Active Directory.

Note

You should consider configuring the LiveSync retry policy to determine how many times a failed modification should be reattempted and what should happen in the event that the modification is unsuccessful after the specified number of attempts. This is discussed in: Integrator's Guide › Synchronizing Data Between Resources › Configuring the LiveSync Retry Policy. If no retry policy is configured, IDM/OpenIDM reattempts the change an infinite number of times, until the change is successful.

High availability

Since you can only point to one DC, using LiveSync is not an option if you require high availability. As an alternative, you can use the timestamp mechanism instead for synchronizing changes and point to a Global Catalog (GC). Timestamps are maintained per entry for create and modify operations; however, delete operations cannot be detected via timestamps. If delete synchronization is a high priority, you should continue to use LiveSync.

There are two constraints you should be aware of if you use a GC:

  • GC only contains a partial attribute set. This means not all attributes are replicated from the DC to GC, although this is configurable.
  • Groups and their members are only properly handled if the groups are universal.
Note

LDAP connector 1.4.0.1 or later is required if you want to use the timestamp mechanism. Suitable versions are included in IDM / OpenIDM 4.x.

To use the timestamp mechanism, you should add the following property to your provisioner configuration file (for example, provisioner.openicf-ldap.json, located in the /path/to/idm/conf directory):

"useTimestampsForSync" : "true",

See Also

Best practice for LiveSync in IDM/OpenIDM (All versions) with multiple DS/OpenDJ instances

Integrator's Guide › Synchronizing Data Between Resources › Types of Synchronization

Connector Reference › Generic LDAP Connector › Using the Generic LDAP Connector With Active Directory

Integrator's Guide › Connecting to External Resources

Samples Guide › Connecting to a MySQL Database With ScriptedSQL

Related Training

N/A

Related Issue Tracker IDs

OPENICF-406 (ldap liveSync not working using TimeStamp strategy)


How do I monitor LiveSync activity using REST in IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on monitoring LiveSync activity in IDM/OpenIDM using the REST API. You can use this information to help you troubleshoot failed synchronization jobs.

Monitoring LiveSync activity

The following example demonstrates returning details for the last two LiveSync schedules and then following up on a specific schedule that failed. You will need to adjust these queries to fit your environment, but they should help you to monitor and resolve your own LiveSync schedule issues.

Note

Using the syncToken endpoint is not a reliable indicator of the health of a job because a failed LiveSync will not update the syncToken. 

Example

Return details of the last two schedule syncs using the following query, where you will need to update userId to match the LiveSync schedule:

$ curl -X GET -H "X-OpenIDM-Username: openidm-admin" -H "X-OpenIDM-Password: openidm-admin" "http://localhost:8080/openidm/audit/access?_queryFilter=userId+sw+\"Scheduled\"&_fields=timestamp,transactionId,userId,response,request&_sortKeys=-timestamp&_pageSize=2"

Example response (where both LiveSync schedules failed):

"result": [
    {
      "_id": "d33e369d-7b82-4f4f-9a70-30d196aacddd-24438",
      "timestamp": "2017-11-14T00:15:45.500Z",
      "transactionId": "d33e369d-7b82-4f4f-9a70-30d196aacddd-24420",
      "userId": "Scheduled d4528b10-cb4f-4b2f-8c13-8deff9677d1b-Fri Nov 10 15:24:30 CDT 2017",
      "response": {
        "status": "FAILED",
        "statusCode": null,
        "elapsedTime": "75475",
        "elapsedTimeUnits": "MILLISECONDS"
      },
      "request": {
        "protocol": "CREST",
        "operation": "ScheduledTask",
        "detail": {
          "taskName": "scheduler-service-group.d4528b10-cb4f-4b2f-8c13-8deff9677d1b"
        }
      }
    },
    {
      "_id": "d33e369d-7b82-4f4f-9a70-30d196aacddd-24382",
      "timestamp": "2017-11-14T00:15:21.500Z",
      "transactionId": "d33e369d-7b82-4f4f-9a70-30d196aacddd-24379",
      "userId": "Scheduled d4528b10-cb4f-4b2f-8c13-8deff9677d1b-Fri Nov 10 15:19:00 CDT 2017",
      "response": {
        "status": "FAILED",
        "statusCode": null,
        "elapsedTime": "75517",
        "elapsedTimeUnits": "MILLISECONDS"
      },
      "request": {
        "protocol": "CREST",
        "operation": "ScheduledTask",
        "detail": {
          "taskName": "scheduler-service-group.d4528b10-cb4f-4b2f-8c13-8deff9677d1b"
        }
      }
    }
  ],
  "resultCount": 2,
  "pagedResultsCookie": "2",
  "totalPagedResultsPolicy": "NONE",
  "totalPagedResults": -1,
  "remainingPagedResults": -1
}

You can then find out more about a failed sync in the audit log using the details returned from the above call. For example, you could use one of the following calls with the corresponding transactionId, depending on what type of events you are interested in:

  • Access events:
    $ curl -X GET -H "X-OpenIDM-Username: openidm-admin" -H "X-OpenIDM-Password: openidm-admin" "http://localhost:8080/openidm/audit/access/d33e369d-7b82-4f4f-9a70-30d196aacddd-24420"
    
    This call will indicate the status of the job (either a SUCCESS or FAILED) and report the latest timestamp. 
  • Activity events:
    $ curl -X GET -H "X-OpenIDM-Username: openidm-admin" -H "X-OpenIDM-Password: openidm-admin" "http://localhost:8080/openidm/audit/activity?_queryFilter=userId+sw+\"Scheduled\"&_status+eq+\"FAILED\"&_transactionId+eq+\"d33e369d-7b82-4f4f-9a70-30d196aacddd-24420\"&_sortKeys=-timestamp"
    
    This call will query the activity logs to see if any failures were reported there.

See Also

How do I read and set the LiveSync syncToken using REST in IDM/OpenIDM (All versions)?

How do I query individual reconciliation synchronization failures using REST in IDM/OpenIDM (All versions)?

Integrator's Guide › Querying Audit Logs Over REST

Integrator's Guide ›Triggering LiveSync Over REST

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I read and set the LiveSync syncToken using REST in IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on reading and setting the LiveSync syncToken using the REST API in IDM/OpenIDM.

Reading and setting the syncToken

You only need to set the syncToken on one instance in the cluster as the syncToken is shared by all instances in a cluster:

  1. Query the _id of the sync data using the following REST call:
    $ curl -X GET -H "X-OpenIDM-Username: openidm-admin" -H "X-OpenIDM-Password: openidm-admin" -H "Content-Type: application/json" http://localhost:8080/openidm/repo/synchronisation/pooledSyncStage?_queryId=query-all-ids
    Example response:
    {
      "result": [
        {
          "_id": "SYSTEMLDAPACCOUNT"
        }
      ],
      "resultCount": 1,
      "pagedResultsCookie": null,
      "totalPagedResultsPolicy": "NONE",
      "totalPagedResults": -1,
      "remainingPagedResults": -1
    }
    
    
  2. Retrieve the last syncToken value using the following REST call, where SYSTEMLDAPACCOUNT is the _id value returned in step 1:
    $ curl -X GET -H "X-OpenIDM-Username: openidm-admin" -H "X-OpenIDM-Password: openidm-admin" -H "Content-Type: application/json" http://localhost:8080/openidm/repo/synchronisation/pooledSyncStage/SYSTEMLDAPACCOUNT
    Example response:
    {
      "_id": "SYSTEMLDAPACCOUNT",
      "_rev": "100",
      "connectorData": {
        "syncToken": 15000,
        "nativeType": "integer"
      }
    }
    
  3. Update the syncToken to the required value (15000 in this example) using the following REST call, where the other connectorData details are the ones returned in step 2 and _rev is set to null to update any revision since it will change every few seconds for small LiveSync intervals:
    $ curl -X PUT -H "X-OpenIDM-Username: openidm-admin" -H "X-OpenIDM-Password: openidm-admin" -H "Content-Type: application/json" -d '{ "connectorData": {  "syncToken": 15000, "nativeType": "integer" }, "_id": "SYSTEMLDAPACCOUNT", "_rev": "null" }' http://localhost:8080/openidm/repo/synchronisation/pooledSyncStage/SYSTEMLDAPACCOUNT
Note

See IDM/OpenIDM (All versions) LiveSync syncToken is out of sync with the DS/OpenDJ changelog number for information on resetting the syncToken to zero instead of to a specific value.

See Also

Best practice for LiveSync in IDM/OpenIDM (All versions) with multiple DS/OpenDJ instances

IDM/OpenIDM (All versions) LiveSync syncToken is out of sync with the DS/OpenDJ changelog number

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I exclude specific users from syncing during LiveSync in IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on excluding specific users from syncing during LiveSync in IDM/OpenIDM using the accountSynchronizationFilter. You can also include specific users using the accountSynchronizationFilter to ensure you only sync the users you choose. This information applies to syncing between an external system (such as an LDAP server like DS/OpenDJ or Active Directory®) and IDM/OpenIDM.

Specifying users to sync

You can specify users to sync during LiveSync by defining the accountSynchronizationFilter in your LDAP provisioner config file (for example, provisioner.openicf-ldap.json), which is located in the /path/to/idm/conf directory. The accountSynchronizationFilter allows you to sync a subset of users by specifying standard LDAP search filters to either include or exclude specific users. Only users matching the accountSynchronizationFilter are synced; by default it is set to null and therefore syncs all users during a LiveSync.

An LDAP search filter consists of one or more criterion which can be joined using AND, OR or NOT operators. Each of these operators is represented by a character, which must be specified before the criteria it applies to; each criterion must be contained within brackets, and the operator and all applicable criteria must also be contained within brackets:

  • AND represented by & - (&(criterion1)(criterion2)) - this operator can be used in a search filter even if there is currently only one criterion.
  • OR represented by | - (|(criterion1)(criterion2))
  • NOT represented by ! - (!(criterion1))

You can combine operators within a single search filter to exactly define your subset of users.

Note

When you are combining operators into a single search filter, you must ensure all opening brackets have corresponding closing brackets for the search filter to work.

Example Search Filters

You can exclude a specific user with the following filter:

"accountSynchronizationFilter" : "(!(uid=userID))"

You can include all users who are a direct member of one of two groups

"accountSynchronizationFilter" : "(|(memberOf=cn=internal,ou=employees,ou=north,dc=example,dc=com)(memberOf=cn=internal,ou=employees,ou=south,dc=example,dc=com))"

Example Search Filters for Active Directory only

For syncing between Active Directory and IDM/OpenIDM, you can make use of the userAccountControl attribute; this is an Active Directory® attribute that provides information about a user's account status.

You can exclude all inactive users (that is, only include active users) with the following filter:

"accountSynchronizationFilter" : "(!(userAccountControl:1.2.840.113556.1.4.803:=2))"

You can include only active users whose organisational unit is customers with the following filter:

"accountSynchronizationFilter" : "(&(!(userAccountControl:1.2.840.113556.1.4.803:=2))(ou=customers))"

See Also

How do I exclude specific users from syncing during reconciliation in IDM/OpenIDM (All versions)?

How do I test LDAP search filters in the Generic LDAP Connector for IDM/OpenIDM (All versions)?

Connectors Guide › Generic LDAP Connector › Controlling What the LDAP Connector Synchronizes

Developer's Guide › Performing LDAP Operations › Searching the Directory

Active Directory: LDAP Syntax Filters

Related Training

ForgeRock Identity Management Core Concepts (IDM-400)

Related Issue Tracker IDs

N/A


Known Issues


IDM/OpenIDM (All versions) LiveSync syncToken is out of sync with the DS/OpenDJ changelog number

The purpose of this article is to provide assistance if the IDM/OpenIDM LiveSync syncToken is out of sync with the DS/OpenDJ changelog number, meaning no changes are detected and LiveSync stops working. If you are using LDAP connector 1.4.1.0 or later, you will see the following error when this happens: "The current SyncToken value (n+x) is greater than the lastChangeNumber value (n)".

Warning

Do not compress, tamper with, or otherwise alter changelog database files directly unless specifically instructed to do so by a qualified ForgeRock technical support engineer. External changes to changelog database files can render them unusable by the server. By default, changelog database files are located under the /path/to/ds/changelogDb directory.

Symptoms

LiveSync fails to detect any changes, and the syncToken and DS/OpenDJ changelog numbers are out of sync.

If you are using LDAP connector 1.4.1.0 or later, the following error is shown when this happens:

WARNING: The current SyncToken value (15,187) is greater than the lastChangeNumber value (12,872)
Sep 21, 2016 8:22:30 AM org.identityconnectors.ldap.LdapConnector doSync

If you are using an older version of the LDAP connector, LiveSync will fail silently.

Recent Changes

Configured or changed your LiveSync configuration.

Updated the DS/OpenDJ instance's changelog after the last successful LiveSync was performed.

Causes

The syncToken is based on the last highest value seen within the DS/OpenDJ changelog and is stored within the IDM/OpenIDM repository.

The syncToken and changelog number can get out of sync for one of the following reasons:

  • IDM/OpenIDM is connected to DS/OpenDJ via a load balancer and the changelog numbers are out of sync between the servers; this is expected in pre-OpenDJ 3.0 as changelog information was not replicated.
  • The repository used by IDM/OpenIDM (repo.jdbc.json) contains old data from a previous IDM/OpenIDM instance, which was pointed to an alternative DS/OpenDJ instance that had a different changelog number.
  • The changelog associated with the DS/OpenDJ instance was modified or purged without having reset the syncToken which is cached in the IDM/OpenIDM repository.

Solution

Note

You should disable the LiveSync schedule if the interval is very small (few seconds) so that the DELETE request can match the rev. The rev changes every few seconds when the LiveSync interval is very small, which would make the DELETE request difficult to apply unless the schedule is disabled.

In IDM 5, there is a resetSyncToken configuration property that can be used to address these possible inconsistencies. See Connectors Guide › Generic LDAP Connector › Configuration Properties for further information on setting this property.

Alternatively, and in earlier versions of OpenIDM, this issue can be resolved by resetting the syncToken to zero (null) using the REST API. You only need to reset the syncToken on one instance in the cluster as the syncToken is shared by all instances in a cluster:

  1. Query the _id of the sync data using the following REST call:
    $ curl -X GET -H "X-OpenIDM-Username: openidm-admin" -H "X-OpenIDM-Password: openidm-admin" -H "Content-Type: application/json" http://localhost:8080/openidm/repo/synchronisation/pooledSyncStage?_queryId=query-all-ids
    Example response:
    {
      "result": [
        {
          "_id": "SYSTEMLDAPACCOUNT"
        }
      ],
      "resultCount": 1,
      "pagedResultsCookie": null,
      "totalPagedResultsPolicy": "NONE",
      "totalPagedResults": -1,
      "remainingPagedResults": -1
    }
    
    
  2. Retrieve the current _rev value of the syncToken using the following REST call, where SYSTEMLDAPACCOUNT is the _id value returned in step 1:
    $ curl -X GET -H "X-OpenIDM-Username: openidm-admin" -H "X-OpenIDM-Password: openidm-admin" -H "Content-Type: application/json" http://localhost:8080/openidm/repo/synchronisation/pooledSyncStage/SYSTEMLDAPACCOUNT
    Example response:
    {
      "_id": "SYSTEMLDAPACCOUNT",
      "_rev": "100",
      "connectorData": {
        "syncToken": 15000,
        "nativeType": "integer"
      }
    }
    
  3. Reset the syncToken to zero using the following REST call, where If-Match is set to the _rev value returned in step 2:
    $ curl -X Delete -H "X-OpenIDM-Username: openidm-admin" -H "X-OpenIDM-Password: openidm-admin" -H "If-Match: 100" http://localhost:8080/openidm/repo/synchronisation/pooledSyncStage/SYSTEMLDAPACCOUNT

The syncToken will detect the correct value next time LiveSync runs and update accordingly.

You should also refer to Best practice for LiveSync in IDM/OpenIDM (All versions) with multiple DS/OpenDJ instances to ensure your configuration is correct to avoid similar issues in the future.

See Also

How do I read and set the LiveSync syncToken using REST in IDM/OpenIDM (All versions)?

Best practice for LiveSync in IDM/OpenIDM (All versions) with multiple DS/OpenDJ instances

Related Training

N/A

Related Issue Tracker IDs

OPENICF-402 (need addition logging when syncToken no longer detects changes)

OPENICF-601 (LDAP connector: need for a "resetSyncToken" flag)


LiveSync fails with ORA-01843 or ORA-01861 error when using the Database Table Connector in IDM/OpenIDM (All versions)

The purpose of this article is to provide assistance if you see one of the following errors when LiveSync fails in IDM/OpenIDM: "ORA-01843: not a valid month" or "ORA-01861: literal does not match format string". These errors occur when using the Database Table Connector to connect to a table on an Oracle® database.

Symptoms

You will see one of the following errors when LiveSync fails depending on the data type as explained in the Causes section:

2017-01-22T12:09:42.062+0000 WARNING org.forgerock.openidm.quartz.impl.SchedulerServiceJob execute org.forgerock.openidm.quartz.impl.SchedulerServiceJob: Scheduled service "scheduler-service-group.sync_systemSunidmAccount_managedUser" invocation reported failure: org.forgerock.json.resource.InternalServerErrorException: Failed to get OperationOptionsBuilder: Can not read from the table or view 'sunidm_openidm.account'.^_ 
org.forgerock.openidm.quartz.impl.ExecutionException: org.forgerock.json.resource.InternalServerErrorException: Failed to get OperationOptionsBuilder: Can not read from the table or view 'sunidm_openidm.account'. 
   at org.forgerock.openidm.provisioner.impl.SystemObjectSetService.execute(SystemObjectSetService.java:412) 
   at org.forgerock.openidm.quartz.impl.SchedulerServiceJob.execute(SchedulerServiceJob.java:133) 
   at org.quartz.core.JobRunShell.run(JobRunShell.java:223) 
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549) 
Caused by: org.forgerock.json.resource.InternalServerErrorException: Failed to get OperationOptionsBuilder: Can not read from the table or view 'sunidm_openidm.account'. 
   at org.forgerock.openidm.provisioner.openicf.impl.OpenICFProvisionerService.liveSynchronize(OpenICFProvisionerService.java:2166) 
   at org.forgerock.openidm.provisioner.impl.SystemObjectSetService.liveSync(SystemObjectSetService.java:449) 
   at org.forgerock.openidm.provisioner.impl.SystemObjectSetService.execute(SystemObjectSetService.java:407) 
... 3 more 
Caused by: org.identityconnectors.framework.common.exceptions.ConnectorException: Can not read from the table or view 'sunidm_openidm.account'. 
   at org.identityconnectors.databasetable.DatabaseTableConnector.sync(DatabaseTableConnector.java:644) 
   at org.identityconnectors.framework.impl.api.local.operations.SyncImpl.sync(SyncImpl.java:73) 
   at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source) 
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
   at java.lang.reflect.Method.invoke(Method.java:498) 
   at org.identityconnectors.framework.impl.api.local.operations.ConnectorAPIOperationRunnerProxy.invoke(ConnectorAPIOperationRunnerProxy.java:104) 
   at com.sun.proxy.$Proxy31.sync(Unknown Source) 
   at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source) 
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
   at java.lang.reflect.Method.invoke(Method.java:498) 
   at org.identityconnectors.framework.impl.api.local.operations.ThreadClassLoaderManagerProxy.invoke(ThreadClassLoaderManagerProxy.java:96) 
   at com.sun.proxy.$Proxy31.sync(Unknown Source) 
   at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source) 
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
   at java.lang.reflect.Method.invoke(Method.java:498) 
   at org.identityconnectors.framework.impl.api.BufferedResultsProxy$BufferedResultsHandler.run(BufferedResultsProxy.java:157) 
Caused by: java.sql.SQLDataException: ORA-01843: not a valid month
Mar 03, 2017 2:50:16 PM org.forgerock.openidm.servlet.internal.ServletConnectionFactory$4 handleException
WARNING: Resource exception: 500 Internal Server Error: "Failed to get OperationOptionsBuilder: Can not read from the table or view 'DB_TABLE'."
org.forgerock.json.resource.InternalServerErrorException: Failed to get OperationOptionsBuilder: Can not read from the table or view 'DB_TABLE'.
   at org.forgerock.openidm.provisioner.impl.SystemObjectSetService.actionInstance(SystemObjectSetService.java:318)
   at org.forgerock.json.resource.InterfaceSingletonHandler.handleAction(InterfaceSingletonHandler.java:34)
...
Caused by: org.forgerock.json.resource.InternalServerErrorException: Failed to get OperationOptionsBuilder: Can not read from the table or view 'DB_TABLE'.
   at org.forgerock.openidm.provisioner.openicf.impl.OpenICFProvisionerService.liveSynchronize(OpenICFProvisionerService.java:2330)
   at org.forgerock.openidm.provisioner.impl.SystemObjectSetService.liveSync(SystemObjectSetService.java:448)
   at org.forgerock.openidm.provisioner.impl.SystemObjectSetService.actionInstance(SystemObjectSetService.java:300)
	... 130 more
Caused by: org.identityconnectors.framework.common.exceptions.ConnectorException: Can not read from the table or view 'DB_TABLE'.
   at org.identityconnectors.databasetable.DatabaseTableConnector.sync(DatabaseTableConnector.java:628)
   at org.identityconnectors.framework.impl.api.local.operations.SyncImpl.sync(SyncImpl.java:73)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at org.identityconnectors.framework.impl.api.local.operations.ConnectorAPIOperationRunnerProxy.invoke(ConnectorAPIOperationRunnerProxy.java:104)
   at com.sun.proxy.$Proxy26.sync(Unknown Source)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at org.identityconnectors.framework.impl.api.local.operations.ThreadClassLoaderManagerProxy.invoke(ThreadClassLoaderManagerProxy.java:96)
   at com.sun.proxy.$Proxy26.sync(Unknown Source)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at org.identityconnectors.framework.impl.api.BufferedResultsProxy$BufferedResultsHandler.run(BufferedResultsProxy.java:157)
Caused by: java.sql.SQLDataException: ORA-01861: literal does not match format string

Recent Changes

Implemented the Database Table Connector.

Causes

The Database Table Connector cannot handle Oracle's TIMESTAMP or DATE data type. This issue will affect you when the field specified as the changeLogColumn in your provisioner configuration is of type TIMESTAMP or DATE.

The error you see is dependent on your data type as follows:

  • TIMESTAMP:
    ORA-01843: not a valid month
  • DATE:
    ORA-01861: literal does not match format string

Solution

This issue can be resolved as follows:

  1. Update your provisioner configuration file (for example, provisioner.openicf-contractordb.json in the /path/to/idm/conf directory) to ensure the allNative and nativeTimestamps configuration properties are set to true as follows:
         "allNative" : "true",
         "changeLogColumn" : "LAST_MODIFIED",
         "nativeTimestamps" : "true",
    
    By default, allNative is set to false and changeLogColumn is typically set to LAST_MODIFIED, which has a data type of TIMESTAMP. You should amend changeLogColumn to match your settings if you use a different field.
  2. Add the following JVM option to the Java execution line in the startup.sh or startup.bat file:
    -Doracle.jdbc.J2EE13Compliant=true

See Also

Connectors Reference › Database Table Connector

Related Training

N/A

Related Issue Tracker IDs

OPENICF-81 (DatabaseTableConnector can't handle timestamp data type from Oracle)

OPENIDM-2265 (Got "ORA-01843: not a valid month" while trying to liveSync from Oracle database)


Implicit Synchronization


How do I provision external accounts in a pre-defined order in IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on provisioning external accounts in a pre-defined order in IDM/OpenIDM using implicit synchronization. Implicit synchronization is referred to as automatic sync in OpenIDM 3.0 but the behaviour and functionality is the same.

Provisioning external accounts in a pre-defined order

If you require provisioning to take place in a specific order, for example, you have multiple accounts that need to be created in a set order or you have LDAP accounts with both a main and sub-entry, where:

  • Main entries must be created before sub-entries.
  • Sub-entries must be deleted before main entries.

You must have a mapping per account or entry and then ensure the mappings in the sync.json file (located in the /path/to/idm/conf directory) are in the required order; provisioning occurs according to the order of the mappings in the sync.json file. For example, if the mapping for the main entry is listed in the sync.json file before the mapping for the sub-entry, the main entry will be provisioned/deprovisioned first, followed by the sub-entry.

Note

The sync-two-external-resources sample (sample5) contains multiple mappings to help you understand more fully how this works.

See Also

How do I merge multiple external accounts to a single managed/user object in IDM/OpenIDM (All versions)?

Integrator's Guide › Synchronizing Data Between Resources › Types of Synchronization

OpenIDM 3 Integrator's Guide › Configuring Synchronization › How Automatic Sync works with onSync

Samples Guide › Synchronizing Data Between Two External Resources

Related Training

ForgeRock Identity Management Core Concepts (IDM-400)

Related Issue Tracker IDs

N/A


How do I prevent updates to specific attributes in DS/OpenDJ when performing an Implicit Synchronization operation in IDM/OpenIDM (All versions)?

The purpose of this article is to provide information on preventing updates to all attributes in DS/OpenDJ when performing an implicit sync operation in IDM/OpenIDM. Implicit sync is designed to update the entire object in the target system (all attributes) even if the change only affects a single attribute, but this article shows how you can remove this behavior from specific attributes.

Overview

You can prevent attributes being updated on the target by setting the NOT_RETURNED_BY_DEFAULT flag on the attribute in the provisioner configuration file (for example, provisioner.openicf-ldap.json), which is located in the /path/to/idm/conf directory. When you set this flag, you can then add a condition to the mapping in the sync.json file (located in the /path/to/idm/conf directory) to selectively apply the property mapping to ensure that it is only updated based on the condition you've set (this condition should evaluate to true when the old and new value differ). 

As of OpenIDM 4.0, you can access the oldSource from the condition script, which makes it easier to ensure the property mapping is only applied after a change to the attribute in the managed object. However, you do not have to use oldSource in your condition.

Other considerations

You should also consider the following depending on your setup: 

  • If the property has a transformation, you also need to include the transformation logic in the condition since the condition is evaluated before the transformation. For example: 
    ( (oldSource.sn !== object.sn) && (oldSource.givenName !== object.givenName) )
  • If an attribute also needs to be synced from LDAP to Managed, you need to do one of the following:
    1. Configure two separate provisioner configuration files; one with the NOT_RETURNED_BY_DEFAULT flag and one without.
    2. Specify a sourceQuery on the LDAP to Managed mapping, which includes the _fields property and explicitly lists all of the attributes. This configuration will cause the attribute with the NOT_RETURNED_BY_DEFAULT flag set to be returned anyway in this scenario.

Preventing updates to an attribute (IDM / OpenIDM 4.x)

The following example demonstrates configuring the mail attribute so that it only gets updated during an implicit sync operation when it has actually changed:

  1. Add the NOT_RETURNED_BY_DEFAULT flag to the mail attribute in the provisioner configuration file:
                   "mail" : {
                        "type" : "string",
                        "nativeName" : "mail",
                        "nativeType" : "string",
                        "flags" : [
                            "NOT_RETURNED_BY_DEFAULT"
                        ]
                    },
    
  2. Add a condition to the mapping for the mail attribute in the sync.json file that evaluates to true when the old and new mail value differs:
                   {
                        "target" : "mail",
                        "source" : "mail",
                        "condition" : {
                            "type" : "text/javascript",
                            "source" : "(oldSource.mail !== object.mail)"
                        }
                    },
    

Preventing updates to an attribute (All versions)

The following example demonstrates configuring the mail attribute so that it only gets updated during an implicit sync operation when it has actually changed:

  1. Add the NOT_RETURNED_BY_DEFAULT flag to the mail attribute in the provisioner configuration file:
                   "mail" : {
                        "type" : "string",
                        "nativeName" : "mail",
                        "nativeType" : "string",
                        "flags" : [
                            "NOT_RETURNED_BY_DEFAULT"
                        ]
                    },
    
  2. Set a flag in the onUpdate trigger in the managed.json file (located in the /path/to/idm/conf directory), for example:
                    "onUpdate" : {
                                    "type" : "text/javascript",
                                    "source" :"newObject.mailUpdate=(newObject.mail!=oldObject.mail);"
                                 },
    
  3. Add a condition to the mapping for the mail attribute in the sync.json file that evaluates to true when the old and new mail value differs based on the onUpdate trigger:
                    {
                        "source" : "mail",
                        "condition" : {
                            "type" : "text/javascript",
                            "source" : "object.mailUpdate;"
                        },
                        "transform" : {
                            "type" : "text/javascript",
                            "source" : "openidm.decrypt(source);"
                        },
                        "target" : "mail"
                    },
    

See Also

FAQ: Scripts in IDM/OpenIDM

Integrator's Guide › Synchronizing Data Between Resources › Types of Synchronization

Related Training

N/A

Related Issue Tracker IDs

OPENIDM-3579 (Implicit sync should not overwrite the entire system object when you patch a selected attribute(s) of managed user)

OPENIDM-7308 (Confusing description implicit sync about changes)


Password Synchronization Plugin


How do I upgrade DS/OpenDJ (All versions) if I have the DS/OpenDJ Password Sync Plugin for IDM/OpenIDM installed?

The purpose of this article is to provide information on upgrading DS/OpenDJ if you have the DS/OpenDJ Password Sync Plugin for IDM/OpenIDM installed.

Background Information

The password plugin configuration is specified in one of the following files depending on the plugin version:

  • Plugin version 3.5.0 and later - openidm-accountchange-plugin-sample-config
  • Plugin versions 1.0.3 and 1.1.1 - openidm-pwsync-plugin-config.ldif
Note

You must use the plugin version that corresponds to your DS/OpenDJ version. See Release Notes › Supported Connectors, Connector Servers, and Plugins for further information.

Upgrading DS/OpenDJ

You can upgrade DS/OpenDJ as follows:

  1. Back up the password plugin configuration file (openidm-pwsync-plugin-config.ldif or openidm-accountchange-plugin-sample-config, which is located in the /path/to/ds/config directory) as this contains your current configuration details.
  2. Update the Default Password Policy to remove the account-status-notification-handler attribute using a dsconfig command such as the following:
    $ ./dsconfig set-password-policy-prop --policy-name "Default Password Policy" --reset account-status-notification-handler --hostname localhost --port 4444 --bindDn "cn=Directory Manager" --bindPassword password --trustAll --no-prompt
  3. Remove the changes made to the cn=config backend when the plugin was installed using a ldapdelete command such as the following:
    $ ./ldapdelete --hostname ds1.example.com --port 4444 --bindDN "cn=Directory Manager" --bindPassword password "cn=OpenIDM Notification Handler,cn=Account Status Notification Handlers,cn=config"
  4. Stop the DS/OpenDJ instance.
  5. Delete the following files that apply to the existing plugin:
    • openidm-pwsync-plugin-config.ldif or openidm-accountchange-plugin-sample-config in the /path/to/ds/config directory.
    • 90-openidm-pwsync-plugin.ldif file in the /path/to/ds/config/schema directory; as of DS 6 and later, located in /path/to/ds/db/schema for new installs.
    • opendj-accountchange-handler-x.x.x.jar in the /path/to/ds/lib/extensions directory.
  6. Restart the DS/OpenDJ instance.
  7. Upgrade to the new version of DS/OpenDJ. See Installation Guide and Upgrading DS/OpenDJ for further information.
  8. Install the new plugin per the instructions in Password Synchronization Plugin Guide › Installing the DS Password Synchronization Plugin.
  9. Update the configuration file (openidm-pwsync-plugin-config.ldif or openidm-accountchange-plugin-sample-config depending on which version of the plugin you installed) with the details contained in the configuration file you backed up if you want to retain the same behavior you had previously.

See Also

DS (All versions) and OpenDJ 3.x fail to start after upgrade if you use the Password Sync Plugin for IDM/OpenIDM

OpenDJ Password Synchronization Plugin 1.0.3 install fails on OpenDJ 3.x

How do I troubleshoot the DS/OpenDJ Password Synchronization plugin in IDM/OpenIDM (All versions)?

Password Synchronization Plugin Guide › Synchronizing Passwords With ForgeRock Directory Services (DS)

Related Training

N/A

Related Issue Tracker IDs

N/A


How do I troubleshoot the DS/OpenDJ Password Synchronization plugin in IDM/OpenIDM (All versions)?

The purpose of this article is to provide assistance if you are experiencing issues with the DS/OpenDJ password sync plugin in IDM/OpenIDM. This article also includes information on enabling debug logging for just the DS/OpenDJ password sync plugin to minimize the size of the resulting DS/OpenDJ logs.

Overview

This article includes the following details that you should check (and rectify) in order to troubleshoot the DS/OpenDJ password sync plugin in IDM/OpenIDM:

Note

If you are still unable to solve your issues, you should attach the collected log files and outputs when you raise a ticket to enable us to help you more quickly. See Sending troubleshooting data to ForgeRock Support for analysis for further information.

DS/OpenDJ password sync flow

The expected DS/OpenDJ password sync plugin flow is as follows:

  1. DS/OpenDJ receives an operation to modify an account password.
  2. The DS/OpenDJ password sync plugin is invoked and passes the cleartext password.
  3. The DS/OpenDJ password sync plugin encrypts the password and performs a Patch operation against the corresponding IDM/OpenIDM managed user's ldapPassword attribute.
  4. The onUpdate trigger within managed.json assigns the decrypted ldapPassword attribute value to the password attribute.
  5. The Implicit sync process is triggered for the managed user to LDAP mapping.
  6. The condition associated with the password attribute mapping is evaluated and returns false as the cleartext ldapPassword and password attribute values are the same.
  7. The LDAP Connector does not perform an update to DS/OpenDJ as there are no modifications to be performed.

Avoiding loops

To ensure this flow is followed correctly and you do not end up with a loop whereby the password change originating from DS/OpenDJ results in an update from IDM/OpenIDM back to DS/OpenDJ for the same LDAP entry, the condition associated with the password attribute mapping (sync.json file) should be similar to the following:

{
    "source" : "password",                  
    "condition" : {
        "type" : "text/javascript",
        "source" : "object.ldapPassword != null && (openidm.decrypt(object.password) != openidm.decrypt(object.ldapPassword))"
},

This mapping applies a condition to the sync of the password attribute, which is dependent upon the managed user's ldapPassword attribute value being different from the password attribute value.

Note

The ldapPassword attribute referred to above is configurable and may be different in your configuration; however, the DS/OpenDJ password sync plugin must be configured to Patch an attribute other than password when performing a bi-directional sync else it is impossible to determine whether an update is needed or not.

If DS/OpenDJ syncs passwords with IDM/OpenIDM in both directions, you need to ensure you have configured IDM/OpenIDM correctly to avoid an infinite loop. See Password Synchronization Plugin Guide › Preventing Infinite Loops for further information.

Keystores and certificates

Incorrect keystore and certificate details can cause issues with the DS/OpenDJ password sync plugin (some of which have been shown in the IDM/OpenIDM log section below). The key things to check are:

  1. The IDM/OpenIDM certificate has been imported into the DS/OpenDJ truststore.
  2. The DS/OpenDJ certificate has been imported into the IDM/OpenIDM keystore. This step only applies if you have implemented mutual SSL authentication (server and client certificate authentication); the password sync plugin is configured to use mutual SSL authentication by default.
  3. The encryption key used for the password is correct and present in both the DS/OpenDJ truststore and the IDM/OpenIDM keystore.

IDM/OpenIDM certificate

This process is documented in the Password Synchronization Plugin Guide: To Import the IDM Certificate into the DS Keystore.

You can check that the IDM/OpenIDM certificate has been imported properly as follows:

  1. Use the keytool command to list the certificate aliases in the IDM/OpenIDM keystore to check which certificate is being used; this could be the openidm-localhost certificate or a CA certificate depending on your setup.
  2. Use the keytool command to list the certificate aliases in the DS/OpenDJ truststore and check that the certificate alias returned in step 1 is present. The following example keytool command shows that the openidm-localhost certificate has been imported into the DS/OpenDJ keystore:
    $ keytool -list -keystore truststore -storetype JKS -storepass password -v
    
    Keystore type: JKS
    Keystore provider: SUN
    
    Your keystore contains 2 entries
    
    Alias name: openidm-localhost
    Creation date: 13/07/2016
    Entry type: trustedCertEntry
    
    Owner: CN=localhost, O=OpenIDM Self-Signed Certificate, OU=None, L=None, ST=None, C=None
    Issuer: CN=localhost, O=OpenIDM Self-Signed Certificate, OU=None, L=None, ST=None, C=None
    Serial number: 152f283769d
    Valid from: Tue Jan 19 14:54:07 AEDT 2016 until: Wed Feb 18 14:54:07 AEDT 2026
    Certificate fingerprints:
         MD5:  D4:B4:E7:A5:3C:E9:42:3C:F5:EB:57:5B:5C:D3:18:DC
         SHA1: A6:82:85:49:95:35:71:DF:33:2F:E4:F5:92:AB:B9:65:51:97:2A:D6
         SHA256: D4:55:A9:E8:B6:61:75:39:F4:E7:FE:EE:AE:B0:A2:46:2D:B2:82:81:4A:97:EB:31:BB:8C:E8:35:1E:80:D3:6C
         Signature algorithm name: SHA512withRSA
         Version: 3
    
    
    *******************************************
    *******************************************
    

DS/OpenDJ certificate

This process is documented in the Password Synchronization Plugin Guide: To Import the DS Certificate into the IDM Truststore and is only required for mutual authentication, although the password sync plugin is configured to use mutual SSL authentication by default.

You can check that the DS/OpenDJ certificate has been imported properly as follows:

  1. Use the keytool command to list the certificate aliases in the IDM/OpenIDM keystore and check that the server-cert certificate (or equivalent CA certificate depending on your setup) is being used.
  2. Check that the certificate DN has been added as a value of the allowedAuthenticationIdPatterns property for the CLIENT_CERT authentication module in the authentication.json file (located in the /path/to/idm/conf directory).

Encryption key

The DS/OpenDJ password is encrypted before it is sent to IDM/OpenIDM. The encryption key used is openidm-localhost by default but could be different if you have changed it.

You should check the encryption key as follows:

  1. Check which encryption key is actually being used by DS/OpenDJ using one of the following methods:
    • Look for the value of the ds-cfg-realm property in the config.ldif file (located in the /path/to/ds/config directory).
    • Check the value of the private-key-alias property using the dsconfig command.
    If this is not the key you expected, you should update the ds-cfg-realm property in the config.ldif file with the correct key.
  2. Check that this encryption key is present in both the DS/OpenDJ truststore and the IDM/OpenIDM keystore, otherwise IDM/OpenIDM will not be able to decrypt the password when it is received. If it is not there, import it into the appropriate stores per the documentation links above.

IDM/OpenIDM log

The IDM/OpenIDM log (openidm.log, located in the /path/to/idm/logs directory) can be useful for identifying any issues with the password synchronization process from the IDM/OpenIDM side. You should increase logging in the IDM/OpenIDM debug log to FINEST (set the .level property to FINEST in logging.properties, which is located in the /path/to/idm/conf directory).

SSL and keys

If you see the following message in your IDM/OpenIDM log after changing a password in DS/OpenDJ, it means the SSL connection is working and the request has been received and successfully decrypted:

FINEST: Request: { "content": { "password": { "$crypto": { "value": { "data": "6m3EOH9Fs7gqKNlTphW8Iw==", "cipher": "AES/ECB/PKCS5Padding", "key": { "data": "SKZIzkHc4wSa56sfivwKpblZrN3a/d8H/M10C48VOiaMFwlGM311ISl6s620ysWWXNXyqU4E/+hQFRx4M9pG80aYWWbHZvaicAtXadEs3kPhC8ffpPu8cYmfYbryKzsgQ0CqR+Ke3qERWdLDXRRX+ionkC9CfY4nMMKGhYVXNW61/TtKBrn3g5/3RvmL+mI9IxDeMDTgPsMuqmJtkSwHe2do/WmgeTnMpzMHb3GuXBcbphilzsTtLQvfGFQMC/0WTgwy0tovASIcY0rPRiEm6ymVrYRZTD+Zqy9pTeSVPH+XrqJTJUafIbIQlA1gf54ZnOrB/7Sauyeo7FxS8CzhIQ==", "cipher": "RSA/ECB/OAEPWithSHA1AndMGF1Padding", "key": "openidm-localhost" } }, "type": "x-simple-encryption" } } }, "resourcePath": "policy/managed/user/7a2594ea-ae24-4084-a963-9fa95e1dccc5", "additionalParameters": { "external": "true" }, "action": "validateProperty", "method": "action", "fields": [  ] }

Although the SSL connection is working, you could still see errors if the provided key is unknown to IDM/OpenIDM, for example:

FINEST: Resource exception: 500 Internal Server Error: "Wrapped org.forgerock.json.JsonException: org.forgerock.json.crypto.JsonCryptoException: key not found: openidm-unknown (/path/to/idm/bin/defaults/script/policy.js#690) in /path/to/idm/bin/defaults/script/policy.js at line number 690 at column number 0"

If the SSL connection is not working, you will see a SSL handshake exception, for example:

javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown 
   at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) 
   at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) 
   at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1959) 
   at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1077) 
   at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312) 
   at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339) 
   at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323) 
Note

If you are experiencing issues with SSL or keys, you can enable SSL debugging as detailed in the SSL debugging section.

Configuration issues

The following message indicates there is an issue with your configuration that is causing a loop whereby the password change originating from DS/OpenDJ results in an update from IDM/OpenIDM back to DS/OpenDJ for the same LDAP entry:

Caused by: org.identityconnectors.framework.common.exceptions.ConnectorException: javax.naming.ServiceUnavailableException: [LDAP: error code 51 - Entry uid=user.1,ou=People,dc=forgerock,dc=com cannot be modified because the server failed to obtain a write lock for this entry after multiple attempts]

To resolve this issue, you must prevent the managed user's password being synced to DS/OpenDJ if the change originated from the DS/OpenDJ password sync plugin. See the DS/OpenDJ password sync flow section above for more details on achieving this. Alternatively, you can change the value of the ds-cfg-update-interval property (in the openidm-pwsync-plugin-config.ldif or openidm-accountchange-plugin-sample-config file) to 1 or 2 seconds; this changes the sync operation to an asynchronous process and may resolve this issue

SSL debugging

SSL debugging traces the SSL handshaking phase. You can enable SSL debugging via the OPENIDM_OPTS environment variable.

On Unix® and Linux® systems:

$ cd /path/to/idm/
$ export OPENIDM_OPTS="-Djavax.net.debug=all"
$ ./startup.sh

On Microsoft® Windows® systems:

C:\> cd \path\to\idm
C:\path\to\idm> set OPENIDM_OPTS=-Djavax.net.debug=all
C:\path\to\idm> startup.bat
Note

You can also edit the startup.sh or startup.bat files to update the default OPENIDM_OPTS values.

DS/OpenDJ configuration files

The following configuration files are useful to check how DS/OpenDJ has been configured and how the password sync plugin has been configured on the DS/OpenDJ side:

  • config.ldif (located in the /path/to/ds/config directory).
  • openidm-pwsync-plugin-config.ldif or openidm-accountchange-plugin-sample-config (located in the /path/to/ds/config directory). The actual file depends on the password sync plugin version.

You can then compare the configuration on the DS/OpenDJ side with the configuration on the IDM/OpenIDM side to ensure they correspond and correct if necessary.

IDM/OpenIDM configuration files

The following configuration files (located in the /path/to/idm/conf directory) are useful to check how synchronization has been configured on the IDM/OpenIDM side and compare with the DS/OpenDJ configuration:

  • sync.json
  • managed.json
  • authentication.json
  • provisioner configuration file for DS/OpenDJ, for example provisioner.openicf-ldap.json.
  • any associated scripts used to manage or transform the password attribute during sync, such as a separate onUpdate script (these are typically located in the /path/to/idm/script directory).
Note

It can be very useful to add logging to your scripts to help identify the point at which an issue can occur. Refer to How do I add logging to JavaScript files in IDM/OpenIDM (All versions)? and How do I add logging to Groovy scripts in IDM/OpenIDM (All versions)? for further information.

DS/OpenDJ logs

The DS/OpenDJ log files (access, errors and replication) can be useful for identifying any issues with the password synchronization process from the DS/OpenDJ side. The logs collected should cover the period during which you are experiencing issues and the timestamps should correspond to the logs collected for IDM/OpenIDM.

Enabling debug logging for the DS/OpenDJ password sync plugin

You can enable debug logging for just the DS/OpenDJ password sync plugin to minimize the size of the resulting DS/OpenDJ logs as follows:

  1. Enable debug logging using the following dsconfig command:
    $ ./dsconfig set-log-publisher-prop --hostname ds1.example.com --port 4444 --bindDN "cn=Directory Manager" --bindPassword password --publisher-name "File-Based Debug Logger" --set enabled:true --no-prompt --trustAll
    
    In OpenDJ 2.6.x, you must include the --set default-debug-level:all option as well.
  2. Set the debug target class using one of the following dsconfig commands depending on your version:
    • OpenDJ 3.5.x / DS:
      $ ./dsconfig create-debug-target --hostname ds1.example.com --port 4444 --bindDN "cn=Directory Manager" --bindPassword password --publisher-name "File-Based Debug Logger" --type generic --target-name org.forgerock.openidm.accountchange.OpenidmAccountStatusNotificationHandler --set enabled:true --no-prompt --trustAll
      
    • OpenDJ 3.0 and earlier:
      $ ./dsconfig create-debug-target --hostname ds1.example.com --port 4444 --bindDN "cn=Directory Manager" --bindPassword password --publisher-name "File-Based Debug Logger" --type generic --target-name org.forgerock.openidm.agent.accountchange.OpenidmAccountStatusNotificationHandler --set enabled:true --no-prompt --trustAll
      

IDM/OpenIDM audit logs

The IDM/OpenIDM audit logs (located in the /path/to/idm/audit directory) can be useful for tracking system activity; the activity.csv file is particularly relevant.

See Also

DS (All versions) and OpenDJ 3.x fail to start after upgrade if you use the Password Sync Plugin for IDM/OpenIDM

How do I upgrade DS/OpenDJ (All versions) if I have the DS/OpenDJ Password Sync Plugin for IDM/OpenIDM installed?

OpenDJ Password Synchronization Plugin 1.0.3 install fails on OpenDJ 3.x

How do I add logging to JavaScript files in IDM/OpenIDM (All versions)?

How do I add logging to Groovy scripts in IDM/OpenIDM (All versions)?

Password Synchronization Plugin Guide

Related Training

ForgeRock Identity Management Core Concepts (IDM-400)

Related Issue Tracker IDs

OPENIDM-3824 (Enhance OpenIDM so that only updated fields are synchronized to OpenDJ, rather than the whole managed object)

OPENIDM-3366 (Password sync loop when LDAP groups change)


How do I troubleshoot Active Directory password synchronization issues in IDM/OpenIDM (All versions)?

The purpose of this article is to provide assistance if you are experiencing issues with the Active Directory® password synchronization service (AD Password Sync Plugin) in IDM/OpenIDM. Likely issues include Active Directory password changes not being populated in IDM/OpenIDM or the plugin not polling for changes.

Troubleshooting synchronization issues

This article includes the following details that you should check (and rectify) in order to troubleshoot the AD password sync plugin in IDM/OpenIDM:

Avoiding loops

If AD syncs passwords with IDM/OpenIDM in both directions, you need to ensure you have configured IDM/OpenIDM correctly to avoid an infinite loop. See Password Synchronization Plugin Guide › Preventing Infinite Loops for further information. 

Note

If you are still unable to solve your issues, you should attach the collected log files and outputs when you raise a ticket to enable us to help you more quickly. See Sending troubleshooting data to ForgeRock Support for analysis for further information.

Keystores and certificates

Incorrect keystore and certificate details can cause issues with the AD password sync plugin (some of which have been shown in the IDM/OpenIDM log section below). The key things to check are:

  • The content of the keystore used by the AD password sync plugin corresponds to the content in the IDM/OpenIDM keystore.
  • The certificate DN has been added to the IDM/OpenIDM configuration.

Finally, you should review the steps in Password Synchronization Plugin Guide › Synchronizing Passwords With Active Directory again to ensure you have not missed a step.

Keystores

You can check that the content of the AD and IDM/OpenIDM keystores correspond as follows:

  1. Use the keytool command to list the certificate aliases in the AD keystore (keystore.pk12) to check which certificate is being used; the default is ad-pwd-plugin-localhost, but could be different depending on your setup.
  2. Use the keytool command to list the certificate aliases in the IDM/OpenIDM keystore to check that the certificate alias returned in step 1 is present. The following example keytool command shows that the ad-pwd-plugin-localhost certificate exists in the IDM/OpenIDM keystore:
    $ keytool -list -keystore security/keystore.jceks -storetype JCEKS -storepass password -v
    
    Keystore type: JCEKS
    Keystore provider: SunJCE
    
    Your keystore contains 4 entries
    
    Alias name: ad-pwd-plugin-localhost
    Creation date: Jan 28, 2017
    Entry type: PrivateKeyEntry
    Certificate chain length: 1
    Certificate[1]:
    Owner: CN=localhost, O=AD-pwd-plugin Self-Signed Certificate
    Issuer: CN=localhost, O=AD-pwd-plugin Self-Signed Certificate
    Serial number: 20be2082
    Valid from: Sat Jan 28 10:08:00 EDT 2017 until: Sat Jan 30 10:08:00 EDT 2027
    Certificate fingerprints:
         MD5:  A7:29:7A:B7:3F:0D:2D:A1:D2:79:9E:77:E9:6E:8A:1E
         SHA1: 54:C2:81:FF:B2:3B:89:6B:D0:AB:D9:A1:3C:95:B1:85:ED:5C:2B:35
         SHA256: 3C:90:BA:9E:3B:78:1B:8D:64:A5:89:4B:64:EC:A2:A6:FA:FA:E3:78:4E:95:26:0A:50:C7:89:65:22:1F:F9:CB
         Signature algorithm name: SHA256withRSA
         Version: 3
    
    *******************************************
    *******************************************
    

Certificate DN

Check that the certificate DN has been added as a value of the allowedAuthenticationIdPatterns property for the CLIENT_CERT authentication module in the authentication.json file (located in the /path/to/idm/conf directory). For example:

{
    "name" : "CLIENT_CERT",
    "properties" : {
        "queryOnResource" : "managed/user",
        "defaultUserRoles" : [
            "internal/role/openidm-cert"
        ],
        "allowedAuthenticationIdPatterns" : [ "CN=localhost, O=AD-pwd-plugin Self-Signed Certificate" ]
    },
    "enabled" : true
}

You will know that authentication has been successful when you see something similar to the following in the openidm.log:

FINE: Authentication client certificate subject CN=localhost, O=AD-pwd-plugin Self-Signed Certificate
Jan 31, 2017 04:27:49 PM org.forgerock.caf.authentication.framework.AuthModules$LoggingAuthModule$4 handleResult
FINE: ClientCert has successfully authenticated the client

Password Synchronization log

Password synchronization messages are logged to idm.log and these messages can help identify the cause of your issues. This file is located in the directory you specified when you installed the AD password sync plugin; you can confirm the location by checking the logPath registry entry under the HKEY_LOCAL_MACHINE\SOFTWARE\ForgeRock\OpenIDM\PasswordSync registry key.

Changing the log level to debug

It is recommended that the log level is set to debug during the install to increase the level of logging. If you didn't set it this way during the install, you can change it by setting the logLevel registry entry under the HKEY_LOCAL_MACHINE\SOFTWARE\ForgeRock\OpenIDM\PasswordSync registry key to debug, for example, "logLevel"="debug". You do not need to restart the domain controller if you change this registry entry as it will be picked up dynamically by the AD password sync plugin.

Common errors

Some common errors messages you might find in the idm.log are as follows, along with possible solutions:

Error message Possible solution
ERROR [468:1732]  encrypt_key(): couldn't import key file "\path\to\ad-pwd-plugin-idm.p12"
Check that pkcs12 file password exists and is valid.
ERROR [468:5372]  password_change_worker(): change request for user "user_ID" failed, network status: 0, response: (empty)
Check that the network connection between the AD password sync plugin and IDM/OpenIDM is working correctly.
ERROR [468:6692]  http_status() error parsing status header

Change the netTimeout registry entry to set the network connection/send/read timeout (in seconds) to prevent requests to IDM/OpenIDM failing if IDM/OpenIDM takes too long to respond.

The netTimeout registry entry is under the HKEY_LOCAL_MACHINE\SOFTWARE\ForgeRock\OpenIDM\PasswordSync registry key; a typical netTimeout setting is between 2 and 6 seconds (for example, "netTimeout"="5" for 5 seconds). You do not need to restart the domain controller if you change the netTimeout registry entry as it will be picked up dynamically by the AD password sync plugin.

IDM/OpenIDM log

The IDM/OpenIDM log (openidm.log, located in the /path/to/idm/logs directory) can be useful for identifying any issues with the password synchronization process from the IDM/OpenIDM side. You should increase logging in the IDM/OpenIDM debug log to FINEST (set the .level property to FINEST in logging.properties, which is located in the /path/to/idm/conf directory).

Some common errors messages you might find in the openidm.log are as follows, along with their meanings:

Error message Meaning
WARNING: JSON resource exception org.forgerock.openidm.objset.InternalServerErrorException: org.forgerock.json.fluent.JsonException: org.forgerock.json.crypto.JsonCryptoException: key not found: ad-pwd-plugin-idm
This message indicates there is a problem with the IDM/OpenIDM configuration; in particular, the key/alias name used to encrypt the password is unknown to IDM/OpenIDM. You should check your configuration and verify that it is correct.
FINE: Client certificate subject CN=localhost, O=AD-pwd-plugin Self-Signed Certificate did not match allowed patterns
This message indicates that the client certificate is not recognized; you must add the certificate DN to the IDM/OpenIDM configuration as detailed in the Keystores and certificates section.
One of the following error messages is shown:
FINE: Failed to evaluate path/to/openidm/bin/defaults/script/router-authz.js script.
FINEST: Resource exception: 403 Forbidden: "Access denied"
org.forgerock.json.resource.ForbiddenException: Access denied
...
Caused by: org.mozilla.javascript.JavaScriptException: [object Object] (/path/to/idm/bin/defaults/script/router-authz.js#484)

You may also see this response:

{"code":403,"reason":"Forbidden","message":"Access denied"}

These messages indicate an issue with the configuration in access.js (located in the /path/to/idm/script directory).

In particular, this can happen if you have changed the default 'password' attribute in the AD password sync plugin configuration. See the Password attribute subsection below for resolution.

AD password sync plugin configuration

You can obtain the AD password sync plugin configuration using one of the following approaches:

  • output settings and validate.
  • export configuration.

Output settings and validate

The idmsync.exe --validate command output provides information about the current password synchronization service settings so these can be checked for any obvious errors. Essentially, this command outputs the values of the registry entries under the HKEY_LOCAL_MACHINE\SOFTWARE\ForgeRock\OpenIDM\PasswordSync registry key and provides feedback about them being valid or not. You should correct any that are invalid.

You can run this command as follows from the command prompt:

C:\> cd \path\to\PasswordSync
C:\path\to\PasswordSync> idmsync.exe --validate

Export configuration

You can export the AD password sync plugin configuration by navigating to the HKEY_LOCAL_MACHINE\SOFTWARE\ForgeRock\OpenIDM\PasswordSync registry key in Regedit, right-clicking on PasswordSync and then selecting export followed by text format. This exports the configuration to a text file.

IDM/OpenIDM configuration files

The following configuration files (located in the /path/to/idm/conf directory) are useful to check how synchronization has been configured on the IDM/OpenIDM side and compare with the AD configuration:

  • sync.json
  • managed.json
  • authentication.json
  • provisioner configuration file for AD, for example provisioner.openicf-ad.json.
  • access.js (located in the /path/to/idm/script directory).
  • any associated scripts used to manage or transform the password attribute during sync, such as a separate onUpdate script (these are typically located in the /path/to/idm/script directory).
Note

It can be very useful to add logging to your scripts to help identify the point at which an issue can occur. Refer to How do I add logging to JavaScript files in IDM/OpenIDM (All versions)? and How do I add logging to Groovy scripts in IDM/OpenIDM (All versions)? for further information.

Password attribute

If you have changed the default 'password' attribute in the AD password sync plugin configuration, you must also update access.js to use the new attribute. For example, if you have changed the attribute to adPassword, you would update the customAuthz field in the following section to replace 'password' with 'adPassword'; this attribute is case sensitive:

// Clients authenticated via SSL mutual authentication
{
 "pattern"   : "managed/user",
 "roles"     : "internal/role/openidm-cert",
 "methods"   : "patch,action",
 "actions"   : "patch",
 "customAuthz" : "isQueryOneOf({'managed/user': ['for-userName']}) && restrictPatchToFields(['adPassword'])"
},

Notification Packages registry entry

If you are using Microsoft Windows 2003, there is a known issue where the password synchronization service may not start if there are blank lines in the Notification Packages registry entry within the list of dlls. You should check this entry (under the HKLM\SYSTEM\CurrentControlSet\Control\Lsa registry key) and remove any blank lines. If you change this registry entry, you must restart the domain controller for this to take effect.

See Also

Password Synchronization Plugin Guide › Synchronizing Passwords With Active Directory

Related Training

ForgeRock Identity Management Core Concepts (IDM-400)

Related Issue Tracker IDs

OPENIDM-6676 (Active Directory Password Sync - password encryption documentation)


Copyright and TrademarksCopyright © 2018 - 2019 ForgeRock, all rights reserved.

This content has been optimized for printing.

Loading...