List index out of range: -1 error in IDM (All versions) when deployed in a cluster

Symptoms

An error similar to one of the following is shown in the IDM logs: 

• Error performing cluster manager thread logic:[81] Nov 01, 2022 4:29:32.172 PM org.forgerock.openidm.cluster.ClusterManager$ClusterManagerThread$1 run SEVERE: Error performing cluster manager thread logic org.forgerock.json.JsonValueException: /: List index out of range: -1   at org.forgerock.json.JsonValue.put(JsonValue.java:1074)    at org.forgerock.json.JsonValue.put(JsonValue.java:1129)    at org.forgerock.openidm.repo.opendj.impl.GenericDJTypeHandler.transformFullObjectOutput(GenericDJTypeHandler.java:160)    at org.forgerock.openidm.repo.opendj.impl.GenericDJTypeHandler.lambda$new$0(GenericDJTypeHandler.java:93)    at org.forgerock.util.promise.Promises$CompletedPromise.then(Promises.java:191)    at org.forgerock.util.promise.Promises$CompletedPromise.then(Promises.java:156) ... Caused by: org.forgerock.json.JsonValueException: /: List index out of range: -1
• Failed to query cluster nodes. Lock balancing aborted:[140] Nov 01, 2022 4:29:32.172 PM org.forgerock.openidm.sync.impl.queue.QueueConsumerFactory lambda$new$1 SEVERE: Failed to query cluster nodes. Lock balancing aborted. org.forgerock.json.resource.InternalServerErrorException: /: List index out of range: -1   at org.forgerock.opendj.rest2ldap.Rest2Ldap.asResourceException(Rest2Ldap.java:505)    at org.forgerock.opendj.ldap.Utils.lambda$toPromise$6(Utils.java:234)    at io.reactivex.rxjava3.internal.observers.ConsumerSingleObserver.onError(ConsumerSingleObserver.java:46)    at io.reactivex.rxjava3.internal.operators.flowable.FlowableLastSingle$LastSubscriber.onError(FlowableLastSingle.java:94)    at io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.onError(BasicFuseableSubscriber.java:101)    at io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.onError(BasicFuseableSubscriber.java:101)    at io.reactivex.rxjava3.internal.operators.flowable.FlowableDoOnEach$DoOnEachConditionalSubscriber.onError(FlowableDoOnEach.java:268)    at io.reactivex.rxjava3.internal.subscribers.BasicFuseableConditionalSubscriber.fail(BasicFuseableConditionalSubscriber.java:110)    at io.reactivex.rxjava3.internal.operators.flowable.FlowableDoOnEach$DoOnEachConditionalSubscriber.tryOnNext(FlowableDoOnEach.java:245)    at io.reactivex.rxjava3.internal.operators.flowable.FlowableObserveOn$ObserveOnConditionalSubscriber.runAsync(FlowableObserveOn.java:639)    at io.reactivex.rxjava3.internal.operators.flowable.FlowableObserveOn$BaseObserveOnSubscriber.run(FlowableObserveOn.java:176) ... Caused by: org.forgerock.json.JsonValueException: /: List index out of range: -1
• Resource exception: 500 Internal Server Error:[72] Nov 01, 2022 4:29:32.172 PM org.forgerock.openidm.servlet.internal.ResourceFilters$3 lambda$handleRequestWithLogging$8 WARNING: Resource exception: 500 Internal Server Error: "/: List index out of range: -1" org.forgerock.json.resource.InternalServerErrorException: /: List index out of range: -1  at org.forgerock.opendj.rest2ldap.Rest2Ldap.asResourceException(Rest2Ldap.java:505) ... Caused by: org.forgerock.json.JsonValueException: /: List index out of range: -1

The following response is received when querying the cluster endpoint using REST: { "code": 500, "reason": "Internal Server Error", "message": "/: List index out of range: -1" }

Recent Changes

N/A

Causes

There are multiple fr-idm-json attributes on one or more entries in the DS repository. These duplicate attributes could be within the config tables (for example, a schedule trigger) or within the user data itself. 

The fr-idm-json attribute is meant to be a single-valued attribute, so it should only exist once on any entry. Improvements have been made in DS 7.2 to improve how DS handles updates to the fr-idm-json attribute to prevent this situation from occurring.

Solution

This issue can be resolved by manually removing the duplicate fr-idm-json attributes from entries in the DS repository. You should work with your DS administrator to remove the oldest attributes where there are duplicates and just leave the newest one. 

Once you have removed the duplicate entries, you should consider upgrading your repository to DS 7.2 or later to take advantage of the improvements made to the way this attribute is handled; you can download this from Backstage. 

Ensure your resulting IDM and DS versions remain compatible: What versions of DS are compatible with IDM?

Finding duplicate entries

You can check for duplicate fr-idm-json attributes by performing a ldapsearch on each of the DS instances, for example:

• DS 7.1 and later: $ ./ldapsearch --hostname localhost --port 1636 --useSsl --usePkcs12TrustStore /path/to/ds/config/keystore --trustStorePassword:file /path/to/ds/config/keystore.pin --bindDN uid=admin --bindPassword password --basedn dc=openidm,dc=forgerock,dc=com -s sub "(fr-idm-json=*)" dn fr-idm-json
• DS 7: $ ./ldapsearch --hostname localhost --port 1636 --useSsl --usePkcs12TrustStore /path/to/ds/config/keystore --trustStorePasswordFile /path/to/ds/config/keystore.pin --bindDN uid=admin --bindPassword password --basedn dc=openidm,dc=forgerock,dc=com -s sub "(fr-idm-json=*)" dn fr-idm-json

Example response showing duplicate fr-idm-json attributes on the uid=node1,ou=states,ou=cluster,dc=openidm,dc=forgerock,dc=com entry:dn: uid=node1,ou=states,ou=cluster,dc=openidm,dc=forgerock,dc=com fr-idm-json: {"recoveryAttempts":1,"detectedDown":"0000000000000000000","type":"state","recoveryFinished":"0000000000000000000","instanceId":"node1","startup":"0000001668594727000","recoveringInstanceId":"null","state":1,"recoveringTimestamp":"0000000000000000000","recoveryStarted":"0000000000000000000","shutdown":"0000000000000000000","timestamp":"0000001668595405000"} fr-idm-json: {"recoveryAttempts":0,"detectedDown":"0000000000000000000","type":"state","recoveryFinished":"0000000000000000000","instanceId":"node1","startup":"0000001668088349000","recoveringInstanceId":null,"state":1,"recoveringTimestamp":"0000000000000000000","recoveryStarted":"0000000000000000000","shutdown":"0000000000000000000","timestamp":"0000001668089407000"}If required, you can use an epoch converter to see which fr-idm-json timestamps are the oldest.

See Also

Best practice for clustering in IDM

IDM in a cluster

fr-idm-json

Related Training

N/A

Related Issue Tracker IDs

OPENDJ-8619 (Rest2LDAP: Do updates to single valued attributes as a MOD Replace instead of MOD Delete/Add)