Solutions
ForgeRock Identity Platform
Does not apply to Identity Cloud

AM 5.x or 6.x fails to connect to the user data store when anonymous access is disabled in DS

Last updated Apr 13, 2021

The purpose of this article is to provide assistance if you experience connectivity failures between AM and the user data store (Identity Repository). Common errors logged include "Connection factory became offline" error and data store module fails with "ldap errorcode=91". This issue will only affect you if you are using DS for your user data store and anonymous access is disabled for that DS instance.


1 reader recommends this article

Symptoms

Users cannot authenticate to AM protected applications and you cannot see your users on the Identities page (previously the Subjects tab) in the console.

An error similar to the following is shown in the IdRepo log when this happens:

LDAPUtils:12/06/2020:03:46:31:049 PM CST: Thread[SystemTimerPool,5,main]  **********************************************  LDAPUtils:12/06/2020:03:46:11:048 PM CST: Thread[SystemTimerPool,5,main]  ERROR: Connection factory became offline:  AuthenticatedConnectionFactory(HeartBeatConnectionFactory(LDAPConnectionFactory(host1.example.com/203.0.113.1:1636)),  SimpleBindRequest(name=uid=openam-admin,ou=sysaccounts,dc=example,dc=com, authentication=simple, controls=[]))  org.forgerock.opendj.ldap.ConnectionException: Connect Error: Connection refused     at org.forgerock.opendj.ldap.ErrorResultException.newErrorResult(ErrorResultException.java:163)     at org.forgerock.opendj.ldap.ErrorResultException.newErrorResult(ErrorResultException.java:125)     at com.forgerock.opendj.ldap.LDAPConnectionFactoryImpl$CompletionHandlerAdapter.adaptConnectionException(LDAPConnectionFactoryImpl.java:183)     at com.forgerock.opendj.ldap.LDAPConnectionFactoryImpl$CompletionHandlerAdapter.failed(LDAPConn  ...  Caused by: java.net.ConnectException: Connection refused     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)     at org.glassfish.grizzly.nio.transport.TCPNIOConnectorHandler.onConnectedAsync(TCPNIOConnectorHandler.java:206)

One of the following errors is shown in the Authentication log, depending on which authentication module you are using:

  • LDAP authentication module: amAuthLDAP:12/06/2020 03:46:26:919 PM BST: Thread[http-bio-8080-exec-57,5,main]  WARNING: Search for User error:  org.forgerock.opendj.ldap.ConnectionException: Connect Error: No operational connection factories available     at org.forgerock.opendj.ldap.ErrorResultException.newErrorResult(ErrorResultException.java:210)  ...  Caused by: org.forgerock.opendj.ldap.ConnectionException: Server Connection Closed: Heartbeat failed     at org.forgerock.opendj.ldap.ErrorResultException.newErrorResult(ErrorResultException.java:210)  ...  Caused by: org.forgerock.opendj.ldap.ErrorResultException: Unwilling to Perform: Rejecting the requested operation because the connection has not been authenticated     at org.forgerock.opendj.ldap.ErrorResultException.newErrorResult(ErrorResultException.java:232)
  • Data Store authentication module: amAuth:12/06/2020 03:28:39:704 PM BST: Thread[http-bio-18080-exec-2,5,main]  Exception :  com.sun.identity.authentication.spi.AuthLoginException: Authentication Failed  Plug-in org.forgerock.openam.idrepo.ldap.DJLDAPv3Repo encountered a ldap exception. ldap errorcode=91     at com.sun.identity.authentication.modules.datastore.DataStore.process(DataStore.java:165)     at com.sun.identity.authentication.spi.AMLoginModule.wrapProcess(AMLoginModule.java:1023)     at com.sun.identity.authentication.spi.AMLoginModule.login(AMLoginModule.java:1197)  ...  Caused by: Message:Plug-in org.forgerock.openam.idrepo.ldap.DJLDAPv3Repo encountered a ldap exception. ldap errorcode=91     at org.forgerock.openam.idrepo.ldap.DJLDAPv3Repo.newIdRepoException(DJLDAPv3Repo.java:2478)     at org.forgerock.openam.idrepo.ldap.DJLDAPv3Repo.handleErrorResult(DJLDAPv3Repo.java:2451)     at org.forgerock.openam.idrepo.ldap.DJLDAPv3Repo.getDN(DJLDAPv3Repo.java:2316)     at org.forgerock.openam.idrepo.ldap.DJLDAPv3Repo.findDNForAuth(DJLDAPv3Repo.java:2266)

Correspondingly, the DS access log shows a CONNECT and DISCONNECT without a SEARCH occurring:

{"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"CONNECT","connId":2},"transactionId":"0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS"},"timestamp":"2020-08-15T16:44:49.400Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-1"} {"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"UNBIND","connId":2,"msgId":3},"transactionId":"5009191b-a09a-4c7f-84cd-e240b2810a67-8","timestamp":"2020-08-15T16:44:49.729Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-10"} {"eventName":"DJ-LDAP","client":{"ip":"203.0.113.0","port":52597},"server":{"ip":"203.0.113.0","port":1389},"request":{"protocol":"LDAP","operation":"DISCONNECT","connId":2},"transactionId":"0","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","reason":"Client Unbind"},"timestamp":"2020-08-15T16:44:49.734Z","_id":"5009191b-a09a-4c7f-84cd-e240b2810a67-12"}

Recent Changes

Disabled anonymous access in DS using one of the following commands depending on version:

  • DS 6.5.x: $ ./dsconfig --hostname ds1.example.com --port 4444 --bindDN "cn=Directory Manager" --bindPassword password set-global-configuration-prop --set unauthenticated-requests-policy:reject --trustAll --no-prompt
  • Pre-DS 6.5: $ ./dsconfig --hostname ds1.example.com --port 4444 --bindDN "cn=Directory Manager" --bindPassword password set-global-configuration-prop --set reject-unauthenticated-requests:true --trustAll --no-prompt

Causes

Completely disabling anonymous access in DS prevents the SEARCH request from succeeding if AM uses the heartbeat mechanism (which it does by default) and causes connections from AM to fail. 

See How does AM 5.x and 6.x use anonymous access calls to DS? for further information.

Solution

This issue can be resolved by allowing anonymous access in DS and then using Access Control Instruction (ACI) to ALLOW anonymous binds for the heartbeat search but disallow other anonymous searches. See Administration Guide › ACI: Disable Anonymous Access for further information.

  • You can use one of the following commands to allow anonymous access depending on version:
    • DS 6.5.x: $ ./dsconfig --hostname ds1.example.com --port 4444 --bindDN "cn=Directory Manager" --bindPassword password set-global-configuration-prop --set unauthenticated-requests-policy:allow --trustAll --no-prompt
    • Pre-DS 6.5: $ ./dsconfig --hostname ds1.example.com --port 4444 --bindDN "cn=Directory Manager" --bindPassword password set-global-configuration-prop --set reject-unauthenticated-requests:false --trustAll --no-prompt
  • You should construct an ACI that allows anonymous binds for the heartbeat search, but only allows access to the root DSE; the following is an example SEARCH request AM normally sends for the heartbeat: SEARCH REQ conn=6 op=2468 msgID=2469 base="" scope=baseObject filter="(objectClass=*)" attrs="1.1" 
  • You can reproduce the heartbeat request using a ldapsearch such as the following: $ ./ldapsearch --port 50389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN "" --searchScope base "objectClass=*" "1.1" 

Alternatively, you can disable heartbeats in AM, although this approach is not recommended as it degrades failover handling. You can disable heartbeats in the data store as follows; you must disable heartbeats for data stores in all affected realms:

  • Console: navigate to: Realms > [Realm Name] > Data Stores > [Data Store Name] > LDAP Connection Heartbeat Interval and enter 0.
  • Amster: follow the steps in How do I update property values in AM (All versions) using Amster? with these values:
    • Entity: OpenDJ
    • Property: openam-idrepo-ldapv3-heartbeat-interval
  • ssoadm: enter the following command: $ ./ssoadm update-datastore -e [realmname] -m [datastorename] -u [adminID] -f [passwordfile] -a openam-idrepo-ldapv3-heartbeat-interval=0 replacing [realmname], [datastorename], [adminID] and [passwordfile] with appropriate values.
Caution

If you choose to disable heartbeats rather than using access controls, you should test this in your pre-production environment first to ensure there are no unwanted side effects.

See Also

How does AM 5.x and 6.x use anonymous access calls to DS?

Data stores in AM

Setup and Maintenance Guide › Setting Up Identity Data Stores

Related Training

N/A

Related Issue Tracker IDs

N/A


Copyright and Trademarks Copyright © 2021 ForgeRock, all rights reserved.