Solutions

Issues with upgrades, Amster exports and registering clients (OAuth2, OIDC and RADIUS) or agents with reference to sunserviceID in AM (All versions)

Last updated Nov 15, 2019

The purpose of this article is to provide assistance if you experience issues with upgrades failing, Amster export failures and problems registering large numbers of clients or agents in AM. You will see "The search results cannot be sorted because the given search request is not indexed" messages when this happens with reference to the sunserviceID attribute. This issue only occurs if you are using an external DS for your configuration store.


Symptoms

You will encounter errors relating to the sunserviceID attribute in a number of situations, including (but not limited to):

  • Upgrading AM.
  • Exporting Amster configurations.
  • Registering large numbers of clients (OAuth2, OIDC or RADIUS clients).
  • Creating large numbers of agents.

Errors similar to the following are shown in AM CoreSystem debug log when this happens:

  • Upgrade example:
    ERROR: An error occurred while trying to look for WebAgents to upgrade
    SMSException Exception Code:5
    Message:Unexpected LDAP exception occurred.
    --------------------------------------------------
    The lower level exception message
    Unavailable Critical Extension: The search results cannot be sorted because the given search request is not indexed
    
  • Registering large numbers of clients and exporting Amster example:
    WARNING: ::SmsCollectionProvider:: SMSException on query
    SMSException Exception Code:5
    Message:Unexpected LDAP exception occurred.
    --------------------------------------------------
    The lower level exception message
    Unavailable Critical Extension: The search results cannot be sorted because the given search request is not indexed
    The lower level exception:
    Unavailable Critical Extension: The search results cannot be sorted because the given search request is not indexed
    

You will find a corresponding error in the DS logs, which provides more insight. For example, one referencing sunserviceID=WebAgent which may be seen on upgrade:

{"eventName":"DJ-LDAP","client":{"ip":"	192.0.2.0","port":8080},"server":{"ip":"192.0.2.10","port":1636},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":67,"msgId":22107,"dn":"ou=default,ou= OrganizationConfig,ou=1.0,ou=AgentService,ou=services,dc=openam,dc=forgerock,dc=com","scope":"one","filter":"(&(&(objectclass=top)(ou=*))(&(objectclass=top)(sunserviceID= WebAgent)))","attrs":["o"]},"transactionId":"d1e21e68-8f64-fa1e-4fbd-16e77cc475ef-62545461","response":{"status":"FAILED","statusCode":"12","elapsedTime":13,"elapsedTimeUnits":"MILLISECONDS","detail":"The search results cannot be sorted because the given search request is not indexed","additionalItems":"unindexed","nentries":0},"timestamp":"2019-10-24T04:15:11.424Z","_id":"d1e21e68-8f64-fa1e-4fbd-16e77cc475ef-62545464"}

When an Amster export fails (with the --failOnError flag set), you will get the following response, which corresponds to the above log snippets:

500 Internal Server Error: Unable to query SMS config: Unexpected LDAP exception occurred.

Recent Changes

Created a large number of clients (OAuth2, OIDC or RADIUS).

Created a large number of agents.

Created a large number of SAML2 entities.

Causes

sunserviceID is the attribute type that stores AM configuration data. By default, it is not indexed.

Each time AM performs a search against things like OAuth2 clients, agents or SAML2 entities, DS tries to do a server-side sort and fails because sunserviceID is not indexed. AM performs searches in various situations, for example, when registering an OAuth2 client, AM will perform a search against existing clients, or when upgrading, AM will perform a search against existing clients, agents and SAML2 entities. If you have a large number of clients, agents or SAML2 entities (for example, running into the thousands), this unindexed attribute will cause the errors seen in the Symptoms section.

Solution

You can workaround this issue by indexing sunserviceID and increasing the search limit as follows:

  1. Create an index for sunserviceID using dsconfig, for example:
    $ ./dsconfig create-backend-index --port 4444 --hostname ds1.example.com --bindDN "cn=Directory Manager" --bindPassword password --backend-name userRoot --index-name sunserviceID --set index-type:equality --trustAll --no-prompt
  2. Increase the search limit for this index using dsconfig. It is suggested you start with a limit of 10000, but you may need to increase this further. Do not set the index-entry-limit to the total size of the backend. For example:
    $ ./dsconfig set-backend-index-prop --port 4444 --hostname ds1.example.com --bindDN "cn=Directory Manager" --bindPassword password --backend-name userRoot --index-name sunserviceID --set index-entry-limit:10000 --no-prompt --trustAll
  3. Rebuild the index using the rebuild-index command. For example:
    $ ./rebuild-index --port 4444 --hostname ds1.example.com --bindDN "cn=Directory Manager" --bindPassword password --baseDN dc=openam,dc=forgerock,dc=org --index sunserviceID --start 0 --trustAll

Important Considerations

Indexing sunserviceID and increasing the index-entry-limit is effectively a workaround for having a large number of clients, agents or entities. Ideally, you should avoid having large numbers of clients/agents. For OAuth2 clients, it is recommended that you have a one-to-many mapping of clients, where one client is associated with many devices/users rather than one client per device/user, which is also best practice advice for keeping clients manageable and easier to administrate.

If you have a large number of clients/agents/entities, it is likely you will hit other scaling and performance issues. You should consider the following, and ensure you performance test and tune your intended setup (AM and DS) before going into production:

  • Index entry limits are set to 4000 by default. Increasing this limit beyond the default (as suggested here) is likely to impact performance as DS will need to maintain entries for very large indexes. There is a blog that explains this setting in more detail: Explaining index-entry-limit in ForgeRock Directory Services.
  • The cursor-entry-limit advanced property (default 100000) specifies the maximum number of entry IDs that can be retrieved by cursoring through an index during a search. Depending on the number of entries, you may need to increase this limit as well. Do not set the cursor-entry-limit to the total size of the backend. See Configuration Reference › cursor-entry-limit for further information.  
  • DS must be tuned for search performance as there are default limits on search operations. You may need to increase these limits (ds-rlim-size-limit and ds-rlim-lookthrough-limit) to allow more search results to be returned. See Administration Guide › Limiting Search Resources for further information on these settings. Additionally, increasing these settings will require more heap, so you will need to ensure your DS heap is tuned appropriately.
  • Clients and agents are stored in in-memory caches, which can lead to heap sizing issues as the number of clients/agents increase. You should ensure your AM heap is tuned appropriately.
  • The AM console cannot cope with large numbers of clients/agents/entities and will hang, respond very slowly or the server will crash with OutOfMemory errors as numbers increase. You should avoid administering clients/agents/entities through the console if this happens.
  • RADIUS clients only: you can change the maximum number of RADIUS clients that can be cached concurrently on an AM server (default 5000) by configuring the org.forgerock.openam.radius.server.context.cache.size advanced server property. See RADIUS Server Guide › RADIUS Server Limitations for further information.
Note

Performance tuning recommendations are outside the scope of ForgeRock support. if you want more tailored advice, consider engaging Deployment Support Services.

See Also

Unindexed searches causing slow searches and poor performance on DS/OpenDJ (All versions) server

Best practice for JVM Tuning

Performance tuning and monitoring ForgeRock products

Administration Guide › Indexing Attribute Values

Related Training

N/A

Related Issue Tracker IDs

OPENAM-15572 (Document sunserviceID index needs to be created under certain case)

OPENAM-15406 (Improve performance for registering and maintaining OAuth2 Clients)

OPENAM-14825 (OAuth2 Dynamic Registration with Software Statement triggers objectClass=* search)

OPENAM-14271 (sunserviceID index is not created and it needs to be indexed )

OPENAM-3996 (AgentsRepo search is suboptimal)



Copyright and TrademarksCopyright © 2019 ForgeRock, all rights reserved.

Recommended Books

Loading...