Solutions
ForgeRock Identity Platform
Does not apply to Identity Cloud

Web Agent (All versions) system log grows rapidly with WebSocket 401 errors

Last updated Jun 3, 2021

The purpose of this article is to provide assistance if your Web Agent system log grows rapidly with WebSocket 401 errors.


Symptoms

You see an increasing number of error messages in the Web Agent system_n.log, with WebSocket 401 errors occurring on a regular basis, for example, every 60 minutes. 

Errors similar to the following are shown:2021-03-13 17:11:52 GMT ERROR [7faefdbb-f13d-364c-d47e-5d2b19657fca]: websocket response 401 2021-03-13 17:11:52 GMT WARNING [7faefdbb-f13d-364c-d47e-5d2b19657fca]: websocket: open status: forbidden 2021-03-13 17:11:52 GMT ERROR [7faefdbb-f13d-364c-d47e-5d2b19657fca]: websocket response 401 2021-03-13 17:11:52 GMT WARNING [7faefdbb-f13d-364c-d47e-5d2b19657fca]: websocket: open status: forbidden 2021-03-13 17:11:52 GMT ERROR [7faefdbb-f13d-364c-d47e-5d2b19657fca]: websocket response 401 2021-03-13 17:11:52 GMT WARNING [7faefdbb-f13d-364c-d47e-5d2b19657fca]: websocket: open status: forbidden 2021-03-13 17:11:52 GMT ERROR [7faefdbb-f13d-364c-d47e-5d2b19657fca]: websocket response 401 2021-03-13 17:11:52 GMT WARNING [7faefdbb-f13d-364c-d47e-5d2b19657fca]: websocket: open status: forbidden 2021-03-13 17:11:52 GMT INFO [7faefdbb-f13d-364c-d47e-5d2b19657fca]: websocket: graceful reconnect failure 2020-03-13 17:11:52 GMT WARNING [0x7f6004fe7700:11181]: session for agent / agentName was not removed 2020-03-13 17:11:53 GMT INFO [7faefdbb-f13d-364c-d47e-5d2b19657fca]: connection established to http://host1.example.com:8080/openam

Recent Changes

You’ve changed the default WebSocket connection interval and/or the session timeout settings.

Causes

This behavior is seen and is expected when the WebSocket connection interval and session timeout settings are set in such a way that they create lots of new sessions, which will result in numerous log messages especially on a busy server.

For example, you might see this behavior when the WebSocket connection interval is less than 30 minutes, and the session timeout is set to 60 minutes. If the agent is idle for 60 minutes, its session gets terminated by AM and you'll see 401 errors in the Web Agent/system_n.log. 

Solution

This issue can be resolved by adjusting one or both of the following settings as needed:

  • The WebSocket connection interval in the web agent's Global properties.
  • The session timeout value in the Server Defaults settings. Potentially, increasing this value to 2880 (2 days) can circumvent the issue.

Change the WebSocket connection interval

You can change the WebSocket connection interval value using either the console or ssoadm:

  • Console: navigate to: Realms > [Realm Name] > Applications > Agents > Web > [Agent ID] > Global >  org.forgerock.openam.agents.config.balance.websocket.connection.interval.in.minutes and amend the required number of minutes, for example, 30 (the default).
  • ssoadm: enter the following command: $ ./ssoadm update-agent -e [realmname] -b [agentname] -u [adminID] -f [passwordfile] -a org.forgerock.openam.agents.config.balance.websocket.connection.interval.in.minutes=[minutes]replacing [realname], [agentname] [adminID], [passwordfile] and [minutes] with appropriate values.

Change the session timeout setting

You can change the session timeout property value using either the console, Amster or ssoadm:

  • Console: navigate to: Configure > Server Defaults > Advanced > com.iplanet.am.session.agentSessionIdleTime and amend the required number of minutes, for example, 2880 (2 days).
  • Amster: follow the steps in How do I update property values in AM (All versions) using Amster? with these values:
    • Entity: DefaultAdvancedProperties
    • Property: com.iplanet.am.session.agentSessionIdleTime
  • ssoadm: enter the following command: $ ./ssoadm update-server-cfg -s default -u [adminID] -f [passwordfile] -a com.iplanet.am.session.agentSessionIdleTime=[minutes]replacing [adminID], [passwordfile] and [minutes] with appropriate values.
Note

You must restart the web application container in which AM runs to apply these configuration changes.

See Also

Out of Memory exception causes AM (All versions) to hang due to increasing number of open Agent sessions

How do I troubleshoot WebSocket issues in Agents (All versions)?

User Guide › Global Properties

Reference › Advanced Properties

Related Training

N/A

Related Issue Tracker IDs

AMAGENTS-4340 (Log level inappropriate when agent reconnects after its token expires)

AMAGENTS-4050 (Saving agent config in quick succession whilst under load can lead to continual 403s)

AMAGENTS-3986 (Web agent is not shutting down correctly, leaving worker processes waiting on semaphores.)



Copyright and TrademarksCopyright © 2021 ForgeRock, all rights reserved.
Loading...