How To

How do I troubleshoot WebSocket issues in Agents 5.x?

Last updated Jun 26, 2019

The purpose of this article is to provide information on troubleshooting WebSocket issues with Agent 5.x in AM.


1 reader recommends this article

Overview

Prior to troubleshooting, it is important to understand that one of the major changes introduced with 5.x Web Agents and Java Agents is an improved notification system that uses the WebSocket protocol which is significantly different from previous versions of the Agents. 

Your environment must support WebSockets in order to use AM with Agents 5.x. Among other things, the WebSocket protocol is used when AM notifies agents about configuration and session state changes. WebSockets must still be supported even if you disable notifications, or you will encounter errors. 

Read the Release Notes to understand the impacts of these changes before troubleshooting:

Initial troubleshooting checks and data to gather

You should perform the following initial checks for WebSocket conformance:

  • Ensure any load balancers or reverse proxies configured in your environment support WebSocket protocols. If they do not, you will see errors such as the following and will need to enable support in your environment:
    WARNING: Failed to create new WebSocket connection, backing off
    org.forgerock.openam.agents.notifications.websocket.WebSocketConnectionException: Failed to create connection
  • If you are using Apache™ HTTP Server as a reverse proxy, ensure you have the mod_proxy_wstunnel module as it's required for proxying the WebSocket protocol.
  • Ensure TLD scanning is either properly configured or disabled in your Apache Tomcat™ installation otherwise the appropriate libraries will not be loaded. See How do I make Tomcat startup faster? for further information.
Note

Configuring load balancers and reverse proxies is outside the scope of ForgeRock support; if you want more tailored advice, consider engaging Deployment Support Services

If all initial checks went well, you should gather the following data for troubleshooting and refer to the rest of the article for common issues:

Notifications endpoint responses

The notifications endpoint applies some login processing logic via a servlet filter. If everything is ok, you will see a 101 response, which indicates the request is valid, it has been upgraded successfully and the WebSockets connection is working correctly. If there is an issue, you will see one of the following responses instead: 401, 403 or 404:

Checking the WebSockets jars are being loaded

You can check the jars are being loaded as follows:

  1. Navigate to the Tomcat bin directory and run the noisy command, for example:
    $ cd /path/to/tomcat/bin
    $ lsof | grep websocket
    • If the WebSocket jars are being loaded, you will see something similar to the following.
      java    1014      root  mem       REG                8,1   225632 2097375 /usr/local/tomcat/lib/tomcat-websocket.jar
      java    1014      root  mem       REG                8,1    36905 2097376 /usr/local/tomcat/lib/websocket-api.jar
      java    1014      root   38r      REG                8,1    36905 2097376 /usr/local/tomcat/lib/websocket-api.jar
      ...
      
    • If the WebSocket jars are not being loaded, you won't see them listed.
  2. You can also verify by adding the following option to the Tomcat startup script (setenv.sh) to view further details about the Java® WebSocket API loading:
    JAVA_OPTS=-verbose:class
  3. Restart the web container.
  4. Check the catalina.out log file for WebSocket details. For example, you would expect to see things like the following if the WebSocket API is available:
    $ cat ../logs/catalina.out | grep websocket
    
    [Loaded org.forgerock.openam.notifications.websocket.JsonValueDecoder from file:/path/to/tomcat/webapps/openam/WEB-INF/lib/openam-notifications-websocket-6.5.0.jar]
    
    [Loaded org.forgerock.openam.notifications.websocket.NotificationsWebSocketConfigurator from file:/path/to/tomcat/webapps/openam/WEB-INF/lib/openam-notifications-websocket-6.5.0.jar]
    
    [Loaded javax.websocket.EncodeException from file:/path/to/tomcat/lib/websocket-api.jar]
    
    ...
    

The Java WebSocket API is bundled with Tomcat 7 and 8 so should be available. 

Note

The openam-notifications-websocket-x.x.x.jar is required for WebSockets to work. If it is missing, you will see 404 responses. If this happens, verify your Tomcat configuration or contact your System Administrator for further assistance.

Running the Agent validation tool (Agents 5.5 and later)

The Agent validation tool was introduced in Agents 5.5 and later. This tool performs a number of checks including WebSocket tests and can be run as follows:

$ agentadmin --V agent_instance

The results of the tests are output to the command line and shows tests as ok or not ok depending on whether they passed, for example:

... 
validate_system_resources: ok 
validate_session_profile: skipped 
Agent websocket open error: error (6) 
validate_websocket_connection: not ok 
...

Further information about the tests and detailed results can be found in the Validator log (validate_xx.log located in the agent /log directory).

See Web Agents User Guide › agentadmin or Java Agents User Guide › agentadmin for further information.

Testing the WebSocket connection via curl

You can test the WebSocket connection by sending the agent's token to the notifications endpoint using curl. This will generate a response similar to what is output in the Validator log. Ensure you use the correct agent name, password, AM FQDN URL and cookie name: 

  1. Authenticate as the agent to return the agent's token:
    $ curl -X POST -H "X-OpenAM-Username: webagentname" -H "X-OpenAM-Password: webagentpassword" -H "Content-Type: application/json" -H "Accept-API-Version: resource=2.1" http://host1.example.com:8080/openam/json/realms/root/authenticate?authIndexType=module&authIndexValue=Application
    
    Example response:
    { "tokenId": "AQIC5wM2LY4SfcxsuvGEjcsppDSFR8H8DYBSouTtz3m64PI.*AAJTSQACMDIAAlNLABQtNTQwMTU3NzgxODI0NzE3OTIwNAEwNDU2NjE0*", "successUrl": "/openam/console", "realm": "/" }
    
  2. Send the agent's token to the notifications endpoint:
    $ curl -v --include --no-buffer -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: agent.example.com:8080" -H "Origin: http://notagent.example.com:8080" -H "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" -H "Sec-WebSocket-Version: 13" -H "iPlanetDirectoryPro: AQIC5wM2LY4Sfcxs...EwNDU2NjE0*" http://host1.example.com:8080/openam/notifications
    Example 101 response when the WebSocket connection is working correctly:
    *   Trying 198.51.100.0...
    * TCP_NODELAY set
    * Connected to host1.example.com (198.51.100.0) port 8080 (#0)
    > GET /openam/notifications HTTP/1.1
    > Host: agent.example.com:8080
    > User-Agent: curl/7.54.0
    > Accept: */*
    > Connection: Upgrade
    > Upgrade: websocket
    > Origin: http://notagent.example.com:8080
    > Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==
    > Sec-WebSocket-Version: 13
    > iPlanetDirectoryPro: AQIC5wM2LY4SfcxsuvGEjcsppDSFR8H8DYBSouTtz3m64PI.*AAJTSQACMDIAAlNLABQtNTQwMTU3NzgxODI0NzE3OTIwNAEwNDU2NjE0*
    
    >
    
    < HTTP/1.1 101
    HTTP/1.1 101
    < X-Frame-Options: SAMEORIGIN
    X-Frame-Options: SAMEORIGIN
    < Upgrade: websocket
    Upgrade: websocket
    < Connection: upgrade
    Connection: upgrade
    < Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=
    Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=
    < Date: Fri, 15 Mar 2019 17:38:15 GMT
    Date: Fri, 15 Mar 2019 17:38:15 GMT

401/403 response from the notifications endpoint

A 401 or 403 response means the agent has failed to establish a WebSocket connection with AM.

You will see a 403 Access Forbidden response in the browser and a 401 response in the logs:

  • Agents 5.5 and later: in the Validator log file, for example:
    2019-03-19 09:27:13  Agent websocket open error: forbidden (3)
    am_timer(): getaddrinfo took 0 seconds
    http request to host1.example.com:8080
    GET /openam/notifications HTTP/1.1
    Host: host1.example.com:8080
    Upgrade: websocket
    Connection: upgrade
    Sec-WebSocket-Key: e7a7rzfsDr7cl3E8EWcyaA==
    Sec-WebSocket-Version: 13
    Sec-WebSocket-Protocol: v1.notifications.forgerock.org
    
    IPlanetDirectoryPro: IHvS8OJUmnyX8vJzI34I5tnuzio.*AAJTSQACMDIAAlNLABx6SkZ2TzVlcUQ5WUlLdFdCSzd0aFc0aTdlUTQ9AAR0eXBlAANDVFMAAlMxAAIwMQ..*
    
    X-ForgeRock-TransactionId: f8525ddb-a383-cb43-ac29-4324b21ebaeb/1
    
    
    http response 401 from host1.example.com:8080
    X-Frame-Options: SAMEORIGIN
    Content-Length: 0
    Date: Tue, 19 Mar 2019 09:27:13 GMT
    websocket response 401
    
  • All versions: in the system_0.log (located in the agent/log directory), for example:
    2019-03-19 09:27:13 GMT ERROR  [fead55aa-f7a6-5641-5b4f-d6d71ebf518d]: websocket response 401
    2019-03-19 09:27:13 GMT WARNING [fead55aa-f7a6-5641-5b4f-d6d71ebf518d]: websocket: open status: forbidden
    2019-03-19 09:27:13 GMT WARNING [3844:3080]: session for agent / webagent was not removed

A common reason for a 401 response is a mismatch between the agent cookie name and the cookie name on the AM server. This is because the agent cookie name (com.sun.identity.agents.config.cookie.name) is used to construct the request sent to the notifications endpoint and must match what AM is expecting. The AM server default cookie name and agent cookie name are both set to iPlanetDirectoryPro by default. If you rename this cookie on the AM server, then you need to update the same on the Agent side.

For example, an agent with a cookie name of IPlanetDirectoryPro will not be able to authenticate to upgrade the request if the AM server has a cookie name of exampleCookie. 

You can check the AM cookie as follows:

  • Agents 5.5 and later: use the Validator log to check the call made to serverinfo:
    http response 200 from host1.example.com:8080
    X-Frame-Options: SAMEORIGIN
    Cache-Control: no-cache
    Content-API-Version: resource=1.1
    ETag: "812974925"
    X-Content-Type-Options: nosniff
    Content-Type: application/json;charset=UTF-8
    Content-Length: 551
    Date: Tue, 19 Mar 2019 09:41:29 GMT
    Connection: close
    {"_id":"*","_rev":"812974925","domains":["example.com"],"protectedUserAttributes":[],"cookieName":"exampleCookie","
    
  • All versions: use the following REST call:
    $ curl -s 'http://host1.example.com:8080/openam/json/serverinfo/*' | tr ',' '\n' | grep cookieName
    
    "cookieName":"exampleCookie"

If your cookies do not match, you can update them them as described in How do I change the session cookie name for AM/OpenAM and Policy Agents (All versions)?

404 response from the notifications endpoint

A 404 response typically means that the WebSocket request has not been upgraded because the AM notifications servlet filter will never throw a 404. If you get a 404, you at least know that the request being sent to the notification servlet filter is valid. A 404 can be caused by a number of things, including network infrastructure; for example, incorrectly configured load balancers or reverse proxies.

To troubleshoot a 404, you can:

  • Make a direct curl request to the notifications endpoint as described in the Testing the WebSocket connection via curl section. This will provide more details and may give you a clue as to what is going wrong.
  • Rename the agent cookie name to verify that you get a 401 response as expected. If you do get a 401, it confirms the servlet filter logic is being applied correctly and means it is a network issue or a Tomcat issue. One possible Tomcat issue is if the openam-notifications-websocket-x.x.x.jar is missing. If this happens, you must redploy AM.

See Also

Troubleshooting AM/OpenAM and Policy Agents

Web Agents User Guide › Notification System

Web Agents User Guide › Preparing the Environment for Reverse Proxies

Web Agents User Guide › Troubleshooting

Java Agents User Guide › Notification System

Java Agents User Guide › Preparing the Environment for Reverse Proxies

Java Agents User Guide › Troubleshooting

RFC 6455 - The WebSocket Protocol

Related Training

N/A

Related Issue Tracker IDs

AMAGENTS-2195 (Add a Cookie Check to Validator)

AMAGENTS-1216 (Agents should get iPlanetDirectoryPro name from OpenAM server)



Copyright and TrademarksCopyright © 2019 ForgeRock, all rights reserved.
Loading...