502 Bad Gateway or SocketTimeoutException when using IG (All versions)
The purpose of this article is to provide assistance if you encounter a 502 Bad Gateway response when routing from IG to a custom endpoint, or an endpoint in IDM or AM using a ClientHandler or ReverseProxyHandler. This response is accompanied by a "java.net.SocketTimeoutException: null" error.
1 reader recommends this article
Symptoms
You will see errors similar to the following depending on how IG is deployed.
IG in Standalone mode (IG 7 and later)
An error similar to the following is shown in the IG debug logs when the endpoint is called from IG (this may be an intermittent error):
<--- (response) id:aa167dcf-461d-513b-a942-9272bad726fb-736 --- HTTP/1.1 502 Bad Gateway [vert.x-eventloop-thread-1] WARN o.f.o.f.LogAttachedExceptionFilter @2-HBEWeb - Response [Status: 502 Bad Gateway] to `https://example.com:8443/app/test` carries an exception [txId:aa167dcf-461d-513b-a942-9272bad726fb-736] io.vertx.core.VertxException: Connection was closed
IG in Web container mode
An error similar to one of the following is shown in the web application container log (for example, catalina.out for Apache Tomcat™) when the endpoint is called from IG (this may be an intermittent error):
--- (response) id:aa167dcf-461d-513b-a942-9272bad726fb-736 --- HTTP/1.1 502 Bad Gateway Context's content : attributes: org.apache.tomcat.util.net.secure_protocol_version: "TLSv1.2" [I/O dispatcher 11] INFO mo.f.o.d.t.T.timer.top-level-handler - Elapsed time: 10025 ms [I/O dispatcher 11] WARN mo.f.o.f.LogAttachedExceptionFilter - Response [Status: 502 Bad Gateway] to 'https://idm.example.com:8444/openidm/endpoint/test' carries an exception [txId:aa167dcf-461d-513b-a942-9272bad726fb-736] java.net.SocketTimeoutException: null at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375) [http-bio-198.51.100.0-8443-exec-2] ERROR mo.f.o.handler.router.RouterHandler - no handler to dispatch to [http-bio-198.51.100.0-8443-exec-1] INFO mo.f.o.d.c.C.c.myClientHandler --- (response) id:aa167dcf-461d-513b-a942-9272bad726fb-736 --- HTTP/1.1 502 Bad Gateway Context's content : attributes: org.apache.tomcat.util.net.secure_protocol_version: "TLSv1.2" [I/O dispatcher 12] WARN o.f.openig.handler.ClientHandler - An error occurred while processing the request java.net.SocketTimeoutException: null at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375) [I/O dispatcher 12] INFO o.f.o.d.c.C.c.TlsClientHandler -Recent Changes
N/A
Causes
The connection from IG to the backend is timing out before a response has been received by the client application. This timeout typically signifies a performance issue on IG or the backend, which stops them from responding in a timely manner. By default, the timeouts for the ClientHandler and ReverseProxyHandler are set to 10 seconds, which should be sufficient in a properly tuned environment; although higher timeouts may be needed for larger requests.
Another potential cause is the load balancer or firewall between IG and the backend is preventing long-lasting connections.
Solution
This issue can usually be resolved by tuning your environment appropriately to address any performance issues. See Tuning Performance and FAQ: IG performance and tuning for further information.
Note
Performance tuning is environment specific and is outside the scope of ForgeRock support; if you want more tailored advice, consider engaging Deployment Support Services.
However, you may want to increase the timeouts for the ClientHandler or ReverseProxyHandler as well; for larger requests, you can create separate handlers with higher timeouts. And if the load balancer or firewall is suspected, there are other settings you can try in the ClientHandler or ReverseProxyHandler.
You can increase the timeouts (connectionTimeout and soTimeout) for the ClientHandler or ReverseProxyHandler in the IG route or globally. For example:
- IG Route
The configuration in a route with the ClientHandler would look similar to this with a 30 second timeout:"handler": { "name": "myClientHandler", "type": "ClientHandler", "config": { "connectionTimeout": "30 seconds", "soTimeout": "30 seconds" } }
- Globally
If you prefer to set these timeouts globally in config.json (located in the $HOME/.openig/config/ directory), you can:
- Update the timeouts for the ClientHandler or ReverseProxyHandler in the heap in config.json, for example: { "handler": { "type": "Router", "name": "_router" }, "heap": [ { "name": "myClientHandler", "type": "ClientHandler", "config": { "connectionTimeout": "30 seconds", "soTimeout": "30 seconds" } } ] }
- Define the ClientHandler or ReverseProxyHandler correctly in the route to use this global configuration, for example: "handler": "myClientHandler",
See Route and Router for further information.
If the issue is caused by the load balancer or firewall not allowing long-lasting connections, then you can try one of the following settings in the ClientHandler or ReverseProxyHandler:
- disableReuseConnection:"disableReuseConnection": true
- connectionTimeToLive:"connectionTimeToLive": "4 minutes"
- retries - configure this to retry when a downstream component returns a
502 Bad Gateway
response code:"retries": { "enabled": true, "condition": "${response.status.code == 502}" }
See ClientHandler and ReverseProxyHandler for further information on these settings.
See Also
Related Training
N/A
Related Issue Tracker IDs
N/A