Troubleshooting

ForgeRock provides support services, professional services, training through ForgeRock University, and partner services to help you set up and maintain your deployments. For information, refer to Getting support.

When you are trying to solve a problem, save time by asking the following questions:

How do you reproduce the problem?
What behavior do you expect, and what behavior do you see?
When did the problem start occurring?
Are their circumstances in which the problem does not occur?
Is the problem permanent, intermittent, getting better, getting worse, or staying the same?

If you contact ForgeRock for help, include the following information with your request:

Description of the problem, including when the problem occurs and its impact on your operation.
The product version and build information.
Steps you took to reproduce the problem.
Relevant access and error logs, stack traces, and core dumps.
Description of the environment, including the following information:
- Machine type
- Operating system and version
- Web server and version
- Java version
- Patches or other software that might affect the problem

The following items are some common issues and solutions:

The agentadmin command offers a validation mode for the agent that can help you troubleshoot issues in your environment; for example, after an agent upgrade or a network change. For more information, refer to the --V[i] option in agentadmin command.

Shared memory errors during upgrade or installation

Question

During upgrade or installation, what should I do if I get shared memory errors?

Answer

Stop the web server where the agent is installed.
Delete the following shared memory files:
- /dev/shm/am_cache_0
- /dev/shm/am_log_data_0
  
  Depending on your configuration, the files can be named differently.
Start the agent.

HTTP headers are ignored

Question

When I map a response or attribute to an HTTP header, using the following properties, why is the header ignored:

Answer

When injecting information into HTTP headers, do not use underscores (_) in the header name. Underscores are incompatible with systems that run CGI scripts, and the header can be silently dropped.

Error installing Agents on Windows

Question

I am trying to install a web agent on Windows, which will connect to an AM server running over HTTPS, but the installer reports the following error:

init_ssl(): ssleay32.dll is not available (error: 87)
init_ssl(): libeay32.dll is not available (error: 87)

Answer

If OpenSSL is correctly installed, on Windows 7 or Windows Server 2008 R2 systems, apply the update provided in Microsoft knowledge base article KB2533623. See Microsoft Security Advisory: Insecure library loading could allow remote code execution.

Error installing Agents With SELinux

Question: I am trying to install Web Agent on a server with SELinux enabled in enforcing mode and I am getting error messages after installation, or the web server does not start up. What happened?
Answer: When installing Web Agent on Linux or Unix servers, you must ensure that the user that runs the web server process has read and write permissions for the agent installation directory and files.

If SELinux is enabled in enforcing mode, you must also ensure that SELinux is configured to allow the web server process to perform read and write operations to the agent installation directory and files. By default, SELinux only allows the web server process to read files in well-known authorized locations, such as the /var/www/html directory.

For environments where security can be more relaxed, consider setting SELinux or the httpd_t context in permissive mode for troubleshooting purposes.

For information about configuring SELinux, refer to the Linux documentation.

Logs Not Written

Question: Why are logs not being written to /log/system_0.log and /log/monitor_0.pipe files? I am seeing this error:

unable to open event channel

Answer: It is likely that the agent does not have permission to be able to write to the /log/system_0.log and /log/monitor_0.pipe log files.

This can occur if you used the agentadmin --V[i] validator command using a user account that is different to the account used to run your web server.

Run the validator command as the same user that runs the web server, for example, by using the sudo command.

To fix the issue, change the ownership of these files to match the user or group that is running your web server.

Port 80 Used as Default

Question: My Apache HTTP server is not using port 80. When I install Web Agent it defaults to port 80. How do I fix this?
Answer: You probably set ServerName in the Apache HTTP Server configuration to the host name, but did not specify the port number.

Instead, set both the host name and port number for ServerName in the configuration. For example, if you have Apache HTTP Server configured to listen on port 8080, then set ServerName appropriately as in the following excerpt:

<VirtualHost *:8080>
ServerName www.localhost.example:8080

Cannot rotate logs

Question: My web server and Web Agent are installed as root, and the agent cannot rotate logs. I am seeing this error:

Could not rotate log file ... (error: 13)

What should I do?

Answer

If the web server is running with a non-root user, for example, the daemon user, you must ensure that user has the following permissions:

Read Permission:
- /web_agents/agent_name/lib
Read and Write Permission:
- /web_agents/agent_name/instances/agent_nnn
- /web_agents/agent_name/log

Apply execute permissions on the folders listed above, recursively, for the user that runs the web server.

For IIS web agents, change the ownership of the files using the agentadmin --o command. For more information, refer to agentadmin command.

You may also see similar issues if SELinux is enabled in enforcing mode, and it is not configured to allow access to agent directories.

Protection Against Phishing Attacks

Question: How do I increase security against possible phishing attacks through open redirect?
Answer: You can specify a list of valid URL resources against which AM validates the goto and gotoOnFail URL using the Valid goto URL Resource service.

AM only redirects a user if the goto and gotoOnFail URL matches any of the resources specified in this setting. If no setting is present, it is assumed that the goto and gotoOnFail URL is valid.

To set the Valid goto URL Resources, use the AM admin UI, and go to Realms > Realm Name > Services > Add > Validation Service, and then add one or more valid goto URLs.

You can use the "*" wildcard to define resources, where "*" matches all characters except "?". For example, you can use the wildcards, such as https://website.example.com/* or https://website.example.com/*?*. For more specific patterns, use resource names with wildcards as described in Configuring success and failure redirection URLs.

Apache Agent Start Up

Question

I have installed the Unix Apache Web Agent, and neither Apache HTTP Server nor the agent start up or log any message. If I remove the agent, the Apache HTTP Server starts again. What can be the problem?

Answer

To troubleshoot Web Agent or a web server that does not start, set the agent logging level to the maximum by performing the following steps:

Set the environment variable AM_SYSTEM_LOG_LEVEL to All in your command line session. For example:
```
$ export AM_SYSTEM_LOG_LEVEL=ALL
```
Restart the Apache HTTP Server.
Check the logs generated in the web_agent/apache24_agent/log/system_n.log.

Web Agent reserves memory for the policy and session cache based on the AM_MAX_SESSION_CACHE_SIZE environment variable. If the server where the agent is installed does not have enough shared memory available, the web agent may log messages like the following:
```
017-11-10 12:06:00.492 +0000   DEBUG [1:7521][source/shared.c:1451]am_shm_create2() about to create block-clusters_0, size 1074008064
2017-11-10 12:06:00.492 +0000   ERROR [1:7521]am_shm_create2(): ftruncate failed, error: 28
```
The error message means the web agent tries to reserve 1074008064 bytes of memory, but there is not enough shared memory available. Several reasons may explain why the shared memory is running low, such as:
- A new application or additional workload may be stretching the server resources to the limit.
  
  In this case, ensure that the server has enough shared memory available to satisfy the need of all the applications.
- A web agent may not have been able to release its shared memory after stopping. Therefore, even if the shared memory is technically not in use, it is still reserved and cannot be reassigned unless freed.
  
  Different operating systems manage the shared memory in different ways. Refer to your operating system documentation for information about checking shared memory usage.
  
  You can reduce the amount of memory the web agent reserves for the session and policy cache by setting the AM_MAX_SESSION_CACHE_SIZE environment variable to a value between 1048576 (1 MB) and 1074008064 bytes (1 GB). For more information, refer to Environment variables.
  
  Troubleshooting a component that does not start and does not generate logs may be difficult to diagnose. Contact the ForgeRock Support team for more help and information.

Infinite Redirection Loops

Question

I have client-based (stateless) sessions configured in AM, and I am getting infinite redirection loops. In the debug.log file I can see messages similar to the following:

... +0000 ERROR [c5319caa-beeb-5a44-a098-d5575e768348]state identifier not present in authentication state
... +0000 WARNING [c5319caa-beeb-5a44-a098-d5575e768348]unable to verify pre-authentication cookie
... +0000 WARNING [c5319caa-beeb-5a44-a098-d5575e768348]convert_request_after_authn_post(): unable to retrieve pre-authentication request data
... +0000 DEBUG [c5319caa-beeb-5a44-a098-d5575e768348] exit status: forbidden (3), HTTP status: 403, subrequest 0

What is happening?

Answer

The redirection loop happens because the client-based (stateless) session cookie is surpassing the maximum supported browser header size. Since the cookie is incomplete, AM cannot validate it.

To ensure the session cookie does not surpass the browser supported size, configure either signing and compression or encryption and compression.

For more information, refer to AM’s Security guide.

Errors After Upgrade

Question: I have upgraded my agent and, in the logs, I can see errors similar to the following:

redirect_uri_mismatch. The redirection URI provided does not match a pre-registered value.
com.iplanet.sso.SSOException: Invalid Agent Root URL
com.iplanet.sso.SSOException: Goto URL not valid for the agent Provider ID

What should I do?

Answer

Web Agent accepts only requests sent to the URL specified by the Agent Root URL for CDSSO property. For example, http://agent.example.com:8080/.

As a security measure, Web Agent prevents you from accessing the agent on URLs not defined in the Agent Root URL for CDSSO property. Add entries to this property when:

Accessing the agent through different protocols. For example, http://agent.example.com/ and https://agent.example.com/.
Accessing the agent through different virtual host names. For example, http://agent.example.com/ and http://internal.example.com/.
Accessing the agent through different ports. For example, http://agent.example.com/ and http://agent.example.com:8080/.

Configuration Not Updated After Upgrade

Question

I have upgraded my Unix Apache or IBM HTTP Server Web Agent, and even though notifications are enabled, the agent does not update its configuration. What is happening?

Answer

Set the web agent logging level to the maximum by performing the following steps:

Set the environment variable AM_SYSTEM_LOG_LEVEL to ALL in your command line session. For example:
```
$ export AM_SYSTEM_LOG_LEVEL=ALL
```
Restart the Apache or IBM HTTP server.
Check the logs generated in the web_agent/agent_type/log/system_n.log file.

Sometimes stopping or upgrading an agent does not clean the pipe file the agent uses to communicate with AM. If the newly started agent cannot create the pipe to communicate with AM because it already exists, the agent would log messages like the following:
```
... UTC   DEBUG [1:10551398][source/monitor.c:503]monitor startup
... UTC   ERROR [102:10551398]monitor unable to get semaphore
... UTC   DEBUG [304:10551398][source/config.c:295]config_initialise():  agent configuration read from cache, agent: / wpa-aix7-Httpd7-32bit
```

If you see similar error messages, perform the following steps to delete the pipe file:

Stop the Apache or IBM HTTP server.
Change directories to the /tmp directory.
Delete the monitor.pipe file.
Restart the Apache or IBM HTTP server.

Custom Pages Not Displayed

Question: After upgrade, the default Apache welcome page appears instead of my custom error pages. What should I do?
Answer: Check your Apache ErrorDocument configuration. If the custom error pages are not in the document root of the Apache HTTP Server, enclose the ErrorDocument directives in Directory elements. For example:

<Directory "/web/docs">
   ErrorDocument 403 myCustom403Page.html
</Directory>

For more information about ErrorDocument, refer to the Apache documentation.

Failure After Installation

Question: After starting a web agent installation, I see a failure in the logs:

[../resources/troubleshooting/troubleshooting.bash:#web-agent-install]

Answer: Web Agent installation, can fail if AM’s validation of the agent configuration exceeds the default timeout of 4 seconds.

You can set the AM_NET_TIMEOUT environment variable to change the default timeout, and then rerun the installation.

Agent Not Protecting a Website

Question: My Web Agent is not protecting my website. In the logs, I can see errors similar to the following:

... -0500  ERROR [86169084-5648-6f4d-a706-30f5343d9220]config_fetch():  failed to load configuration for agent: myagent myagent, error -24
... -0500  ERROR [86169084-5648-6f4d-a706-30f5343d9220]amagent_auth_handler(): failed to get agent configuration instance, error: invalid agent session*

What is happening?

Answer

The Web Agent is unable to log in to AM. Possible causes are:

Network connection between the agent and AM is unavailable.
The AM Connection URL property, which specifies the AM URL may be misconfigured.

Agent Not Protecting a Website

Question: My Web Agent is not protecting my website. In the debug.log file I can see messages similar to the following:

... GMT DEBUG [162ba6eb-cf88-3d7f-f92c-ee8b21971b4c]: (source/oidc.c:265) agent_realm does not have the expected value: JWT
   {
   "sub":"demo",
   "auditTrackingId":"267d1f56-0b97-4830-ae91-6be4b8b7099f-5840",
   "iss":"https://am.example.com:8443/am/oauth2/Customers",
   "tokenName":"id_token",
   "nonce":"D3AE96656D6D634489AF325D90C435A2",
   "aud":"webagent",
   "s_hash":"rxwxIoqDFiwt4MxSwiBa-w",
   "azp":"webagent",
   "auth_time":1561600459,
   "forgerock":{
   "ssotoken":"wi8tHql...MQAA*",
   "suid":"267d1f56-0b97-4830-ae91-6be4b8b7099f-5647"
   },
   "realm":"/Customers",
   "exp":1561607661,
   "tokenType":"JWTToken",
   "iat":1561600461,
   "agent_realm":"/Customers"
   }
... GMT WARNING [162ba6eb-cf88-3d7f-f92c-ee8b21971b4c]: redirect_after_authn(): unable to validate JWT

What is happening?

Answer: If you configured the agent profile in a realm other than AM’s top-level realm (/), you must configure the agent com.sun.identity.agents.config.organization.name bootstrap property with the realm where the agent profile is located. For example, /Customers.

Realm names are case-sensitive. Failure to set the realm name exactly as configured in AM causes the agent to fail to recognize the realm.

Cannot Access Protected Resources

I am getting HTTP 403 Forbidden messages when accessing protected resources, and I can see errors similar to the following in the debug.log file:

... GMT WARNING [69d4632c-82af-b853-0f340vb7b754]: too many pending authentications
... GMT ERROR  [69d4632c-82af-76da-b853-0f340vb7b754]: save_pre_authn_state(): unable to save state for request

What is happening?

Answer: Agents store the progress of authentication with AM in the pre-authentication cookie, agent-authn-tx. This cookie has a maximum size of 4096 bytes, and can fill up if the agent receives many parallel unauthenticated requests to access protected resources.

For more information, refer to Enable Multivalue for Pre-Authn Cookie.

Cannot Access Protected Resources

Question

I am getting HTTP 403 Forbidden messages when accessing the Web Agent.

Answer

Make sure the Web Agent is executable:

In the terminal where the Web Agent is running, go to /opt/web_agents.
Review and, if necessary, change the permissions for the directory:

Web Agents 2023.11.1

Troubleshooting