Solutions

Apache Web Policy Agent 4.1.0 fails to start with no space left on device Configuration Failed error

Last updated Nov 1, 2018

The purpose of this article is to provide assistance if your Apache Web Policy Agent 4.1.0 (patch 4.1.0-28 and earlier) fails to start or install with "status: no space left on device Configuration Failed" error. You may also see the error "am_shm_create(): free disk space on the system is only x bytes, required y bytes".

Symptoms

The following error is shown when starting the Apache agent:

[Mon Nov 06 16:20:45 2017] [error] amagent_init() status: no space left on device 
Configuration Failed

An error similar to the following is shown in the agent debug log:

2017-11-06 16:27:09.422 +0000   ERROR [0x7fc7bf8457e0:18067] am_shm_create(): free disk space on the system is only 969605120 bytes, required 1073741824 bytes
2017-11-06 16:27:09.422 +0000   ERROR [0x7fc7bf8457e0:18067] get_memory_segment(): shared memory error: blocks

You may also encounter this issue when trying to install the Apache agent:

[Fri Nov 10 11:35:43.082435 2017] [amagent:error] [pid 17392:tid 140597884139392] amagent_init() status: no space left on device 
[Fri Nov 10 11:35:43.082453 2017] [:emerg] [pid 17392:tid 140597884139392] AH00020: Configuration Failed, exiting

Recent Changes

Upgraded to, or installed Web policy agents 4.1.0 (patch 4.1.0-28 and earlier).

Added additional Apache instances.

Causes

Web policy agent 4.1.0 shares memory between instances to improve performance and this shared memory (as reported by statfs on the /dev/shm mount) is insufficient. This can happen if you have low memory or you have many Apache instances.

Solution

This issue can be resolved by upgrading to Web Agents 4.2 or later; you can download this from BackStage.

Workaround

You can workaround this issue using one of the following approaches:

Reduce the amount of shared memory used.
Increase the shared memory available.
Reduce the number of Apache instances.

Reduce the amount of shared memory used

Try running the Apache instances with the following environment variables to reduce the amount of memory used:

AM_MAX_SHARED_POOL_SIZE=0x200000
AM_MAX_SESSION_CACHE_SIZE=0x4000000
export AM_MAX_SHARED_POOL_SIZE
export AM_MAX_SESSION_CACHE_SIZE

These variables set a maximum of 2MB of memory to be used for configuration and audit, and a maximum of 128MB for the session cache.

You can set these variables by editing /etc/sysconfig/httpd (or equivalent place such as envvars).

See Web Policy Agent Guide › Configuring Web Policy Agent Environment Variables for further information.

Increase the shared memory available

Firstly you need to determine how much shared memory is required; the required value shown in the error is misleading. You can determine the absolute minimum value by running the agentadmin command with the --v option and then adding up the values shown below in bold:

$ ./agentadmin --v

OpenAM Web Agent for Apache Server 2.2.x
 Version: 4.1.0
 Revision: 31e821e
 Build machine: delacroix
 Build date: Nov 16 2016 11:40:32

System Resources:
 total memory size: 3.2GB
 pre-allocated session/policy cache size: 829.0MB
 log buffer size: 128.7MB
 min audit log buffer size: 2MB, max 2.0GB
 total disk (tmpfs/swap) size: 1.6GB
 free disk (tmpfs/swap) space size: 1.6GB

System contains sufficient resources (with remote audit log feature disabled).

We typically use 1.5GB for testing, so this may be a good starting point.

See How To Resize /dev/shm Filesystem In Linux? for further information on changing /dev/shm. For Docker, you need to set the --shm-size option when running it.

Reduce the number of Apache instances

You should keep the number of Apache instances to the minimum and consider using vhosts instead; each agent can support a maximum of 32 vhosts.

We recommend the following when using vhosts:

Each vhost should have a separate AmAgentConf setting.
Set the AmAgentId property in each Apache instance's httpd.conf file to a different value, for example:
- In the first Apache instance:
```
AmAgentId 0
```
- For the next Apache instance:
```
AmAgentId 1
```
  etc.
Ensure all Apache instances on the operating system pick up the following environment variable:
```
AM_MAX_SESSION_CACHE_SIZE=0xf424000
```

If you choose to use a large number of Apache instances instead of using vhosts, you must monitor memory usage on each additional install and increase shared memory as needed. Additionally, further limiting of AM_MAX_SESSION_CACHE_SIZE may help, but you will need to test performance carefully after changing this. An increase in agent instances will also have an impact on AM/OpenAM servers, so you will need to monitor their CPU and RAM as well.

Related Training

N/A

Related Issue Tracker IDs

AMAGENTS-1259 (DOC: Clarify the requirements on shm size)

AMAGENTS-581 (Agent 4.x has hard limit of 32 instances)

AMAGENTS-383 (Document shared memory commandline variables)

AMAGENTS-315 (Agent fails to start with insufficient shared memory)

AMAGENTS-273 (Improve WebAgent to not using shared memory /dev/shm)