HTTP-Based Monitoring

The following example reads the /alive endpoint anonymously. If the DS server’s internal tests do not find errors that require administrative action, then it returns HTTP 200 OK:

If the server finds that it is subject to errors requiring administrative action, it returns HTTP 503 Service Unavailable.

If there are errors, anonymous users receive only the 503 error status. Error strings for diagnosis are returned as an array of "alive-errors" in the response body, but the response body is only returned to a user with the monitor-read privilege.

When a server returns "alive-errors", diagnose and fix the problem, and then either restart or replace the server.

The following example reads the /healthy endpoint anonymously. If the DS server is alive, as described in Server is Alive (HTTP), and any replication delay is below the threshold configured as max-replication-delay-health-check (default: 5 seconds), then it returns HTTP 200 OK:

If the server is subject to a replication delay above the threshold, then it returns HTTP 503 Service Unavailable. This result only indicates a problem if the replication delay is steadily high and increasing for the long term.

If there are errors, anonymous users receive only the 503 error status. Error strings for diagnosis are returned as an array of "ready-errors" in the response body, but the response body is only returned to a user with the monitor-read privilege.

When a server returns "ready-errors", route traffic to another server until the current server is ready again.

In addition to the examples above, you can monitor whether a server is alive and able to handle requests as Prometheus metrics:

The following example reads a metric to check the delay in replication:

DS replicas measure replication delay as the local delay when receiving and replaying changes. A replica calculates these local delays based on changes received from other replicas. Therefore, a replica can only calculate delays based on changes it has received. Network outages cause inaccuracy in delay metrics.

A replica calculates delay metrics based on times reflecting the following events:

This figure illustrates when these events occur:

Replication keeps track of changes using change sequence numbers (CSNs), opaque and unique identifiers for each change that indicate when and where each change first occurred. The t_n values are CSNs.

When the CSNs for the last change received and the last change replayed are identical, the replica has applied all the changes it has received. In this case, there is no known delay. The receive and replay delay metrics are set to 0 (zero).

When the last received and last replayed CSNs differ:

As long as replication delay tends toward zero regularly and over the long term, temporary spikes and increases in delay measurements are normal. When all replicas remain connected and yet replication delay remains high and increases over the long term, the high replication delay indicates a problem. Steadily high and increasing replication delay shows that replication is not converging, and the service is failing to achieve eventual consistency.

For a current snapshot of replication delays, you can also use the dsrepl status command. For details, see Replication Status.

The following example shows monitoring metrics you can use to check whether the server is running out of disk space:

In your monitoring software, compare free space with the disk low and disk full thresholds. For database backends, these thresholds are set using the configuration properties: disk-low-threshold and disk-full-threshold.

When you read from cn=monitor instead ,as described in LDAP-Based Monitoring, the relevant data are exposed on child entries of cn=disk space monitor,cn=monitor.

The following example shows how you can use monitoring metrics to check whether the server certificate is due to expire soon:

In your monitoring software, compare the expiration date with the current date.

When you read from cn=monitor instead, as described in LDAP-Based Monitoring, the relevant data are exposed on child entries of cn=certificates,cn=monitor.

DS server connection handlers respond to client requests. The following example uses the default monitor user account to read statistics about client operations on each of the available connection handlers:

DS servers have a work queue to track request processing by worker threads, and whether the server has rejected any requests due to a full queue. If enough worker threads are available, then no requests are rejected. The following example uses the default monitor user account to read statistics about the work queue:

To adjust the number of worker threads, see the settings for Traditional Work Queue.

DS servers maintain counts of the number of entries in each backend. The following example uses the default monitor user account to read the counts:

DS server connection handlers respond to client requests. The following example uses the default monitor user account to read active connections on each connection handler:

By default, DS servers return all Prometheus metrics. To limit what the server returns, set one of these HTTP endpoint properties for the /metrics/prometheus:

Set these properties to valid Java regular expression patterns.

The following configuration change causes the server to return only metrics whose names contain connection:

The following configuration change causes the server to exclude metrics whose names start with ds_jvm_. As mentioned in the reference documentation, "The metric name prefix must not be included in the filter." Notice that the example uses the regular expression jvm_.*: