Monitor services

The following sections describe how to set up and maintain monitoring in your deployment, to ensure appropriate performance and service availability:

Access the monitoring endpoints

All ForgeRock products automatically expose a monitoring endpoint to expose metrics in a standard Prometheus format, and as a JSON format monitoring resource.

In PingGateway, metrics are available for each router, subrouter, and route in the configuration. When a TimerDecorator is configured, timer metrics are also available.

Learn more about PingGateway monitoring endpoints and available metrics in Monitoring.

Monitor at the Prometheus Scrape Endpoint

Prometheus metric names are deprecated and expected to be replaced with names ending in _total. The information provided by the metric is not deprecated. Other Prometheus metrics are not affected.

All ForgeRock products automatically expose a monitoring endpoint where Prometheus can scrape metrics, in a standard Prometheus format.

When PingGateway is set up as described in the Quick install, the Prometheus Scrape Endpoint is available at the following endpoints:

By default, no special setup or configuration is required to access metrics at these endpoints. The following example queries the Prometheus Scrape Endpoint for a route.

Tools such as Grafana are available to create customized charts and graphs based on the information collected by Prometheus. For more information on installing and running Grafana, refer to the Grafana website.

Set up PingGateway:

Set up PingGateway for HTTPS, as described in Configure PingGateway for TLS (server-side).

Add the following route to PingGateway:

Linux
Windows

$HOME/.openig/config/routes/myroute1.json

%appdata%\OpenIG\config\routes\myroute1.json

{
  "name": "myroute1",
  "handler": {
    "type": "StaticResponseHandler",
    "config": {
      "status": 200,
      "headers": {
        "Content-Type": [ "text/plain; charset=UTF-8" ]
      },
      "entity": "Hello world, from myroute1!"
    }
  },
  "condition": "${find(request.uri.path, '^/myroute1')}"
}

The route contains a StaticResponseHandler to display a simple message.

Test the setup:
1. Access the route a few times, on https://ig.example.com:8443/myroute1.
2. Query the Prometheus Scrape Endpoint:
  $ curl -v \ --cacert /path/to/secrets/ig.example.com-certificate.pem \ https://ig.example.com:8443/openig/metrics/prometheus/0.0.4
  Metrics for myroute1 and _router are displayed.
  
  Vert.x monitoring is enabled by default to provide additional metrics for HTTP, TCP, and the internal component pool. The metrics provide low-level information about requests and responses, such as the number of bytes, duration, and the number of concurrent requests.

Monitor the Common REST Monitoring Endpoint (deprecated)

All ForgeRock products expose a monitoring endpoint where metrics are exposed as a JSON format monitoring resource.

When PingGateway is set up as described in Quick install, the Common REST Monitoring Endpoint is available at https://ig.example.com:8443/openig/metrics/api?_prettyPrint=true&_sortKeys=_id&_queryFilter=true

By default, no special setup or configuration is required to access metrics at this endpoint. The following example queries the Common REST Monitoring Endpoint for a route, and restricts the query to specific metrics only.

Before you start, prepare PingGateway as described in the Quick install.

Set up PingGateway and some example routes, as described in the first few steps of Monitor the Prometheus Scrape Endpoint.

Query the Common REST Monitoring Endpoint:

$ curl "https://ig.example.com:8443/openig/metrics/api?_prettyPrint=true&_sortKeys=_id&_queryFilter=true"

Metrics for myroute1 and _router are displayed:

{
  "result" : [ {
  "_id" : "gateway._router.deployed-routes",
  "value" : 1.0,
  "_type" : "gauge"
}, {
  "_id" : "gateway._router.route.default.request",
  "count" : 204,
  "_type" : "counter"
}, {
  "_id" : "gateway._router.route.default.request.active",
  "value" : 0.0,
  "_type" : "gauge"
}, {

       . . .

       _id" : "gateway._router.route.myroute1.response.status.unknown",
  "count" : 0,
  "_type" : "counter"
}, {
  "_id" : "gateway._router.route.myroute1.response.time",
  "count" : 204,
  "max" : 0.420135,
  "mean" : 0.08624678327176545,
  "min" : 0.045079999999999995,
  "p50" : 0.070241,
  "p75" : 0.096049,
  "p95" : 0.178534,
  "p98" : 0.227217,
  "p99" : 0.242554,
  "p999" : 0.420135,
  "stddev" : 0.046611762381930474,
  "m15_rate" : 0.2004491450567003,
  "m1_rate" : 2.8726563452698075,
  "m5_rate" : 0.5974045160056258,
  "mean_rate" : 0.010877725092634833,
  "duration_units" : "milliseconds",
  "rate_units" : "calls/second",
  "total" : 17.721825,
  "_type" : "timer"
} ],
  "resultCount" : 11,
  "pagedResultsCookie" : null,
  "totalPagedResultsPolicy" : "EXACT",
  "totalPagedResults" : 11,
  "remainingPagedResults" : -1
}

Vert.x monitoring is enabled by default to provide additional metrics for HTTP, TCP, and the internal component pool. The metrics provide low-level information about requests and responses, such as the number of bytes, duration, the number of concurrent requests, and so on.

Change the query to access metrics only for myroute1: https://ig.example.com:8443/openig/metrics/api?_prettyPrint=true&_sortKeys=_id&_queryFilter=_id+sw+"gateway._router.route.myroute1";.

Note that metric for the router, "_id" : "gateway._router.deployed-routes", is no longer displayed.

Monitor Vert.x metrics

Vert.x metric names are deprecated and expected to be replaced with names ending in _total. The information provided by the metric is not deprecated. Other Prometheus metrics are not affected.

Vert.x monitoring is enabled by default to provide metrics for HTTP, TCP, and the internal component pool. The metrics provide low-level information about requests and responses, such as the number of bytes, duration, the number of concurrent requests, and so on.

To disable Vert.x monitoring, add the following lines to admin.json, and restart PingGateway:

{
  "vertx": {
    "metricsEnabled": false
  }
}

For more information, refer to AdminHttpApplication (admin.json).

Protect monitoring endpoints

By default, no special credentials or privileges are required for read-access to the Prometheus Scrape Endpoint and Common REST Monitoring Endpoint.

To protect the monitoring endpoints, add an admin.json file to your configuration, with a filter declared in the heap and named MetricsProtectionFilter. The following procedure gives an example of how to manage access to the monitoring endpoints.

Set up the procedure in Monitor at the Prometheus Scrape Endpoint, query the Prometheus Scrape Endpoint, and note that metrics for myroute1 and _router are displayed:
```
$ curl -v "https://ig.example.com:8443/openig/metrics/prometheus"
```

Add the following script to the PingGateway configuration:

Linux
Windows

$HOME/.openig/scripts/groovy/BasicAuthResourceServerFilter.groovy

%appdata%\OpenIG\scripts\groovy\BasicAuthResourceServerFilter.groovy

/*
 * This script is a simple implementation of HTTP basic access authentication on
 * server side.
 * It expects the following arguments:
 *  - realm: the realm to display when the user agent prompts for
 *    username and password if none were provided.
 *  - username: the expected username
 *  - passwordSecretId: the secretId to find the password
 *  - secretsProvider: the SecretsProvider to query for the password
*/
import static org.forgerock.util.promise.Promises.newResultPromise;

import java.nio.charset.Charset;
import org.forgerock.util.encode.Base64;
import org.forgerock.secrets.Purpose;
import org.forgerock.secrets.GenericSecret;

String authorizationHeader = request.getHeaders().getFirst("Authorization");
if (authorizationHeader == null) {
    // No credentials provided, return 401 Unauthorized
    Response response = new Response(Status.UNAUTHORIZED);
    response.getHeaders().put("WWW-Authenticate", "Basic realm=\"" + realm + "\"");
    return newResultPromise(response);
}

return secretsProvider.getNamed(Purpose.PASSWORD, passwordSecretId)
        .thenAsync(password -> {
            // Build basic authentication string -> username:password
            StringBuilder basicAuthString = new StringBuilder(username).append(":");
            password.revealAsUtf8{ p -> basicAuthString.append(new String(p).trim()) };
            String expectedAuthorization = "Basic " + Base64.encode(basicAuthString.toString().getBytes(Charset.defaultCharset()));
            // Incorrect credentials provided, return 403 forbidden
            if (!expectedAuthorization.equals(authorizationHeader)) {
                return newResultPromise(new Response(Status.FORBIDDEN));
            }
            // Correct credentials provided, continue.
            return next.handle(context, request);
        },
                noSuchSecretException -> { throw new RuntimeException(noSuchSecretException); });

The script is a simple implementation of the HTTP basic access authentication scheme. For information about scripting filters and handlers, refer to Extend.

Add the following heap configuration for MetricsProtectionFilter to admin.json:

{
  ...,
  "heap": [
    {
      "name": "ClientHandler",
      "type": "ClientHandler"
    },
    {
      "name": "mySecretsProvider",
      "type": "Base64EncodedSecretStore",
      "config": {
        "secrets": {
          "password.secret.id": "cGFzc3dvcmQ="
        }
      }
    },
    {
      "name": "MetricsProtectionFilter",
      "type": "ScriptableFilter",
      "config": {
        "type": "application/x-groovy",
        "file": "BasicAuthResourceServerFilter.groovy",
        "args": {
          "realm": "/",
          "username": "myUsername",
          "passwordSecretId": "password.secret.id",
          "secretsProvider": "${heap['mySecretsProvider']}"
        }
      }
    }
  ],
  ...
}

Notice the following features of the configuration:

The MetricsProtectionFilter uses the script to protect the monitoring endpoint.
The MetricsProtectionFilter requires the username myUsername, and a password provided by the SecretsProvider in the heap.

Restart PingGateway.
Query the Prometheus Scrape Endpoint without providing credentials, and note that an HTTP 401 Unauthorized is returned:
```
$ curl -v "https://ig.example.com:8443/openig/metrics/prometheus"
```
Query the Prometheus Scrape Endpoint by providing correct credentials, and note that metrics are displayed:
```
$ curl -v "https://ig.example.com:8443/openig/metrics/prometheus" -u myUsername:password
```
Query the Prometheus Scrape Endpoint by providing incorrect credentials`, and note that an HTTP 403 Forbidden is returned:
```
$ curl -v "https://ig.example.com:8443/openig/metrics/prometheus" -u myUsername:wrong-password
```

PingGateway 2024.6