Monitor services

This page describes how to set up and maintain monitoring in your deployment.

Push to an OpenTelemetry service

PingGateway can push traces to an OpenTelemetry Protocol (OTLP) endpoint over HTTP. When you use it with other applications with OpenTelemetry support, the service helps you analyze the flows through PingGateway and the other applications to understand performance and system behavior.

This capability is available in Technology preview. It isn’t yet supported, may be functionally incomplete, and is subject to change without notice.

OpenTelemetry support is disabled by default. Enable it with the "openTelemetry" setting in AdminHttpApplication (admin.json). Make sure the OTLP exporter targets the endpoint for the OpenTelemetry service in your deployment. The following example targets the default endpoint:

{
    "openTelemetry": {
        "tracing": {
            "enabled": true,
            "exporter": {
                "type": "otlp",
                "config": {
                    "endpoint": "http://localhost:4318/v1/traces"
                }
            }
        }
    }
}

When tracing is enabled, traces always span requests entering PingGateway and any requests started by PingGateway; for example, proxied requests to the sample application. The span ends when PingGateway receives or returns a response or when an error arises during processing. If a ClientHandler or ReverseProxyHandler has the retry feature enabled, PingGateway records the number of retries as a http.request.resend_count span attribute in the trace.

Trace sampling is alwaysOn by default, meaning PingGateway samples every operation. Adjust this in the configuration settings as required for your deployment and traffic.

Use the "tracing": true decoration on other configuration objects or in "globalDecorators" to activate additional tracing points.

Access the monitoring endpoints

All ForgeRock products automatically expose a monitoring endpoint to expose metrics in a standard Prometheus format, and as a JSON format monitoring resource.

In PingGateway, metrics are available for each router, subrouter, and route in the configuration. When a TimerDecorator is configured, timer metrics are also available.

Learn more about PingGateway monitoring endpoints and available metrics in Monitoring.

Monitor at the Prometheus Scrape Endpoint

Prometheus metric names are deprecated and expected to be replaced with names ending in _total. The information provided by the metric is not deprecated. Other Prometheus metrics are not affected.

All ForgeRock products automatically expose a monitoring endpoint where Prometheus can scrape metrics, in a standard Prometheus format.

When PingGateway is set up as described in the Quick install, the Prometheus Scrape Endpoint is available at the following endpoints:

By default, no special setup or configuration is required to access metrics at these endpoints. The following example queries the Prometheus Scrape Endpoint for a route.

Tools such as Grafana are available to create customized charts and graphs based on the information collected by Prometheus. For more information on installing and running Grafana, refer to the Grafana website.

Set up PingGateway:

Set up PingGateway for HTTPS, as described in Configure PingGateway for TLS (server-side).

Add the following route to PingGateway:

Linux
Windows

$HOME/.openig/config/routes/myroute1.json

%appdata%\OpenIG\config\routes\myroute1.json

{
  "name": "myroute1",
  "handler": {
    "type": "StaticResponseHandler",
    "config": {
      "status": 200,
      "headers": {
        "Content-Type": [ "text/plain; charset=UTF-8" ]
      },
      "entity": "Hello world, from myroute1!"
    }
  },
  "condition": "${find(request.uri.path, '^/myroute1')}"
}

The route contains a StaticResponseHandler to display a simple message.

Test the setup:
1. Access the route a few times, on https://ig.example.com:8443/myroute1.
2. Query the Prometheus Scrape Endpoint:
  $ curl -v \ --cacert /path/to/secrets/ig.example.com-certificate.pem \ https://ig.example.com:8443/openig/metrics/prometheus/0.0.4
  Metrics for myroute1 and _router are displayed.
  
  Vert.x monitoring is enabled by default to provide additional metrics for HTTP, TCP, and the internal component pool. The metrics provide low-level information about requests and responses, such as the number of bytes, duration, and the number of concurrent requests.

Monitor the Common REST Monitoring Endpoint (deprecated)

All ForgeRock products expose a monitoring endpoint where metrics are exposed as a JSON format monitoring resource.

When PingGateway is set up as described in Quick install, the Common REST Monitoring Endpoint is available at https://ig.example.com:8443/openig/metrics/api?_prettyPrint=true&_sortKeys=_id&_queryFilter=true

By default, no special setup or configuration is required to access metrics at this endpoint. The following example queries the Common REST Monitoring Endpoint for a route, and restricts the query to specific metrics only.

Before you start, prepare PingGateway as described in the Quick install.

Set up PingGateway and some example routes, as described in the first few steps of Monitor the Prometheus Scrape Endpoint.

Query the Common REST Monitoring Endpoint:

$ curl "https://ig.example.com:8443/openig/metrics/api?_prettyPrint=true&_sortKeys=_id&_queryFilter=true"

Metrics for myroute1 and _router are displayed:

{
  "result" : [ {
  "_id" : "gateway._router.deployed-routes",
  "value" : 1.0,
  "_type" : "gauge"
}, {
  "_id" : "gateway._router.route.default.request",
  "count" : 204,
  "_type" : "counter"
}, {
  "_id" : "gateway._router.route.default.request.active",
  "value" : 0.0,
  "_type" : "gauge"
}, {
  "...":  "...",
  "_id" : "gateway._router.route.myroute1.response.status.unknown",
  "count" : 0,
  "_type" : "counter"
}, {
  "_id" : "gateway._router.route.myroute1.response.time",
  "count" : 204,
  "max" : 0.420135,
  "mean" : 0.08624678327176545,
  "min" : 0.045079999999999995,
  "p50" : 0.070241,
  "p75" : 0.096049,
  "p95" : 0.178534,
  "p98" : 0.227217,
  "p99" : 0.242554,
  "p999" : 0.420135,
  "stddev" : 0.046611762381930474,
  "m15_rate" : 0.2004491450567003,
  "m1_rate" : 2.8726563452698075,
  "m5_rate" : 0.5974045160056258,
  "mean_rate" : 0.010877725092634833,
  "duration_units" : "milliseconds",
  "rate_units" : "calls/second",
  "total" : 17.721825,
  "_type" : "timer"
} ],
  "resultCount" : 11,
  "pagedResultsCookie" : null,
  "totalPagedResultsPolicy" : "EXACT",
  "totalPagedResults" : 11,
  "remainingPagedResults" : -1
}

Vert.x monitoring is enabled by default to provide additional metrics for HTTP, TCP, and the internal component pool. The metrics provide low-level information about requests and responses, such as the number of bytes, duration, the number of concurrent requests, and so on.

Change the query to access metrics only for myroute1: https://ig.example.com:8443/openig/metrics/api?_prettyPrint=true&_sortKeys=_id&_queryFilter=_id+sw+"gateway._router.route.myroute1";.

Note that metric for the router, "_id" : "gateway._router.deployed-routes", is no longer displayed.

Monitor Vert.x metrics

Vert.x metric names are deprecated and expected to be replaced with names ending in _total. The information provided by the metric is not deprecated. Other Prometheus metrics are not affected.

Vert.x monitoring is enabled by default to provide metrics for HTTP, TCP, and the internal component pool. The metrics provide low-level information about requests and responses, such as the number of bytes, duration, the number of concurrent requests, and so on.

To disable Vert.x monitoring, add the following lines to admin.json, and restart PingGateway:

{
  "vertx": {
    "metricsEnabled": false
  }
}

For more information, refer to AdminHttpApplication (admin.json).

Protect monitoring endpoints

By default, no special credentials or privileges are required for read-access to the Prometheus Scrape Endpoint and Common REST Monitoring Endpoint.

To protect the monitoring endpoints, add an admin.json file to your configuration, with a filter declared in the heap and named MetricsProtectionFilter. The following procedure gives an example of how to manage access to the monitoring endpoints.

Set up the procedure in Monitor at the Prometheus Scrape Endpoint, query the Prometheus Scrape Endpoint, and note that metrics for myroute1 and _router are displayed:
```
$ curl -v "https://ig.example.com:8443/openig/metrics/prometheus"
```

Add the following script to the PingGateway configuration:

Linux
Windows

$HOME/.openig/scripts/groovy/BasicAuthResourceServerFilter.groovy

%appdata%\OpenIG\scripts\groovy\BasicAuthResourceServerFilter.groovy

/*
 * This script is a simple implementation of HTTP basic access authentication on
 * server side.
 * It expects the following arguments:
 *  - realm: the realm to display when the user agent prompts for
 *    username and password if none were provided.
 *  - username: the expected username
 *  - passwordSecretId: the secretId to find the password
 *  - secretsProvider: the SecretsProvider to query for the password
*/
import static org.forgerock.util.promise.Promises.newResultPromise;

import java.nio.charset.Charset;
import org.forgerock.util.encode.Base64;
import org.forgerock.secrets.Purpose;
import org.forgerock.secrets.GenericSecret;

String authorizationHeader = request.getHeaders().getFirst("Authorization");
if (authorizationHeader == null) {
    // No credentials provided, return 401 Unauthorized
    Response response = new Response(Status.UNAUTHORIZED);
    response.getHeaders().put("WWW-Authenticate", "Basic realm=\"" + realm + "\"");
    return newResultPromise(response);
}

return secretsProvider.getNamed(Purpose.PASSWORD, passwordSecretId)
        .thenAsync(password -> {
            // Build basic authentication string -> username:password
            StringBuilder basicAuthString = new StringBuilder(username).append(":");
            password.revealAsUtf8{ p -> basicAuthString.append(new String(p).trim()) };
            String expectedAuthorization = "Basic " + Base64.encode(basicAuthString.toString().getBytes(Charset.defaultCharset()));
            // Incorrect credentials provided, return 403 forbidden
            if (!expectedAuthorization.equals(authorizationHeader)) {
                return newResultPromise(new Response(Status.FORBIDDEN));
            }
            // Correct credentials provided, continue.
            return next.handle(context, request);
        },
                noSuchSecretException -> { throw new RuntimeException(noSuchSecretException); });

The script is a simple implementation of the HTTP basic access authentication scheme. For information about scripting filters and handlers, refer to Extend.

Add the following heap configuration for MetricsProtectionFilter to admin.json:

{
  ...,
  "heap": [
    {
      "name": "ClientHandler",
      "type": "ClientHandler"
    },
    {
      "name": "mySecretsProvider",
      "type": "Base64EncodedSecretStore",
      "config": {
        "secrets": {
          "password.secret.id": "cGFzc3dvcmQ="
        }
      }
    },
    {
      "name": "MetricsProtectionFilter",
      "type": "ScriptableFilter",
      "config": {
        "type": "application/x-groovy",
        "file": "BasicAuthResourceServerFilter.groovy",
        "args": {
          "realm": "/",
          "username": "myUsername",
          "passwordSecretId": "password.secret.id",
          "secretsProvider": "${heap['mySecretsProvider']}"
        }
      }
    }
  ],
  ...
}

Notice the following features of the configuration:

The MetricsProtectionFilter uses the script to protect the monitoring endpoint.
The MetricsProtectionFilter requires the username myUsername, and a password provided by the SecretsProvider in the heap.

Restart PingGateway.
Query the Prometheus Scrape Endpoint without providing credentials, and note that an HTTP 401 Unauthorized is returned:
```
$ curl -v "https://ig.example.com:8443/openig/metrics/prometheus"
```
Query the Prometheus Scrape Endpoint by providing correct credentials, and note that metrics are displayed:
```
$ curl -v "https://ig.example.com:8443/openig/metrics/prometheus" -u myUsername:password
```
Query the Prometheus Scrape Endpoint by providing incorrect credentials`, and note that an HTTP 403 Forbidden is returned:
```
$ curl -v "https://ig.example.com:8443/openig/metrics/prometheus" -u myUsername:wrong-password
```

PingGateway 2024.9