---
title: "Monitor ITRS Analytics using the Gateway template"
description: "Monitor ITRS Analytics in Geneos with the Gateway template included in the Gateway package. The template calls the Ingestion self-monitoring API and covers the ingestion pipeline, metrics pipeline latency, storage volumes, and workloads."
url: "https://docs.itrsgroup.com/docs/geneos/7.12.0/processing/data-access-and-publishing/monitor-itrs-analytics-using-gateway-template/index.html"
---
## Overview

You can monitor ITRS Analytics and display the dataviews in Active Console using the template (`templates/iax-monitoring.xml`) included in the Gateway package.

The template uses the [REST Extractor](https://docs.itrsgroup.com/docs/geneos/7.12.0/collection/rest-extractor/index.html) plugin. Each sampler calls the Ingestion self-monitoring API and converts the JSON response into dataviews for these areas:

- Ingestion pipeline: consumer groups and ingestion rates
- Metrics pipeline latency
- Storage volumes
- Workloads: container status, restarts, CPU and memory, and exit codes

In Gateway Setup Editor, the samplers and rules are in the **IAX Ingestion Monitoring** group.

## Prerequisites

Before you include the template, confirm the following:


**Note**

 The template is available in Gateway version 7.12.2 or later. 


- The Netprobe that runs the samplers is version 7.12.2 or later and meets the [REST Extractor prerequisites](https://docs.itrsgroup.com/docs/geneos/7.12.0/collection/rest-extractor/index.html#prerequisites) (including the Java requirements). The template uses [bearer token authentication](https://docs.itrsgroup.com/docs/geneos/7.12.0/collection/rest-extractor/index.html#bearer-token-authentication).
- The Netprobe can open an HTTPS connection to the ITRS Analytics host. Each request URL is `$(IAX_INSTANCE)/ingestion/api/v1/...`.

## How to monitor ITRS Analytics using the template

To monitor ITRS Analytics, follow these steps:

1. Create an ITRS Analytics API token with the `user` role.
2. Include `templates/iax-monitoring.xml` in the Gateway setup.
3. Set `IAX_INSTANCE` and `IAX_API_TOKEN` in **ITRS Analytics Environment**.
4. Add the **ITRS Analytics Health** type to a managed entity named **ITRS Analytics Health**.

### Create an ITRS Analytics API token

The samplers call the Ingestion self-monitoring API with a bearer token. Create an API token with the `user` role. Enter the token as the value of `IAX_API_TOKEN`.

1. In ITRS Analytics, go to **Admin** > **IAM** > **API Tokens**.
2. Select **Add API Token**.
3. Create a token with the `user` role.
4. Copy the API token.

For more information, see [IAM app API tokens](https://docs.itrsgroup.com/docs/itrs-analytics/app/iam/current/user-guide/iam-configuration/index.html#api-tokens).

### Include the template

To include `iax-monitoring.xml` in the Gateway setup, follow these steps:

1. Open Gateway Setup Editor.
2. In the **Navigation** panel, right-click **Includes** to create a new file.
3. In the **Location** field, enter the path to `iax-monitoring.xml` in the Gateway package `templates` directory.
4. In the **Priority** field, enter any value except `1`. If you enter `1`, the Gateway Setup Editor returns an error.
5. Expand the file location in the **Includes** section.
6. Select **Click to load**.
7. Select **Yes** to load the include file.

For more information, see [Include files](https://docs.itrsgroup.com/docs/geneos/7.12.0/processing/configuration/gse_user_guide/index.html#include-files).

### Set up the variables

The template defines **ITRS Analytics Environment**. Replace each `CHANGE_ME` value. For more information, see [User Variables and Environments](https://docs.itrsgroup.com/docs/geneos/7.12.0/processing/gateway-administration/gateway_user_variables_and_environments/index.html).

| Name | Type | Description |
| --- | --- | --- |
| `IAX_INSTANCE` | string | ITRS Analytics base URL, including `https://` and with no trailing slash. For example, `https://example.analytics.itrsgroup.com`. If HTTPS uses a port other than 443, include the port. Each request URL is `$(IAX_INSTANCE)/ingestion/api/v1/...`. |
| `IAX_API_TOKEN` | password | API token that the samplers send as a bearer token in the `Authorization` header, using REST Extractor [bearer token authentication](https://docs.itrsgroup.com/docs/geneos/7.12.0/collection/rest-extractor/index.html#bearer-token-authentication). Create a password variable and store the token as an AES-encrypted password (`stdAES`), or retrieve it from an external password provider (`extPwd`). The sampler adds the `Bearer` prefix. For more information, see [Secure passwords](https://docs.itrsgroup.com/docs/geneos/7.12.0/security/access-controls/gateway_secure_passwords/index.html#secure-passwords). |

### Add the type

1. Select **Managed entities** in the **Navigation** panel.
2. On the managed entity that monitors ITRS Analytics, set the name to **ITRS Analytics Health**. The sampling status rule applies only to that name.
3. Under **Add types**, select **Add new**, and then select the type **ITRS Analytics Health**.
4. Set the environment for that type to **ITRS Analytics Environment**.

The probe for that managed entity must be able to reach the ITRS Analytics host.

The type references the samplers `consumer-groups`, `ingestion-rates`, `metrics-pipeline-latency`, `volumes`, and `workloads`. The included rules target those names. Keep the names so the rules continue to apply.

### Validate and save the setup

1. Select **Validate current document** ![](https://docscdn.itrsgroup.com/geneos/images/Icons/validate22x22.png "icon") to validate the configuration.
2. Select **Save current document** ![](https://docscdn.itrsgroup.com/geneos/images/Icons/save16x16.png "icon") to apply the changes.

After you save, the dataviews appear in Active Console under the managed entity. For the columns, see [Metrics and dataviews](#metrics-and-dataviews).

## Metrics and dataviews

The five **IAX Ingestion Monitoring** samplers publish these dataviews. Each sampler runs every 60 seconds. For the current API, see [Ingestion API](https://docs.itrsgroup.com/docs/itrs-analytics/app/ingestion/current/user-guide/ingestion-api/index.html#self-monitoring).

| Sampler | Dataviews |
| --- | --- |
| `consumer-groups` | `consumer-groups`, `consumer-groups-unknown` |
| `ingestion-rates` | `ingestion-summary`, `received`, `ingested`, `dropped` |
| `metrics-pipeline-latency` | `metrics-pipeline-latency-latest`, `metrics-pipeline-latency-overall` |
| `volumes` | `volumes-summary`, `volumes-totals` |
| `workloads` | `workloads-summary`, `workloads-restarts` |

### Consumer groups

The `consumer-groups` sampler publishes the lag for each consumer group and topic. Both dataviews use these columns. Rows are sorted by `totalLag`, with the highest lag first.

`consumer-groups` lists groups whose summary does not contain unknown partitions, so `unknowns` is `false`. `consumer-groups-unknown` lists groups whose summary contains unknown partitions, so `unknowns` is `true`.

| Column | Description |
| --- | --- |
| key | Consumer group and topic, separated by `/`. |
| group | Consumer group name. |
| topic | Topic name. |
| numPartitions | Number of partitions in the group for that topic. |
| totalLag | Total lag for the group and topic. |
| minLag | Lowest partition lag. |
| maxLag | Highest partition lag. |
| maxLagPct | Maximum lag ratio returned by the API, converted to a percentage and rounded to two decimal places. |
| unknowns | `true` when the group contains unknown partitions. Otherwise `false`. |

### Ingestion rates

The `ingestion-rates` sampler publishes data point counts for the summary window. In `received`, `ingested`, and `dropped`, counts are for the last hour. The group `GROUP_ALL` is the total for all groups.

#### Ingestion summary

`ingestion-summary` has one row for each field. The `name` column identifies the field.

| Name | Description |
| --- | --- |
| windowStart | Start of the summary window. |
| windowEnd | End of the summary window. |
| receivedRatePerSec | Data points received per second, to two decimal places. |
| receivedCount | Data points received. |
| ingestedCount | Data points ingested. |
| externalCount | External data points received. |
| internalCount | Internal data points received. |
| invalidCount | Invalid data points. |
| uncontainedCount | Uncontained data points. |
| droppedFilteredCount | Filtered data points. |
| ingestedPct | Ingested data points as a percentage of data points received, to two decimal places. The value is 0 when no data points were received. |
| invalidPct | Invalid data points as a percentage of data points received, to two decimal places. The value is 0 when no data points were received. |
| uncontainedPct | Uncontained data points as a percentage of data points received, to two decimal places. The value is 0 when no data points were received. |
| filteredPct | Filtered data points as a percentage of data points received, to two decimal places. The value is 0 when no data points were received. |

#### Received

`received` publishes one row for each data point group. Rows are sorted by `totalReceivedLastHour`, with the highest count first.

| Column | Description |
| --- | --- |
| group | Data point group name. |
| externalLastHour | External data points received. |
| internalLastHour | Internal data points received. |
| totalReceivedLastHour | Total data points received. |

#### Ingested

`ingested` publishes one row for each data point group. Rows are sorted by `totalIngestedLastHour`, with the highest count first.

| Column | Description |
| --- | --- |
| group | Data point group name. |
| externalLastHour | External data points ingested. |
| internalLastHour | Internal data points ingested. |
| totalIngestedLastHour | Total data points ingested. |

#### Dropped

`dropped` lists groups with a dropped count greater than zero, and the `GROUP_ALL` row. Rows are sorted by `totalDroppedLastHour`, with the highest count first.

| Column | Description |
| --- | --- |
| group | Data point group name. |
| invalidLastHour | Invalid data points. |
| uncontainedLastHour | Uncontained data points. |
| filteredLastHour | Filtered data points. |
| totalDroppedLastHour | Sum of invalid, uncontained, and filtered data points. |

### Metrics pipeline latency

The `metrics-pipeline-latency` sampler publishes pipeline latency.

`metrics-pipeline-latency-latest` shows each host and topic in the latest sample. It uses every column in the following table. Rows are sorted by `maxLatencyMs`, with the highest latency first.

`metrics-pipeline-latency-overall` contains one row for all hosts and topics. That row includes `key`, `maxLatencyMs`, and `avgLatencyMs`. The value of `key` is `all`.

| Column | Description |
| --- | --- |
| key | Hostname and topic, separated by `/`. In `metrics-pipeline-latency-overall`, the value is `all`. |
| hostname | Host that produced the sample. |
| topic | Topic name. |
| maxLatencyMs | Maximum pipeline latency, in milliseconds. |
| avgLatencyMs | Average pipeline latency, in milliseconds. |

### Storage volumes

The `volumes` sampler publishes storage size and usage.

`volumes-summary` shows each storage volume and uses every column in the following table. Rows are sorted by `usedPct`, with the highest percentage in use first.

`volumes-totals` contains one row for all volumes. That row includes `key`, `usedPct`, `usedGiB`, and `sizeGiB`. The value of `key` is `all`.

| Column | Description |
| --- | --- |
| key | Pod name and volume name, separated by `/`. In `volumes-totals`, the value is `all`. |
| podName | Pod that owns the volume. |
| volumeName | Volume name. |
| usedPct | Percentage of size in use, rounded to two decimal places. |
| usedGiB | Used space in gibibytes, rounded to two decimal places. |
| sizeGiB | Volume size in gibibytes, rounded to two decimal places. |

### Workloads

The `workloads` sampler publishes container state and resource usage. Both dataviews omit containers whose names start with `linkerd`, and containers whose state is `terminated` or `evicted`. Rows are sorted by `restarts`, with the highest restart count first.

`workloads-summary` lists the remaining containers. `workloads-restarts` lists containers with a restart count greater than zero. `lastExitCode` is only on `workloads-restarts`.

| Column | Description |
| --- | --- |
| key | Pod name and container name, separated by `/`. |
| podName | Pod that owns the container. |
| containerName | Container name. |
| status | Current container state, in uppercase. `RUNNING` is OK under the included rule. |
| restarts | Container restart count. |
| restartAgeMins | Minutes since the current process started. The value is `0` if the container has not restarted or if no start time is available. |
| lastExitCode | Exit code of the previous container state. The cell is empty when the API does not report a code. `0` and `143` are OK under the included rule. |
| cpuRequestPct | Current CPU use as a percentage of the CPU request. |
| cpuLimitPct | Current CPU use as a percentage of the CPU limit. |
| memRequestPct | Current memory use as a percentage of the memory request. |
| memLimitPct | Current memory use as a percentage of the memory limit. |
| cpuRequestCores | Requested CPU, in cores. |
| cpuLimitCores | CPU limit, in cores. |
| memRequestMiB | Requested memory, in mebibytes. |
| memLimitMiB | Memory limit, in mebibytes. |

## Rules

Each rule is in the **IAX Ingestion Monitoring** group.

### Sampling status

The `sampling status` rule applies to the `samplingStatus` headline on every sampler under the managed entity **ITRS Analytics Health**.

| Condition | Severity |
| --- | --- |
| Value is `OK` | OK |
| Any other value | Warning after two samples |

When `samplingStatus` is not `OK`, that sampler did not complete the sample. The headline text describes the HTTP or query failure. Rows in that dataview can be stale or empty. Do not use cell severity in that dataview until the headline is `OK`.

To investigate the headline:

1. Read the headline text. It names the HTTP or query failure.
2. Confirm that `IAX_INSTANCE` is the base URL, including `https://` and with no trailing slash, and that `IAX_API_TOKEN` is a current token with the `user` role.
3. From the Netprobe host, confirm that HTTPS to that URL succeeds.
4. If one sampler fails, the API path for that sampler is failing. If every sampler fails, check the token, the URL, and network connectivity.

### iax-workload-status

The `iax-workload-status` rule applies to the `status` column on `workloads-summary`.

| Condition | Severity |
| --- | --- |
| Status is `RUNNING` | OK |
| Any other status | Critical after two samples |

A Critical value in `status` means that container is not running. The value can be a pending, waiting, or failed state.

To investigate a critical status:

1. Read `podName` and `containerName` on the critical row.
2. Open `workloads-restarts` for the same container, and read `lastExitCode`.
3. If the exit code rule is also Warning, follow the steps in [iax-container-exit-code](#iax-container-exit-code). Exit code 137 means the container was killed because it exceeded its memory limit.
4. If the status stays other than `RUNNING` and the exit code is 0 or 143, the pod was stopped intentionally or is still starting. Check whether a rollout is in progress before you escalate.
5. If the container should be running and does not return to `RUNNING`, collect an ITRS Analytics support bundle and escalate to the platform owner.

### iax-workload-limits

The `iax-workload-limits` rule applies to `cpuLimitPct` on `workloads-summary`, and to `memLimitPct` on `workloads-summary` except rows named `kafka-*/kafka`. It does not use the request columns.

| Condition | Severity |
| --- | --- |
| Value is greater than 95 percent | Warning after two samples |
| Value is 95 percent or less | OK |

A Warning on `memLimitPct` means that container is using more than 95 percent of its memory limit and can be killed. A Warning on `cpuLimitPct` means that container is using more than 95 percent of its CPU limit. The CPU is throttled, which slows ingestion and can increase consumer lag.

The rule skips `memLimitPct` on Kafka broker rows (`kafka-*/kafka`). Memory usage for those brokers includes the page cache, so a healthy broker is near 100 percent. The CPU check still applies to those rows.

To investigate a resource warning:

1. Read the warning column, `podName`, and `containerName`.
2. For `memLimitPct`, open `workloads-restarts`. If `lastExitCode` is 137, the container was killed because it exceeded its memory limit. Raise the memory limit, or find what is using the memory. A restart that keeps the same limit can kill the container again.
3. For `cpuLimitPct`, check `metrics-pipeline-latency-latest` and `consumer-groups` for the same time. If latency is greater than 10 seconds, or if `maxLagPct` is greater than 90, the throttle is already delaying the pipeline. Raise the CPU limit, or reduce the work that container processes.
4. Kafka, ClickHouse, and ingestion containers often run close to their limits. If the same container remains greater than 95 percent and the pipeline is on time, the limit is low for the normal workload. Record that baseline. Raise the threshold only after you and the platform owner agree.

### iax-container-exit-code

The `iax-container-exit-code` rule applies to the `lastExitCode` column on `workloads-restarts`.

| Condition | Severity |
| --- | --- |
| Exit code is 0 or 143 | OK |
| Any other value | Warning |

A Warning means the previous container process ended with an exit code other than 0 or 143. Use the exit code to identify the cause:

| Exit code | Description |
| --- | --- |
| 0 | The previous process exited successfully. |
| 143 | The previous process was stopped with SIGTERM. This is normal for a rollout or a planned stop. |
| 137 | The previous process was killed with SIGKILL. In Kubernetes, this usually means the container exceeded its memory limit. |
| 139 | The previous process ended with a segmentation fault. |
| 125 | The container runtime could not run the command. |

To investigate an exit code:

1. For exit code 137, read `memLimitPct` and `memLimitMiB` for that container. Increase its memory limit. A restart keeps the same limit.
2. For exit code 139 or 125, the container image or the process failed before it could run. Escalate to the platform owner with the pod name, the container name, and the exit code. Try one restart. If the same code occurs again, include the image name and the pod log.
3. For exit code 0 or 143, this rule requires no action. If `status` on `workloads-summary` is not `RUNNING`, follow the steps in [iax-workload-status](#iax-workload-status).
4. For any other exit code, open the pod log for that container.

### iax-consumer-group-maxLagPct

The `iax-consumer-group-maxLagPct` rule applies to the `maxLagPct` column on `consumer-groups`. It does not set severity on `totalLag`.

| Condition | Severity |
| --- | --- |
| `maxLagPct` is greater than 90 | Warning after two samples |
| `maxLagPct` is 90 or less | OK |

A Warning on `maxLagPct` means that, on the most delayed partition for that group and topic, unread messages are greater than 90 percent of the log end offset. The consumer has read little of that partition. Metrics and entities from that topic are late. A consumer this far behind can lose messages when log retention passes its position.

To investigate consumer lag:

1. Read `group`, `topic`, `maxLag`, and `totalLag` on the warning row. Confirm that the lag is a large backlog, rather than a new topic with a short log.
2. Find the pod that consumes that group. On `workloads-summary`, check whether the pod is `RUNNING` and whether it is at its CPU or memory limit.
3. Open `metrics-pipeline-latency-latest` for the same topic. A `maxLatencyMs` warning on that topic means the pipeline is slow, in addition to the consumer.
4. If the consumer pod is stopped, start it. Do not delete the consumer group to clear the lag. Deleting the group removes its committed position.
5. If the pod is running and `maxLagPct` stays greater than 90 for several samples, the consumer is stuck or cannot keep up with the topic. Escalate with the group, the topic, `maxLag`, and `maxLagPct`.

### iax-volume-used-pct

The `iax-volume-used-pct` rule applies to the `usedPct` column on `volumes-summary` and `volumes-totals`.

| Condition | Severity |
| --- | --- |
| Usage is greater than 90 percent | Warning |
| Usage is 90 percent or less | OK |

A Warning on `volumes-summary` means that volume is more than 90 percent full. A Warning on `volumes-totals` means overall volume usage is more than 90 percent. Kafka and ClickHouse stop accepting writes when their volume fills. Consumer lag and ingestion delay then increase.

To investigate volume usage:

1. If the warning is on `volumes-summary`, the row is one pod and one volume. Read `volumeName` to identify the volume.
2. If the warning is only on `volumes-totals`, open `volumes-summary` and sort by `usedPct`. Act on the volume with the highest usage. The totals row is overall usage, not one volume.
3. Check `consumer-groups` for a group that is not reading. A consumer that is not reading can leave Kafka retaining data until the volume fills.
4. Restore the consumer, or expand the volume. If you expand the volume and leave the consumer stopped, the volume can fill again.
5. If `usedPct` continues to increase after the consumer is healthy, escalate with the pod, the volume name, `usedPct`, `usedGiB`, and `sizeGiB`.

### iax-metrics-pipeline-latency

The `iax-metrics-pipeline-latency` rule applies to the `maxLatencyMs` column on `metrics-pipeline-latency-latest` and `metrics-pipeline-latency-overall`. It does not set severity on `avgLatencyMs`.

| Condition | Severity |
| --- | --- |
| Latency is greater than 10,000 milliseconds | Warning after two samples |
| Latency is 10,000 milliseconds or less | OK |

A Warning on `maxLatencyMs` means the slowest message through the metrics pipeline took more than 10 seconds. On `metrics-pipeline-latency-latest`, that value is for one host and one topic in the latest sample. On `metrics-pipeline-latency-overall`, that value is the slowest result in the API response. Metrics on that path are late by at least that long.

To investigate pipeline latency:

1. On `metrics-pipeline-latency-latest`, read `hostname` and `topic`. Investigate that path. Use the overall row only to confirm that a path is slow.
2. Open `consumer-groups` and find that topic. A `maxLagPct` warning means the consumer is also behind.
3. Open `workloads-summary` for the pod on that host. A CPU limit warning means the pod is throttled. A memory limit warning, or exit code 137, means the container was killed because it exceeded its memory limit.
4. If the same host and topic stays greater than 10,000 milliseconds for several samples, escalate with the host, the topic, `maxLatencyMs`, and the consumer lag for that topic.