Monitor ITRS Analytics using the Gateway template

Overview Copied

You can monitor ITRS Analytics and display the dataviews in Active Console using the template (templates/iax-monitoring.xml) included in the Gateway package.

The template uses the REST Extractor plugin. Each sampler calls the Ingestion self-monitoring API and converts the JSON response into dataviews for these areas:

In Gateway Setup Editor, the samplers and rules are in the IAX Ingestion Monitoring group.

Prerequisites Copied

Before you include the template, confirm the following:

Note

The template is available in Gateway version 7.12.2 or later.

How to monitor ITRS Analytics using the template Copied

To monitor ITRS Analytics, follow these steps:

  1. Create an ITRS Analytics API token with the user role.
  2. Include templates/iax-monitoring.xml in the Gateway setup.
  3. Set IAX_INSTANCE and IAX_API_TOKEN in ITRS Analytics Environment.
  4. Add the ITRS Analytics Health type to a managed entity named ITRS Analytics Health.

Create an ITRS Analytics API token Copied

The samplers call the Ingestion self-monitoring API with a bearer token. Create an API token with the user role. Enter the token as the value of IAX_API_TOKEN.

  1. In ITRS Analytics, go to Admin > IAM > API Tokens.
  2. Select Add API Token.
  3. Create a token with the user role.
  4. Copy the API token.

For more information, see IAM app API tokens.

Include the template Copied

To include iax-monitoring.xml in the Gateway setup, follow these steps:

  1. Open Gateway Setup Editor.
  2. In the Navigation panel, right-click Includes to create a new file.
  3. In the Location field, enter the path to iax-monitoring.xml in the Gateway package templates directory.
  4. In the Priority field, enter any value except 1. If you enter 1, the Gateway Setup Editor returns an error.
  5. Expand the file location in the Includes section.
  6. Select Click to load.
  7. Select Yes to load the include file.

For more information, see Include files.

Set up the variables Copied

The template defines ITRS Analytics Environment. Replace each CHANGE_ME value. For more information, see User Variables and Environments.

Name Type Description
IAX_INSTANCE string ITRS Analytics base URL, including https:// and with no trailing slash. For example, https://example.analytics.itrsgroup.com. If HTTPS uses a port other than 443, include the port. Each request URL is $(IAX_INSTANCE)/ingestion/api/v1/....
IAX_API_TOKEN password API token that the samplers send as a bearer token in the Authorization header, using REST Extractor bearer token authentication. Create a password variable and store the token as an AES-encrypted password (stdAES), or retrieve it from an external password provider (extPwd). The sampler adds the Bearer prefix. For more information, see Secure passwords.

Add the type Copied

  1. Select Managed entities in the Navigation panel.
  2. On the managed entity that monitors ITRS Analytics, set the name to ITRS Analytics Health. The sampling status rule applies only to that name.
  3. Under Add types, select Add new, and then select the type ITRS Analytics Health.
  4. Set the environment for that type to ITRS Analytics Environment.

The probe for that managed entity must be able to reach the ITRS Analytics host.

The type references the samplers consumer-groups, ingestion-rates, metrics-pipeline-latency, volumes, and workloads. The included rules target those names. Keep the names so the rules continue to apply.

Validate and save the setup Copied

  1. Select Validate current document to validate the configuration.
  2. Select Save current document to apply the changes.

After you save, the dataviews appear in Active Console under the managed entity. For the columns, see Metrics and dataviews.

Metrics and dataviews Copied

The five IAX Ingestion Monitoring samplers publish these dataviews. Each sampler runs every 60 seconds. For the current API, see Ingestion API.

Sampler Dataviews
consumer-groups consumer-groups, consumer-groups-unknown
ingestion-rates ingestion-summary, received, ingested, dropped
metrics-pipeline-latency metrics-pipeline-latency-latest, metrics-pipeline-latency-overall
volumes volumes-summary, volumes-totals
workloads workloads-summary, workloads-restarts

Consumer groups Copied

The consumer-groups sampler publishes the lag for each consumer group and topic. Both dataviews use these columns. Rows are sorted by totalLag, with the highest lag first.

consumer-groups lists groups whose summary does not contain unknown partitions, so unknowns is false. consumer-groups-unknown lists groups whose summary contains unknown partitions, so unknowns is true.

Column Description
key Consumer group and topic, separated by /.
group Consumer group name.
topic Topic name.
numPartitions Number of partitions in the group for that topic.
totalLag Total lag for the group and topic.
minLag Lowest partition lag.
maxLag Highest partition lag.
maxLagPct Maximum lag ratio returned by the API, converted to a percentage and rounded to two decimal places.
unknowns true when the group contains unknown partitions. Otherwise false.

Ingestion rates Copied

The ingestion-rates sampler publishes data point counts for the summary window. In received, ingested, and dropped, counts are for the last hour. The group GROUP_ALL is the total for all groups.

Ingestion summary Copied

ingestion-summary has one row for each field. The name column identifies the field.

Name Description
windowStart Start of the summary window.
windowEnd End of the summary window.
receivedRatePerSec Data points received per second, to two decimal places.
receivedCount Data points received.
ingestedCount Data points ingested.
externalCount External data points received.
internalCount Internal data points received.
invalidCount Invalid data points.
uncontainedCount Uncontained data points.
droppedFilteredCount Filtered data points.
ingestedPct Ingested data points as a percentage of data points received, to two decimal places. The value is 0 when no data points were received.
invalidPct Invalid data points as a percentage of data points received, to two decimal places. The value is 0 when no data points were received.
uncontainedPct Uncontained data points as a percentage of data points received, to two decimal places. The value is 0 when no data points were received.
filteredPct Filtered data points as a percentage of data points received, to two decimal places. The value is 0 when no data points were received.

Received Copied

received publishes one row for each data point group. Rows are sorted by totalReceivedLastHour, with the highest count first.

Column Description
group Data point group name.
externalLastHour External data points received.
internalLastHour Internal data points received.
totalReceivedLastHour Total data points received.

Ingested Copied

ingested publishes one row for each data point group. Rows are sorted by totalIngestedLastHour, with the highest count first.

Column Description
group Data point group name.
externalLastHour External data points ingested.
internalLastHour Internal data points ingested.
totalIngestedLastHour Total data points ingested.

Dropped Copied

dropped lists groups with a dropped count greater than zero, and the GROUP_ALL row. Rows are sorted by totalDroppedLastHour, with the highest count first.

Column Description
group Data point group name.
invalidLastHour Invalid data points.
uncontainedLastHour Uncontained data points.
filteredLastHour Filtered data points.
totalDroppedLastHour Sum of invalid, uncontained, and filtered data points.

Metrics pipeline latency Copied

The metrics-pipeline-latency sampler publishes pipeline latency.

metrics-pipeline-latency-latest shows each host and topic in the latest sample. It uses every column in the following table. Rows are sorted by maxLatencyMs, with the highest latency first.

metrics-pipeline-latency-overall contains one row for all hosts and topics. That row includes key, maxLatencyMs, and avgLatencyMs. The value of key is all.

Column Description
key Hostname and topic, separated by /. In metrics-pipeline-latency-overall, the value is all.
hostname Host that produced the sample.
topic Topic name.
maxLatencyMs Maximum pipeline latency, in milliseconds.
avgLatencyMs Average pipeline latency, in milliseconds.

Storage volumes Copied

The volumes sampler publishes storage size and usage.

volumes-summary shows each storage volume and uses every column in the following table. Rows are sorted by usedPct, with the highest percentage in use first.

volumes-totals contains one row for all volumes. That row includes key, usedPct, usedGiB, and sizeGiB. The value of key is all.

Column Description
key Pod name and volume name, separated by /. In volumes-totals, the value is all.
podName Pod that owns the volume.
volumeName Volume name.
usedPct Percentage of size in use, rounded to two decimal places.
usedGiB Used space in gibibytes, rounded to two decimal places.
sizeGiB Volume size in gibibytes, rounded to two decimal places.

Workloads Copied

The workloads sampler publishes container state and resource usage. Both dataviews omit containers whose names start with linkerd, and containers whose state is terminated or evicted. Rows are sorted by restarts, with the highest restart count first.

workloads-summary lists the remaining containers. workloads-restarts lists containers with a restart count greater than zero. lastExitCode is only on workloads-restarts.

Column Description
key Pod name and container name, separated by /.
podName Pod that owns the container.
containerName Container name.
status Current container state, in uppercase. RUNNING is OK under the included rule.
restarts Container restart count.
restartAgeMins Minutes since the current process started. The value is 0 if the container has not restarted or if no start time is available.
lastExitCode Exit code of the previous container state. The cell is empty when the API does not report a code. 0 and 143 are OK under the included rule.
cpuRequestPct Current CPU use as a percentage of the CPU request.
cpuLimitPct Current CPU use as a percentage of the CPU limit.
memRequestPct Current memory use as a percentage of the memory request.
memLimitPct Current memory use as a percentage of the memory limit.
cpuRequestCores Requested CPU, in cores.
cpuLimitCores CPU limit, in cores.
memRequestMiB Requested memory, in mebibytes.
memLimitMiB Memory limit, in mebibytes.

Rules Copied

Each rule is in the IAX Ingestion Monitoring group.

Sampling status Copied

The sampling status rule applies to the samplingStatus headline on every sampler under the managed entity ITRS Analytics Health.

Condition Severity
Value is OK OK
Any other value Warning after two samples

When samplingStatus is not OK, that sampler did not complete the sample. The headline text describes the HTTP or query failure. Rows in that dataview can be stale or empty. Do not use cell severity in that dataview until the headline is OK.

To investigate the headline:

  1. Read the headline text. It names the HTTP or query failure.
  2. Confirm that IAX_INSTANCE is the base URL, including https:// and with no trailing slash, and that IAX_API_TOKEN is a current token with the user role.
  3. From the Netprobe host, confirm that HTTPS to that URL succeeds.
  4. If one sampler fails, the API path for that sampler is failing. If every sampler fails, check the token, the URL, and network connectivity.

iax-workload-status Copied

The iax-workload-status rule applies to the status column on workloads-summary.

Condition Severity
Status is RUNNING OK
Any other status Critical after two samples

A Critical value in status means that container is not running. The value can be a pending, waiting, or failed state.

To investigate a critical status:

  1. Read podName and containerName on the critical row.
  2. Open workloads-restarts for the same container, and read lastExitCode.
  3. If the exit code rule is also Warning, follow the steps in iax-container-exit-code. Exit code 137 means the container was killed because it exceeded its memory limit.
  4. If the status stays other than RUNNING and the exit code is 0 or 143, the pod was stopped intentionally or is still starting. Check whether a rollout is in progress before you escalate.
  5. If the container should be running and does not return to RUNNING, collect an ITRS Analytics support bundle and escalate to the platform owner.

iax-workload-limits Copied

The iax-workload-limits rule applies to cpuLimitPct on workloads-summary, and to memLimitPct on workloads-summary except rows named kafka-*/kafka. It does not use the request columns.

Condition Severity
Value is greater than 95 percent Warning after two samples
Value is 95 percent or less OK

A Warning on memLimitPct means that container is using more than 95 percent of its memory limit and can be killed. A Warning on cpuLimitPct means that container is using more than 95 percent of its CPU limit. The CPU is throttled, which slows ingestion and can increase consumer lag.

The rule skips memLimitPct on Kafka broker rows (kafka-*/kafka). Memory usage for those brokers includes the page cache, so a healthy broker is near 100 percent. The CPU check still applies to those rows.

To investigate a resource warning:

  1. Read the warning column, podName, and containerName.
  2. For memLimitPct, open workloads-restarts. If lastExitCode is 137, the container was killed because it exceeded its memory limit. Raise the memory limit, or find what is using the memory. A restart that keeps the same limit can kill the container again.
  3. For cpuLimitPct, check metrics-pipeline-latency-latest and consumer-groups for the same time. If latency is greater than 10 seconds, or if maxLagPct is greater than 90, the throttle is already delaying the pipeline. Raise the CPU limit, or reduce the work that container processes.
  4. Kafka, ClickHouse, and ingestion containers often run close to their limits. If the same container remains greater than 95 percent and the pipeline is on time, the limit is low for the normal workload. Record that baseline. Raise the threshold only after you and the platform owner agree.

iax-container-exit-code Copied

The iax-container-exit-code rule applies to the lastExitCode column on workloads-restarts.

Condition Severity
Exit code is 0 or 143 OK
Any other value Warning

A Warning means the previous container process ended with an exit code other than 0 or 143. Use the exit code to identify the cause:

Exit code Description
0 The previous process exited successfully.
143 The previous process was stopped with SIGTERM. This is normal for a rollout or a planned stop.
137 The previous process was killed with SIGKILL. In Kubernetes, this usually means the container exceeded its memory limit.
139 The previous process ended with a segmentation fault.
125 The container runtime could not run the command.

To investigate an exit code:

  1. For exit code 137, read memLimitPct and memLimitMiB for that container. Increase its memory limit. A restart keeps the same limit.
  2. For exit code 139 or 125, the container image or the process failed before it could run. Escalate to the platform owner with the pod name, the container name, and the exit code. Try one restart. If the same code occurs again, include the image name and the pod log.
  3. For exit code 0 or 143, this rule requires no action. If status on workloads-summary is not RUNNING, follow the steps in iax-workload-status.
  4. For any other exit code, open the pod log for that container.

iax-consumer-group-maxLagPct Copied

The iax-consumer-group-maxLagPct rule applies to the maxLagPct column on consumer-groups. It does not set severity on totalLag.

Condition Severity
maxLagPct is greater than 90 Warning after two samples
maxLagPct is 90 or less OK

A Warning on maxLagPct means that, on the most delayed partition for that group and topic, unread messages are greater than 90 percent of the log end offset. The consumer has read little of that partition. Metrics and entities from that topic are late. A consumer this far behind can lose messages when log retention passes its position.

To investigate consumer lag:

  1. Read group, topic, maxLag, and totalLag on the warning row. Confirm that the lag is a large backlog, rather than a new topic with a short log.
  2. Find the pod that consumes that group. On workloads-summary, check whether the pod is RUNNING and whether it is at its CPU or memory limit.
  3. Open metrics-pipeline-latency-latest for the same topic. A maxLatencyMs warning on that topic means the pipeline is slow, in addition to the consumer.
  4. If the consumer pod is stopped, start it. Do not delete the consumer group to clear the lag. Deleting the group removes its committed position.
  5. If the pod is running and maxLagPct stays greater than 90 for several samples, the consumer is stuck or cannot keep up with the topic. Escalate with the group, the topic, maxLag, and maxLagPct.

iax-volume-used-pct Copied

The iax-volume-used-pct rule applies to the usedPct column on volumes-summary and volumes-totals.

Condition Severity
Usage is greater than 90 percent Warning
Usage is 90 percent or less OK

A Warning on volumes-summary means that volume is more than 90 percent full. A Warning on volumes-totals means overall volume usage is more than 90 percent. Kafka and ClickHouse stop accepting writes when their volume fills. Consumer lag and ingestion delay then increase.

To investigate volume usage:

  1. If the warning is on volumes-summary, the row is one pod and one volume. Read volumeName to identify the volume.
  2. If the warning is only on volumes-totals, open volumes-summary and sort by usedPct. Act on the volume with the highest usage. The totals row is overall usage, not one volume.
  3. Check consumer-groups for a group that is not reading. A consumer that is not reading can leave Kafka retaining data until the volume fills.
  4. Restore the consumer, or expand the volume. If you expand the volume and leave the consumer stopped, the volume can fill again.
  5. If usedPct continues to increase after the consumer is healthy, escalate with the pod, the volume name, usedPct, usedGiB, and sizeGiB.

iax-metrics-pipeline-latency Copied

The iax-metrics-pipeline-latency rule applies to the maxLatencyMs column on metrics-pipeline-latency-latest and metrics-pipeline-latency-overall. It does not set severity on avgLatencyMs.

Condition Severity
Latency is greater than 10,000 milliseconds Warning after two samples
Latency is 10,000 milliseconds or less OK

A Warning on maxLatencyMs means the slowest message through the metrics pipeline took more than 10 seconds. On metrics-pipeline-latency-latest, that value is for one host and one topic in the latest sample. On metrics-pipeline-latency-overall, that value is the slowest result in the API response. Metrics on that path are late by at least that long.

To investigate pipeline latency:

  1. On metrics-pipeline-latency-latest, read hostname and topic. Investigate that path. Use the overall row only to confirm that a path is slow.
  2. Open consumer-groups and find that topic. A maxLagPct warning means the consumer is also behind.
  3. Open workloads-summary for the pod on that host. A CPU limit warning means the pod is throttled. A memory limit warning, or exit code 137, means the container was killed because it exceeded its memory limit.
  4. If the same host and topic stays greater than 10,000 milliseconds for several samples, escalate with the host, the topic, maxLatencyMs, and the consumer lag for that topic.
["Geneos"] ["Geneos > Gateway"] ["User Guide"]

Was this topic helpful?