Monitor ITRS Analytics ingestion pipeline

Overview Copied

You can monitor the ITRS Analytics ingestion pipeline and display the dataviews in Active Console. Use the template (templates/iax-monitoring.xml) included in the Gateway package.

The template uses the REST Extractor plugin. Each sampler calls the ITRS Analytics ingestion monitoring API and converts the JSON response into dataviews. The template provides dataviews for these areas:

• Consumer groups
• Ingestion rates
• Tracer latency
• Storage volumes
• Workloads

The same file includes rules that set severity on workload status, consumer-group lag, volume usage, and tracer latency.

Prerequisites Copied

Before you include the template, confirm the following:

• The Gateway package is version 7.12.1 or later, so templates/iax-monitoring.xml is present.
• The Netprobe that runs the samplers meets the REST Extractor prerequisites, including the Java requirements.
• The Netprobe can open an HTTPS connection to the ITRS Analytics host. The template calls https://<IAX_INSTANCE>/ingestion/api/v1/.

How to monitor the ITRS Analytics ingestion pipeline Copied

To monitor the ITRS Analytics ingestion pipeline, follow these steps:

1. Create an ITRS Analytics API token with the user role.
2. Include templates/iax-monitoring.xml in the Gateway setup.
3. Set IAX_INSTANCE and IAX_API_TOKEN.
4. Add the IAX Ingestion Monitoring samplers to a managed entity.

Create an ITRS Analytics API token Copied

The samplers call the ingestion monitoring API with a bearer token. Create an API token with the user role. Enter the token as the value of IAX_API_TOKEN when you set up the variables.

1. In ITRS Analytics, go to Admin > IAM > API Tokens.
2. Select Add API Token.
3. Create a token with the user role.
4. Copy the API token.

For more information, see IAM app API tokens.

Include the template Copied

To include iax-monitoring.xml in the Gateway setup, follow these steps:

1. Open Gateway Setup Editor.
2. In the Navigation panel, right-click Includes to create a new file.
3. In the Location field, enter the path to iax-monitoring.xml in the Gateway package templates directory.
4. In the Priority field, enter any value except 1. If you enter 1, the Gateway Setup Editor returns an error.
5. Expand the file location in the Includes section.
6. Select Click to load.
7. Select Yes to load the include file.

For more information, see Include files.

Set up the variables Copied

The iax-monitoring.xml template does not define variables. Create the following variables where they are in scope for the samplers, such as Operating environment > Var or an environment applied to the managed entity. For more information, see User Variables and Environments.

Expand table

Add the samplers Copied

1. Select Managed entities in the Navigation panel.
2. On the managed entity that monitors ITRS Analytics, add the samplers from the IAX Ingestion Monitoring sampler group.

The probe for that managed entity must be able to reach the ITRS Analytics host.

The included rules target the sampler names consumer-groups, ingestion-rates, tracer-latency, volumes, and workloads. Keep those names so the rules continue to apply.

Validate and save the setup Copied

1. Select Validate current document to validate the configuration.
2. Select Save current document to apply the changes.

After you save, the dataviews appear in Active Console under the managed entity. For the columns, see Metrics and dataviews.

Rules Copied

The include file adds the IAX Ingestion Monitoring rule group. These rules set cell severity:

Expand table

Metrics and dataviews Copied

The five IAX Ingestion Monitoring samplers publish these dataviews. Each sampler runs every 60 seconds. These columns are produced by templates/iax-monitoring.xml. The template reads the Ingestion self-monitoring API. For the current API, see Ingestion API. The following sections describe the columns in each dataview.

Expand table

Consumer groups Copied

The consumer-groups sampler publishes the lag for each consumer group and topic. Both dataviews use these columns. Rows are sorted by totalLag, with the highest lag first.

consumer-groups lists groups whose summary does not contain unknown partitions, so unknowns is false. consumer-groups-unknown lists groups whose summary contains unknown partitions, so unknowns is true.

Expand table

Ingestion rates Copied

The ingestion-rates sampler publishes one row for each data point group. Both dataviews use these columns.

received-rates shows the count of data points received. ingested-rates shows the count of data points ingested.

Expand table

Tracer latency Copied

The tracer-latency sampler publishes pipeline latency.

tracer-latency-latest shows the latest sample for each host and topic. It uses every column in the following table. Rows are sorted by maxLatencyMs, with the highest latency first.

tracer-latency-overall contains one row for all hosts and topics. That row includes key, maxLatencyMs, and avgLatencyMs. The value of key is all.

Expand table

Storage volumes Copied

The volumes sampler publishes storage size and usage.

volumes-summary shows each storage volume and uses every column in the following table. Rows are sorted by usedPct, with the highest percentage in use first.

volumes-totals contains one row for all volumes. That row includes key, usedPct, usedGiB, and sizeGiB. The value of key is all.

Expand table

Workloads Copied

The workloads sampler publishes container state and resource usage. Both dataviews use these columns and omit containers whose state is terminated or evicted. Rows are sorted by restarts, with the highest restart count first.

workloads-summary also omits containers whose names start with linkerd. workloads-restarts lists containers with a restart count greater than zero.

Expand table