About the Ingestion app

Note

The Ingestion app version 2.12.0 requires ITRS Analytics Platform version 2.18.6 or later.

The Ingestion app allows you to explicitly review and manage data point ingestion into ITRS Analytics.

All data points require a configuration defining the order of their dimensions. Dimensions are the ordered names and values within a data point that uniquely identify the data point’s entity.

ITRS Analytics ships with known dimensions already configured – all Geneos and internal data (Kubernetes) are configured out-of-the-box.

Monitoring Copied

The Monitoring section of the Ingestion app gives you operational visibility into data ingestion and the health of your ITRS Analytics installation. Use it to track data point flow through the ingestion pipeline, monitor cluster storage and running workloads, and investigate Kafka performance.

Ingestion Throughput Copied

This section displays the totals and time-series charts for data points processed by ITRS Analytics:

• Ingested data points — all processed and stored data points.
• Received data points — all data points sent to the system.
• Dropped data points — data points that are dropped due to validation or missing containment rules.
• Filtered data points — data points filtered in one of the active filters.

In addition to the primary totals and time-series charts, secondary ingestion throughput charts are available when you click on the time-series charts. Use these secondary charts when you need more detail than the headline metrics. For example, you can use these to see which data point types account for most of your traffic, or whether issues are coming from internal or external sources.

Each secondary chart group corresponds to a stage or outcome in the ingestion pipeline. Within a group, charts typically show throughput split by data point type such as entities, metrics, events, logs, and traces. You can also switch between Total, Internal, and External views.

Data Volumes Copied

This section shows the size and usage of the volumes in a full ITRS Analytics cluster, including apps. Volumes exceeding 75% capacity are flagged with a warning, and those at or above 90% capacity are flagged as critical.

Workloads Copied

This section displays a table for all the containers running in the ITRS Analytics installation, including apps.

If a container has restarted, its row is highlighted in light red, and the Last Restart column shows when the most recent restart occurred. Hover over that time to see a tooltip with the pod or container’s last restart status, restart reason, exit code, and the exact restart date and time.

When you click on a container row, it shows a drawer displaying the CPU and memory time-series charts, container details, and a Last Status table if there are restarts.

You can also filter containers by enabling the Hide LinkerD or Hide Non-Running options.

Kafka Metrics Copied

Tracer Latency Copied

This tab displays a chart showing the maximum latency values per minute over the past hour for all unique consumer group client and topic combinations.

Tracers are empty messages periodically sent through the raw metrics ingestion pipeline of the ITRS Analytics platform. The summary view shows the maximum value across all partitions, while the detailed view shows values for all partitions of a single consumer group and topic combination. Partitions are ordered by highest latency and a maximum of 24 partitions can be displayed at a time.

Consumer Group Lag Copied

This tab displays a table showing the consumer lag for all unique consumer group client and topic combinations. It shows the number of partitions for each topic, along with the minimum, maximum, and total lag, and the lag ratio.

Note

Lag ratio is calculated as total_lag / log_end_offset.

You can filter data by enabling the Hide Unknown Consumers or Hide Zero Lag options.

Click a row in the table to open a detailed view for that consumer group and topic combination. This view has two tabs:

• Current — shows a snapshot of the lag for each partition in the selected consumer group and topic. Use this view to see which partitions have the highest lag at the moment.

• Historic — shows how lag has changed over the past hour for the selected combination. The chart displays time series data so you can track lag trends. You can filter the chart to show only the partitions you want to include.

Configuration Copied

The Configuration section of the Ingestion app lets you manage how data is ingested, filtered, stored, and retained in ITRS Analytics. Use it to configure dimension ordering for incoming data points, apply inclusion and exclusion rules, set retention durations, and control when inactive entities are removed.

Entity Configuration Copied

Dropped data points Copied

Note

Only users with admin privileges can configure dimensions.

Any unknown dimension is listed in the Dropped Entities tab. These data points are not processed by the Platform.

If you wish to see this data, you must configure the order of dimensions.

To configure the order of unknown dimension keys, click the Order icon in the right-most column. This opens the Order dimension keys dialog, where you can drag the keys into the desired order.

Note

To see a full JSON example of the data point, click Sample.

The dimensions are ordered by hierarchy groups so that you can configure these at the same time. For example, if the following dimensions are listed:

[a][b]
[a][b][c]
[a][b][c][d]

Then configuring the order of [a][b] will also allow you to configure the other dimensions with common keys. It is important that hierarchies are configured following a consistent key order.

Once you have configured one or more dimension keys, the checkbox for the item will be selected in the list. To save the changes, click Configure.

After saving your changes, the configured dimensions are moved from the Dropped Entities tab to the Known Entities tab.

Known entities Copied

There are two sections for resolved dimensions: Resolved and Pre-configured (Read only).

Resolved Copied

These are the dimensions that previously appeared in the Dropped Entities tab. If the items have pending status, it means that they have been configured but no new data has arrived matching these dimensions. Non-pending items are fully configured and working as expected.

To remove resolved dimensions, select the checkbox for the dimension keys, and then click the Remove button.

Note

Removing dimensions from the data points does not automatically populate them in the Dropped Entities tab. They will only appear if new data is published that uses those exact dimensions, which may be unlikely to happen.

Pre-configured (Read only) Copied

The Pre-configured (Read only) tab shows the pre-configured dimensions for known dimension keys. This information is provided for reference only.

Filtering Copied

Dimension Filters Copied

Filters allow you to exclude certain data points based on some matching rules.

1. From the Filtering tab, click the Add button to create a new filter.
2. Input an expression or select the filters.
3. Once the value is set, select one or more options in the Datapoint drop-down list to apply the filter to.
4. Click Save.

To modify or delete a filter, select the checkbox for the filter, and then click Edit or Remove.

Geneos Filters Copied

You can use multiple Geneos filters to manage data points from Geneos that are ingested into ITRS Analytics.

To add your desired Geneos filters, go to the Geneos Filter tab and click Add Filter. You can also reorder, modify, and delete filters in this tab.

All filters added become part of your Geneos Filter set. You can enable or disable single Geneos filters.

• If the filter set is enabled, at least one filter must explicitly include a data point for that data point to be retained. If no filter includes it, it will be excluded.
• If the filter set is disabled, no data points will be excluded.

Note

The filter set is designed to prioritize exclusion.

Filters are applied in the following order to determine which data points are retained or excluded:

• Each filter is analyzed individually, considering both its “include” and “exclude” selectors. Within a single filter, the “exclude” selector always takes precedence over the “include” selector. The order of these selectors does not matter.
• If a filter neither includes nor excludes a data point, it has no impact on the overall results.
• If a filter explicitly includes or excludes a data point, it overrides any previous filter’s results for that data point.
• If no filter specifically selects a data point, the data point is excluded.

Data Retention Policies Copied

This section displays data retention policies, including metric policies and non-metric policies, that are being used in the platform.

You can configure the retention duration for both metric and non-metric policies by clicking a row and setting a value between one hour and 10 years.

Note

When modifying the retention period, a confirmation checkbox is required to acknowledge that reducing the retention period will result in the loss of historical data.

Eviction Rules Copied

Entity eviction rules define when inactive entities and their data are automatically removed. In the Eviction Rules section, you can view all enabled rules, their conditions, types, and details. These are configurable by an admin user in the Ingestion app by clicking Eviction Rules > Add Rule.

There are two types of rules:

1. Fixed Expiration — configure an expiration time for the entity, where inactive entities are evicted after a fixed period of time. This indicates that the entity is retained if the last update is within this duration. For example:

   Entity A is configured with a 2-hour fixed expiration. It is removed two hours after its last update. If the last update was at 13:00, it will be evicted at 15:00; however, if it updates at 14:45, the timer resets and it will be evicted at 16:45.

2. Lifespan-based Expiration — entities are evicted once they have been inactive for a period of time based on how long they were active for.

   • Min Expiration — the minimum time an entity must remain after it becomes inactive before it can be evicted.
   • TTL percent — the percentage of the active lifespan of the entity to be added after inactivity. It determines how long to keep an entity after it goes inactive, based on how long it was active.
   • Max Expiration — optional, but must be greater than the minimum expiration.

   For example, Entity B is configured with the following settings:

   If Entity B was active for four hours and the last update was at 13:00, the computed TTL window is two hours (four hours × 50%). Since two hours falls within the minimum and maximum expiration times, the entity will be evicted at 15:00.

Note

The default eviction rules created during installation can also be restored. Restoring defaults removes all rules created after installation.