Dynamic Entities

Gateways organise data into entities. An entity represents an object being monitored, such as an application or a Kubernetes pod. Gateways format data into the following hierarchy:

• Entity
• Sampler
• Dataview
• Row

However, Collection Agent datapoints do not follow this hierarchy. A datapoint is composed of, and uniquely defined by, a set of dimensions and a metric name. Where a dimension is a key-value pair.

You can configure mappings to construct a Geneos hierarchy from datapoints. The resulting Dynamic Entities are then created in the Gateway. Geneos includes many built-in mappings but you can also create custom mappings to fit your data and workflow.

For each Dynamic Entity that is created, the set of all dimensions belonging to the Dynamic Entity and any attributes it has must be unique. This set of dimensions is referred to as the Dynamic Entity's identifying dimensions.

Custom mappings

Custom mappings allow you to create bespoke mappings to fit your data and workflow. There are two key parts to a custom mapping:

• Filters — you can apply a filter to determine which datapoints are processed by which mapping.

• Geneos items — when a mapping is applied, the Geneos items fields determine how the labels of each datapoint are used to create Dynamic Entities and other items in the Geneos tree structure.

For detailed configuration information, see Dynamic entities > Mapping > Options > custom.

Mapping errors

Dynamic mappings are read by the Netprobe, if a mapping is not compatible this results in an error.

Note the following:

• The names of objects, such as entities or rows, created by a dynamic mapping are determined by the value of the options specified in the mapping. Netprobe accepts names containing any ASCII characters except for !"'()[]^{|}`/*&.

• All cells in a dataview row must have the same dimensions. If a new datapoint is mapped to a row and it does not have the same dimensions as the other datapoints already mapped to that row, then that datapoint will be rejected by the Netprobe.

• If a mapping would map a new datapoint to an existing Dynamic Entity, but the identifying dimensions generated by the datapoint do not match the existing Dynamic Entity, then the datapoint is discarded and an Entity Dimension clash is recorded.

For more information about how Netprobes manage incoming datapoints, see Dynamic Entities Health and Dynamic Entities commands.

Single row pivoting

Any mapping that does not map a label to a row would normally generate a dataview with only a single row, instead this data is pivoted. As a result, when you view this dataview in Active Console there are two columns, name and value, and each named datapoint is mapped to a row. If the data is forwarded to Gateway Hub or Kafka, then it is pivoted back to the original format.

Dynamic entities > Mapping > Local attributes

Dynamic mapping groups allow a set of mappings to be grouped in a Gateway setup.

These groups can be used to provide a more structured view in the Gateway Setup Editor, but they can also be used to simplify the configuration of Dynamic Entities.

A mapping group allows you to set Filters, Geneos items and Local Attributes that are inherited by child mappings. However, children that use built-in mappings only inherit Local Attributes.

To create a new dynamic mapping group:

1. Right-click on the Dynamic Entities > Mapping section in the Navigation tree and select the New Mapping group option. This creates a new mapping group section.

2. Name the mapping group.

3. New mappings can be created in the group using the New Mapping button.

   • Existing mappings can also be added or removed from the group using drag-and-drop in the Navigation tree.

4. (Optional) Add child mapping groups.

   • Child groups can be added by right-clicking on a mapping group in the Navigation tree and selecting the New Mapping group option.

   • Mapping groups can also be added and removed using drag-and drop in the Navigation tree.

5. Configure the Filters, Geneos items and Local Attributes of the mapping groups you have created.

This section allows you to filter which datapoints this mapping applies to based by specifying criteria their labels must meet.

You can apply one or more filters that apply simultaneously.

This section allows you to define how items in the Geneos tree structure are generated from the labels of a datapoint.

When mapping Geneos items, note the following:

• Mapping items, excluding attributes, stack so that each label is appended to the item name in the order listed here. If a mapping exists but the label is missing or empty then it will not be added to the item name. For example, if both Kubernetes namespace and Kubernetes pod name are mapped to Managed Entity and the namespace label is listed first then the Managed Entity name will be <namespace>-<pod_name>. If the Kubernetes namespace label does not exist, the Managed Entity name will be <pod_name>.

• The dataview name is divided into two parts. The first part is defined by the mappings you provide and the second part is generated from the dimensions of the datapoint. This ensures that each dataview row has a unique set of dimensions.

Define zero or more attributes that are applied to every Dynamic Entity created by the mapping.

Available for both Built In and Custom mappings. The attribute values are strings that can include embedded variables. The variables are fetched from the environment specified in the dynamic mapping type.

Mandatory: No

Add a Mapping type to a Netprobe

To add a Mapping type:

1. Expand the Probes top-level section and select a probe.
2. Open the Dynamic Entities tab.
3. In the Dynamic entities section, select a Mapping type from the drop-down list.
4. Click Validate.
5. Click Save.

Dynamic entities > Mapping type > Name

A name for the Mapping type. This name must be unique to avoid ambiguity when using the name in other parts of the Gateway setup. This name is case-sensitive.

Mandatory: Yes

Dynamic entities > Mapping type > Mappings

Specifies the mappings used to create dynamic entities. The Gateway will attempt to find matches using each mapping sequentially in the specified order.

Mandatory: No

Dynamic entities > Mapping type > Samplers

Specifies if any samplers should be created for Dynamic Entities that are created by this Mapping type. A sampler is only created when log data, that the sampler is configured to monitor, is being received.

Caution: You can use only the FKM and State Tracker plugins with Dynamic Entities. Attempting to specify a sampler using a different plugin will result in an error.

Mandatory: No

Dynamic entities > Mapping type > Environment

Specifies the environment used by Dynamic Entities created by using this type.

For more information, see User Variables and Environments.

Mandatory: No

Dynamic entities > Mapping type > Entity timeout

Specifies the duration for which no metric or log data, that contributes to a given Dynamic Entity, must be seen before that entity is considered stale and no longer monitored.

If an entity is not monitored, it does not appear in the Active Console and does not affect rules.

If this option is set too low and no data is arriving that contributes to a Dynamic Entity, then FKM information may disappear before any issues are resolved.

Mandatory: No

Unit: seconds

Dynamic entities > Mapping type > Stale data timeout

Specifies the duration for which no data must be seen before a dataview is marked as stale.

This setting applies only to dynamic dataviews that are created by dynamic Managed Entities and populated with data from a Collection Agent. Dataviews populated by other mechanisms, for example the FKM plugin, are unaffected.

Mandatory: No

Unit: seconds

Command targets

The possible targets of Dynamic Entities commands are:

• Dynamic Entities in the State Tree.
• Cells in the Entities dataview created by the Dynamic Entities Health plugin.
• Cells in the InvalidStreams dataview created by the Dynamic Entities Health plugin.

Show Dynamic Mappings

You can run this command in the Active Console to display the Dynamic Mapping that produced a target Cell.

The value of a Dynamic Entity dataview cell is determined by the value of a datapoint. The command shows each datapoint label and the mapping used to generate the target cell.

Forget Dynamic Entity

You can run this command to remove a Dynamic Entity from Geneos and all associated dataviews from the Active Console.

Dynamic Entities are automatically removed after a timeout period if no new data has been received. This timeout period is determined by the earliest of the Entity timeout and the stale-metrics-threshold in the Collection Agent StatsD plugin configuration file.

This command allows you to remove an entity you have stopped monitoring but still appears in the Active Console since the timeout has not expired. This is common during testing or when reconfiguring production environments.

Note: If you run this command on an entity that is still being monitored, Geneos will recreate the dynamic entity when the next datapoint is received. As a result the command may appear to have no effect.