Dynamic Entities

Overview

Dynamic Entities can be created automatically based on metric datapoints provided by a Collection Agent. This allows you to monitor applications with transient lifetimes.

This type of monitoring is especially useful in an orchestrated environment, such as Kubernetes, where applications and instances can be created and removed dynamically. A mapping template is provided for Kubernetes and OpenShift environments. See, Monitoring strategy in Monitoring strategy.

Differences compared to Managed Entities

Gateways organise data into entities. An entity represents an object being monitored, such as an application or a Kubernetes pod. Gateways format data into the following hierarchy:

Entity
Sampler
Dataview
Row

However, Collection Agent datapoints do not follow this hierarchy and each datapoint can have a different set of dimensions and properties.

You can configure mappings to construct a Geneos hierarchy from datapoints. The resulting Dynamic Entities are then created in the Gateway.

For more information on the Collection Agent, see Collection Agent in Netprobe Overview.

Create Dynamic Entities

To create new Dynamic Entities:

Install and configure a Collection Agent in the environment you want to monitor. For an example of configuring Collection Agent to monitor applications running in Kubernetes, see Installation in a Kubernetes or OpenShift environment.
Create one or more mappings and a mapping type that uses these mappings. For more information see, Dynamic Mapping configuration and Mapping type.
Install and configure a Netprobe to collect data from your Collection Agent and use the mapping type you created. For more information, see Run Collection Agent with Netprobe.

Mapping errors

Dynamic mappings are read by the Netprobe, if a mapping is not compatible this results in an error.

Note the following:

The names of objects, such as entities or rows, created by a dynamic mapping are determined by the value of the options specified in the mapping. Netprobe accepts names containing any ASCII characters except for !"'()[]^{|}`/*&.
All cells in a dataview row must have the same dimensions. If a new datapoint is mapped to a row and it does not have the same dimensions as the other datapoints already mapped to that row, then that datapoint will be rejected by the Netprobe.

For more information about how Netprobes manage incoming datapoints, see Health .

Dynamic Mapping configuration

Dynamic entities > Mapping > Name

A name for the mapping. This name must be unique to avoid ambiguity when using the name in other parts of the Gateway setup. This name is case-sensitive.

Mandatory: Yes

Dynamic entities > Mapping > Options

Specify if this mapping should use a built-in mapping or a custom mapping.

Mandatory: Yes

Dynamic entities > Mapping > Options > builtIn

Select a built-in mapping from the drop down list.

Built-in mappings are included with Gateway and stored in the resources/mappings directory. We aim to provide growing list of built-in mappings for monitoring standard environments.

The following built in mappings are available:

Collection Agent V2 — provides standard mappings for self monitoring data provided by the Collection Agent.

Dynamic entities > Mapping > Options > custom

Custom mappings allow you to create bespoke mappings to fit your data and workflow.

The dimensions and properties of a datapoint are both reference to as labels when creating custom mappings.

Filter

This section allows you to filter which datapoints this mapping applies to based by specifying criteria their labels must meet.

You can apply one or more filters that apply simultaneously.

Field	Description
Label	Specify the name of a dimension or property.
Options	You can apply the following criteria to the specified label to determine if the mapping should be applied: Exists — tests if the datapoint has a label with the specified name. Equals — tests if the value of the named label is equal to a specified value. Match — tests if the value of the named label matches a specified regex pattern.

Field

Description

Label

Specify the name of a dimension or property.

Options

You can apply the following criteria to the specified label to determine if the mapping should be applied:

Exists — tests if the datapoint has a label with the specified name.
Equals — tests if the value of the named label is equal to a specified value.
Match — tests if the value of the named label matches a specified regex pattern.

Geneos items

This section allows you to define how items in the Geneos tree structure are generated from the labels of a datapoint.

When mapping Geneos items, note the following:

Mapping items, excluding attributes, stack so that each label is appended to the item name in the order listed here. If a mapping exists but the label is missing or empty then it will not be added to the item name. For example, if both Kubernetes namespace and Kubernetes pod name are mapped to Managed Entity and the namespace label is listed first then the Managed Entity name will be <namespace>-<pod_name>. If the Kubernetes namespace label does not exist, the Managed Entity name will be <pod_name>.
The dataview name is divided into two parts. The first part is defined by the mappings you provide and the second part is generated from the dimensions of the datapoint. This ensures that each dataview row has a unique set of dimensions.

Field Description

Label Specify the name of a dimension or property.

Options

Field	Description
Label	Specify the name of a dimension or property.
Options	You can map datapoint labels to the following Geneos items: Attribute — mapping a label to an attribute adds a Managed Entity attribute with the label name to the Dynamic Entity created from the datapoint. The value of the Managed Entity attribute will be the value of the label. Multiple labels can be mapped to attributes. Each one will create a new Managed Entity attribute. Dataview — mapping a label to a dataview results in using the value of the label for the dataview name. Multiple labels can be mapped to dataviews, in this case the value of each additional label is appended to the name. Entity — mapping a label to an entity results in using the value of the label for the Dynamic Entity name. Multiple labels can be mapped to an entity, in this case the value of each additional label is appended to the name. Additionally, there are two flags that can be set on each entity mapping. The `Required` option instructs the Netprobe to add an existance filter for the label. The `Use in display name` option can be disabled to hide this label in the display name used by the Active Console. Ignore — mapping a label to this option will ignore the label when creating Geneos items. Row — mapping a label to a row results in using the value of the label as the beginning of the row name. Multiple labels can be mapped to a row, in this case the value of each additional label is appended to the name in the order listed here. Each unmapped label is also appended to the row name in alphabetical order after any explicitly mapped labels. Sampler — mapping a label to a sampler results in using the value of the label for the sampler name. Multiple labels can be mapped to a sampler, in this case the value of each additional label is appended to the name.

You can map datapoint labels to the following Geneos items:

Attribute — mapping a label to an attribute adds a Managed Entity attribute with the label name to the Dynamic Entity created from the datapoint. The value of the Managed Entity attribute will be the value of the label. Multiple labels can be mapped to attributes. Each one will create a new Managed Entity attribute.
Dataview — mapping a label to a dataview results in using the value of the label for the dataview name. Multiple labels can be mapped to dataviews, in this case the value of each additional label is appended to the name.
Entity — mapping a label to an entity results in using the value of the label for the Dynamic Entity name. Multiple labels can be mapped to an entity, in this case the value of each additional label is appended to the name. Additionally, there are two flags that can be set on each entity mapping. The Required option instructs the Netprobe to add an existance filter for the label. The Use in display name option can be disabled to hide this label in the display name used by the Active Console.
Ignore — mapping a label to this option will ignore the label when creating Geneos items.
Row — mapping a label to a row results in using the value of the label as the beginning of the row name. Multiple labels can be mapped to a row, in this case the value of each additional label is appended to the name in the order listed here. Each unmapped label is also appended to the row name in alphabetical order after any explicitly mapped labels.
Sampler — mapping a label to a sampler results in using the value of the label for the sampler name. Multiple labels can be mapped to a sampler, in this case the value of each additional label is appended to the name.

Dynamic entities > Mapping > Local attributes

Define zero or more attributes that are applied to every Dynamic Entity created by the mapping.

Available for both Built In and Custom mappings. The attribute values are strings that can include embedded variables. The variables are fetched from the environment specified in the dynamic mapping type.

Mandatory: No

Mapping type

You can use Mapping types to create logical groupings of mappings. This allows you to easily deploy the same configuration on multiple probes. This is useful when managing large estates.

A Mapping type provides an ordered list of mappings, each datapoint received will be checked against each mapping in the order. Data will be mapped according to the first match made.

Dynamic Entities are created and removed based on Netprobe data. You must specify which mappings each Netprobe should use by applying a Mapping type.

Add a Mapping type to a Netprobe

To add a Mapping type:

Expand the Probes top-level section and select a probe.
Open the Dynamic Entities tab.
In the Dynamic entities section, select a Mapping type from the drop-down list.
Click Validate.
Click Save.

Note: You can also apply Mapping types to probe groups. Probes will inherit the Mapping types specified in their parent group, unless one is specified explicitly on an individual probe. A Mapping type specified on a probe overrides a Mapping type specified on a probe group. For more information, see probes > probeGroup in Probes.

Mapping type configuration

Dynamic entities > Mapping type > Name

A name for the Mapping type. This name must be unique to avoid ambiguity when using the name in other parts of the Gateway setup. This name is case-sensitive.

Mandatory: Yes

Dynamic entities > Mapping type > Mappings

Specifies the mappings used to create dynamic entities. The Gateway will attempt to find matches using each mapping sequentially in the specified order.

Mandatory: No

Dynamic entities > Mapping type > Samplers

Specifies which additional samplers should be created for each Dynamic Entity.

Caution: You can use only the FKM and State Tracker plugins with Dynamic Entities. Attempting to specify a sampler using a different plugin will result in an error.

Mandatory: No

Dynamic entities > Mapping type > Environment

Specifies the environment used by Dynamic Entities created by using this type.

For more information, see User Variables and Environments.

Mandatory: No

Dynamic entities > Mapping type > Entity timeout

Specifies the duration for which no metric or log data, that contributes to a given Dynamic Entity, must be seen before that entity is considered stale and no longer monitored.

If an entity is not monitored, it does not appear in the Active Console and does not affect rules.

If this option is set too low and no data is arriving that contributes to a Dynamic Entity, then FKM information may disappear before any issues are resolved.

Mandatory: No

Unit: seconds

Dynamic entities > Mapping type > Stale data timeout

Specifies the duration for which no data must be seen before a dataview is marked as stale.

This setting applies only to dynamic dataviews that are created by dynamic Managed Entities and populated with data from a Collection Agent. Dataviews populated by other mechanisms, for example the FKM plugin, are unaffected.

Mandatory: No

Unit: seconds

Dynamic Entities commands

Show Dynamic Mappings

You can run this command in the Active Console to display the Dynamic Mapping that produced a target Cell.

The value of a Dynamic Entity dataview cell is determined by the value of a datapoint. The command shows each datapoint label and the mapping used to generate the target cell.

Datapoint information

Column Description

Label Name of the datapoint label.

Value Value of the datapoint label.

Type

Column	Description
Label	Name of the datapoint label.
Value	Value of the datapoint label.
Type	Type of the label. The options are: `Dimension` `Dimension (Built)` `Property` `Property (Built)` Labels created using the label builder are indicated with a `(Built)` tag.

Type of the label.

The options are:

Dimension
Dimension (Built)
Property
Property (Built)

Labels created using the label builder are indicated with a (Built) tag.

The mapping matched to the datapoint and applied is indicated.

Column	Description
Mapping Name	Name of the Dynamic Mapping.
Match Result	Match status. A matched mapping is applied to the incoming datapoint.

Geneos Structure

Column	Description
Geneos structure	Object in the Geneos hierarchy.
Value	Shows the full string of the object name and the mapping that produced each element of the name.

The possible targets of this command are:

Cells in a Dynamic Entity dataview.
Cells in the InvalidMetrics dataview created by the Dynamic Entities Health plugin.

Forget Dynamic Entity

You can run this command to remove a Dynamic Entity from Geneos and all associated dataviews from the Active Console.

Dynamic Entities are automatically removed after a timeout period if no new data has been received. This timeout period is determined by the earliest of the Entity timeout and the stale-metrics-threshold in the Collection Agent StatsD plugin configuration file.

This command allows you to remove an entity you have stopped monitoring but still appears in the Active Console since the timeout has not expired. This is common during testing or when reconfiguring production environments.

Note: If you run this command on an entity that is still being monitored, Geneos will recreate the dynamic entity when the next datapoint is received. As a result the command may appear to have no effect.

The possible targets of this command are:

Dynamic Entities in the State Tree.
Cells in the Entities dataview created by the Dynamic Entities Health plugin.

Appendix: Using regex

Mappings and labels can be created using regular expression (regex) patterns. The following are common patterns that may be useful when creating mappings:

Pattern	Description
`^$`	No matches.
`.+`	Matches any value. Requires that a value is present.
`^A1$`	Matches only `A1` exactly.
`A1`	Matches any value that contains `A1`.