Dynamic Entities

Overview

Dynamic Entities can be created automatically based on metric datapoints provided by a Collection Agent. This allows you to monitor applications with transient lifetimes.

This type of monitoring is especially useful in an orchestrated environment, such as Kubernetes, where applications and instances can be created and removed dynamically. A mapping template is provided for Kubernetes and OpenShift environments. See Data collection in an orchestrated environment in Data collection in an orchestrated environment.

Differences compared to Managed Entities

Gateways organise data into entities. An entity represents an object being monitored, such as an application or a Kubernetes pod. Gateways format data into the following hierarchy:

Entity
Sampler
Dataview
Row

However, Collection Agent datapoints do not follow this hierarchy. A datapoint is composed of, and uniquely defined by, a set of dimensions and a metric name. Where a dimension is a key-value pair.

You can configure mappings to construct a Geneos hierarchy from datapoints. The resulting Dynamic Entities are then created in the Gateway. Geneos includes many built-in mappings but you can also create custom mappings to fit your data and workflow.

For each Dynamic Entity that is created, the set of all dimensions belonging to the Dynamic Entity and any attributes it has must be unique. This set of dimensions is referred to as the Dynamic Entity's identifying dimensions.

For more information on the Collection Agent, see Introduction to Collection Agent.

Create Dynamic Entities

To create new Dynamic Entities:

Install and configure a Collection Agent in the environment you want to monitor. For an example of configuring Collection Agent to monitor applications running in Kubernetes, see .
Create one or more mappings and a mapping type that uses these mappings. For more information, see Dynamic mapping configuration and Mapping type.
Install and configure a Netprobe to collect data from your Collection Agent and use the mapping type you created. For more information, see Collection Agent setup.

Custom mappings

Custom mappings allow you to create bespoke mappings to fit your data and workflow. The key parts to a custom mapping are Filters and Geneos items. For more information, see Options > Custom.

Mapping errors

Dynamic mappings are read by the Netprobe, if a mapping is not compatible this results in an error.

Note the following:

The names of objects, such as entities or rows, created by a dynamic mapping are determined by the value of the options specified in the mapping. Netprobe accepts names containing any ASCII characters except for !"'()[]^{|}`/*&.
All cells in a dataview row must have the same dimensions. If a new datapoint is mapped to a row and it does not have the same dimensions as the other datapoints already mapped to that row, then that datapoint will be rejected by the Netprobe.
If a mapping would map a new datapoint to an existing Dynamic Entity, but the identifying dimensions generated by the datapoint do not match the existing Dynamic Entity, then the datapoint is discarded and an Entity Dimension clash is recorded.

For more information about how Netprobes manage incoming datapoints, see Dynamic Entities Health and Dynamic Entities commands.

Single row pivoting

Any mapping that does not map a label to a row would normally generate a dataview with only a single row, instead this data is pivoted. As a result, when you view this dataview in Active Console there are two columns, name and value, and each named datapoint is mapped to a row. If the data is forwarded to Gateway Hub or Kafka, then it is pivoted back to the original format.

Mapping and mapping group

Dynamic mapping configuration

Field	Description
Name	A name for the mapping. This name must be unique to avoid ambiguity when using the name in other parts of the Gateway setup. This name is case-sensitive. Mandatory: Yes
Options	Specify if this mapping should use a built-in mapping or a custom mapping. Mandatory: Yes
Options > Built in	Select a built-in mapping from the drop-down list. Built-in mappings are included with Gateway and stored in the `resources/mappings` directory. The following are available from a growing list of built in mappings provided for monitoring standard environments: Collection Agent V2 — provides standard mappings for self-monitoring data supplied by the Collection Agent version 2.0.x. Collection Agent V3 — provides standard mappings for self-monitoring data supplied by the Collection Agent version 3.0.x. Azure Monitor V2 — provides mappings to display the Azure Monitor plugin metrics in Geneos. This is only available beginning Geneos 5.8.x. Azure Monitor V3 — provides mappings to display the Azure Monitor plugin metrics Geneos configured with a Collection Agent version 3.0.x. This is only available beginning Geneos 6.0.x. Prometheus V2 — provides mappings to display the metrics from the Prometheus server in Geneos. This is only available beginning Geneos 5.10.x. Prometheus V3 — provides mappings to display the metrics from the Prometheus server in Geneos configured with a Collection Agent version 3.0.x. This is only available beginning Geneos 6.0.x.
Options > Custom	Allows you to create bespoke mappings to fit your data and workflow. There are two key parts to a custom mapping: Filters — you can apply a filter to determine which datapoints are processed by which mapping. For detailed configuration information, see Filter. Geneos items — when a mapping is applied, the Geneos items fields determine how the labels of each datapoint are used to create Dynamic Entities and other items in the Geneos tree structure. For detailed configuration information, see Geneos items. Note: Dimensions and properties of a datapoint are both referred to as labels when creating custom mappings.
Local attributes	Define zero or more attributes that are applied to every Dynamic Entity created by the mapping. Available for both Built In and Custom mappings. The attribute values are strings that can include embedded variables. The variables are fetched from the environment specified in the dynamic mapping type. Mandatory: No

Dynamic mapping group configuration

Dynamic mapping groups allow a set of mappings to be grouped in a Gateway setup.

These groups can be used to provide a more structured view in the Gateway Setup Editor, but they can also be used to simplify the configuration of Dynamic Entities.

A mapping group allows you to set Filters, Geneos items and Local Attributes that are inherited by child mappings. However, children that use built-in mappings only inherit Local Attributes.

To create a new dynamic mapping group:

Right-click on the Dynamic Entities > Mapping section in the Navigation tree and select the New Mapping group option. This creates a new mapping group section.
Name the mapping group.

Note: This name must be unique to avoid ambiguity when using the name in other parts of the Gateway setup. This name is case-sensitive.
Configure the Filters, Geneos items and Local Attributes of the mapping groups you have created. For detailed configuration information, see Mapping options.

New mappings can be created in the group using the New Mapping button. Existing mappings can also be added or removed from the group using drag-and-drop in the Navigation tree.

Child groups can be added by right-clicking on a mapping group in the Navigation tree and selecting the New Mapping group option.

Mapping groups can also be added and removed using drag-and drop in the Navigation tree.

Mapping options

Filter

This section allows you to filter which datapoints this mapping applies to based by specifying criteria their labels must meet.

You can apply one or more filters that apply simultaneously.

Field	Description
Label	Specify the name of a dimension or property.
Options	You can apply the following criteria to the specified label to determine if the mapping should be applied: Exists — tests if the datapoint has a label with the specified name. Equals — tests if the value of the named label is equal to a specified value. Match — tests if the value of the named label matches a specified regex pattern.

Field

Description

Label

Specify the name of a dimension or property.

Options

You can apply the following criteria to the specified label to determine if the mapping should be applied:

Exists — tests if the datapoint has a label with the specified name.
Equals — tests if the value of the named label is equal to a specified value.
Match — tests if the value of the named label matches a specified regex pattern.

Note: If a mapping group and a mapping or a nested mapping group contain a filter for the same label, both will be applied.

Geneos items

This section allows you to define how items in the Geneos tree structure are generated from the labels of a datapoint.

When mapping Geneos items, note the following:

Mapping items, excluding attributes, stack so that each label is appended to the item name in the order listed here. If a mapping exists but the label is missing or empty then it will not be added to the item name. For example, if both Kubernetes namespace and Kubernetes pod name are mapped to Managed Entity and the namespace label is listed first then the Managed Entity name will be <namespace>-<pod_name>. If the Kubernetes namespace label does not exist, the Managed Entity name will be <pod_name>.
The dataview name is divided into two parts. The first part is defined by the mappings you provide and the second part is generated from the dimensions of the datapoint. This ensures that each dataview row has a unique set of dimensions.

Field Description

Label Specify the name of a dimension or property.

Options

Field	Description
Label	Specify the name of a dimension or property.
Options	You can map datapoint labels to the following Geneos items: Attribute — mapping a label to an attribute adds a Managed Entity attribute with the label name to the Dynamic Entity created from the datapoint. The value of the Managed Entity attribute will be the value of the label. Multiple labels can be mapped to attributes. Each one will create a new Managed Entity attribute. Dataview — mapping a label to a dataview results in using the value of the label for the dataview name. Multiple labels can be mapped to dataviews, in this case the value of each additional label is appended to the name. Entity — mapping a label to an entity results in using the value of the label for the Dynamic Entity name. Multiple labels can be mapped to an entity, in this case the value of each additional label is appended to the name. Additionally, there are two flags that can be set on each entity mapping. The `Required` option instructs the Netprobe to add an existance filter for the label. The `Use in display name` option can be disabled to hide this label in the display name used by the Active Console. Ignore — mapping a label to this option will ignore the label when creating Geneos items. Row — mapping a label to a row results in using the value of the label as the beginning of the row name. Multiple labels can be mapped to a row, in this case the value of each additional label is appended to the name in the order listed here. Each unmapped label is also appended to the row name in alphabetical order after any explicitly mapped labels. If no labels are mapped to the row, only a single row is generated which is pivoted when viewed in Active Console. For more information, see Single row pivoting in Dynamic Entities. Sampler — mapping a label to a sampler results in using the value of the label for the sampler name. Multiple labels can be mapped to a sampler, in this case the value of each additional label is appended to the name.

You can map datapoint labels to the following Geneos items:

Attribute — mapping a label to an attribute adds a Managed Entity attribute with the label name to the Dynamic Entity created from the datapoint. The value of the Managed Entity attribute will be the value of the label. Multiple labels can be mapped to attributes. Each one will create a new Managed Entity attribute.
Dataview — mapping a label to a dataview results in using the value of the label for the dataview name. Multiple labels can be mapped to dataviews, in this case the value of each additional label is appended to the name.
Entity — mapping a label to an entity results in using the value of the label for the Dynamic Entity name. Multiple labels can be mapped to an entity, in this case the value of each additional label is appended to the name. Additionally, there are two flags that can be set on each entity mapping. The Required option instructs the Netprobe to add an existance filter for the label. The Use in display name option can be disabled to hide this label in the display name used by the Active Console.
Ignore — mapping a label to this option will ignore the label when creating Geneos items.
Row — mapping a label to a row results in using the value of the label as the beginning of the row name. Multiple labels can be mapped to a row, in this case the value of each additional label is appended to the name in the order listed here. Each unmapped label is also appended to the row name in alphabetical order after any explicitly mapped labels. If no labels are mapped to the row, only a single row is generated which is pivoted when viewed in Active Console. For more information, see Single row pivoting in Dynamic Entities.
Sampler — mapping a label to a sampler results in using the value of the label for the sampler name. Multiple labels can be mapped to a sampler, in this case the value of each additional label is appended to the name.

Note: Mapping items defined at the mapping group level will stack before items defined in nested mapping groups or in mappings and any duplicate items are not removed.

Local attributes

Define zero or more attributes that are applied to every Dynamic Entity created by the mapping.

Available for both Built In and Custom mappings. The attribute values are strings that can include embedded variables. The variables are fetched from the environment specified in the dynamic mapping type.

Mandatory: No

Note: Local attributes defined in nested groups or in mappings will override local attributes defined in their parent mapping groups.

Mapping type

Netprobes process Collection Agent data using a specified Mapping type and create or remove Dynamic Entities accordingly. Mapping types allow you to easily deploy the same configuration on multiple probes. This is useful when managing large estates.

A Mapping type provides an ordered list of mappings, and every datapoint received is checked against each mapping in the order. Data is then mapped according to the first match made.

Add a Mapping type to a Netprobe

To add a Mapping type:

Expand the Probes top-level section and select a probe.
Open the Dynamic Entities tab.
In the Dynamic entities section, select a Mapping type from the drop-down list.

Note: You can also apply Mapping types to probe groups. Probes will inherit the Mapping types specified in their parent group, unless one is specified explicitly on an individual probe. A Mapping type specified on a probe overrides a Mapping type specified on a probe group. For more information, see probes > probeGroup in Probes.

Mapping type configuration

Field	Description
Name	A name for the mapping type. This name must be unique to avoid ambiguity when using the name in other parts of the Gateway setup. This name is case-sensitive. Mandatory: Yes
Collectors	Allows users to add collectors in the mapping type. For more configuration information, see Collectors. Mandatory: No
Mappings	Specifies the mappings used to create dynamic entities. The Gateway will attempt to find matches using each mapping sequentially in the specified order. Mandatory: No Note: You can also specify mappings in Collectors. If mappings are defined in both the mapping type and the collector, then the Netprobe will apply the mappings defined in the mapping type first then the mappings defined in the collectors.
Samplers	Specifies if any samplers should be created for Dynamic Entities that are created by this Mapping type. A sampler is only created when log data, that the sampler is configured to monitor, is being received. Caution: You can use only the FKM and State Tracker plugins with Dynamic Entities. Attempting to specify a sampler using a different plugin will result in an error. Mandatory: No
Environment	Specifies the environment used by Dynamic Entities created by using this type. For more information, see User Variables and Environments. Mandatory: No

Collection Agent parameters

Beginning Geneos 6.0.0, the Collection Agent parameters are now configured under the Dynamic Entities section in the GSE. You may now also configure the parameters from the Collection Agent configuration file in the Gateway Setup Editor if you will use a managed YAML for your Collection Agent.

Collection Agent parameters configuration

Field	Description
Name	Name of the Collection Agent parameter. Mandatory: Yes
Collection Agent	Defines how the Collection Agent will run. The possible values are the following: Unmanaged — Collection Agent will run as a stand-alone process. You should choose this if you wish to run your Collection Agent and Netprobe on different hosts. Managed — the Netprobe will run an instance of the Collection Agent as a Java process. For more configuration information, see Managed Collection Agent .
Reporter port	Specifies the port the Netprobe should listen on to communicate with the Collection Agent using the TcpReporter. Default: `9137` Mandatory: No
Entity timeout	Specifies the duration for which no metric or log data, that contributes to a given Dynamic Entity, must be seen before that entity is considered stale and no longer monitored. If an entity is not monitored, it does not appear in the Active Console and does not affect rules. If this option is set too low and no data is arriving that contributes to a Dynamic Entity, then FKM information may disappear before any issues are resolved. Default: `3600` Unit: seconds Mandatory: No
Stale data timeout	Specifies the duration for which no data must be seen before a dataview is marked as stale. This setting applies only to dynamic dataviews that are created by dynamic Managed Entities and populated with data from a Collection Agent. Dataviews populated by other mechanisms, for example the FKM plugin, are unaffected. Default: `300` Unit: seconds Mandatory: No
Stream buffer size	Sets the maximum number of event messages that the Netprobe holds in memory at a time. The Netprobe holds these messages until they are consumed by another sampler. Default: `1000` Mandatory: No

Managed Collection Agent

Field	Description
YAML	Specifies the YAML file defining the Collection Agent configuration. The possible values are the following: Bundled YAML — the Collection Agent will use the `collection-agent.yml` in the `collection_agent` folder of your Netprobe directory. Custom YAML — the Collection Agent will use the specified YAML file that can have any name. The file can be in any folder but if you are using a relative path, then it should be relative to the working directory. Managed YAML — a `collection-agent.yml` file will be created in your working directory. Choosing this option allows you to define the workflow settings in the Workflow text box and to enable Collection Agent self-monitoring Metrics by ticking the Self monitoring checkbox. These additional configurations and the collectors defined on the mapping type associated to the managing Netprobe will be included in the created `collection-agent.yml` file. Note: If Self monitoring is enabled, the Collection Agent V3 built-in mappings are automatically added to the mappings used by the Netprobe. You may also reference variables in the Workflow field by using the `$(variable_name)` or by choosing a variable from the var drop-down list.
Health port	Specifies the Collection Agent port that the Netprobe communicates with for health checks. For more information, see Health checks in Collection Agent setup. Default: `9136` Mandatory: No
Jvm args	Specifies additional JVM options for when the Collection Agent starts. By default, the Collection Agent runs with the following JVM options: `-Xms512M` — allocates the initial heap size. `-Xmx512M` — allocates the maximum heap size. You can also add additional jars that the plugin needs to the classpath by specifying `-cp <pathToJarFile1>:<pathToJarFile2>` in Jvm args. If a path contains a space, enclose the whole list with quotes. For example, `"<path To Jar File 1>:<path To Jar File 2>"`. For Windows, the separator used is a semicolon, `;`. For example, `"<path To Jar File 1>;<path To Jar File 2>"`. Mandatory: No
Log directory	Specifies the path where the Collection Agent logs are stored. Default: Working directory Mandatory: No

Collectors

Beginning Geneos 6.0.0, you may add or edit a collector for your managed Collection Agent without the need to manually stop and restart your Collection Agent instance.

Note: Collectors can only be added if the Collection Agent is managed and the YAML setting is managedYAML.

When adding or editing a collector, the Netprobe will automatically restart the Collection Agent instance.

Basic configuration

Field	Description
Name	Name of the collector. Mandatory: Yes
Plugin	Specifies the plugin and the collector. The following lists all the possible collectors that can be defined for each plugin: AWS — `ApiDestinationCollector`, `AwsBillingCollector`, `AwsCollector`, `AwsCustomNamespaceCollector`, and `AwsSdkUsageMetricsCollector`. For more information on these services, see Monitored services, logs, and events in AWS. Azure — `AzureEventHubCollector` and `AzureMonitorCollector`. For more information on these services, see Monitored services in Azure. Note: If the Azure collector is configured, the built-in mappings for Azure Monitor V3 are automatically added to the mappings used by the Netprobe. Custom Fluentd Forward — `FluentdForwardCollector` Kubernetes — `KubernetesLogCollector` and `KubernetesMetricsCollector`. For more information, see Collected Metrics in Kubernetes. Open-Telemetry — `OpenTelemetryCollector` Prometheus — `AlertManagerWebhookCollector`, `PrometheusRemoteWriteCollector`, and `PrometheusTargetScrapeCollector` Note: If the Prometheus plugin is configured, the built-in mappings for Prometheus V3 are automatically added to the mappings used by the Netprobe. StatsD — `StatsdServer` Mandatory: Yes
Plugin > Yaml	Allows you to set up your chosen collector in the GSE. This field contains all the mandatory and optional YAML fields that you can configure for a collector. You may also reference variables by using $(variable_name) or by choosing a variable in the var dropdown list. Note: The values of password variables with stdAESPassword or externalPassword types will be passed by the Netprobe as environment variables to the Collection Agent. In effect, the format of the variable is `$(env:GENEOS_variable_name)` in the `collection-agent.yml`. Non-password variables will be replaced with their values in the `collection-agent.yml`. Mandatory: Yes

Advanced configuration

Field	Description
Mappings	Allows you to define which mappings will the collectors use. Note: You can also specify mappings in Collectors. If mappings are defined in both the mapping type and the collector, then the Netprobe will apply the changes in the following order: Mappings defined in the mapping type Mappings specified by the user in the collectors Any built-in mappings added automatically Mandatory: No

Field

Description

Mappings

Allows you to define which mappings will the collectors use.

Note: You can also specify mappings in Collectors. If mappings are defined in both the mapping type and the collector, then the Netprobe will apply the changes in the following order:

Mappings defined in the mapping type
Mappings specified by the user in the collectors
Any built-in mappings added automatically

Mandatory: No

Dynamic Entities commands

Show Dynamic Mappings

You can run this command in the Active Console to display the Dynamic Mapping that produced a target cell.

The value of a Dynamic Entity dataview cell is determined by the value of a datapoint. The command shows each datapoint label and the mapping used to generate the target cell.

The possible targets of this command are:

Cells in a dynamically generated dataview that is not from an FKM or a State Tracker sampler.
Cells in the InvalidStreams dataview created by the Dynamic Entities Health plugin.

Datapoint information

Column	Description
Label	Name of the datapoint label.
Value	Value of the datapoint label. This column will also show a `Clash` warning if a dimension or attribute clash occurs.
Expected	Expected value of the datapoint label, based on the existing dimensions or attributes of an entity or the existing dimensions of a dataview row. This column is only shown when a `Row Dimension clash`, `Entity Dimension clash`, or `Attribute Clash` has occurred.
Type	Type of the label. The options are: `Dimension` `Dimension (Built)` `Property` `Property (Built)` Labels created using the label builder are indicated with a `(Built)` tag.

The mapping matched to the datapoint and applied is indicated.

Column	Description
Mapping Name	Name of the Dynamic Mapping.
Match Result	Match status. A matched mapping is applied to the incoming datapoint.

Geneos Structure

Column	Description
Geneos structure	Object in the Geneos hierarchy.
Value	Shows the full string of the object name and the mapping that produced each element of the name.

Forget Dynamic Entity

You can run this command to remove a Dynamic Entity from Geneos and all associated dataviews from the Active Console.

Dynamic Entities are automatically removed after a timeout period if no new data has been received. This timeout period is determined by the earliest of the Entity timeout and the stale-metrics-threshold in the Collection Agent StatsD plugin configuration file.

This command allows you to remove an entity you have stopped monitoring but still appears in the Active Console since the timeout has not expired. This is common during testing or when reconfiguring production environments.

The possible targets of this command are:

Dynamic Entities in the State Tree.
Cells in the Entities dataview created by the Dynamic Entities Health plugin.

Note: If you run this command on an entity that is still being monitored, Geneos will recreate the dynamic entity when the next datapoint is received. As a result the command may appear to have no effect.

View Log

The View Log command is also available when you right-click the following in the State Tree:

Netprobes with configured Dynamic Entities
Dynamic Managed Entities
Dynamic Entity Health samplers

This command shows the logs of the corresponding Collection Agent of the selected cell. When you right-click on any cell except for the probeName, and select theCollection Agent > View Log... option, a dialog box will open to specify options on how you want to view the logs. The following parameters can be set:

Initial Download size(kb) — sets the initial download size of the log file. This defaults to 2 kb.
Delivery Type — allows you to choose between Snapshot and Continuous delivery type. The snapshot delivery type displays the contents of the log file at the time it was generated, while continuous delivery type displays logs continuously in real-time.
Filter String — allows you to specify strings that will filter the Collection Agent log file.

Sample Output (Snapshot delivery type)

View Configuration

The View Configuration command is also available when you right-click the following in the State Tree:

Netprobes with configured Dynamic Entities
Dynamic Managed Entities
Dynamic Entity Health samplers

When you right-click on any cell except for the probeName, and select theCollection Agent > View Configuration option, a new window displaying the configuration of the corresponding Collection Agent will open.

Sample Output

View Error

When you right-click on a cell in the status column, and select theCollection Agent > View Error option, a new window displaying more inforopen.

Sample Output

Appendix: Using regex

Mapping filters can be created using regular expression (regex) patterns. The following are common patterns that may be useful when creating mappings:

Pattern	Description
`^$`	No matches.
`.+`	Matches any value. Requires that a value is present.
`^A1$`	Matches only `A1` exactly.
`A1`	Matches any value that contains `A1`.