Geneos ["Geneos"]
["Geneos > Netprobe"]["User Guide"]

Prometheus and Alert Manager


The Prometheus plugin is a Collection Agent plugin that gathers metrics from the Prometheus server and alerts from Alert Manager. This plugin collects these data using the Remote Write capability of the Prometheus server, which provides real-time metrics and events.

The plugin can also scrape metrics from applications instrumented to expose Prometheus metrics in Prometheus Exposition Format (PEF). The PEF exposed by scrape targets is not as rich as the remote write data model. However, translation from PEF to data points should closely, if not exactly, match the remote write mappings.


Geneos environment

The latest version of the Prometheus plugin requires Gateway and Netprobe version 5.10.x or higher. The same version must be used for the GSE schema.

The Prometheus plugin binaries are packaged with Netprobe 5.10.x and higher and are stored in the collection_agent folder.

Alternatively, you can download separate binaries for the Prometheus plugin from the ITRS Downloads.

Prometheus environment

The Prometheus plugin requires a running Prometheus server. In addition, you must enable and set up the <remote_write> configuration in your Prometheus server to retrieve real-time metrics. To enable this in a Docker or a Kubernetes environment, add the following in your Prometheus server configuration file:

# prometheus config
  - url: "https://netprobe_location:7654"
    ca_file: /path/to/ca
  - source_labels: [__name__]
    regex: 'up'
    action: keep

Notes: The above configuration:

  • is only valid for certain ways of installing Prometheus. If you are using a Prometheus Operator, then you need to use a different syntax in the configuration file.
  • only sends the up metrics.

The Remote Write capability of the Prometheus server is configured to send only the up metrics to the Prometheus plugin. The Prometheus server can transmit an unbounded number of metrics, and having too many metrics may overload the Netprobe.

You should filter the list of metrics from the Prometheus server to be ingested by the Netprobe. The Netprobe was tested to handle about 1000 metrics per second. Automatic load balancing, which allows ingestion of all metrics, will be supported in a future version.

Alert Manager environment

The Alert Manager is a service that manages the alerts sent from the Prometheus server. The <alertmanager_config> setting enables you to publish these handled alerts from the Alert Manager. To enable this setting, add the following in your Prometheus server configuration file: 

# alert manager config
  - name: 'web.hook'
  - url: 'https://netprobe:5001/'
    ca_file: /path/to/ca

Events from the Alert Manager are published as a stream, which can be monitored using File Keyword Monitor plugin. For more information on how to monitor streams, see files > file > source > stream in File Keywoard Monitor configuration.

Configure Gateway to receive Prometheus data

The Prometheus plugin supports Collection Agent publication into Geneos using dynamic managed entities. To set up the Prometheus plugin in Geneos, follow these steps:

  1. Setup your Collection Agent plugin.

  2. Configure your mappings .

  3. Configure your other Dynamic Entities in the Gateway, see Create Dynamic Entities in Collection Agent setup for a more detailed procedure.

Setup your Collection Agent plugin

Use one of the following options listed below to configure the plugin.

  • Setting up your collector in the Gateway Setup Editor by adding the following configuration in Dynamic EntitiesCollectors. For more information, see Collectors in Dynamic Entities.

  • Adding the following configuration in collection-agent.yml file on your local machine.


Configure your mappings

Use one of the following options listed below to configure your dynamic mappings.

Note: To check if there are any errors in the mapping, you can set up the Dynamic Entities Health, or look at the Collection Agent log file in Collection Agent setup.

Custom mappings

Alternatively, the Prometheus plugin also allows you to create custom mappings so you can modify the display of information in the Geneos dataview. The mappings can be configured differently by using available dimensions or properties.

To create custom mappings for Prometheus Operator:

  1. Navigate to your Prometheus mapping in Dynamic entities > Mappings.

  2. In Options, selection custom.

  3. In the Geneos items > Label, enter the following:

    • job and instance as an entity

    • container, endpoint, namespace, pod, prometheus, prometheus_replica, service, and __itrs_namespace__ as an attribute

Using the custom mappings above, the following dataviews are generated, where the Prometheus plugin monitors the etcd application and the Prometheus server.

Example dataview

Once you set up the plugin with the built-in mapping successfully, the gathered metrics and alerts from the Prometheus server and the Alert Manager, respectively, will be displayed in a dataview in Geneos.

To define how items in the Geneos tree structure are generated from the labels of a datapoint, see Geneos items in Dynamic Entities.