Prometheus and Alert Manager
The Prometheus plugin is a Collection Agent plugin that is used to gather metrics from the Prometheus server and alerts from Alert Manager. This plugin collects these data using the Remote Write capability of the Prometheus server, which provides real-time metrics and events.
The latest version of the Prometheus plugin requires Gateway and Netprobe version 5.10.x or higher. The same version must be used for the GSE schema.
The Prometheus plugin binaries are packaged with Netprobe 5.10.x and are stored in the
Alternatively, you can download separate binaries for the Prometheus plugin from the ITRS Downloads.
The Prometheus plugin requires a running Prometheus server. In addition, you must enable and set up the
<remote_write> configuration in your Prometheus server to retrieve real-time metrics. To enable this in a Docker or a Kubernetes environment, add the following in your Prometheus server configuration file:
To enable the
<remote_write> capability of the Prometheus server in a Docker or a Kubernetes environment, add the following in your Prometheus server configuration file:
# prometheus config remote_write: - url: "https://netprobe_location:7654" tls_config: ca_file: /path/to/ca write_relabel_configs: - source_labels: [__name__] regex: 'up' action: keep
- The above configuration: is only valid for certain ways of installing Prometheus. If you are using a Prometheus Operator, then you need to use a different syntax in the configuration file.
- only sends the
The Remote Write capability of the Prometheus server is configured to send only the
up metrics to the Prometheus plugin. The Prometheus server can transmit an unbounded number of metrics, and having too many metrics may overload the Netprobe.
You should filter the list of metrics from the Prometheus server to be ingested by the Netprobe. The Netprobe was tested to handle about 1000 metrics per second. Automatic load balancing, which allows ingestion of all metrics, will be supported in a future version.
The Alert Manager is a service that manages the alerts sent from the Prometheus server. The
<alertmanager_config> setting enables you to publish these handled alerts from the Alert Manager. To enable this setting, add the following in your Prometheus server configuration file:
# alert manager config receivers: - name: 'web.hook' webhook_configs: - url: 'https://netprobe:5001/' tls_config: ca_file: /path/to/ca
Events from the Alert Manager are published as a stream, which can be monitored using File Keyword Monitor plugin. For more information on how to monitor streams, see files > file > source > stream in File Keywoard Monitor configuration.
The Prometheus plugin supports Collection Agent publication into Geneos using dynamic managed entities. To set up the Prometheus plugin in Geneos, follow these steps:
- Edit the
collection-agent.ymlfile on your local machine where the binaries are stored either by:
- Indicating the required settings in the YAML file.
- Indicating the required settings as environment variables as shown in the example configuration below. See Users Variables and Environments.
collectors: - type: plugin class-name: PrometheusRemoteWriteCollector # Optional but recommended component name. name: prometheus # Port on which to receive incoming remote write requests. port: 7654 # Acceptor thread pool size (default = 2) acceptor-thread-pool-size: 2 # Worker thread pool size (default = 4) worker-thread-pool-size: 4 # Optional TLS configuration. tls-config: cert-file: /path/to/cert_file.pem key-file: /path/to/private_key.pem trust-chain-file: /path/to/trust_chain.pem - type: plugin class-name: AlertManagerWebhookCollector # Optional but recommended component name. name: alertmanager # Port on which to receive incoming alert notifications. port: 5001 # Acceptor thread pool size (default = 2) acceptor-thread-pool-size: 2 # Worker thread pool size (default = 4) worker-thread-pool-size: 4 # Optional, but recommended, mapping from alert 'status' to event severity. # If no mappings are specified, all alerts have default severity (NONE). firing-severities: firing: ERROR resolved: INFO # Optional TLS configuration. tls-config: cert-file: /path/to/cert_file.pem key-file: /path/to/private_key.pem trust-chain-file: /path/to/trust_chain.pem
- Set the mappings in the Gateway. To use the built-in Prometheus mapping, follow the procedures in Create Dynamic Entities in Collection Agent setup. To use a custom mapping, see Custom mappings.
- Enable dynamic managed entities. Alternatively, you can use Self-Announcing Netprobes to enable dynamic entities. See Self-Announcing Netprobe in Collection Agent setup.
NoteTo check if there are any errors in the mapping, you can set up the Dynamic Entities Health, or look at the Collection Agent log file in Collection Agent setup.
Once you set up the plugin with the built-in mapping successfully, the gathered metrics and alerts from the Prometheus server and the Alert Manager, respectively, will be displayed in a dataview in Geneos.
To define how items in the Geneos tree structure are generated from the labels of a datapoint, see Geneos items in Dynamic Entities.
Alternatively, the Prometheus plugin also allows you to create custom mappings so you can modify the display of information in the Geneos dataview. The mappings can be configured differently by using available dimensions or properties.
To create custom mappings for Prometheus Operator:
- Navigate to your Prometheus mapping in Dynamic entities > Mappings.
- In Options, selection custom.
- In the Geneos items > Label, enter the following:
instanceas an entity
itrs_sourceas an attribute
The following metrics dimension or properties can be used to map or filter resources in a custom mapping.
Using the custom mappings above, the following dataviews are generated, where the Prometheus plugin monitors the
etcd application and the Prometheus server.