Prometheus and Alert Manager

Overview

The Prometheus plugin is a Collection Agent plugin that is used to gather metrics from the Prometheus server and alerts from Alert Manager. This plugin collects these data using the Remote Write capability of the Prometheus server, which provides real-time metrics and events.

Prerequisites

Geneos environment

The latest version of the Prometheus plugin requires Gateway and Netprobe version 5.10.x or higher. The same version must be used for the GSE schema.

The Prometheus plugin binaries are packaged with Netprobe 5.10.x and are stored in the collection_agent folder.

Alternatively, you can download separate binaries for the Prometheus plugin from the ITRS Downloads.

Prometheus environment

The Prometheus plugin requires a running Prometheus server. In addition, you must enable and set up the <remote_write> configuration in your Prometheus server to retrieve real-time metrics. To enable this in a Docker or a Kubernetes environment, add the following in your Prometheus server configuration file:

To enable the <remote_write> capability of the Prometheus server in a Docker or a Kubernetes environment, add the following in your Prometheus server configuration file:

# prometheus config
remote_write:
  - url: "https://netprobe_location:7654"
    tls_config:
    ca_file: /path/to/ca
    write_relabel_configs:
  - source_labels: [__name__]
    regex: 'up'
    action: keep

Note

• The above configuration: is only valid for certain ways of installing Prometheus. If you are using a Prometheus Operator, then you need to use a different syntax in the configuration file.
• only sends the up metrics.

The Remote Write capability of the Prometheus server is configured to send only the up metrics to the Prometheus plugin. The Prometheus server can transmit an unbounded number of metrics, and having too many metrics may overload the Netprobe.

You should filter the list of metrics from the Prometheus server to be ingested by the Netprobe. The Netprobe was tested to handle about 1000 metrics per second. Automatic load balancing, which allows ingestion of all metrics, will be supported in a future version.

Alert Manager environment

The Alert Manager is a service that manages the alerts sent from the Prometheus server. The <alertmanager_config> setting enables you to publish these handled alerts from the Alert Manager. To enable this setting, add the following in your Prometheus server configuration file:

# alert manager config
receivers:
  - name: 'web.hook'
    webhook_configs:
  - url: 'https://netprobe:5001/'
    tls_config:
    ca_file: /path/to/ca

Events from the Alert Manager are published as a stream, which can be monitored using File Keyword Monitor plugin. For more information on how to monitor streams, see files > file > source > stream in File Keywoard Monitor configuration.

Configure Gateway to receive Prometheus data

The Prometheus plugin supports Collection Agent publication into Geneos using dynamic managed entities. To set up the Prometheus plugin in Geneos, follow these steps:

1. Edit the collection-agent.yml file on your local machine where the binaries are stored either by:
   • Indicating the required settings in the YAML file.
   • Indicating the required settings as environment variables as shown in the example configuration below. See Users Variables and Environments.

collectors:
  - type: plugin
    class-name: PrometheusRemoteWriteCollector

    # Optional but recommended component name.
    name: prometheus

    # Port on which to receive incoming remote write requests.
    port: 7654
   
    # Acceptor thread pool size (default = 2)
    acceptor-thread-pool-size: 2

    # Worker thread pool size (default = 4)
    worker-thread-pool-size: 4
   
    # Optional TLS configuration.
    tls-config:
      cert-file: /path/to/cert_file.pem
      key-file: /path/to/private_key.pem
      trust-chain-file: /path/to/trust_chain.pem

  - type: plugin
    class-name: AlertManagerWebhookCollector

    # Optional but recommended component name.
    name: alertmanager

    # Port on which to receive incoming alert notifications.
    port: 5001

    # Acceptor thread pool size (default = 2)
    acceptor-thread-pool-size: 2

    # Worker thread pool size (default = 4)
    worker-thread-pool-size: 4

    # Optional, but recommended, mapping from alert 'status' to event severity.
    # If no mappings are specified, all alerts have default severity (NONE).
    firing-severities:
      firing: ERROR
      resolved: INFO

    # Optional TLS configuration.
    tls-config:
      cert-file: /path/to/cert_file.pem
      key-file: /path/to/private_key.pem
      trust-chain-file: /path/to/trust_chain.pem

2. Set the mappings in the Gateway. To use the built-in Prometheus mapping, follow the procedures in Create Dynamic Entities in Collection Agent setup. To use a custom mapping, see Custom mappings.
3. Enable dynamic managed entities. Alternatively, you can use Self-Announcing Netprobes to enable dynamic entities. See Self-Announcing Netprobe in Collection Agent setup.

Note

To check if there are any errors in the mapping, you can set up the Dynamic Entities Health, or look at the Collection Agent log file in Collection Agent setup.

Example dataview

Once you set up the plugin with the built-in mapping successfully, the gathered metrics and alerts from the Prometheus server and the Alert Manager, respectively, will be displayed in a dataview in Geneos.

To define how items in the Geneos tree structure are generated from the labels of a datapoint, see Geneos items in Dynamic Entities.

Custom mappings

Alternatively, the Prometheus plugin also allows you to create custom mappings so you can modify the display of information in the Geneos dataview. The mappings can be configured differently by using available dimensions or properties.

To create custom mappings for Prometheus Operator:

1. Navigate to your Prometheus mapping in Dynamic entities > Mappings.
2. In Options, selection custom.
3. In the Geneos items > Label, enter the following:

• job and instance as an entity
• container, endpoint, namespace, pod, prometheus, prometheus_replica, service, and itrs_source as an attribute

The following metrics dimension or properties can be used to map or filter resources in a custom mapping.

Using the custom mappings above, the following dataviews are generated, where the Prometheus plugin monitors the etcd application and the Prometheus server.