Prometheus and Alert Manager

Overview Copied

The Prometheus plugin is a Collection Agent plugin that gathers metrics from the Prometheus server and alerts from Alert Manager. This plugin collects these data using the Remote Write capability of the Prometheus server, which provides real-time metrics and events.

The plugin can also scrape metrics from applications instrumented to expose Prometheus metrics in Prometheus Exposition Format (PEF). The PEF exposed by scrape targets is not as rich as the remote write data model. However, translation from PEF to data points should closely, if not exactly, match the remote write mappings.

Prerequisites Copied

Geneos environment Copied

The latest version of the Prometheus plugin requires Gateway version 5.10.x or higher and Netprobe version 6.7.x or higher.

The Prometheus plugin binaries are packaged with Netprobe, and are stored in the collection_agent folder. Alternatively, you can download the binaries for the Prometheus plugin from the ITRS Downloads.

Caution

Collection Agent and its plugins is no longer packaged with the Netprobe in Geneos 5.14.7 and the subsequent 5.x versions. If you want to run Collection Agent via Netprobe, please upgrade to the current 6.x version of Geneos.

Remote Write capability Copied

If you are using a running Prometheus server, you must enable and set up the following to retrieve real-time metrics for you Prometheus plugin:

To enable the <remote_write> capability of the Prometheus server in a Docker or a Kubernetes environment, add the following in your Prometheus server configuration file:

# prometheus config
remote_write:
  - url: "https://netprobe_location:7654"
    tls_config:
    ca_file: /path/to/ca
    write_relabel_configs:
  - source_labels: [__name__]
    regex: 'up'
    action: keep

Note

  • The above configuration: is only valid for certain ways of installing Prometheus. If you are using a Prometheus Operator, then you need to use a different syntax in the configuration file.
  • only sends the up metrics.

The Remote Write capability of the Prometheus server is configured to send only the up metrics to the Prometheus plugin. The Prometheus server can transmit an unbounded number of metrics, however having too many metrics may overload the Netprobe.

You should filter the list of metrics from the Prometheus server to be ingested by the Netprobe. The Netprobe was tested to handle about 1000 metrics per second. Automatic load balancing, which allows ingestion of all metrics, will be supported in a future version.

Alert Manager environment Copied

The Alert Manager is a service that manages the alerts sent from the Prometheus server. The <alertmanager_config> setting enables you to publish these handled alerts from the Alert Manager. To enable this setting, add the following in your Prometheus server configuration file:

# alert manager config
receivers:
  - name: 'web.hook'
    webhook_configs:
  - url: 'https://netprobe:5001/'
    tls_config:
    ca_file: /path/to/ca

Events from the Alert Manager are published as a stream, which can be monitored using the File Keyword Monitor plugin. For more information on how to monitor streams, see files > file > source > stream in File Keywoard Monitor configuration.

Target Scrape Copied

If you do not have or do not want to use a Prometheus server, then you can scrape metrics from applications instrumented to expose Prometheus metrics in Prometheus Exposition Format (PEF). To scrape metrics, you must configure and enable the PrometheusTargetScrapeCollector in your Collection Agent Prometheus plugin. For more information, see Configure Gateway to receive Prometheus data.

The PEF exposed by scrape targets is not as rich as the metrics from the remote write data model. However, translation from PEF to data points should closely match the remote write mappings.

Configure Gateway to receive Prometheus data Copied

The Prometheus plugin supports Collection Agent publication into Geneos using dynamic managed entities. To set up the Prometheus plugin in Geneos, follow these steps:

  1. Setup your Collection Agent plugin.
  2. Configure your mappings.
  3. Configure your other Dynamic Entities in the Gateway, see Create Dynamic Entities in Collection Agent setup for a more detailed procedure.

Setup your Collection Agent plugin Copied

Use one of the following options listed below to configure the plugin.

collectors:
  - type: plugin
    className: PrometheusRemoteWriteCollector

    # Optional but recommended component name.
    name: prometheus

    # Port on which to receive incoming remote write requests.
    port: 7654
   
    # Acceptor thread pool size (default = 2)
    acceptorThreadPoolSize: 2

    # Worker thread pool size (default = 4)
    workerThreadPoolSize: 4

    # Optional TLS configuration.
    tlsConfig:
      certFile: /path/to/cert_file.pem
      keyFile: /path/to/private_key.pem
      trustChainFile: /path/to/trust_chain.pem
      # Optional. (default = false)
      # As a client (e.g. PrometheusTargetScrapeCollector)
      # - Skips host verification and renders trustChainFile optional. 
      # As a server (e.g. PrometheusRemoteWriteCollector) 
      # - Renders client authentication optional. 
      insecure: false

  - type: plugin
    className: AlertManagerWebhookCollector

    # Optional but recommended component name.
    name: alertmanager

    # Port on which to receive incoming alert notifications.
    port: 5001

    # Acceptor thread pool size (default = 2)
    acceptorThreadPoolSize: 2

    # Worker thread pool size (default = 4)
    workerThreadPoolSize: 4

    # Optional, but recommended, mapping from alert 'status' to signal event severity.
    # Possible signal event severity values are NONE, WARNING, and CRITICAL.
    # If no mappings are specified, all alerts have default severity (NONE).
    firingSeverities:
      firing: WARNING
      resolved: NONE

    # Optional TLS configuration.
    tlsConfig:
      certFile: /path/to/cert_file.pem
      keyFile: /path/to/private_key.pem
      trustChainFile: /path/to/trust_chain.pem
      # Optional. (default = false)
      # As a client (e.g. PrometheusTargetScrapeCollector)
      # - Skips host verification and renders trustChainFile optional. 
      # As a server (e.g. PrometheusRemoteWriteCollector) 
      # - Renders client authentication optional. 
      insecure: false

  - type: plugin
    className: PrometheusTargetScrapeCollector
   
    # Optional but recommended component name.
    name: prometheus-scraper
   
    # Optional. Collection interval in milliseconds (defaults to 60000).
    collectionInterval: 60000

    # Optional. Scrape target connection timeout in milliseconds
    # (defaults to collectionInterval / 2).
    connectionTimeout: 30000

    # Optional. Max scrape content length in bytes (defaults to 16 MB).
    maxContentLength: 10485760

    # Mandatory. Specify scrape targets.
    scrapeTargets:
      # Optional. Define static scrape targets.
      staticTargets:
          # Mandatory.
        - host: localhost
          # Mandatory.
          port: 4567
          # Optional. Scrape path (defaults to /metrics).
          path: /metrics
          # Optional. Metric name exclude filters.
          # If no nameExcludes are defined then no metrics are excluded.
          # If any nameExcludes are defined then for a metric to be excluded its name
          # must match at least 1 filter.
          # The nameExcludes filters take precedence over all nameIncludes filters.
          # If a metric matches both a nameExcludes filter and a nameIncludes filter, it will be excluded.
          nameExcludes:
            # Prefix filters. Names starting with any prefix are excluded.
            prefixes:
              - promtest
            # Regexp filters. Names matching any pattern are excluded.
            patterns:
              - jvm.*
          # Optional. Metric name include filters.
          # If no nameIncludes are defined then all metrics are included.
          # If any nameIncludes are defined then for a metric to be included its name
          # must match at least 1 filter.
          # The nameExcludes filters take precedence over all nameIncludes filters.
          # If a metric matches both a nameExcludes filter and a nameIncludes filter, it will be excluded.
          nameIncludes:
            # Prefix filters. Names starting with any prefix are included.
            prefixes:
              - promtest
            # Regexp filters. Names matching any pattern are included.
            patterns:
              - jvm.*
          # Optional TLS configuration.
          tlsConfig:
            certFile: /path/to/cert_file.pem
            keyFile: /path/to/private_key.pem
            trustChainFile: /path/to/trust_chain.pem
            # Optional. (default = false)
            # As a client (e.g. PrometheusTargetScrapeCollector)
            # - Skips host verification and renders trustChainFile optional. 
            # As a server (e.g. PrometheusRemoteWriteCollector) 
            # - Renders client authentication optional. 
            insecure: false

      # Optional. Define scrape targets discovered via the kubernetes API.
      kubeTargets:

        # Optional. Define namespaces to be scraped (defaults to all namespaces).
        namespaces:
         - itrs

        # Optional. Define scrape annotation (defaults to prometheus.io/scrape).
        scrapeAnnotation: prometheus.io/scrape

        # Optional. Define scrape port annotation (defaults to prometheus.io/port).
        portAnnotation: prometheus.io/port

        # Optional. Define scrape path annotation (defaults to prometheus.io/path).
        pathAnnotation: prometheus.io/path

        # Optional. Metric name exclude filters.
        # If no nameExcludes are defined then no metrics are excluded.
        # If any nameExcludes are defined then for a metric to be excluded its name
        # must match at least 1 filter.
        # The nameExcludes filters take precedence over all nameIncludes filters.
        # If a metric matches both a nameExcludes filter and a nameIncludes filter, it will be excluded.
        nameExcludes:
          # Prefix filters. Names starting with any prefix are excluded.
          prefixes:
            - promtest
          # Regexp filters. Names matching any pattern are excluded.
          patterns:
            - jvm.*

        # Optional. Metric name include filters.
        # If no nameIncludes are defined then all metrics are included.
        # If any nameIncludes are defined then for a metric to be included its name
        # must match at least 1 filter.
        # The nameExcludes filters take precedence over all nameIncludes filters.
        # If a metric matches both a nameExcludes filter and a nameIncludes filter, it will be excluded.
        nameIncludes:
          # Prefix filters. Names starting with any prefix are included.
          prefixes:
            - etcd
          # Regexp filters. Names matching any pattern are included.
          patterns:
            - grpc.*

        # Optional. Map of dimensions to be added to all datapoints.
        dimensions:
          cluster: ${env:CLUSTER}

        # Optionally disable the publishing of pod labels as entity attributes.
        # Defaults to false.
        disableLabelPublishing: false

      # Optional. Define scrape targets discovered from the ports and addresses exposed by Kubernetes Endpoints/Services
      #           Each target will be decorated with the following dimensions:
      #           - job         Either the defined target name or the Endpoint/Service name.
      #           - instance    A composite of the address and port e.g. `192.1.2.3:9090`.
      #           - namespace   Endpoint namespace.
      #           - node        Node name from pod reference.
      #           - pod         Name of the referenced pod.
      kubeEndpointTargets:
        # Optional. Identifies the target and used as the value of the `job` dimension replacing the K8s Endpoint name.
        - name: apiserver
          # Optional. Scrape path (defaults to /metrics).
          path: /metrics
          # Optional. Define namespaces to be scraped (defaults to all namespaces).
          namespaces:
            - default
            - kube-system
          # Optional. Discover targets using the associated K8s Endpoint and/or K8s Service labels and/or annotations.
          matchers:
            - type: label
              key: component
              value: apiserver
            - type: annotation
              key: itrs/monitor
              value: true
          # Optional TLS configuration.
          tlsConfig:
            certFile: /etc/kube-certs/cert.pem
            keyFile: /etc/kube-certs/private_key.pem
            trustChainFile: /etc/kube-certs/ca.pem
            # Optional. (default = false)
            # As a client (e.g. PrometheusTargetScrapeCollector)
            # - Skips host verification and renders trustChainFile optional. 
            # As a server (e.g. PrometheusRemoteWriteCollector) 
            # - Renders client authentication optional. 
            insecure: false

          # Optional. Metric name exclude filters.
          # If no nameExcludes are defined then no metrics are excluded.
          # If any nameExcludes are defined then for a metric to be excluded its name
          # must match at least 1 filter.
          # The nameExcludes filters take precedence over all nameIncludes filters.
          # If a metric matches both a nameExcludes filter and a nameIncludes filter, it will be excluded.
          nameExcludes:
            # Prefix filters. Names starting with any prefix are excluded.
            prefixes:
              - etcd
            # Regexp filters. Names matching any pattern are excluded.
            patterns:
              - .*bucket.*
          # Optional. Metric name include filters.
          # If no nameIncludes are defined then all metrics are included.
          # If any nameIncludes are defined then for a metric to be included its name
          # must match at least 1 filter.
          # The nameExcludes filters take precedence over all nameIncludes filters.
          # If a metric matches both a nameExcludes filter and a nameIncludes filter, it will be excluded.
          nameIncludes:
            # Prefix filters. Names starting with any prefix are included.
            prefixes:
              - apiserver
            # Regexp filters. Names matching any pattern are included.
            patterns:
              - api.*

Configure your mappings Copied

Use one of the following options listed below to configure your dynamic mappings.

Note

To check if there are any errors in the mapping, you can set up the Dynamic Entities Health, or look at the Collection Agent log file in Collection Agent setup.

Custom mappings Copied

Alternatively, the Prometheus plugin also allows you to create custom mappings so you can modify the display of information in the Geneos dataview. The mappings can be configured differently by using available dimensions or properties.

To create custom mappings for Prometheus Operator:

  1. Navigate to your Prometheus mapping in Dynamic entities > Mappings.
  2. In Options, selection custom.
  3. In the Geneos items > Label, enter the following:
    • job and instance as an entity
    • container, endpoint, namespace, pod, prometheus, prometheus_replica, service, and __itrs_namespace__ as an attribute prometheus-custom-mapping

Using the custom mappings above, the following dataviews are generated, where the Prometheus plugin monitors the etcd application and the Prometheus server.

prometheus-custom-mapping-dv2

prometheus-custom-mapping-dv1

Example dataview Copied

Once you set up the plugin with the built-in mapping successfully, the gathered metrics and alerts from the Prometheus server and the Alert Manager, respectively, will be displayed in a dataview in Geneos.

prometheus-builtin-mapping

To define how items in the Geneos tree structure are generated from the labels of a datapoint, see Geneos items in Dynamic Entities.

["Geneos"] ["Geneos > Netprobe"] ["User Guide"]

Was this topic helpful?