["Geneos > Netprobe"]["User Guide"]

Azure Event Hub

Overview

The Azure Event Hub plugin is a Collection Agent plugin that gets events from an Azure Event Hub and then translates them into Collection Agent datapoints. This plugin uses the Azure Java SDK to consume event hub messages and publishes them to a Netprobe, then it displays data as dataview in Geneos.

The Azure Event Hub plugin comes with the Azure Monitor plugin binaries. In order to use the Azure Event Hub plugin, you only need to add the configuration settings in the Collection Agent YAML file, and create an Event Hub in your Azure Portal. See Azure Event Hub deployment.

Deployment recommendations

Azure Monitor plugin supports the ability to consume real-time metrics from the Azure Event Hub service. This is an alternative to polling Azure Monitor API, enabling Geneos to monitor your Azure cloud environment at scale and in real-time.

Use the Azure Event Hub deployment option in the following cases:

  • If you have Azure diagnostics set up to publish metrics to an Event Hub, you may want to use the Azure Event Hub plugin for gathering metrics. This provides a higher throughput of metrics and is one way to avoid issues caused by Azure API limitations.

  • Your Azure Monitor API usage is near the hourly API limit as your Azure estate grows larger. The best approach to preventing this is identifying the priority of services and resources that you want to monitor, and then slowly migrating them to Event Hub streaming to use the Azure Event Hub plugin.

The Azure Event Hub plugin can also be used in different scenarios. For more information, see Advanced usage of Azure Event Hub.

Prerequisites

Geneos environment

The Azure Monitor plugin and Azure Event Hub plugin has the same Geneos environment requirements. See Geneos environment in Azure Monitor.

Azure environment

The Azure Event Hub plugin requires the following:

Note: You may configure authentication for the port connection to meet your security requirements, but you need to provide the connection string to the plugin. See connection string from Azure.

You can also configure a consumer group if you want to use a consumer group for the Azure Event Hub plugin. If you do not create a consumer group the default consumer group will be used. There should be one consumer group per consumer.

The Azure Event Hub plugin will read all messages from an Event Hub, but it will only process messages that contain metric data published by Azure.

Azure Event Hub deployment

If you have not set up the Azure Monitor plugin, you can refer to the Configure Geneos to deploy Azure Monitor. The set up process is almost the same.

To set up the Azure Event Hub plugin:

  1. You must edit the collection-agent.yml file on your local machine, where the binaries are stored, and add the configuration settings below:

      collectors:
      
        - name: eventhub
          type: plugin
          class-name: AzureEventHubCollector
          
          # Connection string of the Azure Event Hub namespace (required)
          connection-string: <Event Hub connection string>
      
          # Instance name of the Azure Event Hub namespace (required)
          event-hub: <EVENT_HUB_NAME>
      
          # Consumer group name of the Azure Event Hub instance (required)
          consumer-group-name: <CONSUMER_GROUP_NAME>
  2. Set your mappings in the Gateway. Follow the procedures in Create Dynamic Entities in Collection Agent setup. Then, select Azure Monitor V2 in the Built in > Name section.

  3. Link your mapping type to the recently created Azure mapping.

  4. Enable your dynamic managed entities. For reference, see Add a Mapping type in Dynamic Entities.

Note: To check if there are any errors in the mappings, you can set up the Dynamic Entities Health, or look at the Collection Agent log file.

Adjust entity and stale data timeout

Events may be published at different rates. This can cause stale data or missing entities if the publishing rate is slow. If this happens, you can adjust the Entity timeout and Stale data timeout.

Example dataview

Once you set up the plugin and the mappings successfully, the Azure Event Hub plugin metrics will be displayed in a dataview in Geneos.

Advanced usage of Azure Event Hub

This section provides you some scenarios where using the Azure Event Hub plugin can be more helpful to your monitoring.

Filter datapoints published to the Netprobe

The Azure Event Hub plugin will process all diagnostics metrics, and then metrics are sent to the Netprobe for publishing. By default, all metrics read will be sent over, but you can filter which datapoints will be sent to the Netprobe using a workflow. A Collection Agent workflow allows the use of several processors that can be applied to datapoints before these are published. See Collection Agent configuration reference.

For example, if you want to use a drop-filter to drop all metrics that start with Bytes, then add these configuration settings to the Collection Agent YAML file:

workflow:
  store-directory: .
  metrics:
    reporter: tcp
  events:
    reporter: tcp
  common:
    processors:
      # use a drop-filter to drop all metrics that start with "Bytes"
      - type: drop-filter
        matchers:
          - type: name
            name-pattern: Bytes.*

If you want to use a forward-filter to only publish metrics from a specific resource_group, you can use this pattern to further filter by other dimensions (as shown in Show Dynamic Mappings):

workflow:
  store-directory: .
  metrics:
    reporter: tcp
  events:
    reporter: tcp
  common:
    processors:
      - type: forward-filter
        matchers:
         - type: dimension
           key: resource_group
           value: Presales-RG

Monitoring events

When setting up the Netprobe and Collection Agent to connect to an Event Hub, it is useful to also monitor the incoming and outgoing messages in the Event Hub metrics. In an ideal scenario incoming and outgoing messages are mostly equal, which means that events are read by the Azure Event Hub plugin almost immediately.

Note: The load that the Azure Event Hub plugin can consume will vary depending on several factors, such as the processing power of the machine the Netprobe is running on.

You can reduce incoming messages to an Event hub by splitting diagnostics across different services or resource groups, and ensuring that only Azure diagnostics metrics are being published.

Azure Event Hub self monitoring

The Azure Event Hub plugin also provides self monitoring metrics, indicating the plugin name, Event Hub name, consumer group, a few indications of events processed successfully, and events dropped (indicating non-metrics events):

This dataview is created based on the following dimensions and their mappings:

  • resource_group: Azure Event Hub Collector

  • resource_type: selfMonitoring

  • resource_provider: ITRS

The selfMonitoring managed entity displays a dataview that has the following metrics:

Field Description
event_hub_name Name of event hub.
consumer_group Name of consumer group of the event hub instance.
events_processed_per_collection_interval Number of events processed per collection interval.
events_processed_per_hour

Number of events processed per hour.

This metric resets every start of the hour (xx:00).

events_dropped_per_collection_interval Number of events dropped per collection internal.
events_dropped_per_hour

Number of events dropped per hour.

This metric resets every start of the hour (xx:00).

   

Monitor multiple event hubs

You can also setup multiple Azure Event Hub plugins, either in the same Collection Agent instance, or in several Collection Agents. This may be useful when spreading the load across multiple Event Hubs, or multiple Netprobes.

However, if you plan to set up multiple Azure Event Hub plugins connecting to the same Event Hub instance, then you should create individual consumer groups for each plugin.

In the example below, the name for each Event Hub must be unique.

- name: eventhub-synthetic
    type: plugin
    class-name: AzureEventHubCollector
    connection-string: "connection_string"
    event-hub: "mnleventhub-1"
    consumer-group-name: "$Default"
  - name: eventhub-actual
    type: plugin
    class-name: AzureEventHubCollector
    connection-string: "connection_string"
    event-hub: "insights-metrics-pt1m"
    consumer-group-name: "$Default"
  - name: eventhub-central
    type: plugin
    class-name: AzureEventHubCollector
    connection-string: "connection_string"
    event-hub: "central-eventhub-1"
    consumer-group-name: "$Default"