Geneos ["Geneos"]
["Geneos > Netprobe"]["User Guide"]

Azure Monitor

Overview

The Azure Monitor plugin is a Collection Agent plugin that is used to monitor Azure resources. This plugin collects Azure Monitor metrics using a combination of API calls using the Azure Java SDK and REST API, then it displays data as dataview in Geneos.

Note: The latest version of the Azure Monitor plugin is 2.0.3.

The Azure Monitor plugin fetches available metrics for Azure resources. The following limitations apply:

  • Aggregations are performed using the smallest time grain supported by each metric.

  • Only the default aggregation is used for each metric available.

  • Some metrics may not be immediately available, which is the expected behaviour of Azure Monitor. Some metrics may be available for premium resources only, or some may be awaiting actions from the user or system to be done on the resource, for example, pulling a docker image from a container registry makes the SuccessfulPullCount and TotalPullCount metrics available.

  • The Azure Monitor plugin will gather as many metrics as possible for each collection interval. Data already reported from previous collection intervals or timestamp will no longer be gathered.

For more information, refer to the individual Azure resource documentation listed under Services Monitored > Azure Monitor Metrics.

Caution: Since the Azure Monitor plugin works by calling into the Azure API, you should be aware of the API usage limits. If the API limits are frequently reached, you may want to limit or monitor the metrics collected, see Azure Monitor API count monitoring. Alternatively, you can also increase the collection interval time. For more information about API limits, see Request limits in Azure.

Prerequisites

Geneos environment

The latest version of the Azure Monitor Collection Agent plugin requires the following versions of Geneos components:

  • Gateway and Netprobe 5.9.x or higher. The same version must be used for the GSE schema.

  • Collection Agent 2.x. To run Collection Agent, see Collection Agent setup.

The Azure Monitor 2.0.3 binaries are packaged with Netprobe 5.9.x, and are stored in the collection_agent folder.

Alternatively, you can download separate binaries for the Azure Monitor plugin from the ITRS Downloads.

Azure environment

The Azure Monitoring plugin requires an app user to be created which will be used to interface with Azure. In addition, you must configure the appropriate roles to enable access to particular services. Please see the following external documentation:

The Monitoring Reader built-in role has been tested with the plugin and can be used as a base for creating custom roles.

You must make the following changes in your Azure account in order to connect to Azure Monitor. The Azure Monitor plugin requires the following:

  • Client ID

  • Client Secret

  • Tenant ID

  • Subscriptions ID

Please refer to the Azure Monitor plugin configuration file below.

Configure Geneos to deploy Azure Monitor

The Azure Monitor plugin supports Collection Agent publication into Geneos using dynamic managed entities. Setting up this plugin in Geneos involves three primary steps:

  • Configure the Collection Agent YAML file and provide the required settings for your Azure Monitor account.

  • Set the mappings in the Gateway.

  • Enable dynamic managed entities. Alternatively, you can use Self-Announcing Netprobes to enable dynamic entities. See Self-Announcing Netprobe in Collection Agent setup.

To set up the Azure Monitor plugin in Geneos:

  1. You must edit the collection-agent.yml file on your local machine where the binaries are stored either by:

    • Indicating the required settings for client ID, client secret, tenant ID, and subscription ID in the YAML file.

    • Indicating the required settings as environment variables as shown in the example configuration below. See User Variables and Environments.

      collectors:
       
        - name: azure
          type: plugin
          class-name: AzureMonitorCollector
       
          # Interval (in millis) between publications (optional, defaults to five minutes)
          collection-interval: 300000
       
          # The Azure client id that can be used for this application (required)
          client-id: ${env:CLIENT_ID}
       
          # The Azure secret associated with the client id (required)
          client-secret: ${env:CLIENT_SECRET}
       
          # The Azure tenant id to be monitored (required)
          tenant-id: ${env:TENANT_ID}
       
          # The Azure subscription id to be used for monitoring (required)
          subscription-id: ${env:SUBSCRIPTION_ID
      								
          # Filter resources by resource group name according to the provided Java regex
          resource-group-filter: (UAT-group)|(PROD-group)
      
          # Specify a list of key or value pairs which correspond to Azure resource tags and values 
          tag-filter:
             Dev: (Name 1)|(Name 2)
             Owner: .*
      
          # Specify a list of tags which will be published as a datapoint property
          published-tags:
            - Environment
            - Purpose
       
          # Lists the resource type(s) that will be monitored (optional, defaults to listing all available resource types)
          enabled-resources:
            - Microsoft-Compute_disks
            - Microsoft-Network_azureFirewalls

      Setting Description
      enabled-resources To add more Azure resources to monitor, you can add them in enabled-resources following this format: <resource-provider>_<resource-type>. This setting allows you to filter based on the resource type. Without this setting, the Azure Monitor plugin will pick up metrics for all available Azure services. See Monitored Azure services .
      published-tags This is useful if you want to use a tag in the dynamic entities mapping configuration. If this is not set, no tags will be included in the metrics published.
      resource-group-filter If a resource does not have a tag that is listed in the configuration, then the property will not be included in the metrics published.
      tag-filter Allows you to monitor only certain resources with tags that match what is configured. A resource will be monitored only if all tags match the resource, for example, the tag is present and the tag value matches the regex provided.
         

      Caution: Setting the collection-interval to a smaller value than the default could cause the plugin to reach the throttling limit more frequently. For more information about throttling requests, see Throttling Resource Manager requests. You may also want to limit or monitor the metrics collected, see Azure Monitor API count monitoring.

  2. Open Gateway Setup Editor.

  3. In Navigation, click Dynamic entities > Mapping, and then click New Mapping.

  4. Provide a name for this mapping instance.

  5. In Options, select builtIn.

  6. Select Azure Monitor V2 in the Built in > Name section. This provides mappings to display the Azure Monitor metrics in Geneos.

  7. Click Dynamic entities > Mapping types, and then click New Mapping type.

  8. Provide a name for the mapping type.

  9. In the Mappings field, click Add new to add the name of the recently created mapping.

  10. Click Save current document.

  11. Under your probe, go to the Dynamic Entities tab.

  12. Select the Start check box in the Collection agent section.

  13. In the Dynamic entities > Mapping type, select the recently created mapping type.

  14. Click Save current document.

Note: To check if there are any errors in the mapping, you can set up the Dynamic Entities Health, check Azure Monitor API count monitoring, or look at the Collection Agent log file in Collection Agent setup.

Example dataview

Once you set up the plugin and mappings successfully, the monitored Azure services including their metrics will display in a dataview in Geneos. In the example below, the Azure resources are sorted according to resource_provider as an entity, resource_type as a sampler, and resource_group as a dataview.

For detailed information about Azure services, see Supported metrics with Azure Monitor.

To define how items in the Geneos tree structure are generated from the labels of a datapoint, see Geneos items in Dynamic Entities.

Custom mappings

Alternatively, the Azure Monitor plugin also allows you to create custom mappings so you can modify the display of information in the Geneos dataview.

The mappings can be configured differently by using available dimensions or properties. All metrics published by the Azure Monitor plugin have a data_kind of azure_v1.

To create custom mappings:

  1. Navigate to your Azure mapping in Dynamic entities > Mappings.

  2. In Options, selection custom.

  3. In the Geneos items > Label, enter the following:

    • resource_group as an entity

    • resource_provider and resource_type as a sampler

The following metrics dimension or properties can be used to map or filter resources in a custom mapping.

Metric property Description
resource_provider

Resource provider values for each service that can be found in Monitored Azure services .

The builtIn mapping uses this to group similar resource types in an entity.

resource_type

Resource type values for each service that can be found in Monitored Azure services .

The builtIn mapping uses this to group similar resource types in an entity.

resource_group

Corresponds to the resource's resource group.

The builtIn mapping uses this as the dataview name.

region Region where the resource is in.
name Name of the resource.
   

Further, the Azure tags for resources will be transferred to metrics which is useful if the Azure estate has its own way of organising resources by tags.

You can also use Show Dynamic Mappings on the metrics to see the dimensions or properties that can be used for the mapping.

The Azure tags will become a property that can be seen in Show Dynamic Mappings. Additionally, there are several Azure plugin built-in properties:

Azure built-in property Description
Azure Tag

Key-value pair that will be a label-value pair in the dynamic mapping.

For example, the Azure tag Department=Engineering becomes Label : Department Value: Engineering.

aggregation_type Default aggregation type for a specific metric.
region Location where the resource resides when deployed in Azure.
   

Note: Given the nature of the Azure tag having different variations of key-value pairs (depending on how you set this up), when a tag's key clashes with an existing property, the tag will not override the existing property. For example, if you create a tag data_kind=test, this will not override the built-in property data_kind.

Monitored Azure services

The table below lists the Azure Monitor services that this plugin can monitor. Each service has its own set of metrics that can be displayed as data in Geneos dataview.

For detailed information about Azure services, see Supported metrics with Azure Monitor.

Azure service Resource provider Resource type
API Management Service Microsoft-ApiManagement service
App Service Plans Microsoft-Web serverfarms
App Services Microsoft-Web sites
Application Gateways Microsoft-Network applicationGateways
Application Insights Microsoft-Insights components
Automation Accounts Microsoft-Automation automationAccounts
Batch Accounts Microsoft-Batch batchAccounts
Cognitive Services Microsoft-CognitiveServices accounts
Container Groups Microsoft-ContainerInstance containerGroups
Container Registries Microsoft-ContainerRegistry registries
Cosmos DB accounts Microsoft-DocumentDB databaseAccounts
Disks Microsoft-Compute disks
DNS Zones Microsoft-Network dnszones
Event Hub Namepsace Microsoft-EventHub namespaces
Express Route Circuits Microsoft-Network expressRouteCircuits
Firewalls Microsoft-Network azureFirewalls
Front Doors Microsoft-Network frontdoors
Function Apps Microsoft-Web sites
Key Vaults Microsoft-KeyVault vaults
Load Balancers Microsoft-Network loadBalancers
Local/Virtual Network Gateway Connections Microsoft-Network connections
MariaDB Servers Microsoft-DBforMariaDB servers
MySQL Servers Microsoft-DBforMySQL servers
Network Interfaces Microsoft-Network networkInterfaces
Network Watchers/Connection Monitors Microsoft-Network networkWatchers
PostgreSQL Servers Microsoft-DBforPostgreSQL servers
Search Services Microsoft-Search searchServices
Service Bus Namespace Microsoft-ServiceBus namespaces
SQL Server/Databases Microsoft-Sql servers
SQL Server/Elastic Pools Microsoft-Sql servers
Storage Accounts Microsoft-Storage storageAccounts
Traffic Manager Profiles Microsoft-Network trafficManagerProfiles
Virtual Machine Scale Sets Microsoft-Compute virtualMachineScaleSets
Virtual Machine Scale Sets/Virtual Machines Microsoft-Compute virtualMachineScaleSets
Virtual Machines Microsoft-Compute virtualMachines
Virtual Network Gateways Microsoft-Network virtualNetworkGateways
Virtual Networks Microsoft-Network virtualNetworks
     

Additional attributes

In addition to the attributes of metrics that Azure Monitor provides, this plugin also collects the following information for each resource.

Azure service Attribute Attribute description
API Management Services sku_capacity Capacity of the SKU or the number of deployed units of the SKU.
API Management Services sku_name Name of the API Management Service SKU.
App Service Plans capacity Capacity of the App Service Plan.
App Service Plans max_instances Maximum number of instances that can be assigned to the App Service Plan.
App Service Plans number_of_web_apps Number of web apps assigned to the App Service Plan.
App Service Plans operating_system Operating system where the app is running on.
App Services default_hostname Default hostname of the app.
App Services state State of the app.
App Services app_service_plan Name of the app's App Service Plan.
Application Gateways operational_state Operational state of the application gateway.
Application Insights app_id Application Insights Unique ID of the application.
Application Insights app_type Type of application being monitored.
Application Insights provisioning_state State of the app whether it has been provisioned within its defined resource group.
Application Insights sampling_percentage Percentage of the data produced by the application being monitored that is being sampled for Application Insights telemetry.
Automation Account creation_time Account creation date.
Automation Account last_modified_time Account last modified date.
Batch Accounts account_endpoint Batch account endpoint.
Cognitive Services endpoint Endpoint of the created Azure Cognitive Services account.
Cognitive Services kind Resource kind.
Cognitive Services public_network_access States if public endpoint access is allowed.
Cognitive Services sku_name SKU Name of an Azure Cognitive Services account.
Cognitive Services sku_tier SKU Tier of an Azure Cognitive Services account.
Container Groups state State of the container group.
Container Groups os_type Base level OS type required by the containers in the group.
Container Groups ip_address IP address.
Container Registries login_server_url URL that can be used to log into the container registry.
Cosmos DB accounts document_endpoint Connection endpoint for the Cosmos DB database account.
DB For MariaDB fqdn Fully Qualified Domain Name.
DB For MariaDB version MariaDB version.
DB For MySQL fqdn Fully Qualified Domain Name.
DB For MySQL version MySQL version.
DB For PostgreSQL fqdn Fully Qualified Domain Name.
DB For PostgreSQL version PostgreSQL version.
Disks vm_id Resource ID of the virtual machine the disk is attached to.
DNS Zones zone_type DNS Zone type.
DNS Zones record_sets_count Current number of record sets in the DNS zone.
DNS Zones record_sets_max_count Maximum number of record sets that can be created in the DNS zone.
Event Hub Namespaces service_bus_endpoint Service bus endpoint associated with the namespace.
Express Route Circuits circuit_provisioning_state CircuitProvisioningState state of the resource.
Firewalls provisioning_state Provisioning state of the Azure firewall resource.
Firewalls sku_name Name of the Azure Firewall SKU.
Firewalls sku_tier Tier of the Azure Firewall SKU.
Firewalls threat_intel_mode Operation mode for Threat Intelligence.
Frontdoors cName The host that each frontendEndpoint must CNAME to.
Function Apps default_hostname Default hostname of the app
Function Apps state State of the function app.
Function Apps app_service_plan Name of the app's App Service Plan.
Key Vaults N/A  
Load balancers N/A  
Local network Gateways/Connections network_gateway1 Name of the virtual network gateway.
Local network Gateways/Connections network_gateway2 Name of the local network gateway.
Network interfaces vm_id Resource ID of the virtual machine the network interface is associated with.
Network interfaces public_ip_* Public IP address the network interface is associated with.
Network Watchers connection_monitor_count Current number of connection monitors in the network watcher.
Network Watchers/Connection Monitors interval_in_secs Monitoring interval in seconds.
Network Watchers/Connection Monitors status Monitoring status of the connection monitor.
Search Services status Status of the search service.
Service Bus Namespace price_tier Price tier of the service bus.
Service Bus Queue status Status of the Queue under a given service.
Service Bus Topic status Status of the Topic under a given service bus.
SQL Server state State of the SQL Server.
SQL Server/Database elastic_pool Name of elastic pool associated with the SQL database.
SQL Server/Elastic Pools state State of the elastic pool.
Storage accounts primary_endpoint_blob Primary blob endpoint of the storage account.
Storage accounts primary_endpoint_file Primary file endpoint of the storage account.
Storage accounts primary_endpoint_queue Primary queue endpoint of the storage account.
Storage accounts primary_endpoint_table Primary table endpoint of the storage account.
Storage accounts primary_endpoint_dfs Primary dfs endpoint of the storage account.
Storage accounts primary_endpoint_web Primary web endpoint of the storage account.
Traffic Manager Profiles N/A  
Virtual Machine Scale Sets instances Number of virtual machine instances in the scale set.
Virtual Machine Scale Sets/Virtual Machines power_state Power state of the virtual machine instance.
Virtual Machine Scale Sets/Virtual Machines computer_name Computer name of the virtual machine instance.
Virtual Machine Scale Sets/Virtual Machines os_type Operating system of this virtual machine.
Virtual Machines os_type Operating system of this virtual machine.
Virtual Network Gateways gateway_type Virtual Network Gateway type.
Virtual Network Gateways sku_name SKU name of the Virtual network gateway.
Virtual Network Gateways sku_tier SKU tier of the Virtual network gateway.
Virtual Network Gateways/Connections network_gateway1 Name of the first virtual network gateway.
Virtual Network Gateways/Connections network_gateway2 Name of the second virtual network gateway.
Virtual Networks total_address_spaces Total number of address spaces of the virtual network.
Virtual Networks total_peerings Total number of peering of the virtual network.
Virtual Networks total_subnets Total number of subnets of the virtual network.
Virtual Networks ddos_protection_enabled True if DDoS protection is enabled for all protected resources in the virtual network.
Virtual Networks vm_protection_enabled True if VM protection is enabled for all the subnets in the virtual network.
     

Azure Monitor API count monitoring

You can also monitor the self-monitoring API stats and other metrics of the Azure Monitor plugin. This is useful to keep a count of the number of API requests and current monitored resources.

When you run the Netprobe where the Azure Monitor plugin mapping is running with, and assuming that the steps in the Configure Geneos to deploy Azure Monitor are implemented correctly, a selfMonitoring managed entity will be available in the Active Console. The selfMonitoring managed entity does not require any further configuration or parameter to be set in the YAML file.

This dataview is created based on these dimensions and their mappings:

  • resource_group: azure

  • resource_type: selfMonitoring

  • resource_provider: ITRS

The selfMonitoring managed entity displays a dataview that has the following metrics:

Field Description
rowname Plugin name indicated in the collection-agent.yml. This follows the format <plugin name - ITRS>.
subscription_id Subscription ID indicated in the collection-agent.yml.
total_resource_groups

Total number of resource groups available on your Azure estate.

This resets every collection interval.

total_enabled_resource_types Total number of resource types listed on enabled-resources parameter in the collection-agent.yml.
collection_interval Collection interval indicated in the collection-agent.yml.
total_api_count_per_collection_interval Total number of Azure API calls made by Azure Monitor per collection interval.
total_api_count_rate_per_hour

Total number of Azure API calls made by Azure Monitor per hour.