Azure

Overview Copied

The Azure plugin is a Collection Agent plugin that is used to monitor Azure resources. This plugin collects Azure Monitor metrics using a combination of API calls using the Azure Java SDK and REST API, then it displays data as dataview in Geneos.

In addition, the Azure plugin allows you to:

• Combine metrics from Azure with on-premise, multi-cloud, and hybrid environments for end-to-end visibility.
• Monitor real-time alerts using rule and alert capabilities.
• Minimise cost by using a single enterprise tool for monitoring.

To view the versions of Geneos components that are included in the Azure virtual machine image, see Packaged binaries.

Monitored services Copied

You can use the Azure plugin to monitor different services by using the following collectors in the Collection Agent YAML file

• AzureMonitorCollector — for the list of Azure Monitor services that this plugin can monitor, see Monitored Azure services.
• AzureEventHubCollector — for publishing metrics and platform logs from the Azure Event Hub service.

Deployment recommendations Copied

There are multiple approaches in optimising the Azure plugin to monitor a specific estate. In cases where you are nearing the Azure API limits, which is 12000 per hour, you should apply filtering to only monitor the selected services, or deploy other options to run the Azure plugin.

Caution

Since the Azure plugin works by calling into the Azure API, you should be aware of the API usage limits. If the API limits are frequently reached, you may want to limit or monitor the metrics collected, see Azure Monitor API count monitoring. Alternatively, you can also increase the collection interval time. For more information about API limits, see Request limits in Azure.

Deploy Azure through the Azure Marketplace Copied

Use the Deploy Azure Monitor through the Azure Marketplace deployment option for the following reasons:

• An easier way for you to deploy the Azure plugin through the Azure Marketplace with built-in dynamic entity mapping to minimise Gateway configuration.
• This deployment support different scenarios to set up the plugin with minimal configuration.

Configure Geneos to deploy Azure plugin Copied

Use the Configure Geneos to deploy Azure Monitor deployment option in the following cases:

• When you require a native deployment option where you configure the Gateway and Netprobe in Geneos on your local machine.

Configure the Azure Event Hub plugin Copied

Azure Event Hub deployment Copied

Azure plugin supports the ability to consume real-time metrics and platform logs from the Azure Event Hub service. This is an alternative to polling Azure Monitor API, enabling Geneos to monitor your Azure cloud environment at scale and in real-time.

Use the Azure Event Hub deployment option in the following cases:

• If you have Azure diagnostics set up to publish metrics and platform logs to an Event Hub, you may want to use the Azure Event Hub plugin for gathering metrics and platform logs. This provides a higher throughput of metrics and platform logs which is one way to avoid issues caused by Azure API limitations.
• Your Azure Monitor API usage is near the hourly API limit as your Azure estate grows larger. The best approach to preventing this is identifying the priority of services and resources that you want to monitor, and then slowly migrating them to Event Hub streaming to use the Azure Event Hub plugin.

Prerequisites Copied

Geneos environment Copied

The latest version of the Azure Collection Agent plugin requires the following versions of Geneos components:

• Gateway and Netprobe 5.10.x or higher. The same version must be used for the schema.
• Collection Agent 2.x. To run , see setup.

The Azure binaries are packaged with Netprobe, and are stored in the collection_agent folder. Alternatively, you can download separate binaries for the Azure plugin from ITRS Downloads.

Caution

Collection Agent and its plugins is no longer packaged with the Netprobe in Geneos 5.14.7 and the subsequent 5.x versions. If you want to run Collection Agent via Netprobe, please upgrade to the current 6.x version of Geneos.

Azure environment Copied

The Azure plugin requires the following parameters in order to connect to Azure Monitor:

• Client ID
• Client Secret
• Tenant ID
• Subscriptions ID

The Azure plugin requires an app user to be created which will be used to interface with Azure. In addition, you must configure the appropriate roles to enable access to particular services. Please see the following external documentation:

• Quickstart: Register an application with the Microsoft identity platform
• Assign Azure roles using the Azure portal

The Monitoring Reader built-in role has been tested with the plugin and can be used as a base for creating custom roles.

Deploy Azure through the Azure Marketplace Copied

You can deploy the ITRS’s pre-configured virtual machine through the Azure Marketplace that has a running Netprobe with the Azure plugin on port 7036, and optionally, you can run the Gateway on port 7039 that connects to the probe.

The pre-configured virtual machine image created by ITRS contains configuration files and scripts, see Packaged configuration files.

Note

The default behaviour of deploying the virtual machine through the Azure Marketplace does not require any certificates to enable encrypted communication. If you want to switch to secure encrypted communication, see Run Gateway with secure encrypted communication.

Create a virtual machine in Azure Marketplace Copied

Ensure that your Azure account has the necessary credentials in order to access the application. See the requirements for Azure environment.

To create a virtual machine in the Azure Marketplace:

1. Open Azure Portal
2. Click Create a resource under Azure services
3. Search for ITRS Geneos, then click Create. This opens the Create a virtual machine page
4. Provide the required information in all tabs under the Create a virtual machine screen. Some fields are mandatory and the value must not be empty

   Note

   You can configure the remaining required fields depending on your machine requirements. For example, you can freely adjust the size of your virtual machine accordingly and create your own resource group. We only require you to follow the settings below when configuring your virtual machine.

   • In the Basics tab, ensure that the Image field is set to ITRS Geneos Azure plugin - Gen2.
   • In the Advanced tab, add the script below in the Custom data field. This will run the Gateway in demo mode. For other deployment scenarios, you can refer to the Supported scenarios for deploying your virtual machine.

   #cloud-config
   runcmd:
   - bash /geneos/set-azure-parameters.sh --clientID <"client-id"> --clientSecret <"client-secret"> --tenantID <"tenant-id"> --subsID <"subscription-id"> --gatewayArgs "-demo"
   

   In this example, the virtual machine will run the Collection Agent, Netprobe, and Gateway because the --gatewayArgs "-demo" was used in the parameters script. The Gateway will run in demo mode that does not require licencing and with minimal configuration needed.
5. Once the configuration for the virtual machine is completed, click Review + create.
   • If you select SSH public key on the Basics tab as the authentication type, Azure will ask you to download private key.
   • Azure will validate all the requirements that you submitted to create a virtual machine.
   • Click Create.
6. Wait until the deployment status is completed. Then, you can open your resource.

7. After the virtual machine is created, go to the resource and add the inbound port rules for port 7036.
   • In the Networking > Add inbound port rule for 7036, change the Destination port ranges to 7036.
   • In Protocol > TCP, update the name.
   • In the Networking > Add inbound port rule for 7039, change the Destination port ranges to 7039.
   • In Protocol > TCP, update the name.
8. Add the inbound port rules for port 7039.
   • In the Networking > Add inbound port rule for 7039, change the Destination port ranges to 7039.
   • In Protocol > TCP, update the name.
9. Navigate to Overview, copy the public IP address provided to connect via SSH client.
10. Open Active Console Settings > Connections.
    • In the Connections section, add the hostname or the public IP address of your virtual machine.
    • Click OK. This displays the monitored Azure metrics and the Azure Monitor API count monitoring. See Example dataview.

Note

Once you successfully create your virtual machine, and connect Gateway from the Active Console, check if the monitored Azure resources display in the dataview. If not, then you should check the status of the Gateway and Netprobe in the virtual machine, see Check the Gateway and Netprobe status.

Supported scenarios for deploying your virtual machine Copied

This section provides you a list of scenarios to deploy the Azure plugin through the Azure Marketplace. If you want to update your virtual machine, see Update virtual machine using parameters.

Run Gateway with secure encrypted communication Copied

You can run a secured Netprobe in the virtual machine by retrieving these files from a key vault:

Note

You can only run a secured Netprobe or Gateway after creating a virtual machine. Then, you must move the certificate files below to your virtual machine.

• SSL certificate
• SSL certificate key
• SSL certificate chain

You should manually configure the inbound port in the following cases:

• When you enable a secured Gateway, add the port that you set in gateway.setup.xml of your virtual machine which is a secured port.
• When you add the -port in --netprobeArgs, add the inbound port that you set in -port.
• When you add the -port in --gatewayArgs, add the inbound port that you set in -port.

To configure the inbound port, navigate to your virtual machine in Azure Portal, then click Settings > Networking.

To view the parameters that you can use in the --netprobeArgs parameter, see –netprobeArgs, and for --gatewayArgs, see –gatewayArgs.

Run this script in your virtual machine:

bash /geneos/set-azure-parameters.sh --gatewayArgs "-licd-host <host> -licd-port <port> -ssl-certificate <file> -ssl-certificate-key <file> -ssl-certificate-chain <file>" --netprobeArgs "-secure -ssl-certificate <file> -ssl-certificate-key <file> -ssl-certificate-chain <file>"

Run Gateway connecting to an existing Licence Daemon Copied

In this mode, the virtual machine will run the Collection Agent, Netprobe, and Gateway. The Gateway must be connected to an existing LICD that requires normal licencing. Then, you can connect Active Console to the Gateway using the public IP address of your virtual machine on port 7039.

Run this script in your virtual machine:

bash /geneos/set-azure-parameters.sh --clientID <"client-id"> --clientSecret <"client-secret"> --tenantID <"tenant-id"> --subsID <"subscription-id"> --gatewayArgs "-licd-host <hostname> -licd-port <port>"

The Gateway in the virtual machine will use its default values, a port on 7039, and LICD localhost on 7041 if none are provided.

Note

When the default values are not used, for example, if --gatewayArgs "-licd-port 7040" is run, you must create your own Gateway setup and configure the inbound port to allow connection through port 7040 in your Azure virtual machine.

Run Self-announcing Netprobe connecting to an existing Gateway Copied

In this mode, the virtual machine will run the Collection Agent and Netprobe as a Self-announcing Netprobe. The SAN probes need to connect to your own Gateway.

You must add the mapping type named Azure Metrics to your Gateway to see the data in your Active Console. For guidance on how to add the mapping type, refer to the steps in Configure Geneos to deploy Azure Monitor.

Run this script in your virtual machine:

bash /geneos/set-azure-parameters.sh --clientID <"client-id"> --clientSecret <"client-secret"> --tenantID <"tenant-id"> --subsID <"subscription-id"> --san "gwHost=<host>;gwPort=<port>;gwSecure=<value>"

When --san is enabled, you should not use the -setup option in the --netprobeArgs parameter. If you use the -setup option, this will be ignored.

Run normal Netprobe connecting to an existing Gateway Copied

In this mode, the virtual machine will run the Collection Agent and Netprobe in normal mode. This requires you to configure the Gateway to connect to the Netprobe on port 7036.

You must add the mapping type named Azure Metrics to your Gateway to see the data in the Active Console. For guidance on how to add the mapping type, refer to the steps in Configure Geneos to deploy Azure Monitor.

Run this script in your virtual machine:

bash /geneos/set-azure-parameters.sh --clientID <"client-id"> --clientSecret <"client-secret"> --tenantID <"tenant-id"> --subsID <"subscription-id">

Update virtual machine using parameters Copied

You can update your Azure Monitor plugin parameters in the Dynamic Entities > Collectors section of your Gateway Setup Editor. For more information, see Collectors in Dynamic Entities.

You can also test or update your newly created virtual machine in the Azure Portal. To do this, you can use the supported arguments to the set-azure-parameters.sh script.

1. Open your virtual machine under Recent resources.
2. Click Run command > RunShellScript. This opens the Run Command Script screen.
3. Update the script in the Linux Shell Script field. Then, click Run.

Here is the list of parameters that you can use for the set-azure-parameters.sh configuration:

• The client ID, client secret, subscription ID, and tenant ID must all be provided when updating credentials for the Azure plugin.
• To update your logger level, list of enabled resources, collection interval, or SAN options without updating the Azure credentials, you can run any of the following in the Run command > RunShellScript:

bash set-azure-parameters.sh --enabledResources <resources>
bash set-azure-parameters.sh --loggerLevel <logger-level>
bash set-azure-parameters.sh --collectionInterval <interval>
bash set-azure-parameters.sh --san
bash set-azure-parameters.sh --san <san-options>
bash set-azure-parameters.sh --enabledResources <resources> --loggerLevel <logger-level> --collectionInterval <interval> --san <san-options>

• Setting the collection-interval to a smaller value could cause the plugin to reach the throttling limit more frequently. For more information about throttling requests, see Request limits in Azure.
• If no parameter is provided, the shell script will exit.

Packaged configuration files Copied

The pre-configured virtual machine image created by contains the following:

• Gateway, Netprobe, Collection Agent, and Azure plugin inside /geneos with the following configuration files:

  • collection-agent.yml is modified to call on variables set from shell script.
  • gateway.setup.xml is pre-configured to automatically connect the to the in the virtual machine.
  • logback.xml is modified to allow log rollover.
  • san.xml setup file for running the Netprobe in self-announce mode.

• Helper scripts:

  • /geneos/get-diagnostics.sh
  • /geneos/set-azure-parameters.sh
  • /geneos/set-logger-info.sh
  • /geneos/add-enabled-resources.py
  • /geneos/update-collection-interval.py
  • /geneos/update-delay.py

• Service files:

  • /usr/bin/start-netprobe.sh
  • /usr/bin/start-gateway.sh
  • /lib/systemd/system/netprobe.service
  • /lib/systemd/system/gateway.service

Packaged binaries Copied

The pre-configured virtual machine image created by ITRS contains the latest versions of the following Geneos components:

• Geneos Gateway
• Geneos Netprobe
• Collection Agent
• Azure plugin

Troubleshooting Copied

Check the Gateway and Netprobe status Copied

You can run the following commands to check the state of the Gateway and Netprobe in your virtual machine.

Note

Both the Gateway and Netprobe on the virtual machine are run as a systemd service.

1. Open a terminal session in your virtual machine and connect to SSH.
2. Run this command to check if the Gateway and Netprobe are running in the virtual machine:

   # Check the status of Netprobe
   systemctl status netprobe
   
   # Check the status of Gateway
   systemctl status gateway
   

3. Run this command to restart the Gateway and Netprobe if they fail during the initialisation of the virtual machine:

   # Restart the Netprobe
   systemctl restart netprobe
   
   # Restart the Gateway
   systemctl restart gateway
   

Get the diagnostic files Copied

For troubleshooting purposes, you can download a diagnostic file from the Azure virtual machine storage account with the following contents:

• Collection Agent
• Collection Agent logs
• Netprobe logs

To set up your Storage account:

1. Navigate to the Azure Portal
2. Click Storage Accounts under the Azure services.
3. Search for your storage account. If you have not created a storage account, click Create.
4. Enter a name for your container, and set the Public access level to Blob. Then, click Create.
5. Once the container is created, click Shared access tokens under Settings, and then provide the following information:
   • Under Signing method, select the Account key button.
   • On the Permissions menu, select Create.
   • Set the Start and Expiry date and time.
   • Set the allowed IP addresses, if necessary.
   • Under Allowed protocols, select the HTTPS only button.
6. Click Generate SAS token and URL, then copy the Blob SAS token.
7. In your container, go to Settings > Properties, then copy the URL which is needed for the storage account container URL.
8. Navigate to the Azure Portal, and open your virtual machine.
   • Go to the Operations > Run command > RunShellScript. This opens the Run Command Script screen.
   • Enter the following in the Linux Shell Script field:

   bash /geneos/get-diagnostics.sh --url "<Azure storage account container URL>" --token "<Blob SAS token>"
   

9. Once successful, your diagnostics file will be stored in the Storage container > Storage Explorer with a filename netprobe-logs-<timestamp>.tar.gz.

Access Azure cloud through a proxy host Copied

Accessing Azure cloud through a proxy can be configured by adding the https.* properties in the JVM arguments. For example, you can access the cloud via a proxy host and port by adding the following properties:

-Dhttps.proxyHost=webcache.example.com -Dhttps.proxyPort=8080

For more information on adding JVM arguments, see Managed Collection Agent.

To learn more about the available properties to enable proxy access, see Java Networking and Proxies.

Configure Geneos to deploy Azure Copied

The Azure Monitor plugin supports Collection Agent publication into Geneos using dynamic managed entities. Setting up this plugin in Geneos involves three primary steps:

1. Set up your Collection Agent plugin.
2. Configure your mappings.
3. Configure your other Dynamic Entities in the Gateway, see Create Dynamic Entities in Collection Agent setup for a more detailed procedure.

Set up your Collection Agent plugin Copied

Use one of the following options listed below to configure the plugin.

• Setting up your collector in the Gateway Setup Editor by adding the following configuration in Dynamic Entities > Collectors. The available collectors for the Azure Monitor plugin are AzureEventHubCollector and AzureMonitorCollector. For more information on how to set up your collectors, see Collectors in Dynamic Entities.
• Adding the following configuration in collection-agent.yml file on your local machine.

collectors:
 
  - name: azure
    type: plugin
    className: AzureMonitorCollector
 
    # Interval (in millis) between publications (optional, defaults to five minutes)
    collectionInterval: 300000
 
    # The Azure client id that can be used for this application (required)
    clientId: ${env:CLIENT_ID}
 
    # The Azure secret associated with the client id (required)
    clientSecret: ${env:CLIENT_SECRET}
 
    # The Azure tenant id to be monitored (required)
    tenantId: ${env:TENANT_ID}
 
    # The Azure subscription id to be used for monitoring (required)
    subscriptionId: ${env:SUBSCRIPTION_ID}
    
    # Filter resources by resource group name according to the provided Java regex
    resourceGroupFilter: (UAT-group)|(PROD-group)

    # Specify a list of key or value pairs which correspond to Azure resource tags and values 
    tagFilter:
      Dev: (Name 1)|(Name 2)
      Owner: .*

    # Specify a list of tags which will be published as a datapoint property
    publishedTags:
      - Environment
      - Purpose
 
    # Lists the resource type(s) that will be monitored (optional, defaults to listing all available resource types)
    enabledResources:
      - Microsoft.Compute/disks
      - Microsoft.Network/azureFirewalls

Configure your mappings Copied

Use one of the following options listed below to configure your dynamic mappings.

• Choosing a built-in mapping in Dynamic Entities Health > Mapping. For more information, see Mapping and mapping group in Dynamic Entities.
• Setting up a custom mapping in Dynamic Entities Health > Mapping. For more information, see Custom mappings.

Note

To check if there are any errors in the mapping, you can set up the Dynamic Entities Health, check Azure Monitor API count monitoring, or look at the Collection Agent log file.

Custom mappings Copied

Alternatively, the Azure plugin also allows you to create custom mappings so you can modify the display of information in the Geneos dataview.

The mappings can be configured differently by using available dimensions or properties. All metrics published by the Azure plugin have a data_kind of azure_v1.

To create custom mappings:

1. Navigate to your Azure mapping in Dynamic entities > Mappings.
2. In Options, selection custom.
3. In the Geneos items > Label, enter the following:

• resource_group as an entity
• resource_provider and resource_type as a sampler

The following metrics dimension or properties can be used to map or filter resources in a custom mapping.

Further, the Azure tags for resources will be transferred to metrics which is useful if the Azure estate has its own way of organising resources by tags.

You can also use Show Dynamic Mappings on the metrics to see the dimensions or properties that can be used for the mapping.

The Azure tags will become a property that can be seen in Show Dynamic Mappings. Additionally, there are several Azure plugin built-in properties:

Note

Given the nature of the Azure tag having different variations of key-value pairs (depending on how you set this up), when a tag’s key clashes with an existing property, the tag will not override the existing property. For example, if you create a tag data_kind=test, this will not override the built-in property data_kind.

Example dataview Copied

Once you set up the plugin and mappings successfully, the monitored Azure services including their metrics will display in a dataview in Geneos. In the example below, the Azure resources are sorted according to resource_provider as an entity, resource_type as a sampler, and resource_group as a dataview.

For detailed information about Azure services, see Services Monitored > Azure Monitor Metrics.

To define how items in the Geneos tree structure are generated from the labels of a datapoint, see Geneos items in Dynamic Entities.

Monitored Azure services Copied

The Azure plugin fetches available metrics for Azure resources. The following limitations apply:

• Aggregations are performed using the smallest time grain supported by each metric.
• Only the default aggregation is used for each metric available.
• Some metrics may not be immediately available, which is the expected behaviour of Azure Monitor. Some metrics may be available for premium resources only, or some may be awaiting actions from the user or system to be done on the resource, for example, pulling a docker image from a container registry makes the SuccessfulPullCount and TotalPullCount metrics available.
• The Azure plugin will gather as many metrics as possible for each collection interval. Data already reported from previous collection intervals or timestamp will no longer be gathered.

For more information, refer to the individual Azure resource documentation listed under Services Monitored > Azure Monitor Metrics.

Caution

Since the Azure plugin works by calling into the Azure API, you should be aware of the API usage limits. If the API limits are frequently reached, you may want to limit or monitor the metrics collected, see Azure Monitor API count monitoring. Alternatively, you can also increase the collection interval time. For more information about API limits, see Request limits in Azure.

The table below lists the Azure Monitor services that this plugin can monitor. Each service has its own set of metrics that can be displayed as data in Geneos dataview.

For detailed information about Azure services, see Services Monitored > Azure Monitor Metrics.

Dimensions Copied

In addition to the attributes of metrics that Azure Monitor provides, this plugin also collects the following information for each resource.

Additional attributes Copied

In addition to the attributes of metrics that Azure Monitor provides, this plugin also collects the following information for each resource.

Azure API count monitoring Copied

You can also monitor the self monitoring API stats and other metrics of the Azure plugin. This is useful to keep a count of the number of API requests and current monitored resources.

A selfMonitoring managed entity will be available in the Active Console under the ITRS managed entity when you configure correctly the Azure plugin. The selfMonitoring managed entity does not require any further configuration, or parameter to be set in the YAML file.

This dataview is created based on these dimensions and their mappings:

• resource_group: Azure Monitor Collector
• resource_type: selfMonitoring
• resource_provider: ITRS

The selfMonitoring managed entity displays a dataview that has the following metrics: