Geneos ["Geneos"]
["Geneos > Netprobe"]["User Guide"]

Amazon CloudWatch

Overview

Amazon CloudWatch integration with Geneos enables you to retrieve live Amazon Elastic Compute Cloud (EC2) instance information, list CloudWatch metrics, monitor volumes and other interactions configured in Amazon Web Services, and display them as dataviews in Geneos.

The AWS monitoring integration provides a Gateway configuration file that enables monitoring of the AWS environment through a set of pre-configured Toolkit samplers. The script can be configured by a set of environment variables and command line options.

This guide walks you through the integration process.

Audience

This user guide is intended for users who will be setting up, configuring, troubleshooting and maintaining this integration. This is also intended for users who will be using Active Console to monitor data from the AWS environment.

You should be familiar with Python and AWS Command Line Interface (AWS CLI), and with general administration of the AWS environment.

Prerequisites

The following requirements must be met prior to the installation and setup of the template:

  • Python 2.7 or higher, or Python 3.7 or higher.
  • Python 3.3 or higher.
  • Boto, Boto 3 and AWS Command Line Interface (AWS CLI).
  • This requires an additional licence to use. Please contact your ITRS Account Manager for more information.
  • Connection to the AWS services using the security credentials: Access Key ID, Secret Access Key and default region. The keys are generated from your AWS account. When you run aws configure command, it will create a flat file and locate it in your home directory. This file is used by this integration.
  • Note: You can also set up the connection using environment variables. For more information see Environment configuration.

Installation procedure

The AWS monitoring integration is largely pre-configured and involves three primary steps:

  1. Download the integration files from the resources site.
  2. Load the integration file in the Gateway Setup Editor.
  3. Associate the samplers with a managed entity.

Note: The include files are intended to be read-only and should not be edited (unless the integration specifically requires for them to be modified). This is because include files can be shared between a few instances of a Gateway and best practice is to share them in their default state. It is also to make the upgrades and maintenance process of the integration files easier. All configuration should be carried out in your local Gateway files instead.

Download the integration files

  1. Download the integration package geneos-integration-aws-<version>.zip from the ITRS Downloads site.
  2. As a best practice, save the files on the same machine where your Gateway is running.

The integration package includes a number of files. These are the files that determine the integration setup:

  • include/awsMonitoring.xml — Gateway configuration file (also called an include file) containing Toolkit samplers configured for AWS monitoring.
  • Python scripts that call AWS API and return a JSON format. The JSON values are then processed in a CSV format and displayed in Active Console. The scripts are:
    • scripts/Get_CloudMetrics.py — list of available metrics for your services
    • scripts/CostUsage.py — monitoring of AWS costs and usage.
    • scripts/List_AMIs.py — statistics for Amazon Machine Images (AMIs) that you have.
    • scripts/List_EC2s.py — statistics for Amazon Elastic Compute Cloud (EC2) instances that you started.
    • scripts/List_EC2_Vols.py — statistics for Amazon EC2 volumes that you use.
    • scripts/EC2s_Commands.py — list of AWS commands for Geneos. The commands allow you to:
      • Create an AMI based on a running EC2 instance.
      • Start and stop an EC2 instance from an AMI ID.

Load the integration files

To load the integration files into the Gateway Setup Editor, follow these steps:

  1. Open Gateway Setup Editor.
  2. In the Navigation panel, click Includes to create a new file.
  3. Enter the location of the file to include in the Location field. In this example, it is the include/awsMonitoring.xml.
  4. Update the Priority field. This can be any value except 1. If you input a priority of 1, the Gateway Setup Editor will return an error.
  5. Expand the file location in the Include section.
  6. Select Click to load.
  7. Click Yes to load the new include file.

Gateway configuration now appears in the Includes section.

Associate the samplers with a managed entity

After you have loaded the configuration file, the AWS monitoring samplers become available for use. These samplers are associated with a pre-defined Type. To associate the samplers with a managed entity, follow these steps:

  1. In the Gateway Setup Editor, create a new managed entity by right-clicking the Managed entities folder.
  2. Under the Basic tab for the new managed entity, add a pre-configured AWS type.
  3. Specify the environment associated with the samplers. By default, this is AWS.
  4. Apply the managed entity to the running Netprobe.
  5. Click Validate current document to check your configuration.
  6. Click Save current document to apply the changes.

After you save your changes to the managed entity, the AWS dataviews appear under the managed entity on the state tree.

For more information about include files, see the 'Include' files in Gateway Setup Editor.

Environment configuration

The default environment configuration is created by the awsMonitoring.xml script. You can create a new environment or modify the default AWS environment variables according to your system.

Note: Standard variable scoping rules apply when variable reference is used in include files. Please take note of that when creating your own variables. For more information, see Variable Scoping and Resolution in User Variables and Environments.

Environment variables

The awsMonitoring.xml script provides the variables that are set in the Environments section. They are configured to ensure that the integration monitors your target AWS environment. Note that only three variables are Geneos specific, the remaining ones are official environment variables for AWS that can be used in Gateway Setup Editor configuration.

For more information on AWS specific environment variables, see AWS documentation.

Variable Type Description
AWS_COST_DAYS Integer

Geneos specific variable. Used to determine the number of days for which the bill will be displayed in the AWS Cost Usage dataview.

DRILL_AWS_SERVICE String Geneos specific variable. Used for creating Custom CloudMetrics dataviews.
AWS_BY_ID String Geneos specific variable. Used for listing out the accounts in AWS Cost Usage dataview.
AWS_SAMPLE_INT Integer

Geneos specific variable. Frequency of sampling. If checks are executed manually, the value is shown as 0.

Note: The default configuration of this integration is to perform manual sampling. This is because Amazon charges for each request that you make, for example for pulling out metrics from CloudWatch or asking for your billing cycle.

AWS_COST_SAMPLE_INT Integer

Geneos specific variable. Frequency of sampling specific to the AWS Cost Usage sampler. If checks are executed manually, the value is shown as 0.

Note: The default configuration of this integration is to perform manual sampling. This is because Amazon charges for each request that you make, for example for pulling out metrics from CloudWatch or asking for your billing cycle.

AWS_ACCESS_KEY_ID String AWS access key ID. Use this variable if you want to connect to AWS environment with environment variables instead of AWS CLI.
AWS_SECRET_ACCESS_KEY String AWS secret key. Access and secret key variables override credentials stored in credential and configuration files. Use this variable if you want to connect to AWS environment with environment variables instead of AWS CLI.
AWS_DEFAULT_REGION String AWS region. This variable overrides the default region of the profile that is in use. Use this variable if you want to connect to AWS environment with environment variables instead of AWS CLI.
AWS_SESSION_TOKEN String Specify a session token if you are using temporary security credentials.
AWS_DEFAULT_OUTPUT String

AWS Command Line (CLI) output formatting.

Example: JSON, text or table.

AWS_PROFILE String Name of the AWS CLI profile to use. This can be the name of a profile stored in a credential or configuration file. If you want to use the default profile, then input Default.
AWS_CA_BUNDLE String Path to a certificate bundle to use for HTTPS certificate validation.
AWS_SHARED_CREDENTIALS_FILE String Location of the file that the AWS CLI uses to store credentials, for example, access keys.
AWS_CONFIG_FILE String

Location of the file that the AWS CLI uses to store configuration profiles.

     

Metrics and dataviews

The sections below list metrics and example dataviews available with the AWS integration.

CloudMetrics

This sampler provides a list of your available Amazon CloudWatch and metrics you can display for them. For getting a particular set of Amazon CloudWatch metrics, specify the service using the DRILL_AWS_SERVICE environment variable in your toolkit configuration. To see the sample metrics configured with this integration, see Custom CloudMetrics.

Headline field Description
execDuration

Elapsed time for the Toolkit plug-in to run the script.

Unit: seconds

   
Column Name Description
Services Services available for your AWS account.
Available Metrics Metrics available for each service.
   

AWS CloudMetrics dataview

Custom CloudMetrics

This sampler provides a sample list of Amazon CloudWatch available with your account and metrics you can display for these services. You can create additional samplers with services available in your Amazon account using the DRILL_AWS_SERVICE environment variable in your toolkit configuration.

AMIs

This sampler shows your backed up Amazon Machine Images (AMIs) within your account. It is also used for creating new EC2 instances bases on a pre-built AMI using custom commands.

Headline field Description
execDuration

Elapsed time for the Toolkit plug-in to run the script.

Unit: seconds

Column Name Description
AMI ID ID of the Amazon Machine Image (AMI)
Image Name Name of the AMI.
Arch

Architecture of the AMI.

Example: 32-bit (i386) or 64-bit (x86_64)

OS Platform Platform on which the AMI is based.
Location Location of the AMI.
State

Current state of the AMI.

Example: available.

State Reason Reason for the current state. For example, if the state shows as terminated, the reason can be: volume limit exceeded.
Root Device Name Name of the root device volume.
Root Device Type

Type of the root device volume.

Example: Amazon EBS or instance store.

Creation Date Time and date when the AMI was created.
Owner ID ID of the owner of the AMI.

EC2 Instances

This sampler displays information of your running Elastic Compute Cloud (EC2) instances. It is also used for starting and stopping the EC2 instances by using custom commands.

Headline field Description
execDuration

Elapsed time for the Toolkit plug-in to run the script.

Unit: seconds

Column Name Description
Instance ID ID of the EC2 instance.
Name Name of the EC2 instance.
Instance Type

Type and model of the EC2 instance.

Example: t2.large, t3.small, m5.xlarge.

Instance State

Current state of the instance.

Example: running, stopped.

State Reason Reason for the displayed state. For example, if the state shows as stopped, the reason can be: user initiated shutdown.
OS Platform Platform on which the EC2 instance is based.
Public IPv4 Public IP of the EC2 instance.
Private IPv4 Private IP of the EC2 instance.
Elastic IPs Static IP of the EC2 instance.
Interface ID Resource identifier of elastic network interface.
Availability Zone Availability zone within the region.
Key Name

Name of the key pair used to launch the EC2 instance.

Security Group (Group ID) Name and ID of the security group assigned to the EC2 instance.

EBS Volumes

This sampler displays information of volumes running with the EC2 instances. It also shows their capacity.

Headline field Description
execDuration

Elapsed time for the Toolkit plug-in to run the script.

Unit: seconds

Column Name Description
Instance ID - Volume ID

ID and name (in brackets) of the EC2 Instance to which the volume is connected.

ID of the volume (indented).

Device name Name of the root device volume.
Size

Volume size.

Unit: GBs

Volume type

Volume type.

Example: gp2 (General Purpose SSD), io1 (Provisioned IOPS SSD)

Device state

The attachment state of the volume.

Example: attached — this means that the volume is attached to an EC2 instance.

State

The state of the attached volume.

Example: in-use.

Attached date The time stamp when the attachment initiated.
Delete on termination Indication whether the volume be deleted when an EC2 instance is terminated.

AWS Cost Usage

This sampler displays information about your EC2 costs. By default, it shows 30 days of cost. You can override this value using the AWS_COST_DAYS variable.

Headline field Description
execDuration

Elapsed time for the Toolkit plug-in to run the script.

Unit: seconds

Column Name Description
Service Name of a service linked to your account that generates costs.
Amount Amount due.
Unit

Currency in which the costs are displayed.