Geneos ["Geneos"]
["Geneos > Netprobe"]["User Guide"]

Google Stackdriver

Overview

Google Stackdriver integration enables you to retrieve all metrics and metadata available in your Stackdriver setup and display them as dataviews in Geneos.

The integration provides a Gateway configuration file that enables monitoring of Stackdriver through a set of pre-configured Toolkit samplers. You can query and list the available metrics for your Google Cloud environment and running instances.

This guide walks you through the integration process to get the Toolkit sampler running with default Stackdriver views.

Audience

This user guide is intended for experienced Geneos users who want to set up and configure an integration between Geneos and Stackdriver. This is also intended for Geneos users who are monitoring data from Stackdriver.

As a user, you should be familiar with Stackdriver as well as the Google Cloud Platform (GCP) services.

Prerequisites

The following requirements must be met prior to the installation and setup of the template:

  • Python 3.6 or higher.
  • Python package manager (pip).
  • Google API Client for Python.
  • Google Authentication Library.
  • Google Cloud Platform service account key in a form of a JSON file.
  • This requires an additional licence to use. Please contact your ITRS Account Manager for more information.

Installation procedure

The Stackdriver integration is largely pre-configured and involves three primary steps:

  1. Download the integration files from the resources site.
  2. Load the integration file in the Gateway Setup Editor.
  3. Configure the environment.
  4. Associate the samplers with a managed entity.

Note: The include files are intended to be read-only and should not be edited (unless the integration specifically requires for them to be modified). This is because include files can be shared between a few instances of a Gateway and best practice is to share them in their default state. It is also to make the upgrades and maintenance process of the integration files easier. All configuration should be carried out in your local Gateway files instead.

Download the integration files

  1. Download the integration package geneos-integration-gcp-<version>.zip from the ITRS Downloads site.
  2. As a best practice, save the files on the same machine where your Gateway is running.

The integration package includes a number of files. These are the files that determine the integration setup:

  • include/gcpMonitoring.xmlGateway configuration file (also called an include file) containing Toolkit samplers configured for Stackdriver integration.
  • scripts/list_instances.py — Python script that calls Stackdriver API and return a JSON format. The JSON values are then processed in a CSV format and displayed in Active Console. This displays available metrics for your services.
  • scripts/requirements.txt — list of the required Google libraries. They can be installed by running the pip command against this requirements file:
    $ pip install -r requirements.txt

Load the integration files

To load the integration files into the Gateway Setup Editor, follow these steps:

  1. Open Gateway Setup Editor.
  2. In the Navigation panel, click Includes to create a new file.
  3. Enter the location of the file to include in the Location field. In this example, it is the include/gcpMonitoring.xml.
  4. Update the Priority field. This can be any value except 1. If you input a priority of 1, the Gateway Setup Editor will return an error.
  5. Expand the file location in the Include section.
  6. Select Click to load.
  7. Click Yes to load the new include file.

Success: Gateway configuration now appears in the Includes section.

Environment variables

The gpsMonitoring.xml script provides the variables that are set in the Environments section. They are configured to ensure that the integration monitors your Stackdriver environment.

Variable Type Description
GOOGLE_APPLICATION_CREDENTIALS String

Path of the JSON file with your Google Cloud Platform service key.

GOOGLE_MANUAL Integer

Frequency of sampling. If checks are executed manually, the value is shown as 0.

Note: The default configuration of this integration is to perform manual sampling. This is because Google charges for each request that you make,.

GOOGLE_PROJECT_ID String

ID of the Google project you are connecting to.

GOOGLE_DRILL_METRIC String Key path of a specific metric used for creating dataviews of a single Stackdriver metric.
GOOGLE_DRILL_VIEW String A sub-string used to match a set of metric key paths. Used for creating dataviews of multiple metrics.
PYTHON_PATH String Path of the Python library located on the machine where the Netprobeis running.
     

Configure the environment

The default environment configuration is created by the gcpMonitoring.xml script. You should create a new environment in your main Gateway setup based on the default one.

  1. In the Gateway Setup Editor, locate your Gateway under the Includes section.
  2. Copy the default GCP environment.
  3. In the Gateway Setup Editor, create a new environment by right-clicking the Environment folder. Paste the copied environment.
  4. Rename the new environment to distinguish it from the default one. Example: GCP_env
  5. Modify the environment variables according to your system.

Note: Standard variable scoping rules apply when variable reference is used in include files. Please take note of that when creating your own variables. For more information, see Variable Scoping and Resolution in User Variables and Environments.

Associate the samplers with a managed entity

After you have loaded the configuration file, the Stackdriver monitoring samplers become available for use. These samplers are associated with a pre-defined Type. To associate the samplers with a managed entity, follow these steps:

  1. In the Gateway Setup Editor, create a new managed entity by right-clicking the Managed entities folder.
  2. Under the Basic tab for the new managed entity, add a pre-configured GCP type.
  3. Specify the environment associated with the samplers. This is the new environment that you created in your main Gateway setup. Example: GCP_env.
  4. Apply the managed entity to the running netprobe.
  5. Under the Basic tab for the new managed entity, add a pre-configured GCP type.
  6. Apply the managed entity to the running Netprobe.
  7. Click Validate current document to check your configuration.
  8. Click Save current document to apply the changes.

Success: After you save your changes to the managed entity, the Google Stackdriver dataviews appear under the managed entity on the state tree.

For more information about include files, see the 'Include' files in Gateway Setup Editor.

Dataviews configuration

This integration provides a number of default dataviews, but you can add more based on which metrics and services your Stackdriver is monitoring. The default StackDriver-Monitoring dataview lists all the available metrics and their respective key paths. These key paths are used to configure the aggregated mode and detail views.

Configure a sampler with a single metric

To configure a sampler with a particular Stackdriver metric, use a GOOGLE_DRILL_METRIC environment variable. The value of that variable is a key path of the metric you want to see the details of. Key paths are displayed by the StackDriver-Monitoring dataview.

Configure a sampler with multiple metrics

To configure a sampler with a set of Stackdriver metrics, use a GOOGLE_DRILL_VIEW environment variable. The value of that variable is a sub-string of the target key paths. Key paths are displayed by the StackDriver-Monitoring dataview.

For example, to display a set of metrics referring to CPU, you can use the compute.googleapis.com/instance/cpu sub-string and it will cross reference the following metrics:

  • compute.googleapis.com/instance/cpu/reserved_cores
  • compute.googleapis.com/instance/cpu/usage_time
  • compute.googleapis.com/instance/cpu/utilization

Metrics and dataviews

Stackdriver integration enables you to display all the metrics and running instances that are available in your Google Cloud Platform environment. The dataviews are divided into three categories:

  • Configuration mode — dataview that displays all the available services and metrics along with their respective key paths. These key paths are used to configure the aggregated mode and detail views. This integration provides a default configuration mode dataview named StackDriver-Monitoring.
  • Aggregated mode — dataviews that display service level views. In this mode, instances are always displayed as rows and all available metrics are displayed as columns. Example configuration to show all compute engine instances and available metrics:
    key path = compute.googleapis.com
  • Detail view — dataviews that display individual metrics such as CPU utilization or disk read bytes. The configuration uses key paths specific to the individual metric. Example configuration to only show a detailed view of CPU utilization for each instance of the compute engine.
    key path = compute.googleapis.com/instance/cpu/utilization

The sections below list metrics and example dataviews available with the Stackdriver integration.

StackDriver-Monitoring

This is a configuration mode sampler that displays all available metrics with their key paths and descriptions. These key paths are then used to create dataviews that display details of the selected metrics.

Headline field Description
execDuration

Elapsed time for the Toolkit plug-in to run the script.

Unit: seconds

 

Column Name Description
Display Name Name of the available metric.
Description Description of the metric.
Key Path

Key path of the specific metric. The key path is used to drill down to more details of the selected metric.

StackDriver-GoogleComputeEngine

This sampler displays an aggregated view of all metrics and their values for each instance. To see details of the metrics you can create more aggregated views or detail views. Sections below present examples of such views.

StackDriver-CPU

This sampler displays an aggregated view of all CPU instances. To see the detailed metrics of each instance, use the detail views.

StackDriver-CPU-UsageTime

This sampler displays a detail view of the CPU instances providing metrics of the usage time.

StackDriver-CPU-Utilization

This sampler displays a detail view of the CPU instances providing metrics of the CPU utilization.

StackDriver-Disk

This sampler displays an aggregated view of all metrics available to monitor disks. To see the detailed values for each metric, use the detail views.

StackDriver-Disk-Read-Bytes

This sampler displays a detail view of the read_bytes_count metric for each instance and disk.

StackDriver-Firewall

This sampler displays an aggregated view of all metrics available for firewall. To see the detailed values for each metric, use the detail views.

StackDriver-Firewall-Dropped-Bytes

This sampler displays a detail view of the dropped_bytes_count metric for each instance.

StackDriver-Firewall-Dropped-Packets

This sampler displays a detail view of the dropped_packets_count metric for each instance.