Retrieve Data from Gateway Hub

Overview

One of the most important functions of Gateway Hub is to retrieve metric data recorded by the Gateway. This data can be used by upstream applications such as Grafana for custom dashboards or any other tool which requires metric data.

This document is a worked example explaining how to retrieve data from Gateway Hub for use in an external application. The examples provided use cURL but any HTTP client or library is suitable.

Prerequisites

The Gateway is publishing to Gateway Hub and has Single Sign On enabled using Kerberos for authentication.

The Gateway Hub in this example is located at hub.example.com.

Authenticating

Before interacting with Gateway Hub and accessing the stored data, you must acquire a security token. This token must be passed as an HTTP header along with any subsequent requests.

To request a security token, the endpoint GET /token must be accessed.

Request a security token

First, invoke the following endpoint to receive a security token:

curl -k -s --negotiate -u : https://hub.example.com:/authorize?response_type=token?client_id=rest&state=rest

The following parameters are used in the command above:

  • response_type=token, because a JSON Web Token is expected as the response.
  • client_id=rest, because this is a REST request being validated.
  • state=rest, because the context of the request is for a REST request.

The endpoint returns a JSON Web Token, which looks like the following:

{
  "access_token": "eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJjZGF2aWVzIiwiYXVkIjoicmVzdCIsImlzcyI6IklUUlMgU1NPIEFnZW50IiwiZ3JvdXBzIjpbIkxETiJdLCJzdGF0ZSI6IklHTk9SRSIsImV4cCI6MTU2NjI5Njc1NSwiZGlzcGxheV9uYW1lIjoiQ2hyaXMgRGF2aWVzIiwiaWF0IjoxNTY2Mjk2NDU1LCJub25jZSI6IklHTk9SRSIsInJlcXVlc3Rfc291cmNlIjoiMTkyLjE2OC4xMC4xMzgiLCJlbWFpbCI6ImNkYXZpZXNAaXRyc2dyb3VwLmNvbSJ9.fVe2_a3surVzdzNE8KpWnk7Ph68qP1NzI8ehdSDVDToA3KlSEIohjLH4eLsvUEIPQN4snWM_RG52kwW4AqYSdCQiOuTGJaifwSVKAAykG7QaYcdI_NcEz34W4uBP132sBYoy48h6mUcJkcGDgNIM0nDw6vfubuL8Pv3naalG258WvhOiHTkhnaI35i8nJSXs-6hr5l5ahF6CNcsn5CpMbWSOqrfmbhNVDj-gOj71X89ubC0E5UhlIMKV8hJOUBLHAbCkVp3NeZZ6lFtIJpPjHQlTnlwiSlZKHdxnDMCRC2iBGhE-E2bqvld8EKYsryOOfh1RrAeSKfn2lyyhNA4hqA",
  "expires": 1566296755,
  "token_type": "Bearer"
} 

The access_token is required for any subsequent REST commands to Gateway Hub, and must be returned in the following form:

“Authorization”: “Bearer <access_token>”

With a valid token, Gateway Hub can be queried for the required metric data.

Metric Query Example

For this example there is a Gateway running a Netprobe configured with a CPU plugin.

You want to know for the Managed Entity LDN_KGX Gateway, the average, 95th percentile, and max of the percentUtilisation metric for Average_cpu for over the last hour, in 5 minute buckets.

To do this, construct a query to submit with POST /metrics/query.

Note: Queries are formulated as JSON objects. Since double quotes are a special character in JSON you will need to use escape characters when typing quotes, these have been included in the examples below. When using a GUI you do not need to use escape characters.

For this query provide the following parameters:

  • filter
    • entities
      • You are interested in the metrics from a single server named LDN_KGX Gateway so use the following entity query:

        "entities": "itrs.name = \"LDN_KGX Gateway\""
        

        Note: name is one of the built in attributes so it has the prefix itrs.

    • range
      • You are interested in an hours' data, so in this example, use the following time range:

              "range":{  
                 "from":"2019-06-14T11:00:00Z",
                 "to":"2019-06-14T12:00:00Z"
              }

        Although UTC is used here, any time zone is supported. Be aware that the output of this query is zoned according to the zoneId parameter.

  • zoneId
    • The zoneId is used to determine the timezone to align the output time buckets, as well as the optional time window filter (not used in this example). Here use UTC:
      "zoneId": "UTC"
  • metrics
    • You are interested in a single metric from a single dataview. The identifier field is used to provide an alias when several dataview cells are being combined, and because you are interested in a single metric only, you can use the cell name percentUtilisation here. The include array allows a series of dataview cell references, but because you are only interested in a single cell this is just:
         "metrics":[  
            {  
               "identifier":"percentUtilisation",
               "include":[  
                  "cpu/cpu/Average_cpu/percentUtilisation"
               ]
            }
         ]
    • To query for all rows, use the rowname wildcard:
      	"metrics":[  
      	  {  
      	   	"identifier":"percentUtilisation",
      		"include":[  
      		   "cpu/cpu/{rowname}/percentUtilisation"
      		]
      	  }
      	]

      Note: rowname is the only part of the path that allows using wildcards.

  • aggregations
    • You want the average, the 95th percentile, and the maximum values for each time bucket, therefore specify the following:

         "aggregations":[  
            "itrs.avg",
            "itrs.tdigest.percentile(0.95)",
            "itrs.max"
         ]

      Note: t-digest is an efficient online algorithm for generating close approximates for percentiles from streaming data.

  • bucketing
    • A hours' worth of raw CPU data is not very useful for analysis or charting, so request the data to be aggregated into 5 minute buckets:
         "bucketing":{  
            "duration":"5 minute"
         }
  • grouping
    • You are interested in the metric data of a single entity only, and therefore do not need to perform any additional groupings.

All of these parameters combined give the following HTTP request body:

{  
   "zoneId":"UTC",
   "filter":{  
      "entities":"itrs.name = \"LDN_KGX Gateway\"",
      "range":{  
         "from":"2019-06-14T11:00:00Z",
         "to":"2019-06-14T12:00:00Z"
      }
   },
   "metrics":[  
      {  
         "identifier":"percentUtilisation",
         "include":[  
            "cpu/cpu/Average_cpu/percentUtilisation"
         ]
      }
   ],
   "aggregations":[  
      "itrs.avg",
      "itrs.tdigest.percentile(0.95)",
      "itrs.max"
   ],
   "bucketing":{  
      "duration":"5 minute"
   }
}

Next, POST the query using the following command:

curl -H "Content-Type: application/json" -X POST -d '<request body>' https://hub.example.com:9999/v1alpha/metrics/query

The results are returned in a single JSON object consisting of two parts, the output schema and the output data:

  • schema provides the structure of the output and is used by downstream applications to properly interact with the data.
  • data provides the metric data requested, in the buckets specified.
{
    "schema": {
        "bucketing": {
            "duration": "5 minute"
        },
        "grouping": [],
        "metrics": {
            "percentUtilisation": {
                "itrs.avg": {
                    "type": "float64",
                    "unit": "fraction"
                },
                "itrs.tdigest.percentile_0.95": {
                    "type": "float64",
                    "unit": "fraction"
                },
                "itrs.max": {
                    "type": "float32",
                    "unit": "fraction"
                }
            }
        }
    },
    "data": [
        {
            "bucket": "2019-06-14T11:00:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09108000049988428,
                    "itrs.tdigest.percentile_0.95": 0.12105000019073486,
                    "itrs.max": 0.1219
                }
            }
        },
        {
            "bucket": "2019-06-14T11:05:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09681999981403351,
                    "itrs.tdigest.percentile_0.95": 0.12765000015497208,
                    "itrs.max": 0.1295
                }
            }
        },
        {
            "bucket": "2019-06-14T11:10:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09960666646560033,
                    "itrs.tdigest.percentile_0.95": 0.13364999741315842,
                    "itrs.max": 0.1344
                }
            }
        },
        {
            "bucket": "2019-06-14T11:15:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09436666667461395,
                    "itrs.tdigest.percentile_0.95": 0.129600003361702,
                    "itrs.max": 0.1376
                }
            }
        },
        {
            "bucket": "2019-06-14T11:20:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.0928266669313113,
                    "itrs.tdigest.percentile_0.95": 0.11765000224113464,
                    "itrs.max": 0.1177
                }
            }
        },
        {
            "bucket": "2019-06-14T11:25:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09883333345254262,
                    "itrs.tdigest.percentile_0.95": 0.12640000134706497,
                    "itrs.max": 0.1269
                }
            }
        },
        {
            "bucket": "2019-06-14T11:30:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09623999993006388,
                    "itrs.tdigest.percentile_0.95": 0.11945000290870667,
                    "itrs.max": 0.1203
                }
            }
        },
        {
            "bucket": "2019-06-14T11:35:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09350666751464208,
                    "itrs.tdigest.percentile_0.95": 0.1218000017106533,
                    "itrs.max": 0.122
                }
            }
        },
        {
            "bucket": "2019-06-14T11:40:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09303999990224839,
                    "itrs.tdigest.percentile_0.95": 0.11589999869465828,
                    "itrs.max": 0.1183
                }
            }
        },
        {
            "bucket": "2019-06-14T11:45:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09660000056028366,
                    "itrs.tdigest.percentile_0.95": 0.12773749884217978,
                    "itrs.max": 0.1301
                }
            }
        },
        {
            "bucket": "2019-06-14T11:50:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09768666625022888,
                    "itrs.tdigest.percentile_0.95": 0.12480000406503677,
                    "itrs.max": 0.1253
                }
            }
        },
        {
            "bucket": "2019-06-14T11:55:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09436666568120321,
                    "itrs.tdigest.percentile_0.95": 0.12285000085830688,
                    "itrs.max": 0.125
                }
            }
        }
    ]
}

Aggregations

Aggregations are statistical operations performed over each group or bucket of data.

Aggregation Description
itrs.tdigest.cdf

The approximate Cumulative Distribution Function (CDF) of values in the group using the T-Digest algorithm.

This approximation is highly accurate in tail regions of the distribution but not in non-tail regions, and should not be used if you need exact calculations.

itrs.percentile

The exact percentile of values in the group.

This may be slower than T-Digest based operations.

itrs.count The count of all values in the group.
itrs.sum The sum of all values in the group.
itrs.tdigest.percentile

The approximate percentile (inverse CDF) of values in the group using the T-Digest algorithm.

This approximation is highly accurate in tail regions of the distribution but not in non-tail regions, and should not be used if you need exact calculations.

itrs.min The smallest (minimal) value in the group.
itrs.max The largest (maximal) value in the group.
itrs.stddev The standard deviation of all values in the group.
itrs.avg The mean of all values in the group.
itrs.var The variance of all values in the group.