Retrieve Data from Gateway Hub

Overview

One of the most important functions of Gateway Hub is to retrieve metric data recorded by the Gateway. This data can be used by upstream applications such as Grafana for custom dashboards or any other tool which requires metric data.

This document is a worked example explaining how to retrieve data from Gateway Hub for use in an external application. The examples provided use cURL but any HTTP client or library is suitable.

Prerequisites

The Gateway is publishing to Gateway Hub.

The Gateway Hub in this example is located at hub.example.com.

Authentication

In production environments, you must ensure that the Gateway Hub has Single Sign-On enabled and is using Kerberos for authentication.

Before interacting with Gateway Hub and accessing the stored data, you must acquire a security token. This token must be passed as an HTTP header along with any subsequent requests.

To request a security token, the endpoint GET /token must be accessed.

Request a security token

First, invoke the following endpoint to receive a security token:

curl -k -s --negotiate -u : https://hub.example.com:8080/authorize?response_type=token?client_id=rest&state=rest

The following parameters are used in the command above:

response_type=token, because a JSON Web Token is expected as the response.
client_id=rest, because this is a REST request being validated.
state=rest, because the context of the request is for a REST request.

The endpoint returns a JSON Web Token, which looks like the following:

{
  "access_token": "eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJjZGF2aWVzIiwiYXVkIjoicmVzdCIsImlzcyI6IklUUlMgU1NPIEFnZW50IiwiZ3JvdXBzIjpbIkxETiJdLCJzdGF0ZSI6IklHTk9SRSIsImV4cCI6MTU2NjI5Njc1NSwiZGlzcGxheV9uYW1lIjoiQ2hyaXMgRGF2aWVzIiwiaWF0IjoxNTY2Mjk2NDU1LCJub25jZSI6IklHTk9SRSIsInJlcXVlc3Rfc291cmNlIjoiMTkyLjE2OC4xMC4xMzgiLCJlbWFpbCI6ImNkYXZpZXNAaXRyc2dyb3VwLmNvbSJ9.fVe2_a3surVzdzNE8KpWnk7Ph68qP1NzI8ehdSDVDToA3KlSEIohjLH4eLsvUEIPQN4snWM_RG52kwW4AqYSdCQiOuTGJaifwSVKAAykG7QaYcdI_NcEz34W4uBP132sBYoy48h6mUcJkcGDgNIM0nDw6vfubuL8Pv3naalG258WvhOiHTkhnaI35i8nJSXs-6hr5l5ahF6CNcsn5CpMbWSOqrfmbhNVDj-gOj71X89ubC0E5UhlIMKV8hJOUBLHAbCkVp3NeZZ6lFtIJpPjHQlTnlwiSlZKHdxnDMCRC2iBGhE-E2bqvld8EKYsryOOfh1RrAeSKfn2lyyhNA4hqA",
  "expires": 1566296755,
  "token_type": "Bearer"
}

The access_token is required for any subsequent REST commands to Gateway Hub, and must be returned in the following form:

“Authorization”: “Bearer <access_token>”

With a valid token, Gateway Hub can be queried for the required metric data.

Entity queries

Entity information can be accessed using the /entities endpoint. To use this endpoint make a request to the hub using a URL of the form https://hubhost:8081/v0/entities replacing the hostname as appropriate.

For more information about creating entity queries, see Entity filter syntax.

Metric queries

In this example there is a Gateway running a Netprobe configured with a CPU plugin.

You want to know for the Managed Entity LDN_KGX Gateway, the average, 95th percentile, and max of the percentUtilisation metric for Average_cpu for over the last hour, in 5 minute buckets.

You can access metric information using the /metrics/query endpoint. To do this, make a POST request to Gateway Hub using a url of the form https://hubhost:8081/v0/metrics/query replacing the hostname as appropriate.

Note: Queries are formulated as JSON objects. Since double quotes are a special character in JSON you will need to use escape characters when typing quotes, these have been included in the examples below. When using a GUI you do not need to use escape characters.

For this query provide the following parameters:

filter
- entities
  - You are interested in the metrics from a single server named LDN_KGX Gateway so use the following entity query:
```
"entities": "itrs.name = \"LDN_KGX Gateway\""
```
    Note: name is one of the built in attributes so it has the prefix itrs.
- range
  - You are interested in an hours' data, so in this example, use the following time range:
```
      "range":{  
         "from":"2019-06-14T11:00:00Z",
         "to":"2019-06-14T12:00:00Z"
      }
```
    Although UTC is used here, any time zone is supported. Be aware that the output of this query is zoned according to the zoneId parameter.
zoneId
- The zoneId is used to determine the timezone to align the output time buckets, as well as the optional time window filter (not used in this example). Here use UTC:
```
"zoneId": "UTC"
```
metrics
- You are interested in a single metric from a single dataview. The identifier field is used to provide an alias when several dataview cells are being combined, and because you are interested in a single metric only, you can use the cell name percentUtilisation here. In the general case, note that the identifier field can only contain letters, numbers, hyphens, dots, or underscores. The include array allows a series of dataview cell references, but because you are only interested in a single cell this is just:
```
   "metrics":[  
      {  
         "identifier":"percentUtilisation",
         "include":[  
            "cpu/cpu/Average_cpu/percentUtilisation"
         ]
      }
   ]
```
- To query for all rows, use the rowname wildcard:
```
	"metrics":[  
	  {  
	   	"identifier":"percentUtilisation",
		"include":[  
		   "cpu/cpu/{rowname}/percentUtilisation"
		]
	  }
	]
```
  Note: rowname is the only part of the path that allows using wildcards.
aggregations
- You want the average, the 95th percentile, and the maximum values for each time bucket, therefore specify the following:
```
   "aggregations":[  
      "itrs.avg",
      "itrs.max"
   ]
```
bucketing
- A hours' worth of raw CPU data is not very useful for analysis or charting, so request the data to be aggregated into 5 minute buckets:
```
   "bucketing":{  
      "duration":"PT5M"
   }
```
  For more information about available bucket sizes, see Query metrics in Gateway Hub REST API.
grouping
- You are interested in the metric data of a single entity only, and therefore do not need to perform any additional groupings.

All of these parameters combined give the following HTTP request body:

{  
   "zoneId":"UTC",
   "filter":{  
      "entities":"itrs.name = \"LDN_KGX Gateway\"",
      "range":{  
         "from":"2019-06-14T11:00:00Z",
         "to":"2019-06-14T12:00:00Z"
      }
   },
   "metrics":[  
      {  
         "identifier":"percentUtilisation",
         "include":[  
            "cpu/cpu/Average_cpu/percentUtilisation"
         ]
      }
   ],
   "aggregations":[  
      "itrs.avg",
      "itrs.max"
   ],
   "bucketing":{  
      "duration":"PT5M"
   }
}

If SSO is enabled then you must also include the access token in this request body.

Next, POST the query using the following command:

curl -H "Content-Type: application/json" -X POST -d '<request body>' https://hub.example.com:8080/v1alpha/metrics/query

The results are returned in a single JSON object consisting of two parts, the output schema and the output data:

schema provides the structure of the output and is used by downstream applications to properly interact with the data.
data provides the metric data requested, in the buckets specified.

{
    "schema": {
        "bucketing": {
            "duration": "PT5M"
        },
        "grouping": [],
        "metrics": {
            "percentUtilisation": {
                "itrs.avg": {
                    "type": "float64",
                    "unit": "fraction"
                },
                "itrs.max": {
                    "type": "float32",
                    "unit": "fraction"
                }
            }
        }
    },
    "data": [
        {
            "bucket": "2019-06-14T11:00:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09108000049988428,
                    "itrs.max": 0.1219
                }
            }
        },
        {
            "bucket": "2019-06-14T11:05:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09681999981403351,
                    "itrs.max": 0.1295
                }
            }
        },
        {
            "bucket": "2019-06-14T11:10:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09960666646560033,
                    "itrs.max": 0.1344
                }
            }
        },
        {
            "bucket": "2019-06-14T11:15:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09436666667461395,
                    "itrs.max": 0.1376
                }
            }
        },
        {
            "bucket": "2019-06-14T11:20:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.0928266669313113,
                    "itrs.max": 0.1177
                }
            }
        },
        {
            "bucket": "2019-06-14T11:25:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09883333345254262,
                    "itrs.max": 0.1269
                }
            }
        },
        {
            "bucket": "2019-06-14T11:30:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09623999993006388,
                    "itrs.max": 0.1203
                }
            }
        },
        {
            "bucket": "2019-06-14T11:35:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09350666751464208,
                    "itrs.max": 0.122
                }
            }
        },
        {
            "bucket": "2019-06-14T11:40:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09303999990224839,
                    "itrs.max": 0.1183
                }
            }
        },
        {
            "bucket": "2019-06-14T11:45:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09660000056028366,
                    "itrs.max": 0.1301
                }
            }
        },
        {
            "bucket": "2019-06-14T11:50:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09768666625022888,
                    "itrs.max": 0.1253
                }
            }
        },
        {
            "bucket": "2019-06-14T11:55:00Z",
            "group": {},
            "metrics": {
                "percentUtilisation": {
                    "itrs.avg": 0.09436666568120321,
                    "itrs.max": 0.125
                }
            }
        }
    ]
}

Aggregations

Aggregations are statistical operations performed over each group or bucket of data.

Aggregation	Description
`itrs.percentile(p)`	Exact percentile of values in the group. The value of `p` is the floating point number indicating the desired percentile, for example `0.95`.
`itrs.count`	Count of all values in the group.
`itrs.sum`	The sum of all values in the group.
`itrs.min`	Smallest (minimal) value in the group.
`itrs.max`	Largest (maximal) value in the group.
`itrs.stddev`	Standard deviation of all values in the group.
`itrs.avg`	Mean of all values in the group.
`itrs.var`	Variance of all values in the group.
`itrs.first`	Value of the sample with the earliest timestamp in the group.
`itrs.last`	Value of the sample with the latest timestamp in the group.