Data Sets (Time series)

Overview Copied

Data sets represent a placeholder for different kinds of data collected by the Gateway. Time series is the only type available.

Time Series data sets can then be used as part of Anomaly Detection and Breach predictor functionality.

Create a Time Series Copied

To create a Time Series:

Select Data sets from the navigation tree in the GSE.
Click New Time series.
Select the type of the time series. The type specifies if the time series is database driven or gathers data from Gateway Hub.
Provide all required information. See Configuration reference for more information about available options.
Validate and click Save current document to save the changes.

Gateway Hub driven Time series Copied

Geneos can use data sets generated by Gateway Hub. The specification for these data sets is set up using GSE, and the Gateway automatically manages their generation and retrieval.

To use Anomaly Detection rules with data sets generated from Gateway Hub you should enable SSO authentication in Gateway Hub. For more information, see Connecting to Gateway Hub

The Gateway uses SSO tokens to access the data set endpoints in Gateway Hub. It is possible to connect to a Gateway Hub without authentication but this is insecure and should not be used in production environments. See Unauthenticated usage with an insecure Gateway Hub in Centralised Gateways User Guide.

To use this method, select Gateway Hub driven as Type when setting up time series. For more information, see Type — Hub.

Database driven Time Series Copied

The time series can be set up by a process external to the Gateway and stored in two tables in the database used for database logging. The tables are:

The schema for these tables is available in the Gateway resources directory provided as part of the Gateway bundle. The data is read from the database at Gateway start time and at the reload time defined in the setup.

It is up to an external process to maintain and update the time series tables. This can be controlled using the Gateway scheduled command.

To use this method, select databaseDriven as Type when setting up time series. For more information, see Configuration reference.

Table name Description

Table name	Description
`time_series_user_table`	Stores a set of names and unique IDs. The names are used to map the names of the time series defined in the Gateway setup to the IDs used in the `time_series_data_user_table`. There are two values per row: `name` — corresponds to the data set name defined in the Gateway and is unique. `time_series_id` — a unique ID used in the `time_series_data_user_table`.
`time_series_data_user_table`	Stores the time series data. There are three values per row: `time_series_id` — ID used to link the data back to the name in the `time_series_data_user_table`. `start_time` — start time for the `value` of a point in the time series data. Time in seconds since the start of the day. `value` — value of the time series at `start_time`.

time_series_user_table

Stores a set of names and unique IDs. The names are used to map the names of the time series defined in the Gateway setup to the IDs used in the time_series_data_user_table.

There are two values per row:

name — corresponds to the data set name defined in the Gateway and is unique.
time_series_id — a unique ID used in the time_series_data_user_table.

time_series_data_user_table

Stores the time series data.

There are three values per row:

time_series_id — ID used to link the data back to the name in the time_series_data_user_table.
start_time — start time for the value of a point in the time series data. Time in seconds since the start of the day.
value — value of the time series at start_time.

Prerequisites Copied

Before you can configure the database driven time series, you need to configure the database tables:

Configure database logging. For more information, see MySQL configuration in Gateway Database Logging.
Ensure that the tables defined in <gateway directory>/resources/database/<database type>/time-series-schema-1.0.sql exist.
Insert data into those tables.

Configuration reference Copied

Type — Gateway Hub driven Copied

This section provides more information about configuration options if you select your time series to be generated from Gateway Hub.

Algorithm — Seasonal-quick Copied

You can specify the algorithm to use to generate the dataset. There is currently one algorithm available, Seasonal-quick:

Setting	Description
Entity query	Specifies the entity query using the entities filter syntax. For example, `user.COMPONENT=EMS`. This query finds all entities with the user-defined managed entity attribute `COMPONENT=EMS`. For more information on how to use the entity filter syntax, see Entity Filter Syntax. Note: In GSE you should only use quotes. Do not escape the quotes with backslashes because the JSON formatter in Gateway does that for you.
Metrics	Data you want to query. This is a sequence of raw metric names to be included in the resulting metric time series. Example: `cpu/cpu/Average_cpu/percentUtilisation` For more information about retrieving metric data, see Metric Query Example Note: In GSE you should only use quotes. Do not escape the quotes with backslashes because the JSON formatter in Gateway does that for you.
Aggregations	Aggregations are calculations you wish to perform on the data. The options are: `count` `sum` `min` `max` `stddev` `avg` `var` `percentile` For aggregations that can be set to a specified level (for example, `percentile`), you must also specify an `alias` to distinguish between multiple entries. Percentage levels are set between `0` and `100`. Aggregations are provided by Gateway Hub, for more information see Aggregations in Retrieve data from Gateway Hub. Note: The percentage aggregation available here uses the `itrs.tdigest.percentile` operation provided by the Gateway Hub API.
Granularity	Seasonal granularity setting is used to create Time Series buckets of selected granularity. It allows you to choose how long an interval each value in the Time Series represents. You can choose from the drop-down menu, or enter a number followed by a length of time. The options in the drop-down menu are: `1 minute` `5 minutes` `15 minutes` `1 hour` `3 hours` `12 hours` `1 day`
Period	The period of cycle which defines seasonality. It allows you to choose the period over which you want the values to repeat. The options are: `Day` `Week`
Periods	Number of periods. It allows you to say how many recent periods (days or weeks) you want the data to be based on.

Period settings example Copied

Here’s an example of how to use period settings (granularity, period, and periods):

You want to write a rule which will compare the current value of an item with a typical value for this time of day, based on data from the last 60 days.

Granularity allows you to choose how long an interval each value in the time series represents. In this case, you should choose 5 minutes or 15 minutes.
Period allows you to choose the period over which you expect the values to repeat: does your time series represent a typical day or a typical week? In this use case, you want to choose a period of day. If the value you are monitoring varies a lot between working and non-working days, and you have enough historical data available, you should choose week.
Periods setting allows you to say how many recent days or weeks you want the data to be based on. In this use case, you want 60 days, so you should set this parameter to 60.

Time to live Copied

This specifies the length of time the time series is valid for. The options are:

1 day
1 hour
1 week
3 hours

When the data is passed from Gateway to Gateway Hub, the default value is 1 day.

The data set is automatically deleted after the time to live value expires.

Setting	Description	Mandatory	XML schema path
Time series	Time series model a day's worth of data uploaded from the database.	No	timeSeries
Name	Specifies a name that you want to identify each time series with. If your time series are database driven, the name must correspond to one of the database tables you have configured.	Yes	timeSeries > name
Description	Specifies additional information about the time series. You can enter multi-line comments in the description field.	No	timeSeries > description
External	Specifies how the external data access is managed.		timeSeries > external
External > Reload time	Specifies the time of the day that data should be uploaded from the database or Gateway Hub every day.	No (default value is the current time during time series creation).	timeSeries > external > reloadTime
Type	Specifies if the time series is database driven or gathers data from Gateway Hub. This setting has two options: Database driven Gateway Hub driven. See Type Hub.	No	timeSeries > type

Obcerv Time Series Copied

Obcerv Time Series builds on the ideas behind Database driven Time Series and Gateway Hub driven Time series. However, unlike these two methods, you do not define a dataset as you would for Database and Gateway Hub-driven time series. Instead, you define a Query Template. This template is specified in the Rules section of the GSE.

Additionally, you need to ensure that you have configured a connection to Obcerv. You need both a Publishing connection so that Obcerv has data to use to generate the seasonal statistics and the Data access connection which is used to query the seasonal statistics from Obcerv. For more information, see Connect Geneos to Obcerv

Obcerv Time Series template Copied

rules > obcervTimeSeriesTemplate > name Copied

Specifies the name of the Obcerv Time Series.
Mandatory: Yes

rules > obcervTimeSeriesTemplate > periodicity Copied

Period of cycle which defines seasonality. This can be set to either one day or one week.
Mandatory: Yes

rules > obcervTimeSeriesTemplate > periods Copied

Number of periods to query.
If the periodicity has been set to one week, and periods is set to six, then the time series will be created using up to six weeks of historic data. If no data is available for a specific bucket, then the data returned from Obcerv for that bucket will be empty. Otherwise, Obcerv will build values with the available data.
Mandatory: Yes

rules > obcervTimeSeriesTemplate > bucketSize Copied

This setting enables the creation of time series buckets with a chosen size. It allows you to choose how long an interval each value in the time series represents. It can assume any of the following options:

5 mins
10 mins
15 mins
30 mins
1 hour
2 hours
3 hours
4 hours
6 hours
12 hours

Mandatory: Yes

rules > obcervTimeSeriesTemplate > reloadTime Copied

Designated time for the Gateway to reload data from Obcerv. The time is applied using the timezone of Gateway. Reloading is done once every day.
Mandatory: Yes

rules > obcervTimeSeriesTemplate > timezone Copied

Timezone of the data. This can either be Gateway, Netprobe, or a specific timezone.
Data will be queried from the previous midnight in the specified timezone, and the day and week periods will be associated with the end time. Mandatory: No Default: Gateway Timezone

Obcerv Time Series Rules Copied

Once a template has been defined, a rule can be created to use a path alias to reference the time series. When the rule is applied to a cell, the Gateway will construct a query and generate a new timeseries for the data referenced in the path alias.

A time series path alias is comprised of an xpath that appends the time series information to an xpath that references a single cell. So the following xpath ./timeSeries[(@obcervTimeSeriesTemplate="Daily")][(@aggregation="avg")] will cause the Gateway to create an Obcerv time series and query the data from Obcerv. The query will include all the information in the obcervTimeSeriesTemplate together with the source data to use, which in this case is the current cell the rule will run on. Other xpaths can be used:

../cell[(@column="latency")]/timeSeries[(@obcervTimeSeriesTemplate="Daily")][(@Aggregation="max")] - this gets the Obcerv Time Series that returns the maximum value of the data for the column called latency in the same row of the dataview.
/geneos/gateway/directory/probe[(@name="P")]/managedEntity[(@name="me")]/sampler[(@name="s")]/dataview[(@name="dv")]/rows/row[(@name="r")]/cell[(@column="c")]/timeSeries[(@obcervTimeSeriesTemplate="Weelky")][(@Aggregation="count")] - this gets the Obcerv Time Series that returns the number of metric values that are in a specific bucket for a specific cell.

To see the value, we can use Compute Engine to set the value of a cell. In that case, if we create a new column on the dataview next to the cell we are interested in, we can set the xpath to ../cell[(@column="Col1")]/timeSeries[(@template="Daily")][(@aggregation="avg")].

When cells matching rules referring to Obcerv Timeseries arrive, the Gateway will initiate queries to Obcerv, generating Time Series data that it will maintain up-to-date.

Click Continue to start the interactive demo below.

Create a Line Chart using Obcerv Time Series data Copied

It is also possible to plot Obcerv time series data in a chart in a similar way that we plot a database time series.

When you right-click on a cell that is the source of time series data that has been queried by Gateway, Obcerv will add an Obcerv Time Series menu item to the context menu. This option will display all the time series queried by Gateway for that specific cell. You can then choose a destination, which can be either a new or existing chart, and then it will be presented with a dialog box.

This allows you to set the start time (Time from) and the end time (Time to) of the chart together with the choice of query result (Graph Type).

Note that it is permissible and often correct for the End Time to precede the Start Time. In this case, the graph will run from Start Time to the end of the day, and then to End Time.

The data is then queried and placed in an Active Chart. The chart will re-query its data if any of the following occur:

Gateway connection drops and reconnects
Template parameters change
Time moves from before the start time to after the start time

The workspace saves enough data that if the AC restarts and connects to the Gateway, it will re-query the Obcerv data and reset the chart.

Previous article Next article