Obcerv Time Series for Dynamic Thresholds
Overview Copied
Dynamic threshold-based monitoring rules in the Gateway can automatically adjust the threshold based on the historical behavior of the metric. Unlike static thresholds, which remain constant, dynamic thresholds take into consideration patterns and fluctuations in data over time. This means they can adapt to changes and identify anomalies or trends that could indicate potential issues.
Dynamic threshold rules are particularly suitable for metrics that exhibit cyclical variations over a day or a week. For example, the number of connected users, trade volume, and order latency. They are also useful where it is harder to identify the correct threshold values across varying environments. For example, 80% CPU utilization for a known busy server is normal, while the same on an underutilized server could indicate an issue.
Dynamic Thresholds offer number of benefits:
-
Ability to take into account seasonal variations and trends. This means thresholds can automatically adjust to match normal expected behaviour.
-
Better understanding of the normal expected behaviour will minimise false alerts and reduce alert noise.
-
Reduction in alert noise will improve efficiency and reduce operational cost.
Prerequisites Copied
Before you configure dynamic threshold rules in the Gateway, make sure you have the following:
-
An instance of Obcerv to store Gateway data and generate the seasonal time series statistics.
- Requires an Obcerv Platform version 2.3.0 or newer.
- Requires an Obcerv API Gateway application version 2.2.0 or newer.
-
A Gateway configured to publish data to Obcerv and also query time series data from Obcerv.
- Requires Gateway version 6.8.x or newer.
For more information, see Connect Geneos to Obcerv
How to use Dynamic Thresholds in Gateway rules Copied
Click Continue to start the interactive demo below.
Create a new Obcerv Time Series template Copied
Obcerv Time Series builds on the ideas behind Database driven Time Series and Gateway Hub driven Time series. However, unlike these two methods, you do not define a dataset as you would for Database and Gateway Hub-driven time series. Instead, you define a Query Template. This template is specified in the Rules section of the GSE.
rules > obcervTimeSeriesTemplate > name Copied
Specifies the name of the Obcerv Time Series.
Mandatory: Yes
rules > obcervTimeSeriesTemplate > periodicity Copied
Period of cycle which defines seasonality. This can be set to either one day or one week.
Mandatory: Yes
rules > obcervTimeSeriesTemplate > periods Copied
Number of periods to query.
If the periodicity has been set to one week, and periods is set to six, then the time series will be created using up to six weeks of historic data. If no data is available for a specific bucket, then the data returned from Obcerv for that bucket will be empty. Otherwise, Obcerv will build values with the available data.
Mandatory: Yes
rules > obcervTimeSeriesTemplate > bucketSize Copied
This setting enables the creation of time series buckets with a chosen size. It allows you to choose how long an interval each value in the time series represents. It can assume any of the following options:
- 5 mins
- 10 mins
- 15 mins
- 30 mins
- 1 hour
- 2 hours
- 3 hours
- 4 hours
- 6 hours
- 12 hours
Mandatory: Yes
rules > obcervTimeSeriesTemplate > reloadTime Copied
Designated time for the Gateway to reload data from Obcerv. The time is applied using the timezone of Gateway. Reloading is done once every day.
Mandatory: Yes
rules > obcervTimeSeriesTemplate > timezone Copied
Timezone of the data. This can either be Gateway, Netprobe, or a specific timezone.
Data will be queried from the previous midnight in the specified timezone, and the day and week periods will be associated with the end time.
Mandatory: No
Default: Gateway Timezone
Create a new rule that uses the Obcerv Time Series template Copied
Once a template has been defined, a rule can be created to use a path alias to reference the time series. When the rule is applied to a cell, the Gateway will construct a query and generate a new timeseries for the data referenced in the path alias.
A time series path alias is comprised of an xpath that appends the time series information to an xpath that references a single cell. So the following xpath ./timeSeries[(@obcervTimeSeriesTemplate="Daily")][(@aggregation="avg")]
will cause the Gateway to create an Obcerv time series and query the data from Obcerv. The query will include all the information in the obcervTimeSeriesTemplate together with the source data to use, which in this case is the current cell the rule will run on. Other xpaths can be used:
-
../cell[(@column="latency")]/timeSeries[(@obcervTimeSeriesTemplate="Daily")][(@Aggregation="max")]
- this gets the Obcerv Time Series that returns the maximum value of the data for the column called latency in the same row of the dataview. -
/geneos/gateway/directory/probe[(@name="P")]/managedEntity[(@name="me")]/sampler[(@name="s")]/dataview[(@name="dv")]/rows/row[(@name="r")]/cell[(@column="c")]/timeSeries[(@obcervTimeSeriesTemplate="Weelky")][(@Aggregation="count")]
- this gets the Obcerv Time Series that returns the number of metric values that are in a specific bucket for a specific cell.
To see the value, we can use Compute Engine to set the value of a cell. In that case, if we create a new column on the dataview next to the cell we are interested in, we can set the xpath to ../cell[(@column="Col1")]/timeSeries[(@template="Daily")][(@aggregation="avg")]
.
When cells matching rules referring to Obcerv Timeseries arrive, the Gateway will initiate queries to Obcerv, generating Time Series data that it will maintain up-to-date.
Visualise the time series data on a chart Copied
It is also possible to plot Obcerv time series data in a chart in a similar way that we plot a database time series.
When you right-click on a cell that is the source of time series data that has been queried by Gateway, Obcerv will add an Obcerv Time Series menu item to the context menu. This option will display all the time series queried by Gateway for that specific cell. You can then choose a destination, which can be either a new or existing chart, and then it will be presented with a dialog box.
This allows you to set the start time (Time from) and the end time (Time to) of the chart together with the choice of query result (Graph Type).
Note that it is permissible and often correct for the End Time to precede the Start Time. In this case, the graph will run from Start Time to the end of the day, and then to End Time.
The data is then queried and placed in an Active Chart. The chart will re-query its data if any of the following occur:
- Gateway connection drops and reconnects
- Template parameters change
- Time moves from before the start time to after the start time
The workspace saves enough data that if the AC restarts and connects to the Gateway, it will re-query the Obcerv data and reset the chart.