Signals
You can view triggered signals and its history generated by the Forecaster app from the Alerting app.
Warning / Critical (triggered)
The Forecaster app can trigger either of these signals:
-
Forecast signal — when the projected value of an observed metric is greater than or equal to (>=) the specified threshold within the specified horizon. If both warning and critical signal condition are satisfied, a critical signal is generated. Forecast signals are labeled with the
forecastcategory, which can be viewed from the Alerting app. -
Static signal — when the current value of an observed metric is above or equal to (>=) the specified threshold.
Note
The static and forecast signals are independent of each other. The same entity can have both forecast and static signals triggered at the same time.
Cleared
A forecast signal is cleared if it was previously triggered and the projected forecast within the specified duration is below the threshold. A static signal is cleared if it was previous triggered and the current status is now below the specified threshold. If a config is removed, signals triggered from entities that match the removed config will also be cleared.
Illustrative example
Refer to the following thresholds as an example:
- Warning: 80% in 2 hours
- Critical: 90% in 1 hour
The example illustrates the lifecycle of signals triggered on an entity by the Forecaster app.
Instrumentation
The Forecaster app leverages an in-house StatsD client to record metrics regarding its internal state. The following is a complete list of metrics collected by the app:
| Metric | Type | Description |
|---|---|---|
| Total Processed | gauge | The total number of metrics processed in the last minute. |
| Total Excluded | gauge | The total number of metrics excluded for forecasting in the last minute. |
| Total Forecasts | gauge | The total number of metrics forecasted in the last minute. |
| Forecast Signals Generated | gauge | The total number of forecast signals generated in the last minute. |
| Static Signals Generated | gauge | The total number of static signals generated in the last minute. |
| Preprocessing Time | duration | The duration of preprocessing. |
| Forecast Time | duration | The duration of forecasts. |