Notifications

The Notifications app closes the loop between data collection, signal generation, and alerting. When the condition of your IT estate needs attention, the app conveniently notifies you through its integration with external systems such as Slack.

You can configure notifications in two ways: either for groups of entities or for individual entities. Grouped notifications are enabled if there is at least one grouping in the configuration.

Grouped notification lifecycle

Grouped notifications are formed by bundling Obcerv entities that have common characteristics. For example, if you group by container, it will create a group per Kubernetes container that grows or shrinks seamlessly (as entities such as pods or volumes match or cease to match the grouping parameters).

Grouped notifications are triggered when at least one entity in the group exceeds the configured warning or critical trigger interval. Once a notification is triggered, reminders are periodically sent following the reminder trigger interval.

The notification is cleared when all entities in the group exceed the clear trigger interval and no additional entities in the group have been triggered.

Entity notification lifecycle

Entity notifications are triggered when an individual entity exceeds the configured warning or critical trigger interval. Once a notification is sent, reminders are periodically sent following the reminder trigger interval. The notification is cleared when the entity exceeds the clear trigger interval.

Configuration

To create a new configuration, follow these steps:

1. From the Web Console, select Alerting > Notifications.
2. Click Add New.

3. Specify the Name for the new configuration.
4. On the Conditions field, provide conditions to include or exclude entities from being taken into consideration when evaluating notifications for individual entities or groups.
5. On the Group by field, filter entities with common attributes or dimensions that will be grouped together. A single notification will be sent for each group.

A group is an implicit filter so any entities that do not have the group attributes or dimensions will be excluded. For example:

Given the above entities and grouping configuration OS, the following groups will be created with these entities:

Only one notification per group will be sent and entity Dev_None will be disregarded since it doesn’t have the group attribute OS.

5. Click Add Target, and then choose a connection type. Provide which settings to use. For more information, see Triggers and Messages. When done, click OK.
6. Click Save to create the new configuration.

Triggers

A notification is triggered and cleared based on an entity’s severity.

Warning / Critical (triggered)

If an entity has had a critical or warning severity for longer than the supplied duration, then it is considered triggered and a notification is sent, either individually for the entity or as part of a group (if groups are configured). Where a severity is rapidly changing between warning and critical states, the warning trigger also considers entities with critical severity.

Cleared

If an entity has been previously triggered and has had no severity for the supplied duration, then it is considered cleared and a notification is sent indicating that the entity is healthy.

Reminders

Reminder notifications apply to triggered entities and groups, and are periodically sent as a reminder that the entity/group is still in a triggered state.

Messages

Notification messages must be configured for each enabled trigger. It consists of the main message body and an optional short title. The title’s usage depends on the integration. For example, for Slack notifications, it is used as a header and as the text of push notifications. It is not used for Webhooks.

Messages may contain placeholders of the form ${placeholder} that will be interpolated by the app. The available placeholders can be seen by clicking .

Some placeholders are only available for grouped notifications while others are only available for entity notifications. The supported placeholders can be seen below.

Illustrative example

Consider the above timeline of an entity’s severity where the numbers represent minutes, and assuming that the configuration of all triggers is set to 1 minute.

Integrations

The app integrates with the following third-party systems:

• Slack
• Webhook

Instrumentation

The app leverages an in-house StatsD client to record metrics regarding its internal state. The following is a complete list of metrics collected by the app:

Notification storage and retrieval

Notifications are currently recorded in logs and can be retrieved from the Logs screen of the Web Console.

From the Logs screen, set the dimension filter in the From field by inputting the string {container="obcerv-app-notifications-notifier"}|logfmt followed by additional parameters for the information you would like to extract.

Examples

All triggered notifications that have been sent:

{container="obcerv-app-notifications-notifier"}|logfmt|state="SENT"|type="TRIGGERED"

All notifications that were attempted to be sent but failed:

{container="obcerv-app-notifications-notifier"}|logfmt|state="FAILED"

All triggered notifications for the configuration called “Obcerv License”:

{container="obcerv-app-notifications-notifier"}|logfmt|state="SENT"|type="TRIGGERED"|notificationName="Obcerv License"

The following filtering tags are available: