Notifications

The Notifications app closes the loop between data collection, signal generation, and alerting. When the condition of your IT estate needs attention, the app conveniently notifies you through its integration with external systems such as Slack.

You can configure notifications in two ways: either for groups of entities or for individual entities. Grouped notifications are enabled if there is at least one grouping in the configuration.

Grouped notification lifecycle Copied

Grouped notifications are formed by bundling Obcerv entities that have common characteristics. For example, if you group by container, it will create a group per Kubernetes container that grows or shrinks seamlessly (as entities such as pods or volumes match or cease to match the grouping parameters).

Grouped notifications are triggered when at least one entity in the group exceeds the configured warning or critical trigger interval. Once a notification is triggered, reminders are periodically sent following the reminder trigger interval.

The notification is cleared when all entities in the group exceed the clear trigger interval and no additional entities in the group have been triggered.

Entity notification lifecycle Copied

Entity notifications are triggered when an individual entity exceeds the configured warning or critical trigger interval. Once a notification is sent, reminders are periodically sent following the reminder trigger interval. The notification is cleared when the entity exceeds the clear trigger interval.

Configuration Copied

To create a new configuration, follow these steps:

1. From the Web Console, select Notifications.
2. Click Add Notification.

3. Specify the Name for the new configuration. You can also add a description.
4. On the Filter field, set whether to include or exclude entities when evaluating notifications for individual entities or groups.
5. On the Group by field, select entities with common attributes or dimensions that will be grouped together. A single notification will be sent for each group.

A group is an implicit filter so any entities that do not have the group attributes or dimensions will be excluded. For example:

Given the above entities and grouping configuration OS, the following groups will be created with these entities:

Only one notification per group will be sent and entity Dev_None will be disregarded since it doesn’t have the group attribute OS.

6. Under Targets, select a target type, and then choose an option from the Targets drop-down list. Depending on the target type, you can set more options. For example, if you selected SLACK, you will see the Reply in Slack thread toggle.

Note

If you need to create a new target, go to Notifications > Targets, and then click beside the target type. For more information about the available target types, see Integrations.

7. Check the options under the Triggers and Messages section. For more information, see Triggers and Messages.
8. Click Save to add the new configuration.

Triggers Copied

A notification is triggered and cleared based on an entity’s severity.

Warning / Critical (triggered) Copied

If an entity has had a critical or warning severity for longer than the supplied duration, then it is considered triggered and a notification is sent, either individually for the entity or as part of a group (if groups are configured). Where a severity is rapidly changing between warning and critical states, the warning trigger also considers entities with critical severity.

Cleared Copied

If an entity has been previously triggered and has had no severity for the supplied duration, then it is considered cleared and a notification is sent indicating that the entity is healthy.

Reminders Copied

Reminder notifications apply to triggered entities and groups, and are periodically sent as a reminder that the entity/group is still in a triggered state.

Messages Copied

Notification messages must be configured for each enabled trigger. It consists of the main message body and an optional short title. The title’s usage depends on the integration. For example, for Slack notifications, it is used as a header and as the text of push notifications. It is not used for Webhooks.

Messages may contain placeholders of the form ${placeholder} that will be interpolated by the app. The available placeholders can be seen by clicking .

Some placeholders are only available for grouped notifications while others are only available for entity notifications. The supported placeholders can be seen below.

Illustrative example Copied

Consider the above timeline of an entity’s severity where the numbers represent minutes, and assuming that the configuration of all triggers is set to 1 minute.

Integrations Copied

The app integrates with the following third-party systems:

• Slack
• Webhook
• ServiceNow

Instrumentation Copied

The app leverages an in-house StatsD client to record metrics regarding its internal state. The following is a complete list of metrics collected by the app:

Notification storage and retrieval Copied

Notifications are currently recorded in logs and can be retrieved from the Logs screen of the Web Console.

From the Logs screen, set the dimension filter in the From field by inputting the string {container="obcerv-app-notifications-notifier"}|logfmt followed by additional filtering parameters for the information you would like to extract.

Notification examples Copied

All triggered notifications that have been sent:

All notifications that were attempted to be sent but failed:

All triggered notifications for the configuration called “Obcerv License”:

The following filtering tags are available:

Audit log storage and retrieval Copied

Audit logs can be retrieved from the Logs screen of the Web Console.

From the Logs screen, set the dimension filter in the From field by inputting the string {container="obcerv-app-notifications"}|logfmt|class="AUDIT" followed by additional filtering parameters for the information you would like to extract.

Audit log examples Copied

All notifications that have been created, updated, or deleted by user djohn:

All targets that were deleted by user admin:

Notifications or targets that were updated by any user:

The following filtering tags are available: