Manage notifications

Background information

This page describes advanced configuration options of notifications. For background information on notifications, see the following:

• For a general description of how notifications work in OP5 Monitor, see Checks, alerts, and notifications in Key concepts of OP5 Monitor.
• For descriptions of the notification options you can set on hosts and services, see Host options in Manage templates and time periods and Service options in Manage templates and time periods.

Notification process

OP5 Monitor determines whether it needs to send out notifications through the service check and host check logic. It sends out host and service notifications in the following instances:

• When a hard state change occurs.
• When a host or service remains in a hard non-OK state and the time specified by the configuration setting notification_interval in the host or service configuration has passed since the last notification was sent out for that specific host or service.
• If the notification filters permit it. For more information, see Notification filters.

Note: If a host is in a soft or hard DOWN or UNREACHABLE state, the service notifications for services belonging to that host will be blocked.

Each host and service definition has a contact and contact_groups option that specifies which contacts receive notifications for that particular host or service.

When OP5 Monitor sends out a host or service notification, it notifies each individual contact or member of any contact groups specified in the contact_groups option of the service definition. If a contact is a member of more than one contact group, OP5 Monitor removes duplicate contact notifications before sending notifications out.

The default behaviour is that if a host has contacts configured in the host configuration, the specified contacts receive notifications for the host and the services on the host. There are exceptions to this default behaviour:

• If a service on a host has different contacts set to the host, the contacts on the host receive all the notifications, except for the service that has contacts defined.
• If a service in a host group has contacts set, only those specific contacts receive the notification.

The following notification options inherit their values from the host they are associated with, if you do not specify a value on the service:

• contacts and contact_groups
• check_period
• notification_interval
• notification_period

Notification filters

OP5 Monitor uses a number of filters to determine whether to send a notification or not. You can find more information about time periods and service, host, and contact settings in Manage templates and time periods.

Naemon notification documentation

The notifications in OP5 Monitor follow an extensive rule set that is inherited from the core Naemon daemon. You can find more information about Naemon notifications in the Naemon notifications documentation.

Update the host name link in the notification email

The notification email includes a link back to the OP5 Monitor server that sent the notification. You can configure the email settings to link to a different host.

This is useful if OP5 Monitor is configured for load-balanced monitoring, where notifications can be sent out from different sources, but one of the peers is the preferred system for configuration and viewing data. This is the recommended way to configure load-balanced environments.

You can update the URL linking back to OP5 Monitor by updating file /etc/op5/notify.yml with a different hostname value:

hostname: master1.op5.com

For example, a notification sent from master2.op5.com in a load-balanced configuration appears as follows:

op5 Monitor

Service CUSTOM detected  2016-07-03  16:36:11.
'Certificate Expiration Check'  on host  'master01'  has passed the CRITICAL threshold.
https://master01.op5.com/monitor/index.php/status/service/master01

Additional info;
CRITICAL - File /opt/plugins/custom/certificate-expire is  21770079  seconds old

Host:    master01
Address:  172.27.0.12
Alias:   OP5 Monitor Server
Status:  UP
Comment: /etc/op5/notify.yml configured on master2

The link in the notification email takes you to master1 to view details of the problem.

Update the sender email address in the notification email

By default, OP5 Monitor sends notifications from email address op5monitor without any domain. The mail transfer agent (MTA) adds the local domain name, which by default is @localhost.localdomain.

To change the email address that OP5 Monitor sends notifications from, you can use the --from argument in the notification command, or reconfigure your MTA and host name in OP5 Monitor to send the message from the correct domain.

To change the sender email address in the notification command from op5monitor@localhost.localdomain to op5notification@mycompany.com:

1. In the OP5 Monitor user interface, click Manage > Configure.
2. In the Checks and Notifications section, click Commands.
3. In the Command to edit field, search for host-notify or select the command from the drop-down list, then click Go.
4. In the command_line field, add --from op5notification@mycompany.com, as in the following example:

   Command name

   Command line

   host-notify

   $USER3$/notify/notify --from op5notification@mycompany.com -c "$CONTACTNAME$" -h "$HOSTNAME$" -f "$NOTIFICATIONTYPE$" -m "$CONTACTEMAIL$" -p "$CONTACTPAGER$" "HOSTALIAS=$HOSTALIAS$" HOSTADDRESS=$HOSTADDRESS$ " " HOSTSTATE=$HOSTSTATE$ " " HOSTSTATEID=$HOSTSTATEID$ " " HOSTSTATETYPE=$HOSTSTATETYPE$ " " HOSTATTEMPT=$HOSTATTEMPT$ " " HOSTLATENCY=$HOSTLATENCY$ " " HOSTEXECUTIONTIME=$HOSTEXECUTIONTIME$ " " HOSTDURATION=$HOSTDURATION$ " " HOSTDURATIONSEC=$HOSTDURATIONSEC$ " " HOSTDOWNTIME=$HOSTDOWNTIME$ " " HOSTPERCENTCHANGE=$HOSTPERCENTCHANGE$ " " HOSTGROUPNAME=$HOSTGROUPNAME$ " " HOSTGROUPALIAS=$HOSTGROUPALIAS$ " " LASTHOSTCHECK=$LASTHOSTCHECK$ " " LASTHOSTSTATECHANGE=$LASTHOSTSTATECHANGE$ " " LASTHOSTUP=$LASTHOSTUP$ " " LASTHOSTDOWN=$LASTHOSTDOWN$ " " LASTHOSTUNREACHABLE=$LASTHOSTUNREACHABLE$ " " HOSTOUTPUT=$HOSTOUTPUT$ " " HOSTPERFDATA=$HOSTPERFDATA$ " " HOSTACKAUTHOR=$HOSTACKAUTHOR$ " " HOSTACKCOMMENT=$HOSTACKCOMMENT$ " " NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$ " " CONTACTALIAS=$CONTACTALIAS$ " " DATETIME=$DATETIME$ " " SHORTDATETIME=$SHORTDATETIME$ "DATE=$DATE$" "TIME=$TIME$" "TIMET=$TIMET$" "HOSTACTIONURL=$HOSTACTIONURL$" "HOSTNOTESURL=$HOSTNOTESURL$" "ADMINPAGER=$ADMINPAGER$" "ADMINEMAIL=$ADMINEMAIL$" "NOTIFICATIONCOMMENT=$NOTIFICATIONCOMMENT$"

5. Click Submit.
6. Click Save at the top of the configuration page.

Note: To change this for service notifications, repeat the steps above for the command service-notify.

The SMS service.PROBLEM skin

The SMS service.PROBLEM skin is very simple, since there is a limit to how much data you can send by SMS:

#SERVICEDESC# on #HOSTNAME# is #SERVICESTATE#. #SERVICEOUTPUT#

The mail service.PROBLEM skin

Mail notifications can contain a lot more data, so the mail skin is more complex:

From: op5Monitor To: #CONTACTEMAIL# Subject: [op5]
#NOTIFICATIONTYPE#: '#SERVICEDESC#' on '#HOSTNAME#' is
#SERVICESTATE#
#extra_host_vars#
op5 Monitor
Service #NOTIFICATIONTYPE# detected #LASTSERVICESTATECHANGE#.
'#SERVICEDESC#' on host '#HOSTNAME#' has passed the #SERVICESTATE#
threshold.
#STATUS_URL
#Additional info;
#SERVICEOUTPUT
#Host: #HOSTNAME# Address: #HOSTADDRESS# Alias: #HOSTALIAS# Status: 
#HOSTSTATE# Comment: #NOTIFICATIONCOMMENT#
Service: #SERVICEDESC# Status : #SERVICESTATE# Latency: Check was 
#SERVICELATENCY# seconds behind schedule Misc : Check took 
#SERVICEEXECUTIONTIME# seconds to complete
Additional links (requires configuration);Host actions: #HOSTACTIONURL# Host notes: #HOSTNOTESURL#Service 
actions: #SERVICEACTIONURL# Service notes: #SERVICENOTESURL#

Keywords

Skins contain keywords in the following format:

• #SERVICEDESC#
• #HOSTNAME#

Keywords are replaced with the value of a command line argument in the format X=Y. So a command line such as this generates a keyword with the name X having the value Y.

If a notification macro, or other object supplying a keyword value, is missing from the notification command, it does not stop the notification from being sent; only the replacement value will be missing.

Create a custom notification skin

If you need to change the default notification skins, you need to create them in a different directory.

To create custom notification skins:

1. From the OP5 Monitor command line, change to the /notify directory:

   cd /opt/monitor/op5/notify

2. Create a directory called custom-skins:

   mkdir custom-skins

3. Copy the skins.* directories to the custom-skins directory:

   cp -R skins.* custom-skins/

4. Make your changes to the skins. The new skins are used directly after you save the changes.

Note: If you have a peered configuration, you need to make these changes on all peers.