Manage notifications

Overview

The procedures below describe how to perform some advanced notification configuration tasks. Only advanced users with a good knowledge of how OP5 Monitor notifications work should make the updates described.

Background information

This page describes advanced configuration options of notifications. For background information on notifications, see the following:

Notification process

OP5 Monitor determines whether it needs to send out notifications through the service check and host check logic. It sends out host and service notifications in the following instances:

  • When a hard state change occurs.
  • When a host or service remains in a hard non-OK state and the time specified by the configuration setting notification_interval in the host or service configuration has passed since the last notification was sent out for that specific host or service.
  • If the notification filters permit it. For more information, see Notification filters.

Each host and service definition has a contact and contact_groups option that specifies which contacts receive notifications for that particular host or service.

When OP5 Monitor sends out a host or service notification, it notifies each individual contact or member of any contact groups specified in the contact_groups option of the service definition. If a contact is a member of more than one contact group, OP5 Monitor removes duplicate contact notifications before sending notifications out.

The default behaviour is that if a host has contacts configured in the host configuration, the specified contacts receive notifications for the host and the services on the host. There are exceptions to this default behaviour:

  • If a service on a host has different contacts set to the host, the contacts on the host receive all the notifications, except for the service that has contacts defined.
  • If a service in a host group has contacts set, only those specific contacts receive the notification.

The following notification options inherit their values from the host they are associated with, if you do not specify a value on the service:

  • contacts and contact_groups
  • check_period
  • notification_interval
  • notification_period

Notification filters

OP5 Monitor uses a number of filters to determine whether to send a notification or not. You can find more information about time periods and service, host, and contact settings in Create or update a host, service, or contact template.

Filter Description
Program-wide Whether notifications are enabled on a program-wide basis. Program-wide notification settings are managed in the Operations menu of Manage > Process information. For more details, see Program-wide management.
Service and host filters
  • Is the host or service in scheduled downtime?
  • Is the host or service in a flapping state?
  • Are the host or service notification options enabled for this type of notification?
  • Are we in the right time period for notifications at the moment?
  • Have we already sent a notification about this alert? Has the host or service remained in the same non-OK state as it was when the last notification went out?
Contact filters
  • Do the contact notifications options says that this type of notification is supposed to be sent?
  • Are the contact notification options enabled for this type of notification?

Naemon notification documentation

The notifications in OP5 Monitor follow an extensive rule set that is inherited from the core Naemon daemon. You can find more information about Naemon notifications in the Naemon notifications documentation.

Notification scripts

Notification commands use scripts in the same way as normal check commands. The information below describes the content of the scripts, which you can use for guidance when updating the scripts.

All default notification scripts delivered with OP5 Monitor are located in /opt/monitor/op5/notify.

Notification macros

Many of the arguments sent to the notification commands are macros. Macros are a type of variable containing, in most cases, a program-wide value. You can find more information about macros in the Naemon macro documentation.

One of the most important macros used with notifications is $NOTIFICATIONTYPE$. This macro tells you what type of notification is to be sent.

The $NOTIFICATIONTYPE$ macro can have one of the following values:

Notification Type Description
PROBLEM The service or host has just entered (or is still in) a problem state.
RECOVERY The service or host has recovered from a problem state.
ACKNOWLEDGEMENT A service or host in a problem state has been acknowledged by a user.
FLAPPINGSTART The host or service has entered a flapping state.
FLAPPINGSTOP The host or service has left a flapping state.
FLAPPINGDISABLED Host or service flapping detection has stopped and it has therefore left a flapping state.
DOWNTIMESTART The host or service has entered scheduled downtime.
DOWNTIMESTOP The host or service has left scheduled downtime.
DOWNTIMECANCELLED The scheduled downtime for a host or service has been cancelled.

Change notification email settings

Update the host name link in the notification email

The notification email includes a link back to the OP5 Monitor server that sent the notification. You can configure the email settings to link to a different host.

This is useful if OP5 Monitor is configured for load-balanced monitoring, where notifications can be sent out from different sources, but one of the peers is the preferred system for configuration and viewing data. This is the recommended way to configure load-balanced environments.

You can update the URL linking back to OP5 Monitor by updating file /etc/op5/notify.yml with a different hostname value:

hostname: master1.op5.com

For example, a notification sent from master2.op5.com in a load-balanced configuration appears as follows:

op5 Monitor

Service CUSTOM detected 2016-07-03 16:36:11.
'Certificate Expiration Check' on host 'master01' has passed the CRITICAL threshold.
https://master01.op5.com/monitor/index.php/status/service/master01

Additional info;
CRITICAL - File /opt/plugins/custom/certificate-expire is 21770079 seconds old

Host:    master01
Address: 172.27.0.12
Alias:   OP5 Monitor Server
Status:  UP
Comment: /etc/op5/notify.yml configured on master2

The link in the notification email takes you to master1 to view details of the problem.

Update the sender email address in the notification email

By default, OP5 Monitor sends notifications from email address op5monitor without any domain. The mail transfer agent (MTA) adds the local domain name, which by default is @localhost.localdomain.

To change the email address that OP5 Monitor sends notifications from, you can use the --from argument in the notification command, or reconfigure your MTA and host name in OP5 Monitor to send the message from the correct domain.

To change the sender email address in the notification command from op5monitor@localhost.localdomain to op5notification@mycompany.com:

  1. In the OP5 Monitor user interface, click Manage > Configure.
  2. In the Checks and Notifications section, click Commands.
  3. In the Command to edit field, search for host-notify or select the command from the drop-down list, then click Go.
  4. In the command_line field, add --from op5notification@mycompany.com, as in the following example:
    Command nameCommand line
    host-notify
    $USER3$/notify/notify --from op5notification@mycompany.com -c "$CONTACTNAME$" -h "$HOSTNAME$" -f "$NOTIFICATIONTYPE$" -m "$CONTACTEMAIL$" -p "$CONTACTPAGER$" "HOSTALIAS=$HOSTALIAS$" HOSTADDRESS=$HOSTADDRESS$ " " HOSTSTATE=$HOSTSTATE$ " " HOSTSTATEID=$HOSTSTATEID$ " " HOSTSTATETYPE=$HOSTSTATETYPE$ " " HOSTATTEMPT=$HOSTATTEMPT$ " " HOSTLATENCY=$HOSTLATENCY$ " " HOSTEXECUTIONTIME=$HOSTEXECUTIONTIME$ " " HOSTDURATION=$HOSTDURATION$ " " HOSTDURATIONSEC=$HOSTDURATIONSEC$ " " HOSTDOWNTIME=$HOSTDOWNTIME$ " " HOSTPERCENTCHANGE=$HOSTPERCENTCHANGE$ " " HOSTGROUPNAME=$HOSTGROUPNAME$ " " HOSTGROUPALIAS=$HOSTGROUPALIAS$ " " LASTHOSTCHECK=$LASTHOSTCHECK$ " " LASTHOSTSTATECHANGE=$LASTHOSTSTATECHANGE$ " " LASTHOSTUP=$LASTHOSTUP$ " " LASTHOSTDOWN=$LASTHOSTDOWN$ " " LASTHOSTUNREACHABLE=$LASTHOSTUNREACHABLE$ " " HOSTOUTPUT=$HOSTOUTPUT$ " " HOSTPERFDATA=$HOSTPERFDATA$ " " HOSTACKAUTHOR=$HOSTACKAUTHOR$ " " HOSTACKCOMMENT=$HOSTACKCOMMENT$ " " NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$ " " CONTACTALIAS=$CONTACTALIAS$ " " DATETIME=$DATETIME$ " " SHORTDATETIME=$SHORTDATETIME$ "DATE=$DATE$" "TIME=$TIME$" "TIMET=$TIMET$" "HOSTACTIONURL=$HOSTACTIONURL$" "HOSTNOTESURL=$HOSTNOTESURL$" "ADMINPAGER=$ADMINPAGER$" "ADMINEMAIL=$ADMINEMAIL$" "NOTIFICATIONCOMMENT=$NOTIFICATIONCOMMENT$"
  5. Click Submit.
  6. Click Save at the top of the configuration page.

Note: To change this for service notifications, repeat the steps above for the command service-notify.

Notification skins

The three basic notification types (email, SMS and HTML POST notifications) all use notification skins. The notification skins are templates describing how the notification is supposed to look.

The /notify directory includes the following skins directories:

  • skins.htmlpost/
  • skins.mail/
  • skins.sms/

Each directory contains a number of notification skins divided into host and service notification filters:

  • host.ACKNOWLEDGEMENT
  • host.FLAPPINGSTART
  • host.FLAPPINGSTOP
  • host.PROBLEM
  • host.RECOVERY
  • service.ACKNOWLEDGEMENT
  • service.FLAPPINGSTART
  • service.FLAPPINGSTOP
  • service.PROBLEM
  • service.RECOVERY

The SMS service.PROBLEM skin

The SMS service.PROBLEM skin is very simple, since there is a limit to how much data you can send by SMS:

#SERVICEDESC# on #HOSTNAME# is #SERVICESTATE#. #SERVICEOUTPUT#

The mail service.PROBLEM skin

Mail notifications can contain a lot more data, so the mail skin is more complex:

From: op5Monitor To: #CONTACTEMAIL# Subject: [op5]
#NOTIFICATIONTYPE#: '#SERVICEDESC#' on '#HOSTNAME#' is
#SERVICESTATE#
#extra_host_vars#
op5 Monitor
Service #NOTIFICATIONTYPE# detected #LASTSERVICESTATECHANGE#.
'#SERVICEDESC#' on host '#HOSTNAME#' has passed the #SERVICESTATE#
threshold.
#STATUS_URL
#Additional info;
#SERVICEOUTPUT
#Host: #HOSTNAME# Address: #HOSTADDRESS# Alias: #HOSTALIAS# Status: 
#HOSTSTATE# Comment: #NOTIFICATIONCOMMENT#
Service: #SERVICEDESC# Status : #SERVICESTATE# Latency: Check was 
#SERVICELATENCY# seconds behind schedule Misc : Check took 
#SERVICEEXECUTIONTIME# seconds to complete
Additional links (requires configuration);Host actions: #HOSTACTIONURL# Host notes: #HOSTNOTESURL#Service 
actions: #SERVICEACTIONURL# Service notes: #SERVICENOTESURL#

Keywords

Skins contain keywords in the following format:

  • #SERVICEDESC#
  • #HOSTNAME#

Keywords are replaced with the value of a command line argument in the format X=Y. So a command line such as this generates a keyword with the name X having the value Y.

If a notification macro, or other object supplying a keyword value, is missing from the notification command, it does not stop the notification from being sent; only the replacement value will be missing.

Create a custom notification skin

If you need to change the default notification skins, you need to create them in a different directory.

To create custom notification skins:

  1. From the OP5 Monitor command line, change to the /notify directory:
    cd /opt/monitor/op5/notify
  2. Create a directory called custom-skins:
    mkdir custom-skins
  3. Copy the skins.* directories to the custom-skins directory:
    cp -R skins.* custom-skins/
  4. Make your changes to the skins. The new skins are used directly after you save the changes.

Note: If you have a peered configuration, you need to make these changes on all peers.

Configure SNMP trap notifications

You can send OP5 Monitor notifications as SNMP traps. To configure SNMP notifications, you need to perform the following:
  • Add SNMP notification commands.
  • Configure contacts.

Add SNMP notification commands

You need to add one command for host notifications and one for service notifications.

  1. In the OP5 Monitor user interface, click Manage > Configure.
  2. In the Checks and Notifications section, click Commands.
  3. Add two new commands with the following arguments, changing snmp.trap.host and SNMPCOMMUNITY to their correct values in both commands.
    Command nameCommand line
    host_notify_by_snmp
    $USER3$/notify/notify_by_snmp.pl -h snmp.trap.host -C SNMPCOMMUNITY -t nHostNotify "NOTIFICATIONTYPE=$NOTIFICATIONTYPE$" "NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$" "HOSTACKAUTHOR=$HOSTACKAUTHOR$" "HOSTACKCOMMENT=$HOSTACKCOMMENT$" "HOSTNAME=$HOSTNAME$" "HOSTSTATEID=$HOSTSTATEID$" "HOSTSTATETYPE=$HOSTSTATETYPE$" "HOSTATTEMPT=$HOSTATTEMPT$" "HOSTDURATIONSEC=$HOSTDURATIONSEC$" "HOSTGROUPNAME=$HOSTGROUPNAME$" "LASTHOSTCHECK=$LASTHOSTCHECK$" "LASTHOSTSTATECHANGE=$LASTHOSTSTATECHANGE$" "HOSTOUTPUT=$HOSTOUTPUT$"
    service_notify_by_snmp
    $USER3$/notify/notify_by_snmp.pl -h snmp.trap.host -C SNMPCOMMUNITY -t nSvcNotify "NOTIFICATIONTYPE=$NOTIFICATIONTYPE$" "NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$" "SERVICEACKAUTHOR=$SERVICEACKAUTHOR$" "SERVICEACKCOMMENT=$SERVICEACKCOMMENT$" "HOSTNAME=$HOSTNAME$" "HOSTSTATEID=$HOSTSTATEID$" "SERVICEDESCRIPTION=$SERVICEDESCRIPTION$" "SERVICESTATEID=$SERVICESTATEID$" "SERVICEATTEMPT=$SERVICEATTEMPT$" "SERVICEDURATIONSEC=$SERVICEDURATIONSEC$" "SERVICEGROUPNAME=$SERVICEGROUPNAME$" "LASTSERVICECHECK=$LASTSERVICECHECK$" "LASTSERVICESTATECHANGE=$LASTSERVICESTATECHANGE$" "SERVICEOUTPUT=$SERVICEOUTPUT$"

  4. Click Submit.
  5. Click Save at the top of the configuration page.

Configure contacts

To configure the notification contacts:

  1. In the OP5 Monitor user interface, click Manage > Configure.
  2. In the Checks and Notifications section, click Contacts.
  3. Either open an existing contact or create a new one.
  4. From the host_notification_commands drop-down list, select host_notify_by_snmp.
  5. From the service_notification_commands drop-down list, select service_notify_by_snmp.
  6. Set contactgroups to a contact group associated with the required monitoring objects.
  7. Click Submit.
  8. Click Save at the top of the configuration page.