OP5 Monitor ["OP5 Monitor"]
["Technical Reference"]

Manage notifications

Overview

The procedures below describe how to perform some advanced notification configuration tasks. Only advanced users with a good knowledge of how OP5 Monitor notifications work should make the updates described.

Background information

This page describes advanced configuration options of notifications. For background information on notifications, see the following:

Notification process

OP5 Monitor determines whether it needs to send out notifications through the service check and host check logic. It sends out host and service notifications in the following instances:

  • When a hard state change occurs.
  • When a host or service remains in a hard non-OK state and the time specified by the configuration setting notification_interval in the host or service configuration has passed since the last notification was sent out for that specific host or service.
  • If the notification filters permit it. For more information, see Notification filters.

Note: If a host is in a soft or hard DOWN or UNREACHABLE state, the service notifications for services belonging to that host will be blocked.

Each host and service definition has a contact and contact_groups option that specifies which contacts receive notifications for that particular host or service.

When OP5 Monitor sends out a host or service notification, it notifies each individual contact or member of any contact groups specified in the contact_groups option of the service definition. If a contact is a member of more than one contact group, OP5 Monitor removes duplicate contact notifications before sending notifications out.

The default behaviour is that if a host has contacts configured in the host configuration, the specified contacts receive notifications for the host and the services on the host. There are exceptions to this default behaviour:

  • If a service on a host has different contacts set to the host, the contacts on the host receive all the notifications, except for the service that has contacts defined.
  • If a service in a host group has contacts set, only those specific contacts receive the notification.

The following notification options inherit their values from the host they are associated with, if you do not specify a value on the service:

  • contacts and contact_groups
  • check_period
  • notification_interval
  • notification_period

Notification filters

OP5 Monitor uses a number of filters to determine whether to send a notification or not. You can find more information about time periods and service, host, and contact settings in Manage templates and time periods.

Filter Description
Program-wide Whether notifications are enabled on a program-wide basis. Program-wide notification settings are managed in the Operations menu of Manage > Process information. For more details, see Program-wide management.
Service and host filters
  • Is the host or service in scheduled downtime?
  • Is the host or service in a flapping state?
  • Are the host or service notification options enabled for this type of notification?
  • Are we in the right time period for notifications at the moment?
  • Have we already sent a notification about this alert? Has the host or service remained in the same non-OK state as it was when the last notification went out?
Contact filters
  • Do the contact notifications options says that this type of notification is supposed to be sent?
  • Are the contact notification options enabled for this type of notification?

Naemon notification documentation

The notifications in OP5 Monitor follow an extensive rule set that is inherited from the core Naemon daemon. You can find more information about Naemon notifications in the Naemon notifications documentation.

Notification scripts

Notification commands use scripts in the same way as normal check commands. The information below describes the content of the scripts, which you can use for guidance when updating the scripts.

All default notification scripts delivered with OP5 Monitor are located in /opt/monitor/op5/notify.

Notification macros

Many of the arguments sent to the notification commands are macros. Macros are a type of variable containing, in most cases, a program-wide value. You can find more information about macros in the Naemon macro documentation.

One of the most important macros used with notifications is $NOTIFICATIONTYPE$. This macro tells you what type of notification is to be sent.

The $NOTIFICATIONTYPE$ macro can have one of the following values:

Notification Type Description
PROBLEM The service or host has just entered (or is still in) a problem state.
RECOVERY The service or host has recovered from a problem state.
ACKNOWLEDGEMENT A service or host in a problem state has been acknowledged by a user.
FLAPPINGSTART The host or service has entered a flapping state.
FLAPPINGSTOP The host or service has left a flapping state.
FLAPPINGDISABLED Host or service flapping detection has stopped and it has therefore left a flapping state.
DOWNTIMESTART The host or service has entered scheduled downtime.
DOWNTIMESTOP The host or service has left scheduled downtime.
DOWNTIMECANCELLED The scheduled downtime for a host or service has been cancelled.

Change notification email settings

Update the host name link in the notification email

The notification email includes a link back to the OP5 Monitor server that sent the notification. You can configure the email settings to link to a different host.

This is useful if OP5 Monitor is configured for load-balanced monitoring, where notifications can be sent out from different sources, but one of the peers is the preferred system for configuration and viewing data. This is the recommended way to configure load-balanced environments.

You can update the URL linking back to OP5 Monitor by updating file /etc/op5/notify.yml with a different hostname value. For example:

/etc/op5/notify.yml

...
# The notification email includes a link back to the OP5 Monitor server that
# sent the notification. You can configure the email settings to link to a
# different host.
# If not specified, the system hostname is used.
hostname: master1.op5.com
...

In the above example, a notification sent from master2.op5.com in a load-balanced configuration appears as follows:

op5 Monitor

Service CUSTOM detected 2016-07-03 16:36:11.
'Certificate Expiration Check' on host 'master01' has passed the CRITICAL threshold.
https://master01.op5.com/monitor/index.php/status/service/master01

Additional info;
CRITICAL - File /opt/plugins/custom/certificate-expire is 21770079 seconds old

Host:    master01
Address: 172.27.0.12
Alias:   OP5 Monitor Server
Status:  UP
Comment: /etc/op5/notify.yml configured on master2

The link in the notification email takes you to master1 to view details of the problem.

For more information on /etc/op5/notify.yml, see Default configuration file.

Update the sender email address in the notification email

By default, OP5 Monitor sends notifications from email address op5monitor without any domain. The mail transfer agent (MTA) adds the local domain name, which by default is @localhost.localdomain.

To change the email address that OP5 Monitor sends notifications from, you can use the --from argument in the notification command, or reconfigure your MTA and host name in OP5 Monitor to send the message from the correct domain.

To change the sender email address in the notification command from op5monitor@localhost.localdomain to op5notification@mycompany.com:

  1. In the OP5 Monitor user interface, click Manage > Configure.
  2. In the Checks and Notifications section, click Commands.
  3. In the Command to edit field, search for host-notify or select the command from the drop-down list, then click Go.
  4. In the command_line field, add --from op5notification@mycompany.com, as in the following example:
    Command nameCommand line
    host-notify
    $USER3$/notify/notify --from op5notification@mycompany.com -c "$CONTACTNAME$" -h "$HOSTNAME$" -f "$NOTIFICATIONTYPE$" -m "$CONTACTEMAIL$" -p "$CONTACTPAGER$" "HOSTALIAS=$HOSTALIAS$" HOSTADDRESS=$HOSTADDRESS$ " " HOSTSTATE=$HOSTSTATE$ " " HOSTSTATEID=$HOSTSTATEID$ " " HOSTSTATETYPE=$HOSTSTATETYPE$ " " HOSTATTEMPT=$HOSTATTEMPT$ " " HOSTLATENCY=$HOSTLATENCY$ " " HOSTEXECUTIONTIME=$HOSTEXECUTIONTIME$ " " HOSTDURATION=$HOSTDURATION$ " " HOSTDURATIONSEC=$HOSTDURATIONSEC$ " " HOSTDOWNTIME=$HOSTDOWNTIME$ " " HOSTPERCENTCHANGE=$HOSTPERCENTCHANGE$ " " HOSTGROUPNAME=$HOSTGROUPNAME$ " " HOSTGROUPALIAS=$HOSTGROUPALIAS$ " " LASTHOSTCHECK=$LASTHOSTCHECK$ " " LASTHOSTSTATECHANGE=$LASTHOSTSTATECHANGE$ " " LASTHOSTUP=$LASTHOSTUP$ " " LASTHOSTDOWN=$LASTHOSTDOWN$ " " LASTHOSTUNREACHABLE=$LASTHOSTUNREACHABLE$ " " HOSTOUTPUT=$HOSTOUTPUT$ " " HOSTPERFDATA=$HOSTPERFDATA$ " " HOSTACKAUTHOR=$HOSTACKAUTHOR$ " " HOSTACKCOMMENT=$HOSTACKCOMMENT$ " " NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$ " " CONTACTALIAS=$CONTACTALIAS$ " " DATETIME=$DATETIME$ " " SHORTDATETIME=$SHORTDATETIME$ "DATE=$DATE$" "TIME=$TIME$" "TIMET=$TIMET$" "HOSTACTIONURL=$HOSTACTIONURL$" "HOSTNOTESURL=$HOSTNOTESURL$" "ADMINPAGER=$ADMINPAGER$" "ADMINEMAIL=$ADMINEMAIL$" "NOTIFICATIONCOMMENT=$NOTIFICATIONCOMMENT$"
  5. Click Submit.
  6. Click Save at the top of the configuration page.

Note: To change this for service notifications, repeat the steps above for the command service-notify.

Change notification date/time format

Beginning OP5 Monitor 8.3.2, you can change the date/time format for notifications.

By default, notification time stamps follow the RFC 3339 format. You can change the date/time settings for notifications in /etc/op5/notify.yml. For more information, see Default configuration file.

Where no date/time format is defined, notifications will assume the YYYY-MM-DD HH:MM:SS format.

Options for date/time format

The following configuration settings are available for /etc/op5/notify.yml:

Option Description
timestamp_format

Sets the time format following strftime(3). For more information, see strftime(3) — Linux manual page.

The following presets are provided in the /etc/op5/notify.yml:

  • RFC 3339

    • %Y-%m-%d %H:%M:%S%z

    • Example output: 2021-05-25 14:16:26+0100

    • Default: Enabled

  • System locale preferred format

    • %c

    • Default: Disabled

  • US-style time stamp with time zone

    • %d/%m/%Y %I:%M:%S %p %Z

    • Example output: 25/05/2021 03:16:26 PM CET

    • Default: Disabled

utc_timestamp

If set to yes, then notifications will use the UTC time stamps instead of the local system time.

Default value: no

   

Naemon macros that use Unix time

There are a few Naemon macros that get reported in Unix time (time_t) by default. These are converted to your configured date/time format:

  • LASTHOSTCHECK

  • LASTHOSTSTATECHANGE

  • LASTHOSTUP

  • LASTHOSTDOWN

  • LASTHOSTUNREACHABLE

  • LASTSERVICECHECK

  • LASTSERVICESTATECHANGE

  • LASTSERVICEOK

  • LASTSERVICEWARNING

  • LASTSERVICEUNKNOWN

  • LASTSERVICECRITICAL

  • TIMET

  • PROCESSSTARTTIME

  • EVENTSTARTTIME

Caution: Macros not included in this list do not necessarily reflect Unix time. Before you convert a reported time stamp, verify the date/time format used for the macro.

Default configuration file

Beginning OP5 Monitor 8.3.2, the following configuration file is shipped with OP5 Monitor by default:

/etc/op5/notify.yml

---
# Configuration for OP5 Monitor notifications.
# Location: /etc/op5/notify.yml
# Lines starting with # are ignored.
 
# The notification email includes a link back to the OP5 Monitor server that
# sent the notification. You can configure the email settings to link to a
# different host.
# If not specified, the system hostname is used.
#hostname: "monitor-server-hostname.example.com"
 
 
# Format in notifications for timestamp macros like LASTHOSTCHECK, LASTSERVICESTATECHANGE etc.
# See strftime(3) for details of formatting (https://man7.org/linux/man-pages/man3/strftime.3.html).
 
# Internet Date/Time Format (rfc3339)
timestamp_format: "%Y-%m-%d %H:%M:%S%z"
 
# Use system locales preferred format:
#timestamp_format: "%c"
 
# US style timestamp, with timezone:
#timestamp_format: "%d/%m/%Y %I:%M:%S %p %Z"
 
 
# Set to yes to use UTC (Universal Coordinated Time/GMT) timestamps in
# notifications instead of systems local time.
utc_timestamps: no

If you have already defined /etc/opt/notify.yml in the past, then this default file does not affect your existing implementation. Instead, the package manager appends .rpmnew to the default /etc/op5/notify.yml being shipped with the updated OP5 Monitor package. In this case, you must manually merge your existing /etc/op5/notify.yml file with the contents of the new /etc/op5/notify.yml.rpmnew file to use the new date/time formats available to notifications.

Notification skins

The three basic notification types (email, SMS and HTML POST notifications) all use notification skins. The notification skins are templates describing how the notification is supposed to look.

The /notify directory includes the following skins directories:

  • skins.htmlpost/
  • skins.mail/
  • skins.sms/

Each directory contains a number of notification skins divided into host and service notification filters:

  • host.ACKNOWLEDGEMENT
  • host.FLAPPINGSTART
  • host.FLAPPINGSTOP
  • host.PROBLEM
  • host.RECOVERY
  • service.ACKNOWLEDGEMENT
  • service.FLAPPINGSTART
  • service.FLAPPINGSTOP
  • service.PROBLEM
  • service.RECOVERY

The SMS service.PROBLEM skin

The SMS service.PROBLEM skin is very simple, since there is a limit to how much data you can send by SMS:

#SERVICEDESC# on #HOSTNAME# is #SERVICESTATE#. #SERVICEOUTPUT#

The mail service.PROBLEM skin

Mail notifications can contain a lot more data, so the mail skin is more complex:

From: op5Monitor To: #CONTACTEMAIL# Subject: [op5]
#NOTIFICATIONTYPE#: '#SERVICEDESC#' on '#HOSTNAME#' is
#SERVICESTATE#
#extra_host_vars#
op5 Monitor
Service #NOTIFICATIONTYPE# detected #LASTSERVICESTATECHANGE#.
'#SERVICEDESC#' on host '#HOSTNAME#' has passed the #SERVICESTATE#
threshold.
#STATUS_URL
#Additional info;
#SERVICEOUTPUT
#Host: #HOSTNAME# Address: #HOSTADDRESS# Alias: #HOSTALIAS# Status: 
#HOSTSTATE# Comment: #NOTIFICATIONCOMMENT#
Service: #SERVICEDESC# Status : #SERVICESTATE# Latency: Check was 
#SERVICELATENCY# seconds behind schedule Misc : Check took 
#SERVICEEXECUTIONTIME# seconds to complete
Additional links (requires configuration);Host actions: #HOSTACTIONURL# Host notes: #HOSTNOTESURL#Service 
actions: #SERVICEACTIONURL# Service notes: #SERVICENOTESURL#

Keywords

Skins contain keywords in the following format:

  • #SERVICEDESC#
  • #HOSTNAME#

Keywords are replaced with the value of a command line argument in the format X=Y. So a command line such as this generates a keyword with the name X having the value Y.

If a notification macro, or other object supplying a keyword value, is missing from the notification command, it does not stop the notification from being sent; only the replacement value will be missing.

Create a custom notification skin

If you need to change the default notification skins, you need to create them in a different directory.

To create custom notification skins:

  1. From the OP5 Monitor command line, change to the /notify directory:
    cd /opt/monitor/op5/notify
  2. Create a directory called custom-skins:
    mkdir custom-skins
  3. Copy the skins.* directories to the custom-skins directory:
    cp -R skins.* custom-skins/
  4. Make your changes to the skins. The new skins are used directly after you save the changes.

Note: If you have a peered configuration, you need to make these changes on all peers.

Configure SNMP trap notifications

You can send OP5 Monitor notifications as SNMP traps. To configure SNMP notifications, you need to perform the following:
  • Add SNMP notification commands.
  • Configure contacts.

Add SNMP notification commands

You need to add one command for host notifications and one for service notifications.

  1. In the OP5 Monitor user interface, click Manage > Configure.
  2. In the Checks and Notifications section, click Commands.
  3. Add two new commands with the following arguments, changing snmp.trap.host and SNMPCOMMUNITY to their correct values in both commands.
    Command nameCommand line
    host_notify_by_snmp
    $USER3$/notify/notify_by_snmp.pl -h snmp.trap.host -C SNMPCOMMUNITY -t nHostNotify "NOTIFICATIONTYPE=$NOTIFICATIONTYPE$" "NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$" "HOSTACKAUTHOR=$HOSTACKAUTHOR$" "HOSTACKCOMMENT=$HOSTACKCOMMENT$" "HOSTNAME=$HOSTNAME$" "HOSTSTATEID=$HOSTSTATEID$" "HOSTSTATETYPE=$HOSTSTATETYPE$" "HOSTATTEMPT=$HOSTATTEMPT$" "HOSTDURATIONSEC=$HOSTDURATIONSEC$" "HOSTGROUPNAME=$HOSTGROUPNAME$" "LASTHOSTCHECK=$LASTHOSTCHECK$" "LASTHOSTSTATECHANGE=$LASTHOSTSTATECHANGE$" "HOSTOUTPUT=$HOSTOUTPUT$"
    service_notify_by_snmp
    $USER3$/notify/notify_by_snmp.pl -h snmp.trap.host -C SNMPCOMMUNITY -t nSvcNotify "NOTIFICATIONTYPE=$NOTIFICATIONTYPE$" "NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$" "SERVICEACKAUTHOR=$SERVICEACKAUTHOR$" "SERVICEACKCOMMENT=$SERVICEACKCOMMENT$" "HOSTNAME=$HOSTNAME$" "HOSTSTATEID=$HOSTSTATEID$" "SERVICEDESCRIPTION=$SERVICEDESCRIPTION$" "SERVICESTATEID=$SERVICESTATEID$" "SERVICEATTEMPT=$SERVICEATTEMPT$" "SERVICEDURATIONSEC=$SERVICEDURATIONSEC$" "SERVICEGROUPNAME=$SERVICEGROUPNAME$" "LASTSERVICECHECK=$LASTSERVICECHECK$" "LASTSERVICESTATECHANGE=$LASTSERVICESTATECHANGE$" "SERVICEOUTPUT=$SERVICEOUTPUT$"

  4. Click Submit.
  5. Click Save at the top of the configuration page.

Configure contacts

To configure the notification contacts:

  1. In the OP5 Monitor user interface, click Manage > Configure.
  2. In the Checks and Notifications section, click Contacts.
  3. Either open an existing contact or create a new one.
  4. From the host_notification_commands drop-down list, select host_notify_by_snmp.
  5. From the service_notification_commands drop-down list, select service_notify_by_snmp.
  6. Set contactgroups to a contact group associated with the required monitoring objects.
  7. Click Submit.
  8. Click Save at the top of the configuration page.