OP5 Monitor ["OP5 Monitor"]
["OP5 Monitor > Slim Poller"]["User Guide"]

Set up Slim Poller

Overview

The Slim Poller is a scaled-down version of the poller. For more information on its contents and limitations, see Slim Poller in Scalable monitoring.

To deploy a Slim Poller you need to have:

  • A good knowledge of Docker and the Docker ecosystem.
  • A working Docker environment or container orchestration tool.

Note: ITRS does not support Docker or any of its services, such as swarm clusters and container orchestration. For more information on how to build and manage Docker services, refer to the Docker and Kubernetes documentation.

Container overview

The Slim Poller consists of two containers, which include the following components:

  • slim-poller_naemon-core — contains Naemon, the MerlinNaemon Eventbroker Module and check plugins; it is responsible for executing all checks.
  • slim-poller_naemon-merlin — contains the Merlin daemon; the daemon communicates with the Eventbroker Module in the slim-poller_naemon-core container, and is responsible for sending all check results back to the master server.

The two containers have a one-to-one relationship. You cannot set up multiple slim-poller_naemon-core containers with one slim-poller_naemon-merlin container.

Before you begin

Before you add a Slim Poller to a master server, ensure the following:

  • Any peers are fully connected and synchronised. For guidance, see Check cluster state information in Scale up your monitoring environment.
  • The host groups which you want the Slim Poller to monitor exist on the master.
  • Folder /opt/monitor/.ssh exists on the master; if not, create it with the following commands:
    mkdir -p /opt/monitor/.ssh
    chown monitor /opt/monitor/.ssh
    chmod 700 /opt/monitor/.ssh				

Install SSH keys

The Slim Poller container must have SSH keys installed that are authorized at the master server. One way to achieve this is to create a new Docker image from the Slim Poller image.

Caution: The SSH key added in this image must also be added to /opt/monitor/.ssh/authorized_keys on the designated master server.

The following are example commands for installing the SSH:

Copy
FROM op5com/slim-poller_naemon-core:8.3.0

COPY --chown=monitor:root id_rsa /opt/monitor/.ssh/id_rsa
COPY --chown=monitor:root id_rsa.pub /opt/monitor/.ssh/authorized_keys

RUN chmod 600 /opt/monitor/.ssh/id_rsa
RUN chmod 644 /opt/monitor/.ssh/authorized_keys

Automatic setup

Beginning OP5 Monitor 8.3.x, it is possible to make use of the following Kubernetes features:

  • Autoscaling in Kubernetes. For more information, see Horizontal Pod Autoscaler.

  • kubectl command to manually increase the number of Slim Pollers running.

Setup for autoscaling

In order to achieve autoscaling, the following is set up:

  • A container entry script that connects to a designated master, and registers on the cluster to all relevant masters and peers.

  • The cluster_update module is set up to detect any changes to the cluster. A connection is established with the designated master, and the cluster configuration is updated.

  • Slim Pollers are identified by UUID on the master, but not towards peers.

  • The address of each Slim Poller must be an address that is reachable from within the Kubernetes cluster; for example, the pod IP. It is not necessary for this IP to be reachable from masters outside the Kubernetes cluster.

A number of environment variables are used to configure this, such as the master IP, poller hostgroup, and so on. For more information, see Setting environment variables.

Setting environment variables

For autoscaling to work correctly, you need to set up the following environment variables:

Environment variable Description
MASTER_ADDRESS The address of the designated master node.
MASTER_NAME Name of the master node.
MASTER_PORT

Merlin port of the master node.

By default, this is set to 15551.

POLLER_ADDRESS

The address that this poller should use. Use the Kubernetes pod IP.

POLLER_NAME Name of the poller. In autoscaling, this name is generated by Kubernetes.
POLLER_HOSTGROUPS

One or more hostgroups that the poller is responsible for. If there are multiple hostgroups, specify them in a comma-separated list.

These hostgroups must exist on the master server prior to container startup.

FILES_TO_SYNC Optional. Comma-separated list of paths to sync from the master server.
LOG_LEVEL

Set the log level of various setup scripts. This does not change the log output of Naemon and Merlin.

Mandatory: No

Possible values: debug, info, error, critical

Default value: info

   

The following example shows a YAML file with the environment variables configured for autoscaling in Kubernetes:

Copy
example-autoscaling.yaml
         env:
         - name: MASTER_ADDRESS
           value: "IP_ADDRESS_HERE"
         - name: MASTER_NAME
           value: "master"
         - name: MASTER_PORT
           value: "15551"
         - name: POLLER_ADDRESS
           valueFrom:
             fieldRef:
               fieldPath: status.podIP
         - name: POLLER_NAME
           valueFrom:
             fieldRef:
               fieldPath: metadata.name
         - name: POLLER_HOSTGROUPS
           value: "GROUP1, GROUP2"
         - name: FILES_TO_SYNC
           value: "/opt/plugins/custom/, /opt/plugins/check_nagios"

 

Scaling to a higher number of replicas

After starting a Slim Poller deployment, you can scale up the replicas manually by using the kubectl command:

Copy
kubectl scale deployment.v1.apps/op5-slim-poller --replicas=2

 

You can also use Kubernetes autoscaling. For more information, see Horizontal Pod Autoscaler.

Example Kubernetes deployment file

Copy
slim-poller-kubernetes.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: op5-slim-poller
  name: op5-slim-poller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: op5-slim-poller
      name: op5-slim-poller
  template:
    metadata:
      labels:
        app: op5-slim-poller
        name: op5-slim-poller
    spec:
      volumes:
        - name: ipc
          emptyDir: {}
        - name: merlin-conf
          emptyDir: {}
      terminationGracePeriodSeconds: 240
      containers:
       - name: naemon-core
         image: op5com/slim-poller_naemon-core:8.3.0
         volumeMounts:
          - mountPath: /var/run/naemon
            name: ipc
          - mountPath: /opt/monitor/op5/merlin
            name: merlin-conf
         livenessProbe:
           exec:
             command:
             - /usr/bin/mon
             - containerhealth
             - core
           initialDelaySeconds: 120
           periodSeconds: 30
           failureThreshold: 5
         env:
         - name: MASTER_ADDRESS
           value: "MASTER_IP"
         - name: MASTER_NAME
           value: "master"
         - name: MASTER_PORT
           value: "15551"
         - name: POLLER_ADDRESS
           valueFrom:
             fieldRef:
               fieldPath: status.podIP
         - name: POLLER_NAME
           valueFrom:
             fieldRef:
               fieldPath: metadata.name
         - name: POLLER_HOSTGROUPS
           value: "group1, group2"
         - name: FILES_TO_SYNC
           value: "/opt/plugins/custom/"
       - name: naemon-merlin
         image: op5com/slim-poller_naemon-merlin:8.3.0
         livenessProbe:
           exec:
             command:
             - /usr/bin/mon
             - containerhealth
             - merlin
           initialDelaySeconds: 20
           periodSeconds: 30
         ports:
          - containerPort: 15551
            name: op5-merlin
            protocol: TCP
         volumeMounts:
          - mountPath: /var/run/naemon
            name: ipc
          - mountPath: /opt/monitor/op5/merlin
            name: merlin-conf
      restartPolicy: Always

Manual setup

Set up volumes and containers

Volume overview

In order to create stateless containers, you need to have a few volumes set up, as summarised below. These volumes are also included in the deployment examples.

Volume Containers Mount point Persistence Required Description
ipc

slim-poller_naemon-core

slim-poller_naemon-merlin

/var/run/naemon/ Not required Yes Needed for sharing a Linux socket between the naemon-core and naemon-merlin containers.
merlin-conf

slim-poller_naemon-core

slim-poller_naemon-merlin

/opt/monitor/op5/merlin/ Required Yes The Merlin configuration files.
naemon-conf slim-poller_naemon-core /opt/monitor/etc/ Recommended No The Naemon configuration files. If no changes to the default configuration are required, you can omit this volume. Merlin will fetch the required object configuration from the master at startup.
status slim-poller_naemon-core /opt/monitor/var/status/ Required Yes Saves the Naemon status files, which contain state information such as comments, downtimes, and acknowledgements.

Docker Compose quick start

To get started with Docker Compose:

  1. Save the Docker Compose YAML configuration below into a file called docker-compose.yml in a new directory on your system.
  2. Run the following command to start the Slim Poller containers:
    docker-compose up

Deployment examples

Note that the following deployment examples have a specific version tag set on the Slim Poller container images. You must ensure you are setting the same version tag as the version of your OP5 Monitor master.

Click the name of each YAML file below to see its contents.

Closeddocker-compose.yml

Kubernetes/OpenShift

Closedslim-poller.yml

Closedmerlin-conf-pvc.yml

Closednaemon-conf-pvc.yml

Closedstatus-pvc.yml

Configure the master

In a load-balanced environment with peered masters, you must perform these steps on the master and all of its peers.

  1. Add the Slim Poller to the master using the following command, replacing the variables between angled brackets (<>) with your own values:
    mon node add <poller_name> type=poller hostgroup=<selected_hostgroups> connect=no notifies=no takeover=no address=<poller_IP>

    Note: If the master is on the same network as the Slim Poller and can monitor the same components as the Slim Poller, then you do not need to specify takeover=no.

    For example:

    mon node add poller type=poller hostgroup=pollergroup connect=no notifies=no takeover=no address=192.168.1.2
  2. Restart the master.
    mon restart

Success: You can now see file /var/cache/merlin/config/<POLLER_NAME>.cfg on your master server. If the file does not exist, the master cannot correctly generate the poller configuration.

Configure the Slim Poller

To configure the Slim Poller:

  1. Open a shell on the slim-poller_naemon-core container.
  2. Use the following convenience script to set up the master server on the Slim Poller, replacing the variables between angled brackets (<>) with your own values:
    setup.sh --master-name <master_name> --master-address <master_IP> --poller-name <poller_name>	

    If you prefer to set up the master server manually using mon commands, see Poller configuration.

Note: Beginning OP5 Monitor 8.3.x, you can enable node identification using UUID instead of using IP. For guidance, see UUID identification in Scale up your monitoring environment.

Plugin state retention

Some plugins save state between each invocation, such as check_by_snmp_cpu, which saves retention data at location /var/check_by_snmp_cpu. Other plugins, such as Plugin reference, might require a login file saved on the system.

For cases such as these, we recommend that you define a persistent storage volume in your deployment files. As a starting point for creating new persistent storage, you can use the status volume from the deployment examples in Deployment examples.