Scale up your monitoring environment

Check cluster state information

When you add new servers to scale up an existing environment, it is important to check that the environment is stable. You can check the state of the current cluster by using OP5 Monitor's integrated command-line back-end tool mon, as follows:

mon node status

OP5 Monitor displays all known nodes, including the local node, peers, and pollers, with their current state. A properly synchronised and online cluster displays all nodes as active. It displays any problems in red.

For more information on the mon tool, see Mon command reference.

Prerequisites

In any distributed or load-balanced monitoring environment, all servers must have:

• The same operating system version, with the same 32 or 64 bit architecture.
• The same version of OP5 Monitor installed.

Before you begin

Before you add a new poller or peer to your environment, you must ensure that you:

• Have suitable servers installed with the same architecture and OP5 Monitor version as the other servers in the environment, with the following configuration:
  • The following TCP ports open on the poller nodes, to allow master nodes to successfully communicate with poller nodes:
    • 22 (SSH) — used for distributing configuration from master to poller nodes.
    • 15551 (Merlin) — used for state communication, such as check results.
  • All server names resolvable by DNS, or manually using /etc/hosts.
  • All server system clocks synchronised, preferably by NTP. For more information, see Install NTP and synchronise servers in Additional server and software setup.
  • All other mandatory and recommended configuration completed on the hosts. For more information, see Additional server and software setup.
• Create all host groups for which each poller and Slim Poller will be responsible on the master, containing at least one host, with at least one contact and one service. For more information, see Manage hosts and services.

Tip: A simple way to set up servers with the same server configuration as other servers in the same environment is to clone them and subsequently update the host details.

Restrict shell access to nodes

When logged in to the server, an unprivileged user can read sensitive files, or even read and write to databases used by OP5 Monitor. It is also possible for unprivileged users to affect the monitoring process directly, thereby circumventing any access control by the OP5 Monitor software.

Check that only users with full administrative privileges have shell access (for example, via SSH) to ITRS OP5 Monitor nodes.

Caution: Granting user permissions for Test this host and Test this service effectively grants shell access to the monitoring host. For more information, see Update user permissions in Manage users, contacts, and permissions.

You can set up peering between two nodes for load-balancing or redundancy. Note that ITRS only supports peering between nodes in the same location, and preferably on the same rack.

Caution: Do not duplicate host names. If you are merging two previously independent OP5 Monitor nodes as peers, they will by default have themselves listed as an object called monitor. Before starting this configuration, give both of these monitor objects more descriptive names. If you fail to do this, SNMPv3 monitoring will break, as one key pair will disappear when merging these identical objects. If you encounter this issue, you must resolve it by creating a new SNMP user. For guidance, see Configure an SNMPv3 user in Additional server and software setup.

Add a new peer to another OP5 Monitor server

In this procedure we will set up a load-balanced monitoring environment with two peered nodes.

Note that in the command line examples below, in scenarios where you are converting an already running standalone server to a load balanced setup:

• peer01 is your existing OP5 Monitor server.
• peer02 is the new peer.

Caution: It is essential to get this right to avoid pushing the new peer's empty host and service object configuration to the existing server and overwriting your configuration. If in doubt, please contact ITRS Support.

Add a new peer to an existing load-balanced setup

In the command line examples below:

• peer01 and peer02 are your existing peers.
• peer03 is the new peer.

1. Log on to the new peer as root, using SSH.
2. Add the previously existing peers to the new peer:

   mon node add peer01 type=peer
   mon node add peer02 type=peer

3. Set up SSH connectivity towards all the new peer's configured peers:

   mon sshkey push --type=peer
   asmonitor mon sshkey push --type=peer

4. Add the new peer to all other nodes:

   mon node ctrl --type=peer mon node add peer03 type=peer

5. Log on to the first existing peer in as root, using SSH, and set up the SSH connectivity towards all its configured peers, including the new peer:

   mon sshkey push --type=peer
   asmonitor mon sshkey push --type=peer

6. Log on to the second existing peer as root, using SSH, and set up the SSH connectivity towards all its configured peers, including the new peer:

   mon sshkey push --type=peer
   asmonitor mon sshkey push --type=peer

7. On either existing peer, push the server configuration to the new peer:

   asmonitor mon oconf push peer03

8. On any of the three peers, trigger a full OP5 Monitor restart on all nodes:
   

   mon node ctrl --self -- mon restart

9. After a few minutes, check that the peers are fully connected and synchronised. For guidance, see Check cluster state information.

Before you begin

Before you run the master server configuration commands, ensure that any peers are fully connected and synchronised. For guidance, see Check cluster state information.

Configure the master

In a load-balanced environment with peered masters, you must perform these steps on the master and all of its peers.

1. Log in as root, using SSH.
2. Verify that the host group exists and print its current host members:

   mon query ls hostgroups -c members name=se-gbg

   Caution: Assigning a non-existent host group to a poller will cause OP5 Monitor to crash.

3. Add the new poller to the configuration:

   mon node add poller01 type=poller hostgroup=se-gbg takeover=no

   Note: Before you continue, ensure that the master can resolve the poller name by DNS or manually using /etc/hosts.

4. Set up SSH connectivity between the master and the poller:
   1. Confirm that OpenSSH's client and server parts are installed:

      rpm -qa | grep openssh

   2. If they are not installed, install them:

      yum install openssh
      

   3. Edit file /etc/ssh/sshd_config with a text editor, and check that PasswordAuthentication is set to yes:

      edit /etc/ssh/sshd_config

   4. After you have saved your changes, push the SSH configuration to the poller:

      mon sshkey push poller01 
      asmonitor mon sshkey push poller01

5. (Optional) For increased security, ITRS recommends creating the public SSH key for the OP5 Monitor user and placing it within the authorized_keys file. This enables OP5 Monitor to securely exchange configuration data without first requiring root keys to be exchanged. To do this, log on to both the master and the poller as root, using SSH, and run these commands:

   mkdir -p /opt/monitor/.ssh
   chown monitor:root /opt/monitor/.ssh
   su monitor
   ssh-keygen -t rsa
   scp ~/.ssh/id_rsa.pub root@poller.company.com:

   mkdir -p /opt/monitor/.ssh
   chown monitor:root /opt/monitor/.ssh
   mv ~/id_rsa.pub /opt/monitor/.ssh
   su monitor
   touch ~/.ssh/authorized_keys
   cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
   more ~/.ssh/authorized_keys
   rm ~/id_rsa.pub

6. On the master, add the master to the poller's configuration:

   mon node ctrl poller01 mon node add master01 type=master connect=no

Push the configuration

In a load-balanced environment with peered masters, perform these steps on the master server only.

1. Restart Naemon on the master:

   systemctl restart naemon

2. Push the configuration from the master to the new poller:

   asmonitor mon oconf push poller01

3. Restart OP5 Monitor on the new poller:

   mon node ctrl poller01 mon restart

4. Restart OP5 Monitor on the master and all its peers:

   mon node ctrl --self --type=peer mon restart

Before you begin

Before you add a Slim Poller to a master server, ensure the following:

• Any peers are fully connected and synchronised. For guidance, see Check cluster state information.
• The host groups which you want the Slim Poller to monitor exist on the master.
• Folder /opt/monitor/.ssh exists on the master; if not, create it with the following commands:

  mkdir -p /opt/monitor/.ssh
  chown monitor /opt/monitor/.ssh
  chmod 700 /opt/monitor/.ssh				

Set up volumes and containers

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
version: "3.1"
services:
  naemon-core:
    depends_on:
      - naemon-merlin
    volumes:
      - ipc:/var/run/naemon
      - merlin-conf:/opt/monitor/op5/merlin
      - naemon-conf:/opt/monitor/etc
      - ssh-conf:/opt/monitor/.ssh
      - status:/opt/monitor/var/status/
    image: op5com/slim-poller_naemon-core:8.1.0
  naemon-merlin:
    volumes:
      - ipc:/var/run/naemon
      - merlin-conf:/opt/monitor/op5/merlin
    image: op5com/slim-poller_naemon-merlin:8.1.0
volumes:
  ipc:
  merlin-conf:
  naemon-conf:
  ssh-conf:
  status:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
apiVersion: apps/v1
kind: Deployment
metadata:
  name: op5-slim-poller
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: op5-slim-poller
    template:
      metadata:
        labels:
          app: op5-slim-poller
      spec:
        volumes:
        - name: ipc
          emptyDir: {}
        - name: merlin-conf
          persistentVolumeClaim:
            claimName: merlin-conf-pvc
        - name: naemon-conf
          persistentVolumeClaim:
            claimName: naemon-conf-pvc
        - name: ssh-conf
          persistentVolumeClaim:
            claimName: ssh-conf-pvc
        - name: status
          persistentVolumeClaim:
            claimName: status-pvc
        containers:
        - image: op5com/slim-poller_naemon-core:8.1.0
          name: naemon-core
          resources: {}
          volumeMounts:
          - name: ipc
            mountPath: /var/run/naemon
          - name: merlin-conf
            mountPath: /opt/monitor/op5/merlin
          - name: naemon-conf
            mountPath: /opt/monitor/etc
          - name: ssh-conf
            mountPath: /opt/monitor/.ssh
          - name: status
            mountPath: /opt/monitor/var/status
        - image: op5com/slim-poller_naemon-merlin:8.0.10
          name: naemon-merlin
          resources: {}
          volumeMounts:
          - name: ipc
            mountPath: /var/run/naemon
          - name: merlin-conf
            mountPath: /opt/monitor/op5/merlin
        restartPolicy: Always
status: {}

1
2
3
4
5
6
7
8
9
10
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: merlin-conf-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Mi

1
2
3
4
5
6
7
8
9
10
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: naemon-conf-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

1
2
3
4
5
6
7
8
9
10
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ssh-conf-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Mi

1
2
3
4
5
6
7
8
9
10
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: status-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 150Mi

Configure the master

In a load-balanced environment with peered masters, you must perform these steps on the master and all of its peers.

1. Add the Slim Poller to the master using the following command, replacing the variables between angled brackets (<>) with your own values:

   mon node add <poller_name> type=poller hostgroup=<selected_hostgroups> connect=no notifies=no takeover=no address=<poller_IP>

   Note: If the master is on the same network as the Slim Poller and can monitor the same components as the Slim Poller, then you do not need to specify takeover=no.

   For example:

   mon node add poller type=poller hostgroup=pollergroup connect=no notifies=no takeover=no address=192.168.1.2

2. Restart the master.

   mon restart

3. If your monitor user needs a password to sync SSH keys from the Slim Poller, create it with the following command:

   passwd monitor

   After the SSH keys are synced, you can delete the password with the following command:

   passwd --delete monitor

Success: You can now see file /var/cache/merlin/config/<POLLER_NAME>.cfg on your master server. If the file does not exist, the master cannot correctly generate the poller configuration.

Configure the Slim Poller

To configure the Slim Poller:

1. Open a shell on the slim-poller_naemon-core container.
2. Sync the SSH keys with the master server with the following command, replacing the variable between angled brackets (<>) with your own values:

   mon sshkey push <master_IP>	

3. Use the following convenience script to set up the master server on the Slim Poller, replacing the variables between angled brackets (<>) with your own values:

   setup.sh --master-name <master_name> --master-address <master_IP> --poller-name <poller_name>	

   If you prefer to set up the master server manually using mon commands, see Poller configuration.

Plugin state retention

Some plugins save state between each invocation, such as check_by_snmp_cpu, which saves retention data at location /var/check_by_snmp_cpu. Other plugins, such as Plugin reference, might require a login file saved on the system.

For cases such as these, we recommend that you define a persistent storage volume in your deployment files. As a starting point for creating new persistent storage, you can use the status volume from the deployment examples in Deployment examples.

Before you begin

Before you run the master server configuration commands, ensure that:

• All peers (if applicable) are fully connected and synchronised. For more information, see Check cluster state information.
• The host group you are adding to the poller already exists.

Configure the master server

In a load-balanced environment with peered masters, you must perform these steps on the master and all of its peers.

1. Log on as root, using SSH.
2. Edit file /opt/monitor/op5/merlin/merlin.conf using a text editor. For example:

   edit /opt/monitor/op5/merlin/merlin.conf

3. Find the configuration block related to the poller and append the new host group to the hostgroup setting value, prefixed by a comma. For example, to add a host group called op5-hg2, change the line from:

   hostgroup = op5-hg1

   to:

   hostgroup = op5-hg1,op5-hg2

   Caution: Host groups must be comma-separated only, without any white space. If you fail to do this, error Incompatible object config (sync triggered) may occur during Naemon restart. Remember that adding a non-existent host group will also cause OP5 Monitor to crash.

4. After you have saved your changes, restart OP5 Monitor:

   mon restart

Configure the master

In a load-balanced environment with peered masters, perform these steps on the master and all of its peers.

1. Log on to the master as root, using SSH.
2. Edit file /opt/monitor/op5/merlin/merlin.conf using a text editor. For example:

   edit /opt/monitor/op5/merlin/merlin.conf

3. Find the configuration block related to the poller and insert the option notifies = no at the end of the block:

poller poller01 {
 address = 192.0.2.50
 port = 15551
 takeover = no
 notifies = no
}

4. After you have saved your changes, restart OP5 Monitor:

mon restart

Configure the poller

1. Log on to the poller as root, using SSH.
2. Edit the file /opt/monitor/op5/merlin/merlin.conf using a text editor. For example:

   edit /opt/monitor/op5/merlin/merlin.conf

3. Find the module configuration block and insert the option notifies = no at the end of the block:

module {
  log_file = /var/log/op5/merlin/neb.log;
  notifies = no
}

4. After you have saved your changes, restart OP5 Monitor:

   mon restart

OP5 Monitor has limited support for synchronising files between peers and between masters and pollers.

For example, when you add a new user to OP5 Monitor on one of your masters, you can use this feature to automatically synchronise the user database files on all other peers and pollers.

asmonitor mon oconf push

Configure synchronisation

For file synchronisation between peers, you need to repeat the procedure on all peers.

In this example:

• The master will synchronise files to its poller, poller01.
• The following files will be synchronised:
  • /etc/op5/auth_users.yml
  • /etc/op5/auth_groups.yml
• The contents of the following directory will also be synchronised:
  • /opt/plugins/custom/

To configure synchronisation:

1. Log on to the source node (master) as root, using SSH.
2. Edit file /opt/monitor/op5/merlin/merlin.conf using a text editor:

   edit /opt/monitor/op5/merlin/merlin.conf 

3. Find the configuration block related to the destination node, in this case poller01. Within this block, insert a new sync sub-block, saving your changes when you are done.

   poller poller01 {
    hostgroup = se-gbg 
    address = 192.0.2.50
    port = 15551
    takeover = no
    sync {
    /etc/op5/auth_users.yml
    /etc/op5/auth_groups.yml
    /opt/plugins/custom/
    }
   }

   Note: The trailing slash at the end of /opt/plugins/custom/ in the example above indicates that the contents of the directory must be synchronised, rather than the directory itself. This is the recommended way of synchronising directories.