Troubleshooting

This page describes how to investigate Opsview Monitor problems.

Opsview Components Copied

The first place you should go to when you are encountering issues with Opsview Monitor is the Configuration > My System > System Overview tab. If there are problems with a component, or some missing components, this page will highlight them.

System overview

Diagnostics Copied

On the Monitoring Engine tab there is also a button labeled Download Now, which will download a .tar.gz file with all the relevant syslogs, and others:

This diagnostic information will be useful when you raise issues with our support team.

Web user interface is not working Copied

If the web interface is the problem then you will not be able to access these web pages. If this is the case, run the command /opt/opsview/watchdog/bin/opsview-monit summary as the root user, as seen below:

# /opt/opsview/watchdog/bin/opsview-monit summary
Monit 5.24.0 uptime: 4d 5h 34m
┌─────────────────────────────────┬────────────────────────────┬───────────────┐
│ Service Name                    │ Status                     │ Type          │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ tvoon-centos7-test              │ OK                         │ System        │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ opsview-watchdog                │ Running                    │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ opsview-web                     │ Running                    │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ opsview-timeseriesrrdupdates    │ Running                    │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ opsview-timeseriesrrdqueries    │ Running                    │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ opsview-timeseriesenqueuer      │ Running                    │ Process       │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
[snipped]

If you have any issues with the processes above, for example, opsview-web, then you can restart an individual process with the command:

# /opt/opsview/watchdog/bin/opsview-monit restart opsview-web

You should also check to make sure your database is up, running and accessible.

You can also view detailed information about each process by running the following command as the root user:

# /opt/opsview/watchdog/bin/opsview-monit status opsview-web
The Monit daemon 5.14 uptime: 4d 6h 54m
Process 'opsview-web'  status                            Running
monitoring status                 Monitored  
pid                               3451  
parent pid                        1  
uid                               999  
effective uid                     999  
gid                               998  
uptime                            3h 21m  
children                          3  
memory                            250.8 MB  
memory total                      1.0 GB  
memory percent                    5.0%  
memory percent total              20.7%  
cpu percent                       0.0%  
cpu percent total                 0.0%  
data collected                    Tue, 15 Sep 2015 16:16:39

If you encounter the error:

# /opt/opsview/watchdog/bin/opsview-monit summary
Monit: the monit daemon is not running

Then the opsview-watchdog service is not running. To start it, as root, run the commands:

# pkill -u opsview
# systemctl start opsview-watchdog
# /opt/opsview/watchdog/bin/opsview-monit start all

This will kill any leftover processes, start the daemon, and then restart all the services which the watchdog is controlling.

If the watchdog process starts but the processes it is monitoring do not, check your sudo configuration (using the command visudo) does not have Defaults requiretty enabled. If it does, disable it (by commenting it out with a ‘#’ character) and rerun the following:

# /opt/opsview/watchdog/bin/opsview-monit validate

Finally, if your watchdog services start (as per the ‘summary’ command), but suddenly shutdown after a minute, you may not have enough free disk space. Opsview Monitor requires a MINIMUM of 2GB free space. If this threshold is breached, Opsview Monitor will elegantly shutdown instead of crashing and leaving the system in a problematic state when the disk space issue is resolved.

To confirm you are encountering the disk space issue, run the command:

# cat /var/log/syslog | grep "resource limit"
Sep 15 12:06:47 ov-author opsview-monit[4362]: 'rootfs' space free 1.7 GB matches resource limit [space free>2.0 GB]
Sep 15 12:06:47 ov-author opsview-monit[4362]: 'varfs' space free 1.7 GB matches resource limit [space free>2.0 GB]
Sep 15 12:06:47 ov-author opsview-monit[4362]: 'optfs' space free 1.7 GB matches resource limit [space free>2.0 GB]

If you see these errors, you should check your free disk space using df:

# df -h
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/ovauthorvg-rootlv  9.3G  7.6G  1.2G  87% /
none                           4.0K     0  4.0K   0% /sys/fs/cgroup
udev                           2.5G  4.0K  2.5G   1% /dev
tmpfs                          497M  352K  496M   1% /run
none                           5.0M     0  5.0M   0% /run/lock
none                           2.5G     0  2.5G   0% /run/shm
none                           100M     0  100M   0% /run/user
/dev/mapper/ovauthorvg-bootlv  233M   38M  179M  18% /boot

Logs Copied

Logs are always a good place to start when it comes to troubleshooting. See the Logging section for information about how logs are generated.

Databases Copied

We have seen issues where a database has a bad schema and indexes are given the wrong name. This causes problems for the upgrade scripts as they expect specific names to exist when upgrading.

Follow this process to reset the schema while retaining the existing data. You should not normally have to do this.

  1. Stop all Opsview Components
  2. Take a backup of the opsview database: /opt/opsview/coreutils/bin/db_opsview db_backup > /tmp/opsview.db
  3. Take another backup, for comparing differences: mysqldump -u {user} -p{password} --default-character-set=utf8mb4 --skip-extended-insert opsview | sed 's/character_set_client = utf8 /character_set_client = utf8mb4 /' > /tmp/opsview.diff
  4. Export just data from the database: mysqldump --default-character-set=utf8mb4 --skip-extended-insert -t -c -u {user} -p{password} opsview | sed 's/character_set_client = utf8 /character_set_client = utf8mb4 /' > /tmp/opsview.data
  5. Create the database from scratch: /opt/opsview/coreutils/bin/db_opsview db_install
  6. Export the schema information from a fresh install: mysqldump -d -u {user} -p{password} --default-character-set=utf8mb4 opsview | sed 's/character_set_client = utf8 /character_set_client = utf8mb4 /' > /tmp/opsview.schema
  7. Delete and recreate just the database: echo 'drop database opsview; create database opsview' | mysql -u {user} -p{password}
  8. Import the fresh schema information: mysql -u {user} -p{password} opsview < /tmp/opsview.schema
  9. Import the data: mysql -u {user} -p{password} opsview < /tmp/opsview.data
  10. Take a new backup: /opt/opsview/coreutils/bin/db_opsview db_backup > /tmp/opsview_post.db
  11. Take another backup, for comparing: mysqldump -u {user} -p{password} --default-character-set=utf8mb4 --skip-extended-insert opsview | sed 's/character_set_client = utf8 /character_set_client = utf8mb4 /' > /tmp/opsview2.diff
  12. Compare to check differences: diff -u /tmp/opsview.diff /tmp/opsview2.diff
  13. Start all Opsview Components.

CentOS/RHEL/OL — Automatic dependencies Copied

yum should automatically resolve all dependencies when installing Opsview Monitor. However, in some instances during installation, if the Opsview Monitor packages don’t include opsview-base, opsview-perl and so on, then ensure yum-updatesd-helper is not running and execute the following commands:

# yum remove opsview
# yum clean all
# yum makecache

Finally, running the command shown below should show the correct dependencies and allow Opsview Monitor to install correctly:

# yum deplist opsview
# yum install opsview

Access denied for some files within the repositories Copied

If you replicate our public repository to a server on your own network, you may find you get Access Denied errors when trying to copy some files.

This is expected behavior as some files within the repository are restricted to customers that purchase additional modules.

["Opsview On-premises"] ["User Guide"]

Was this topic helpful?