Manage logging

Overview

OP5 Monitor can act as a centralised syslog logging server for event correlation, pattern-based alerts, and audit purposes. This page describes how to configure clients for remote logging and configure the built-in Logger functionality in OP5 Monitor. It also includes details of logging files and logging levels.

Licence

If you want to use Logger for syslog logging, you need to purchase a Logger licence from ITRS. Without a licence, you can still use it for logging on your local server.

Configure remote logging on your host servers

You can set up any system with remote syslog capabilities to send its logs to your OP5 Monitor host for storage analysis. If your system is not covered in the sections below, refer to your vendor documentation for more information on configuring syslog.

Configure remote logging on Windows

The Windows syslog agent runs as a Windows service. It sends the Windows event log content to the IP address or fully qualified domain name (FQDN) of your OP5 Monitor system. It can also send plain text log files for applications that keep their own logs.

Install and configure the Windows syslog agent

Install and configure the agent as follows:

Download the Windows syslog agent from the ITRS downloads page.
Double-click the downloaded MSI file and follow the on-screen instructions to install it.
Open the newly installed SyslogAgentConfig on your Windows server.
In the Syslog server field, enter your OP5 Monitor IP address.
Check the Enable forwarding of event logs checkbox.
Configure any additional options, as required:
- Check Enable mirror delivery and specify the IP address in Mirror Syslog server to send logs to this additional IP address.
- Change the ports from the default 514 UDP port in the Syslog server and Mirror Syslog server fields. You can specify different ports for each server.
- Select UDP transport after ping to ping the syslog server before sending the logs.
- Check the options to enabling forwarding of event logs, application logs, or both.
- In Filter out these EventIDs, enter a comma-separated list of event IDs to filter out before sending. You can specify a maximum of 30 IDs.
Click Start Service.

For more information on the Windows syslog agent, click the Help button in the application.

Export the configuration to another machine

You can export the configuration from the Windows Registry to a .reg file, so that the settings can be pushed out as a group policy or automated for scripting.

Note: If you do not delete the LastRun key as detailed below, the configuration may not work on other computers.

To export the configuration:

Type regedit in the Windows Start menu or command prompt.
Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Datagram\SyslogAgent.
Right-click on the SyslogAgent folder and select Export.
Save the file in a location of your choice.
Right-click on the file and choose Edit.
If there is a LastRun line, remove the whole line and save the file.\

Configure remote logging on Unix

Most Unix and Linux systems have built-in logging capability, so you do not need to install any extra software. The main types of standard logging software on Unix systems are:

rsyslog
syslog-ng
syslogd

To configure your logging software to work with Logger, you need to update the specific configuration file for the logging software with details of your OP5 Monitor host. The files are as follows:

rsyslog — /etc/rsyslog.conf
syslogd — /etc/syslog.conf
syslog-ng — /etc/syslog-ng.conf

For guidance on configuring syslog, refer to the online documentation for your logging software or your system manual. For example:

man syslog.conf

Send text files to Logger

Some applications do not send their logs to syslog but store them in a file on disk instead. If it is not possible to configure an application to use syslog, you can use one of the following workarounds:

Define a log source in the configuration file. For example, in the case of syslog-ng, you can add a new log section with that source in the messages from the kernel section.
Use tail and logger to read the log file and send appended lines to syslog. For example, the following command reads /var/log/myapp.log and sends it to syslog as facility daemon and severity info:
```
# tail -f /var/log/myapp.log | logger -p daemon.info
```

You need to configure your system to execute the command on reboot, by adding the command to the relevant file, for example /etc/rc.local.

Configure Logger in OP5 Monitor

You can use the Logger configuration page to configure the following:

The database retention period.
Log archiving.

Database retention period

OP5 Monitor comes with a default database retention period of 192 hours, which is 8 days. This is designed to be sufficiently short to not require too much disk space, while keeping enough data to allow you to evaluate storage requirements and determine suitable retention times. OP5 Monitor self-monitoring is enabled during installation, and warns you if the server is running out of disk space.

Log archive

You can use a log archive for long-term log storage. The archive does not have a retention time.

In the default configuration archiving is disabled, to avoid filling up disk space on the OP5 Monitor server. Configuring remote storage for the archive is recommended, for reasons of security, hardware failure, and storage.

Notes:
Archived files are not rotated automatically, so logs are stored indefinitely. You must put in place a retention and clean-up policy to clear out old log files.
If archiving is turned off, existing archive files are kept on disk.

Searching the archive

You can search the archive using the Run query on archive feature from any log message list view:

It is important to note the following about searching the log archive:

The archive search feature searches on text files instead of the database.
Archive logging only starts logging messages from the moment archive logging is turned on, so the archive search feature can only return results from that period.

You can also generate a report of archive files. For more information, see Log message archive report.

Configure Logger

To configure Logger:

In the OP5 Monitor user interface, click Manage > Configure
Click Logger configuration.
Specify the number of hours for database retention in the Keep in database for field.
Check the Archive logs as text files checkbox and specify an absolute Storage path on the OP5 Monitor server.

Add a reference to the log file from the command line

By default, OP5 Monitor includes a reference to the log file in log messages for some of its modules. You can enable it for other modules by setting Reference to True in configuration files in /etc/op5.

Warning: This is a costly operation that you must use with extreme caution.

For more information on which files include a reference by default, see the Default reference column of the table in the section below.

Log files

Log files by feature and function

The tables below give details of log files for specific OP5 Monitor features and functions, and the type of information they log. For more details of logging levels available for each log configuration file, see Logging levels.

Note: Paths are only for guidance, and may differ by operating system.

Feature or function	Module	Log configuration file	Default log file path	Default debug level	Default reference	Content
Authorisation and authentication	Auth	`/etc/op5/log.yml`	`/var/log/op5/auth.log`	Error	True	PHP errors
Business services	Synergy	`/etc/op5/log.yml` `/opt/synergy/etc/config.lua`	`/var/log/op5/synergy.log` See Syslog	Error See Syslog	True See Syslog	PHP errors Only on/off configuration available, everything else managed by syslog-ng
Configuration	Nacoma	`/etc/op5/log.yml`	`/var/log/op5/nacoma.log` `/var/log/op5/nachos.log`	Error	True	PHP errors
GUI	Ninja	`/etc/op5/log.yml`	`/var/log/op5/ninja.log`	Error	True	PHP errors
HTTP API	HTTP API	`/etc/op5/log.yml`	`/var/log/op5/http_api.log`	Error	True	PHP errors
Mayi	Mayi	`/etc/op5/log.yml`	`/var/log/op5/mayi.log`	Error	False	PHP errors
SMS	-	`/etc/smsd.conf`	`/var/log/smsd/smsd.log` `/var/log/smsd/smsd_trouble.log` (only if `smart_logging` enabled, which creates a separate log file for errors)	Notice	-
Syslog		`/etc/rsyslog.conf` `/etc/syslog.conf` `/etc/syslog-ng.conf`	`/dev/console` `/var/log/messages` `/var/log/secure` `/var/log/maillog` `/var/log/spooler` `/var/log/boot.log` `/var/log/cron` `/var/log/kern`		-
Distributed and load-balanced setups	Merlin	`/opt/monitor/op5/merlin/merlin.conf`	`/var/log/op5/merlin/daemon.log` `/var/log/op5/merlin/neb.log`	Info	-	Merlin communication and module logs

Logging levels

These tables show the logging levels and types for the three main logging files: log.yml, smsd.log, and syslog-ng. Each level automatically includes all levels with lower granularity. For example, if the level is Warning, then Warning, Error, and Critical events are logged.

log.yml

Level	Type	Description
1	Error	Errors that have already occurred.
2	Warning	Potentially harmful situations.
3	Notice	Informational messages.
4	Debug	Fine-grained information events.

smsd.log

Level	Type	Description
7	Debug	All AT commands and modem answers and other detailed information useful for debugging.
6	Info	Information regarding current occurrences. Not detailed enough for debugging but potentially of interest.
5	Notice	Information regarding when a message was received or sent and abnormal occurrences which do not prevent SMS from working, such as a wrong destination number in an SMS file.
4	Warning	Warning message when there is a problem sending a single short message.
3	Error	Error message for a temporary problem, such as a modem answered with an error during initialisation, or a file cannot be accessed.
2	Critical	Error message for a permanent problem, such as sending failed multiple times, or wrong permissions for a queue.

syslog-ng

Level	Type	Description
0	emerg	System is unusable.
1	alert	Action must be taken immediately.
2	crit	Critical conditions.
3	err	Error conditions.
4	warning	Warning conditions.
5	notice	Normal but significant conditions.
6	info	Informational.
7	debug	Debug-level messages.

Example: Configure business services for logging

You need to modify the following two configuration files to debug business service logging: /etc/op5/log.yml and /opt/synergy/etc/config.lua.

Open file /etc/op5/log.yml and change the default logging level settings to the following:
```
...synergy: file: /var/log/op5/synergy.log level: debug reference: true...
```
Open file /opt/synergy/etc/config.lua and change the default logging level settings to the following:
```
... -- If true logs debugging to syslog debug = true,...
```
Change the rsyslog logging level in file /etc/rsyslog.conf, if required.
Note: Configuring rsyslog requires advanced knowledge of the rsyslog product. For guidance, see the vendor documentation.

Restart Synergy and rsyslog:

systemctl restart synergy
systemctl restart rsyslog

Remember to restore the original settings and restart the services when you no longer need to troubleshoot business services.