Results Exporter
Opsview Results Exporter exports results to services such as Splunk, syslog, local files, and arbitrary HTTP endpoints.
Known issue
When logging data to Rsyslog via TCP, the first message has the correct priority, but all subsequent messages use theuser.notice
priority.
Installation Copied
The Opsview Results Exporter component can be installed on the Orchestrator system.
To install the Results Exporter:
# Debian
sudo apt-get install opsview-results-exporter
# RPM
sudo yum install opsview-results-exporter
Next, ensure that your component configuration (see Configuration section below) is updated to contain the correct:
- Message Queue encoder key and password in the
results_queue
andcommand_queue
sections. - Database credentials in the
runtime_database
section. - Registry password in the
registry
section.
These will be the values set when deploying your Opsview system, as specified in /opt/opsview/deploy/etc/user_secrets.yml
.
# Once the dependencies are installed, configured and running,
# run the following command to allow opsview-monit to monitor
# the new component
sudo -iu root /opt/opsview/watchdog/bin/opsview-monit reload
Configuration Copied
The opsview-results-exporter
component ships with a default configuration file (/opt/opsview/resultsexporter/etc/resultsexporter.defaults.yaml
). Any changes to this file will be overwritten on package updates so you should not edit this file if you want to keep your changes across package updates.
Create /opt/opsview/resultsexporter/etc/resultsexporter.yaml
for any custom settings if it does not already exist. resultsexporter.yaml
is not overwritten across package updates and overrides settings in resultsexporter.defaults.yaml
.
With resultsexporter.yaml
permissions, ensure /opt/opsview/resultsexporter/etc/resultsexporter.yaml
is accessible by the opsview group with read permissions. For example:
chown root:opsview /opt/opsview/resultsexporter/etc/resultsexporter.yaml
chmod 640 /opt/opsview/resultsexporter/etc/resultsexporter.yaml
Example contents of /opt/opsview/resultsexporter/etc/
:
drwxr-x--- 2 root opsview 4096 Jan 21 11:53 .
drwxr-x--- 7 root opsview 4096 Jan 21 11:31 ..
-rw-r----- 1 root opsview 625 Jan 18 16:33 resultsexporter.defaults.yaml
-rw-r----- 1 root opsview 2343 Jan 21 17:04 resultsexporter.yaml <----- make your changes here
The following options can be overridden in resultsexporter.yaml
:
results_queue
— results message queue configuration.command_queue
— command message queue configuration.runtime_database
— runtime database connection configuration.outputs
— list of outputs to export the results to. Results can be exported via syslog, file, and HTTP.registry
— connection configuration for the registry.default_fields
— default message fields to export. See Field Mapping.default_filter
— default filter that is applied to messages. See Filtering.
Credentials
Ensure that your component configuration has the correct Message Queue encoder key, password, database password, and registry password during installation. These values were generated when you deployed Opsview and can be found in/opt/opsview/deploy/etc/user_secrets.yml
.
When overriding settings in resultsexporter.yaml
, refer to the structure used in resultsexporter.defaults.yaml
. For example, to override the results Message Queue or registry settings within resultsexporter.yaml
:
resultsexporter:
...
results_queue:
messagequeue:
encoder_key: ZW5jb2RlcmR1bW15a2V5MTIzNA==
host: 127.0.0.1
port: 35672
management_port: 45672
user: opsview
password: opsview
registry:
host: 127.0.0.1
port: 12379
user: opsview
password: opsview
Refer to the examples as described on Exporting Results on how to add your own outputs.
Advanced Configuration Copied
Worker settings Copied
The number of parallel component workers, and the timeout before component worker restart (if encountering errors) can be set as follows:
resultsexporter:
...
worker_timeout: <secs> # default 30
initial_worker_count: <num> # default to 0, which will use the number of CPU cores
Note the following considerations:
- On a multi-core system, the more workers in use, the lower the load and elapsed time will be, with an upper bound on the number of cores available in your system.
- The more workers in use, the greater the overall component memory usage will be used.
- The memory usage of workers is multiplicative with the number of outputs configured.
Data Cache settings Copied
The Results Exporter can cache Opsview data if the dal_fetchall
or dal_fetchone
field operations are in use (see Exporting Results).
Override the data_access_cache
section in the configuration file for full control over the cache settings:
resultsexporter:
...
data_access_cache:
- <data model type>: # e.g. RuntimeHosts
cache_items: <yes|no> # whether to cache data for the model
max_items: <num> # if caching, the maximum number of items to cache (if field unset, defaults to no limit)
max_item_size: <num> # if caching, the maximum size of any cached item (if field unset, defaults to no limit)
pre_load: <yes|no> # whether to pre-load data on component start up (default no)
refresh_on: reload # set if the cache should be cleared for the model when Apply Changes completes
Note the following considerations:
- The maximum number of items refers to the number of individual queries (for example, individual field operation executions after message fields are populated) that can be cached at once. For example, if a query was configured to run using a message field and it was ran against 2 messages with different values for that field, it would count as 2 separate cache items.
- The maximum size of each item refers to the number of data model results the cache will accept. For example, if a query matches 2
RuntimeHosts
in the system, it would be a cache item of size 2. - Setting a maximum number or size on items will reduce the memory usage of the component, but it will increase the load and elapsed time to process individual messages.
- Enabling preloading will increase load and elapsed time on component startup (and when Apply Changes occurs, if refreshing on reload is set), but it will decrease load and elapsed time during processing.
- Using message filters to decrease the number of messages processed will decrease the overall load and elapsed time to process all messages.
By default, the component uses the following cache configuration:
opsview_results_exporter_data_access_cache:
- RuntimeHashtags:
cache_items: no
- RuntimeHosts:
cache_items: yes
max_items: 8_000
max_item_size: 1
pre_load: yes
refresh_on: reload
- RuntimeServicechecks:
cache_items: yes
max_items: 32_000
max_item_size: 1
pre_load: yes
refresh_on: reload
The component will generate a warning if it encounters an unexpected model type.
Data from unsupported model ... could not be queried from cache
If intentionally not caching a model type, then add a cache_items: no
entry for the model to the cache settings to suppress the logging.
This will require overriding the entire section and then adding your new entries.
Logging Settings Copied
The Results Exporter logging settings can be set by overriding the following section:
resultsexporter:
...
logging:
sensitive_trace: <yes|no> # enable low-level trace logging (default no)
data_access_cache:
collisions: <yes|no> # whether to log data cache collisions (default no)
loggers:
opsview:
level: NOTICE # overall component log level
The sensitive_trace
will only have an effect if the component is also set at the DEBUG
log level.
Monitoring Copied
To apply the self monitoring Host Template for this component, run the following:
host1:~# cd /opt/opsview/deploy/
# configure monitoring of the Opsview Results Exporter component
host1:/opt/opsview/deploy# ./bin/opsview-deploy lib/playbooks/setup-monitoring.yml
Management Copied
Configuration Copied
DPKGs Copied
Watchdog service files are now managed by the package, doing a remove would leave the watchdog service file behind with a .save extension. Purging the package will remove it. The package managed config files are as follows:
/opt/opsview/watchdog/etc/services/opsview-resultsexporter.conf
RPMs Copied
Watchdog service files are now managed by the package. Any modifications will be saved at upgrade and remove processes with the .rpmnew
and .rpmsave
extensions correspondingly.
/opt/opsview/watchdog/etc/services/opsview-resultsexporter.conf
Service administration Copied
As root, start, stop and restart the service using:
/opt/opsview/watchdog/bin/opsview-monit <stop|start|restart> opsview-resultsexporter
Troubleshooting Copied
Credentials Copied
If you see this error message in the log, please check that the component configuration has the correct password for the registry. The Results Exporter will need to be restarted if any changes are made.
[ERR] RegistryError<status_code=401, error_code=110, index=0, message=u'The request requires user authentication', raw_message='{"errorCode":110,"message":"The request requires user authentication","cause":"Insufficient credentials","index":0}\n'>
A message containing the following error will be seen on start-up if the Message Queue password is incorrect:
[ERR] Opsview Results-Exporter launcher failed to start: (libev) epoll_wait: Invalid argument
Messages containing the following errors will be seen when processing messages if the Message Queue encoder key is incorrect:
[ERR] Cannot verify HMAC signature! ...
[ERR] Dumping unparsable message: ...
If the database credentials are incorrect, the following error will be displayed on start-up:
[ERR] Failed to connect to Data Access Layer: (1045, "Access denied for user ...
Component restarting too frequently Copied
The error message below indicates that the component is encountering a recurring error that is causing it to restart too frequently:
[ERR] Opsview Results-Exporter launcher failed to start: (libev) epoll_wait: Bad file descriptor
If this is the case, look for an earlier error message in the logs that matches one of the examples on this page.
Configuration file formatting Copied
If you receive this error message, with the /var/log/opsview/opsview
, please check the /opt/opsview/resultsexporter/etc/resultsexporter.yaml
filter syntax is correct.
For the error below, an extra parenthesis was present. The opsview-resultsexporter
will need to be restarted if any change is made.
Jul 22 12:02:24 orchestrator-main opsview : [ERR] Opsview Results-Exporter launcher failed to start: Output 'syslog.test': Filter Syntax Error - char: ')', line: '1,113'.