Configuring Dynamic Thresholds

Exporting data to Obcerv Copied

Exporting data to Obcerv requires configuring the Results Exporter in Opsview and using the most recent Obcerv version for data ingestion.

Configure Obcerv Copied

Obcerv version 2.6 introduced support for ingesting Opsview data. It is currently the only version supported for Dynamic Thresholds.

Create an Obcerv User Copied

To create and configure users in Obcerv, follow the instructions in Role-based access control (RBAC) page from the Obcerv documentation.

To import Dynamic Thresholds from an Obcerv instance, the Obcerv Importer process user must be configured with the credentials of an Obcerv user having appropriate permissions. For on-premises Obcerv instances, creating a dedicated user for this purpose is recommended.

In this example, a user named dynamic_thresholds is created.

1. Log in to the Obcerv instance’s Keycloak page as administrator.

2. Create a new user with the username dynamic_thresholds under the obcerv realm.

3. Add a password to the dynamic_thresholds user by navigating to User > Credentials > Set Password. Ensure the Temporary option is not selected.

4. Add the user role to the dynamic_thresholds user by navigating to User > Role Mappings > Assign Roles.

   Note

   To keep the system secure, it is recommended to create a new policy in Obcerv’s Keycloak to allow the dynamic_thresholds user access to only the necessary entities.

5. Navigate to Clients > obcerv-platform > Authorisation > Policies, and then create the following user policies. See examples in Obcerv documentation for reference.

   Name:    allow-ov-dynamic-thresholds
   Users:   dynamic_thresholds
   Logic:   Positive
   

   Name:    deny-ov-dynamic-thresholds
   Users:   dynamic_thresholds
   Logic:   Negative
   

6. Navigate to the Authorisation > Scopes tab, and then create the following scope:

   Name: hierarchy = [host, service]
   Display Name: Opsview Dynamic Thresholds rules
   

7. Navigate to the Authorisation > Resources tab, and then open the entities resource and add the newly created authorization scope to the Authorization scopes field.

8. Create a new permission to deny a user the read scope for the resource. Navigate to the Authorisation > Permissions tab, and then click Create scope-based permission to provide the following:

   Name:                    deny-entities-read-scope
   Resource:                entities
   Authorization scopes:    read
   Policies:                deny-ov-dynamic-thresholds
   Decision Strategy:       Unanimous
   

9. At this point the user has no available scopes and cannot view any metrics from the query service. So, add a permission to allow the authorisation scope created above. Return to the Authorisation > Permissions tab, and then click Create Permission > Create scope-based permission to provide the following:

   Name:                    allow-entities-for-ov-dynamic-thresholds
   Resource:                entities
   Authorization scopes:    hierarchy = [host, service]
   Policies:                allow-ov-dynamic-thresholds
   Decision Strategy:       Unanimous
   

Configure backend Copied

The Dynamic Thresholds backend is configured by Opsview Deploy. While several options are available for backend configuration, the following must be specified in the user variables file at /opt/opsview/deploy/etc/user_vars.yml.

# Hostname of the Obcerv server to connect to for importing dynamic thresholds
opsview_dynamic_thresholds_obcerv_host: obcervinstance.domain.com

# Username of the Obcerv user used to connect to the Obcerv server
opsview_dynamic_thresholds_obcerv_username: dynamic_thresholds

# Password of the Obcerv user used to connect to the Obcerv server
opsview_dynamic_thresholds_obcerv_password: amFtaWVsdWNrZXR0NzA3

Further optional configuration options are documented in /opt/opsview/deploy/etc/examples/user_variables-example.yml. Once the configuration is added, run the sys-install.yml playbook with Opsview Deploy to apply the changes.

cd /opt/opsview/deploy
./bin/opsview-deploy ./lib/playbooks/sys-install.yml

Retrieve Obcerv ingestion credentials Copied

You need to retrieve the Obcerv ingestion password and ingestion certificate to configure the Results Exporter to export data to Obcerv successfully.

Ingestion certificate Copied

The ingestion certificate contents can be copied from the Obcerv UI at Admin > Ingestion Service under CERT and saved to a file on your orchestrator system. This file must be readable by the opsview user and in valid PEM format.

Ingestion password Copied

You can retrieve the password for the ingestion-api user by running the following kubectl command against the Obcerv cluster.

kubectl -n <obcerv resources namespace> get secret iam-initial-credentials -o jsonpath='{.data.obcerv-ingestion-api}' | base64 -d

Configure Results Exporter in Opsview Copied

Dynamic Thresholds relies on metric timeseries from an Obcerv instance. To export this data to an Obcerv system, you must configure the Results Exporter component.

To export data to an Obcerv instance, add the following configuration to your Results Exporter configuration file at /opt/opsview/deploy/etc/user_results_exporter.yml. If opsview_results_exporter_outputs is already defined, then add the obcerv output to the existing configuration.

opsview_dynamic_thresholds_ingestion_host: obcervinstance.domain.com
opsview_dynamic_thresholds_ingestion_password: your-ingestion-api-password

opsview_results_exporter_outputs:
  obcerv:
    dynamic_threshold_metrics:
      type: dynamic_thresholds
      filter: '*'
      parameters:
        host: '{{ opsview_dynamic_thresholds_ingestion_host }}'
        port: 443
        cert_path: /path/to/obcerv/ingestion/cert.pem
        username: ingestion-api
        password: '{{ opsview_dynamic_thresholds_ingestion_password }}'

Once the configuration is added, run the results-exporter-install.yml playbook with Opsview Deploy to apply the changes.

cd /opt/opsview/deploy
./bin/opsview-deploy ./lib/playbooks/results-exporter-install.yml

Configure Service Checks Copied

Dynamic Thresholds are configured on a per-host-service basis. You can enable or disable Dynamic Thresholds for service checks through the Opsview Configuration > Dynamic Thresholds page.

This page is only visible to users with the ADMINACCESS permission. See the section Adding a New Role for more information. Additionally, the Dynamics Threshold configuration page will only show Host-Services the current Opsview user has access to.

When you first access the Dynamic Thresholds configuration page, you will not see any service checks listed. No host-services are enabled for Dynamic Thresholds by default.

1. To enable Dynamic Threshold for a service check or multiple service checks, use the search bar to find the desired services. Pressing ENTER will display a list of matching services in the table.

   

2. Select the service checks you want to enable Dynamic Thresholds for by clicking the checkbox, then click Apply threshold changes. To disable Dynamic Thresholds for a service, uncheck the box and click Apply threshold changes again.

   

3. Once enabled, the service check will be listed in the table. The Current Thresholds column displays the latest threshold data for the service check (or Awaiting Data if no data has been imported yet).

   

Limitations Copied

• Dynamic Thresholds are only available for service checks that return a single metric value.
• Dynamic Thresholds currently only support service checks that calculate thresholds as VALUE > THRESHOLD_VALUE. Less-than thresholds are not supported.
• Dynamic Thresholds only support service checks that include the Dynamic Thresholds macros: $WARNINGGT or $CRITICALGT as values for warning and critical threshold arguments.
  • Currently only a subset of the bundled Service Checks support Dynamic Thresholds.
  • Dynamic Threshold macros are written in the format $WARNINGGT:DEFAULTVALUE$, where DEFAULTVALUE is the default threshold value if Dynamic Thresholds are not available/not configured.
• Threshold prediction accuracy will improve based on the length of time data has been exported to Obcerv. While one week of exported data is recommended before using Dynamic Thresholds, optimal accuracy is achieved with up to four weeks of data.
  • For newly exported host-service metric data, Dynamic Thresholds may not be available for a brief period until sufficient data is collected.

Troubleshooting Copied

Dynamic Thresholds configuration page is missing Copied

The Dynamic Thresholds configuration page may not be accessible in the navigation bar if the Dynamic Thresholds feature is not included in your Opsview license.

Please check Managing your Subscription/Checking your Subscription Details for more information.

Host-services are missing in the Dynamic Thresholds search Copied

Host-services must be compatible with the Dynamic Thresholds feature to appear in the configuration page search results. Compatibility is determined by the following criteria:

• Host-service must be returning performance metrics.
• Service check definition must contain one of the Dynamic Thresholds macros: $WARNINGGT or $CRITICALGT.

My enabled host-services are in an Awaiting Data state Copied

If a host-service’s dynamic threshold configuration is showing Awaiting Data, it’s possible that service metric data isn’t reaching Obcerv. Verify that the correct host-services are being exported through the Obcerv Web Console and check Opsview log files for any related errors.

Log message details Copied

These are common log messages you might see when configuring Dynamic Thresholds encounters an error or issue.

Nothing to do right now Copied

INFO [opsview.opsview.processors.obcervimporter] ObcervImporterProcessor checking objects ...
INFO [opsview.opsview.processors.obcervimporter] Nothing to do right now

All host-services using Dynamic Thresholds have been refreshed within the last min_object_update_interval_secs seconds, and there have been no changes to which host-services are enabled or disabled for this feature. The Obcerv Importer will perform another check later.

Failed to extract result for 'Host.Service (oid=707) metric: response = {} Copied

INFO [opsview.opsview.processors.obcervimporter] ObcervImporterProcessor checking objects ...
WARNING [opsview.obcervimporter.opsview.obcervimporter.obcervclient] Failed to extract result for 'hostA.'ServiceA' (oid=7) metricA': response={}
WARNING [opsview.obcervimporter.opsview.obcervimporter.obcervclient] Failed to extract result for 'hostB.'ServiceB' (oid=97) metricB': response={}

Note

As discussed in the Limitations section, Dynamic Thresholds work optimally with host-services that have at least a week’s worth of data in Obcerv. For newly monitored host-services, sufficient data for threshold calculations might not yet be available. In such cases, the Obcerv Importer will periodically retry and log a warning until enough data is collected.