Configuring Dynamic Thresholds

Before configuring Dynamic Thresholds, ensure that the prerequisites have been met.

Exporting data to ITRS Analytics Copied

Exporting data to ITRS Analytics requires configuring the Results Exporter in Opsview.

Create an ITRS Analytics User Copied

To create and configure users in ITRS Analytics, follow the instructions in Role-based access control (RBAC) page from the ITRS Analytics documentation.

To import Dynamic Thresholds from an ITRS Analytics instance, the ITRS Analytics Importer process user must be configured with the credentials of an ITRS Analytics user having appropriate permissions. For on-premises ITRS Analytics instances, creating a dedicated user for this purpose is recommended.

In this example, a user named dynamic_thresholds is created.

1. Log in to the ITRS Analytics instance’s Keycloak page as administrator.

2. Switch to the obcerv realm by selecting it from the dropdown menu.

3. Create a new user with the username dynamic_thresholds.

4. Add a password to the dynamic_thresholds user by navigating to User > Credentials > Set Password. Ensure the Temporary option is not selected.

5. Add the user role to the dynamic_thresholds user by navigating to User > Role Mappings > Assign Roles. You may need to filter by realm roles to see the user role.

   Note

   To keep the system secure, it is recommended to create a new policy in ITRS Analytics’s Keycloak to allow the dynamic_thresholds user access to only the necessary entities.

6. Navigate to Clients > obcerv-platform > Authorization > Policies, and then create the following client policies of user type. See examples in ITRS Analytics documentation for reference.

   Name:    allow-ov-dynamic-thresholds
   Users:   dynamic_thresholds
   Logic:   Positive
   

   Name:    deny-ov-dynamic-thresholds
   Users:   dynamic_thresholds
   Logic:   Negative
   

7. Navigate to the Authorization > Scopes tab, and then create the following scope:

   Name: hierarchy = [host, service]
   Display Name: Opsview Dynamic Thresholds rules
   

8. Navigate to the Authorization > Resources tab, and then open the entities resource and add the newly created authorization scope to the Authorization scopes field. You may need to add a display name to save changes.

9. Create a new permission to deny a user the read scope for the resource. Navigate to the Authorization > Permissions tab, and then click Create scope-based permission to provide the following:

   Name:                    deny-entities-read-scope
   Resource:                entities
   Authorization scopes:    read
   Policies:                deny-ov-dynamic-thresholds
   Decision Strategy:       Unanimous
   

10. At this point, the user has no available scopes and cannot view any metrics from the query service. So, you must add a permission to allow the authorization scope created above. Return to the Authorization > Permissions tab, and then click Create Permission > Create scope-based permission to provide the following:

    Name:                    allow-entities-for-ov-dynamic-thresholds
    Resource:                entities
    Authorization scopes:    hierarchy = [host, service]
    Policies:                allow-ov-dynamic-thresholds
    Decision Strategy:       Unanimous
    

Configure backend Copied

The Dynamic Thresholds backend is configured by Opsview Deploy. While several options are available for backend configuration, the following must be specified in the user variables file at /opt/opsview/deploy/etc/user_vars.yml.

# Hostname of the Analytics server to connect to for importing dynamic thresholds
opsview_dynamic_thresholds_analytics_host: analyticsinstance.domain.com

# Username of the Analytics user used to connect to the Analytics server
opsview_dynamic_thresholds_analytics_username: dynamic_thresholds

# Password of the Analytics user used to connect to the Analytics server
opsview_dynamic_thresholds_analytics_password: amFtaWVsdWNrZXR0NzA3

Optionally, you can enable or disable server certificate checking. This defaults to true, and it uses the system CA certificate store to verify the ITRS Analytics server certificate:

# Whether to check the CA certificate of the server
opsview_dynamic_thresholds_obcerv_verify_cert: false

Further optional configuration options are documented in /opt/opsview/deploy/etc/examples/user_variables-example.yml. Once the configuration is added, run the sys-install.yml playbook with Opsview Deploy to apply the changes.

cd /opt/opsview/deploy
./bin/opsview-deploy ./lib/playbooks/sys-install.yml

Retrieve ITRS Analytics ingestion credentials Copied

You need to retrieve the ITRS Analytics ingestion password and ingestion certificate to configure the Results Exporter to export data to ITRS Analytics successfully.

Ingestion host and certificate Copied

Access the ITRS Analytics UI and navigate to Admin > Certificate.

You can extract the ingestion host from the present URL by removing the scheme and port sections; for example, my.host.com is the ingestion host taken from the URL https://my.host.com:123.

The ingestion certificate contents can then be copied and saved to a file on your orchestrator system. This file must be readable by the opsview user and in a valid PEM format.

Ingestion password Copied

You can retrieve the password for the ingestion-api user by running the following kubectl command against the ITRS Analytics cluster.

kubectl -n <analytics resources namespace> get secret iam-initial-credentials -o jsonpath='{.data.obcerv-ingestion-api}' | base64 -d

Configure Results Exporter in Opsview Copied

Dynamic Thresholds relies on metric timeseries from an ITRS Analytics instance. To export this data to an ITRS Analytics system, you must configure the Results Exporter component.

To export data to an ITRS Analytics instance, add the following configuration to your Results Exporter configuration file at /opt/opsview/deploy/etc/user_results_exporter.yml. If opsview_results_exporter_outputs is already defined, then add the analytics output to the existing configuration.

opsview_dynamic_thresholds_ingestion_host: analyticsinstance.domain.com
opsview_dynamic_thresholds_ingestion_password: your-ingestion-api-password

opsview_results_exporter_outputs:
  analytics:
    dynamic_threshold_metrics:
      type: dynamic_thresholds
      filter: '*'
      parameters:
        host: '{{ opsview_dynamic_thresholds_ingestion_host }}'
        port: 443
        cert_path: /path/to/analytics/ingestion/cert.pem
        username: ingestion-api
        password: '{{ opsview_dynamic_thresholds_ingestion_password }}'

Once the configuration is added, run the results-exporter-install.yml playbook with Opsview Deploy to apply the changes.

cd /opt/opsview/deploy
./bin/opsview-deploy ./lib/playbooks/results-exporter-install.yml

Configure Service Checks Copied

Note

The Dynamic Thresholds page is only visible to users with ADMINACCESS permission. For more information, see Adding a New Role. Additionally, the page will only show Host-Services the current Opsview user has access to.

Dynamic Thresholds are configured on a per-host-service basis. You can enable or disable Dynamic Thresholds for service checks in the Configuration > Dynamic Thresholds page.

Troubleshoot Service Checks Copied

Test your Dynamic Thresholds configuration in selected service checks through the Monitoring > Navigator page.

When the service check runs, the $WARNINGDT;DEFAULTVALUE;HARDLIMIT$ and $CRITICALDT;DEFAULTVALUE;HARDLIMIT$ macros are replaced with the current threshold values if Dynamic Thresholds has been correctly configured.

If Dynamic Thresholds is not configured properly or applied to the check, the default value will be used; the value is never allowed to exceed the hard limit.

Limitations Copied

• Dynamic Thresholds are only available for service checks that return a single metric value.
• Dynamic Thresholds currently only support service checks that calculate thresholds as VALUE > THRESHOLD_VALUE. Less-than thresholds are not supported.
• Dynamic Thresholds only support service checks that include the Dynamic Thresholds macros: $WARNINGDT or $CRITICALDT as values for warning and critical threshold arguments.
  • Currently only a subset of the bundled Service Checks support Dynamic Thresholds.
  • Dynamic Threshold macros are written in the format $WARNINGDT;DEFAULTVALUE$, where DEFAULTVALUE is the default threshold value if Dynamic Thresholds are not available/not configured.
  • Optionally, you can also set a hard limit for Dynamic Threshold macros. This is defined as $WARNINGDT;DEFAULTVALUE;HARDLIMIT$, where HARDLIMIT will cap any calculated dynamic thresholds at its value. Note that hard limits only apply to checks with Dynamic Thresholds enabled and do not affect default values.
• Threshold prediction accuracy will improve based on the length of time data has been exported to ITRS Analytics. While one week of exported data is recommended before using Dynamic Thresholds, optimal accuracy is achieved with up to four weeks of data.
  • For newly exported host-service metric data, Dynamic Thresholds may not be available for a brief period until sufficient data is collected.

Troubleshooting Copied

To help resolve any issues you may encounter during the configuration, refer to the Dynamic Thresholds issues troubleshooting documentation.