Use Agentic SRE for operations and support in ITRS Analytics

Beta

The ITRS Agentic SRE app is currently available as a beta release. Features, behavior, and documented procedures are subject to change without prior notice during ongoing development. Starting with Agentic SRE app version 0.6.0, your license must include the Agentic SRE entitlement, and the app must be enabled in the KOTS Admin Console. For details, see Enable Agentic SRE with a license entitlement. For information about major and minor updates, see the Agentic SRE app release notes.

Important

Starting with Agentic SRE app version 0.5.0, provider API keys and tokens are expected to be stored in the Secrets app instead of directly in the LLM configuration. If you upgrade from an earlier beta release, review any previously configured LLMs and recreate or update them to use API Key (Secret).

About the ITRS Agentic SRE app Copied

The ITRS Agentic SRE app is an AI-assisted capability within the ITRS Analytics Web Console. It helps users investigate operational conditions, ask product-related questions, and work from monitored entity context within a single interface.

The app provides an AI chat experience for operational and support workflows, together with an Admin > AI area where administrators can configure LLMs, test model connections, review service readiness, and review evaluation results before making models available for wider use.

The Agentic SRE app provides the following capabilities for day-to-day use:

• Specialized agent modes such as SRE, Insights, RCA, and Support, each intended for a different operational or support task.
• Entity-aware investigations that can find relevant monitored entities and help keep a conversation focused on selected services, hosts, or components.
• Inline entity references that let users add one or more monitored entities to a prompt from recently visited entities, entity cards, or the @ menu.
• Structured responses that can include reasoning steps, supporting sources, tool activity, and generated outputs to help users review the response in context.
• Response feedback controls that let users approve or disapprove an answer and optionally explain why an answer was not useful.
• Chat history controls that let users reopen, rename, archive, and restore previous conversations when chat history is enabled.
• Entity-focused dashboard workflows that can help users continue analysis from a relevant dashboard or entity card.
• LLM configuration workflows that support Google GenAI and secure API-key references through the Secrets app.
• Eval workflows that let administrators load tests by tags, compare results, and upload saved eval reports for review.
• Audit workflows, when enabled, that let administrators review AI conversations and export conversation records.
• Status monitoring, when enabled, that lets administrators review LLM readiness and the availability of the documentation and entity indexes.

Note

Available agent modes and admin features can vary by deployment, user permissions, and enabled capabilities.

Prerequisites Copied

Before using the Agentic SRE app, ensure that the following requirements are met:

• Your license includes the Agentic SRE entitlement, and the app is enabled in the KOTS Admin Console. For details, see Enable Agentic SRE with a license entitlement.
• ITRS Analytics Platform 2.17.0 or later and Web Console 3.13.0 or later.
• At least one LLM is configured in Admin > AI > LLMs. The AI chat is not usable until an LLM is available for selection.
• The Secrets app must be available in your deployment so provider credentials can be stored and referenced securely.
• If you want to use administrative features such as LLMs, Eval, or Audit, you must have access to Admin > AI.

If no LLM is configured, start by adding your chosen AI provider in the LLMs section before using the AI chat.

Use case scenarios Copied

Use the Agentic SRE app when you want faster guidance without switching between multiple tools or documentation sources.

Note

Use SRE when you want a general entry point for the Agentic SRE app. The SRE agent acts as an orchestrator that can handle different types of requests and route them to the appropriate specialist workflow when needed.

Investigate what is happening in your environment Copied

Use Insights when you want a quick operational summary for one or more selected entities. To request a platform-wide summary without entity context, state explicitly that you want an overview of the system or environment.

Typical tasks include:

• Starting with a single agent for operational, support, and investigation queries.
• Summarizing what changed in the last few minutes for one or more selected entities or services.
• Requesting an overview of the whole environment by stating the intended scope.
• Reviewing related signals, metrics, and surrounding context before escalation.
• Building a first-pass explanation of what is happening in the environment.

If Insights cannot resolve the entity named in a request, refine the entity reference or try the request again instead of relying on a platform-wide summary.

In most cases, you can send all tasks through SRE and let it direct the request to the most relevant workflow. This is useful when you need to move from an alert to a working hypothesis quickly, or when you are not sure which specialist agent to choose first.

Perform root cause analysis Copied

Use RCA when you already have an incident, service, or entity in scope and want a more structured investigation.

Typical tasks include:

• Narrowing the likely cause of a service issue by following related telemetry.
• Comparing current symptoms with the surrounding operational context.
• Focusing the investigation on the most relevant evidence first.

For example, you might ask, “What changed in the last 30 minutes for this service?” after selecting the affected entity in Entity Viewer.

Get product and setup guidance Copied

Use Support when you need help with product features, setup tasks, or product documentation.

Typical tasks include:

• Asking how to complete a configuration task in ITRS Analytics.
• Requesting product guidance for onboarding or troubleshooting.
• Searching across supported documentation sources without leaving the Web Console.

For example, you can ask for help creating a configuration template or for guided steps to complete a setup task.

You can also start these requests in SRE and allow it to route the query to the appropriate support-oriented workflow.

Create dashboards from AI results Copied

Use the app when you want to create a dashboard for a selected entity.

Add the entity to the prompt and ask the AI chat to create an entity dashboard. When the dashboard is created, the response includes a link that you can use to open it for further analysis.

Entity cards in a response provide separate actions for opening the related entity context or adding the entity to a prompt.

How to use the AI chat in the ITRS Analytics Web Console Copied

Follow these steps to start using the AI chat in the Web Console:

1. In the ITRS Analytics Web Console, open the AI chat from the header.
2. In the prompt toolbar, choose the agent mode that best matches your task.
3. Select the LLM that you want to use.
4. Enter your prompt.
5. If you are already working in Entity Viewer, select the relevant entity to make it available among the recently visited entities in AI chat.
6. Add each relevant entity to the prompt before sending it.

Use the agent modes as follows:

• SRE — Use this as the primary agent for most requests. It can handle general operational questions and broad troubleshooting, and it can route the request to the appropriate specialist workflow when needed.
• Insights — Use this for entity summaries, change investigation, and explicitly requested system-wide overviews.
• RCA — Use this for focused incident investigation and root cause analysis.
• Support — Use this for product how-to guidance, setup help, and documentation-based questions.

Add entity context to a prompt Copied

Add entity references when you want an answer to focus on specific monitored entities. Entity references appear as chips within the prompt, so you can review the scope before sending the request.

To add entity context:

1. Open or select an entity in the Web Console. Recently visited entities appear above the prompt.

2. Add an entity by using one of the following methods:

   • select Add to chat on the entity chip
   • open Contexts to apply and add one or all recently visited entities
   • enter @ in the prompt and select an entity from the filtered list
   • select Add to chat on an entity card in an AI response

3. Repeat the process to include more than one entity.

4. Review the entity chips within the prompt, then send the request.

You can select an entity chip to open the related entity.

Review the AI response Copied

After you send a prompt, review the response details before acting on the recommendation:

• Check the main answer for the recommended next step.
• Review reasoning steps when you want to understand how the answer was built.
• Review sources when the response includes documentation links or cited references.
• Review tool activity and generated outputs when the app provides structured results.
• Click entity cards to continue the investigation from the related entity context.

Provide feedback on an AI response Copied

If response feedback is enabled, use the thumbs-up or thumbs-down controls below an AI response to record whether it was useful.

• Select thumbs up to approve the response
• Select thumbs down to disapprove the response and optionally explain what was wrong
• Select the active control again to remove your feedback

Submitted feedback remains associated with the response and is available to administrators in Audit.

Use chat history if it is enabled Copied

If chat history is enabled in your deployment, you can reopen earlier conversations from the history panel.

Use chat history to:

• Start a new chat from the history panel.
• Resume a previous investigation.
• Rename a conversation so it is easier to find later.
• Archive conversations that you no longer need in the active list.
• Open the archived list and restore a conversation when you need it again.

Configure AI settings in Admin Copied

Use Admin > AI to prepare the Agentic SRE app for operational users. From this area, you can configure LLMs, review service readiness and evaluation reports, and view AI audit activity.

Admin access

The procedures in this section are intended for users with access to Admin > AI. Available sections, such as Eval and Audit, can vary by deployment and assigned permissions.

Add an LLM Copied

Use the LLMs section to add, test, and manage the language models that the Agentic SRE app can use for chat, retrieval, and evaluation.

Important

Starting with Agentic SRE app version 0.5.0, provider API keys and tokens are expected to be stored in the Secrets app instead of directly in the LLM configuration. If you upgrade from an earlier beta release, review any previously configured LLMs and recreate or update them to use API Key (Secret).

Use this section when you need to:

• add a new model provider for chat or evaluation
• update model settings such as endpoint, model name, or credentials
• verify that a configured model is ready before making it available to users
• review which models are available for selection in chat

Use this procedure to add a model and make it available in chat:

1. In the Web Console, go to Admin > AI > LLMs.

2. Click Add LLM to create a new entry, or copy an existing LLM configuration.

3. Select the provider that you want to use.

4. Enter the required provider settings, such as model name, endpoint or base URL, credentials, and any provider-specific options.

5. In API Key (Secret), select the secret that stores the provider credential.

6. If no secrets are available, create the required secret first in the Secrets app, then return to the LLM configuration and select it.

7. Use the masked credential fields and reveal control only when you need to verify an entered value.

8. Click Test to verify that the model connection works as expected. The control shows progress followed by a success or failure icon. Point to the control to review the response or failure reason, and click it again when you want to rerun the test.

   

9. Save the LLM.

10. Open the AI chat and select the new LLM from the prompt toolbar.

Before you save a model, confirm the following:

• the provider is correct
• the model name matches the model that you intend to use
• the endpoint or base URL is valid for that provider
• the credential is stored as a secret reference
• the Test action succeeds

Depending on your deployment, available providers can include managed services and self-hosted options.

Configure a Google GenAI model Copied

Google GenAI is available as an LLM provider. When adding a Google GenAI model, configure the following required settings:

• API Key (Secret) — The secret that stores the Google GenAI API key
• Model — The Gemini model that the app uses
• Base URL — The Google GenAI service endpoint

You can also configure Temperature, Max Tokens, Top P, and Top K. Leave optional settings unchanged to use the provider defaults.

For supported Gemini models, use either Thinking Level or Thinking Budget to limit how much reasoning the model performs. Do not configure both settings for the same LLM. Leave both settings unset to use the model default.

Configure reasoning for an OpenAI model Copied

For OpenAI models that support reasoning, use Reasoning Effort to control how much reasoning the model performs. The available values depend on the selected model. Models that do not support this setting do not show the field.

Review service readiness in Status Copied

If Status is enabled in your deployment, use Admin > AI > Status to review whether the services required by the Agentic SRE app are ready.

The Status page provides live information for:

• LLM Readiness — Shows the number of configured LLMs and whether chat is ready. You can test individual LLMs from this card
• Documentation Index — Shows whether the index, documents, and embedding model are loaded
• Entity Index — Shows whether the entity index is available, its current build phase, and the number of indexed entities

The overall status can be Operational, Degraded, or Unavailable. Review the individual cards when the app reports reduced capability or an unavailable component.

The AI chat header also shows a status indicator. Administrators can select this indicator to open the Status page.

Review AI activity in Audit Copied

If Audit is enabled in your deployment, use it to review recorded AI conversations and related activity.

Use Audit and chat history for different purposes:

• Chat history is a user-facing way to reopen, rename, archive, and restore previous conversations from the chat interface.
• Audit is an administrative view of recorded AI conversations and related activity.

Chat history contains your own AI chat conversations. Audit provides administrators with tenant-wide activity, including conversations from all users and activity that is not added to user chat history, such as Eval runs and LLM connection tests.

• use chat history when you want to continue your own earlier work

• use Audit when you want to review tenant-wide activity in more detail, inspect turns, or export records

  

1. Go to Admin > AI > Audit.
2. Review the list of recorded conversations. Each entry can show details such as Title, LLMs Used, Exchanges, Tokens, Created, and Status.
3. Use the available filters to narrow the list when you need to find a specific conversation or time range.
4. Use Refresh to reload the Audit list.
5. Open a conversation from the list to review the recorded exchange in more detail.
6. In the conversation detail view, review the summary information, including the LLMs used, exchange count, average latency, total tokens, user and team information, and conversation status.
7. Review the conversation replay to inspect the recorded prompts and responses.
8. If turn replay is available, select Inspect turn on a turn entry to open the turn inspection drawer.
9. In Turn inspection, review the event timeline for that turn. This view can help you examine the recorded sequence of events for the selected exchange in greater detail.
10. Review the feedback indicator for a turn to see whether the user approved or disapproved the response. Point to the indicator to review an optional feedback comment.
11. Export the conversation record in the available format when you need to keep or share the audit output.

Use conversation details as follows:

• Review LLMs Used to confirm which model or models were involved in the conversation.
• Review Exchanges to understand how many turns were recorded.
• Review Tokens to understand the scale of the interaction.
• Review Created and Status to confirm when the conversation occurred and whether it completed successfully.

If turn replay is available, each turn can also show:

• Turn status
• Duration
• Token count
• User prompt
• LLM used for the response
• Assistant response

Use Inspect turn when you need a more detailed view of a single exchange. In the turn inspection drawer, review the event timeline and use the available categories to focus on the part of the turn that you want to inspect, such as LLM activity, tools, retrieval, or routing.

For example, if you need to review a production support conversation, open the audit entry and confirm which LLM answered, how many exchanges occurred, how many tokens were used, and whether the conversation completed successfully. If a specific turn needs closer review, use Inspect turn to open the detailed event timeline for that exchange.

Run an Eval campaign and review the results Copied

Use Eval to compare outputs, validate model behavior, and review answer quality before wider rollout.

Use Eval when you want to:

• compare one LLM against another
• test whether a model answers Support questions correctly
• check whether required words or phrases appear in an answer
• review the quality of a model before wider use

1. Go to Admin > AI > Eval.

2. Click Add Eval to open the Add New Eval wizard.

   

3. In Configuration, select the target LLM, eval type, and number of iterations. Choose the model explicitly rather than relying on a default selection.

4. In Graders, select one or more graders for the run.

5. In Tests, load tests from one or more test sets, or define custom tests for your scenario.

6. If your test sets use tags, filter the loaded questions by tag to narrow the scope of the run.

7. For string-content checks, choose whether every expected string must match or the best-matching string determines the score.

8. Select the tests that you want to include, then click Run Eval.

9. Monitor the run status and wait for completion. Click Cancel Evaluation if you need to stop an active run.

10. Open the generated report to compare scores, inspect test results, filter results by tag, and review answer quality.

11. Repeat the run with adjusted settings if you want to compare models or tune the configuration.

Understand Eval results Copied

After an Eval run completes, open the generated report to review the results in detail.

Example use case:

Use Eval to validate Support answers before rolling out a new LLM.

Example test prompt:

Can Geneos monitor Linux and Windows hosts?

Example expected answer:

Yes. Geneos can monitor both Linux and Windows hosts. The answer should clearly mention both operating systems and provide a concise explanation.

Example expected strings:

• Linux
• Windows

This type of test checks two things at the same time:

• whether the answer is broadly correct and helpful
• whether it includes the exact terms that the answer must mention

Use the report to:

• Review the overall score ranges across the selected tests.
• Compare the score ranges for each configured grader.
• Open individual tests to review prompts, expected answers, and returned answers.
• Filter the report by tags when you want to focus on a specific group of tests.

When you open a test answer, the view can include:

• Expected and Actual panels so you can compare the target answer with the returned answer.
• Answer Reasoning when the run stores reasoning for that answer.
• Grader tabs such as LLM, String Content, and other configured grader types.
• A percentage score on each grader tab. This percentage shows the score returned by that grader for the selected answer. Higher percentages indicate a closer match to the grader’s criteria.
• Structured feedback cards that explain why the answer received that score, for example under criteria such as correctness, completeness, clarity, relevance, or similarity.
• An Overall summary that helps you decide whether the answer is acceptable for your use case.

Use the grader tabs as follows:

• LLM — Shows the score from the LLM-based grader. Use this to assess whether the answer is semantically useful, clear, and aligned with the expected outcome.
• String Content — Shows the score for expected-string matching. Use this when your test requires specific terms, values, or phrases to appear in the answer.
• Routing — Shows whether the request was routed to the expected agent.
• Tool Selection — Shows whether the expected tools were used.
• Routing + Tool — Shows the combined routing and tool-selection result.
• Grounding — Shows whether the answer is supported by the retrieved information.

Available Eval types and graders depend on the enabled capabilities. By default, production deployments provide Support evals. Agent-targeted evals for SRE, Insights, and RCA are available only when enabled for the deployment.

If a grader does not apply to a test, the result is shown as N/A instead of a percentage. An N/A result does not indicate a failed test.

For example, if a Support eval answer shows LLM 68% and String Content 100%, you can interpret the result as follows:

• The answer included the expected strings successfully.
• The answer was only partially aligned with the broader expected quality or meaning.

Use this combination of scores and feedback to decide whether you should adjust the prompt, refine the test, or compare the same test against another LLM.

Upload and review Eval reports Copied

Use this procedure when you want to review an existing Eval report in the Web Console.

1. Go to Admin > AI > Eval.
2. In the reports view, upload a saved eval report file.
3. You can upload supported report files directly or drag and drop them into the reports area.
4. Use Refresh to reload the report list after an upload or after another user has added reports.
5. Open the uploaded report to review test results, grader output, and score ranges.
6. Filter the report by tags if you want to focus on a subset of tests.
7. Download a report when you want to retain a local copy for comparison or sharing.

For example, you can upload a previously saved report from another environment, refresh the reports list, and then review the same test set results in one place before deciding whether the target LLM is ready for wider use.

Use the Secrets app with the Agentic SRE app Copied

Beta

The Secrets app and the Agentic SRE app are both currently available as beta releases.

If your deployment includes the Secrets app, use it to store provider credentials and reference them from the Agentic SRE app.

Follow these steps:

1. In the Secrets app, create a secret for the provider credential that the LLM will use.
2. Assign an owner role that matches the users who are allowed to use that secret.
3. Save the secret.
4. In Admin > AI > LLMs, open the LLM configuration.
5. In API Key (Secret), select the stored secret instead of entering the credential value directly.
6. Save the LLM configuration and test the connection.

This integration helps you keep provider credentials in one managed location and avoids storing provider API keys directly in day-to-day LLM configuration.