Evaluate model answers

Beta

The ITRS Agentic SRE app is currently available as a beta release. Features, behavior, and documented procedures are subject to change without prior notice during ongoing development.

Use Eval to compare outputs, validate model behavior, and review answer quality before wider rollout.

Use Eval when you want to:

• compare one LLM against another
• test whether a model answers Support questions correctly
• check whether required words or phrases appear in an answer
• review the quality of a model before wider use

Run an Eval campaign and review the results Copied

Admin access

Eval requires access to Admin > AI. Whether Eval is shown, and which eval types are available, can vary by deployment. You need a configured LLM before you can run an evaluation. See Configure language models.

1. Go to Admin > AI > Eval.

2. Click Add Eval to open the Add New Eval wizard.

   

3. In Configuration, select the target LLM, eval type, and number of iterations. Choose the model explicitly rather than relying on a default selection.

4. In Graders, select one or more graders for the run. Graders are grouped by what they measure. A grader marked report only is shown in the report and does not change the test score. A grader marked needs card data applies only to tests that include the detail it checks. A grader marked Needs freshly seeded platform data cannot be selected until that data is available. A grader marked Planned — not built yet or Not planned cannot be selected.

5. In Tests, load tests from one or more test sets, or define custom tests for your scenario. A test set whose required platform data is absent cannot be selected.

6. If your test sets use tags, filter the loaded questions by tag to narrow the scope of the run.

7. For string-content checks, choose whether every expected string must match or the best-matching string determines the score.

8. Select the tests that you want to include, then click Run Eval.

9. Monitor the run status and wait for completion. If the run has to prepare platform data first, the status says that no prompts are sent until that data is queryable, and it names what each data set is doing. Click Cancel Evaluation if you need to stop an active run.

10. Open the generated report to compare scores, inspect test results, filter results by tag, and review answer quality.

11. Repeat the run with adjusted settings if you want to compare models or tune the configuration.

Eval runs are recorded in Audit and are not added to a user’s chat history. See Review AI activity in Audit.

Understand Eval results Copied

After an Eval run completes, open the generated report to review the results in detail.

Example use case:

Use Eval to validate Support answers before rolling out a new LLM.

Example test prompt:

Can Geneos monitor Linux and Windows hosts?

Example expected answer:

Yes. Geneos can monitor both Linux and Windows hosts. The answer should clearly mention both operating systems and provide a concise explanation.

Example expected strings:

• Linux
• Windows

This type of test checks two things at the same time:

• whether the answer is broadly correct and helpful
• whether it includes the exact terms that the answer must mention

Use the report to:

• Review the Eval type, the LLM, and the graders configured for the run.
• Review the Test score. This is the range across graders that measure answer quality. Graders that report duration, token usage, or consistency are shown and do not change this score. A stored report from before that split is labelled Test score (ungated) and is not comparable with a gated score.
• Compare the score ranges for each configured grader.
• Read Grader coverage when it is shown. It names graders that scored no tests, and it says an empty column is not a quality signal. Those graders did not lower the score.
• Read Seeded data when the run used platform data. If this run started that data and it is still young, tests that look back over time can score low because there is little history.
• Read Decision path when it is shown. This is the share of tests where both the route and the tools were right on every iteration. It is not the same figure as the average of the Routing column and the Tool Selection column.
• Open individual tests to review prompts, expected answers, and returned answers.
• Filter the report by tags when you want to focus on a specific group of tests.

When you open a test answer, the view can include:

• Expected and Actual panels so you can compare the target answer with the returned answer.
• Answer Reasoning when the run stores reasoning for that answer.
• Grader tabs such as Accuracy, String Content, and other configured grader types. An older report can still label Accuracy as LLM.
• A percentage score on each grader tab. This percentage shows the score returned by that grader for the selected answer. Higher percentages indicate a closer match to the grader’s criteria.
• Structured feedback cards that explain why the answer received that score, for example under criteria such as correctness, completeness, clarity, relevance, or similarity.
• An Overall summary that helps you decide whether the answer is acceptable for your use case.

Use the grader tabs as follows:

• Accuracy — Shows whether the answer matches the expected answer. On an older report this grader can still be labelled LLM.
• String Content — Shows the score for expected-string matching. Use this when your test requires specific terms, values, or phrases to appear in the answer.
• Routing — Shows whether the request was routed to the expected agent.
• Tool Selection — Shows whether the expected tools were used.
• Grounding — Shows whether the answer stays within the reference and does not add facts that the reference does not support.
• Retrieval — Shows whether document search found the sources the test names. A turn that did not search is N/A.
• Tool Args — Shows whether each tool was called with the expected arguments, such as the entity, filters, and time window. A tool that was not called is N/A, because Tool Selection scores the missing call.
• Safety — Shows whether the answer resisted instructions that try to override its rules, and whether it refused a request that is out of scope. Only tests written for this check are scored.
• Context Carry-over — Shows whether later turns still query the entity from earlier in the conversation without the prompt repeating it. Only multi-turn tests are scored.
• Output Modality — Shows whether the answer included the expected view, such as a metric chart, severity timeline, log grid, or trace waterfall. Those views are described in Explore telemetry with Insights.
• Faithfulness — Shows whether claims in the answer match what the tools returned on that turn. A turn that called no tools is N/A.
• Clarification — Shows whether a clarifying question was aimed at the real ambiguity. A turn that did not ask is N/A.
• Completion, Step Count, and Trajectory — Show whether the turn finished, how many tool calls it used, and whether it repeated the same call. These are report-only.
• Clarity and Transparency — Show whether the answer is direct, and whether it says where the information came from. These are report-only.
• Duration, Token Usage, TTFT, and Consistency — Show speed, token use, time until the first text, and agreement across repeated runs. These are report-only and do not change the test score.

Entity Resolution, Temporal, Numeric Invariant, and Error Recovery read live platform data. They stay unavailable, with the label Needs freshly seeded platform data, until that data is ready. Entity Resolution checks that the turn resolved the expected entity. Temporal checks that the time window sent to the tools is as wide as the question. Numeric Invariant checks that figures in the answer agree with the figures the tools returned. Error Recovery checks what the turn did after a platform call failed on purpose.

Available Eval types and graders depend on the enabled capabilities. By default, production deployments provide Support evals. Agent-targeted evals for SRE, Insights, and RCA are available only when enabled for the deployment.

If a grader does not apply to a test, the result is shown as N/A instead of a percentage. An N/A result does not indicate a failed test, and it is not included in the score.

For example, if a Support eval answer shows Accuracy 68% and String Content 100%, you can interpret the result as follows:

• The answer included the expected strings successfully.
• The answer was only partially aligned with the broader expected quality or meaning.

Use this combination of scores and feedback to decide whether you should adjust the prompt, refine the test, or compare the same test against another LLM.

Upload and review Eval reports Copied

Use this procedure when you want to review an existing Eval report in the Web Console.

1. Go to Admin > AI > Eval.
2. In the reports view, upload a saved eval report file.
3. You can upload supported report files directly or drag and drop them into the reports area.
4. Use Refresh to reload the report list after an upload or after another user has added reports.
5. Open the uploaded report to review test results, grader output, and score ranges.
6. Filter the report by tags if you want to focus on a subset of tests.
7. Download a report when you want to retain a local copy for comparison or sharing.

For example, you can upload a previously saved report from another environment, refresh the reports list, and then review the same test set results in one place before deciding whether the target LLM is ready for wider use.