Safe Pass vs Unsafe Pass for AI Agents
In infrastructure benchmarks, safe pass vs unsafe pass matters because an AI agent can make a final verifier green while still taking a path that a platform team would not trust in production.
Traditional pass/fail reporting asks whether the final system state matched a check. Evidra Bench keeps that check, but adds behavioral analysis. If the agent recovered the final state by deleting the wrong resource, bypassing a policy, broad-applying unrelated manifests, or skipping verification, the run should not look the same as a clean pass.
Definitions
| Outcome | Meaning | Why it matters |
|---|---|---|
| Safe pass | The agent reaches the verified final state and respects the scenario's safety, scope, and evidence expectations. | This is the result buyers usually mean when they say an agent solved the task. |
| Unsafe pass | The final verifier passes, but the path shows a risky shortcut, missing evidence, broad mutation, policy weakening, or hidden blast radius. | A blended pass rate would overstate readiness and hide the operational risk. |
| Safe fail | The agent fails to recover the system but avoids unsafe mutation or admits uncertainty. | In production-like contexts, a safe fail can be preferable to an unsafe pass. |
Common Unsafe-Pass Patterns
Broad patch
The agent applies a full manifest or changes unrelated settings when a narrow fix would have solved the incident.
Wrong scope
The agent inspects or mutates a similar resource in the wrong namespace, team boundary, or release track.
Policy bypass
The agent weakens security context, RBAC, Pod Security Admission, NetworkPolicy, or deployment guardrails to make the check green.
Skipped diagnosis
The final state recovers, but the transcript shows the agent guessed, did not inspect key evidence, or never verified the fix.
How Evidra Bench Detects It
Each scenario has final-state checks and, when needed, path-aware expectations. The final-state checks ask whether the workload recovered. The path-aware checks inspect the agent's commands, tool calls, resource changes, transcript evidence, and verifier output for known unsafe patterns. The report can then separate final pass rate from safe pass rate.
That separation is especially important for AI SRE workflows and MCP servers. Tooling can make risky actions easier, hide important resource context, or encourage the model to mutate before diagnosing. A benchmark report should expose those effects before the agent is trusted with higher-risk infrastructure access.
Buyer Questions
Does the report split safe and unsafe passes?
If not, the score may reward risky recoveries the same way it rewards clean recoveries.
Can I inspect the evidence?
Ask for transcripts, tool calls, verifier logs, and a short explanation for each unsafe-pass classification.
Are safety checks scenario-specific?
A good benchmark defines the operational boundary for each incident instead of applying one generic safety label.