Review Failure Modes Before an Agent Acts Autonomously

Start with an event you can investigate

Consider a support agent that can repeat an account action after receiving an ambiguous tool response. “AI risk: high” does not explain the problem. A useful scenario is narrower: the service times out after an action succeeds, the agent retries, and a second action reaches the customer.

The NIST Measure playbook discusses assessment and uncertainty. The register below is a proposed review aid, not a calibrated scoring model or a legal conclusion.

Keep the dimensions separate

Dimension	Question	Evidence to seek
Consequence	What could the repeated action affect?	The actual tool behavior and affected process
Likelihood	How often can the triggering condition occur?	Controlled tests and relevant operational observations
Uncertainty	Which assumptions have not been tested?	Explicit gaps in data or system knowledge
Mitigation	What interrupts the failure path?	A demonstrated approval, deduplication, or stop mechanism
Residual issue	What remains after the control?	Recovery limits and unresolved decisions

Do not invent a probability or loss estimate merely to fill a risk matrix. Unknown likelihood can coexist with a potentially serious consequence.

Test the proposed control

A policy saying “a person can intervene” is not the same as demonstrating intervention before the action occurs. Test the relevant permissions and failure path in a controlled environment. Record the version, test conditions, result, and who reviewed the evidence.

These are engineering risk questions. Whether a particular event creates a legal duty or liability needs a separate source- and fact-specific assessment.

Make a decision that can be revisited

Document the scenario, proposed owner, evidence, remaining uncertainty, and reason for the release or escalation decision. If autonomy, tools, or affected users change, revisit that reasoning rather than carrying over the earlier label.

Carver’s assessment may identify potential regulatory issues alongside this work. It does not supply a validated numerical risk score or make the organization’s risk-acceptance decision.