How do I tell whether my AI agent is getting worse over time?
Last updated
Watch for the qualified people in your company reversing what it does more often, and compare that against something that should not have moved. One signal is never enough, and a rate on its own tells you nothing.
The hard case is the agent that gets worse while every check still passes: every line it relies on is current, every one has an owner, every output is allowed. Nothing breaks, and it is still drifting.
Five signals, none conclusive alone
- Reversals by people qualified to make them. A deal desk lead undoing a price, legal rewording a term. The main signal — and you decide who counts as qualified, because a junior rep undoing an expert's work is noise.
- Escalations moving, in either direction. Fewer asks can mean the agent has learned, or that it has stopped asking. Falling escalations with rising reversals is the worst combination there is.
- Near misses climbing: things allowed only because a line happened to be current that day.
- What it writes changing shape: longer, more certain, more absolute.
- Somebody saying it feels off. Worth writing down even with nothing to measure it against.
Reading them honestly
- Compare against something that should not have changed. A rise across every agent at once is about your market or your data, not this agent.
- Group before you count. Four bad emails with the same wrong phrase are one problem with one cause. Fix the line it came from rather than demoting the agent four times.
- Let explained failures fade, and unexplained ones stay. A problem nobody understood has not gone away because it stopped appearing.
What to do about it
Tighten what the agent may do, alert somebody, and let a person decide. Drift is an inference from a pattern, and an agent halted on an inference is how a control becomes something people work around.
And never turn it into a judgement about a person. The moment reversals become a way to grade the rep behind an agent, people stop reversing things and the main signal disappears.
What we have built
Reynforce records what each agent asked, relied on and was told, which is the raw material for every signal above. Reporting drift from it is designed and not built yet, and this page will say when that changes.
/what-should-an-ai-agent-be-allowed-to-promise.md covers how permissions open and close.