When an operations team deploys agents on a high-volume queue, average handle time often rises within the first month. The natural conclusion is that the deployment made things worse.
In many cases the opposite is true. The increase is a measurement effect, and it is common enough to plan for.
Why Handle Time Rises
Agents handle cases that settle at first pass: deterministic, high-volume, rule-governed work with a documented decision. When they absorb those cases, people are left with the harder half. The work that returns to a person is more complex, more ambiguous and slower on average.
The average therefore rises even as the total work completed improves. Fewer cases reach a human, the cases that do are the difficult ones, and a dashboard built around average handle time reports a successful deployment as a regression.
Absorbed, Assisted and Retained Work
The solution is to measure the queue as three populations instead of one. Once agents and people share the work, these three kinds of work cannot be distinguished in the case record unless they are tracked separately.
Absorbed. Work the agent handled end to end, with no human touch.
Assisted. Work a person completed with agent help.
Retained. Work that stayed with a person.
Measured per agent and per intent, this split is the most reliable indicator of whether a deployment succeeded. It is not visible in the case platform, because the case platform records that a case closed and not who or what did the work.
With the three categories in place, a rising handle time on retained work is no longer a concern. The team can see how much work left the human queue, what people now spend time on, and where released capacity went. Capacity that cannot be located was not released.
An Example Comparison
Consider a queue where, before the deployment, every case takes about the same time and is worked by a person. After the deployment, the agent settles the straightforward cases and a person works the rest. The people are not slower. They are doing a different mix of work.
Comparing the two periods on average handle time makes the second look worse. Comparing them on cases completed per hour of human effort, with the absorbed work counted, makes it look better. The data is the same and only the question differs. The question has to be chosen before the deployment, not after the first dashboard shows an unexpected result.
Measuring People and Agents Together
The second component is how each person works with the agent. For every suggestion there are three possible responses: accept, override or ignore.
Recorded per person instead of per team, these responses carry useful information. An override means either the agent is wrong or the person is, and both are worth knowing. A pattern of overrides by experienced people points to a design defect in the agent or in the standard it was built from. A pattern of ignored suggestions from a few users points to adoption and training needs.
The screen below, from the Pyze demo environment, shows this kind of comparison. It sets throughput for adopters of an AI copilot against non-adopters, and it states its own caveat: the cohorts are not fully controlled for claim complexity and experience, so the difference is directional evidence best validated with a controlled rollout. Handling time per claim is similar across the two groups. The gain appears in throughput, which is why one metric is not enough.
Metrics to Track
The following measures avoid the problem:
- The absorbed, assisted and retained split, per agent and per intent.
- Accept, override and ignore rates, per person.
- Retained effort and cycle time on the work people still do, with case mix controlled rather than averaged across a changing population.
- Where the released capacity went, so it is not assumed.
All of these are compared against a baseline taken before the deployment, using the same measurement method. A baseline reconstructed afterwards cannot be verified, so the baseline should be captured first. We cover this in Why Modernization Programs Struggle to Prove Results.
Where to Go Next
- See how AI Effectiveness separates absorbed, assisted and retained work.
- Read AI Discovery and Effectiveness for the full loop from picking use cases to proving the result.
- Read the broader discussion in Why AI Agent Deployments Need Execution Intelligence.