- Under tool churn, the strongest evaluated combination reaches 42.3% success: budget for maintaining the tool layer, not only the model.
- On stateful CRM tasks, strict success ranges from 64.32% to 83.60% across systems: actions that change a record need human sign-off.
- Rankings shift across models and tasks: a vendor figure does not transfer to your environment, only your own evaluation set does.
The figure that was missing
Until this autumn, no public measurement said what an agent actually completes in an enterprise setting. Demonstrations worked, feedback stayed qualitative, and the available figures came from the vendors themselves. Three weeks of October 2026 produced four independent benchmark sets, published as preprints, all pointing the same way.
Their value is not to crown a winner. It is to give orders of magnitude to set against sales promises, and to name the conditions under which an agent degrades. Those conditions describe the normal life of an information system: tools that change, tasks that modify records, languages that are not English.
When tools change, success falls
Δτ-Bench, published on 25 October 2026, evaluates 1,588 multi-turn tasks across 42 domains while moving the environment through six successive states: tools are deprecated, replaced, replaced again. Under those conditions, the strongest evaluated model-and-harness combination reaches 42.3% success. Most tasks fail.
This is the most useful result for framing a project, because it describes an ordinary situation. A payment gateway changes its interface, a bank connector evolves, a CRM updates its API. An agent wired to those tools does not hold its initial level. The cost of a deployment therefore includes continuous maintenance of the tool layer, not only the choice of model.
On a CRM, the failure band still has to be covered
CRM-Agent-Bench, published on 23 October 2026, measures 555 stateful CRM tasks across nine business domains, scoring outcome, trajectory, operational viability and policy compliance. Across sixteen agent systems, strict success ranges from 64.32% to 83.60%.
A stateful task modifies a record: creating an opportunity, changing a stage, recording a discount. The matching failure band, 16 to 36 points depending on the system, cannot be recovered afterwards. That is the technical argument for a simple rule: the agent prepares, a person approves before writing. The rule has a cost, lower than that of a corrupted sales pipeline.
A vendor figure does not transfer
A third preprint, dated 23 October 2026, crossed four general-purpose harnesses with four models, plus two native systems, across six benchmarks: 111 settings and 52,168 trajectories. It finds that rankings shift by model and by task, and that raw performance and valid-completion rate do not necessarily favour the same pairing.
The authors give no split of responsibility between the model and its harness, and we will not invent one. The practical consequence is enough: a score announced by a vendor was obtained in its own environment, with its own tooling. The only figure that should drive your decision is the one you measure on your own tasks, with your tools and your rules.
The Moroccan case is not measured
LoWAgent, published on 19 October 2026, built six fully localised sites in five low-resource languages and 3,000 task instances. All eight evaluated models do worse than on English, with a sharper drop for open-source agents, worsening further in the presence of cookie notices and ad pop-ups.
This work should be read for what it is. The five languages are English, Hindi, Bengali, Urdu and Amharic. Neither Arabic nor French appears. The benchmark indicates a direction, it does not measure a Moroccan site. If an agent is to operate in French, Arabic or darija, the verification remains to be done at your end: nobody has published it.
What this changes in a discovery phase
These measurements do not say to give up on agents. They say where to put the effort. Three decisions follow: reserve human approval for actions that write a record, put tool-layer maintenance in the running budget, and build an evaluation set from your own cases before comparing suppliers.
That last point is the one most often dropped for lack of time. It is the only one that produces a figure you can hold a supplier to. Thirty real cases, replayed at each model or tool change, cost less than a badly framed pilot and let you decide on facts rather than on a demonstration.
| Benchmark | What it measures | Published result |
|---|---|---|
| Δτ-Bench | 1,588 tasks, 42 domains, tools deprecated then replaced | 42.3% for the strongest evaluated combination |
| CRM-Agent-Bench | 555 stateful CRM tasks, 9 business domains, 16 systems | From 64.32% to 83.60% strict success |
| EGI-BENCH | Eight enterprise benchmarks behind one runner, 14 models | Scores from 39.6 to 65.6 out of 100 |
| LoWAgent | Six localised sites, five low-resource languages | All eight models drop outside English |
Sources & methodology
This insight combines cited publications with editorial analysis. Illustrative examples are not client results. Vendor features and terms may change.