> ## Content Index
> Fetch the complete content index at: https://thepaymentsoperator.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Visa's Agents Failed 43% of the Time. In Half the Failures, They Bought Anyway.
- URL: https://thepaymentsoperator.com/visas-agents-failed-43-of-the-time-in-half-the-failures-they-bought-anyway/
- Published: 2026-11-09T09:34:00.000Z
- Updated: 2026-11-18T11:14:09.000Z
- Description: Visa tested 22 models as shoppers. The number that matters is what they did after they failed.
- Author: Armando

*The headline was the failure rate. The number an issuer should read twice is what the agents did after they failed.*

Visa published something on 5 November that I did not expect a card network to publish, and almost every account of it reported the wrong number.

The Visa Agentic Commerce Benchmark tested 22 AI models from 12 labs across 15 shopping tasks — buying a television, a laptop, a personal care appliance. Models passed 57.2% of runs. The press took the complement and ran it as the story: AI shopping agents fail 42.8% of the time.

That is a true number and it is not the interesting one.

Here is the interesting one. **In about 54% of the failed runs, the agent completed the purchase anyway.**

Stop on that. More than half the time an agent got it wrong, it did not hesitate, did not ask another question, did not decline. It bought something — either a product that did not meet the customer's requirements, or one chosen without having uncovered what the requirements were.

There are two ways a system can fail. It can fail by not acting, which costs you a sale. Or it can fail by acting confidently and incorrectly, which costs you a dispute, a refund, a call, a chargeback, and a customer who now distrusts the channel.

The benchmark says this failure mode is the second one, in the majority of cases. That is not a model quality finding. That is a payments finding, and it landed in an industry that was still debating whether agentic commerce was going to be large.

## What the test actually required

It is worth understanding the pass criteria, because they are stricter than most benchmarks and that makes the result more credible rather than less.

To pass, a model had to identify the customer's needs through conversation, uncover every requirement the task defined, and then make the right call — which sometimes meant *declining to buy*, because nothing fit. Visa notes this is the first agentic commerce benchmark that requires both complete discovery and a correct purchase-or-no-purchase decision. You cannot pass by guessing well.

The example scenario Visa describes is realistic in a way that benchmarks usually are not. A customer wants help choosing a television. They mention a brand a relative recommended and a membership with a third-party supplier. They leave everything else unsaid. The agent has to surface the budget, the format compatibility, HDR10+ for family videos, the gaming requirement — none of which the customer thought to mention, all of which determine whether the purchase was any good.

That is what buying is actually like. People do not arrive with a specification. They arrive with a fragment and an assumption that the other party will ask.

And the results track that directly: the five models that uncovered the most hidden requirements averaged a 70.2% pass rate, against 44.4% for the five that uncovered the least. Asking good questions is most of the job.

## The same agent, twice, two answers

The second finding is the one that will make rules hard to write, and it got no coverage at all.

Each model got three attempts at each scenario. All three attempts passed only **34.2%** of the time.

Read that carefully, because it is not the same as a 57% pass rate. It means that for roughly two thirds of model-and-scenario combinations, the outcome changed between attempts. The same agent, given the same task, sometimes passed and sometimes failed. Visa's own description of the example scenario notes that models often failed when repeating a scenario they had already passed.

I have spent twenty years in systems where the first question after any incident is *can you reproduce it*. A system whose output changes across identical inputs is not a system we know how to operate. It is not that it is unreliable — plenty of things are unreliable in known ways, and we build around them with retries, limits and reconciliation. It is that it is unreliable in an unpatterned way.

And consider what that does to a dispute.

A cardholder calls and says the agent bought the wrong thing. Under any of the frameworks the networks have published, the defence is the audit trail: here is what you authorized, here is what the agent recorded, here is the mandate. But an audit trail of a non-deterministic process tells you what happened once. It does not tell you whether it was supposed to happen. The same instruction, run again, might have produced the right purchase — and that fact is not evidence of anything, for either side.

We are being handed better records of a process whose correctness cannot be established by replaying it. That is genuinely new, and nothing in our dispute machinery was built for it.

## Somebody else's margin decision sets my error rate

The third finding is the one I cannot stop thinking about.

The five most expensive models per run passed **67.1%** of runs. The five cheapest passed **50.2%**.

That is a seventeen-point spread in purchase correctness, and it correlates with what the model costs to run.

Now trace who makes that choice. Not the cardholder — they will never know which model served them. Not the merchant — the agent arrives at the checkout already carrying a decision. Not the issuer, and certainly not us.

The choice belongs to whoever operates the agent, and it is an infrastructure cost decision. A platform deciding between a cheaper model and a better one is optimising its own gross margin. The externality of that decision is a seventeen-point difference in how often the purchase is wrong — and the cost of a wrong purchase does not land on the platform. It lands on the merchant as a refund, on the acquirer as a chargeback, on the issuer as a provisional credit, and on a contact centre as a call.

We have seen this structure before. Any time the party choosing the control level is not the party absorbing the loss, the control level drifts down. That is not cynicism about AI companies; it is what happens by default, to everybody, absent a rule that reconnects the two.

So here is the uncomfortable question, and I do not think anyone has asked it out loud: should the model tier behind an agent purchase be disclosed in the transaction?

It sounds absurd when you first say it. It stops sounding absurd when you notice it is exactly what we do everywhere else. We disclose whether a transaction was authenticated. We disclose the entry mode. We disclose whether a credential was stored. Every one of those fields exists because it predicts risk and because risk has to be priced and allocated.

If inference tier predicts error rate at seventeen points, it is a risk signal, and risk signals belong in the message.

## The gap consumers already priced

One more figure, from Visa's Trust Index in September: 72% of US consumers had used an AI assistant. Only 23% trusted one to handle a payment on their behalf.

A fifty-point gap between using the thing and letting it pay.

The usual reading is that consumers are behind and will catch up with familiarity. Put it next to the benchmark and a different reading appears: the public's instinct is roughly calibrated. They are using agents, finding them useful, and declining to hand over the card — which, given a 42.8% failure rate where more than half the failures still complete a purchase, is a defensible position rather than a backward one.

That 23% is not an education problem. It is a performance assessment, and it will move when the performance moves.

## Credit where it is due

I want to say plainly what Visa did here, because it is unusual.

Visa sells agentic commerce infrastructure. Intelligent Commerce is its product line for exactly this market. And Visa published a benchmark showing that the agents meant to run on that infrastructure fail nearly half the time and buy anyway in most of those failures.

The company is explicit that the benchmark is separate from the product portfolio, and it frames the result carefully — that this does not mean agents are unready for commerce, but that consistency and reliability remain challenges. It has shared the benchmark with model providers for feedback and plans to extend the testing into checkout, payment and order placement.

You could read that as managed messaging. I read it as a network behaving the way a network is supposed to: setting a measurement standard for a category it also sells into, and publishing the uncomfortable result rather than waiting for somebody else to.

That is the opposite of what the last two years of announcements have mostly been. It deserves to be noticed, and it deserves to be copied.

## From a Panamá desk

Everything in the benchmark was tested in English, on US retail scenarios, buying US consumer electronics.

I do not say that as a complaint. It is a first round, and you start somewhere. But it does mean the published failure rate is the rate under the most favourable possible conditions: the language these models were trained on most, the product catalogues best represented in that training, the review ecosystems richest in data.

Now run the same test in Spanish, on a regional retailer, with local brands, local sizing, local warranty terms, instalment options and a shipping cost that depends on whether the address is inside the city. I have no data on what that number would be. Neither does anybody else, which is exactly the point — and I would be surprised if it were better than 57.2%.

That matters more here than it would in a large market, because the agent traffic reaching our BINs will be cross-border by default. Our cardholders will be buying from US merchants, through agents built for US scenarios, in a language and a catalogue the agent handles well. Then, later, the same agents will be pointed at regional commerce, and the error rate will be whatever it is, discovered by our dispute teams.

So the regionally specific action is small and worth doing: somebody should run this benchmark, or something like it, in Spanish, on Latin American catalogues. Visa has published the method. A regional association, a university, a processor — any of them could do it. The number would be ours, it would be citable, and it would give this region something it currently does not have in any agentic commerce conversation: data of its own.

## My judgment

The 42.8% will improve. Models get better quickly and this is the worst these agents will ever be, which is the standard and correct thing to say.

The 54% is a different kind of problem. A system that fails by completing the transaction is not primarily a capability problem — it is a design choice about what to do under uncertainty. An agent that declined to buy when it had not resolved the requirements would convert less and dispute less. An agent that buys anyway converts more and disputes more. Both are shippable products. Nothing in the current rules makes the second one more expensive to ship than the first.

That is the gap, and it is the same gap I have been writing about all year under different headlines. The networks have shipped credentials, tokens, intent data and now measurement. What has not been shipped is a rule that makes the party choosing the failure mode carry the cost of it.

Until that exists, the economically rational agent is the one that buys when unsure. And we are the ones who find out.

## Three questions for the people who own this

**1.** If 54% of an agent's failures end in a completed purchase, what does that do to your dispute volume forecast for agent-initiated traffic — and does your forecast currently assume agents fail by abstaining?

**2.** The same agent passed and failed the same task across retries two thirds of the time. When a cardholder disputes and the merchant produces a mandate as evidence, what does your adjudication do with a record of a process that would not reproduce?

**3.** Inference tier correlated with a seventeen-point spread in getting the purchase right. Should that be disclosed in the transaction, the way authentication and entry mode are? If not, what is the principled reason — and who is carrying the cost of the difference today?

Reply to this email with a line or two. The sharpest answers will shape the next issue.

## Sources

**Visa, 5 November 2026** — the Visa Agentic Commerce Benchmark: 22 models from 12 labs across 15 shopping challenges; a 57.2% overall pass rate; best 82.2% and worst 28.9%; the five models uncovering the most hidden requirements at 70.2% versus 44.4%; the five most expensive models per run at 67.1% versus 50.2% for the cheapest; all three retries passing only 34.2% of the time; roughly 54% of failed runs still ending in a purchase; the Superintelligence Sweep scenario; Rubail Birwadker's comments; and the September Trust Index figures of 72% and 23%. [Link](https://investor.visa.com/news/news-details/2026/Visa-Releases-First-of-Its-Kind-Industry-Benchmark-for-AI-Shopping-Almost-Half-of-Test-Runs-Fail/default.aspx?ref=thepaymentsoperator.com)

Every figure above is Visa's. The reading of the 54% as a failure-mode finding rather than a capability finding is mine, as is the argument about inference tier as an undisclosed risk signal. Visa makes no claim about either.