agentic commerce

Last verified: 7 November 2026. Benchmark figures are from Visa’s announcement. Model rankings are from Fortune.

A shopper asks for a TV. The agent finds a well-reviewed 65-inch set at a fair price and buys it. It never asked whether the TV needed to work with a child’s games console, or whether the shopper needed a 30-day return window. The purchase looks reasonable. It’s also wrong.

That’s the failure Visa set out to measure. On 5 November 2026, Visa published results from its new Agentic Commerce Benchmark, which put AI shopping agents built on 22 models through 15 shopping tasks. The agents failed 42.8% of test runs. In more than half of those failures, they decided to buy anyway.

Key Takeaways

• AI shopping agents failed 42.8% of Visa’s 990 test runs, with model pass rates ranging from 28.9% to 82.2%.
• Requirement discovery was the clearest dividing line. Models that uncovered the most hidden needs passed 70.2% of runs, against 44.4% for those that uncovered the fewest.
• 54% of failed runs still ended in a purchase decision, so many agents acted without enough information.
• Consistency is the bigger problem. All three attempts passed for only 34.2% of model and scenario pairs.
• Cheaper per run isn’t cheaper per success. Once wrong purchases have to be unwound, a higher pass rate can outweigh a higher model price.
• Consumers want control as well as accuracy. In FIS’s U.S. survey, half want to approve every AI transaction and 58% want a refund guarantee.
• Visa’s first round didn’t test checkout or payment. It measures discovery and decisions, not payment readiness.

---

Quick Navigation

---

What Visa actually asked AI shopping agents to do

The first round tested 22 AI models from 12 labs on 15 shopping scenarios, with three attempts per model per scenario. The products were everyday purchases: a TV, a laptop and a personal-care appliance (Fortune describes it as a hair appliance).

The design choice that matters is that the shopper didn’t say everything up front. According to Fortune’s report, the shopper was itself an agent, given a list of required attributes, which volunteered only some of them. For a TV, Visa’s example requirements were budget, format compatibility, HDR10+ support and gaming capability. Fortune’s examples included compatibility with a child’s console, a 30-day return window and Dolby Vision.

A run passed only if the agent did two things. It had to uncover every requirement defined for that task, and it had to make the correct decision: buy a suitable product, or buy nothing if nothing fit. Visa says it added the second condition “to prevent lucky guesses”. An agent that happens to pick the right TV without establishing why still fails.

Visa says this is the first agentic commerce benchmark to require both. It’s a strict rule, and the right one.

What the first round didn’t test matters just as much. Visa’s release says it “will extend testing to checkout, payment, and order placement” in future rounds. So the “purchase” in these results is the agent’s decision to buy, scored by the benchmark, not a completed order or payment. Visa’s release doesn’t describe the product catalogue or execution environment further, and the whitepaper it mentions wasn’t linked when we checked.

How often AI shopping agents failed, and what each number measures

Visa’s figures are easy to misread, because several of them use different denominators.

Measurement	Result	What it actually counts
Overall pass rate	57.2%	Share of all test runs that passed
Overall failure rate	42.8%	Share of all test runs that failed
Best-performing model	82.2%	That model’s share of runs passed
Worst-performing model	28.9%	That model’s share of runs passed
All three attempts passed	34.2%	Share of model and scenario pairs where every attempt passed
Failed runs ending in a purchase	54%	Share of failed runs, not of all purchases
Top five at uncovering hidden requirements	70.2%	Average pass rate of those five models
Bottom five at uncovering hidden requirements	44.4%	Average pass rate of those five models
Five most expensive models per run	67.1%	Average pass rate of those five models
Five least expensive models per run	50.2%	Average pass rate of those five models

The structure implies 990 runs in total: 22 models × 15 scenarios × 3 attempts. That’s our arithmetic, not a figure Visa published, but the reported percentages fit it. Each model had 45 runs, so 82.2% is 37 passes and 28.9% is 13. Across all models, roughly 424 runs failed, and about 229 of those ended in a purchase decision.

Visa says the results aren’t a claim that agents aren’t ready for commerce, only that consistency and reliability remain challenges. That’s fair: 82.2% is capable, and 28.9% is not.

Which models did best

Visa’s release doesn’t name the 22 models. Fortune reported that agents built on Google’s Gemini and on a Meta model tied for first at 82%, a Claude agent placed third at 76%, and an OpenAI agent placed sixth at 64%. The Next Web and others have repeated those rankings, citing Fortune.

Reported rank	Agent	Pass rate	Source
Joint 1st	Agent built on a Google Gemini model	82%	Fortune
Joint 1st	Agent built on a Meta model	82%	Fortune
3rd	Claude agent	76%	Fortune
6th	OpenAI agent	64%	Fortune

Fortune’s text gives version names we couldn’t match to published models, so we’ve left them out. These are agents built on particular model versions at a particular time, not verdicts on each lab. With 45 runs per model, the gap between 82% and 76% is about three runs.

Why the best AI shopping agents asked better questions

The most useful finding in the benchmark isn’t the failure rate. It’s this: the five models that uncovered the most hidden buying factors through conversation passed 70.2% of runs. The five that uncovered the fewest passed 44.4%.

Shopping involves several distinct jobs. Product search finds items that match a query, and recommendation ranks them. Preference discovery works out what the shopper cares about, and constraint identification separates hard requirements from nice-to-haves. The purchase decision comes last.

Search and ranking are mature. Visa’s results suggest the middle two are where agents struggle. A shopper says “a good 65-inch TV under $1,000”. But if the shopper’s child plays on a console that needs a particular HDMI feature, if the shopper owns a streaming box that only outputs one HDR format, or if they need delivery before a date, the best-reviewed TV on the list can still be the wrong one.

That doesn’t mean more questions are better. A shopper facing 15 questions before seeing a TV will leave. The skill is asking the few that change the decision: compatibility with what the shopper owns, hard budget limits, return conditions, timing. Visa’s metric measures requirements uncovered, not conversation length.

When AI shopping agents buy anyway

Visa reported that about 54% of failed runs still ended with the AI making a purchase. Visa describes two ways that happened: the agent recommended a product that didn’t meet the shopper’s identified requirements, or it bought before uncovering all of them.

The first is a matching failure: the agent knew what was needed and chose badly. The second is a judgment failure: it acted without enough information. The second is more worrying, because the agent didn’t recognise its own uncertainty. A careful sales assistant who isn’t sure asks another question. These agents committed.

Be precise about what the 54% means. It isn’t the share of all purchases that were wrong. It’s the share of failed runs that ended in a decision to buy, roughly 229 of about 424. The other 46% ended without a purchase, a better way to fail because nothing has to be unwound.

In this benchmark, a bad purchase decision stopped at the decision, since order placement isn’t tested yet. In a live deployment, the same decision moves through stages, each more expensive to reverse:

Stage	What has happened	What it takes to reverse
Incorrect recommendation	The agent suggested the wrong product	Nothing, if the shopper notices
Purchase decision	The agent decided to buy	Nothing yet, if a human confirms first
Order placed	The merchant has the order	A cancellation, possibly a fee
Payment completed	Money has moved	A refund or chargeback
Product delivered	The wrong item is in the house	A return, shipping, restocking and support time

That’s why the 54% figure matters more as agents gain autonomy. Most shopping assistants today stop at the first or second row. Visa’s later rounds, and the industry’s agent payment products, aim lower down, where every failed decision becomes a return, a support call or a dispute. As Fortune noted, who bears liability for an agent’s mistaken or fraudulent purchase is still an open question.

Why AI shopping agents can’t repeat a good decision

Here’s the number that should worry developers most. Across all models and scenarios, all three attempts passed only 34.2% of the time.

That’s a consistency measure, not overall success. In about two-thirds of model and scenario pairs, at least one of the three attempts failed.

A quick check shows that failures weren’t purely random. If every run were an independent coin flip with a 57.2% chance of passing, all three attempts would pass about 18.7% of the time. The observed 34.2% is higher, which suggests some model and scenario pairs are reliably solved while others fail more often. That’s our arithmetic on Visa’s averages, and it’s only indicative, since pass rates vary by model.

Variation between runs isn’t automatically a fault. As our explainer on LLM determinism notes, reproducibility isn’t correctness. An agent can phrase questions differently or suggest a different but equally suitable TV and still pass. What matters is whether the final decision stays correct.

In a shopping conversation, small differences compound. A different first question leads to a different second one, and the agent may never get round to return policy. Sampling, prompt sensitivity, the order of tool calls and the products a search happens to return can all shift the outcome. Academic work points the same way: the τ-bench paper introduced a pass^k metric because agents that succeed on one attempt often fail on repeats. In its retail domain, pass^8 was below 25%.

A shopper doesn’t experience the average. They experience one run, and can’t tell whether they got the good one.

The cost of an AI shopping agent getting it wrong

Visa also compared model cost with performance. The five models with the highest average cost per run passed 67.1% of runs, and the five cheapest passed 50.2%. Visa didn’t publish per-model prices with the release, and the figures are averages across groups, so they can’t be divided into exact cost comparisons for individual models.

They do raise the right question. The useful measure isn’t cost per run but cost per successful task:

Cost per successful task = total execution cost ÷ number of successful tasks

For a single agent, that’s the cost per run divided by its pass rate. Our guide to LLM cost per task sets out the same method for API work in general.

Here’s an illustrative example with made-up numbers. Agent A costs $0.20 a run and passes 50% of the time. Agent B costs $1.00 a run and passes 80% of the time.

Measure (hypothetical)	Agent A	Agent B
Cost per run	$0.20	$1.00
Pass rate	50%	80%
Model cost per successful task	$0.40	$1.25
Wrong purchases per run (failures × 54%)	0.27	0.11
Total cost per successful task, at $15 to unwind each wrong purchase	$8.50	$3.28

On model cost alone, Agent A is three times cheaper. Once a hypothetical $15 cost to handle each wrong purchase is included, through returns, refunds and support time, Agent B is less than half the price per correct outcome. The $15 is an assumption chosen only to show the effect; real figures vary widely by product and retailer. We’ve also applied Visa’s 54% figure to both agents, though in practice it may differ by model.

The expensive agent doesn’t always win. With human approval before every purchase, failures cost far less and a cheaper model may be right. But API spend is the most visible cost, not the largest.

Shoppers will take advice from AI, not hand over their wallets

Consumer research shows the same divide between help and authority. Visa’s own Trust Index, a Harris Poll survey of 2,065 U.S. consumers carried out in May and published in September, found that 72% had used an AI assistant. Only 23% trusted generative AI to handle payment transactions on their behalf.

The FIS 2026 Consumer Pulse Survey, released on 5 November, explains where that trust comes from. FIS and Kantar surveyed 2,000 U.S. adults online between 16 and 30 September 2026:

• 50% want to approve every transaction an AI assistant makes.
• 57% are comfortable with AI flagging or freezing suspicious charges without asking first.
• 58% chose a refund guarantee for mistakes as a top-three condition for trusting AI with money.
• 56% chose access to a human who can review or override decisions.

Both surveys cover U.S. adults only, and FIS notes its data are unweighted and self-reported. Within that limit, people are more comfortable with AI that protects their money than AI that spends it. What reassures them isn’t mainly accuracy. It’s control, a human to appeal to and a way to undo mistakes.

Even an agent that passed every run wouldn’t meet those conditions on its own. Benchmarks measure accuracy; trust also needs accountability.

What a trustworthy AI shopping agent looks like

The distinction worth holding onto is between being able to buy something and being authorised to. An agent connected to a payment method can technically complete a purchase. Whether it should is a separate decision that belongs to the shopper and the payment system, not the model.

It helps to think about reliability in four dimensions. This is our framework, built from the research above, not an industry standard:

Dimension	Question it answers	Covered by Visa’s first round?
Discovery accuracy	Did the agent understand what the shopper needs?	Yes
Decision accuracy	Did it choose the right product, or decline?	Yes
Execution reliability	Did it carry out the authorised action correctly?	No, planned
Financial accountability	Can a mistake be traced, attributed and reversed?	No

Visa’s first round measures the top two rows. The bottom two depend on agent design and payment infrastructure, and no single safeguard covers them. A confirmation prompt helps, but a shopper approving dozens of purchases a week stops reading them.

A layered design spreads the job across several points. The model handles discovery and decides when it lacks information. The application enforces limits the model can’t override: spending caps, approved merchants and product categories, and approval thresholds above which a human must confirm. The payment layer adds its own controls. Visa Intelligent Commerce, announced in 2025, swaps card details for tokenised credentials that authorised agents can use within consumer-set limits. After the purchase, transaction logs tied to a specific agent identity make it possible to see what happened, and revocable permissions let the shopper switch an agent off.

This is the same principle we covered in AI agent authorization: a login proves access, not permission. An agent should get narrow, logged, revocable rights, enforced where money actually moves rather than in the agent’s instructions.

What Visa’s benchmark doesn’t tell us

Visa’s benchmark is a useful early measurement. It’s still a first round, with clear limits.

• The scope is small. Fifteen scenarios across three product types can’t represent the range of things people buy, from groceries to flights. Three attempts give a rough view of consistency, not a precise one.
• The setup isn’t fully documented. The release doesn’t explain how product information was supplied, how the simulated shopper decided what to reveal, or how hidden requirements were scored. It also doesn’t say whether every model ran in the same agent framework with the same tools.
• Cost per run isn’t defined. It isn’t clear whether “cost per run” includes only model calls or the full agent system.
• There’s no independent replication yet. Visa says it shared the benchmark with model providers for feedback and made it available to try on Visa Payments Frontier. Outside groups haven’t published their own results.
• Real shoppers are messier. A simulated shopper with a fixed list of requirements is consistent in a way real people aren’t. Real shoppers change their minds, contradict themselves and leave mid-conversation.

None of this makes the results unreliable. They show how agents did on Visa’s tasks, not how ready agentic commerce is overall. And for a payments company with a stake in agentic commerce, Visa published notably critical findings.

What retailers and developers should measure next

Visa’s results point to a short list of measurements for anyone building or deploying a shopping agent:

• Pass rate across repeated runs, not single runs. Track how often a scenario passes every time.
• Requirement coverage. Measure how many decision-relevant constraints the agent establishes before it recommends anything.
• Wrong-purchase rate. Separate failures that end in a purchase from failures that end in no purchase. Only the first kind costs money.
• Cost per successful task, including failure handling. Add returns, refunds and support time to model costs.
• Escalation behaviour. Check whether the agent asks for help or declines when it lacks information.

Measure them on your own catalogue. A benchmark shows what an agent can do; your own data shows what it will do.

If an AI shopping agent can spend someone’s money, it needs more than a good average. It has to get the same task right repeatedly, know when it doesn’t know and work inside limits that make its mistakes cheap to reverse. Visa’s first round shows the industry isn’t there yet, and gives it a way to measure progress.

FAQ

What are AI shopping agents?

AI shopping agents are AI systems that find, compare and sometimes buy products on a shopper’s behalf. Unlike search tools, they hold a conversation, weigh requirements and can take actions such as placing an order.

What did Visa’s AI shopping benchmark test?

It tested whether AI models could uncover a shopper’s requirements through conversation and make the right purchase or no-purchase decision. Visa ran 22 models from 12 labs on 15 scenarios, three times each.

Why did AI shopping agents fail 42.8% of Visa’s test runs?

A run failed if the agent missed any task-defined requirement or made the wrong decision. Visa’s results tie failure closely to missing the shopper’s unstated needs.

Why did some failed AI shopping agents still make purchases?

They either chose a product that didn’t meet the identified requirements or decided to buy before finding all of them. That happened in 54% of failed runs.

Which AI models performed best in Visa’s benchmark?

According to Fortune, agents built on Google’s Gemini and on a Meta model tied for first at 82%, with a Claude agent third at 76%. Visa’s own release didn’t name the models.

Are expensive AI shopping agents more reliable?

On average, yes in this benchmark: the five costliest models passed 67.1% of runs and the five cheapest 50.2%. But price alone doesn’t decide value, so compare cost per successful task.

Does Visa’s benchmark test actual payments?

No. The first round covered product discovery and the purchase decision only. Visa says checkout, payment and order placement will be added later.

Can AI shopping agents make purchases without human approval?

Technically, some can, if they’re connected to a payment method. Whether they’re allowed to depends on the limits and approval rules set by the shopper, the application and the payment network.

How can retailers make AI shopping safer?

Use spending limits, merchant and category restrictions, approval thresholds, tokenised credentials, transaction logs and easy refunds. Then measure agents on repeated runs over your own catalogue.

Are AI shopping agents ready for everyday use?

For finding and comparing products, often yes. For unsupervised spending, Visa’s own results suggest caution: even the best model failed about one run in six.

---

Keep reading

Here are the latest posts from the blog.