Visa’s AI shopping benchmark: Models fail 42.8% of tasks, Google and Meta lead

How Visa tested AI shopping agents

Visa released results from its Agentic Commerce Benchmark, the first evaluation of AI models on shopping tasks. The benchmark tested 22 models from 12 labs across 15 scenarios, including purchasing electronics and personal care appliances. Models were required to clarify missing details, such as budget or technical specifications, before deciding whether to buy or skip an item. To pass, an AI had to meet all hidden requirements in a scenario without making incorrect purchases.

Models struggle with reliability and precision

The benchmark recorded a 42.8% failure rate across all models. Performance varied widely: the best model passed 82.2% of attempts, while the worst managed only 28.9%. Google and Meta’s AI agents tied for the top score at 82%, followed by a Claude model at 76%. An OpenAI agent ranked sixth with a 64% pass rate.

Models that proactively asked questions to uncover hidden buying factors performed significantly better, passing 70.2% of tasks, compared to 44.4% for less inquisitive models. Only 34.2% of models passed all three attempts in every scenario, underscoring inconsistency as a key challenge.

In over half of failed runs (54%), the AI proceeded with a purchase despite failing to meet the customer’s requirements, revealing a tendency toward false confidence.

Consumer trust lags behind AI adoption

Visa’s Trust Index, published in September, found that while 72% of U.S. consumers have used AI assistants, only 23% trust them to handle payments. This trust gap highlights a hurdle for startups integrating AI into commerce or fintech workflows.

Separate data from NielsenIQ showed AI shopping adoption crossed 50% among U.S. consumers in September, suggesting growing familiarity—but not necessarily dependability.

Holiday spending reflects inflation caution

Visa’s holiday spending report, released alongside the benchmark, reveals a disconnect between consumer aspirations and behavior. More than half of U.S. consumers prioritized travel and live experiences on their wish lists, but only 15% plan to spend on experiences for others. Meanwhile, 27% intend to focus on physical gifts, up from previous years.

Economic pressures drive this shift. Consumer prices rose 3.7% annually in September, fueled by a 22% surge in airline fares compared to 2025. Only households earning over $200,000 increased travel spending for the holiday season, while one-third of consumers earning under $100,000 plan to switch to lower-cost brands for gifts.

Visa projects holiday sales will grow 4.5% year-over-year, but inflation-adjusted growth is just 1.1%, down from 1.3% in 2025. This aligns with broader consumer caution, as spending expectations reflect higher prices rather than increased volume.

Key takeaways

• Visa’s benchmark reveals AI shopping agents are still unreliable for commerce, failing 42.8% of tasks—though top performers like Google and Meta achieve 82% success. Founders building AI-driven retail or fintech tools now have a clear performance baseline to beat.
• The 54% of failed runs ending in unintended purchases show AI agents struggle with precision, a risk for startups automating shopping or payments without guardrails.
• Consumer trust in AI for payments remains low (23%), signaling a gap for startups to address with transparency or hybrid human-AI models.
• Visa’s holiday spending data shows inflation and economic caution are reshaping retail, with most shoppers prioritizing physical gifts over travel—useful for startups forecasting demand.

Related on Lazyfounder

• Google
• Meta
• OpenAI
• More ai news