AI Commerce Agents Hallucinate Order Completions; GPT-4o Performs Worst Among Models Tested
Key Takeaways
- ▸"Journey hallucinations" are a distinct AI failure mode where agents falsely confirm completed transactions, triggering wrong automated workflows and damaging customer experiences
- ▸False completion claims occurred in up to 8.5% of sessions across four models tested, with GPT-4o showing the worst performance
- ▸Reliability degrades significantly under repetition: GPT-4o dropped from 60% accuracy on a single task to below 25% when asked to repeat the same task eight times
Summary
Research presented at ACM UMAP 2026 discovered a new category of AI hallucination in commerce transactions called "journey hallucinations," where AI agents falsely claim orders were completed even when event logs show no transaction occurred. Across 90 real shopping sessions and four foundation models, ungrounded prompting produced false completion claims at rates up to 8.5%, with OpenAI's GPT-4o performing worse than the other three models tested.
Journey hallucinations differ from typical factual hallucinations because they cause immediate downstream business damage: when an AI agent incorrectly claims an order was placed, it triggers automated workflows like cart recovery emails sent to customers who never added items, or post-purchase messaging sent to people who never completed purchases. Testing revealed that reliability degrades sharply under repetition—a GPT-4o retail agent scoring above 60% accuracy on a single task attempt fell below 25% when asked to complete the same task eight times in succession.
The paper, authored by researchers from Rezolve AI Labs and the University of Sarajevo, identifies four specific transactional claims that AI agents most commonly misstate and that have the most direct business impact. The authors recommend implementing a "Trace Test" of five verification checks before deploying any AI agent in commerce scenarios, and provide five critical questions for evaluating vendor solutions.
- The problem can be mitigated by implementing trace verification checks that validate agent claims against actual event logs before responses reach customers
Editorial Opinion
This research exposes a critical vulnerability in how AI commerce agents are validated today: companies test accuracy against isolated scenarios rather than stress-testing repetitive real-world usage patterns. The business damage extends well beyond poor UX—false order confirmations harm customer trust through misaligned automated workflows that customers can't understand. Most troubling is that this failure mode went unnamed until now, suggesting deployed systems are silently generating these errors without explicit monitoring.
