A/B testing is great for finding the better of two variants. It is bad at finding the things that were never variants in the first place. Here are three quiet conversion killers I see all the time, none of which would ever show up in a Convert or VWO test report.
1. The slow tap-to-paint cycle. A customer taps your “add to cart” button. The button shows no visual feedback for 600 milliseconds while a third-party app debounces. The customer taps again. Now the cart has two of something they wanted one of. They get confused, remove one, get confused, abandon.
Your analytics layer shows this as a normal session with one add-to-cart and a remove. Your A/B testing tool can’t test a fix because there’s no variant to compare. The only way to find this is to watch someone use the site on a slow phone. Almost nobody does.
2. The trust drop in the second cart. Returning customers see their saved address from a previous order. New customers see a friendly empty checkout. But returning customers who haven’t shopped in six months sometimes see their old address but in the format of two years ago — before they moved, before the apartment number was added correctly. They start to wonder if you have the rest of their data straight. They abandon.
An A/B test could never surface this. A heatmap won’t show it. The only signal is a slightly elevated abandon rate among 6-12 month returning customers, which most founders don’t even segment for.
3. The Klaviyo SMS opt-in that fires at the wrong moment. A customer is reading your refund policy because they’re about to buy a $200 item and they want to know they can return it. Klaviyo’s SMS popup fires because they’ve been on the site for 90 seconds. The popup covers the refund policy. The customer dismisses it, loses their place, scrolls up, gives up, leaves.
You can’t A/B test this because it’s context-dependent. The fix isn’t a new popup design — it’s suppressing the popup on legal pages entirely. Which requires somebody to think about it, not test it.
The pattern across all three: the bug is in the seam between systems. Between the platform and the app. Between the customer’s memory and your data. Between marketing’s opt-in goals and the buyer’s reading flow. A/B testing only finds the kinds of bugs that fit within one system at a time.
The way to find these is the unglamorous one: watch real customers use the site on a real phone, segment your funnel data by lifecycle stage, and stop treating “ran an A/B test” as synonymous with “did CRO.” The two aren’t the same.