How to know if a discount code actually increased sales
Every tool that hands out a code reports the orders that used it. None of that tells you whether those orders needed the code. The difference is the entire question, and there is one way to answer it that does not rely on an opinion.
Why is the number in my dashboard too high?
A popup, an email flow, or a coupon campaign reports the same thing: orders that used its code. Call that attributed revenue.
Inside that figure are two different people. One was leaving and came back because of the offer. The other had a card out already and took a discount that was waved at them. Both orders carry the code, and the dashboard counts them identically.
The second group is not a rounding error. On a store where most discounted orders come from shoppers who were already converting, the offer is not recovering revenue, it is reducing margin on revenue you had.
What is a holdout test?
Pick a random slice of the shoppers who would normally qualify for the offer, and do not show it to them. Keep everything else the same.
Over time you have two comparable groups. The held-back group tells you the conversion rate you get with no offer at all, which is the thing you otherwise have to guess. Subtract one from the other and what is left is what the discount caused.
The word for that in statistics is a randomised control group, and it is the same mechanism used to test a drug. The only reason it is unusual in ecommerce tooling is that it requires deliberately not selling to some people, which no vendor volunteers to do.
How do I run a holdout test on Shopify?
- Decide what qualifies. The shopper has to be eligible for the offer under the same rules in both groups, or you are comparing two different audiences.
- Split randomly, not by segment. Not by device, country, or traffic source. Any rule that correlates with intent poisons the comparison.
- Hold back enough to measure, not more. A ten percent holdout costs you ten percent of the offer and takes a long time to reach significance. Twenty to thirty percent reaches an answer sooner, and you stop once you have it.
- Count orders, not clicks. The outcome is a completed order tied to a real Shopify order ID. Anything earlier in the funnel can move without revenue moving.
- Wait. The most common mistake is reading the result after two days because the first number looked good.
How much data before the answer means anything?
There is no universal threshold, but there is a useful floor: if the smaller group has fewer than a few hundred sessions, or fewer than a handful of conversions, a single extra order swings the result more than the effect you are measuring.
A practical check is to ask what happens to your conclusion if one order moves from one group to the other. If the answer flips, you do not have a result yet.
What does it mean if the holdout shows no difference?
A holdout that comes back with no difference is the most valuable outcome and the least welcome one. It means the orders were happening anyway and the discount was margin handed over for nothing.
The useful response is not to abandon offers but to narrow them: fewer shoppers, later in the session, or a smaller percentage. Then measure again. A test you can repeat is worth more than a number you believed once.
Where this fits
If you are trying to work out which shoppers you can reach in the first place, the distinction between abandoned carts and abandoned checkouts decides that. If you are looking at where sessions end before any of this applies, start with exit rate.
We build an app that runs this holdout automatically, which is why we care about the question. The method above works without it, and a store large enough to split its own traffic should run it either way.
Common questions
How do I know if a discount actually increased sales?
Hold a random slice of eligible shoppers back from the offer and compare conversion between the two groups. Orders in the group that never saw the offer were going to happen anyway, so the difference between the groups is the part the discount caused.
Why is attributed revenue not the same as incremental revenue?
Attributed revenue counts every order that used the code, including shoppers who had already decided to buy and picked up a discount on the way. Incremental revenue counts only the orders that would not have happened otherwise, which is always the smaller number.
How large does a holdout group need to be?
Large enough that the comparison is not noise. As a rough floor, you want a few hundred sessions in the smaller group and at least a handful of conversions in it. Below that, a difference of one or two orders moves the result by more than the effect you are trying to measure.
What if the holdout shows no difference?
Then the discount was not causing those orders and the margin given away was a straight cost. That is a useful answer, not a failed test, and it is the one most stores never get because nothing in a standard setup measures it.
Try it on your own store
Clinch is free to install. The Free plan costs $0 per month and takes 10% of revenue it actually recovers, so if it recovers nothing you pay nothing.
Install on Shopify