Retail teams are asked to justify a lot with very little. A conversion rate slipped, a return rate rose, a campaign underdelivered. The meeting that follows usually produces an opinion, a redesign, and a decision made by whoever spoke with most confidence. The tools to answer the question properly have been around for years. What is usually missing is the habit of asking for evidence before spending money.
For a technology-and-society audience the gap is worth naming. The discipline that governs how a business handles customer data should also govern how it changes what customers see: decide the question first, collect the minimum you need, and be able to explain afterwards why you made the change. An experiment is an accountability mechanism that happens to pay for itself.
The default method fails for a structural reason
The usual approach is a before-and-after comparison. Change the page in March, compare April’s numbers with March’s, treat the difference as the effect. It rarely is. Advertising spend moves, a promotion ends, a competitor launches, payday changes who is shopping, a product goes out of stock and buyers substitute something else. Every one of those lands in the same number as the change you made, and none can be separated out afterwards.
Two versions running at the same time, with visitors assigned at random, remove the problem by construction. Both groups meet the same market in the same week. The change under test is the only difference left between the two sets of numbers. It is a small methodological point with a large consequence: it turns a claim into a measurement.
What the platform does on its own, and where it stops
Shopify ships a testing feature, and its documentation states the boundaries precisely. Under Markets > Rollouts there are three rollout types, and the Experiment type compares a control against a treatment. A rollout can change your online store theme, your checkout and accounts configuration, or your product catalogs. Experiments need the Grow plan or higher, Liquid template changes are not supported inside a rollout, vintage themes cannot be tested at all, and headless or custom storefront checkouts are not supported.
That covers a theme rebuild and a checkout configuration change, on a plan many merchants already pay for. It also marks where the native option stops. A price is not a theme setting. A shipping rate is not a catalog field. A single button on a product page is neither. Those are the changes merchants ask for most, and each needs a tool that alters what a visitor sees at the element level.
The framing matters here. The platform is scoping a feature to the surfaces it owns, which is a reasonable thing for a platform to do. Teams that assume more discover the limit after they have already promised a result.
Data minimisation applies to experiments too
- An experiment collects data about people, which means it inherits the same obligations as any other collection. In practice that comes down to a few unglamorous decisions.
- Test what serves the customer’s decision. If the question is whether a shipping threshold converts, the answer does not require storing behavioural profiles for a year.
- Keep the variables minimal. A test that changes price, imagery and copy at once produces a result nobody can act on, and it exposes more visitor segments than the question needed.
- Report uncertainty honestly. A dashboard showing a raw percentage difference with no confidence attached invites a decision the data does not support. Statistical significance exists to prevent that conversation.
- Write down what you will do with each outcome before you launch. A decision rule agreed in advance is the cheapest ethical control in an experiment programme, because it stops the result being reinterpreted later
None of this needs a data science team. It needs the willingness to say no to tests that cannot end in a decision.
The uncomfortable arithmetic of testing
Most experiments do not produce a winner, and any programme that pretends otherwise is selling something. Analytics-Toolkit’s meta-analysis of 1,001 A/B tests found 33.5% with a statistically significant positive result, a mean lift of 15.9% among the winners and a median of 7.5%. Two in three tests return nothing conclusive.
That ratio is the price of the good third. A test also earns its keep in a second way: one that shows no difference closes a question permanently, which is more than most redesign debates manage.
What this looks like in practice
For a merchant, the responsible version of growth work is unremarkable.
- State the hypothesis as a sentence about a metric the customer’s experience produces. Revenue per visitor, average order value, checkout completion.
- Change one variable.
- Set the sample size and the run length before launch, and run whole weeks so a full shopping cycle is covered.
- Judge the outcome on revenue per visitor, so a discount that lifts orders while shrinking baskets does not get recorded as a win.
- Log the decision as well as the result.
For Shopify stores, reliable A/B testing depends on more than showing visitors two different versions of a page. Each visitor needs to see the same version across sessions, and any tested price must remain consistent from the product page through the cart, checkout, order and analytics. Without that consistency, it’s difficult to trust the results.
Shopify A/B testing apps make it possible to run these experiments without building a custom testing system. Elevate A/B Testing, for example, lets merchants test prices, pages, shipping rates, product images, split URLs and checkout changes. Plans start at $49 per month, with a 14-day free trial.
Price testing is often a good place to start. Stores can show two different prices to randomly assigned groups of visitors and measure how each performs against revenue, conversion and profitability. Tests run on a portion of live traffic, and the original price can be restored with a click if results aren’t going in the right direction.
Shipping is the other lever with an outsized effect on margin, and it works the same way. One threshold against another, with conversion, basket size and revenue per visitor reported side by side.
The habit matters more than the software
A retailer running two controlled experiments a quarter will make better decisions than one running none, whichever software it uses. The value sits in the habit: a question written down before the money goes out, a comparison designed so the answer can be trusted, and a record of what was decided.
For organisations that talk about responsible technology, retail is an unusually clear place to start. The customer data is already being processed. The decision is already being made. All that is left to settle is whether it gets made on evidence or on the confidence of whoever spoke last.
The short version
- Before-and-after comparisons measure the market. Run both versions at once with random assignment.
- Shopify’s own rollout feature covers themes, checkout configuration and catalogs on Grow and above. Price, shipping and element-level tests sit outside it.
- Apply data minimisation to experiments: few variables, the minimum data, a decision rule agreed in advance.
- Expect roughly one clear winner in three tests, and count the conclusive negatives as results.
Judge outcomes on revenue per visitor, and record the decision alongside the number.
