Measured results, reproducible from one command
Every budget picks some people and skips others. Who should it pick?
Retention offers, outreach calls, fraud reviews, follow-up appointments. The money covers a fraction of the population, so somebody chooses the fraction. The usual answer is to rank people by risk and treat the top of the list. Sometimes that works. Sometimes it is worse than picking at random. This page shows both, on real data, and how to tell which case you are in.
Six datasets, eight ways of choosing, one budget slider. Nothing is modelled in your browser: every number was computed in advance by the benchmark and is checkable in the repository.
The idea in one table
If you could see what your action does to each person, there would be four kinds of people. Only one kind is worth the money.
| Kind of person | If you do nothing | If you act | Worth the budget? |
|---|---|---|---|
| Persuadables | bad outcome | good outcome | Yes. They are the entire return. |
| Sure things | good outcome | good outcome | No. They were fine anyway. |
| Lost causes | bad outcome | bad outcome | No. Nothing on offer helps. |
| Sleeping dogs | good outcome | bad outcome | No. Acting does damage. |
Ranking by risk sorts on the first column alone. It puts lost causes at the top, because they really are the highest risk, and it cannot see sleeping dogs at all. Persuadables sit in the middle of a risk list and get missed. Uplift modelling tries to sort on the difference between the two columns instead, which is the thing you actually care about and the thing you can never observe for any individual, because only one of the two ever happens to them.
Every method against random targeting
One line per way of choosing who to treat. Higher is better. The red dashed line is random targeting, the floor: any method below it is worse than picking names out of a hat. Change the dataset to see how completely the picture changes.
Pick a method and move the budget
The same measurement, one method at a time, with its uncertainty band and the list of who actually gets treated. If the band crosses the red line, the method has not been shown to beat random targeting at that budget.
Who is in the treated set
| # | Row | Predicted uplift | Cost |
|---|
Units are row numbers in a public dataset's held-out split. No personal or feature values are shipped to your browser.
Could your own data answer this at all?
Most data cannot, and finding out afterwards is expensive. Two very different questions hide behind one word, "enough":
- Does the intervention do anything? The easy question, and the one most pilots are sized for.
- Is targeting better than picking at random? Much harder, because you now need to tell one group apart from the average rather than from zero.
These are floors, not estimates. They price measuring a ranking you already have, and a real study also has to learn the ranking from the same rows, which costs more. How much better the top group responds is the thing the study exists to discover, so it cannot be known in advance, which is why the answer is a range rather than a number.
The same arithmetic, run on the six datasets above from four summary numbers each and no model at all, gets every one of the six verdicts right: the four that could be ranked on, and the two that could not. That check is a test in the repository, so it cannot quietly stop being true.
Running this on your own data
1. Check you have the right kind of data
You need an intervention that already happened to some people and not others, and a record of what happened to both. Not a customer list. Nothing here, and nothing anywhere, can tell you the effect of an action nobody has ever taken.
Ideally who got it was decided at random. If it was not, the numbers mix what the intervention did with whatever made those people likelier to be chosen, and the tool will say so rather than quietly pretending otherwise.
2. If you have no data yet, size the pilot
Use the calculator above, then run a holdout of that size. One command does the arithmetic:
uv run itx power \
--base-rate 0.10 \
--relative-effect 0.10 \
--budget 0.2It reports the easy threshold and the hard one separately, because sizing for the easy one is the usual mistake.
3. Spend one model finding out if it is worth continuing
Before fitting anything sophisticated, ask whether risk ranking already does the job on your population. It costs a single model:
uv run itx diagnose \
--csv customers.csv \
--treatment got_offer \
--outcome churned \
--features age,tenure,plan \
--categorical plan \
--design randomised \
--outcome-polarity lower-is-betterThe last two have no defaults and are never guessed. The same file labelled "churned" and "retained" gave opposite answers until that was fixed.
4. Only then, run the full benchmark
If the diagnostic says there is room, fit everything and compare it against both baselines with intervals on every number, the way this page does.
uv run itx benchmark \
--dataset hillstromAnd pass --no-tune. The tuning search was measured four times and bought nothing.
What this project found
Risk ranking is neither a trap nor safe
It depends on the data, the difference is worth a change of sign, and nobody can guess which case they are in. On ACIC it destroys value. On Criteo it matches every sophisticated method. Both results came out of the same code.
You can tell which case you are in, cheaply
One outcome model, before any of the expensive work. That diagnostic is the most useful thing this project produced, and it was not what the project set out to build.
The fraud case was rigged against risk ranking, and lost
It was designed so that the highest-risk transactions are the ones where review does least good. Risk ranking still beat every uplift model at every budget. A team that kept its existing model and ignored this repository would be better off.
Most data cannot answer the question
Lenta has 687,029 customers in a real randomised trial and still cannot support a targeting decision. Size alone is not enough, and the calculator above says so before you spend the money.
Every number here has an interval
A result without one is treated as a defect in this codebase. That rule is what kept the conclusions standing through six defects found during the build: every corrected number stayed inside the range already published.
The checks caught real mistakes
Rebuilding every result from a fresh copy found a bug that made part of a table impossible for anyone to reproduce. It was fixed, remeasured, and written up next to the result rather than quietly corrected.
How to read these numbers
What "outcome bought per head" means
It is measured per person in the whole population, not per person treated. That is deliberate: it lets a 5% budget and a 40% budget sit on the same axis and be compared honestly. Treating 5% of people very well can still buy less in total than treating 40% adequately.
Why every number has a range after it
The range is a 95% confidence interval: where the answer would likely fall if you ran the whole thing again on fresh data. A method whose range crosses the random line has not been shown to beat random targeting, whatever its headline number says. Reporting the number alone would make several results on this page look far more certain than they are.
This page shows one split, the repository averages five
The results table in the repository averages five random splits of each dataset. This page shows the first of them, because a slider has to move through a single ranking of actual people rather than an average of five different ones. So these numbers sit near the table's and will not equal them. On ACIC at a 10% budget this split gives risk ranking -0.2434 where the five-split average is -0.1971.
Where the numbers come from
The random line is the closed-form expectation: the budget share times the value of treating everybody. The repository's random row is the average of 200 actual random draws, so the two sit close together and differ slightly.