How to make money with a propensity model

Same model, three decisions

Results before theory. Same 10,000 customers, same model, same AUC of —. Three ways to use the score, with a call costing €8 and a margin of €120 per sale:

The rest of this entry explains that math, shows where it works and, above all, where it breaks.

Where this came from

A mentee came to me with a project that, on paper, was finished. It had the stages every course teaches (data, preparation, model, deployment) plus one that almost nobody draws on the diagram: a folder called POLICY.

The model estimated each customer's propensity to buy. The policy folder turned that estimate into action: propensity bands, a threshold, the list of who gets the offer. The business decision lived there, not in the model. His numbers don't appear here; the problem is the same, rebuilt with synthetic data.

The physicist's version of the problem

Before any ROC curve, an order of magnitude. Ten thousand customers, a purchase rate of 6%: some 600 people who will buy, scattered across the customer base. Each contact costs C; each sale leaves a margin of V (what's left after product cost, taxes, commission and discount, not the price).

Calling a customer with probability p of buying has an expected value of p·V − C. It pays off when that is positive:

This is classical decision theory, in the form Elkan (2001) made the standard reference in machine learning: if not calling costs nothing, the optimal threshold is the ratio C/V. With €8 and €120, t* ≈ 0.067. Not 0.5. Nor is it, in general, the Youden point or any other "pretty" point on the ROC curve. With a calibrated score, the Youden point amounts to cutting at the base rate: it bakes in a cost ratio that matches yours only by coincidence. Here, by coincidence, it almost does (the base rate is 6.4% and C/V is 6.7%); change V on the bench and watch the two drift apart. (The cost-weighted ROC point, on the other hand, does lead to the same t*.)

The math hides two conditions. First: p has to be a real probability. Second: people who don't get a call don't buy on their own. Keep both in mind, because that's where the money leaks.

Where the money is

1. Stop paying for calls that don't pay for themselves

Going from the "call everyone" strategy to the C/V rule, the campaign makes — fewer calls and profit goes from — to —. The calls avoided save —; along the way, — sales are lost, sales that together cost more in calls than they brought in. No new sale was invented: the gain comes from no longer paying for customers below C/V.

The default 0.5 threshold errs in the other direction. With this customer base, —. By equation (1), 0.5 would only be optimal if a sale were worth exactly twice a call (V = 2C); here it's worth 15 times as much. The error is a cost error, inherited from the library default. The scikit-learn documentation itself warns that this fixed threshold is almost never the right one.

2. Same team, more sales

With fixed capacity, the question shifts from "who is worth calling" to "who is most worth calling". The threshold becomes a position in the queue, and what decides who gets in is the quality of the ranking at the top of the list (precision in the first 1,000 positions, which is what sets the lift), not the AUC over the whole customer base. Calibration still tells you whether the queue is worth filling.

Sales with 1,000 calls. Lift: —. Change the capacity in Bench 1 and these bars follow.

3. The bar is set by the business, not by IT

The optimal threshold is a function of C and V, not of the model. Move the C and V sliders in Bench 1 and watch the green zone shift while the model stays exactly the same. When the product margin changes, or the team's cost changes, the list has to change too. Try C = 40 and V = 80, or any pair with V = 2C: it's the only kind of world where equation (1) puts the threshold at 0.5. And notice what the bench shows in that case: in this customer base almost nobody scores above 0.5, so not even the right threshold can save a campaign with nobody to call.

Reproduce it yourself

The same steps as Bench 1, in Python. The numbers come out different from the page's because the random generator is different (they move by a few thousand euros from one seed to the next); the conclusions don't.

import numpy as np

rng = np.random.default_rng(7)
N, C, V = 10_000, 8, 120            # customers, cost per call, value per sale

score = rng.beta(0.8, 12, N)        # score calibrated by construction
buyer = rng.random(N) < score       # who actually buys

def profit(t, capacity=None):
    order = np.argsort(-score)
    chosen = order[score[order] >= t][:capacity]
    return buyer[chosen].sum() * V - len(chosen) * C

grid = np.arange(0, 0.5, 0.005)
best = grid[np.argmax([profit(t) for t in grid])]
print(f"C/V rule = {C/V:.3f} | empirical best = {best:.3f}")
print(f"everyone = {profit(0)} | threshold 0.5 = {profit(0.5)} | rule = {profit(C/V)}")
print(f"rule, capped at 1,000 calls = {profit(C/V, 1000)}")