We build models that score a record — a customer, an account, a machine, an invoice — and return a number you can rank. The model is the easy part. Knowing how often that number is right, and what it costs you when it is wrong, is the work.
In short: These models turn rows of history into a probability for each record. What decides whether one is worth building is the number of times the outcome actually happened, not the number of rows in the table. We report every accuracy figure next to a baseline, on data the model never saw during training.
What these models are asked to do
The shape of the problem is always the same. You have a table where each row is a thing, and columns describe that thing over time. You want a number saying how likely a particular future is for each row. Then you sort by that number and do something about the top of the list.
Churn and cancellation
Which accounts are likely to leave inside a defined window, ranked so a retention team works a short list instead of the whole book.
Demand and volume
How much will be needed, by location and by week, so stock levels and shift plans rest on a forecast rather than last year plus a feeling.
Failure and downtime
Which machine, part or vehicle is drifting toward a fault, read from sensor history, usage hours and past service records.
Credit and payment risk
How likely an applicant or an invoice is to go unpaid, with the reasoning traceable enough to explain to a customer or an auditor.
Customer lifetime value sits in the same family, with one difference. The answer is an amount rather than a yes or no, so the way it gets scored changes and the errors are read differently. Everything else about the build is the same.
Fifty thousand customers can be an eleven-example problem
This is the first thing we check, and it decides more projects than any modeling decision that follows. A table of fifty thousand customers sounds like plenty of data. If eleven of them churned in the period you care about, the model has eleven chances to learn what leaving looks like. Eleven. The other 49,989 rows only teach it what staying looks like, which it already does well enough by default.
So we count the outcome, not the rows. A few hundred clean examples of the event is a workable place to start. A few dozen usually is not, and when that is what exists we say so before anyone writes code. Sometimes the fix is patience: define the event properly, record it consistently, and revisit in two quarters with real history. Sometimes the fix is a wider definition of the outcome that keeps the business meaning intact. Sometimes there is no model here at all, and a simple rule does the job for a fraction of the cost.
Every accuracy figure comes with a baseline
Accuracy on its own is close to meaningless when the outcome is rare. Take a fault that happens to two machines in a thousand. A model that predicts no fault every single time, forever, is right 99.8 percent of the time. It is also worthless. That number is the baseline: what you would score by always guessing the most common answer.
We print it next to every result we give you. If the model cannot beat it by a margin that justifies the build and the upkeep, that is the finding, and you get told it plainly. We also report where the errors fall, because a false alarm and a missed fault cost different amounts and you are the one who knows the ratio. All of it is measured on a slice of history held back from training, so the model is being graded on records it has genuinely never seen.
Leakage: the reason a beautiful score means nothing
A churn model quietly reading the cancellation-reason field will score magnificently. It will also be useless. That field is only populated once the customer has already left, so at the moment you actually need a prediction it is empty for everyone. The model learned to read the answer off the back of the paper.
Leakage is rarely that obvious. It hides in a status flag updated after the fact, a timestamp that only exists post-event, an aggregate computed over the full history including the future, a record that was backfilled during a migration. We audit every input against one question: would this value have been available, with this content, at the moment the prediction needs to be made? Anything that fails gets removed and the model gets scored again, usually lower and finally honest. A clean pipeline through your data integration layer makes this far easier to guarantee.
What happens after it goes live
A model is a snapshot of how the world behaved during the period it was trained on. The world moves. Pricing changes, a competitor arrives, a product line retires, and the patterns the model relies on stop holding. The scores do not announce this. They keep coming, confident as ever, and slowly get worse.
So we ship the checks with the model: input distributions compared against training data, prediction rates tracked over time, and outcomes fed back so accuracy can be measured against reality rather than against a test set from months ago. That feeds ongoing monitoring and a sensible retraining schedule. Budget for it. A model nobody watches is a model nobody should be acting on.
What we won’t do
We won’t build a model on a handful of outcome examples
If your history holds a dozen instances of the thing you want predicted, we will tell you and propose rules, better recording, or a wait — instead of a model that will look confident and be wrong.
We won’t quote accuracy without the baseline beside it
On a rare outcome, a model that always says no scores in the high nineties. Any figure we hand you is paired with what guessing the majority answer would have scored on the same held-out data.
We won’t ship a score nobody can explain
If no one can say why one account ranked above another, the team meant to act on the list stops trusting it within a month. Where the reasoning cannot be traced, we say so before you commit to the approach.
Do you have enough of the outcome to model it?
Send us the table and a clear definition of the event you care about. We will count the real examples and tell you whether a model is worth building, before anyone writes a line of code.
The row count matters far less than the number of times the outcome actually occurred. A few hundred clean examples of the event is a workable starting point; a few dozen usually is not. We count the positive examples before we quote, and we will tell you if the answer is that you should wait and record better data first.
How do you prove the model works?
We hold back a slice of history the model never sees during training and score it there. That result is compared against a baseline that predicts the most common outcome every single time. If the model cannot beat the baseline by a margin worth the build and the upkeep, we tell you rather than dressing the number up.
What is data leakage and why does it matter?
Leakage is a field that only exists because the outcome already happened. A churn model reading the cancellation-reason field looks near perfect in testing and is useless in production, because that field is empty for every customer who has not cancelled yet. We audit each input against one test: would this value have been available, with this content, at the moment the prediction is needed?
Does the model stay accurate over time?
No. Behavior shifts, products change, and scores drift away from reality without any warning sign in the output itself. We ship monitoring on input distributions and prediction rates, and we feed real outcomes back so accuracy is measured against what happened rather than an old test set. Plan for periodic retraining as part of the running cost.
