Retraining is easy. The gate before it ships is the work
We refresh models on recent data at a cadence that matches how fast your world actually moves. Every new version is measured against the one running in production. If it does not win, it does not ship.
In short: We retrain on recent data at a cadence set by how quickly your inputs and your business change, then validate each candidate against the version currently live before anything is switched over. The comparison gate matters more than the schedule.
Four decisions, not one button
Teams tend to talk about retraining as a single scheduled job. In practice it is a set of choices, and the schedule is the least interesting of them. What data goes in, what the candidate gets compared against, who signs off, and how you get back to the previous version — those decide whether retraining makes the system better or slowly wears it down. Get them right once and the cadence question mostly answers itself.
What goes into the training data
Recent examples, cleaned and checked for the periods where something upstream misbehaved. Fresh is not the same as good, and a bad month deserves to be excluded on purpose.
How often it runs
Set by how fast your inputs shift. Some systems need a monthly refresh. Others have been fine on the same model for a year and would gain nothing from touching it.
What it is compared against
The version in production, scored on the same frozen evaluation set, on the same cases. Not against a number written down last quarter and not against itself.
How you get back
Old versions and their training data are kept and can be restored. A rollback that takes ten minutes changes how confidently you can try things.
A pipeline on a timer will cheerfully learn from a broken month
This is the failure we see most. Somebody sets up automatic retraining on last month’s data. It works for a while. Then an upstream form changes, a field starts arriving empty, a integration partner alters a code, or a promotion skews three weeks of behavior into something unrepresentative. The pipeline does not know any of that. It trains on what it was given, produces a model that faithfully reflects a period when things were wrong, and pushes it live because that is what it was told to do.
Nobody notices immediately. The job succeeded, the logs are clean, the new version deployed. What changed is the quality of the answers, and that only surfaces when someone downstream starts complaining. By then two more cycles may have run on top of it. Automation is not the problem here. Automation without a check is.
Nothing switches until the new version wins
Every candidate gets scored against the model currently serving traffic, on an evaluation set that does not change between runs. We look at the headline number and then at the breakdown, because an average can improve while a segment you care about gets worse. If the candidate does not beat the incumbent on the measures that matter, it stays on the shelf and the current model keeps running. That is a normal outcome, not a failed cycle.
Where the stakes are high, we go further before a full switch — shadow the candidate against live traffic without acting on its output, or route a small share of real cases to it and watch. The point of all of this is simple. You should never find out that a retrain made things worse by reading it in a support ticket. Pair this with continuous monitoring and the gate has real numbers behind it instead of a hunch.
Setting your cadence
We work backward from change. How often does your product catalog turn over? How often do the people or the rules or the vocabulary shift? A support classifier at a company shipping new features every fortnight ages faster than a document extractor working on forms that have looked the same since 2018. Drift measurements from monitoring give you the honest answer, and it is often slower than teams expect. Retraining more often than your data changes adds risk and cost without adding accuracy. We would rather set a slower schedule with a strict gate than a fast one with none.
What we won’t do
Ship a candidate straight to production without a comparison
Fully hands-off retraining is faster to build and we will not build it. The gate is the part that protects you, and removing it to save a day of work trades a small convenience for a failure mode nobody sees coming.
Sell you a retraining schedule you do not need
Plenty of models are performing fine and would gain nothing from a monthly refresh. If the drift numbers say your inputs have barely moved, we will tell you that, even when it means a smaller engagement.
Promise every cycle produces an improvement
Some retrains come back flat or worse. That is the system working correctly — you learned something and kept the better model. Anyone guaranteeing a gain every cycle is either not measuring or not telling you the whole story.
When did your model last get checked against a newer one?
Tell us what the model does and how your data has changed since it was trained. We will tell you whether a refresh is worth doing yet.
As often as your data changes, and no more. We work it out from drift measurements rather than picking a round number. Some systems need a monthly refresh because the inputs move quickly. Others have run on the same model for a year and would gain nothing from touching it. Retraining faster than your world changes adds cost and risk without adding accuracy.
What is wrong with fully automatic retraining?
Nothing, as long as there is a gate before deployment. A pipeline that trains on whatever arrived last month has no way of knowing that an upstream field was broken for three weeks or that a promotion skewed behavior. It will learn from that period and push the result live. Automation is fine. Automation without a comparison against the current model is the problem.
What happens if the new version is worse?
It does not ship. The model currently in production keeps running and we look at why the candidate underperformed, which is usually informative in itself. A cycle that ends with no change is a successful cycle. We keep previous versions and their training data so you can also roll back quickly if something slips through.
Do you need our production data to do this?
We need access to representative training and evaluation data, which is often production data. Where that is sensitive we work within your environment, use redacted or synthetic samples for development, and keep the handling rules documented. What is workable depends on your obligations, so it is a conversation we have early rather than an assumption we make.
