Most machine learning projects do not fail at the modelling stage. They fail in the year afterwards, when nobody can reproduce the training run, nobody owns the model, and nobody notices it has quietly stopped being right. The MLOps lifecycle is the practice that prevents that, and nearly all of its difficulty lives outside the notebook.
This guide walks the lifecycle stage by stage: what you version, what you watch, and what breaks if you skip it. It then covers the part written about least, what changes when you go from one model retrained by hand to thirty across several teams.
What the MLOps lifecycle actually contains
The textbook version is five boxes: data, train, validate, deploy, monitor, with an arrow curving back. Not wrong, just missing everything that makes each box hard.
Data collection and versioning
Versioning data is harder than versioning code and matters more. Code is text, it diffs cleanly, Git was built for it. A training set is millions of rows assembled by a query reading tables that are still being written to, and "the data as of last March" may no longer exist anywhere.
The failure is specific. Someone reports odd behaviour on a certain input, you rerun the query to reproduce the training run, upstream tables have moved, and you get different weights. The bug is now unreachable.
Make every run reference an immutable snapshot: a content-addressed store like DVC, a table format with time travel like Delta or Iceberg, or a dated Parquet dump with the extraction query committed beside it. What matters is that the dataset behind run 412 can be materialised exactly, months later, by someone who was not there. Version the schema too. A column that silently moved from cents to dollars trains fine and serves nonsense.
Feature engineering and training-serving skew
Training-serving skew is the most common reason a model that scored well offline disappoints in production, and it is plumbing rather than modelling.
Features get computed twice. A data scientist writes a transformation in Python against a historical table; an engineer reimplements it in the application language against a live request. They agree on the happy path and diverge at the edges: how nulls are filled, whether an unseen category maps to "unknown" or throws, whether a rolling window is 30 days or 28, whether timestamps are UTC on one side and local on the other.
A feature store exists for exactly this. Its real job is not storage, it is guaranteeing that one transformation produces both the training value and the serving value, and that historical lookups are point-in-time correct: a training row for an event on 3 March carries only what was knowable on 3 March. Join carelessly against a current-state table and you leak the future into training, so offline metrics look excellent and production does not. You do not need a feature store to fix this. You need one implementation of each transformation, called from both paths.
Experiment tracking and reproducibility
A run is reproducible when you can name the commit, the data snapshot, the hyperparameters, the environment and the seed and get the same artefact back. Miss one and you have a record, not a reproduction. Environment is the one people drop: scikit-learn>=1.3 is not pinned, and a container digest is what makes a run repeatable next year rather than next week.
Validation past a single accuracy number
One aggregate metric hides everything worth knowing. A model at 94 percent overall can sit at 61 percent on the segment producing most of your revenue, and the average will never say so. A gate worth having checks:
- Performance by slice, split by segment, geography, device, or any axis where the business behaves differently.
- The incumbent model on the same held-out data, not a number from an older experiment on a different split.
- A trivial baseline. If predicting the majority class scores nearly as well, the model is not earning its operational cost.
- Behavioural tests: assertions about specific inputs and expected directional behaviour, like unit tests on any other function.
- Latency under realistic load, because a model that misses the request budget cannot ship.
Write it as an automated gate. A checklist someone is supposed to remember is not a gate. Pulling the stages together:
| Stage | Version this | Monitor this | What breaks if you skip it |
|---|---|---|---|
| Data collection | Dataset snapshot, extraction query, schema | Row counts, null rates, schema changes, freshness | Training stops being reproducible; schema changes reach the model silently |
| Feature engineering | Transformation code, feature definitions, point-in-time joins | Feature distributions in training versus serving | Skew: strong offline scores, weak production, no obvious cause |
| Training | Commit, hyperparameters, environment digest, seed, dataset ID | Duration, cost, convergence, resource use | You cannot rebuild the model behind a decision someone is questioning |
| Validation | Evaluation set, metric definitions, thresholds | Sliced metrics against the incumbent and a baseline | A model worse for your best segment ships because the average looked fine |
| Deployment | Artefact, serving image, config, rollout policy | Error rate, latency percentiles, traffic split | Rollback becomes a rebuild; a bad model stays live for hours |
| Serving | API contract, input schema, preprocessing code | p95 and p99 latency, timeouts, cost per thousand predictions | The model becomes the slowest dependency and nobody can say why |
| Monitoring | Alert thresholds, dashboards, on-call ownership | Input and output distributions, null and default rates | Degradation arrives as a customer complaint, not an alert |
| Retraining | Trigger conditions, approval path, promotion criteria | Time since last retrain, trigger rate, post-promotion results | Models go stale on a schedule nobody chose, or retrain on broken data |
Deployment patterns: shadow, canary, staged rollout
Do not switch traffic in a single step. Offline validation tells you how the model behaves on data you already have. It says nothing about live traffic, real latency, or the preprocessing code that actually runs in production.
Shadow mode runs the new model on production traffic while the old one still serves every response. Predictions are logged and compared, never used. This catches what offline testing cannot: preprocessing that differs from the training pipeline, features that are stale or missing at request time, latency that only appears at production concurrency.
Canary serves the new model's answers to one to five percent of traffic. This is where you learn whether the decisions have the business effect you expected, which shadow mode cannot tell you because nobody acted on shadow predictions. Fix promotion and rollback criteria as numbers first, and make rollback a config change rather than a deployment.
Staged rollout widens in steps, holding at each long enough to see the metric you care about. If the model influences a slow outcome like a renewal or a repayment, each hold must be long enough for that outcome to appear. Going to full traffic in an afternoon on a metric that resolves in three weeks is a guess, not a rollout.
All three need both models runnable at once with traffic split by configuration. Build that before you need it, or you will be doing a full redeploy to roll back at exactly the moment speed matters.
Batch or real time: decide before you design anything
Batch scoring runs on a schedule, writes predictions to a table, and the application reads them. Cheaper, easier to monitor, trivially retried, and it lets you inspect a whole day of predictions before anyone sees them. It is correct whenever the answer does not depend on the last few minutes: churn scores, lead ranking, demand forecasts, most recommendations.
Real-time inference is needed when the input arrives with the request: fraud decisions on a transaction, pricing on a basket, anything with a user waiting. It costs more, puts a model in your latency budget, and needs fresh feature values in single-digit milliseconds, usually harder than the model itself.
Default to batch. Ask what the user does differently with a prediction that is six hours old, and the honest answer is often nothing.
System health and model quality are two separate monitoring jobs
A model service can be healthy by every infrastructure metric and completely wrong. System health is ordinary service monitoring: latency percentiles, error rates, throughput, saturation, cost. Your existing observability stack covers it.
Model quality is different, because ground truth arrives late or never. You predict churn today and learn in ninety days. You predict fraud and only hear about disputed cases. So quality monitoring cannot wait for labels. What you can watch immediately:
- Input distributions. Mean, spread and null rate per feature against the training reference. Earliest signal, and the one most teams lack.
- Output distributions. If a fraud model flagged 0.8 percent of transactions last month and flags 6 percent today, something changed, confirmed cases or not.
- Prediction confidence. Drift toward the uncertain middle usually precedes a measurable accuracy drop.
- Default and fallback rates. Rising defaults mean an upstream pipeline is failing quietly and the model is scoring on filler.
- Delayed accuracy, backfilled against the prediction actually served as labels arrive.
Monitoring inputs is the cheapest high-value item there. A broken upstream job sending nulls shows up in feature distributions within an hour and in accuracy metrics in three months.
Data drift and concept drift are different problems
Data drift means the inputs changed while the learned relationship still holds. A new market launches, a campaign brings a different demographic, an app release changes which fields get filled. Detection is statistical and needs no labels: compare live feature distributions against the training reference on a rolling window, per feature, with population stability index or a Kolmogorov-Smirnov test. Retraining on recent data usually fixes it.
Concept drift means the relationship itself changed, so the same input now warrants a different answer. Fraud patterns adapt because fraudsters adapt. What counted as a good lead before a repositioning is not one after it. This can happen while input distributions look perfectly stable, so distribution monitoring will not catch it. You need outcomes, which means labels, which means detection is delayed by however long ground truth takes.
Retraining fixes data drift reliably. It fixes concept drift only if the new relationship is already visible in recent labelled data. When the world changed last week and labels take ninety days, retraining now trains on the old world with extra steps. That case needs a human decision and sometimes a feature redesign.
On thresholds: drift tests on a large sample fire constantly if you set them by textbook p-value. Calibrate against your own history instead, because an alert nobody acts on is worse than none. It teaches the team to ignore the channel.
The retraining loop: what triggers it, who approves it
Every team asks how often to retrain. The better question is what should cause one.
Scheduled retraining runs weekly or monthly. Predictable and easy to plan capacity for, and either wasteful or too slow, because the cadence is chosen by intuition rather than any measurement of decay. Still the right starting point, since it proves the pipeline works end to end.
Triggered retraining fires on a condition: drift crossing a threshold, accuracy under a floor, or enough new labelled data. More efficient, and it carries a failure mode worth naming. If the trigger is drift and the drift comes from a broken upstream pipeline, automatic retraining will faithfully train on broken data and promote it. Any automatic trigger needs data validation upstream that can veto the run.
Either way, promotion should be automated and strict: the candidate is compared to the incumbent on a fixed evaluation set with sliced metrics and a threshold it must clear. Decide separately whether promotion needs a human. For a recommendation ranker, automatic promotion behind a canary is reasonable. For a credit decision or anything a regulator can ask about, a named person signs off and that signature is part of the audit record. The engineering is identical. The governance is not.
What changes when you go from one model to thirty
This is where MLOps investment either pays off or is wasted. One model maintained by the person who built it is a different problem from thirty across four teams, and the difference is coordination rather than size.
You need a registry and a named owner per model. With one, the owner is obvious and the current version is whatever is deployed. With thirty, nobody can answer "what is in production, what trained it, who owns it, when was it last evaluated" without a record per version holding the training run, dataset snapshot, evaluation results, stage and owner. Without it you accumulate forgotten endpoints still serving traffic and models whose author left a year ago. The owner matters more than the registry row: a model with no owner will not be retrained, evaluated or decommissioned. It will just keep answering.
Reproducibility stops being optional. Someone will ask why a specific decision was made eighteen months ago, and that question comes from a regulator, a lawyer, an enterprise audit or an incident review. Answering means reconstructing the exact model version and the exact features it saw for that request, which requires logging the model version alongside every prediction. One line of code, and nearly impossible to backfill.
Serving cost becomes a real line item. One model on a modest instance is a rounding error. Thirty, several on GPUs and several kept warm for latency, is a number finance will ask about. The useful questions are unglamorous: which could be batch, which endpoints are provisioned for a peak that happens twice a year, which models still run for a removed feature.
The failure mode changes. With one model you plan for "the model is wrong", and you catch it because its author is watching. With thirty, the dominant failure is "nobody noticed the model went stale". No outage, no error spike, no alert. It gets worse over months while dashboards stay green, because they measure the service and not the predictions. By the time anyone notices, it has been making slightly worse decisions for two quarters, and the cost is invisible in the way that makes it large. The defence is organisational: an owner per model, a review with a date on it, a decommission path, and quality metrics someone is accountable for.
Budget for that. On Denti360, our own live dental practice management SaaS, the planning figure we publish for keeping conventional software current is 15 to 20 percent of the original build cost per year, before new features. ML systems carry all of that plus retraining, data pipelines and the monitoring above.
MLOps best practices for teams without thirty models yet
Most teams do not need the full stack on day one. Installing platform machinery before you have models in production is a common and expensive mistake: it eats months, adds concepts nobody needs yet, and often gets abandoned when the first real requirement does not fit a tool chosen in the abstract.
A sane minimum, in build order:
- Version data and code together. Every run records the commit and an immutable reference to its dataset.
- Track experiments automatically. Parameters, metrics and artefacts captured by the training code, not by anyone remembering.
- Be able to roll back fast. The previous artefact stored and promotable by config in minutes, without a rebuild. Practise it once.
- Monitor inputs as well as outputs. Feature distributions and null rates against the training reference.
- Log the model version with every prediction. Cheap now, unrecoverable later.
That is most of the value and none of it needs a platform. Add a feature store when you have real skew problems or several models sharing features. Add orchestration when manual runs are genuinely the bottleneck. Add a registry when you cannot list your models from memory. Let each piece be pulled in by a problem you can name.
The same restraint applies to the product around the model. Prediction quality rarely decides whether an ML feature succeeds. What the user does with an uncertain answer, and how the system behaves when the model is down, matter more, which makes this end-to-end product work rather than a modelling exercise with a UI attached.
When the model is an API you do not own
Generative AI features now sit beside classical models in the same products, and the lifecycle changes shape when you call a third-party model instead of training one.
You version prompts and retrieval sources instead of weights. Behaviour comes from the prompt template, system instructions, tool definitions, and the retrieval corpus with its chunking and embedding config. All of it belongs in version control, reviewed like code, with a record of which version served which request. Prompts edited in a vendor console are undeployable and unauditable.
Evaluation is harder because outputs are open-ended. There is no accuracy score for a paragraph. You need a fixed set of representative inputs graded by rubric-scored human review or by a model judge whose agreement with human judgement you have checked, run on every prompt change the way you run tests. Skip it and you are editing prompts by feel, regressing behaviour you cannot name.
Cost scales with usage rather than being fixed. A trained model on an instance costs roughly the same at ten requests or ten thousand. A token-billed API does not. Cost becomes a per-request property moving with prompt length, retrieved context and output length, so a small system-prompt change multiplied across production traffic is a budget event. Track it per feature.
The provider can change the model underneath you. Behaviour you tested can shift with no deploy on your side. Pin explicit versions where they are offered, keep the evaluation suite runnable on demand so you can measure a change instead of arguing about it, and design fallbacks for rate limits, outages and deprecations.
What carries over: shadow and canary rollout matter more here, since offline evaluation of open-ended output is weaker. Monitoring still applies, on refusal rates, response lengths, latency, retrieval hit rates and user corrections. Drift still exists, in the corpus and in what users ask for. We build these as AI and agentic development work, and this scaffolding separates a demo from something you can run for two years. If you are staffing it, an embedded engineering team owning pipeline, serving and monitoring together produces a system someone can actually maintain.
Trying to work out what an ML or AI feature will take to run, not just to build? A Scoping Sprint ($2,300, two weeks) ends with a lifecycle and serving plan made for your case, a prototype, and a fixed quote. Or just start a conversation.
