A field guide · part four / keeping watch after deployment
A model that forecasts beautifully in the lab is only half the job. Once it is live, the world drifts out from under it, the sensors it depends on fail in their own ways, and faults hide in the joint behaviour of channels that each look fine alone. This part is about the unglamorous discipline that keeps a deployed system honest: watching the watcher.
Section 01
The first three parts of this series built a model: they turned a sensor into features, forecast its next value, and projected its failure. All of that assumed the model and its inputs stay the way they were when you validated them. In production that assumption quietly dies. The operating regime shifts, a new component changes the baseline, a sensor starts lying, and the carefully tuned detector keeps reporting that all is well while the ground moves beneath it.
Monitoring is the practice of catching these failures of the system around the model, and it needs its own tools, because the model's own outputs will not raise the alarm. Three problems dominate, and each gets a working simulation here: faults that are invisible in any single channel but obvious in the joint behaviour of several; the slow drift of the live data away from the distribution the model was trained on; and the maddening question of whether a strange reading means the machine is sick or merely the sensor.
Fourth in a series
This part assumes the earlier vocabulary. Sensory Data built features and residuals, Forecasting built backtests and intervals, and Remaining Useful Life built degradation models. Here those same signals are watched after deployment, where the enemy is not a hard forecasting problem but a world that refuses to hold still.
Section 02
Set a threshold on each channel alone and you will miss the most dangerous faults. Two sensors that normally move together, say a motor's current and its temperature, can each sit comfortably inside their usual range while their combination is something the machine has never produced in health. The fault lives in the correlation, and a per-channel limit is blind to it.
The classical fix is Hotelling's T-squared, which measures distance in a way that accounts for the covariance between channels. It learns the healthy cloud's mean and covariance, then scores each new point by its Mahalanobis distance: how many standard deviations away it is along the directions the data actually varies. A control limit drawn from the chi-square distribution turns that distance into an alarm. The simulation builds a healthy cloud of two correlated channels and lets you place a test point; watch a reading that is normal on each axis light up as a joint anomaly once it strays off the correlation ridge.
Section 03
Every model in this series carried the same warning: retrain on drift. This is how you know when. Concept drift is the change over time in the distribution of the live data, and even a perfect model decays as the world wanders away from its training set. You catch it not by waiting for accuracy to collapse but by comparing the recent data against a fixed reference window and measuring how far they have diverged.
Two measures dominate. The population stability index (PSI) bins both distributions and sums their disagreement; a rough convention treats below 0.1 as stable, 0.1 to 0.25 as a moderate shift worth watching, and above 0.25 as a significant drift that usually warrants retraining. The Kullback-Leibler divergence measures the same idea in the language of information. The simulation holds a reference distribution fixed and lets you shift and stretch the current one, reporting both scores as the overlap erodes.
Section 04
A monitor screams. Before anyone touches the machine, one question decides everything: is the equipment actually degrading, or has the sensor itself failed? Acting on a faulty sensor wastes a teardown; ignoring a real fault risks the asset. The two are distinguishable, but only if you have planned for it, usually with redundancy and a few cheap features.
Sensor faults have signatures. A stuck sensor goes flat, its local variance collapsing to nothing. A drift fault pulls one channel slowly away from its redundant twin, growing the residual between them. A spike is a single impossible jump; a dropout falls to zero. The tell that separates all of these from a real fault is the cross-sensor residual: when the machine degrades, every healthy sensor sees it and moves together, so the residual between redundant sensors stays small. When one sensor lies, only it moves, and the residual blows up. The simulation injects each fault type and runs simple detectors to reach a verdict.
Section 05
A monitor is judged differently from a forecaster. Its currency is alarms, and the costs are asymmetric in a way that breeds a specific failure mode. Too sensitive and it cries wolf; the operators learn to ignore it, and alarm fatigue quietly disables the whole system. Too lax and it misses the event it existed to catch. The tuning lives on that tradeoff.
Of the alarms raised, how many were real; of the real events, how many were caught. Monitoring usually cannot maximise both, so the operating point is a deliberate choice.
How long before the event the alarm fires. A correct alarm that arrives too late to act on is worth little more than a miss.
Alarms per unit time during healthy operation. The single number most responsible for whether operators keep trusting the system.
Once a fault begins, how many samples until the alarm. Tighter limits detect faster but raise the false-alarm rate, the same tension seen with CUSUM in part one.
The discipline is to set these deliberately, with the cost of a miss and the cost of a false alarm both written down, rather than chasing a single accuracy number. A monitor nobody trusts is worse than no monitor, because it provides the illusion of safety while the operators have stopped looking.
Section 06
A monitor lives or dies on whether the people it serves keep trusting it. These are the habits that keep a deployed monitor useful long after the novelty of the dashboard has worn off.
The one-sentence summary
A production monitor is a promise to notice when the world has changed, so watch the inputs as carefully as the outputs, tune every threshold against the cost of a false alarm and a miss, give it the redundancy to say why something looks wrong, and back it with a runbook, because a monitor nobody trusts is worse than none at all.
Section 07
References point to further study; the explanations and simulations above are written and implemented from first principles rather than drawn from any single source.