Machine Learning · Diagnostics · VHM
The workhorse classifiers that name the fault. From one decision tree you can read, to random forests and gradient boosting — XGBoost, LightGBM, CatBoost — in plain words, with a worked example.
Anomaly detection tells you something is wrong. A classifier tells you which fault it is. In vehicle health monitoring the classifiers that earn their keep are almost all built from decision trees: a single tree you can read like a flowchart, and then ensembles of hundreds of trees that vote or correct one another — Random Forest, AdaBoost, gradient boosting (XGBoost, LightGBM, CatBoost), and Extra Trees. They are strong on the kind of data a fleet actually produces — wide tables of mixed sensor features, with nonlinear effects and interactions — and they hand you a ranked list of which signals drove the decision, almost for free. This monograph builds them up from the single tree, explains how each ensemble differs and when to reach for it, and runs one worked example end to end: classifying an EV charging fault from tabular features. Plain words, block diagrams, a flowchart, Python you can run, practical notes for the messy realities of fleet data, and references.
A fleet sends home a wide table: one row per vehicle (or per charge, or per drive), and a column for every feature you can compute from its sensors.
Battery temperature spread, charge-connector resistance, peak charge current, how often the car charges in the cold, firmware version, pack age in cycles, a dozen vibration features from the drivetrain — each becomes a number in a column. The question a classifier answers is simple to state: given this row of numbers, which fault is this? Healthy, or a firmware cold-throttle bug, or a high-resistance connector, or a cell-balancing fault, and so on.
That is a supervised classification problem. "Supervised" means you train it on rows where the answer is already known — from teardowns, warranty claims, seeded-fault test rigs, or confirmed past diagnoses. The model learns the pattern that separates the classes, then labels new rows.
This kind of data — a wide table of mixed-meaning columns, where the signal lives in nonlinear thresholds ("resistance above 8 mΩ and cold") and in interactions between columns — is exactly where tree-based models shine, and where the fashionable deep networks usually do not pay their way. That is why, across VHM, the model that finally puts a name on the fault is so often a Random Forest or a gradient-boosted ensemble.
Two stages, two model types. Unsupervised anomaly detection (PCA, Isolation Forest, autoencoders) flags that a car looks abnormal — no labels needed. Then a supervised tree ensemble names the specific fault — labels needed. This post is about the second stage.
Start with the single building block: a decision tree (the classic algorithm is called CART, for Classification And Regression Trees). It is a flowchart the computer writes for itself. Each box asks one yes/no question about one feature — "is connector resistance above 8 mΩ?" — and sends you left or right. You follow the questions down until you reach a leaf, which carries the answer.
How does the tree pick which question to ask? It is greedy. At each box it tries every feature and every threshold, and keeps the split that makes the two resulting groups as pure as possible — meaning each group is dominated by a single class. Purity is measured by Gini impurity or entropy; both are just numbers that fall to zero when a group is all one class. It repeats, splitting each group again, until the leaves are clean enough or it runs out of room.
A single tree has two great virtues for VHM: you can read it — every decision is a chain of plain thresholds an engineer can sanity-check — and it needs almost no data preparation (no scaling, handles mixed feature types, shrugs at outliers). That makes it the honest baseline: always train one first, because if a model nobody can argue with already hits 90%, you have learned something.
Its great vice: a single deep tree overfits. It will happily carve the training data into tiny perfect leaves that memorize noise, and it is unstable — change a handful of training rows and the whole flowchart can rearrange. The cure for both is to stop relying on one tree and grow a crowd.
An ensemble trains many trees and combines them. The reason it works is a single idea worth holding onto: errors that disagree cancel out. If you grow many different trees and each makes mistakes in different places, then when you average their votes the mistakes wash out and the shared signal survives.
There are two ways to build the crowd, and they fix two different problems.
Bagging (Random Forest, Extra Trees) grows many full-depth trees in parallel, each on a slightly different slice of the data, then lets them vote. Each tree alone overfits, but their errors point in random directions, so the vote is stable. Bagging mainly reduces variance — the jitter that made one tree untrustworthy.
Boosting (AdaBoost, gradient boosting) grows trees in sequence, each small and each focused on the rows the previous trees got wrong. The crowd is built deliberately, not randomly. Boosting mainly reduces bias — it keeps chipping away at the error the team still makes — and tends to be the most accurate when tuned with care.
The Random Forest is the safe default — the model to reach for first when a single tree is not enough.
It grows hundreds of full trees and lets them vote, with two tricks that keep the trees different from one another so their errors cancel:
The combination is the whole point: the trees are decorrelated, so averaging their votes cancels the noise and keeps the signal. The forest is far more stable and accurate than any tree inside it, and it is hard to overfit — adding more trees never hurts, it just costs compute.
input: table X (rows×features), labels y, number of trees T
for t = 1 … T:
bag ← sample rows of (X, y) # with replacement
tree_t ← grow a full tree on bag,
at each split consider only a random subset of features
return the set {tree_1 … tree_T}
# to predict a new row x:
predict(x) = majority vote of tree_1(x) … tree_T(x)
Because each tree skips about a third of the rows (they were not in its bag), those rows are a ready-made test set for that tree. Averaged over the forest, this out-of-bag (OOB) error gives an honest accuracy estimate without holding data back — handy when labeled faults are scarce.
Boosting builds the crowd one tree at a time, and each new tree is asked a pointed question: where is the team still wrong, and can you fix it?
AdaBoost, the original, does this by re-weighting rows: after each small tree, the rows it got wrong get heavier, so the next tree pays them more attention. The final answer is a weighted vote, with more accurate trees counting more.
Gradient boosting is the modern, more powerful version. Instead of re-weighting rows, each new tree is trained to predict the residual — the leftover error — of the team so far, and a small step of that correction is added in. "Gradient" just means it follows the slope of the error downhill, one tree at a time. The trees are deliberately shallow, because the strength comes from many gentle corrections rather than a few aggressive ones.
input: X, y, number of trees M, learning rate η (small, e.g. 0.05)
F ← a constant first guess # e.g. the class frequencies
for m = 1 … M:
r ← gradient of the loss at F # "where and how F is wrong"
h ← grow a small tree predicting r from X
F ← F + η · h # take a small step
return F
These three are all gradient boosting; they differ in engineering, not in the core idea. All are excellent. The short version:
| Library | What's special | Reach for it when |
|---|---|---|
| XGBoost | The well-tested standard. Strong regularization, robust, huge community, runs everywhere. | Default for a tabular fault classifier; you want a known quantity. |
| LightGBM | Fastest on big data. Grows trees leaf-wise and bins features, so it trains quickly on millions of fleet rows. | Large fleets, many features, you retrain often and care about speed. |
| CatBoost | Handles categorical columns natively (supplier, plant, firmware id) and needs little tuning to do well. | Lots of categorical build attributes; you want strong results out of the box. |
| AdaBoost | The original boosting; simple, fewer knobs, less accurate than the gradient trio. | A light baseline, or teaching the idea. |
Random Forest first (it just works and barely needs tuning). If you need to squeeze out more accuracy and can afford to tune, move to gradient boosting — XGBoost as the safe pick, LightGBM if speed matters, CatBoost if you have many categorical columns.
Extra Trees (Extremely Randomized Trees) is a Random Forest with the randomness turned up one more notch. Where a Random Forest searches for the best threshold on each candidate feature, Extra Trees picks the thresholds at random and keeps the best of those. It usually also trains each tree on the full data rather than a bootstrap bag.
The effect: the trees are even more decorrelated and a little more biased individually, but the ensemble is often as accurate as a Random Forest and noticeably faster to train, because it skips the expensive search for the perfect split. In VHM it is a fine swap for Random Forest when training time matters and you want to retrain frequently.
Two practical questions remain: which of these models do you start with, and what do you actually turn to make it better?
You do not tune everything. A handful of knobs carry almost all the benefit; the rest are noise. Here are the ones worth knowing, and a sane place to start for a VHM fault classifier.
| Knob | Model | What it does | Start at |
|---|---|---|---|
| n_estimators | Random Forest | Number of trees. More is safer (never overfits), just slower. | 300–500 |
| n_estimators | Boosting | Number of rounds. Pair with a low learning rate; let early stopping pick it. | 500–2000 + early stop |
| learning_rate | Boosting | Step size. Lower = more accurate but needs more trees. | 0.03–0.1 |
| max_depth | Random Forest | Tree depth. Deep is fine — the vote tames it. | None (full) |
| max_depth | Boosting | Depth of each small tree. Keep it shallow. | 3–6 |
| min_samples_leaf / min_child_weight | Both | Minimum data in a leaf. Raise to smooth and de-overfit. | raise if overfitting |
| max_features | Random Forest | Features tried per split — the decorrelation knob. | "sqrt" |
| subsample / colsample_bytree | Boosting | Random rows / features per tree. Adds regularization. | 0.7–0.9 |
| reg_lambda / reg_alpha | Boosting | L2 / L1 penalties. Raise to fight overfitting. | small, then raise |
| class_weight / scale_pos_weight | Both | Counter rare-fault imbalance. | balanced / class ratio |
For boosting, almost all the gain comes from one recipe: lower the learning rate, raise the number of trees, and use early stopping to find where the validation score stops improving. Tune depth and subsampling after that, and stop when the curve flattens.
A diagnosis you cannot explain is hard to act on. Tree ensembles hand you the explanation almost for free.
Because every split is made on one feature to reduce impurity, you can add up how much each feature helped across all the trees. That gives a ranked list — a feature importance — telling you which signals the model leaned on. For a fault classifier that list is gold: it points the engineer at the physical cause, not just the label.
There are three flavours, in rising order of trustworthiness:
Importance tells you what the model used, not what physically causes the fault. A feature can rank high because it is correlated with the true cause (a confounder), not because it is the cause. Treat the ranking as a lead to investigate, not a verdict — the same lesson as the rest of the diagnosis series.
One fleet, one charging code, three answers: healthy, a firmware cold-throttle bug, or a high-resistance connector lot. Let's train the classifier that tells them apart.
This is the running scenario from the diagnosis series — 6,000 EVs throwing the same P0AE0 charging code, with two real hidden causes. We have a labeled history (from teardowns and confirmed past cases) and a table of features per car. The code below trains a Random Forest as the baseline, then XGBoost, reads off the confusion matrix, and prints the feature importance.
Each row is one car; each column a feature computed from its telemetry and build record. Keep the meaning obvious so the importances are readable.
# features per vehicle — one row each import pandas as pd from sklearn.model_selection import train_test_split cols = [ "connector_resistance", # mΩ, from charge logs "firmware_version", # numeric, e.g. 5.0 / 5.1 / 5.2 "cold_charge_fraction", # share of charges below 5 °C "supplier_code", # categorical: A / B "peak_charge_current", # A "pack_age_cycles", # full-charge equivalents ] df = pd.read_parquet("fleet_features.parquet") X = pd.get_dummies(df[cols]) # one-hot the categorical column y = df["label"] # healthy / firmware_bug / connector_lot X_tr, X_te, y_tr, y_te = train_test_split( X, y, test_size=0.25, stratify=y, random_state=0)
from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import classification_report, confusion_matrix # --- baseline: Random Forest, almost no tuning --- rf = RandomForestClassifier( n_estimators=400, class_weight="balanced", # faults are rare oob_score=True, random_state=0, n_jobs=-1) rf.fit(X_tr, y_tr) print("OOB accuracy:", round(rf.oob_score_, 3)) print(classification_report(y_te, rf.predict(X_te))) # --- stronger: XGBoost --- from xgboost import XGBClassifier xgb = XGBClassifier( n_estimators=600, max_depth=4, learning_rate=0.05, subsample=0.8, colsample_bytree=0.8, eval_metric="mlogloss", random_state=0) xgb.fit(X_tr, y_tr.cat.codes) print(confusion_matrix(y_te.cat.codes, xgb.predict(X_te)))
The confusion matrix is the honest scorecard: rows are the true class, columns what the model guessed. The diagonal is correct; everything off-diagonal is a mix-up worth understanding.
| true ↓ / pred → | healthy | firmware bug | connector lot | recall |
|---|---|---|---|---|
| healthy | 1182 | 9 | 9 | 98.5% |
| firmware bug | 6 | 171 | 5 | 94.0% |
| connector lot | 7 | 4 | 165 | 93.8% |
Read the off-diagonal as questions. The handful of firmware-vs-connector mix-ups (5 and 4 cars) are the genuinely ambiguous cars — cold-charging cars with a Supplier-B connector, where both causes leave a similar trace. That is not a bug in the model; it is the de-mixing problem the fleet posts warned about, and it tells you exactly which cars need a confirming measurement.
# honest, model-agnostic importance from sklearn.inspection import permutation_importance imp = permutation_importance(rf, X_te, y_te, n_repeats=10, random_state=0) for name, score in sorted( zip(X.columns, imp.importances_mean), key=lambda t: -t[1]): print(f"{name:24s} {score:.3f}") # per-car reasons, for the report attached to a diagnosis import shap explainer = shap.TreeExplainer(xgb) shap_values = explainer.shap_values(X_te) # why THIS car got THIS label
The importance ranking comes back led by connector_resistance and firmware_version — the two real causes (Figure 5). The model has not just labeled the cars; it has handed the engineer the two signals to go and confirm. That is the whole loop: detect, classify, and explain, with the explanation pointing back at the physics.
A 30-line baseline reached ~96% accuracy on a three-way fault classification and named the two driving signals — with no feature scaling, no neural network, and a model an engineer can interrogate. That ratio of effort to result is why tree ensembles are the workhorse of VHM fault classification.
A classifier outputs a number. Acting on a fleet — flagging cars for a campaign — needs that number to mean something, and needs a threshold chosen by what mistakes actually cost.
Two traps sit between the model and the decision.
Trap one: the score is not a probability. A tree ensemble's "0.7" is not a calibrated 70% chance — boosting tends to push scores toward 0 and 1, forests toward the middle. If a number is going to drive a recall, calibrate it first so that, among cars scored 0.7, about 70% truly have the fault.
from sklearn.calibration import CalibratedClassifierCV cal = CalibratedClassifierCV(xgb, method="isotonic", cv=5) cal.fit(X_tr, y_tr) p = cal.predict_proba(X_te)[:, fault_idx] # now a trustworthy probability
Trap two: 0.5 is almost never the right cutoff. The default threshold treats a missed fault and a false alarm as equally bad. In VHM they are wildly unequal. A missed high-resistance connector can mean a thermal event and a warranty claim; a false flag means a needless service visit or an unwarranted campaign. Put numbers on it and the threshold falls out.
Suppose a missed connector fault costs about $4,000 per car (warranty + reputation + safety exposure) and a false flag costs about $250 per car (service time, owner goodwill). Then for each candidate threshold you can compute the expected cost over the fleet and pick the lowest. Because misses are ~16× costlier than false alarms, the best cutoff sits far below 0.5.
This is the same logic as the fleet decision to launch a campaign, pushed down to the per-car score. Calibrate, write down the cost of each error, and let the numbers set the threshold — then the model's output is a decision, not just a guess.
Swap the leaf from a label to a number and the very same ensembles do prognostics — predicting how much life a part has left, not just which fault it has.
A classification tree's leaf holds a class; a regression tree's leaf holds a number — the average of the training values that landed there. Everything else is identical, so RandomForestRegressor and XGBRegressor predict a continuous target such as Remaining Useful Life (RUL) in cycles or days from the same tabular features.
from sklearn.ensemble import RandomForestRegressor rul = RandomForestRegressor(n_estimators=400, random_state=0, n_jobs=-1) rul.fit(X_tr, y_rul_tr) # y = cycles until failure threshold pred = rul.predict(X_te) # point estimate of remaining life
A point estimate alone is dangerous for a maintenance call — you want a band. Two easy ways: read the spread of the individual trees' predictions in a forest, or train a gradient-boosting model with a quantile loss to predict the 10th and 90th percentiles directly, giving an honest prediction interval.
from sklearn.ensemble import GradientBoostingRegressor lo = GradientBoostingRegressor(loss="quantile", alpha=0.1).fit(X_tr, y_rul_tr) hi = GradientBoostingRegressor(loss="quantile", alpha=0.9).fit(X_tr, y_rul_tr) # report "RUL between lo(x) and hi(x)" — a band, not a false-precise number
Regression trees have a hard ceiling for prognostics: they cannot predict a value outside the range they were trained on. A part more degraded than anything in the training data will get an answer that is too optimistic. That single limitation is important enough to earn its own section.
Tree ensembles are the right default for tabular VHM — but defaults have edges. Knowing where they fail is part of using them well.
The honest list of where to reach for something else:
The algorithm is the easy part. These are the things that decide whether a VHM classifier survives contact with a real fleet.
class_weight="balanced" (or scale_pos_weight in XGBoost), resample with care, and judge by precision, recall, and the confusion matrix — not accuracy.Breiman, L. (2001). Random Forests. Machine Learning, 45(1). The paper that started it — bagging plus random feature subsets.
Breiman, Friedman, Olshen & Stone (1984). Classification and Regression Trees (CART). The single-tree foundation.
Geurts, Ernst & Wehenkel (2006). Extremely Randomized Trees. Machine Learning, 63(1).
Freund & Schapire (1997). A Decision-Theoretic Generalization of On-Line Learning (AdaBoost).
Friedman, J. (2001). Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics, 29(5). The gradient-boosting paper.
Chen & Guestrin (2016). XGBoost: A Scalable Tree Boosting System. KDD.
Ke et al. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. NeurIPS.
Prokhorenkova et al. (2018). CatBoost: Unbiased Boosting with Categorical Features. NeurIPS.
Lundberg & Lee (2017). A Unified Approach to Interpreting Model Predictions (SHAP). NeurIPS.
Niculescu-Mizil & Caruana (2005). Predicting Good Probabilities with Supervised Learning. ICML. Why tree-ensemble scores need calibration.
Meinshausen, N. (2006). Quantile Regression Forests. JMLR, 7. Prediction intervals (RUL bands) from forests.
Hastie, Tibshirani & Friedman (2009). The Elements of Statistical Learning, 2nd ed. Chapters 9, 10, 15 — trees, boosting, forests.
Pedregosa et al. (2011). scikit-learn: Machine Learning in Python. JMLR, 12. The library used above.
Lei et al. (2020). Applications of machine learning to machine fault diagnosis: A review and roadmap. Mechanical Systems and Signal Processing, 138. Tree ensembles in the VHM/PHM context.