Learning Time Series with Neural Networks

Section 01

Why let the features learn themselves

The earlier parts of this series were a catalogue of hand-built features. Someone who understood bearings decided that an envelope spectrum was the right lens; someone who understood control decided that an innovation sequence was the right residual. That expertise is precious, and where it exists it is hard to beat. Neural networks earn their place exactly where it runs out: when the signal is high-dimensional, the fault is subtle, or nobody yet knows which feature matters.

A neural network is a stack of simple adjustable functions, trained by showing it examples and nudging its millions of small parameters until its output matches the truth. The nudging is the whole story, and it has a single engine, gradient descent, which this part starts with and then dresses in the four architectures that dominate time-series work: convolutions that learn filters, recurrence that learns memory, autoencoders that learn what normal looks like, and attention that learns where to look. Every simulation here computes the real thing in your browser, no recordings, so you can watch a network actually learn.

Fifth in a series

This is the methodological companion to the operational arc. Where Sensory Data and Forecasting engineered features and models by hand, this part lets the network find them. The lessons of the earlier parts do not disappear; the best systems usually feed hand-built features into a learned model, marrying domain knowledge to capacity.

Section 02

The learning machine: gradient descent

Strip a neural network to its core and you find one idea repeated. Each parameter has a knob; turning it changes the output and so changes the error. Backpropagation computes, for every knob at once, which way and how hard to turn it to reduce the error, and gradient descent takes a small step in that direction. Repeat a few thousand times and the network has shaped itself to the data.

The simulation below is a genuine network, one input, a layer of tanh units, one output, learning to reproduce a waveform from scratch by full gradient descent, training live as you watch the fit improve and the loss fall. The controls expose the three levers that decide whether learning succeeds: too few hidden units and the network underfits, unable to bend into the shape; too high a learning rate and it diverges; raise the noise and watch a large network start to overfit, chasing the noise instead of the signal.

forward: ŷ = Σ_j w²_j · tanh(w¹_j x + b¹_j) + b² · update: w ← w − η ∂L/∂w

FIGURE 1 · a network learning a waveform by gradient descent · live training

Hidden units 10

Learning rate η 0.12

Training noise 0.08

training data network output loss curve

Top: rust dots are the noisy training points; the teal curve is the network's current output, redrawn every few gradient steps as it learns. Bottom: the training loss, falling as the fit improves. Drop the hidden units to three and the curve cannot capture the fast wiggle; push the learning rate near its maximum and the loss spikes and thrashes; raise the noise with many units and the curve starts threading individual points, the signature of overfitting.

Section 03

Convolutions: a learnable filter bank

Part one designed filters by hand: a smoother here, a band-pass there, a matched template for bearing impacts. A convolutional network learns those filters instead. Its core operation is the one part one used for matched filtering, sliding a short kernel along the signal and taking a dot product at every position, but the kernel's coefficients are parameters, trained until they detect whatever discriminates the classes in the data.

The simulation runs a genuine 1D convolution, then the two operations that make it a network layer: a ReLU that keeps only positive responses, and max-pooling that summarises each neighbourhood by its strongest activation, shrinking the signal while keeping what fired. Switch the kernel between a smoother, an edge detector and an impact-matched template, and watch the feature map light up where the kernel resonates. A trained network discovers the impact template on its own; here you can hand it the answer and see why it works.

feature map: (k ∗ x)[t] = Σ_τ k[τ] · x[t − τ] → ReLU → max-pool

FIGURE 2 · convolution, ReLU and pooling on vibration · interactive

kernel: smoother edge impact-matched

Background noise 0.25

input vibration feature map (conv + ReLU) pooled

Top: the raw vibration, with periodic impacts hidden in noise. Middle: the feature map after convolving with the chosen kernel and clipping negatives. Bottom: the max-pooled summary. With the impact-matched kernel the feature map spikes cleanly at each impact and pooling preserves those spikes; the smoother blurs everything, and the edge detector fires on every sharp transition including the noise.

Section 04

Memory: recurrent networks

Convolutions see a fixed window. To carry information across an arbitrarily long sequence, a network needs memory, and a recurrent network supplies it by keeping a hidden state that it updates at every step and feeds back into itself. The simulation runs the essential recurrence, a hidden state that blends its previous value with the new input, which is a genuine one-unit recurrent network and the seed of every fancier variant.

One knob, the recurrent weight, sets how much past the state retains, and so sets the effective memory length: near zero the unit just echoes the input, near one it integrates over a long horizon. The simulation also searches for the recurrent weight that best predicts the next sample, a tiny act of learning, and marks it. Real LSTM and GRU cells replace this single fixed knob with learned gates that decide, per step and per feature, what to remember and what to forget, which is how they hold information across hundreds of steps without it decaying away.

hidden state: h_t = a · h_t−1 + (1 − a) · x_t · effective memory ≈ 1 / (1 − a) steps

FIGURE 3 · a recurrent hidden state and its memory · interactive

input: noisy signal step pulse train

Recurrent weight a 0.80

input x hidden state h best-memory weight

Grey: the input sequence. Teal: the recurrent hidden state. Push the recurrent weight toward zero and the state copies the input, keeping no memory; push it toward one and the state becomes a long, smooth integral of the past. The readout reports the effective memory length and the prediction error, and the ochre marker shows the recurrent weight that best predicts the next sample, the value a one-parameter network would learn.

Section 05

Autoencoders: learning normal

Faults are rare, and labelled faults rarer still. An autoencoder sidesteps the problem by learning only what normal looks like. It squeezes each input through a narrow bottleneck and then tries to reconstruct it; trained on healthy data alone, it becomes good at rebuilding healthy patterns and bad at rebuilding anything else. The reconstruction error then doubles as an anomaly score, large exactly when the input does not resemble the training distribution.

The simulation makes this exact and genuine. Healthy windows of vibration live near a low-dimensional subspace; the autoencoder's job is to find that subspace and project onto it. A linear autoencoder provably learns the same subspace as principal component analysis, so the simulation computes the principal components of the healthy data directly and reconstructs through a bottleneck of the size you choose. Feed it a healthy window and the reconstruction overlays the input; inject an anomaly and the reconstruction cannot follow it, and the error climbs past the alarm line.

reconstruct: x̂ = μ + Σ_i≤k (v_i · (x − μ)) v_i · score = ∥ x − x̂ ∥

FIGURE 4 · reconstruction error as an anomaly score · interactive

input: healthy anomalous

Bottleneck size k 2

Anomaly strength 1.0

input window reconstruction

Rust: the input window. Teal: the autoencoder's reconstruction through a k-dimensional bottleneck. On a healthy window a bottleneck of two or three rebuilds it almost perfectly. Switch to an anomalous window and the reconstruction, trapped in the healthy subspace, cannot match it, so the residual and the error score jump. Too small a bottleneck reconstructs even healthy data poorly; too large a one starts reconstructing anomalies too, the same bias-variance tension seen throughout the series.

Section 06

Attention: what the model looks at

Recurrence carries memory forward but smears it; by the time a distant event matters, its trace has faded. Attention, the mechanism behind transformers and the foundation models part two mentioned, takes a different route. Instead of a single rolling state, it lets every position look directly at every other position and decide, with a learned similarity, how much each one matters right now.

The simulation computes genuine self-attention over a short sequence. Each step is given a small query and key; their dot products, passed through a softmax, become the attention weights, how strongly the step doing the looking attends to each step being looked at. The heatmap shows the full matrix, and the selected row shows where one position focuses. Here the queries and keys are simple fixed features so the mechanism is visible; a real transformer learns these projections, but the move is the same: attend most to the past that most resembles the present.

attention: α_ij = softmax_j( q_i · k_j / √d ) · output_i = Σ_j α_ij v_j

FIGURE 5 · a self-attention weight matrix · interactive

Query position 23

Sharpness (temperature) 1.0

attention weight selected query attended steps

Top: the attention matrix, row i showing how strongly step i attends to each step j, darker teal for more weight. The rust-outlined row is the query position you selected. Bottom: the signal, with the steps that query attends to most marked in ochre. Raise the sharpness and the attention collapses onto a few most-similar steps; lower it and the focus spreads evenly. The model looks hardest at the moments that resemble the one it is trying to explain.

Section 07

Shrinking the network for the edge

A network that runs in a data centre is one thing; one that must run on a microcontroller bolted to a motor is another. Part one's edge pattern, keep the raw data local and send only the conclusion, demands that the model fit in kilobytes and run in milliseconds. The dominant tool is quantization: storing the weights not as 32-bit floating point but as 8-bit, or fewer, integers.

The simulation quantizes the parameters of a small model that reconstructs a waveform, rounding each weight to one of a limited set of levels and rebuilding the output. At 8 bits the quantized output is visually indistinguishable from the original at a quarter of the size, which is why INT8 is the workhorse of embedded inference. Drop toward 3 or 2 bits and the reconstruction visibly degrades as the levels grow too coarse to represent the weights. The readout tracks the model size and the error, the tradeoff every edge deployment has to price.

quantize: q = round( (w − min) / Δ ) · Δ + min, Δ = (max − min) / (2ⁿ − 1)

FIGURE 6 · weight quantization and the size-accuracy tradeoff · interactive

Bit depth n 8

full-precision output quantized output

Grey: the model's output at full 32-bit precision. Rust: the output after quantizing every weight to n bits. From 16 down to 8 bits the two are almost identical while the model shrinks fourfold; below about 4 bits the coarse levels start to distort the output. The readout shows the model size and the reconstruction error, so you can read off how much accuracy each byte saved costs.

Section 08

Practical notes

Neural networks reward data and punish carelessness in ways the hand-built methods do not. These are the lessons that decide whether a learned model is an asset or a liability in a monitoring system.

Earn the network. A neural model is only worth its complexity when a simpler one has genuinely failed. Benchmark against the hand-built features and classical models from the earlier parts first, and adopt the network only when it clearly wins on an honest test.
Data hunger is real. Capacity needs examples. With a few hundred labelled events, classical methods usually win; networks pull ahead when data is plentiful, and starve quietly when it is not.
Normalize, and remember how. Networks are sensitive to input scale. Standardise every channel, and store those scaling constants with the model, because the same normalisation must run at inference or the network sees nonsense.
Leakage is the silent killer. Split train and test by time or by unit, never at random across a sequence, or the network memorises the future and your accuracy is a fiction that collapses on deployment.
Watch for overfitting. A model that nails training and fails validation has learned the noise. Hold out data honestly, use early stopping and regularisation, and treat a too-good training score as a warning, not a triumph.
Calibrate the confidence. A network's raw output is not a probability. If a decision rests on its confidence, calibrate it and validate the calibration, exactly as part two did for forecast intervals.
Quantize and measure. Do not assume INT8 is free. Quantize, then re-validate accuracy on real data, because a model that passed in floating point can fail once its weights are coarsened for the edge.

The one-sentence summary

A neural network is a feature learner, not a magic box, so spend your effort on honest data, honest splits and honest baselines, let the network earn its complexity against the hand-built methods that came before it, and remember that everything the earlier parts taught about leakage, calibration and the edge applies with more force, not less, once the features are learned.

Section 09

References and further reading

Goodfellow, I., Bengio, Y., & Courville, A. Deep Learning. MIT Press. The standard graduate text on the architectures and training methods in this part.
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). "Learning representations by back-propagating errors." Nature. The paper that made gradient descent practical for multilayer networks.
LeCun, Y., Bengio, Y., & Hinton, G. (2015). "Deep learning." Nature. A compact overview of convolutional and recurrent models and their reach.
Hochreiter, S., & Schmidhuber, J. (1997). "Long short-term memory." Neural Computation. The LSTM cell and the gated memory behind it.
Vaswani, A., et al. (2017). "Attention is all you need." NeurIPS. The transformer and the self-attention mechanism of section six.
Hinton, G. E., & Salakhutdinov, R. R. (2006). "Reducing the dimensionality of data with neural networks." Science. Autoencoders and their link to principal component analysis.
Jacob, B., et al. (2018). "Quantization and training of neural networks for efficient integer-arithmetic-only inference." CVPR. The basis of practical INT8 deployment.
Fawaz, H. I., et al. (2019). "Deep learning for time series classification: a review." Data Mining and Knowledge Discovery. A survey of architectures applied specifically to time series.

References point to further study; the explanations and simulations above are written and implemented from first principles rather than drawn from any single source.

That completes the methodological detour. The operational arc, Sensory Data to Forecasting to Remaining Useful Life to Monitoring, tells you what to do; this part tells you how a learned model does it underneath, and where its appetite for data and discipline makes it worth the trouble.