Physics-Informed and Transformer Models for Time Series

Section 01

Two ways to add structure

Part five let networks learn their features from scratch, which works when data is plentiful and fails when it is scarce. Most engineering problems are scarce. A motor that has failed three times does not offer the millions of examples a generic network wants, and yet you often know a great deal about it: the differential equations its windings obey, the conservation laws its temperatures respect, the periodic structure of its vibration. The art is to feed that knowledge into the model so it does not have to rediscover physics from a handful of measurements.

There are two complementary ways to do it. The first builds structure in explicitly: a physics-informed network is trained to satisfy a governing equation, a grey-box hybrid corrects a physics model with a small learned term, and a Koopman embedding makes nonlinear dynamics behave linearly. The second builds structure into the architecture: transformers bias a model toward attending across a sequence, patching reshapes that sequence efficiently, and dilated convolutions reach far back in time cheaply. Both routes reduce the data a model needs to be right, and every claim below is computed live in the page.

Sixth in a series

This part sits beside part five as the second half of the deep-learning material. It leans on the control and dynamics language of the prognostics work and the attention sketch from part five, and it closes the loop with part two by ending on probabilistic forecasts, the calibrated intervals a decision actually consumes.

Section 02

Physics-informed neural networks

Suppose you have a few noisy measurements of a vibrating structure and you know it obeys a damped-oscillator equation. A plain network fit to the data alone will track the points it has seen and then wander off into nonsense the moment it runs out of them. A physics-informed network refuses to wander, because its training loss has a second term: the residual of the governing equation, evaluated at many points where there is no data at all. The model is rewarded for fitting the data and, just as hard, for obeying the physics everywhere.

The simulation solves this genuinely. The model is a flexible function whose derivatives are known in closed form, so the equation residual is exact; the data loss and the physics loss combine into one objective that is solved by least squares. Start with the physics weight near zero and you have a pure data fit: it nails the sparse points in the early window and then diverges wildly in the data-free region. Turn the physics weight up and the governing equation takes over, carrying the solution correctly through the region where no data exists at all.

loss = Σ_data (u − y)² + λ · Σ_colloc ( u″ + 2ζω u′ + ω² u )²

FIGURE 1 · data fit vs physics-constrained extrapolation · interactive

Physics weight λ 1.0

Data points 6

Measurement noise 0.04

noisy data (early window) model solution true solution

Rust dots: noisy measurements, all in the shaded early window. Teal: the model's solution. Grey: the true damped oscillation. With the physics weight near zero the teal curve fits the dots and then flies apart in the unshaded data-free region. Raise the weight and the differential equation pins the solution down everywhere, so it keeps oscillating and decaying correctly long after the data has run out, the whole point of a physics prior.

Section 03

Grey-box hybrids

Pure physics models are never quite right. A real bearing has friction the equations idealise; a real battery has chemistry the circuit model simplifies. Pure data models, conversely, have no backbone and extrapolate badly. A grey-box hybrid takes the best of each: run the physics model, which captures the gross behaviour, then train a small network to learn the correction, the part the physics got wrong.

The simulation has a true system, a physics model with a deliberately wrong parameter, and a learned correction fit to the gap between them. The physics-only prediction is biased, tracking the right shape with the wrong amplitude and decay. The correction, learned only from training-region data, repairs that bias and, because it sits on top of a model with real structure, the hybrid extrapolates beyond the data far better than a free-form fit could. Make the physics more wrong and watch the correction work harder; shrink the training data and watch the hybrid hold up where a pure data fit would collapse.

hybrid: ŷ(t) = physics(t) + correction(t), correction learned from ( y − physics )

FIGURE 2 · physics model plus a learned correction · interactive

Physics model error 0.40

Training data fraction 0.50

Noise 0.05

data physics only hybrid

Rust dots: measured data. Grey: the physics model alone, biased by its wrong parameter. Teal: the hybrid, physics plus a correction learned from the gap in the shaded training region. The hybrid matches the data and keeps tracking the truth into the unshaded region, because the physics backbone carries the structure and the correction only has to clean up a small, smooth residual rather than invent the whole signal.

Section 04

Koopman: making nonlinear dynamics linear

Nonlinear systems are hard because the tools of linear analysis, eigenvalues, superposition, the whole comfortable toolbox, do not apply. Koopman theory offers a trade: lift the state into a richer set of observables, functions of the state, and in that lifted space the dynamics can become linear, even though the original system was not. A nonlinear flow in two variables can be an exactly linear flow in three well-chosen observables.

The simulation takes a genuinely nonlinear system and learns its Koopman operator from data by dynamic mode decomposition: collect snapshots, lift them with the observables, and find the single linear matrix that best advances them one step. Predicting forward is then just repeated matrix multiplication, and the lifted prediction tracks the true nonlinear trajectory. For contrast, a naive linear model fit to the raw state alone, no lifting, cannot capture the nonlinear coupling and peels away. The lift is what buys the linearity, an idea that runs straight back to the control-theoretic heart of this whole field.

lift g = [x, y, x²] · learn A from data: g_k+1 ≈ A g_k · predict by iterating A

FIGURE 3 · linear prediction of a nonlinear flow, with and without lifting · interactive

Start x₀ -1.0

Start y₀ 1.0

true nonlinear flow Koopman (lifted, linear) naive linear (no lift)

Phase portrait, x against y. Grey: the true nonlinear trajectory. Teal: the prediction from the learned linear Koopman operator in the lifted space, which tracks the curve faithfully because the lift makes the dynamics genuinely linear. Rust dashed: a linear model fit to the raw state with no lifting, which misses the curvature of the slow manifold and drifts off. Move the start point and watch the lifted model stay accurate where the naive one cannot.

Section 05

The transformer block

Part five showed a single head of attention. A real transformer adds two ingredients that make it work on sequences. First, because attention is otherwise blind to order, every position is stamped with a positional encoding, a fixed pattern of sines and cosines at many frequencies that uniquely labels where each step sits. Second, attention runs in parallel heads, each free to specialise: one head can track content similarity while another tracks position, and their outputs are combined.

The simulation computes both genuinely. The top panel is the real sinusoidal positional encoding, a bank of waves whose interference gives every position a distinct fingerprint. Below it, two attention heads run on the same sequence: a content head that attends to steps with similar values, and a positional head that attends to nearby steps, producing visibly different patterns. A full block wraps these in a residual connection and normalisation, which keep deep stacks trainable, but the attention and the encoding are the heart of it.

PE(pos, 2i) = sin( pos / 10000^2i/d ) · multi-head: heads attend in parallel, then combine

FIGURE 4 · positional encoding and two attention heads · interactive

Attention sharpness 1.0

Encoding frequencies 6

encoding / attention weight content head position head

Top: the sinusoidal positional encoding, positions down the rows and encoding dimensions across, each column a wave of a different frequency so that no two positions share a fingerprint. Bottom left: a head attending by content, lighting up between steps of similar value. Bottom right: a head attending by position, banded along the diagonal. Different heads, different jobs, which is why transformers use many of them at once.

Section 06

Patching a series into tokens

Attention costs grow with the square of the sequence length, which is ruinous for the long records typical of condition monitoring. The trick that made transformers practical for time series, popularised by PatchTST, is patching: chop the series into short segments and treat each segment as a single token, the way a sentence is a sequence of words rather than letters. A thousand samples become a few dozen patch-tokens, and the quadratic cost falls accordingly.

The simulation shows the reshaping directly. Choose a patch length and the long signal is sliced into patches, each becoming one column of a token grid that the transformer would then attend over. Longer patches mean fewer tokens and cheaper attention, but each token blurs more of the fine detail inside it; shorter patches keep detail at the cost of more tokens. That tradeoff, granularity against compute, is the single dial patching gives you, and it is why patch length is the first hyperparameter anyone tunes.

length L, patch P → tokens = L / P · attention cost ∝ tokens²

FIGURE 5 · slicing a long series into patch-tokens · interactive

Patch length P 16

input series patch boundaries token grid

Top: the long input series, with teal lines marking the patch boundaries for the chosen patch length. Bottom: the token grid, one column per patch, each column the stacked samples of that patch. Widen the patch and the boundaries spread out, the grid loses columns, and the attention cost, which scales with the square of the token count, drops sharply, at the price of coarser tokens. Narrow it and you recover detail but pay in tokens.

Section 07

Dilated convolutions: long memory

Transformers are not the only way to reach far back in a sequence. A temporal convolutional network stacks dilated causal convolutions: causal so each output sees only the past, never the future, and dilated so each successive layer skips over an exponentially growing gap. A handful of layers then sees thousands of steps back, with none of the vanishing-gradient trouble that plagues recurrence and none of the quadratic cost of attention.

The simulation makes the receptive field visible. Each layer doubles its dilation, one, two, four, eight, so the set of input samples that can influence the final output widens geometrically with depth. Add layers and watch the cone of influence reach further into the past; the readout reports exactly how many steps back the output can now see. This exponential reach for a linear cost in layers is why TCNs remain a strong, efficient baseline for long sequences.

dilations 1, 2, 4, … , 2^L−1 · receptive field = 1 + Σ_l (k − 1) · 2^l = 2^L for k=2

FIGURE 6 · the growing receptive field of a dilated TCN · interactive

Layers 4

node active connection receptive field

Each row is a layer, the bottom the input and the top the output. The dilated causal connections, rust, skip over a gap that doubles every layer, so the single output node at the top draws, through the stack, on an exponentially wide span of inputs, shaded in ochre. Add a layer and the receptive field roughly doubles. The readout gives the exact reach, the number of past steps the output can see.

Section 08

Probabilistic forecasts and pinball loss

A forecast that is a single number hides exactly what a decision needs: how uncertain it is. Deep forecasters earn their keep by predicting a distribution, and the simplest honest way to do that is to predict several quantiles directly. Train one output for the tenth percentile, one for the median, one for the ninetieth, and the spread between them is a calibrated forecast interval, the same object the conformal methods of part two produced by a different route.

The trick is the loss. The pinball loss is an asymmetric penalty, tilted so that minimising it drives an output to the requested quantile rather than the mean. The simulation trains three quantile predictors by genuine gradient descent on the pinball loss, on data whose spread deliberately widens to the right. The result is a fan that hugs the points where they are tight and flares where they scatter, with the band capturing close to its nominal coverage, which the readout checks against the data.

pinball_τ(e) = max( τ·e, (τ−1)·e ), e = y − ŷ · minimised at the τ quantile

FIGURE 7 · a quantile forecast fan trained on pinball loss · interactive

Spread growth 1.0

Noise 1.0

data median (q50) q10 / q90 band

Rust dots: data whose scatter widens to the right. Teal: the median forecast. Ochre band: the tenth-to-ninetieth percentile interval, each edge a separate predictor trained on its own pinball loss. The band stays tight where the data is tight and flares where it spreads, exactly tracking the changing uncertainty, and the readout reports the fraction of points it actually captures against the eighty percent it targets.

Section 09

Practical notes

Structured models reward the knowledge you bring and punish the structure you get wrong. These are the lessons that decide whether a physics prior or a fancy architecture earns its place.

Wrong physics is worse than none. A physics-informed model is only as good as its equation. If the governing law is mis-specified, the prior fights the data and the result is confidently wrong; verify the physics before you trust the constraint.
Weight the physics deliberately. The balance between data loss and physics loss is a real hyperparameter, not a detail. Too little and the constraint does nothing; too much and the model ignores measurements it should respect.
Grey-box beats both extremes. When you have partial physics, a learned correction on top of it almost always outperforms either pure physics or pure data, and it extrapolates because the backbone carries structure.
Lift, then check linearity. Koopman embeddings are powerful when a good set of observables exists, but a poor lift gives a linear model that fits the training trajectory and fails elsewhere. Validate the linear prediction on held-out initial conditions.
Transformers are data-hungry. Attention has little built-in bias, so it needs either large data or the help of patching, pretraining and strong positional structure. On small datasets a TCN or a classical model often wins.
Patch length is the first dial. For long series, set the patch length before anything else; it sets both the compute budget and the granularity the model can see.
Predict intervals, calibrate them. A quantile head or a conformal wrapper turns a point forecast into a usable one, but a predicted interval is only worth its nominal coverage if you check that coverage on real held-out data.

The one-sentence summary

Structure is the cheapest data you will ever get, so inject what you genuinely know, the governing equation, a partial model, a sequence-aware architecture, weigh it honestly against the measurements rather than letting either dominate, and remember that a prior is a liability the moment it is wrong, which is why every structured model still has to be validated on data it has never seen.

Section 10

References and further reading

Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). "Physics-informed neural networks." Journal of Computational Physics. The paper that defined the data-plus-equation training objective.
Karniadakis, G. E., et al. (2021). "Physics-informed machine learning." Nature Reviews Physics. A broad survey of physics priors and grey-box hybrids.
Brunton, S. L., & Kutz, J. N. Data-Driven Science and Engineering. Cambridge University Press. Koopman theory, dynamic mode decomposition, and the lifting idea of section four.
Schmid, P. J. (2010). "Dynamic mode decomposition of numerical and experimental data." Journal of Fluid Mechanics. The DMD algorithm used to learn the linear operator.
Vaswani, A., et al. (2017). "Attention is all you need." NeurIPS. The transformer, positional encoding, and multi-head attention.
Nie, Y., et al. (2023). "A time series is worth 64 words." ICLR. PatchTST and the patching strategy of section six.
Bai, S., Kolter, J. Z., & Koltun, V. (2018). "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling." arXiv. The temporal convolutional network and its dilated receptive field.
Koenker, R., & Bassett, G. (1978). "Regression quantiles." Econometrica. Quantile regression and the asymmetric loss behind probabilistic forecasting.

References point to further study; the explanations and simulations above are written and implemented from first principles rather than drawn from any single source.

Two routes to the same destination: physics builds structure in, attention learns it. Both let a model say more from less, and both, like everything in this series, are only as trustworthy as the data they are checked against. With part five, this completes the deep-learning arc that sits beneath the operational chain of sensing, forecasting, prognosis and monitoring.