Operator Learning / Scientific ML
How FNO and KNO learn to map one whole function to another, so a trained network can replace a slow physics solver, and stay correct even when you change the grid.
A normal neural network learns a map between vectors of fixed size. Photo in, label out. The input is a fixed list of numbers, and the output is a fixed list of numbers.
A neural operator learns something bigger: a map between two functions. You feed it a whole field, for example temperature across a metal plate, and it returns another whole field, for example that temperature one second later. The input is a function. The output is a function. The thing that turns one into the other is called an operator.
Why care? Because the laws of physics are written as partial differential equations (PDEs), and the solution of a PDE is exactly this kind of map: give it the starting condition and material properties (functions), and it produces the solution (a function). A classical solver computes that map slowly, step by step, on a fine mesh. A neural operator learns the map once, then applies it in a single fast pass.
But there is an honest catch worth seeing up front. The grid only stays harmless if it is fine enough to actually resolve the wiggles in your function. Sample too coarsely and a fast feature disguises itself as a slow one. This is aliasing, and it is the real limit behind every grid based method.
The teal curve is the true function (seven full cycles). The dots are the samples a grid actually sees, joined by straight lines in rust. Slide down toward few samples and watch the rust reconstruction collapse into a completely wrong slow wave.
The rule of thumb (Nyquist): you need more than two samples per cycle to capture a feature at all. Below that line, no method recovers the truth, and a neural operator is only invariant across grids that both clear this bar. Above it, the operator genuinely does not care which grid you used.
FNO and KNO are two flavors of the same skeleton. Almost every neural operator is built from three moves:
Formally each layer updates the feature field v like this:
Read it plainly: the new value at point x equals a nonlinearity σ applied to a local part (just point x) plus a global part K that gathers information from everywhere. What is K, and how do we make it cheap? That single question is the whole story of FNO and KNO.
FNO answers the question with one clever trick: do the global mixing in frequency space.
The global part K is meant to be a convolution: blend each point with its neighbors using some learned blur pattern. Convolving directly over a big grid is expensive. But there is a famous shortcut, the convolution theorem:
Inside each layer, FNO takes the feature field and:
R (this is the only thing trained in the global path),W·v as a skip path, then applies the nonlinearity.N·log N.This is the exact move at the heart of FNO. Drag the slider to choose how many of the lowest frequency modes to keep. The teal curve is the true signal; the rust curve is what you can rebuild from just those modes.
energy in each frequency mode (kept modes glow, dropped modes fade)
Notice how just a handful of modes already capture the broad shape, and adding more only sharpens fine detail. FNO bets that for physics, the broad shape is most of the answer, so it keeps a small kmax and trains weights only on those.
Here is the full journey of one prediction, top to bottom.
KNO answers the global mixing question from a different angle: make the dynamics linear, then advancing in time is just repeated matrix multiplication.
Real systems are nonlinear and tangled, which is what makes them hard to predict far ahead. Koopman theory offers a beautiful escape hatch: even a wildly nonlinear system becomes a linear system if you stop watching the raw state and instead watch the right set of observables, that is, smart measurement functions of the state. The price is that this linear world can be very high dimensional. The reward is that linear systems are trivial to push forward in time.
z → K·z. Then forecasting t steps ahead is just applying the operator t times: K·K··K. No step by step nonlinear solver needed.
KNO learns the coordinate change with a neural network and learns the linear step too:
z, the space where life is linear.K. A learned linear map advances z one step forward in time. To capture the natural oscillation modes efficiently, this step is often computed in frequency space, so a Fourier transform appears here too, but its job is to move the linear dynamics, not to do convolution like in FNO.K as many times as the number of future steps you want. Because it is linear, this is cheap and stable for long horizons.Here is the payoff, made visible. Both a Koopman style linear operator and an ordinary one step nonlinear predictor can be rolled forward in time. The difference is how their error behaves. A nonlinear predictor with a tiny gain error compounds it every step, so small mistakes snowball. A linear operator whose modes (eigenvalues) are pinned to the right place stays disciplined far longer.
Teal is the true signal of a decaying oscillation. Rust is a Koopman style linear march; ochre is a nonlinear predictor that re-applies its own slightly imperfect step over and over. Drag the horizon and watch which one holds.
This is a toy, not a trained model, but the mechanism is real: the danger in long rollouts is compounding, and KNO fights it by fitting the spectrum of one linear operator instead of re-running a nonlinear step. The flip side, in the limitations section below, is what happens if those eigenvalues drift outside the safe zone.
FNO and KNO both transform a field as a whole. DeepONet reaches the same goal from a different direction, by splitting the job into two small networks that meet at the end.
It rests on a classic result (the universal approximation theorem for operators): any operator can be written as a sum of products, one factor depending on the input function, the other on where you want to read the answer. DeepONet simply learns those two factors with two networks:
y (where you want the output) and produces a matching list of basis values. It answers: where are we asking?y. Ask at many y and you rebuild the whole output function.The neat part is the trunk. Because it takes any location y, you can query the output anywhere, including points you never trained on. That is DeepONet's own route to being grid free. Its weak spot: the input has to be sampled at a fixed sensor layout, so unlike FNO it is not naturally invariant to changing the input resolution.
None of this is magic. You teach a neural operator the same way you teach any network: show it examples and correct its mistakes. The only twist is that each example is a pair of functions.
Two details carry most of the weight. First, the loss is usually a relative L2 error over the entire output field, so the network is graded on the whole function at once. Second, the expensive solver runs only during data generation; at deployment the trained operator replaces it and runs in a single fast pass. That trade, slow once to fast forever, is the entire economic argument for operator learning.
Same goal, learn an operator. Different bet on what makes it cheap.
| FNO | KNO | DeepONet | |
|---|---|---|---|
| Core idea | Global convolution as a multiply in frequency space. | Lift to coordinates where dynamics are linear, then iterate. | Split into branch (input) and trunk (location), combine by dot product. |
| What is learned | Per mode weights R, plus lift, local, project. | Encoder, decoder, and the linear operator K. | Two networks: branch and trunk. |
| Best at | One shot field to field; spatial PDEs. | Long horizon time forecasting. | Flexible input to output maps; query anywhere. |
| Grid | Invariant across resolutions. | Mesh free, works from samples. | Output query free; input fixed at sensors. |
| One line | σ( Wv + IFFT(R·FFT(v)) ) | Decode( Kt·Encode(u) ) | u(y) = Σ bk(a)·tk(y) |
Costs are per forward solve on a grid of N points, for L layers or T time steps.
| Method | Cost per solve | Resolution invariant | Mesh free |
|---|---|---|---|
| Classical PDE solver | large, many iterations to converge | no | depends on mesh |
| FNO | about L · N log N | yes | no (regular grid) |
| KNO | encode + T linear steps + decode | effectively | yes |
| DeepONet | one branch pass + one trunk pass per query | partly | output yes |
The headline is the same in every row: the heavy cost moves to a one time training stage, and each later prediction is cheap.
A fair monograph names the failure modes, not just the wins. None of these are dealbreakers, but each one shapes when you reach for which tool.
The FFT assumes a regular grid and periodic boundaries. Real parts have holes, curved walls and irregular meshes.
Fix: Geo-FNO warps odd geometry into a regular grid; GINO adds a graph layer at the messy boundary.
Keeping only low modes is also a blind spot. Sharp shocks, cracks and fine textures live in the high modes FNO throws away.
Fix: keep more modes where it matters, or add local layers that carry the fine detail.
The whole long horizon promise rests on the Koopman eigenvalues sitting inside the unit circle. If training nudges one outside, the linear march amplifies every step and the forecast blows up, exactly the ochre curve in the demo above.
Fix: constrain or regularize the spectrum so the operator stays stable.
Every operator is trained on a family of inputs. Feed it a case far outside that family and it extrapolates poorly, often without warning.
Fix: cover the operating range when generating training pairs, and watch the residual at run time.
For vehicle and prognostics work, neural operators are not just a PDE curiosity. They are a fast surrogate for the physics inside a running system, which is exactly what a health monitor needs.
Three connections do most of the work:
A neural operator learns a map from one whole function to another, which is exactly what solving a PDE asks for, and unlike a CNN it stays valid when you change the grid, as long as the grid clears the Nyquist bar. Every such network lifts, mixes globally and locally a few times, then projects. FNO makes the global mix cheap by jumping into frequency space, keeping a few low modes, multiplying by learned weights, and jumping back. KNO learns coordinates where the system is linear, then marches that linear operator forward in time. DeepONet splits the work into a branch and a trunk that meet at a dot product. All three pay for a slow solver once, then run fast forever, which is what lets a trained operator stand in as a physics surrogate, expose faults through its residual, and forecast remaining life.