Transformer Surrogates for Physics: Transolver & GeoTransolver

What we are asking the network to do

The goal is a neural surrogate: a network that looks at a car geometry and predicts a physical field over its whole surface and surrounding volume — the pressure pushing on every panel, the velocity of air in every part of the flow — all in one forward pass.

Today those fields come from a CFD solver, which grinds through the fluid equations on a fine grid and can take hours or days per design. A surrogate is trained on many solved cases and then predicts the answer for a new shape in seconds. It does not replace CFD; it lets engineers screen many design variations quickly and reserve the expensive solver for the promising ones.

The geometry arrives as a mesh or point cloud: a collection of N points in space, each carrying coordinates and perhaps a few features (a surface normal, a signed distance, a marker for the inlet or wall). For a detailed vehicle, N can climb into the millions. Whatever architecture we choose has to swallow that many points and produce one prediction per point.

• Input: a mesh / point cloud with N points and their coordinates + features.
• Output: a target field value at each of the N points (for example surface pressure, wall-shear-stress, or volume velocity).
• Constraint: it must stay fast and memory-sane even when N is very large.

Why a plain transformer chokes

Transformers are built around self-attention, where every element looks at every other element to decide what matters. That is exactly the "everything talks to everything" behaviour you want for physics — but it comes at a price.

If there are N points, standard self-attention forms a comparison between every pair of points. That is N × N comparisons — the cost grows like N² (quadratic). The trouble is easy to see with real numbers:

• At 1,000 points, that is a million comparisons — fine.
• At 1,000,000 points, that is a trillion comparisons — and a matrix that cannot fit in memory.

A car mesh sits at the bad end of that scale. So the raw transformer, for all its appeal, is simply unworkable here without a change to how attention is computed. The rest of this article is about that change.

Physics-Attention and "slices"

Transolver (Wu, Luo, Wang & Long, Tsinghua University, ICML 2024) starts from a physical observation: points in similar physical states behave alike. Two mesh points sitting in the same calm wake, or in the same high-pressure stagnation region, will respond to their surroundings in nearly the same way.

So instead of treating all N points as separate individuals, the model groups points that share a physical state and treats them together. It learns to softly assign each mesh point to one of a small, fixed number M of slices — soft clusters of physically-similar points. "Soft" means a point is not forced into exactly one slice; it gets a weight for each slice, and the model learns those weights from data rather than from a hand-drawn rule.

Once the points are gathered into M slice tokens, attention runs among the slices, not among the raw points. Because M is small and fixed (think tens or low hundreds), that attention is cheap. The result for each slice is then scattered back to the points that belong to it.

One Physics-Attention block, step by step

A single Physics-Attention block does four things. Stack several of them, the way a normal transformer stacks its layers, and you have Transolver's core.

1. Soft assignment. Map each point's features to a set of slice weights — how strongly this point belongs to each of the M slices.
2. Points → slices (aggregate). Use those weights to gather all the points into M slice tokens. This is like a cross-attention where the slices ask the points, "what is your state?"
3. Self-attention among slices. Let the M slice tokens attend to each other. This is where global physical interactions are modelled — and it is cheap because M is small.
4. Slices → points (scatter). Send each updated slice token back to the points that belong to it, using the same soft weights, so every point gets an updated feature.

Why the cost is now linear

The two heavy operations — gathering N points into M slices, and scattering back — each cost about N × M. The attention in the middle costs about M × M, which is tiny. So the total is dominated by N × M, and since M is a small fixed number, the block scales linearly with N instead of quadratically. That is the difference between "runs on a million points" and "runs out of memory." Recent work frames this Physics-Attention as a form of linear attention, connecting it to the broader family of efficient transformers.

Transolver as a complete model

Zoom out from the single block and Transolver looks just like a transformer, with Physics-Attention swapped in for ordinary attention.

• Encoder. Embed each point's coordinates and features into a working representation.
• Body. A stack of Physics-Attention blocks — the transformer "layers" of this architecture.
• Decoder. Read out the target field value at every point.

Because the whole stack is linear in N, it can be applied to large, real geometries rather than toy grids.

Results, as reported

The Transolver paper reports state-of-the-art accuracy across a suite of standard partial-differential-equation (PDE) benchmarks, with about a 22% average reduction in relative error across six of them versus prior methods. It is shown to work on industrial car and airfoil designs, not just clean academic domains. A follow-up, Transolver++, extends the same slicing idea to million-scale geometries, pushing the approach toward the point counts real vehicles demand.

Making it geometry-aware

GeoTransolver (NVIDIA, released in PhysicsNeMo, 2025) keeps everything good about Transolver — the physics-aware attention on learned slices — and adds a strong sense of geometry and operating conditions that a plain slice attention can miss.

The core change is GALE (Geometry-Aware Latent Embeddings). Where Transolver's block does plain attention among slices, GeoTransolver adds cross-attention to a shared context. That context is a compact latent representation that encodes:

• the geometry of the object,
• the global parameters of the case (for example inflow speed or a design variable), and
• the boundary conditions.

The context is built once and reused in every block, so the model does not have to re-derive "what shape am I looking at, and under what conditions" at every layer. It is computed from multi-scale ball queries around points — an idea borrowed from DoMINO: for each location you gather neighbours at several radii, so you capture both the fine near-surface detail (small radius) and the far-field effects (large radius) at the same time.

What GeoTransolver adds over Transolver

The practical benefits are better accuracy, more robustness when the geometry or the operating regime shifts away from the training set, and better data efficiency (good results from less training data). It is benchmarked on DrivAerML and the Luminary SHIFT-SUV / SHIFT-Wing datasets against DoMINO, Transolver, and AB-UPT, comparing both integrated quantities (drag and lift) and Relative L1 field errors.

Using these models in practice

• Inputs. A mesh or point cloud with per-point coordinates (plus any available features such as normals or region markers).
• Outputs. Surface fields like pressure and wall-shear-stress, and volume fields like velocity and pressure.
• Training data. Both models learn from CFD-solved cases; the quality and coverage of that data sets the ceiling on what the surrogate can do.
• The selling point. Linear-in-N cost. That is what lets these models touch million-point geometries where a plain transformer cannot.
• The main knob. The number of slices M is a capacity dial — more slices mean more expressive power and more compute; fewer means faster and lighter. Tune it for your problem.
• Availability. Both are available in NVIDIA PhysicsNeMo, with documentation for external aerodynamics on irregular meshes.

References