HotPOT: How Heterogeneous Physics Experts Solve the PDE Pretraining Interference Problem

Large-scale PDE pretraining attempts to cover data from multiple partial differential equations with a single model, but different equations differ greatly in their governing laws, physical variables, and the ways variables interact, and simply mixing data does not always improve prediction performance. Existing PDE mixture-of-experts (MoE) models alleviate cross-task interference through learned routing, but they generally adopt homogeneous expert architectures—all experts share the same network structure, making it difficult to directly reflect the essential differences among equations in state evolution. HotPOT was proposed precisely to address this problem at the architectural level.

The Bottleneck of Homogeneous Experts and the Heterogeneous Idea

The core assumption of mixture-of-experts models in PDE pretraining is that different equations share some common representations while experts are needed to absorb their respective special characteristics. The problem is that if all experts are homogeneous, then "special characteristics" can only be expressed through parameter differences, not through computational structure. PDE state evolution is often determined by numerical update rules, such as finite differences, spectral methods, or finite volume schemes, and these rules themselves encode structural information about the equation. Homogeneous experts can only learn these rules implicitly from data, which is less efficient and more data-hungry.

HotPOT seizes precisely this gap: it gives each PDE family a neural-numerical expert that pairs a learnable predictor with a fixed numerical update. The fixed numerical update contains no trainable parameters and is responsible for computing the next state according to the equation's known numerical scheme; the learnable predictor outputs the quantities required by the numerical update. This division of labor separates "what to learn" from "how to compute," so that experts no longer differ purely in parameters but in computational workflow.

Mixture-of-experts (MoE) core idea is to train multiple "expert" subnetworks and use a routing network to decide which experts process each input. In PDE pretraining scenarios, routing usually assigns different equations to different experts based on equation type or input features, thereby reducing cross-task interference. However, experts in traditional MoE often share the same network architecture (such as homogeneous MLP or Transformer blocks) and differ only in parameters. This means that differences among experts rely entirely on data-driven learning rather than structural priors. For PDEs with known physical laws, this homogeneous design cannot directly exploit structured knowledge such as numerical schemes, leading to lower learning efficiency and higher data requirements. HotPOT's heterogeneous experts target this limitation by embedding numerical update rules as fixed components within experts, creating essential differences in computational workflow among experts.

HotPOT's Architecture Design

HotPOT adopts an overall structure of a shared encoder plus heterogeneous experts. The shared encoder is responsible for capturing common information across PDEs, avoiding the need for each equation to learn general representations from scratch. Each PDE family corresponds to a neural-numerical expert, in which a learnable predictor generates the quantities required for the numerical update based on the encoder output, and the fixed numerical update then computes the next state from these quantities.

The direct benefit of this design is that different PDEs can share the same encoder while retaining their own learnable predictors and numerical updates. The shared part reduces the data pressure of pretraining, while the heterogeneous part preserves structural differences at the equation level. Compared with purely data-driven homogeneous MoE, HotPOT embeds numerical schemes as priors inside experts, reducing the space in which experts must implicitly infer physical laws from data.

Neural-numerical mixture of experts is a design paradigm that combines learnable neural networks with fixed numerical solvers. The fixed numerical update usually corresponds to some known discretization scheme (such as finite differences or spectral methods); it contains no trainable parameters and is responsible for advancing the state according to the equation's structured rules. The learnable predictor is responsible for outputting the intermediate quantities required by the numerical scheme (such as fluxes, source terms, or closure terms). This division of labor enables the model to decouple "known physical laws" from "unknown constitutive relations," reducing the burden on neural networks to implicitly learn numerical schemes from data. Compared with purely data-driven experts, neural-numerical experts are more data-efficient in scenarios where physical laws are known but some terms are unknown, and they also provide structured inductive bias for cross-equation generalization.

Experimental Performance: Winning All 24 Comparisons

According to the paper, the research team conducted experiments on five PDE families under six settings, covering 24 same-scale comparisons before and after fine-tuning, and HotPOT outperformed MoE-POT in all comparisons, reducing rollout error by an average of 50.7%. Even more noteworthy is the relationship between model scale and performance: the 17M-parameter HotPOT model also surpassed the significantly larger MoE-POT model, indicating that the gains from architectural heterogeneity are not offset by the parameter gap.

The performance gains also extend to zero-shot prediction under unseen physical conditions, meaning that what HotPOT learns is not overfitting to the training equation families but transferable numerical-physical representations. The paper's code has been open-sourced on GitHub at cruiseresearchgroup/HotPOT-code.

Implications for the PDE Pretraining Roadmap

The value of HotPOT lies not only in a set of experimental numbers. It suggests a path to improving pretraining that differs from "stacking parameters and stacking data": embed the known numerical structure of equations as non-trainable fixed components within experts, and let the learnable part focus on quantities that numerical updates cannot directly provide. This neural-numerical hybrid expert design is naturally suited to scenarios where physical laws are known but constitutive relations or closure terms are unknown.

Of course, this path also has its applicable boundaries. Fixed numerical updates require that each PDE family have a clear numerical scheme available; for problems where numerical schemes are not yet mature or the equation form is uncertain, the construction cost of heterogeneous experts will rise. In addition, the paper's experiments cover five PDE families, and generalization ability across more equation types still needs further validation. But at least on the core pain point of cross-equation interference, HotPOT provides a structurally clear and reproducible solution.

One-line Summary

HotPOT uses a shared encoder plus heterogeneous neural-numerical experts and wins all 24 PDE pretraining comparisons.

Paragraph Summary

To address the difficulty that homogeneous mixture-of-experts (MoE) models have in reflecting structural differences among equations in large-scale PDE pretraining, HotPOT proposes a shared encoder plus heterogeneous neural-numerical expert architecture. Each PDE family has an expert that pairs a learnable predictor with a fixed numerical update, embedding numerical schemes as priors and reducing the data pressure of implicitly learning physical laws. Across five PDE families and six settings, HotPOT outperformed MoE-POT in all 24 same-scale comparisons, reducing rollout error by an average of 50.7%; its 17M-parameter model surpassed larger baselines, and zero-shot prediction also demonstrated transferability. This work offers a new path for PDE pretraining: embedding numerical structure rather than simply stacking parameters.