Symmetry of Differential Neural Tangent Kernels under the Orthogonal Group

Introduction

Differential Neural Tangent Kernel (DNTK) is an important tool for analyzing the training dynamics of overparameterized physics-informed neural networks (PINNs). A recent theoretical study delves into the symmetry of infinite-width DNTK under the orthogonal group, providing universal conclusions for two-layer and deep networks based on explicit expressions and recursive formulas of DNTK. This work covers all activation functions with and without bias terms, offering new insights into the structure of infinite-width DNTK.

Neural Tangent Kernel (NTK): NTK is a kernel function that describes the dynamics of infinitely wide neural networks trained by gradient descent. In the infinite-width limit, the network output function can be approximated as a linear model with respect to initial parameters, and its kernel (i.e., NTK) remains constant during training, thereby simplifying the training dynamics of nonlinear networks to kernel regression problems. Differential Neural Tangent Kernel (DNTK) further extends this framework to scenarios requiring computation of network output derivatives with respect to inputs (such as differential operators in physics-informed neural networks), characterizing the inner product of gradients of outputs and their derivatives with respect to parameters. Explicit expressions and recursive formulas of DNTK are fundamental tools for analyzing its spectral properties and generalization behavior.

Core Contributions and Theoretical Framework

The study first derives the symmetry properties of infinite DNTK under the action of the orthogonal group. For two-layer networks, the authors directly analyze the transformation behavior using explicit expressions of DNTK; for deep networks, they characterize it layer by layer through recursive formulas. Interestingly, these results do not depend on the specific form of the activation function, nor are they affected by the presence of bias terms, thus having broad applicability. This lays a theoretical foundation for subsequent analysis of the spectral properties of DNTK.

Spectral Equivalence of Integral Operators

The paper further proves that integral operators on the unit sphere induced by operators on the same group orbit have the same spectrum, and their reproducing kernel Hilbert spaces (RKHS) differ only by an orthogonal transformation. This means that under the action of the orthogonal group, the integral operators corresponding to DNTK are essentially equivalent, thereby simplifying comparisons between different network structures.

Reproducing Kernel Hilbert Space (RKHS): Given a positive definite kernel (K), there exists a unique Hilbert space (\mathcal{H}K) such that (K(\cdot, x)) is an element of this space and satisfies the reproducing property (\langle f, K(\cdot, x)\rangle{\mathcal{H}_K} = f(x)). In kernel methods, learning problems are equivalent to finding the optimal function in the RKHS. Two kernels whose RKHS differ only by an orthogonal transformation means that the function spaces they induce are geometrically isometric, so the solutions to corresponding kernel regression or regularization problems have the same structural properties. This provides a rigorous equivalence framework for comparing DNTK under different network architectures.

Zonal Symmetry and Eigenvalue Decay of Symmetric Operators

For symmetric operators, the paper proves that the DNTK fixed by the operator exhibits zonal symmetry on the unit sphere. Based on this, the authors derive the eigenvalue decay rate of the integral operators for shallow networks (with RePU activation and zero-bias initialization). This result connects symmetry with spectral decay, providing a quantitative tool for understanding the generalization behavior of DNTK.

Orthogonal Group and Group Action: The orthogonal group (O(n)) consists of all linear transformations that preserve the Euclidean inner product, i.e., the set of matrices satisfying (Q^\top Q = I). In the context of kernel functions, considering the orthogonal group action means applying an orthogonal transformation (x \mapsto Qx) to input coordinates and examining the relationship between (K(Qx, Qy)) and (K(x, y)). If a kernel remains invariant under any orthogonal transformation, it is said to have orthogonal invariance. This symmetry analysis can reveal structural constraints of kernel functions, such as representing them as radial or zonal functions, thereby greatly simplifying spectral analysis.

Zonal Symmetry and Eigenvalue Decay: On the unit sphere, if the kernel of an integral operator depends only on the inner product of two unit vectors (i.e., (K(x, y) = f(x^\top y))), the operator is said to have zonal symmetry. The eigenfunctions of such operators are exactly spherical harmonics, and eigenvalues can be explicitly given by the Legendre expansion coefficients of the kernel. The eigenvalue decay rate directly determines the smoothness of the RKHS and generalization error bounds: faster decay means a "smaller" function space, and the model tends to learn smoother functions. For shallow networks with RePU activation, the decay rate given in the paper quantifies the concentration of the DNTK spectrum and is a key indicator for analyzing PINN training dynamics and generalization performance.

Significance and Outlook

This study reveals the structural patterns of infinite-width DNTK from the perspective of symmetry, with conclusions applicable to a wide range of activation functions and network depths. The established spectral equivalence and eigenvalue decay rates provide new theoretical support for analyzing the training dynamics and generalization performance of physics-informed neural networks. Future work may further explore symmetries under other group actions and similar properties in more complex network architectures.

One-line Summary

Infinite-width DNTK has symmetry under the orthogonal group; its integral operators are spectrally equivalent and exhibit zonal symmetry, and eigenvalue decay rates for RePU networks are provided.

Paragraph Summary

This study systematically analyzes the symmetry properties of infinite-width Differential Neural Tangent Kernel (DNTK) under the action of the orthogonal group. Using explicit expressions for two-layer networks and recursive formulas for deep networks, the authors prove that the symmetry conclusions of DNTK do not depend on the specific form of the activation function, nor are they affected by the presence of bias terms. Furthermore, the paper proves that integral operators on the unit sphere induced by operators on the same group orbit have the same spectrum, with their reproducing kernel Hilbert spaces differing only by an orthogonal transformation; the DNTK fixed by symmetric operators exhibits zonal symmetry on the unit sphere, and based on this, eigenvalue decay rates for shallow networks with RePU activation and zero bias are derived. These results provide new theoretical tools for understanding the training dynamics and generalization performance of physics-informed neural networks.