GHARP: Real-time Gaussian Head Animation with Decoupled Identity and Expression for Second-level Driving

From a Few Photos to Real-time Driving: GHARP's Approach

Bringing digital humans to life has long been a popular direction in computer graphics, but achieving real-time, high-fidelity expression driving on ordinary hardware with only a few input images remains full of challenges. GHARP (Real-time Gaussian Head Animation from Large-scale Reconstruction Prior) offers a clear engineering compromise: split the whole problem into two stages—offline construction of an identity representation, and runtime prediction of expression residuals.

This decoupled design is very clever. The identity stage is responsible for capturing the person's geometry and appearance details from a few images, such as skin texture, facial contours, and lighting characteristics; the animation stage focuses on generating dynamic offsets based on driving signals (such as expression coefficients or facial landmarks). The offline stage can run slowly and compute heavily, while the online stage must be lightweight and fast. The computational budgets of the two do not interfere with each other, leaving room for real-time performance.

Expression coefficients and 3D parametric face models are the prerequisite for understanding the driving signals in this article. 3DMM (3D Morphable Model) and its subsequent variants (such as FLAME and FaceVerse) use a set of low-dimensional coefficients to separately control identity shape, expression deformation, and albedo. Expression coefficients are usually regressed from audio, video, or landmarks. Such parametric models provide a semantically clear, extremely low-dimensional control interface, but linear bases struggle to express high-frequency details such as wrinkles and dimples. Therefore, the mainstream approach is "parametric model as the base, neural representation for details": use expression coefficients to drive coarse geometry, then let the network learn the remaining high-frequency residuals. GHARP's residual prediction belongs to this paradigm, with the difference that the representation carrying details is replaced by Gaussian point clouds, making online inference lighter.

Gaussian Splatting as the Core Representation

The name GHARP contains "Gaussian," hinting that it relies on 3D Gaussian Splatting, an emerging representation. Compared with Neural Radiance Fields (NeRF), Gaussian Splatting has a natural advantage in rendering speed because each Gaussian sphere can be projected and rasterized in parallel without dense sampling along rays. This is a key prerequisite for real-time animation.

According to the paper description, the identity stage uses large-scale reconstruction priors to initialize or constrain the Gaussian point cloud. This means the model does not need to guess the head shape from scratch but stands on the "shoulders" of existing reconstruction data, converging faster to a high-quality identity representation. The introduction of this prior allows the method to remain stable even with very few input views.

The animation stage then predicts expression-related residuals on top of this static representation—which can be understood as displacement, rotation, or scale corrections for each Gaussian point as the expression changes. Since the residuals are local adjustments relative to the identity representation, the network does not need to re-encode the entire head, greatly reducing computation.

3D Gaussian Splatting is a scene representation and rendering method proposed at SIGGRAPH 2023. It explicitly models the scene with a large number of anisotropic 3D Gaussian ellipsoids: each Gaussian sphere has a position, a covariance matrix (controlling shape and orientation), opacity, and color (usually expressed as spherical harmonic coefficients for view-dependent appearance). During rendering, all Gaussian spheres are first projected into screen space, then depth-sorted and rasterized with differentiable alpha blending. The entire process is fully parallelized and does not rely on point-wise queries from a neural network.

Unlike NeRF's volume rendering with dense sampling along rays, the rendering cost of Gaussian Splatting mainly depends on the number of Gaussian spheres and their screen coverage, so it can easily reach hundreds of frames per second. This is why it is regarded as new infrastructure for real-time graphics. The cost is higher memory usage, sensitivity of sorting and blending in overlapping regions, and artifacts at extreme viewpoints. For head animation, the explicitness of Gaussian spheres brings an additional benefit: expression residuals can be applied directly to physical quantities such as position, rotation, and scale, without going through the black-box modulation of an implicit field.

Engineering Benefits of Decoupling

Separating identity and animation is not just a training strategy choice; it directly affects deployment experience. The offline stage can spend minutes or even longer on optimization, and the user only needs to wait once; the online stage requires inference within milliseconds per frame to support real-time interaction. GHARP's architecture allows resource allocation on both sides to be independent.

This model is also easy to extend. If the user wants to change the expression driving signal or increase the animation frame rate, only the animation stage needs to be modified, without rerunning expensive identity reconstruction. Conversely, improvements in identity representation quality (such as higher-resolution Gaussian point clouds) will not directly slow down online inference, as long as the residual network remains lightweight.

"Priors" in avatar reconstruction usually refer to general facial knowledge pretrained on large amounts of 3D or video data, such as a generic head template, point cloud distribution in canonical space, or transferable deformation priors. Few-view reconstruction is a typical ill-posed problem: a few photos cannot constrain occluded regions and back-side geometry, and pure per-person optimization easily overfits to the input viewpoints. Introducing a large-scale reconstruction prior is equivalent to adding a strong regularization term to optimization, restricting the solution space to "reasonable shapes that look like human faces." This is consistent with the "pretraining + fine-tuning" idea of general large models. The cost is that the distribution of prior data determines the generalization boundary—if the target person differs greatly from the prior (such as exaggerated makeup or special face shape), the prior may instead introduce bias.

Real-world Imagination for Real-time Head Animation

From an application perspective, the most direct scenarios for methods like GHARP are dynamic avatars in virtual streamers, video conferencing, and social applications. Traditional solutions either rely on professional motion capture equipment or require large amounts of training data, while GHARP emphasizes "few input images," greatly lowering the acquisition threshold. Users may only need a few multi-angle selfies to obtain a real-time drivable 3D head.

Another direction worth watching is mobile. Gaussian Splatting's rendering pipeline is relatively friendly, and combined with the decoupled design, putting heavy computation offline in the cloud and lightweight animation on the terminal is a feasible hybrid architecture. Of course, the paper abstract does not give specific frame rates or hardware metrics, and actual performance still needs to be seen from subsequent experimental data.

Challenges and Points to Be Verified

The decoupled architecture also has costs. The interface between the identity stage and the animation stage—that is, how residuals are expressed and aligned with Gaussian attributes—needs careful design; otherwise, texture drift or geometric distortion can easily occur during expression driving. In addition, the coverage of the large-scale reconstruction prior determines the method's generalization ability to unseen people. If the prior data differs greatly from the target person, the effect may be compromised.

Currently, public information is still limited. Key metrics such as specific real-time numbers, robustness to extreme expressions, and multi-view consistency have not yet been disclosed. For researchers and developers interested in 3D facial animation, GHARP's approach is worth learning from, but actual performance still awaits community verification after the full paper and code are released.