PCHMR Benchmark: Federated Learning for Human Mesh Recovery Under Privacy Constraints

Human Mesh Recovery (HMR) models are increasingly being deployed on real devices, but making these models continuously better typically requires aggregating sensitive user-side human images onto a central server—which is itself a privacy risk. The newly proposed PCHMR benchmark directly confronts this tension: can HMR still be effectively evaluated and improved without centralizing raw images?

The Centralized Training Assumption Is Breaking Down

Existing HMR training pipelines are almost all built on the assumption of "unimpeded access to centralized data." PCHMR's approach is to remove this assumption, training state-of-the-art HMR models with federated learning and secure aggregation variants, then evaluating their performance across a range of client scales and degrees of data heterogeneity.

The value of this work lies not just in producing a number, but in systematically documenting what happens to representative HMR backbones when data is no longer centralized. For teams that rely on a "collect first, then train" paradigm, this benchmark amounts to a stress test.

Federated learning and secure aggregation are key to understanding PCHMR's premise. Federated learning means multiple clients train models locally on their own data and upload only model updates (such as gradients or parameters) to a server for aggregation, so raw data never leaves the local device. Secure Aggregation further ensures that even the server cannot see the plaintext of any individual client's updates—it can only obtain the sum of all updates, thereby preventing user information from being reverse-engineered from updates. Combined, the two make it possible for "collaborative training" and "non-centralized data" to coexist, which is the foundation for PCHMR's ability to evaluate HMR models without touching raw human images.

How HMR Performs Under Federated Learning

PCHMR covers a range of scales from few to many clients and deliberately introduces data heterogeneity—different users vary widely in pose, scene, and body type distribution. The paper evaluates federated learning and secure aggregation variants side by side; neither centralizes raw images, allowing the cost of different collaborative training paths to be compared under equivalent privacy constraints.

It should be noted that the source material does not disclose specific accuracy loss figures, so no quantitative conclusion can be given here. What is certain is that the benchmark's goal is to make "how and by how much HMR performance degrades under privacy constraints" a measurable question rather than one left to intuition.

Data Heterogeneity is often called the Non-IID problem in federated settings: data distributions differ across clients. For HMR, this means different users differ greatly in body type, pose habits, camera viewpoint, and background environment, so local data cannot represent the global distribution. The higher the degree of Non-IID, the more the globally aggregated model tends to skew toward certain clients, leading to overall performance degradation or worsened fairness. PCHMR deliberately varies client scale and heterogeneity precisely to measure the impact of this distribution shift on HMR backbones, rather than reporting only average accuracy under idealized conditions.

DePoser: Unblocking the Local Annotation Bottleneck with Depth Foundation Models

Privacy constraints bring not only changes to training methods but also a data bottleneck: without centralized data, local high-quality annotations are lacking. To address this, the authors propose DePoser, a local annotation and fine-tuning pipeline combined with depth foundation models that enables HMR models obtained through collaborative training to adapt to end users' own data distributions.

On in-the-wild data, fine-tuning with DePoser annotations can improve personalized accuracy. It is important to note that the evaluation is based on pseudo-ground truth generated by DePoser, so this improvement should be understood as an improvement relative to pseudo-ground truth, not as independent validation against absolute ground truth.

Depth Foundation Model refers to models pretrained on large-scale depth data that can estimate high-quality depth maps from monocular images or video. Compared with traditional depth estimation, such models generalize better in in-the-wild scenes and can provide more reliable geometric cues. In DePoser, the depth foundation model's role is to generate 3D geometric supervision signals for local data, which are then combined with human priors or 2D keypoints to automatically produce pseudo-annotations of human meshes, thereby bypassing the "no local annotations" bottleneck. Note that the quality ceiling of the pseudo-ground truth is determined by the depth foundation model, and its errors propagate into subsequent fine-tuning.

What This Benchmark Means

PCHMR's positioning is clear: to provide public support for HMR models under privacy constraints and their real-world deployment and evaluation. It simultaneously reveals problems at two levels—federated feasibility at the training level and local personalization capability at the deployment level. For teams working on on-device human perception, virtual try-on, fitness pose analysis, and similar applications, the significance of such a benchmark is that it turns "privacy compliance" from a product constraint into a comparable research variable.

At the current stage, the combination of federated training and local fine-tuning still requires more public reproductions to confirm its generalizability, and the cost of DePoser's reliance on depth foundation models for annotation is not elaborated in the source material. These are directions worth tracking going forward.

FAQ

Does PCHMR require centralizing user images? No. Both its federated learning and secure aggregation training paths avoid centralizing raw images, which is the premise of the entire benchmark.

Is the fine-tuning effect of DePoser trustworthy? The paper reports personalized accuracy improvements relative to pseudo-ground truth generated by DePoser. This is a self-consistent evaluation method, but it is not equivalent to conclusions under independent ground-truth annotations.

Who is this benchmark for? Researchers concerned with HMR training and deployment under privacy constraints, and product teams that need on-device human mesh capabilities.

One-line Summary

PCHMR evaluates and improves HMR under federated learning and secure aggregation without centralizing raw human images.

Paragraph Summary

PCHMR is a privacy-constrained benchmark for Human Mesh Recovery (HMR). Addressing the widespread reliance of existing HMR training on centralized data, it trains state-of-the-art HMR models with federated learning and secure aggregation variants, and systematically evaluates their performance across a range of client scales and degrees of data heterogeneity. Because raw images are not centralized, local annotation becomes a bottleneck, so the authors further propose DePoser—a local annotation and fine-tuning pipeline combined with depth foundation models that enables personalized adaptation to end-user data distributions. On in-the-wild data, fine-tuning based on DePoser pseudo-ground truth can improve personalized accuracy, but this improvement is a self-consistent evaluation relative to pseudo-ground truth, not independent ground-truth validation. Overall, the benchmark turns privacy compliance into a measurable research variable, serving deployment scenarios such as on-device human perception.