Eye-Tracking + Psychophysical Attention Distribution: A New Approach to Reward Model Preference Prediction

From Eye-Tracking to Attention Distribution: A New Signal for Reward Modeling

Integrating human feedback into language models is a key path to aligning model outputs with human preferences, and RLHF (Reinforcement Learning from Human Feedback) is the most representative framework for this. In recent years, some research has attempted to use eye-tracking (ET) data as an additional supervision signal, grounding reward model learning in real human reading behavior. Building on this, a new work further introduces well-established findings from cognitive psychology on human letter recognition, modeling visual attention as a spatially graded distribution centered on the current fixation point, and incorporates this representation into reward model training.

Core conclusion up front: Experiments show that reward models trained with this attention distribution achieve higher preference prediction accuracy than baseline models; some experimental results are qualitatively consistent with existing findings on human attention allocation during reading, suggesting the model may capture cognitively meaningful attention allocation characteristics.

Why Eye-Tracking Data Is Worth Introducing into Reward Models

Traditional RLHF preference annotations are typically "holistic judgments"—annotators view a pair of responses and conclude which is better. This signal is sparse and loses details from the reading process: which sentence a person lingered on longer, where they looked back, where they skipped. Eye-tracking data fills exactly this gap, decomposing "preference" into observable reading behavior.

Earlier fixation-centric models already validated the feasibility of this path, but their handling of fixation points was relatively coarse: they often treated the current fixation position as a single point, or considered only discrete fixation events. The problem is that human attention during reading does not fall precisely on a single letter, but forms a graded perceptual range around the fixation point—this is precisely the classic conclusion of letter recognition research in cognitive psychology. What the new work does is bring this "graded distribution" assumption into the reward model.

RLHF (Reinforcement Learning from Human Feedback) is the mainstream paradigm for aligning large language models today, with a typical three-step process: first, collect human preference comparisons on model outputs (e.g., ranking two responses to the same prompt) and train a reward model to fit human preferences; then use reinforcement learning algorithms (such as PPO) to optimize the language model so that its outputs receive higher reward scores. The reward model plays the role of a "human preference proxy" in this framework, and its quality directly determines the final alignment outcome. Therefore, how to provide richer, less noisy supervision signals for the reward model is one of the core problems in RLHF research. The introduction of physiological behavioral data such as eye-tracking follows this line of thinking, attempting to supplement sparse holistic preference labels with fine-grained behavioral evidence from the reading process.

Fixation-centric model refers to methods that model visual attention using individual fixation points or discrete fixation events as the basic unit. Data recorded by eye trackers is essentially a sequence of fixation coordinates and durations; early research often treated each fixation as an independent attention landing point, or only counted event-level features such as fixation duration and regression counts. The limitation of such approaches is that they ignore the spatial perceptual range around the fixation point: the human eye still obtains clear visual information near the fixation point, and recognition ability declines smoothly with increasing eccentricity. The new work replaces discrete fixation points with a continuous spatial decay distribution, precisely correcting this simplified assumption of fixation-centric models.

Method: Modeling Attention as a Spatially Graded Distribution

The specific approach of this research is to represent visual attention as a distribution centered on the current fixation position that decays with spatial distance, rather than as isolated fixation points. This distribution is then encoded into the reward model training process, acting as an additional supervision signal alongside preference labels.

The cleverness of this design lies in its "psychophysical" foundation. Letter recognition research has long shown that the accuracy of character recognition at different positions around the fixation point exhibits systematic differences, with attention resources allocated by spatial position. Injecting this prior into the model is equivalent to providing the reward model with a structured inductive bias that is closer to human perceptual mechanisms, rather than letting the model guess from scratch how eye-tracking data should be interpreted.

From an engineering perspective, this representation is also easier to integrate with existing reward model pipelines: it does not change the overall framework of preference learning, but only enriches the form of the input signal.

The "visual span" and eccentricity effect in letter recognition are classic findings in cognitive psychology: the foveal region of the human eye has the highest resolution, and as stimuli deviate from the fixation point (increasing eccentricity), character recognition accuracy systematically declines, forming the so-called "visual span" range. This phenomenon can be described by a Gaussian or similar decay function, i.e., attention resources are continuously allocated around the fixation point according to spatial distance. Parameterizing this prior as a distribution centered on the fixation point that decays with distance is essentially injecting a validated perceptual regularity from psychophysics as an inductive bias into the model, so that the reward model does not need to learn from scratch from limited eye-tracking data "how attention varies with position," thereby reducing sample requirements and improving generalization.

Experimental Results: Accuracy Improvement and Cognitive Consistency

The main quantitative result reported in the paper is: reward models trained with the proposed fixation distribution achieve higher preference prediction accuracy than baseline models. It should be noted that the original abstract does not disclose specific accuracy values or improvement magnitudes, but the directional conclusion of "higher" is clear.

More interesting are the qualitative findings. Some experimental results are qualitatively consistent with existing research on human attention allocation during reading, and the researchers accordingly believe that the model may indeed capture cognitively meaningful aspects of human attention allocation. This is more noteworthy than a mere metric improvement—it suggests that what the model learns is not accidental statistical patterns in the data, but something isomorphic to human reading mechanisms.

However, caution is also warranted: qualitative consistency does not equal identical mechanisms, and evidence at the abstract level is not yet sufficient to prove that the model internally truly reproduces the human attention allocation process.

Significance and Limitations: A Path Worth Pursuing

The value of this work lies in demonstrating the potential of "cognitive science priors + multimodal supervision signals" in reward modeling. One of the core bottlenecks of RLHF is the noise and sparsity of preference signals; eye-tracking data provides finer-grained behavioral evidence, and psychological models provide the correct interpretive framework for this evidence. The combination of the two brings the alignment objective of reward models closer to "how humans actually read and judge," rather than abstract preference labels.

Limitations are also evident. Eye-tracking data collection is costly and limited in scale, and the scalability of such methods remains to be validated; whether the parameterization of attention distribution applies to non-reading tasks is also an open question. The paper itself currently provides only directional conclusions, lacking public disclosure of specific improvement magnitudes.

For teams focused on alignment research, this path is worth tracking: it suggests that the next improvement in reward models may come not only from larger preference datasets, but also from more refined modeling of human perceptual and cognitive mechanisms.

Inductive bias refers to the structural assumptions beyond the training data itself that a model relies on during learning, such as the locality assumption of convolutional networks or the temporal assumption of recurrent networks. Appropriate inductive bias can narrow the hypothesis space and improve sample efficiency and generalization; inappropriate bias can limit model expressiveness. This research injects the psychological prior that "attention decays with spatial distance" as an inductive bias into the reward model, effectively telling the model how eye-tracking signals should be interpreted rather than letting it discover patterns from data on its own. This also explains why the design is more feasible in eye-tracking scenarios with limited data scale, while suggesting that its applicability may be limited to reading-type tasks where this prior holds.

One-line Summary

Research models eye-tracking attention as a spatially graded distribution and injects it into reward models, improving preference prediction accuracy.

Paragraph Summary

This research focuses on improving supervision signals for reward models in RLHF, proposing to introduce eye-tracking data into reward model training. Addressing the coarse treatment of fixation points as discrete points in previous fixation-centric models, the authors draw on the conclusion from cognitive psychology letter recognition research that "attention is spatially graded around the fixation point," modeling visual attention as a distribution centered on the current fixation position that decays with distance, and using it as an additional supervision signal alongside preference labels to train the reward model. Experiments show that reward models trained with this method achieve higher preference prediction accuracy than baselines, and some results are qualitatively consistent with existing findings on human attention allocation during reading. The article also notes that eye-tracking data collection is costly and scalability remains to be validated, and whether the parameterization of attention distribution can be extended to non-reading tasks remains an open question.