RLCD

The RLHF (Reinforcement Learning from Human Feedback) training method optimizes for the model's overconfidence deficiency, training the model to proactively express uncertainty when uncertain.

Jev claims to use the RLCD training method.

RLCD is a policy-gradient-based reinforcement learning method (an unofficial name coined by the author) that optimizes decision models for strictly proper scoring rules, granting maximum reward only when the model outputs mathematically calibrated probabilities.

RLCD constructs positive and negative sample pairs for contrastive learning, combines reinforcement learning signals to distill model output distributions, and has advantages in scenarios that do not require manually annotated rewards.

RLCD (reinforcement learning with calibrated decision-making) training adopts a teacher-student distillation approach: the teacher gives a probability distribution (such as 0.7/0.2/0.1), the student's raw scores after softmax preserve the ranking but deviate in distribution, and the goal is to make the student's distribution approach the teacher's.

RLCD's reward normalization mechanism amplifies the noisy score function term by 3.6 to 15 times, which is the main reason for its performance gap.

Using an unbiased leave-one-out estimator can recover most of RLCD's performance gap.

The RLCD (Reinforcement Learning from Contrastive Distillation) recipe lags behind standard cross-entropy by 2.5 to 3.0 percentage points.

Jev's training method is called RLCD (Reinforcement Learning for Calibrated Decisions), which uses calibration as the optimization objective.

RLCD (Reinforcement Learning for Calibrated Decisions) is the method behind TypeSafe's product Jev, designed to address RLVR's disregard for confidence.

The key modification in RLCD is replacing the 0/1 reward with a proper scoring rule.

The author speculates that RLCD uses the Brier score as its scoring rule, but TypeSafe has not yet publicly disclosed the specific reward function implementation.

On top of the Brier score, RLCD adds a per-bin calibration penalty term similar to the ECE approach, reducing rewards in overconfident intervals and increasing rewards in underconfident intervals.

The truly effective approach is per-bin attribution of the calibration penalty; only by creating reward differences across different confidence intervals can a meaningful gradient signal be produced.

There is an unverified risk with calibration rewards: they may simply flatten the confidence of all outputs toward a middle value, performing well on the ECE metric while losing discriminative information.

RLCD switches the optimization objective from human satisfaction to decision calibration, meaning the model's output confidence highly matches its actual accuracy rate.

Diogo Almeida spent two years developing a new training method called RLCD (Reinforcement Learning for Calibrated Decisions).

Laya is an open-source decision model evolved from the Jev architecture that surpasses Jev on multiple Jev benchmarks through the RLCD training method.

The architecture being compared against (which the author calls Jev) uses parallel sampling, is trained via RLCD, and outputs confidence distributions and schema selections.

Both the PPO and RLCD approaches attempt to move away from the autoregressive paradigm and use reinforcement learning to produce structured probabilistic or confidence outputs.