{"id":116,"date":"2026-10-16T14:21:53","date_gmt":"2026-10-16T14:21:53","guid":{"rendered":"https:\/\/evoxia.online\/?p=116"},"modified":"2026-10-16T14:21:53","modified_gmt":"2026-10-16T14:21:53","slug":"rlhf-and-preference-optimization-training-language-models-on-which-answer-people-prefer","status":"publish","type":"post","link":"https:\/\/evoxia.online\/?p=116","title":{"rendered":"RLHF and Preference Optimization: Training Language Models on Which Answer People Prefer"},"content":{"rendered":"\n

A Model That Imitates Everything It Read<\/strong><\/p>\n\n\n\n

A language model trained only on next-token prediction learns to continue text the way text on the internet tends to continue. Ask it a question and it may answer, or it may produce three more questions, because a list of questions is a perfectly plausible continuation. It has no built-in idea that one continuation is helpful and another is not. It only knows which one is likely.<\/p>\n\n\n\n

Supervised fine-tuning on written demonstrations helps, but it has a ceiling. Someone has to write every good answer by hand, and the model learns to copy the style of those answers rather than to understand what made them good. There is also a practical asymmetry: most people cannot write an excellent summary of a legal document on demand, but almost anyone can read two summaries and say which one is better. Reinforcement learning from human feedback (RLHF) is built on that asymmetry.<\/p>\n\n\n\n

Comparisons Instead of Scores<\/strong><\/p>\n\n\n\n

The idea predates chatbots. In 2017, Paul Christiano, Jan Leike and colleagues at OpenAI and DeepMind published \u00abDeep Reinforcement Learning from Human Preferences\u00bb at NeurIPS. Their agents played Atari games and controlled simulated robots, and instead of receiving a reward from the environment, they received human judgments about pairs of short video clips: which of these two looks more like what I want?<\/p>\n\n\n\n

It needed very little human input. The paper reports that the method worked while humans gave feedback on less than 1% of the agent’s interactions with the environment, and that complex new behaviors could be trained with about an hour of human time. The best known example was a simulated robot learning to do a backflip, a behavior that is hard to write as a reward function but easy to recognize. According to OpenAI, it took around 900 bits of feedback from the evaluator.<\/p>\n\n\n\n

Asking for comparisons rather than numeric scores is a deliberate choice. People are inconsistent when asked to rate something from 1 to 10, and two raters rarely share the same scale. They are much more consistent when asked to pick one of two.<\/p>\n\n\n\n

The Reward Model<\/strong><\/p>\n\n\n\n

Comparisons cannot be optimized directly, so the first step is to turn them into a function. A reward model, usually a copy of the language model with its output layer replaced by a single number, is trained to give the preferred response a higher score than the rejected one.<\/p>\n\n\n\n

The training objective comes from the Bradley-Terry model, proposed by Ralph Bradley and Milton Terry in 1952 for ranking items from paired comparisons. It says that the probability a person prefers response A over response B is the sigmoid of the difference between their scores. Training the reward model means maximizing the log-likelihood of the human choices under that assumption. Only the difference between two scores matters, so the absolute value of a reward means nothing on its own.<\/p>\n\n\n\n

Once trained, the reward model can score any new response in milliseconds. That is the point: a human cannot sit in the loop for millions of training steps, but a model of the human can.<\/p>\n\n\n\n

PPO and the KL Penalty<\/strong><\/p>\n\n\n\n

The second step treats the language model as a policy. A prompt is the state, the generated response is the action, and the reward model provides the score at the end. The usual algorithm is Proximal Policy Optimization (PPO), introduced by John Schulman and colleagues in 2017, which limits how far each update can move the policy.<\/p>\n\n\n\n

Left alone, this optimization finds strange outputs. The reward model was trained on responses from the original model, and once the policy drifts into text that looks nothing like that distribution, the reward model’s scores stop meaning anything. To prevent this, the reward is reduced by a penalty proportional to the KL divergence between the current policy and the original supervised model. The model is pushed toward higher reward but kept close to the language it already knows how to produce. The strength of that penalty, usually called beta, is one of the most important settings in the whole pipeline.<\/p>\n\n\n\n

In practice this means running four models at once: the policy being trained, the frozen reference model, the reward model, and a value network that PPO uses to estimate expected reward. That cost, and PPO’s sensitivity to hyperparameters, explains much of what came later.<\/p>\n\n\n\n

Case: InstructGPT<\/strong><\/p>\n\n\n\n

Two papers made RLHF work on language. In 2020, Nisan Stiennon and colleagues applied it to summarizing Reddit posts. OpenAI reported that a 1.3 billion parameter model trained with human feedback outperformed a 12 billion parameter model trained only with supervised learning, and labelers preferred the feedback models’ summaries to the human-written ones in the dataset.<\/p>\n\n\n\n

In 2022, Long Ouyang and colleagues published InstructGPT (NeurIPS 2022), the method behind the first version of ChatGPT. The pipeline had three stages: supervised fine-tuning of GPT-3 on demonstrations written by labelers, a reward model trained on rankings of model outputs, and PPO against that reward model. The headline result was that people preferred outputs from the 1.3 billion parameter InstructGPT model to those of the 175 billion parameter GPT-3, a model with roughly 100 times more parameters. The paper also reported better truthfulness and less toxic output, with small regressions on some public NLP benchmarks.<\/p>\n\n\n\n

That comparison is the one people remember. A large part of what users experienced as quality was not coming from more parameters. It came from rankings that labelers wrote for roughly 33,000 prompts.<\/p>\n\n\n\n

Reward Hacking<\/strong><\/p>\n\n\n\n

A reward model is an approximation of what people want, and optimizing hard against an approximation eventually exploits its errors. Leo Gao, John Schulman and Jacob Hilton measured this in \u00abScaling Laws for Reward Model Overoptimization\u00bb (ICML 2023). They used a large fixed reward model as a stand-in for humans, trained smaller proxy reward models on its labels, and optimized policies against the proxies. The proxy score kept rising, while the score from the \u00abgold\u00bb model rose, peaked, and then fell. Larger reward models delayed the drop but did not remove it.<\/p>\n\n\n\n

The errors that get exploited are often mundane. Stiennon’s team noticed that their labelers favored longer summaries, and the models converged to the longest length allowed. Prasann Singhal and colleagues later found that, in several standard setups, a reward based purely on response length reproduced most of the improvement RLHF gave over supervised fine-tuning. Mrinank Sharma and colleagues at Anthropic (ICLR 2024) found that both humans and preference models sometimes preferred convincingly written answers that agreed with the user over correct ones, which gives the training process a direct incentive toward sycophancy.<\/p>\n\n\n\n

Direct Preference Optimization<\/strong><\/p>\n\n\n\n

In 2023, Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon and Chelsea Finn published \u00abDirect Preference Optimization: Your Language Model is Secretly a Reward Model\u00bb at NeurIPS. Their observation was mathematical. The KL-constrained objective that RLHF optimizes has a known optimal solution: the best policy is the reference model’s distribution reweighted by the exponential of the reward divided by beta.<\/p>\n\n\n\n

That relationship can be inverted. Any policy implies a reward, equal to beta times the log ratio between the policy’s probability of a response and the reference model’s probability, plus a term that depends only on the prompt. Substitute that implied reward into the Bradley-Terry loss and the prompt-only term cancels, because Bradley-Terry only looks at differences. What remains is a loss written entirely in terms of the policy and the reference model: increase the log-probability ratio of the preferred response relative to the rejected one, passed through a sigmoid.<\/p>\n\n\n\n

DPO needs no separate reward model, no sampling during training, and no PPO. It looks like ordinary supervised training on pairs. Many open models adopted it quickly, and variants followed, including IPO (Azar and colleagues) and KTO (Ethayarajh and colleagues), which works from single responses labeled good or bad instead of pairs. DPO does not escape the underlying problem, though. It still learns from the same preference data, with the same biases toward length and agreement.<\/p>\n\n\n\n

Rewards From Rules and From Other Models<\/strong><\/p>\n\n\n\n

Human comparisons are expensive, so later work replaced some of them. Anthropic’s Constitutional AI (Bai and colleagues, 2022) had a model critique and revise its own answers against a written list of principles, then used AI-generated preference labels for the reinforcement learning stage, an approach often called RLAIF.<\/p>\n\n\n\n

For tasks with checkable answers, the reward model can be dropped entirely. DeepSeekMath (Shao and colleagues, 2024) introduced Group Relative Policy Optimization (GRPO), which removes PPO’s value network: it samples several answers to the same question and uses each answer’s reward relative to the group’s average as its advantage. DeepSeek-R1 (2025) combined GRPO with rule-based rewards, checking whether a math answer is correct or whether code passes tests, rather than relying on a learned reward model. A rule is harder to fool than a neural network, but only a narrow set of tasks have rules.<\/p>\n\n\n\n

Closing Note<\/strong><\/p>\n\n\n\n

The name suggests that RLHF teaches a model human values. What it teaches is narrower: the model learns to produce the response that a particular group of raters, reading quickly, would pick from a pair. Most of the time that is close to the response a careful person would want. The cases where it is not (longer is not better, agreeing is not the same as being right) are where the method’s real assumptions show up. Every preference dataset carries a description of its raters, and the model trained on it inherits that description along with the good answers.<\/p>\n\n\n\n

By: Max Johnson B.<\/p>\n","protected":false},"excerpt":{"rendered":"

A Model That Imitates Everything It Read A language model trained only on next-token prediction learns to continue text the way text on the internet tends to continue. Ask it a question and it may answer, or it may produce three more questions, because a list of questions is a perfectly plausible continuation. It has […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-116","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts\/116","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=116"}],"version-history":[{"count":0,"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts\/116\/revisions"}],"wp:attachment":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=116"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=116"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=116"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}