--- title: "SimPO fine-tuning: reference-free preference optimization" description: "SimPO drops the reference model and adds length normalization. +6.4 AlpacaEval 2 and +7.5 Arena-Hard over DPO at lower training memory." source: "https://www.kensink.com/llm/fine-tuning/methods/simpo/" canonical: "https://www.kensink.com/llm/fine-tuning/methods/simpo/" --- [← All methods](https://www.kensink.com/llm/fine-tuning/methods/) · 07 · REFERENCE-FREE DPO · SPECIALISED ★ SimPO Specialised METHOD · SIMPLE PREFERENCE OPTIMIZATION # SimPO. DPO minus the reference model, plus length normalization. Reference-free preference optimization with a length-normalized log-probability reward. Reports +6.4 AlpacaEval 2 and +7.5 Arena-Hard over DPO on the same training data (Meng et al., NeurIPS 2024). The strongest DPO challenger as of 2026. TRL PEFT HuggingFace [Talk to our team →](https://www.kensink.com/contact) [Fine-tuning hub](https://www.kensink.com/llm/fine-tuning/) Gain vs DPO +6.4 AlpacaEval 2, +7.5 Arena-Hard Memory Lower than DPO (no reference model) Trade Slightly higher hyperparameter sensitivity \[WHY THIS EXISTS\] ## DPO carries a reference model and has length bias. DPO needs the SFT reference model in memory for the loss anchor. SimPO drops it: the reward is just the length-normalized log-probability of the response. Same preference data, one less model in memory, length bias controlled. - Reward = mean log-prob of the response under the policy - No reference model = lower memory and faster training - Length normalization stops the policy from rewarding longer outputs - Same (chosen, rejected) data as DPO \[THE PIPELINE\] ## SimPO, end to end. Same data shape as DPO. No reference model in memory. Train. SFT checkpoint Preference pairs SimPO loss (length-normalized) Train 1 to 2 epochs Eval (Arena-Hard) Ship 01 ### SFT base + preference data Same starting point as DPO. SimPO does not change the data layer. 02 ### TRL CPOTrainer with simpo\_gamma + simpo\_alpha TRL supports SimPO via CPOTrainer with loss\_type='simpo'. Defaults are reasonable. Tune gamma (target margin) and the length normalization weight if results look off. \[THE STACK WE'D DEPLOY\] ## What we run in production for SimPO. TRL (CPOTrainer with loss\_type='simpo') PEFT HuggingFace Datasets \[ACCURACY · COST · TRADE\] ## The numbers we measure SimPO on. AlpacaEval 2 vs DPO +6.4 points Same training data Arena-Hard vs DPO +7.5 points Memory at training Lower (no reference model) When it earns the build High-quality preference data, when memory at training is tight, when DPO has been benchmarked and the upgrade is worth the extra hyperparameter discipline. When it doesn't Without an SFT base or preference data, when noisy preference labels make the length normalization unstable. \[ OUR TAKE \] ## Strongest DPO challenger. We benchmark it alongside DPO on new projects. On clean preference data SimPO often wins by the paper's numbers. On noisy data DPO can be more robust. We run both on the same data and pick by held-out evals. \[READ AT THE SOURCE\] ## Papers, docs, and primary sources. - [↗ Meng et al., SimPO (NeurIPS 2024)](https://arxiv.org/abs/2405.14734) - [↗ TRL CPOTrainer (SimPO loss)](https://huggingface.co/docs/trl/main/en/cpo_trainer) \[COMMON QUESTIONS\] ## What buyers ask before they sign. SimPO vs DPO: which should we use? Benchmark both. SimPO has the better headline numbers, DPO has more battle testing. On a new project we run both with the same data and pick by golden-set eval. \[RELATED FINE-TUNING TOPICS\] ## Worth a look next. [ 02 · FINE-TUNING ### Data pipeline Sourcing, PII redaction (Presidio), synthetic data (Distilabel, Nemotron), DEITA quality scoring, MinHash + SemDedup, labeling vendors, feedback loops. Read more](https://www.kensink.com/llm/fine-tuning/data-pipeline/)[ 03 · FINE-TUNING ### Platforms OpenAI RFT, Anthropic on Bedrock, Vertex, Azure Foundry, Databricks Mosaic, Together, Predibase, NeMo Customizer, Modal, Lambda. Side-by-side with our take. Read more](https://www.kensink.com/llm/fine-tuning/platforms/)[ 04 · FINE-TUNING ### By data + compute scale Under 1k examples to over 1M, single A10G to 128 B200. Indicative cost, recommended method, hardware tier. Read more](https://www.kensink.com/llm/fine-tuning/by-scale/)[ 05 · FINE-TUNING ### Custom model build Continued pretraining, SFT, preference optimization (DPO, SimPO, ORPO), reasoning distillation (R1 lineage), model merging (TIES, DARE). The full build pipeline. Read more](https://www.kensink.com/llm/fine-tuning/custom-models/)[ 06 · FINE-TUNING ### Compliance EU AI Act (Article 25 substantial-modification trap), GDPR, HIPAA, FedRAMP, Colorado AI Act, India DPDP, China GenAI Measures. Region-by-region for tuned LLMs. Read more](https://www.kensink.com/llm/fine-tuning/compliance/) FINE-TUNING · KENSINK LABS ## Considering SimPO? Let's pressure-test it first. We benchmark the cheap method first, name the trade, and only deploy the expensive one when the numbers force it. Sized to your data, your evals, your residency. [Start a conversation →](https://www.kensink.com/contact) [All fine-tuning topics](https://www.kensink.com/llm/fine-tuning)