--- title: "KTO fine-tuning: preference learning from thumbs" description: "Kahneman-Tversky Optimization trains on single-response binary labels. The right method when production feedback is thumbs, not pairwise comparisons." source: "https://www.kensink.com/llm/fine-tuning/methods/kto/" canonical: "https://www.kensink.com/llm/fine-tuning/methods/kto/" --- [← All methods](https://www.kensink.com/llm/fine-tuning/methods/) · 09 · BINARY-FEEDBACK PREFERENCE · SPECIALISED ★ KTO Specialised METHOD · KAHNEMAN-TVERSKY OPTIMIZATION # KTO. Preference learning from thumbs up/down, not pairs. KTO (Ethayarajh et al., 2024) trains on individual binary signals: this response was good, this one was bad. Drops the (chosen, rejected) pair requirement of DPO and SimPO. Matches reality: production feedback is thumbs, not side-by-side comparisons. TRL PEFT [Talk to our team →](https://www.kensink.com/contact) [Fine-tuning hub](https://www.kensink.com/llm/fine-tuning/) Data shape (prompt, response, good/bad) Source Production thumbs, edit deltas Trade Needs class balance discipline \[WHY THIS EXISTS\] ## Most production feedback is single-response, not paired. DPO needs a (chosen, rejected) pair. Most production captures thumbs on one response at a time. KTO uses the single-response binary signal directly, with a Kahneman-Tversky utility function (loss aversion) to balance the asymmetry between good and bad. - (prompt, response, label) triples, where label is thumbs up or thumbs down - Loss weights bad more heavily than good (loss aversion) - Requires a desirable:undesirable ratio between roughly 1:10 and 10:1 \[THE PIPELINE\] ## KTO, end to end. Capture thumbs in production. Train KTO. Repeat. Production thumbs (prompt, response, label) KTO loss Train Ship + next round 01 ### Capture thumbs from day one Production thumbs and edit deltas (the user's edit becomes the desirable response, the original becomes undesirable). This is the feedback loop the fine-tune runs on. 02 ### Train KTO with desirable\_weight, undesirable\_weight Balance the loss for the class ratio. TRL handles the math. \[THE STACK WE'D DEPLOY\] ## What we run in production for KTO. TRL (KTOTrainer) PEFT \[ACCURACY · COST · TRADE\] ## The numbers we measure KTO on. Data shape Single response + label Loss asymmetry Bad weighted ~2x good Tunable When it earns the build When production feedback is thumbs, when you cannot run pairwise comparisons, when iterative shipping is the discipline. When it doesn't When you have clean pairwise data (use DPO or SimPO), when class balance is extreme (1:100+). \[ OUR TAKE \] ## Underrated. The right answer when feedback is thumbs. Most production teams have thumbs data and no pairwise data. KTO is the method that maps onto that reality. \[READ AT THE SOURCE\] ## Papers, docs, and primary sources. - [↗ Ethayarajh et al., KTO (2024)](https://arxiv.org/abs/2402.01306) - [↗ TRL KTOTrainer](https://huggingface.co/docs/trl/main/en/kto_trainer) \[COMMON QUESTIONS\] ## What buyers ask before they sign. KTO vs DPO when we have both kinds of data? If pairs are clean and abundant, DPO. If thumbs are abundant and pairs are sparse, KTO. If you have both, benchmark; KTO often wins on production data because the distribution matches deployment. \[RELATED FINE-TUNING TOPICS\] ## Worth a look next. [ 02 · FINE-TUNING ### Data pipeline Sourcing, PII redaction (Presidio), synthetic data (Distilabel, Nemotron), DEITA quality scoring, MinHash + SemDedup, labeling vendors, feedback loops. Read more](https://www.kensink.com/llm/fine-tuning/data-pipeline/)[ 03 · FINE-TUNING ### Platforms OpenAI RFT, Anthropic on Bedrock, Vertex, Azure Foundry, Databricks Mosaic, Together, Predibase, NeMo Customizer, Modal, Lambda. Side-by-side with our take. Read more](https://www.kensink.com/llm/fine-tuning/platforms/)[ 04 · FINE-TUNING ### By data + compute scale Under 1k examples to over 1M, single A10G to 128 B200. Indicative cost, recommended method, hardware tier. Read more](https://www.kensink.com/llm/fine-tuning/by-scale/)[ 05 · FINE-TUNING ### Custom model build Continued pretraining, SFT, preference optimization (DPO, SimPO, ORPO), reasoning distillation (R1 lineage), model merging (TIES, DARE). The full build pipeline. Read more](https://www.kensink.com/llm/fine-tuning/custom-models/)[ 06 · FINE-TUNING ### Compliance EU AI Act (Article 25 substantial-modification trap), GDPR, HIPAA, FedRAMP, Colorado AI Act, India DPDP, China GenAI Measures. Region-by-region for tuned LLMs. Read more](https://www.kensink.com/llm/fine-tuning/compliance/) FINE-TUNING · KENSINK LABS ## Considering KTO? Let's pressure-test it first. We benchmark the cheap method first, name the trade, and only deploy the expensive one when the numbers force it. Sized to your data, your evals, your residency. [Start a conversation →](https://www.kensink.com/contact) [All fine-tuning topics](https://www.kensink.com/llm/fine-tuning)