--- title: "DoRA fine-tuning: weight-decomposed LoRA" description: "DoRA decomposes weights into magnitude and direction. Up to +4.4% over LoRA at the same trainable parameter count. Drop-in replacement in PEFT and Unsloth." source: "https://www.kensink.com/llm/fine-tuning/methods/dora/" canonical: "https://www.kensink.com/llm/fine-tuning/methods/dora/" --- [← All methods](https://www.kensink.com/llm/fine-tuning/methods/) · 06 · WEIGHT-DECOMPOSED LORA · SPECIALISED ★ DoRA Specialised METHOD · WEIGHT-DECOMPOSED LORA # DoRA. LoRA with magnitude and direction split. Decomposes the weight matrix into magnitude (a scalar per column) and direction (a unit vector). LoRA modulates only the direction, the magnitude is trained separately. Reports +1 to +4.4% over LoRA on commonsense benchmarks (LLaMA-7B/13B, LLaMA3-8B). Our default replacement for plain LoRA in 2026. PyTorch PEFT TRL Unsloth [Talk to our team →](https://www.kensink.com/contact) [Fine-tuning hub](https://www.kensink.com/llm/fine-tuning/) Gain vs LoRA +1 to +4.4 points Cost Same as LoRA Code change One flag in PEFT (use\_dora=True) When Drop-in for LoRA \[WHY THIS EXISTS\] ## LoRA conflates magnitude and direction. A weight update has two parts: how much (magnitude) and which way (direction). LoRA's low-rank decomposition mixes them. DoRA decomposes the pretrained weight first, then trains magnitude and direction separately. The intuition matches what full SFT does naturally and what LoRA approximates with friction. - Weight = magnitude \* direction (per output column) - LoRA on direction, scalar on magnitude - Same trainable param budget, better accuracy - Toggle on existing LoRA configs (use\_dora=True in PEFT) \[THE PIPELINE\] ## DoRA, the same as LoRA with one extra config. Use the same data, the same hyperparameters, the same eval. Set use\_dora=True and the trainer handles the decomposition. Same data as LoRA PEFT config use\_dora=True Train Eval Ship adapter 01 ### Reuse the LoRA config, add use\_dora Rank 16, alpha 32, all-linear, use\_dora=True. PEFT 0.10+ supports DoRA natively. No data or LR change needed. 02 ### Train, eval, ship Same pipeline as LoRA. Adapter file is the same size. vLLM and LoRAX serve it the same way. \[THE STACK WE'D DEPLOY\] ## What we run in production for DoRA. PEFT (use\_dora=True) TRL Unsloth (DoRA support) vLLM (multi-LoRA) \[ACCURACY · COST · TRADE\] ## The numbers we measure DoRA on. LLaMA-7B commonsense +3.7% vs LoRA Per Liu et al., ICML 2024 LLaMA-13B commonsense +1.0% vs LoRA LLaMA3-8B (multi-task) +4.4% vs LoRA Trainable params Same as LoRA When it earns the build Anywhere you would use LoRA. The gain is a flag flip. When it doesn't When the framework does not support it (older PEFT, custom training stacks). \[ OUR TAKE \] ## Our 2026 default LoRA variant. We turn on use\_dora unless the runtime does not support it. The accuracy gain is consistent and the cost is zero. \[READ AT THE SOURCE\] ## Papers, docs, and primary sources. - [↗ Liu et al., DoRA (ICML 2024)](https://arxiv.org/abs/2402.09353) - [↗ PEFT DoRA docs](https://huggingface.co/docs/peft/main/en/developer_guides/lora#weight-decomposed-low-rank-adaptation-dora) \[COMMON QUESTIONS\] ## What buyers ask before they sign. Why is DoRA not just always the default in libraries? It is becoming so. PEFT, Unsloth, and Axolotl all support it. Some custom training stacks have not caught up. If your stack supports it, use it. Does DoRA add inference cost? Negligible. The magnitude scalar fuses into the adapter at serving time. \[RELATED FINE-TUNING TOPICS\] ## Worth a look next. [ 02 · FINE-TUNING ### Data pipeline Sourcing, PII redaction (Presidio), synthetic data (Distilabel, Nemotron), DEITA quality scoring, MinHash + SemDedup, labeling vendors, feedback loops. Read more](https://www.kensink.com/llm/fine-tuning/data-pipeline/)[ 03 · FINE-TUNING ### Platforms OpenAI RFT, Anthropic on Bedrock, Vertex, Azure Foundry, Databricks Mosaic, Together, Predibase, NeMo Customizer, Modal, Lambda. Side-by-side with our take. Read more](https://www.kensink.com/llm/fine-tuning/platforms/)[ 04 · FINE-TUNING ### By data + compute scale Under 1k examples to over 1M, single A10G to 128 B200. Indicative cost, recommended method, hardware tier. Read more](https://www.kensink.com/llm/fine-tuning/by-scale/)[ 05 · FINE-TUNING ### Custom model build Continued pretraining, SFT, preference optimization (DPO, SimPO, ORPO), reasoning distillation (R1 lineage), model merging (TIES, DARE). The full build pipeline. Read more](https://www.kensink.com/llm/fine-tuning/custom-models/)[ 06 · FINE-TUNING ### Compliance EU AI Act (Article 25 substantial-modification trap), GDPR, HIPAA, FedRAMP, Colorado AI Act, India DPDP, China GenAI Measures. Region-by-region for tuned LLMs. Read more](https://www.kensink.com/llm/fine-tuning/compliance/) FINE-TUNING · KENSINK LABS ## Considering DoRA? Let's pressure-test it first. We benchmark the cheap method first, name the trade, and only deploy the expensive one when the numbers force it. Sized to your data, your evals, your residency. [Start a conversation →](https://www.kensink.com/contact) [All fine-tuning topics](https://www.kensink.com/llm/fine-tuning)