PEFT
PEFT (Parameter-Efficient Fine-Tuning) is a collective term for fine-tuning methods that adapt a large language model to a specific task with minimal computational resources and data, by updating only a subset of the model's parameters rather than all of them.
Attempting to fine-tune an LLM with tens of billions of parameters by training all parameters from scratch would take more than half a day even with 8 A100s running in parallel. In environments with limited budgets and hardware, this approach is often not even a viable option. PEFT breaks through this barrier by "freezing the majority of the model and training only a small number of added parameters."
The major techniques can be summarized as follows:
- LoRA (Low-Rank Adaptation) — Inserts low-rank deltas into weight matrices. The added parameters account for roughly 0.1–1% of the entire model. Currently the most widely adopted approach.
- QLoRA — Combines LoRA with 4-bit quantization to further compress GPU memory to less than half. Enables training of 7B models even on consumer GPUs.
- Prefix Tuning / Prompt Tuning — Adds trainable vectors on the input side. Makes no modifications to the model itself whatsoever.
- Adapter — Inserts bottleneck layers between Transformer layers. Was the dominant approach before LoRA emerged, but has somewhat faded from prominence today.
The author's team fine-tuned a 7B parameter LLM using LoRA on a single A100 in approximately 3 hours, improving task-specific accuracy by 15–20% over the base model. Full fine-tuning would have required A100×8 for 12 hours on the same task, making the cost difference stark.
That said, PEFT is not a silver bullet. It is difficult to use PEFT alone to acquire capabilities the model does not originally possess—such as generation in unsupported languages—and in such cases it becomes necessary to combine it with Continued Pre-training.
A common question is "which should I use, PEFT or RAG?"—but the two serve fundamentally different roles. RAG handles retrieval of external knowledge, while PEFT handles adjustment of the model's behavior and style. A practical starting point for choosing between them is: use RAG when you need accurate citation of internal knowledge, and use PEFT when you want to standardize the tone or format of responses. Combining both is not uncommon either.
Articles covering this term
- What is PEFT (Parameter-Efficient Fine-Tuning)? A Technology That Reduces AI Model Customization Costs by 90%Learn how PEFT (Parameter-Efficient Fine-Tuning) works—covering LoRA, QLoRA & Adapter methods—for decision-makers. Includes real-world GPU deployment cases & investment insights.
- Choosing Between Fine-Tuning and RAG: A Practical Guide Comparing Cost, Accuracy, and Use CasesFine-tuning vs RAG: which should you choose? We compare both across 4 axes—cost, accuracy, update frequency, and security—to guide your decision by use case.
- Introduction to Fine-Tuning — The Fundamentals and Decision Criteria B2B Companies Should Know Before Building Their Own LLMLearn how fine-tuning works from the basics. Covers PEFT/LoRA, when to use RAG, cost estimates, and a checklist for B2B companies evaluating custom model development.
- Local LLM / SLM Deployment Comparison — AI Utilization Without Cloud API DependencyCompare open-weight models like GPT OSS, Phi-4, and Llama 4 Scout against cloud APIs on GPU requirements, task accuracy, and TCO. A local AI deployment guide for data sovereignty and cost optimization.
Related Terms
Context Window
A context window refers to the maximum number of tokens an LLM can process at one time, indicating t
Fine-tuning
Fine-tuning refers to the process of providing additional training data to a pre-trained machine lea
Generative AI (Generative AI)
Generative AI is a collective term for AI models capable of autonomously generating content such as
LoRA
LoRA (Low-Rank Adaptation) is a technique that inserts low-rank delta matrices into the weight matri