Quantization (Quantization)
An optimization technique that compresses model size by reducing parameter precision from 16-bit to 4-bit or similar, enabling inference with limited computational resources.
What is Quantization?
Quantization is an optimization technique that reduces the numerical precision of a model's weight parameters (e.g., 32-bit floating point → 4-bit integer) to compress model size and memory usage.
Intuitive Understanding
It is similar to how reducing a photo's image quality decreases its file size. While the amount of information per parameter decreases, the model's overall performance is maintained to a surprisingly high degree. Applying 4-bit quantization to a 70B parameter model shrinks VRAM consumption from approximately 140GB to around 35GB, making inference possible without expensive GPU clusters.
Types of Quantization
| Method | Characteristics |
|---|---|
| Post-Training Quantization (PTQ) | Quantizes an already-trained model as-is. Straightforward, but may result in significant accuracy degradation. |
| Quantization-Aware Training (QAT) | Trains with quantization in mind. More accurate than PTQ, but requires training costs. |
| GPTQ / AWQ / GGUF | Quantization formats optimized for LLMs. Widely adopted as distribution formats for local LLMs. |
QLoRA is a technique that combines quantization with LoRA, enabling fine-tuning in a 4-bit quantized state.
Practical Decision Criteria
Multiple research findings have reported that "quantizing a larger model" yields higher performance than "using a smaller model at full precision." When selecting a model for edge AI environments, finding the optimal solution involves exploring combinations of model size and quantization bit-width.
Articles covering this term
- Local LLM / SLM Deployment Comparison — AI Utilization Without Cloud API DependencyCompare open-weight models like GPT OSS, Phi-4, and Llama 4 Scout against cloud APIs on GPU requirements, task accuracy, and TCO. A local AI deployment guide for data sovereignty and cost optimization.
- Latency Budget Design for AI Agents — How to Control the Trade-off Between Thinking Time and Response TimeExplore how "thinking time" causes response delays in multi-step reasoning agents, and compare latency budget allocation strategies and implementation patterns based on task complexity.
- What is SLM Distillation? How to Create Small Specialized Models from Large LLMsLearn how SLM distillation uses large LLMs as teachers to build cost-effective, task-specific private models—covering the mechanism and implementation steps.
- What is Edge AI? How On-Device LLMs Work and How to Choose One for BusinessLearn the basics of edge AI & on-device LLMs, and how to design workflows requiring low latency, data privacy, or unstable connectivity—where cloud LLMs fall short.
Related Terms
Context Window
A context window refers to the maximum number of tokens an LLM can process at one time, indicating t
Fine-tuning
Fine-tuning refers to the process of providing additional training data to a pre-trained machine lea
Generative AI (Generative AI)
Generative AI is a collective term for AI models capable of autonomously generating content such as
LoRA
LoRA (Low-Rank Adaptation) is a technique that inserts low-rank delta matrices into the weight matri