What is SLM Distillation? How to Create Small Specialized Models from Large LLMs

SLM distillation is a technique for efficiently transferring the knowledge learned by a large language model (LLM) into a smaller language model (SLM). As one approach to knowledge distillation, it has evolved from a paper published by Hinton et al. in 2015, and in recent years it has been widely used to build task-specialized models based on SLMs of 7B parameters or fewer.

This article is written for engineers and architects considering private AI deployment in on-premises or edge AI environments. By the end, you will systematically understand the following three points.

Conclusion: SLM distillation is a technique for transferring the knowledge of a large LLM into a smaller model, and it is a core method for building private AI.

This approach applies the concept of knowledge distillation to LLMs, and it is grounded in the principles proposed by Hinton et al. in 2015.

At first, it is easy to assume that "knowledge transfer and knowledge distillation mean the same thing," but in reality distillation is one implementation of knowledge transfer, and the two differ clearly in purpose, procedure, and output.

The main differences between the two are as follows.

• Knowledge transfer: A broad term referring to the general concept of moving knowledge between people, organizations, and systems. It also includes documentation, training, and copying model weights
• Knowledge distillation: A concrete machine learning technique that trains a student model using the output distribution (soft labels) of a teacher model. It was formalized in the 2015 paper "Distilling the Knowledge in a Neural Network" (arXiv:1503.02531) by Geoffrey Hinton et al.

The heart of distillation lies in "soft labels." In standard supervised learning, hard labels are used that assign 1 to the correct class and 0 to all others. In distillation, by contrast, the entire probability distribution output by the LLM (e.g., "cat: 0.7, dog: 0.2, tiger: 0.1") is used directly as the learning signal, so implicit knowledge such as similarity relationships between classes is also conveyed to the student model.

This difference directly affects implementation decisions.

In knowledge distillation, the large LLM is called the "teacher model" and the lightweight SLM (Small Language Model) is called the "student model." Their roles are clearly separated: the teacher is the source of knowledge, and the student is the model that receives that knowledge and operates in a production environment.

The main roles of the teacher model are as follows.

• Generate "soft labels" (the probability distribution of each token) for a given input
• Provide rich information that includes not only correct/incorrect answers but also similarity relationships between classes
• Convey the "implicit knowledge" that the student should imitate, carried within the probability distribution

The main roles of the student model are as follows.

• Learn from the teacher's soft labels as the training signal and reproduce a similar output distribution
• Achieve accuracy close to the teacher on a specific task while keeping the parameter count low
• Run inference in resource-constrained environments such as on-premises or edge settings

An important axis for decision-making is the generality of the task. If you want to cover a broad range of domains, choose a large open-weight model as the teacher; if you are focusing on internal documents or specific business tasks, it is more efficient to train intensively on synthetic data generated from the teacher's output.

In training the student, a "distillation loss" is used that incorporates both the teacher's output probability distribution and the correct labels into the loss function.

Many engineers are puzzled: "SLMs and LLMs are both Transformer-based, so why do they behave so differently?" Understanding the differences between the two is the starting point for designing distillation.

The most fundamental difference is the parameter count. Whereas LLMs have tens to hundreds of billions of parameters, SLMs—as noted in research literature—are often the focus of studies targeting 7B (7 billion) parameters or fewer, so the model sizes differ by orders of magnitude.

The main architectural differences are as follows.

• Number of layers and heads: LLMs have large numbers of both Transformer blocks and attention heads, giving them high expressive power. SLMs significantly reduce these to lower the memory footprint
• Context window: LLMs have a wide context window so they can handle long contexts, but in SLMs it is often reduced, creating constraints on processing long conversation histories or documents
• Whether MoE (Mixture of Experts) is adopted: Some large models adopt an MoE structure to reduce inference cost, but SLMs predominantly use a Dense Model configuration

These constraints translate directly into the difficulty of distillation, because the rich intermediate representations held by the teacher model must be compressed and transferred into a student model that has fewer layers.

Conclusion: With the three requirements of cost, privacy, and edge deployment overlapping, interest in SLM distillation is rising rapidly.

Three needs are surfacing simultaneously: breaking away from dependence on cloud APIs, securing data sovereignty through on-premises operation, and deploying lightweight models to edge devices. Each H3 section explores the background and specific use context of each in detail.

As the use of generative AI spreads, the voices saying "we don't want to send our internal data to the cloud" remain deeply rooted in the field. In heavily regulated industries such as healthcare, finance, and manufacturing, passing personal information or trade secrets to external APIs can itself pose a compliance risk in many cases.

At first it is easy to think "using a cloud API and controlling it with prompts is sufficient," but in reality obligations to audit data transmission logs and contractual constraints become barriers, and an increasing number of organizations are choosing to bring everything fully in-house on premises or in a private cloud. SLM distillation is a powerful means of meeting this requirement.

The main challenges when running private AI on premises are as follows.

• Inference cost: Running a large model as-is on premises requires high-performance GPU servers, which tends to make procurement and maintenance costs high
• Latency: In real-time integration with internal systems, there are situations where round-trip delay to the cloud is unacceptable
• Data governance: Keeping logs and inference history under your own management makes it easier to respond to audits and to GDPR and industry regulations

By using SLM distillation, you can condense the knowledge of a large model into a small model of 7B parameters or fewer, making practical inference possible even in existing on-premises servers or environments with just one or two GPUs. The ability to maintain task-specialized accuracy while cutting cloud dependence is the essential reason this technique is attracting attention.

The benefits of SLM distillation appear most prominently in edge AI deployment scenarios.

The small model obtained through distillation can complete inference on-device without relying on cloud APIs. This yields three advantages at once: elimination of network latency, reduction of communication costs, and keeping data entirely on premises. It is especially effective for use cases that demand both low latency and privacy, such as anomaly detection on production lines or real-time diagnosis at medical sites.

From the perspective of GPU cost, the following branching conditions serve as decision criteria.

• When performing inference at high volume and continuously: Pay-as-you-go charges for cloud GPUs tend to accumulate, and running an SLM of 7B or fewer parameters on a small on-premises GPU (e.g., a T4-class card instead of an NVIDIA A100) tends to keep total costs lower.
• At the PoC stage with low inference frequency: It saves development cost to use the cloud API as-is and migrate to an SLM only just before going into production.

The training (distillation) phase itself can also compress the GPU time needed to train the student model by pre-generating the teacher model's inference as offline synthetic data. There are reports of cases where this can be handled with spot instances or burst usage without permanently securing a large GPU cluster.

Many teams struggle: "When distillation, fine-tuning, and quantization are all on the table, which should I tackle first?" Understanding the purpose and application phase of each makes the decision easier.

First, fine-tuning is a technique that updates the parameters of an existing model through additional training to adapt it to a specific domain. Since the model size itself does not change, it suits situations where "I want to improve accuracy but keep the scale the same." Quantization lowers the numerical precision of the weights (for example, from FP32 to INT8), reducing memory usage and computational cost at inference time. Its characteristic is that it can be made lightweight while preserving the model structure. And knowledge distillation is a technique that trains a smaller student model from scratch using the output distribution of a large teacher model as a guide; it differs in nature from the other two in that it can compress the very scale of the model.

To summarize, the basic guideline is: fine-tuning if the goal is domain adaptation, quantization if you want to make an existing model faster and lighter as-is, and knowledge distillation if you want to port the capabilities of a large model into a small one.

Moreover, these techniques are not mutually exclusive. Applying fine-tuning to further adapt a student model that has been made smaller through distillation, and finally using quantization to reduce inference cost, is an approach commonly taken in practice. If you want to optimize accuracy, speed, and size simultaneously, this stacking is worth considering.

Conclusion: The key to successful distillation lies in the preparation phase before training begins.

Three preparations—selecting the teacher model, managing the quality of synthetic data, and building the MLOps pipeline—determine the final quality of the student model.

Selecting the teacher model is the first decision that shapes the quality of the entire distillation process. At first it is easy to think "you should just use the highest-performing cloud API model as the teacher," but in reality, running an open-weight model in a local environment is often easier to balance across the three points of licensing, cost, and data sovereignty.

The main axes for selection are the following three points.

• Task fit: Confirm whether the teacher model has sufficient knowledge of the target domain (legal, medical, manufacturing, etc.). In specialized areas that a general-purpose model cannot handle, it is effective to choose a model that has been additionally trained on the same domain
• License compatibility: If you will use the generated soft labels or synthetic data commercially, confirm in advance whether the teacher model's license permits it. Even open-weight models may have restrictions on commercial use depending on the license
• Inference cost and output speed: Because generating soft labels requires processing a large number of samples, a model with low inference throughput becomes a bottleneck for the entire pipeline

A practical approach to leveraging open-weight models is to refer to a staged size lineup such as the LLaMA family (7B / 13B / 33B / 65B). Using a 13B-class model as the teacher and distilling into a student of 7B or fewer is a design that makes it easier to maintain quality while keeping GPU costs down.

Even if the teacher model is high quality, if the training data is poor, the effectiveness of distillation is greatly diminished. In practice, internal data alone is often insufficient in quantity, making the use of synthetic data essential.

Two main approaches are used to generate synthetic data.

• Generation by the teacher itself: A method of feeding the teacher model diverse prompts and having it output answers and soft labels simultaneously. By preparing domain-specialized question templates, you can efficiently collect data suited to your business
• Conversion of existing corpora: A method of passing internal manuals and FAQs to an LLM and converting them into question-answer pairs. It is important to confirm in advance the copyright and confidentiality of the source data

For quality control, you should avoid using generated data as-is. As concrete filtering criteria, it is recommended to establish the following perspectives.

• Consistency check: Verify that the teacher does not return contradictory answers to the same question
• Hallucination screening: Exclude rows containing claims that are difficult to fact-check
• Ensuring diversity: Reduce duplication of similar samples so that the student model does not learn a biased distribution

From the perspective of branching conditions, when the domain is narrow and internal data is sufficient, the conversion approach is low-cost and effective; but for new domains or when internal data is scarce, centering on teacher generation and designing diverse prompts tends to lead more readily to improved accuracy.

"Do I have to prepare the teacher model's inference environment and the student's training environment separately?"—in teams tackling distillation for the first time, questions like this tend to be the first barrier.

In designing an MLOps pipeline, you broadly build the following three layers.

• Data layer: Generation, filtering, and version control of synthetic data. Using a feature store or data version control tools such as DVC makes it easier to track quality changes
• Training layer: Separate, as a pipeline, the soft-label generation job by the teacher model and the training job for the student model. When GPU memory is limited, it is effective to run teacher inference as a preceding batch process and cache the labels
• Evaluation and deployment layer: Incorporate automatic computation of evaluation metrics, registration in a model registry, and deployment to on-premises or edge environments into the CI/CD flow

On the infrastructure side, NVIDIA A100-class GPUs are ideal, but by combining PEFT (parameter-efficient fine-tuning) and LoRA, there are cases where training is possible even on smaller GPUs. When on-premises operation is a premise, it is important to build in from the start a design that does not send out model artifacts and training data to an external cloud.

Conclusion: Implementation proceeds in three steps: "generate soft labels → train the student → evaluate and prepare." The order of each step and its quality control determine the accuracy of the final model.

Once preparation is complete, you move on to the concrete implementation phase. We explain in order the three steps: generating soft labels from the teacher model, training the student using PEFT and LoRA, and evaluation and model card preparation.

Generating soft labels is the most important step, one that determines the quality of the entire distillation.

At first it is easy to think "you can just use the teacher model's hard labels (the correct class only) as-is," but in reality, using soft labels (the probability distribution of each token) greatly improves the generalization performance of the student model. This is because hard labels discard all information other than the correct answer, whereas soft labels condense the semantic closeness and ambiguity between vocabulary items that the teacher model has learned.

Basic procedure for generating soft labels

• Set the temperature parameter (Temperature): Smooth the output probability distribution with temperature T. At T=1 the distribution stays as-is; setting T>1 puts information onto low-probability tokens as well, making it easier for the student to learn. Hinton et al.'s original paper (arXiv:1503.02531) also reports that increasing T improves transfer efficiency.
• Save the logits: At inference time, write out the logits of the final layer (the raw scores before softmax) to a file or feature store. By retaining them as a per-token probability distribution, you can compute the KL divergence loss in the subsequent training step.
• Estimate batch size and inference cost: When processing a large corpus, the teacher model's inference cost can balloon unexpectedly. Using synthetic data to narrow the inference targets to your business domain makes it easier to balance cost and quality.

Once you have prepared the soft labels, the next phase is training the student model. Because full-parameter retraining is expensive in GPU terms, in practice it is common to combine LoRA, a representative method of PEFT (parameter-efficient fine-tuning).

LoRA is an approach that adds low-rank delta matrices to the weight matrices, allowing the parameters to be updated to be narrowed to roughly a few percent of the whole. As a result, there are cases where even a 7B-class student model can complete training on a single consumer GPU.

As axes for deciding training settings, the following branching conditions are useful in practice.

• When domain-specialized data is abundant: Set the LoRA rank higher (e.g., rank=64) and increase the number of training epochs to deeply absorb specialized knowledge
• When the amount of data is limited: Keep the rank low (e.g., rank=8–16) and rigorously apply early stopping and monitoring of validation loss to prevent overfitting

The standard loss function configuration combines KL divergence against the soft labels with cross-entropy against the hard labels. The hyperparameters that adjust the ratio between the two (such as the temperature T of temperature scaling) are tuned while watching performance on the validation set.

During training, continuously monitor the following points.

The most common failure pattern is to skip the evaluation phase the moment you think, "distillation is done, all that's left is to deploy." The quality of the student model cannot be guaranteed without a comparative evaluation against the teacher model.

Key metrics for evaluation

• Task accuracy: Compare the teacher's and student's scores on benchmarks for the target business (QA, classification, summarization, etc.)
• Hallucination rate: In domains that require fact-checking, separately measure the error rate
• Latency and throughput: Measure actual inference speed in edge AI or on-premises environments and confirm whether it meets requirements
• Context window handling: Test whether accuracy degrades with long inputs

For evaluation data, it is fundamental to prepare a hold-out set that was not used in training. Because evaluating only with synthetic data makes it harder to see divergence from actual operation, it is recommended to include a certain number of samples close to real business tasks.

Preparing the model card (Model Card)

Once evaluation is complete, record the following in the model card.

• The name and version of the teacher model
• The provenance of the training data and the proportion of synthetic data
• Evaluation metrics and the values achieved
• Known limitations (unsupported domains, languages, etc.)
• License and usage restrictions

The model card also directly supports AI governance and internal audits.

Conclusion: Distillation failures boil down to three patterns: quality degradation, token mismatch, and reduced inference accuracy. Knowing the countermeasures in advance prevents rework.

As you proceed with implementation, distinctive problems such as increased hallucination and token distribution mismatch tend to surface. Each H3 section explains representative failure patterns and how to avoid them in concrete terms.

The first problem you tend to face with a student model after distillation is an increase in hallucination. There are reports of cases where the teacher model's "ability to answer while preserving ambiguity" is not sufficiently transferred during the distillation process.

At first it is easy to think "increasing the scale of the student model will solve it," but in reality improving data quality and strengthening the evaluation loop is often more effective. Rather than increasing model size, the approach of filtering data by the confidence scores of the soft labels generated by the teacher and excluding low-quality samples leads to suppressing quality degradation.

When the vocabularies of the teacher model and the student model do not match, the problem arises that the token probability distribution of the soft labels cannot be transferred as-is. Because BPE tokenizers (Byte-Pair Encoding Tokenizers) differ in vocabulary size and splitting rules from model to model, the same word is often converted into different token sequences.

Typical situations where this problem tends to occur are as follows.

• When the teacher uses a 32,000-vocabulary tokenizer and the student uses a 16,000 vocabulary, gaps arise in the probability mapping
• In Japanese and multilingual domains, a tokenizer trained mainly on English may split kanji and kana too finely, causing a single character to become multiple tokens
• When specialized terms (medical, legal, manufacturing, etc.) are treated as unknown tokens (UNK), the amount of information in the soft labels drops significantly

As an axis for deciding on countermeasures, if the teacher and student share a tokenizer from the same lineage you can use the probability distribution directly; but if they use different tokenizers, consider either re-alignment processing of the token sequences or switching to hard-label (correct-label) based distillation rather than soft labels.

Here is a summary of the implementation caveats.

Have you ever wondered, "Why does the answer quality of a post-distillation model suddenly drop when I hand it a long document?" This is a typical problem caused by a reduced context window.

There are cases where the teacher model has a context window spanning tens of thousands of tokens, but as the student model reduces its parameter count, the number of tokens it can handle also tends to become constrained. If, during training, soft labels were generated only with short input sequences, the student model cannot sufficiently learn long-range dependency reasoning patterns.

Specific scenarios where problems tend to occur are as follows.

• Cross-referencing multiple documents: Summarization and Q&A tasks that require applying context from the first half to answers in the second half
• Multi-step reasoning: Cases in problems involving CoT (chain of thought) where intermediate reasoning steps get truncated
• Searching long internal documents: Cases where, when combined with RAG, the chunk size exceeds the window limit

Effective approaches for avoidance are shown below.

• Intentionally include long input samples in the distillation dataset.

Q1. Should you distill or fine-tune first?

As a general rule, the recommended order is to distill first—creating a small model specialized for your business domain—and then layer on task-specific fine-tuning. By transferring the general knowledge of the teacher model through distillation and then fine-tuning with a small amount of high-quality business data, it becomes easier to balance accuracy and cost. However, when an existing open-weight model is sufficient as a base, there are cases where distillation is skipped and the need is met with fine-tuning alone.

---

Q2. How do you decide between a local LLM and SLM distillation?

A local LLM is a method of running an open-weight model as-is on premises or on your own servers, and its advantage is ease of deployment. SLM distillation, on the other hand, is an approach to building a custom model specialized for a specific business task, suited to situations where further optimization in terms of inference speed, cost, and privacy is required. It is practical to first evaluate existing models in Local LLM / SLM Deployment Comparison — Using AI Without Depending on Cloud APIs, and to proceed to distillation only when accuracy and speed do not meet requirements.

---

Q3. How much GPU resource does distillation require?

It varies greatly depending on the parameter scale of the student model. When targeting an SLM of 7B parameters or fewer, GPUs equivalent to the NVIDIA A100 are commonly used in the training phase. By combining PEFT methods such as LoRA or QLoRA, memory usage tends to be greatly reduced compared to full fine-tuning. In the inference phase, there are reports of cases where operation is possible even on a typical server GPU if quantization is applied.

---

Q4. Can distillation be accomplished with synthetic data alone?

Training is possible even with only the synthetic data generated by the teacher model, but if quality control is inadequate, there is a risk that hallucination increases. In practice, mixing in a small amount of real data or human-reviewed samples with the synthetic data is considered effective for stabilizing quality. It is important to record the generation conditions of the synthetic data (the temperature parameter, prompt design, etc.) and to state them explicitly in the model card to ensure reproducibility and transparency.

---

Q5. How do you maintain the quality of a distilled model over time?

After going into production, it is recommended to incorporate AI observability mechanisms and perform periodic monitoring of output quality. When the distribution of business data changes, a cycle of generating additional synthetic data with the teacher model and responding with incremental fine-tuning is effective. By building an evaluation and retraining flow into the MLOps pipeline, you can continuously prevent the model from becoming obsolete.

Many cases start by trying to begin with fine-tuning, but in reality distilling first is often more advantageous in terms of both final accuracy and efficiency.

The reason lies in the "starting point" of learning. Fine-tuning uses the knowledge held by the base model (foundation model) as-is, but when the student model is small to begin with, the base model alone tends to lack the density of knowledge needed for business tasks. By first having the student model learn the soft labels of the teacher LLM through distillation, the student model's expressive power is raised, and the subsequent fine-tuning functions more effectively.

The recommended order is as follows.

• Step 1 — Distillation: Generate soft labels from the teacher LLM and transfer general knowledge into the student model
• Step 2 — Fine-tuning (PEFT / LoRA): Perform additional training on domain-specific data to specialize for business tasks
• Step 3 — Quantization: Further compress the model size for edge AI or on-premises environments

However, when basing your work on an open-weight model that is already sufficiently large in scale, you may reach the target accuracy with fine-tuning alone, skipping distillation.

It is easier to organize the two by thinking along the axis of "use an off-the-shelf product, or build a custom one."

A local LLM is an approach that runs an open-weight model as-is on premises or in an edge environment. It can be used immediately without additional training and delivers sufficient performance for general-purpose question answering and summarization tasks. SLM distillation, on the other hand, transfers the knowledge and output style of a specific domain into the student model, so it incurs an initial build cost but yields task-specialized accuracy.

The branching points for the decision can be organized as follows.

• When immediate operation is needed for general-purpose tasks, adopt a local LLM as-is
• When high accuracy is required in a specific domain (legal, medical, manufacturing, etc.), build a custom model with SLM distillation
• For edge AI use cases where you want to cut inference cost and latency to the absolute minimum, a distilled small model tends to be advantageous

A staged approach is also effective: verify requirements with a local LLM at the PoC stage, and migrate to distillation if accuracy or response speed does not meet your standards. Starting with distillation from the outset risks spending too much effort on synthetic data generation and building the MLOps pipeline.

Note that the two methods are not mutually exclusive. A configuration in which you use a local LLM as the teacher model and derive a small model from it through distillation is also a practical option.