Model Fine-tuning

When you need a model specifically for generating e-commerce product descriptions, there are three main options.

• The first is pre-training from scratch. This requires collecting massive text data (possibly trillions of tokens), renting dozens of A100 GPUs to train for months, and spending millions of dollars. This is what giants like OpenAI and Google do, and it is unrealistic for the vast majority of people.

• The second is Prompt engineering. Give a general large model a carefully designed prompt, such as "You are a professional e-commerce copywriter. Please generate an attractive description based on the following product information..." This approach has zero cost and is immediately usable, but the problems are: you have to write long prompts every time, the results are unstable, it is easily disturbed by irrelevant input, and token consumption is high.

• The third is fine-tuning. Take a trained general model, continue training it for a short while with your proprietary data, and let it "learn" your specific task. The resulting model retains general capabilities while performing excellently on your task, and the cost is controllable.

Comparison of the three approaches:

Use a simple analogy:

• Pre-training from scratch is like cultivating a doctor from zero by yourself, which takes more than a decade.
• Prompt engineering is like giving an ordinary doctor a detailed operation manual.
• Fine-tuning is like letting an already licensed doctor pursue specialist training for a few months and become a specialist expert.

Why Fine-tuning is Needed

Not all tasks need fine-tuning; first clarify when to use it and when not to.

Limitations of Prompt Engineering

Prompt engineering is powerful, but it has several ceilings that are hard to break through.

• The first limitation is the context window limit.Your prompts and examples must fit into the model's context window (e.g., 4K, 8K, 128K, 1024K tokens). If your task requires hundreds of examples to describe clearly, the prompt cannot contain them.

• The second limitation is unstable results.The same prompt may produce very different results when phrased differently. When the model is in a "good mood" (sampling randomness), the output is good; when not, it may go completely off-topic.

• The third limitation is high inference cost.Each inference requires sending the entire long prompt in, consuming many tokens and slowing response time. In high-concurrency scenarios, costs can skyrocket.

• The fourth limitation is the forgetting problem.Although the model has read your prompt, the massive amount of pretraining data before it may still be dragging things down. For example, if you ask it to output JSON, it may still add a lot of natural language explanations.

Applicable Scenarios for Fine-tuning

Fine-tuning is worth considering when your task matches the following characteristics.

• Characteristic 1: Clear task definition and fixed output format. For example, "translate technical documentation into an easy-to-understand blog," "generate ticket summaries from customer service records," "convert natural language into SQL queries."

• Characteristic 2: You have a few hundred to a few thousand high-quality labeled data samples. Too little data won't produce results from fine-tuning, and too much is unnecessary—generally 500-5,000 samples is the ideal range.

• Characteristic 3: You need professional performance in a specific domain. For example, medical report interpretation, legal document summarization, financial news analysis. General-purpose models may not understand industry terminology and conventions; fine-tuning can help them "enter the industry."

• Characteristic 4: You are sensitive to response speed and cost. A small fine-tuned model may outperform a general large model, with 10x faster inference speed and 100x lower cost.

• Characteristic 5: You need strict adherence to output format. For example, output must be JSON, must use a specific tone, must include certain fields. Fine-tuning lets the model "remember" these requirements without having to repeat them in the prompt every time.

Cost-Benefit Analysis of Fine-tuning

Before doing fine-tuning, do the math first.

Cost mainly comes from three aspects: data preparation, computing resources, and manual debugging. Data preparation usually takes the most work—you need to collect, clean, and label data. Computing resources are now very cheap; with LoRA + QLoRA, a consumer-grade GPU (RTX 3090/4090) can fine-tune 7B/13B models.

Benefits are reflected in several aspects: better results, lower inference costs, faster response times, and more stable output. If your model will provide services externally, these benefits will continue to accumulate.

When should you not fine-tune? If tasks change rapidly and data formats change every week, prompt engineering is more flexible. If you're only doing a one-time exploration, or you only have a few dozen samples, it's not worth fine-tuning either.

Full Fine-tuning

First understand the most traditional fine-tuning method, then look at why we rarely use it today.

Principles and Process

The idea of full fine-tuning is simple: take all the parameters of the pretrained model and continue training on your dataset, updating all the weights.

The process is roughly like this:

• First step: prepare the data. Organize your task data into "input-output" pairs.

• Second step, load the pre-trained model. For example, download open-source models such as LLaMA-2-7B and Mistral-7B.

• Step three, set the training parameters. The learning rate should be set very small (e.g., 1e-5 to 5e-5), because we don't want to completely overwrite the knowledge learned during pretraining.

• Step 4: Start training. Let the model run a few epochs on your data (complete passes through the dataset several times).

• Step five, save the model. After training, you get a brand new weight file, the same size as the original model.

Catastrophic Forgetting Problem

Full fine-tuning has a serious problem: catastrophic forgetting.

What does it mean? When you train on your data, the model gradually "forgets" the general knowledge it learned during pretraining. It's like a person who frantically does math problems to prepare for a math exam and ends up forgetting the Chinese and English they had learned before.

Why does this happen? Because full fine-tuning updates all parameters. The knowledge learned during pretraining is encoded in these parameters, and drastic modifications can destroy the original knowledge.

The result is: the fine-tuned model performs well on your task, but its performance on other general tasks declines. Even worse, it may lose some basic abilities, such as following instructions and understanding complex problems.

How to mitigate this? You can mix in pretraining data for joint training, but that brings us back to the old problem of "needing massive amounts of data." Alternatively, you can use regularization to constrain parameters from changing too much, but the effect is limited.

VRAM Requirement Calculation

Another problem with full fine-tuning is that the VRAM requirements are too large. Let's calculate:

VRAM required to train a model = model parameters + gradients + optimizer states + activations.

For a LLaMA-7B model, in full precision (FP32), 7 billion parameters × 4 bytes = 28 GB. Gradients also take 28 GB, and the optimizer states (Adam optimizer) require 56 GB (twice the gradients). Activations vary with batch size and sequence length, and need at least a few GB as well.

Adding it all up, full fine-tuning a 7B model requires about 100+ GB of VRAM — this means at least 8 A100 GPUs (40GB) or 2 A100 GPUs (80GB) are needed.

If half precision (FP16/BF16) is used, the memory requirement can be halved, but it still requires about 50-60 GB of VRAM. This is still not something an ordinary person can afford.

Full fine-tuning is not unusable, but it is costly and risky. Unless you have a compelling reason (e.g., a very large amount of data and a need to change the model's fundamental behavior), you should give priority to parameter-efficient fine-tuning (PEFT) methods.

Parameter-Efficient Fine-tuning (PEFT)

PEFT (Parameter-Efficient Fine-Tuning) is an umbrella term for a class of methods that train only a small number of parameters. It retains the effectiveness of fine-tuning while greatly reducing cost.

Core Idea of PEFT

The core insight of PEFT is:The pretrained model already contains sufficient knowledge. We don't need to modify all parameters; we only need to "adjust" a small portion of parameters to adapt to the new task.。

It's like a piano that has already been built. You don't need to rebuild the entire piano; you just need to fine-tune a few strings to produce the tone you want.

The benefits of PEFT are obvious:

• First, memory requirements are greatly reduced. It used to need 60 GB, now it might only need 6 GB.

• Second, no catastrophic forgetting. The original model parameters remain unchanged; we just add a little something new, so existing knowledge is not destroyed.

• Third, low storage cost. PEFT typically only saves a small number of newly added parameters (a few MB to a few hundred MB), without having to save the entire model.

• Fourth, it enables "task composition". Train multiple small adapters, load them on demand during inference, and one model can serve as multiple models.

Overview of Mainstream PEFT Methods

PEFT is not a single method, but a collective term for a class of methods. Let's look at several mainstream methods:

• The first is Adapter. Insert small neural network modules in the middle of each Transformer layer. During training, only these adapters are trained, and the original model parameters are frozen. This is an early PEFT method and is now used less.

• The second is Prefix Tuning. Add learnable "prefix" vectors before each layer of the model. These prefix vectors occupy only a small portion, but they can guide the model's output.

• The third is Prompt Tuning. Add learnable soft prompts only at the input layer. This is the simplest method, but its effect is relatively limited.

• The fourth is LoRA (Low-Rank Adaptation). Add low-rank matrices alongside the model's attention layers. This is currently the most popular and effective PEFT method.

• The fifth is QLoRA (Quantized LoRA). On top of LoRA, it adds model quantization to further reduce memory requirements. This is today's mainstream choice.

Today's best practice is clear:Prefer LoRA, and use QLoRA if memory is insufficient. These two methods are effective, low-cost, and have a mature ecosystem, making them the optimal choice in most cases.

LoRA（Low-Rank Adaptation）

LoRA is currently the most mainstream fine-tuning method, and understanding its principle is important.

Principle of Low-Rank Decomposition (Intuitive Understanding)

First, understand what "rank" is and what "low-rank" means.

Suppose you have a 100 × 100 matrix, and the data inside it may not be completely random. If this data can be composed of just a few "patterns", we say the matrix has a low rank.

For example: if every row is a multiple of the first row, then the rank of this matrix is 1. If every row is a linear combination of the first two rows, then the rank is 2. And so on.

The core insight of LoRA is:When a large model adapts to a new task, the change in weights is usually "low-rank" — that is, this change can be represented by very few parameters.。

This is like, although there are 1000 different e-commerce product descriptions, their writing styles may be determined by just a few core factors: tone, length, points of emphasis. You don't need 1000 different directions to describe the variation; 8, 16, or 32 is enough.

So LoRA's approach is: freeze all parameters of the original model, and add two small matrices A and B next to the attention layers. A is d × r, B is r × d, where r is the rank (Rank), usually a small number like 8, 16, or 32.

Original model output = original model forward pass.

LoRA output = original model forward pass + (input × A × B) × scaling factor.

During training, only A and B are trained; the original model parameters remain completely unchanged.

LoRA Mathematical Derivation

Don't be afraid, the mathematical derivation is simple, only a few lines.

Assume a certain layer of the original model is a linear transformation: h = W₀x, where W₀ is the pretrained weight matrix with shape d × d.

During fine-tuning, we want to update W₀, but not by too much. LoRA's approach is to represent the change in W₀ as the product of two small matrices: ΔW = BA.

So the output after fine-tuning is: h = W₀x + (BA)x.

Here B has shape d × r, A has shape r × d, and r is the rank (Rank).

More precisely, the formula in the LoRA paper is:

h = W₀x + α * (BA)x

where α is a scaling factor, usually set to some multiple of r, so that when r is changed there is no need to readjust hyperparameters.

During training, W₀ is completely frozen; only A and B are trainable.

At initialization, A is initialized with a random Gaussian distribution, and B is initialized to zero, so that at the start of training the LoRA part outputs zero and does not affect the original model's behavior.

See, LoRA only needs to train 0.4% of the parameters, yet it can achieve results close to full fine-tuning. That's the magic of it.

Choosing r (Rank) and α Parameters

LoRA has two key hyperparameters: the rank r and the scaling coefficient α.

r determines LoRA's capacity. The larger r is, the more complex changes it can represent, but the parameter count also increases.

Common r values: 8, 16, 32, 64. For most tasks, r = 8 or 16 is sufficient. Going beyond r = 64 usually yields diminishing returns and increases training cost.

How to choose r? A practical suggestion: try r = 8 first, and if results are poor, increase it to 16 or 32. Don't start with a very large r.

α is the scaling coefficient. A common setting is α = r, so that the scale of ΔW = BA remains roughly stable. Alternatively, set α = 1 and let the user adjust it themselves.

Other parameters to consider:

Target modules: Which layers should LoRA be applied to? Usually it works well to only add the query and value projections (q_proj, v_proj) of the attention layers. It can also be added to more layers, but that increases the parameter count.

Dropout: LoRA can add dropout to prevent overfitting; common values are 0.05 or 0.1.

Merging LoRA Weights

After training LoRA, there are two options at inference time.

• The first is to keep LoRA separate. At inference, load the original model, then load the LoRA weights, and dynamically compute the merged output. The advantage is that multiple LoRAs can be flexibly switched; the disadvantage is that inference is slightly slower (usually negligible).

• The second is to merge the LoRA weights. Add LoRA's ΔW = BA to the original model's W₀, yielding W = W₀ + α×BA. This produces a single model file, and inference works just like a normal model.

The advantage of merging is fast inference and no special LoRA support needed. The disadvantage is that once merged, it cannot be undone, and if there are multiple LoRAs, each must be merged into a separate model file.

QLoRA (Quantized LoRA)

QLoRA takes LoRA's low cost to the extreme—fine-tuning 70B models is possible on consumer-grade GPUs.

4-bit Quantization + LoRA

QLoRA's idea is simple: quantize the model to 4-bit, then perform LoRA fine-tuning on top.

What is quantization? It converts floating-point numbers (FP16/BF16, each taking 2 bytes) into lower-precision representations. 4-bit quantization uses 4 bits (0.5 bytes) to represent a weight.

This reduces model size by 75%: a 7B model drops from 13 GB to about 3.5 GB, a 13B model from 26 GB to about 6.5 GB, and a 70B model from 140 GB to about 35 GB.

But direct quantization loses precision. QLoRA's innovation is a data type called NF4 (Normalized Float 4), which better preserves the information in neural network weights.

QLoRA's approach:

• 1. Quantize the pretrained model to 4-bit (NF4 format).

• 2. Add LoRA adapters on top.

• 3. During training, the 4-bit quantized base model is completely frozen; only the LoRA FP16 weights are trained.

The result: VRAM requirements drop by another 75%, with almost no loss in performance.

bitsandbytes Library

bitsandbytes is the core library for quantization, developed by Tim Dettmers.

It provides:

• 1. 4-bit quantization (NF4).

• 2. Double Quantization—quantizes the quantization constants as well, saving a bit more VRAM.

• 3. Paged Optimizer—moves some data to CPU memory when VRAM is insufficient.

Ordinary users don't need to interact directly with bitsandbytes; the peft and transformers libraries already wrap it. But it's useful to know it exists.

Consumer-Grade GPU Fine-tuning Solutions

With QLoRA, ordinary people can afford to do fine-tuning too.

RTX 3090/4090 (24GB VRAM) can easily fine-tune 7B/13B models, and can even attempt 70B models (requiring longer sequence lengths or smaller batch sizes).

VRAM requirement estimate:

QLoRA is a game changer. It turned large model fine-tuning from "only big companies can do it" into "ordinary people can do it on their own gaming laptops." This is one of the most important advances in open-source AI.

Training Data Preparation

Data preparation is the most time-consuming and critical part of the fine-tuning process. Garbage in, garbage out—data quality determines the upper limit of the model.

Data Formats (Alpaca, ShareGPT Formats)

There are several common data formats; choose one and keep it consistent.

The first is the Alpaca format. It is one of the most popular formats, popularized by the Stanford Alpaca project.

The format is: instruction (instruction), input (input, optional), output (output).

The second is the ShareGPT format. This format is used for dialogue data and supports multi-turn conversations.

The format is: conversations (dialogue list), each turn has from (role) and value (content).

There is also a simple input-output pair format, suitable for simple tasks.

Which format is best? The Alpaca format is the most common; most training frameworks support it. For conversation tasks, ShareGPT is more suitable.

Key Points for Data Cleaning

Data cleaning is much more important than you might think. 80% of your model's performance depends on data quality.

• First, deduplication. Duplicate data will cause the model to overfit to those repeating patterns. Check and remove completely identical or highly similar data.

• Second, filter low-quality data. Remove data that is too short, too long, poorly formatted, or has obvious errors. For example, outputs with only a few characters, or with garbled text.

• Third, unify formats. For example, some outputs use Chinese quotation marks, some use English quotation marks; some end with a period, some don't. Try to unify them so the model learns faster.

• Fourth, balance the distribution. If some categories of data are particularly abundant and others are scarce, the model will be biased toward the majority category. Consider oversampling the minority class, or undersampling the majority class.

• Fifth, check for data leakage. Ensure that the test set and validation set do not appear in the training set.

Heuristic Rules for Data Volume

How much data do you need? This is one of the most frequently asked questions.

The answer is: it depends on task difficulty.

• Simple tasks (e.g., making a piece of text more conversational): 100-300 entries may be enough.

• Medium tasks (e.g., generating summaries in a specific format): 300-1000 entries is suitable.

• Complex tasks (e.g., Q&A in professional domains): 1000-5000 entries is ideal.

Of course, more data is usually better, but with diminishing marginal returns. 1000 entries is much better than 500, but 10000 may not be much better than 5000.

A practical suggestion: first prepare 500 high-quality data entries and run a trial. If the results are poor, analyze whether it's due to insufficient data, poor data quality, or hyperparameter issues, and then make targeted improvements.

Data quality >> data quantity. 1000 carefully crafted entries work much better than 10000 hastily assembled ones.

Spending more time in the data preparation phase is worthwhile. Model training is just a "press a button" matter, but data quality determines whether pressing that button is valuable.

Supervised Fine-tuning (SFT) in Practice

Finally, we've reached the hands-on section. We'll use Unsloth or LLaMA-Factory to implement the complete fine-tuning workflow.

Environment Configuration

Step 1: Install the necessary libraries.

Using Unsloth/LLaMA-Factory

Unsloth is an optimized training library that is fast and memory-efficient. LLaMA-Factory is an all-in-one training framework that supports multiple models and methods.

Let's first look at the code for training with Unsloth + peft:

If you want a simpler approach, you can use LLaMA-Factory, which provides both a Web UI and a command-line interface:

Training Hyperparameter Settings

Choosing hyperparameters is an art, but there are some empirical rules.

Learning Rate: This is the most important hyperparameter. For LoRA, 2e-5 to 5e-5 is a safe starting point. If training is unstable (loss oscillates), decrease it a bit. If training is too slow (loss decreases very slowly), increase it a bit.

Batch Size: The larger the better, but it is limited by VRAM. per_device_train_batch_size can be set to 2, 4, or 8. If VRAM is insufficient, use gradient_accumulation_steps to accumulate gradients. For example, batch_size=2, accumulation_steps=4 is equivalent to batch_size=8.

Epochs: 2-5 epochs is appropriate. Too few and the model won't learn enough; too many may cause overfitting. You can watch the eval loss, and stop when the eval loss stops decreasing.

LoRA Rank (r): 8-64 are all fine. 16 is a good starting point. Use 8 for small tasks, and 32 or 64 for complex tasks.

Max Sequence Length: Decide based on your data. 512, 1024, 2048, 4096 are all common. The longer it is, the more VRAM is required.

Training Monitoring and Loss Interpretation

During training, pay attention to a few metrics.

• Train Loss: the loss on the training set, should decrease steadily. If it decreases quickly, it means the learning rate may be too large or there is too little data. If it decreases slowly, it means the learning rate is too small or the model capacity is insufficient.

• Eval Loss: the loss on the validation set, this is the most important metric. It should decrease together with train loss. If eval loss starts to rise while train loss continues to decrease, it means overfitting, and training should be stopped.

Signs of overfitting: eval loss rises after first falling, while train loss keeps decreasing. Solutions: reduce epochs, increase dropout, add data, decrease LoRA rank.

Signs of underfitting: both train loss and eval loss are high, and they decrease slowly. Solutions: increase epochs, increase LoRA rank, add data, slightly increase the learning rate.

Evaluating Fine-tuning Effects

After training, how do you know if the model is good?

Selection of Evaluation Metrics

The evaluation method depends on your task type.

The first is manual evaluation. Have people read the model's output and score its quality. This is the most reliable method, but also the most time-consuming.

You can make a simple scoring table:

The second is automatic evaluation. Use metrics such as BLEU, ROUGE, and BLEURT. These metrics are not perfect, but they allow for rapid iteration.

The third is A/B testing. Mix the outputs of the old model and the new model together for blind evaluation to see whether the new model is really better.

Preventing Overfitting

Overfitting is the most common problem in fine-tuning. It shows good performance on the training set but poor performance on new data.

Methods to prevent overfitting:

1. Early stopping: Monitor the eval loss, and stop training when it stops decreasing.

2. Add more data: More and more diverse data is the best way to prevent overfitting.

3. Data augmentation: Rewrite existing data to create variation.

4. Reduce model capacity: Decrease the LoRA rank and reduce the number of LoRA layers.

5. Increase dropout: Set lora_dropout in the LoRA configuration to 0.05 or 0.1.

6. Weight decay: Add weight_decay=0.01 to the training parameters.

Deploying Fine-tuned Models

Once the model is trained, how do you use it?

Loading LoRA for Inference

The first way is to load the base model, then load the LoRA adapter.

Deployment Methods

There are several common deployment options.

• The first is to use vLLM. It is a high-performance inference engine that supports LoRA, continuous batching, and other features. Recommended for production environments.

• The second is to use Text Generation Inference. HuggingFace's official inference service, feature-rich.

• The third is to wrap it yourself with FastAPI. Simple and direct, suitable for small-scale deployment.

• The fourth is to use Ollama. Convert the model to GGUF format and run it with Ollama, suitable for local deployment.