Regularization

Imagine we are learning to ride a bicycle. At the beginning, we may be very nervous, gripping the handlebars tightly, body stiff, trying to remember every detail of the movement. This state of over-focusing on details and trying to perfectly control every tiny movement is called in machine learningoverfitting。

Our model (like you as a beginner) is too complex, perfectly memorizing every sample in the training data, including noise and random fluctuations, which causes it to perform poorly when faced with new, unseen data (such as actually riding on the road), lacking generalization ability.

Regularizationis the core technology designed to solve this problem. Its core idea is:Cool down the model's learning enthusiasm, prevent it from getting hung up on details, and thereby improve its adaptability to new environments.In simple terms, regularization adds an extra penalty term to the objective function (loss function) during model training, thereby limiting the model's complexity and preventing it from over-relying on specific patterns in the training data.

This article will take you through an in-depth understanding of the principles of regularization, common methods, and their applications in engineering practice.

Basic Concepts: The Bias-Variance Tradeoff

Before diving into regularization, we need to understand the two core sources of error in machine learning models:BiasandVariance. This helps us understand what regularization is actually adjusting.

• Biasmeasures the model's ownsystematic error.High bias means the model is too simple, failing to even learn the basic patterns in the training data (underfitting).
• Variancemeasures the model's sensitivity torandom fluctuations in the training data.High variance means the model is too complex, treating noise in the training data as patterns to learn (overfitting).

Our goal is to find thebias-variance tradeoffoptimal point to minimize the total error. Regularization is a method that, byadding a bit of bias (making the model slightly simpler), significantly reduces variance, and thereby effectively improves the overall generalization performance of the model.

L1 and L2 Regularization

The most classic regularization method is to directly add a penalty term based on the model's weight parameters to the loss function. Depending on the calculation method of the penalty term, it is mainly divided into L1 and L2 regularization.

Changes in the Loss Function

Unregularized loss function (using Mean Squared Error, MSE, as an example):Loss = (1/n) * Σ(真实值 - 预测值)²

Loss function after adding the regularization term:Loss_正则化 = Loss + λ * Penalty(权重)

where:

• λ(lambda) is theregularization strength coefficient,a hyperparameter greater than 0. It controls the strength of the penalty.λThe larger λ is, the heavier the penalty on model complexity, and the simpler the model becomes.
• Penalty(权重)is the penalty term. L1 and L2 have different definitions.

L1 Regularization (Lasso Regression)

• Penalty Term: The sum of the absolute values of all weight parameters in the model.
• Formula： Penalty = Σ|w_i|, wherew_iis the i-th weight.
• Loss Function： Loss_L1 = Loss + λ * Σ|w_i|

Core Features and Effects：

• Feature Selection: L1 regularization tends to produce asparse weight matrix,, i.e., it will directly compress the weights of many unimportant features to00. This is equivalent to automatically completing feature selection, where the model only retains the most important features.
• Geometric Interpretation: Geometrically, its constraint region is a 'diamond' (a rhombus in two dimensions). The optimal solution point is more likely to touch a 'corner' of this diamond, and points at the corner mean some coordinates are 0.

Code Example：

Output:

Lasso 模型系数：
  特征 0: 16.6855
  特征 1: 54.0447
  特征 2: 5.0302
  特征 3: 63.5492
  特征 4: 93.4587
  特征 5: 70.5421
  特征 6: 86.9569
  特征 7: 10.2711
  特征 8: 3.0697
  特征 9: 70.7835

非零权重的特征数量： 10 / 10

L2 Regularization (Ridge Regression)

• Penalty Term: The sum of squares of all weight parameters in the model.
• Formula： Penalty = Σ(w_i)²
• Loss Function： Loss_L2 = Loss + λ * Σ(w_i)²

Core Features and Effects：

• Weight Decay: L2 regularization tends to make all weight parametersapproach 0, but usually do not equal 0.It shrinks all weights uniformly, preventing any single weight from becoming too large.
• Improving Ill-Conditioned Problems: For data with multicollinearity (high correlation) among features, ordinary linear regression may be unstable. L2 regularization can effectively improve this problem and make the solution more stable.
• Geometric Interpretation: Geometrically, its constraint region is a 'circle' (a circle in two dimensions). The optimal solution point is more likely to touch the 'edge' of this circle rather than a sharp corner.

Code Example：

L1 vs L2 Comparison Summary

Elastic Net

Elastic Net is a compromise between L1 and L2 regularization, containing penalty terms of both.Loss_ElasticNet = Loss + λ1 * Σ|w_i| + λ2 * Σ(w_i)²

It combines L1's feature selection capability with L2's stability, making it suitable for cases with very high feature dimensions and correlations among features.

Other Regularization Techniques

In addition to directly modifying the loss function, there are also methods that achieve regularization by changing the training process or model structure.

Dropout (for Neural Networks)

Dropout is an extremely effective regularization technique in neural networks. During thetraining process,it randomly causes a portion of neurons in the network to temporarily 'deactivate' (set their output to 0).

How It Works：

• In each training batch, with probabilityp(e.g., 0.5), randomly drop a portion of neurons.
• Forward propagation and backpropagation are performed only in the remaining neurons.
• During testing or prediction, all neurons are used, but the output of a neuron is multiplied by(1-p)to keep the expected value consistent.

Core Idea: To prevent complex co-adaptation between neurons, forcing the network to learn morerobustanddispersedfeature representations. This is like a team: it cannot always rely on a few core members; everyone needs to be able to work independently, so that even if someone is absent, the team can still function normally.

Code example (using TensorFlow/Keras)：

Early Stopping

Early stopping is a simple and efficient regularization strategy. It does not modify the loss function, but insteadmonitors the model's performance on the validation set.。

Steps：

• Split the data into training and validation sets.
• Train the model on the training set, and periodically evaluate performance on the validation set (e.g., after each training epoch).
• Once it is found that the performance (e.g., loss) on the validation set has not improved over multiple consecutive epochsor has even started to decline,stop training immediately.

Core idea:Stop training at the moment just before the model begins to overfit the training data (i.e., when the validation error starts to rise), thereby obtaining the model weights with the best generalization ability.

Practice Exercise: Comprehensive Comparison of Regularization Effects

Let's use a complete example to compare the effects of different regularization methods on a regression task.

Practice tasks：

• Run the above code and observe how the unregularized model fluctuates wildly to fit noise (overfitting), while the regularized model curves are smoother and closer to the true function.
• Try adjustingdegree(the polynomial degree) and each model'salpha(regularization strength) parameters, and observe their impact on the model fitting results.
• (Advanced) Split the data into training and test sets, compute the MSE of each model on the test set, and verify the improvement in generalization from regularization.

Summary and Engineering Recommendations

Regularization is an essential tool in a machine learning engineer's toolbox. To apply it effectively, keep the following points in mind:

• Understand the nature of the problem:First, use learning curves, validation set performance, etc., to determine whether the model faces an overfitting (high variance) problem.
• Start simple:Usually, you can first tryL2 regularization,because it is stable and easy to tune. If the feature dimensionality is extremely high and feature selection is needed, then considerL1orelastic net.。
• Tuning is key:The regularization strengthλ(oralpha) is a crucial hyperparameter. It must becross-validationto carefully select.
• Use in combination:In practice, regularization techniques are often used in combination. For example, when training deep neural networks,Dropout + L2 weight decay + early stoppingis an extremely common combination.
• Domain adaptation:For computer vision tasks, Dropout and Batch Normalization (which also has a certain regularization effect) are very effective. For sequence models (such as RNN, Transformer), Dropout and weight decay are commonly used.

The ultimate goal of regularization is to guide the model from "rote memorization" of training data toward "deep understanding" of the universal patterns behind the data, so as to make more reliable predictions in the real world. Master it, and you hold the key to improving model generalization.