Singular Value Decomposition (SVD) -- Decomposing Any Matrix
Eigendecomposition only applies to square matrices, but SVD does not have this limitation—Any matrix can undergo SVD。
It is the mathematical engine behind technologies such as recommender systems, image compression, and LoRA fine-tuning.
SVD: Breaking a Matrix into Three Parts
(m×n)
(m×m)
Rotation
(m×n)
Scaling
(n×n)
Rotation
Geometric intuition: any linear transformation = rotation + scaling along coordinate axes + rotation again.
The diagonal elements of Σ are calledsingular values, arranged from largest to smallest, representing the "energy" of the matrix in each orthogonal direction.
Truncated SVD: Approximating with the First k Singular Values
\[ A \approx U_k \Sigma_k V_k^T \]The smaller k is = the higher the compression rate = the rougher the approximation. This is the core idea of dimensionality reduction and compression.
Life Example
Compress an Image
A 1000×1000 table (1 million numbers), after SVD decomposition, uses only the first 10 singular values to reconstruct.
You only need to store about 10×1000×2 = 20,000 numbers to get a decent approximation.
This is the power of SVD compression—discard unimportant "energy" and preserve the core structure.
Mathematical Definition
\[ A = U \Sigma V^T \]The columns of U are left singular vectors (orthogonal), the rows of V are right singular vectors (orthogonal), and the diagonal elements of Σ, σ₁ ≥ σ₂ ≥ ... ≥ σ_r > 0, are the singular values.
Python Hands-on Practice
Example
A = np.array([[1,0,0,0,2],[0,0,3,0,0],[0,0,0,0,0],[0,4,0,0,0]])
U, S, VT = np.linalg.svd(A)
print("Singular values:", np.round(S, 2)) # [4. 3. 2.24 0.]
print("Number of nonzero singular values = rank =", np.sum(S > 1e-10))
# Image compression demo: an image of rank 1
img = np.outer(np.array([1,2,3,2,1]), np.array([2,4,6,4,2]))
U_i, S_i, VT_i = np.linalg.svd(img)
print("\nSingular values of the image matrix:", np.round(S_i, 2))
# Perfectly reconstructed using only 1 singular value (because rank = 1)
k = 1
approx = U_i[:,:k] @ np.diag(S_i[:k]) @ VT_i[:k,:]
print("k=1 reconstruction error:", np.linalg.norm(img - approx))
奇异值: [4. 3. 2.24 0.] 非零奇异值个数 = 秩 = 3 图像矩阵的奇异值: [78.26 0. 0. 0. 0.] k=1重建误差: 7.11e-14 （几乎完美）
Application Scenarios in AI
Mathematical Foundation of LoRA Fine-tuning
LoRA assumption: \( \Delta W \) is low-rank—this is equivalent to assuming that in the SVD of \(\Delta W \), only the first r singular values are significant, and the rest can be ignored. The smaller r is, the fewer trainable parameters there are, but the lower the approximation accuracy. In practice, r=8 or 16 is often the best balance between effectiveness and efficiency, stemming from empirical observations of the decay rate of singular values of weight matrices.
Recommender Systems: Matrix Factorization Collaborative Filtering
Perform truncated SVD on the user-item rating matrix R (m×n): \( R \approx U_k \Sigma_k V_k^T \). Each row of U_k is a user's k-dimensional latent vector, and each row of V_k is an item's k-dimensional latent vector. The predicted rating of user u for item i = the dot product of u's latent vector and i's latent vector.
Model Compression and Distillation
After performing SVD on the weight matrix W of a fully connected layer, use truncated SVD to approximate: \( W \approx U_k \Sigma_k V_k^T \). The original m×n matrix has mn parameters, and after decomposition it becomes k(m+n+1). When k << min(m,n), the number of parameters is greatly reduced. This is very useful when deploying large models on mobile devices.
SVD Implementation of PCA
sklearn's PCA uses SVD rather than eigendecomposition at its core—it directly performs SVD on the centered data matrix, and the columns of the right singular vectors V are the principal component directions. SVD is numerically more stable and does not require explicitly computing the covariance matrix.
Other extensions