Linear Equations and Matrix Rank
From the chickens-and-rabbits problem to AI model training, systems of linear equations are everywhere. The "rank" of a matrix tells us how much truly useful information is contained in the system.
Any system of linear equations can be written as Ax = b
A is the coefficient matrix (m equations, n unknowns), x is the unknown vector, and b is the constant vector.
For example, the chickens-and-rabbits problem: \( \begin{cases} x+y=10 \\ 2x+4y=28 \end{cases} \) written in matrix form:
\[ \begin{bmatrix} 1 & 1 \\ 2 & 4 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 10 \\ 28 \end{bmatrix} \]Rank: The Number of Independent Pieces of Information
Rank = the number of truly "independent" rows (or columns) in a matrix.
If an equation can be obtained by multiplying other equations by coefficients, it is redundant — it adds no new information.
The rank determines the solution cases of a system of equations:
Full rank
Unique solution
No redundancy among equations
Rank-deficient
Infinitely many solutions
There are redundant equations
Contradictory
No solution
The equations contradict each other
Real-life Example
Chickens-and-rabbits problem: full rank, unique solution
10 heads + 28 feet → 6 chickens, 4 rabbits. The two equations provide two independent pieces of information, exactly enough to solve for the two unknowns.
Redundant information: rank-deficient, infinitely many solutions
Equation 1: x + y = 5. Equation 2: 2x + 2y = 10 (which is simply equation 1 multiplied by 2).
The two equations are actually one piece of information — there are infinitely many pairs (x, y) that satisfy the condition.
Mathematical Definition
Determining the Rank
| Condition | Solution cases |
|---|---|
| rank(A) = rank([A|b]) = n | Unique solution |
| rank(A) = rank([A|b]) < n | Infinitely many solutions |
| rank(A) < rank([A|b]) | No solution |
Here [A|b] is the augmented matrix formed by appending b to the right of A.
Python Hands-on Practice
Example
# Chicken and rabbit in the same cage: full rank
A = np.array([[1,1],[2,4]])
b = np.array([10,28])
x = np.linalg.solve(A, b)
print("Solution:", x, "→ Chicken=6, Rabbit=4")
print("Rank:", np.linalg.matrix_rank(A)) # 2 (full rank)
# Redundant equation
A_red = np.array([[1,2],[2,4]]) # Row 2 = 2 × Row 1
print("\n"Rank of redundant matrix:", np.linalg.matrix_rank(A_red)) # 1
# Rank of a random large matrix
big = np.random.randn(100, 50)
print("Rank of 100×50 random matrix:", np.linalg.matrix_rank(big)) # 50
解: [6. 4.] → 鸡=6, 兔=4 秩: 2（满秩） 冗余矩阵的秩: 1 100×50 随机矩阵的秩: 50
Application Scenarios in AI
LoRA Fine-tuning = Low-rank Assumption
The core assumption of LoRA (Low-Rank Adaptation) is that the fine-tuning update \Delta W of pretrained weights is low-rank. Therefore, the full update \Delta W = AB can be approximated by the product of two small matrices A (d×r) and B (r×d), where r << d.
For example, for GPT-3's 12288-dimensional weight matrix, r = 8 or 16 is enough — the rank drops from 12288 to 8, reducing the number of parameters by more than a thousandfold. This is using the concept of "rank" for parameter-efficient fine-tuning.
Feature Collinearity Detection
If the rank of the data matrix is much smaller than the number of columns → there is high collinearity (multicollinearity) among features → X^TX in linear regression is non-invertible or near singular → regularization (Ridge/Lasso) or dimensionality reduction (PCA) is needed. In real ML projects, this manifests as an excessively large condition number of the feature matrix.
Matrix Completion in Recommendation Systems
The user-item rating matrix usually has only a few ratings (sparse), but it is assumed to be generated by a low-rank structure (similar users have similar tastes). Matrix Completion uses rank constraints to infer missing ratings from known ratings. This is the core idea of the winning algorithm in the Netflix Prize competition.
Other extensions