Deep Learning Basics

Deep learning is not some mysterious black technology; it is essentially a mathematical function fitter—give it large amounts of input-output data, and it automatically finds the complex relationships between them.

For example: you give it a million pictures of cats, with each picture corresponding to the label "cat". Deep learning automatically figures out what combinations of pixels look like a cat.

Another example: you give it a billion sentences of human language, with the next word of each sentence as the target. Deep learning automatically learns, given the preceding context, what the next word is most likely to be.

The core advantage of deep learning is that as data volume increases, its performance continues to improve.This is something traditional machine learning cannot do.

In this chapter, we will start from scratch and break down every component of deep learning step by step:

• Mathematical Foundations of Neural Networks
• The Computational Process of Forward Propagation
• The Role of Activation Functions
• Loss Function Design
• The Principle of Backpropagation
• Optimizer Selection
• Regularization Techniques
• Normalization Methods
• Learning Rate Scheduling

Finally, we implement a complete multi-layer perceptron (MLP) with PyTorch, connecting all the knowledge points together.

This chapter involves some mathematics, but don't worry—we will explain every formula intuitively rather than asking you to memorize it. The focus is on understanding why it is designed this way, not how to calculate.

Mathematical Foundations of Neural Networks

Deep learning is built on three branches of mathematics: linear algebra, calculus, and probability theory.

You don't need to be a math expert, but you do need to understand a few core concepts.

Linear Algebra: Matrix Multiplication

The most basic operation in neural networks is matrix multiplication.

Why use matrices? Because they can concisely represent the process of "multiple inputs being transformed by multiple neurons."

Let's first look at an intuitive example:

The geometric meaning of matrix multiplication is a "linear transformation"—it can rotate, scale, and stretch vector spaces.

But linear transformations alone are not enough.

If every layer is just matrix multiplication, then no matter how deep the network is, the entire network is equivalent to a single-layer network, because the composition of multiple linear transformations is still a linear transformation.

This is why we need activation functions—to introduce nonlinearity.

Calculus: Derivatives and the Chain Rule

The core of training neural networks is "gradient descent"—finding the direction of parameters that minimizes the loss.

The gradient is the "derivative of a multivariable function"—it tells us how much the loss changes when each parameter changes a little.

Let's look at a simple example:

For multivariable functions, we need to compute the partial derivative with respect to each variable, then combine them into a vector—this is the gradient.

The chain rule is the mathematical foundation of backpropagation. It allows us to "break down layer by layer" the derivatives of composite functions.

The core idea of the chain rule is:Break a complex function into simple functions, differentiate each, then multiply them together.。

Backpropagation is the application of the chain rule in neural networks—starting from the output, it propagates gradients backward layer by layer.

Probability Theory: Conditional Probability

In classification problems, neural networks often output a "probability distribution"—given an input, what is the probability of each class.

Conditional probability P(Y|X) represents "the probability of Y occurring given that X has occurred."

:.2%}

:.2%}

Core points of three mathematical foundations:

Forward Propagation

Forward propagation is the computation process where input data "flows through" the neural network, from the input layer to the hidden layer to the output layer.

Computation Graph

Representing a neural network as a computation graph makes the logic of forward propagation and backpropagation clearer.

A computation graph consists of nodes (operations) and edges (data flow).

Computation graph of a simple two-layer neural network:

The advantage of the computation graph is that every step is clear, and during backpropagation you just compute gradients in the reverse order.

Neural Networks from the Perspective of Matrix Multiplication

From the perspective of matrix multiplication, a neural network is a series of "linear transformations + nonlinear activations".

Each layer can be abstracted as:

aᵢ = activation(Wᵢ · aᵢ₋₁ + bᵢ)

where a₀ is the input x.

This formula is concise yet powerful—it can represent almost all neural network structures, from simple logistic regression to complex Transformers.

Note the dimension matching for matrix multiplication: if the input vector is n-dimensional and the next layer has m neurons, then the weight matrix W must have shape m×n. This way, after W·x, you get an m-dimensional vector.

The Role of Activation Functions

As mentioned earlier: without activation functions, a deep network is equivalent to a single-layer network.

Use a simple example to prove this:

Activation functions are the key component that makes deep networks meaningful—they introduce nonlinearity and allow the network to learn complex patterns.

Activation Functions in Detail

An activation function determines how a neuron's output responds to its input.

A good activation function should satisfy several conditions:

• Nonlinearity: this is a must
• Differentiability: so gradients can be computed
• Simple computation: both forward and backward propagation should be fast
• No saturation: gradients will not vanish or explode

Let's look at the most commonly used activation functions.

Sigmoid: Saturation and Vanishing Gradients

Sigmoid is one of the earliest activation functions. It compresses any real number to between (0, 1).

Formula: σ(x) = 1 / (1 + e⁻ˣ)

The problems with Sigmoid are obvious:

• When |x| > 6, the gradient is almost 0, causing vanishing gradients
• The output is not centered at 0, which affects optimization
• Computing the exponential is relatively slow

Due to these problems, Sigmoid is rarely used in hidden layers in modern deep learning.

ReLU and Variants

ReLU (Rectified Linear Unit) is the most commonly used activation function today.

Formula: ReLU(x) = max(0, x)

Simple but very effective.

Comparison of several common activation functions:

Selection Principles

Recommended choices for activation functions:

• Hidden layer: Prefer ReLU, simple and effective
• If you encounter dead ReLU: Try Leaky ReLU or GELU
• Transformer: GELU or SwiGLU is the standard choice
• Binary classification output layer: Sigmoid (outputs probability)
• Multi-class classification output layer: Softmax (outputs probability distribution)
• Regression output layer: No activation (outputs any real number)

Don't overthink the choice of activation function. Usually ReLU is good enough. When you confirm ReLU is the bottleneck, it's not too late to switch to others.

Loss Functions

The loss function measures the gap between the "model's prediction" and the "true answer".

The goal of training is to make this gap as small as possible.

Cross-Entropy Loss (Classification)

The most commonly used loss function for classification problems is Cross-Entropy Loss.

Its intuitive meaning is: "how different is the probability distribution predicted by the model from the true distribution."

A simplified version of cross-entropy loss: when labels are class indices instead of one-hot encoding, a more efficient computation method can be used:

loss = -log(probs[target_class])

This is why in PyTorch, CrossEntropyLoss directly accepts class indices as labels.

MSE Loss (Regression)

For regression problems (predicting continuous values), the most commonly used loss is Mean Squared Error (MSE).

Formula: MSE = mean((pred - target)²)

Next-Word Prediction Loss for Language Models

The training objective of large language models (e.g., GPT) is simple: given the preceding context, predict the next word.

This is essentially a multi-class classification problem—each word in the vocabulary is a class.

Summary of common loss functions:

Backpropagation

Backpropagation is the core algorithm for training neural networks.

Its goal is to compute the gradient of the loss function with respect to each parameter, and then use gradient descent to update the parameters.

Derivation via the Chain Rule

We will use a simple example to derive backpropagation step by step.

Although this example is simple, it contains all the core ideas of backpropagation:

1. beforetoward传播：ComputeAllmiddleVariable Forward propagation: compute all intermediate variables
2. 反toward传播: fromOutputstart，use链formula法ThenlayerlayerCompute梯degree Backpropagation: starting from the output, use the chain rule to compute gradients layer by layer
3. ParameterUpdate:use梯degreeBottom降UpdateParameter Parameter update: update parameters using gradient descent

Gradient Flow in Computation Graphs

formulti-layer神经Networking，梯degreeWhen computingGraphMedium反towardflow动。 For multi-layer neural networks, gradients flow backward through the computational graph.

everyonelayerof梯degreeDependencyinafteronelayer传comeof梯degree。 The gradient of each layer depends on the gradient passed from the next layer.

ThisExamplesshows反toward传播ofCompleteProcess. This example demonstrates the complete process of backpropagation.

The core rule is: The core rules are:

• forLine性layer y = Wx + b：dL/dW = (dL/dy)·xᵀ，dL/db = dL/dy For a linear layer y = Wx + b: dL/dW = (dL/dy)·xᵀ, dL/db = dL/dy
• foractivateFunction y = f(z)：dL/dz = dL/dy ⊙ f'(z)（⊙ Yes逐elementMultiplication） For an activation function y = f(z): dL/dz = dL/dy ⊙ f'(z) (⊙ is element-wise multiplication)
• 梯degreefromafter往before传，everyonelayeralluse链formula法Then"积累"梯degree Gradients propagate from back to front, and each layer uses the chain rule to "accumulate" gradients.

OKMessageYes：presentgenerationDeep LearningFramework(such as PyTorch、TensorFlow）willAutomaticisyouCompute反toward传播。you只requiresdefinitionbeforetoward传播，FrameworkKnow how to useAutomaticmicroDivide（autograd）搞decideAll梯degreeCalculate. The good news is: modern deep learning frameworks (such as PyTorch, TensorFlow) automatically compute backpropagation for you. You only need to define the forward pass, and the framework will handle all gradient computation with automatic differentiation (autograd).

Gradient Descent Optimizers

Compute出梯degreeAfter,I们requiresuseExcellenttransformdevicecomeUpdateparameters. After computing the gradients, we need an optimizer to update the parameters.

SimplestofExcellenttransformdeviceYesRandom梯degreeBottom降（SGD）, but还Yes很many更AdvancedofExcellenttransformdevice. The simplest optimizer is stochastic gradient descent (SGD), but there are many more advanced optimizers.

SGD: Stochastic Gradient Descent

Formula: θ = θ - η·∇L(θ) Formula: θ = θ - η·∇L(θ)

where η is the learning rate. where η is the learning rate.

SGD SimpleButYesDisadvantages: SGD is simple but has drawbacks:

• Uses the same learning rate for all parameters
• Convergence can be slow
• Easily gets stuck in local optima or saddle points.
• Sensitive to noise

Momentum: Acceleration with Momentum

Momentum 引入already"Speed"ofConcepts——Let梯degreeUpdate带Yes惯性。 Momentum introduces the concept of "velocity" — giving gradient updates inertia.

Formula: Formula:

v = β·v + (1-β)·∇L(θ)
θ = θ - η·v

β is usually set to 0.9. β is usually set to 0.9.

Adam: Adaptive Learning Rate

Adam (Adaptive Moment Estimation) is currently one of the most commonly used optimizers.

It combines the ideas of Momentum (first moment) and RMSProp (second moment) to maintain an adaptive learning rate for each parameter.

Formula:

m = β₁·m + (1-β₁)·∇L(θ)
v = β₂·v + (1-β₂)·(∇L(θ))²
m̂ = m / (1-β₁ᵗ)
v̂ = v / (1-β₂ᵗ)
θ = θ - η·m̂ / (√v̂ + ε)

Default hyperparameters: β₁=0.9, β₂=0.999, ε=1e-8, η=1e-3.

AdamW: Weight Decay Correction

AdamW is an improved version of Adam — it separates weight decay from the gradient.

This is useful for regularization.

Optimizer comparison:

Optimizer selection suggestion: Try Adam first, with the learning rate set to 1e-3 (3e-4 is a safer choice). If results are unsatisfactory, try tuning hyperparameters or switching to another optimizer.

Regularization Techniques

The goal of regularization is to prevent overfitting — helping the model perform well on unseen data.

L1/L2 Regularization

L1 and L2 regularization are the most basic regularization methods.

They add a penalty on parameters to the loss function:

L_total = L + λ·L_reg

L1 regularization (Lasso): L_reg = |w|

L2 regularization (Ridge): L_reg = w²

Dropout

Dropout is a simple but effective regularization technique: randomly "turning off" a portion of neurons during training.

This forces the model to learn more robust features — it cannot rely on any specific neuron.

Data Augmentation

Data augmentation is one of the most effective regularization methods—it creates more training samples by transforming existing data.

For images, common data augmentation techniques include: flipping, rotation, scaling, cropping, color jittering, etc.

For text, common data augmentation techniques include: synonym replacement, random insertion, random deletion, back-translation, etc.

Summary of regularization techniques:

Batch Normalization vs Layer Normalization

Normalization techniques can speed up training and make optimization more stable.

Internal Covariate Shift

When training deep networks, the input distribution of each layer changes as the parameters of the previous layer change. This is "internal covariate shift" (Internal Covariate Shift).

This slows down training because each layer has to constantly adapt to new distributions.

The goal of normalization techniques is to keep the input distribution of each layer stable.

Batch Normalization

Batch Normalization (BatchNorm) normalizes along the batch dimension.

Formula:

μ = 1/N · sum(xᵢ)
σ² = 1/N · sum((xᵢ - μ)²)
x̂ᵢ = (xᵢ - μ) / √(σ² + ε)
yᵢ = γ·x̂ᵢ + β

where γ and β are learnable parameters.

Advantages of batch normalization:

• Speeds up training
• Allows larger learning rates
• Reduces sensitivity to initialization
• Has a slight regularization effect

Disadvantages of batch normalization:

• Depends on batch size—works poorly with too-small batches
• Behavior differs between training and testing
• Not suitable for sequence models (e.g., RNN, Transformer)

Layer Normalization

Layer Normalization (LayerNorm) normalizes along the feature dimension instead of the batch dimension.

This makes it independent of batch size and very suitable for sequence models.

Comparison of the two normalization methods:

For Transformer and large language models, LayerNorm is the standard choice. It computes statistics independently at each position, does not depend on batch size, and does not need to maintain running statistics.

Learning Rate Scheduling

Learning rate is one of the most important hyperparameters — too large and it won't converge, too small and convergence is too slow.

Learning rate scheduling is dynamically adjusting the learning rate during training.

Warmup

In the early stage of training, use a small learning rate to "warm up" and let the model stabilize, then increase to the target learning rate.

This is especially important for Transformer.

Cosine Annealing

Cosine annealing gradually decreases the learning rate following the shape of the cosine function.

The Impact of Learning Rate on Training

The impact of learning rate:

• Too large: training unstable, may diverge
• Too small: slow convergence, may get stuck in local optima
• Appropriate: fast and stable convergence

A common strategy is the "learning rate finder": first train a few steps with learning rates increasing from small to large, and see at which learning rate the loss decreases fastest.

Summary of learning rate scheduling methods:

Hands-On: Implementing an MLP from Scratch with PyTorch

Now let's integrate all the previous knowledge points, use PyTorch to implement a complete multi-layer perceptron (MLP), and train it on a simple classification task.

This example contains all standard components of deep learning training:

• Data processing: Dataset, DataLoader
• Model definition: MLP with LayerNorm and Dropout
• Loss function: CrossEntropyLoss
• Optimizer: AdamW
• Learning rate scheduler: Warmup + Cosine
• Training loop: training phase + validation phase
• Model saving and inference

This workflow can be directly extended to more complex tasks and models.