Transformer Model

Transformer is a deep learning model based on the attention mechanism, first proposed by Vaswani et al. in their 2017 paper "Attention is All You Need".

It has revolutionized the field of natural language processing (NLP) and gradually expanded to almost all AI areas such as computer vision.

The core idea of Transformer is to completely abandon the traditional word-by-word processing method (RNN), and instead use the attention mechanism to let the model read the entire sentence at once while determining the relationship between each word and other words, thereby achieving faster training and stronger comprehension ability.

Why do we need Transformer?

Before the advent of Transformer, the NLP field mainly relied on RNN (Recurrent Neural Network) series models (such as LSTM, GRU), which process text sequentially and have two key shortcomings.

Limitations of RNN

RNN processes text like a human "reading word by word", which brings the following problems:

Vanishing gradient:When processing long text, the model "forgets" earlier information. For example, in "I borrowed a book about quantum physics from the library yesterday", by the time it reaches "book", it has long forgotten that "I" is the subject, making long-distance dependencies extremely difficult to capture.
No parallelism:RNN must process each word sequentially and cannot leverage the parallel computing capability of GPUs, making training on extremely long text very slow.

Transformer's solution

Through the self-attention mechanism, the model processes all words simultaneously and dynamically calculates the strength of association between each pair of words, completely solving the above two problems.

RNN vs Transformer comparison diagram

Comparison of RNN sequential processing vs Transformer parallel + global attention

Transformer Overall Architecture

Transformer consists of two main parts: an Encoder and a Decoder, each stacked from multiple layers of identical modules.

Analogy: The encoder is like a "reader" that understands an input Chinese sentence as a set of semantically rich vectors; the decoder is like a "translator" that references these vectors and generates English output word by word.

The following is the Transformer architecture diagram, with the encoder on the left and the decoder on the right.

Architecture Overview SVG

Complete architecture of Transformer encoder + decoder (orange arrows indicate Cross-Attention information flow)

The Transformer model consists of two parts, an Encoder and a Decoder, each composed of multiple stacked identical modules.

Encoder

The encoder is stacked from N identical modules, and each layer contains two sublayers:

Multi-Head Self-Attention mechanism:Computes the relevance of each word to other words in the input sequence.
Feed-Forward Neural Network:Performs an independent nonlinear transformation on each word.

Each sublayer is followed by a Residual Connection and Layer Normalization.

Decoder

The decoder is also stacked from N identical modules, and each layer contains three sublayers:

Masked Multi-Head Self-Attention mechanism:Computes the relevance of each word to preceding words in the output sequence (using masks to prevent future information leakage).
Encoder-Decoder Attention mechanism:Computes the relevance between the output sequence and the input sequence.
Feed-Forward Neural Network:Performs an independent nonlinear transformation on each word.

Similarly, each sublayer is followed by residual connection and layer normalization.

Before the emergence of the Transformer model, the mainstream models in NLP were RNN-based architectures, such as Long Short-Term Memory networks (LSTM) and Gated Recurrent Units (GRU). These models capture dependencies in sequences by processing input data sequentially, but they have the following problems:

Vanishing gradient problem: Long-distance dependencies are difficult to capture.
Limitation of sequential computation: It cannot fully utilize the parallel computing capability of modern hardware, resulting in low training efficiency.

Transformer solves these problems by introducing the self-attention mechanism, allowing the model to process the entire input sequence simultaneously and dynamically assign different weights to each position in the sequence.

Core: Self-Attention Mechanism

The self-attention mechanism is the most important component of Transformer. It answers the question: "When processing this word, which other words in the sentence should I focus on?"

What are Q, K, V?

The vector of each word is linearly transformed into three roles: Query, Key, and Value.

Q K V Diagram

The origin of Q, K, V and the attention computation process

Use a search engine analogy to understand Q/K/V:

Q (Query) = the keyword you enter in the search box, representing "What do I want to look up?"

K (Key) = the title tag of each webpage, representing "Which words can I match?"

V (Value) = the actual content of each web page, representing "What information can I provide?"

Attention weights = the similarity between Q and each K; the final result = a weighted sum of all V using the weights.

Attention Formula

\[ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V \]

where:

\(Q\)is the query matrix,\(K\)is the key matrix,\(V\)is the value matrix.
\(d_k\)is the dimension of the vectors, used to scale the dot product and prevent gradient explosion.
softmax converts the raw scores into a probability distribution between 0 and 1 (with weights summing to 1)

Multi-Head Attention

A single attention perspective is limited, just like looking at a problem from only one angle.

Multi-head attention splits the input into h subspaces; each "head" independently learns a different attention pattern, and the results are concatenated at the end.

Positional Encoding

Transformer processes all words simultaneously and inherently has no "sense of order" — "cat eats fish" and "fish eats cat" would be treated the same.

Positional encoding is like adding a "seat number" to each word, telling the model the position of each word in the sentence.

Analogy: just as you mark "Question 1, Question 2" on an exam paper, positional encoding lets Transformer know that "I" is the 1st word, "love" is the 2nd word.

Since Transformer has no explicit sequence information (such as time steps in RNN), positional encoding is used to add position information to each word in the input sequence. Typically, sine and cosine functions are used to generate positional encodings:

\[ PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) \] \[ PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) \]

where:

\(pos\) is the position of the word, and \(i\) is the dimension index.

Positional encoding visualization

Positional encoding is added to word embeddings to inject position information into the model.

Encoder-Decoder Architecture

The Transformer model consists of two parts: an encoder and a decoder:

Encoder:Converts the input sequence into a series of hidden representations. Each encoder layer contains a self-attention mechanism and a feed-forward neural network.
Decoder:Generates the target sequence based on the encoder's output. Each decoder layer contains two attention mechanisms (self-attention and encoder-decoder attention) and a feed-forward neural network.

Residual Connection and Layer Normalization

After each sublayer (self-attention, feed-forward network) outputs, two operations are performed: residual connection and layer normalization, which help stabilize training in deep networks.

Residual connection diagram

Residual connections allow gradients to "short-circuit" through, and layer normalization stabilizes training.

Residual connection:Adds the sublayer's input directly to its output (output = F(x) + x), avoiding vanishing gradients in deep networks and allowing the model to "selectively ignore" a layer's transformation.
Layer normalization:Normalizes the activations of each layer, making training more stable and convergence faster.

Advantages of Transformer

Compared with traditional RNN architectures, Transformer has the following notable advantages:

Parallel computation

Processes the entire sequence at once, fully utilizing GPU parallelism, with training speed far exceeding RNN.

Long-range dependencies

The self-attention mechanism keeps the "communication distance" between any two words at 1, no longer limited by sequence length.

Strong scalability

Stacking more layers and increasing dimensions can improve performance, giving rise to powerful models such as BERT and GPT.

Applications of Transformer

The Transformer architecture has been widely applied across various AI fields; the following introduces them by domain.

Natural Language Processing (NLP)

Machine translation (Google Translate) Text generation (GPT series) Text classification Question answering systems Sentiment analysis Summarization

Computer Vision (CV)

Image classification (Vision Transformer / ViT) Object detection Image generation

Multimodal Tasks

Image-text alignment (CLIP) Text-to-image generation (DALL-E / Stable Diffusion) Video understanding

Applications of Transformer

Natural Language Processing (NLP)：
- Machine translation (e.g., Google Translate)
- Text generation (e.g., GPT series models)
- Text classification, question answering, etc.
Computer Vision (CV)：
- Image classification (e.g., Vision Transformer)
- Object detection, image generation, etc.
Multimodal tasks：
- Tasks combining text and images (e.g., CLIP, DALL-E).

PyTorch Implementation Example

The following is a complete PyTorch Transformer example with detailed comments to help beginners understand each step:

Example

import torch
import torch.nn as nn
import torch.optim as optim

# --- Define the Transformer model ---

class TransformerModel(nn.Module):
def __init__(self, input_dim, model_dim, num_heads, num_layers, output_dim):
super(TransformerModel, self).__init__()

# Word embedding: maps word indices to model_dim-dimensional vectors
self.embedding = nn.Embedding(input_dim, model_dim)

# Positional encoding: learnable position vectors, max supporting length 1000
self.positional_encoding = nn.Parameter(
torch.zeros(1, 1000, model_dim)
)

# PyTorch built-in Transformer (includes encoder + decoder)
self.transformer = nn.Transformer(
d_model=model_dim, # Vector dimension
nhead=num_heads, # Number of heads for multi-head attention
num_encoder_layers=num_layers, # Number of encoder layers
num_decoder_layers=num_layers # Number of decoder layers
)

# Final linear layer: maps vectors back to vocabulary size (for predicting the next word)
self.fc = nn.Linear(model_dim, output_dim)

def forward(self, src, tgt):
src_seq_length = src.size(1)
tgt_seq_length = tgt.size(1)

# Word embedding + positional encoding (add the two together)
src = self.embedding(src) + self.positional_encoding[:, :src_seq_length, :]
tgt = self.embedding(tgt) + self.positional_encoding[:, :tgt_seq_length, :]

# Pass through Transformer (encoder reads src, decoder generates tgt)
transformer_output = self.transformer(src, tgt)

# Linear layer outputs vocabulary probabilities for each position
output = self.fc(transformer_output)
return output

# --- Hyperparameter settings ---

input_dim = 10000 # Vocabulary size (how many distinct words there are)
model_dim = 512 # Vector dimension for each word (the original paper uses 512)
num_heads = 8 # Number of multi-head attention heads (must divide model_dim)
num_layers = 6 # Number of encoder/decoder layers (the original paper uses 6)
output_dim = 10000 # Output dimension (same as vocabulary size)

# --- Initialize model, loss function, and optimizer ---

model = TransformerModel(input_dim, model_dim, num_heads, num_layers, output_dim)
criterion = nn.CrossEntropyLoss() # Multi-class cross-entropy loss
optimizer = optim.Adam(model.parameters(), lr=0.001) # Adam optimizer

# --- Construct example data (replace with real corpus in actual use) ---

# src: source sequence (e.g., Chinese), shape = (sequence_length=10, batch_size=32)
src = torch.randint(0, input_dim, (10, 32))
# tgt: target sequence (e.g., English), shape = (sequence_length=20, batch_size=32)
tgt = torch.randint(0, input_dim, (20, 32))

# --- Forward pass ---

output = model(src, tgt)
# output.shape = (20, 32, 10000): predicted distribution over vocabulary at each position

# --- Compute loss ---

# view(-1, output_dim) flattens (20,32,10000) into (640, 10000)
loss = criterion(output.view(-1, output_dim), tgt.view(-1))

# --- Backpropagation + update weights ---

optimizer.zero_grad() # Clear gradients from the previous step
loss.backward() # Compute gradients
optimizer.step() # Update parameters

print(f"loss value: {loss.item():.4f}")

The code above uses random data only to demonstrate the workflow. For actual training, you need to:

1) Prepare real parallel corpora (e.g., Chinese-English translation pairs);

2) Perform tokenization and vocabulary construction;

3) Implement Decoder step-by-step inference (autoregressive generation);

4) Adjust the learning rate scheduling strategy (the original paper uses Warmup).

Summary

The evolution from RNN to Transformer represents an important paradigm shift in deep learning.

Evolution path diagram

Evolutionary path from RNN to Transformer

Transformer's three major innovations completely transformed the deep learning landscape:

Self-attention mechanism—allowing any two words to directly "talk," bidding farewell to vanishing gradients.
Full parallelization—a qualitative leap in training speed, making large-scale pretraining possible.
Universal architecture—from NLP to CV and multimodal, Transformer has become the "universal building block" of the AI era.

Master Transformer, and you hold the key to understanding modern AI models like GPT, BERT, and Stable Diffusion.

Terminology Quick Reference

Term	English	Description
Self-attention	Self-Attention	A mechanism that computes correlations between elements within a sequence
Multi-head attention	Multi-Head Attention	Multiple attention heads compute in parallel to capture information from different feature subspaces
Positional encoding	Positional Encoding	A technique that provides the model with positional information of elements in a sequence
Residual connection	Residual Connection	Directly adds input to sublayer output to alleviate vanishing gradients
Layer normalization	Layer Normalization	Normalizes activation values of each layer to accelerate training convergence
Encoder	Encoder	Encodes the input sequence into contextual representations
Decoder	Decoder	Generates the target sequence based on the encoder output
Cross-attention	Cross-Attention	An attention mechanism where the decoder attends to the encoder output
Masked attention	Masked Attention	A mechanism in the decoder that prevents attending to future positional information
Feed-forward network	Feed-Forward Network	Fully connected layer applied independently to each position

Other extensions