Understanding the Transformer Architecture: Attention Is All You Need

Why Do We Need the Transformer?

Before the Transformer emerged, sequence modeling mainly relied on RNN / LSTM / GRU. These architectures had a fundamental problem: they could not be parallelized. Each step of computation depends on the hidden state from the previous step, causing training to be extremely slow.

In 2017, Google published "Attention Is All You Need", proposing an architecture entirely based on the Attention mechanism, which completely solved this problem.

Self-Attention Mechanism

The core idea of Self-Attention is simple: allow every token in the sequence to directly "see" all other tokens.

Computation process:

Generate three vectors for each input token: Query (Q), Key (K), Value (V)
Compute the dot product similarity between Q and all Ks → get Attention Score
Normalize with Softmax → get Attention Weight
Compute the weighted sum of all V → get output

Attention(Q, K, V) = softmax(QK^T / √d_k) V

Dividing by √d_k prevents the dot product from becoming too large and causing the Softmax gradient to vanish.

The four steps above are the description in words. The diagram below draws the same thing for a concrete four-token sentence — the n×n grid in the middle is the attention weight matrix, and you can step through how each cell is computed.

1/9Input: four tokens

The model does not see words, it sees tokens. Here, a four-token sentence.

Diagram: the full flow inside one Transformer layer. Left column is the attention computation; right column is the rest of the layer (residual, LayerNorm, feed-forward). The weights in the matrix are illustrative; the point is that every token scores against every token.

Multi-Head Attention

Rather than performing Attention once, perform it multiple times (multiple heads), with each head focusing on different features:

Some heads focus on syntactic relationships
Some heads focus on semantic associations
Some heads focus on positional proximity

The results from each head are concatenated and then passed through a linear transformation.

Positional Encoding

Because Attention itself does not care about position (it is "unordered"), positional information must be injected separately. The original paper uses sine/cosine functions:

PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

From Transformer to GPT

GPT is essentially a Transformer Decoder-only architecture:

It uses only Masked Self-Attention (only looks at left-side context)
It performs autoregressive generation through Next Token Prediction
After scaling up, powerful language abilities emerge

Practical Application Scenarios

Scenario 1: Machine Translation System Transformer was originally used for German-English translation. Suppose you want to translate "The cat sat on the mat" into German. Self-Attention lets each English word directly relate to its corresponding German word, even when the word order is completely different (German verbs are often placed at the end of the sentence). Each head focuses on a different aspect: grammatical structure, part of speech, and positional relationships.

Scenario 2: Code Generation (GPT Series) When you enter "Write a Python function to sort a list," GPT's Masked Self-Attention lets each generated token see all previously generated code. This is why LLMs can generate syntactically correct code: at each step, they "review" the complete context.

Hands-On Demo: Implementing Self-Attention in Python

import torch
import torch.nn.functional as F

def self_attention(Q, K, V, d_k):
    # Q, K, V shape: [batch_size, seq_len, d_model]
    scores = torch.matmul(Q, K.transpose(-2, -1)) / torch.sqrt(d_k)
    attention_weights = F.softmax(scores, dim=-1)
    output = torch.matmul(attention_weights, V)
    return output

# Test: 2 sentences, each with 5 tokens, model dimension 64
Q = torch.randn(2, 5, 64)
K = torch.randn(2, 5, 64)
V = torch.randn(2, 5, 64)
output = self_attention(Q, K, V, d_k=64)
print(output.shape)  # torch.Size([2, 5, 64])

Read these first