If you’ve trained a language model in the past five years, you’ve used a transformer. If you’ve fine-tuned BERT, run inference through GPT-2, or implemented any variant of the encoder-decoder stack, you’ve operated the architecture that Vaswani et al. introduced in 2017. And yet the internal mechanics — what attention is actually computing, why positional encoding is necessary, how the encoder and decoder differ in their masking — are frequently glossed over in favor of high-level narrative.
This article is written for practitioners who want the mechanical picture. We’ll walk through each component of the transformer from first principles, keeping the math at a conceptual level while being precise enough to be useful. Code snippets are intentional omissions here; the goal is the mental model, not a PyTorch tutorial.
We assume familiarity with feedforward networks, backpropagation, and some exposure to sequence modeling. We do not assume prior transformer experience.
What Problem Did the Transformer Solve?
Before 2017, the dominant approach to sequence modeling was the recurrent neural network (RNN) — and its gated variants, the LSTM and GRU. These architectures process tokens one at a time in sequence order, maintaining a hidden state that is updated at each step. The problem is that hidden state is a fixed-size bottleneck. Everything the network knows about token 1 must be compressed into the same vector that carries information about tokens 50, 100, and 500 — and then that vector is fed into the prediction for token 501.
Two practical pathologies follow from this. First, gradient flow degrades across long sequences. Even LSTMs, designed specifically to mitigate vanishing gradients, struggle to propagate useful signal across hundreds of tokens. Second, and critically for modern training pipelines, RNNs are inherently sequential. You cannot compute the hidden state at step t without the hidden state at step t−1. This serialization means that, regardless of how much GPU memory you have, the forward pass cannot be parallelized across the time dimension. Training is slow.
The transformer Vaswani et al., 2017 eliminated both problems by replacing recurrence entirely with attention. Every position in the sequence attends directly to every other position in a single matrix operation. There is no hidden state propagated step by step. The architecture is trivially parallelized across both the sequence and the batch dimensions, which is why transformers scaled so readily when hardware caught up.
How Does Attention Work?
The core operation of a transformer is scaled dot-product attention. Despite the name, the mechanics are closer to a soft lookup table than anything biological.
Given an input sequence, the attention mechanism produces three derived representations for each token: a query (Q), a key (K), and a value (V). These are computed by multiplying the input embeddings by three separate learned weight matrices — W_Q, W_K, and W_V. The matrices are learned during training; the names are suggestive of their role, not prescriptive.
The attention score between token i and token j is the dot product of i’s query vector with j’s key vector:
score(i, j) = Q_i · K_j
This score is then divided by the square root of the key dimension (d_k), typically 64 in the original paper. This scaling prevents the dot products from growing large when d_k is large, which would push softmax outputs into regions with very small gradients. After scaling, a softmax is applied across all j positions, producing a probability distribution over the sequence. The output for token i is then a weighted sum of all value vectors:
Attention(Q, K, V) = softmax(QK^T / √d_k) · V
What this computes, intuitively: for each token, how much should this position “look at” every other position in the sequence, and then what information should it collect from those positions? The query asks a question; the keys are potential answers; the values are the information retrieved. The softmax determines how attention weight is distributed.
Because Q, K, and V are all derived from the same input sequence in the basic case, this is called self-attention. No external memory is required.
What Is Self-Attention, Specifically?
Self-attention is attention applied within a single sequence, rather than between a source and target sequence. In an encoder, every token attends to every other token in the input — bidirectionally. The word “bank” in “river bank” and “bank account” will have different attention distributions over surrounding tokens, allowing the model to contextualize its representation.
This is the mechanism that replaced the RNN’s sequential hidden state. Instead of passing information forward through time step by step, the transformer computes a new representation for each token that is an aggregation of all token representations in a single parallel operation. For a sequence of length n, the self-attention layer computes n × n attention weights, then uses those weights to mix the value vectors — all in matrix form, all parallelizable.
The cost of this approach is quadratic. The attention matrix is n × n, so both memory and compute scale as O(n²) with sequence length. For sequences of 512 or 2048 tokens, this is manageable. For very long sequences — document-level tasks, genomic data, time series with tens of thousands of steps — the quadratic cost is a genuine constraint. A substantial body of subsequent research (Longformer, BigBird, FlashAttention, linear attention approximations) addresses exactly this.
What Is Multi-Head Attention?
Running a single attention operation captures one type of relationship between positions. Multi-head attention runs h attention operations in parallel, each with its own learned Q, K, V projection matrices, then concatenates the results and projects them back to the model dimension.
In the original paper, the model dimension d_model is 512, and the number of heads is 8. Each head operates in a subspace of dimension 512/8 = 64. The heads are free to specialize: one head might learn syntactic dependencies, another coreference, another positional proximity. The concatenation and final projection then mix these representations.
Formally:
MultiHead(Q, K, V) = Concat(head_1, ..., head_h) · W_O
where head_i = Attention(Q·W_Q_i, K·W_K_i, V·W_V_i)
The key practical point is that multi-head attention does not increase computational cost proportionally — the per-head dimension is reduced, so the total compute is comparable to a single full-dimension attention. But the capacity to attend along multiple axes simultaneously is what makes the mechanism expressive enough to be the primary representational workhorse of modern deep learning.
Why Does Positional Encoding Matter?
Self-attention has no intrinsic notion of order. The attention computation between token i and token j is the same regardless of whether i comes before or after j in the sequence. Permute the tokens, and you get the same attention scores — which is catastrophically wrong for any task where sequence order matters (i.e., essentially all language tasks).
To inject positional information, the original transformer adds a positional encoding to each token embedding before it enters the attention layers. The original formulation uses fixed sinusoidal functions:
PE(pos, 2i) = sin(pos / 10000^(2i / d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model))
Each dimension of the positional encoding oscillates at a different frequency, so the encoding for any position is a unique vector. Importantly, because sinusoidal functions satisfy certain linearity properties, the model can generalize to sequence lengths not seen in training — a property that purely learned positional embeddings (used in GPT-2, for example) do not share.
In practice, most modern architectures have moved away from fixed sinusoidal encodings toward learned absolute position embeddings or, more recently, rotary position embeddings (RoPE) and ALiBi (Attention with Linear Biases). These later approaches encode relative rather than absolute position, which has proven more effective for long-context generalization. But the underlying problem they’re solving — making the model position-aware without breaking the attention mechanism — is the same one the original sinusoidal encoding addressed.
How Do the Encoder and Decoder Differ?
The original transformer was designed for sequence-to-sequence tasks (machine translation). It has two stacks: an encoder and a decoder.
The encoder processes the input sequence through N identical layers (6 in the original paper). Each layer has two sublayers: a multi-head self-attention mechanism (bidirectional — every token attends to every other), followed by a position-wise feedforward network. Residual connections wrap each sublayer, and layer normalization is applied. The encoder produces a sequence of contextualized representations that is then passed to the decoder.
The decoder also has N layers, but each decoder layer has three sublayers. The first is a masked multi-head self-attention layer. The masking is causal: when generating token t, the decoder can only attend to positions 0 through t−1. This prevents the model from “seeing the future” during training on the target sequence. The second sublayer is cross-attention (sometimes called encoder-decoder attention): the decoder’s queries attend to the encoder’s keys and values, pulling in relevant information from the source. The third sublayer is the same position-wise feedforward network as in the encoder.
The practical consequence of this split: encoder-only models (BERT, RoBERTa) are well-suited to tasks that require understanding a full sequence at once — classification, NER, question answering over a context. Decoder-only models (GPT series) are well-suited to generation tasks where left-to-right causal modeling is appropriate. Encoder-decoder models (T5, BART) are the natural choice for sequence-to-sequence tasks: translation, summarization, structured generation.
What the Transformer Does Not Do Well
Honest assessment of an architecture includes its failure modes.
Long sequences remain a constraint. Quadratic attention cost is not a solved problem. FlashAttention (Dao et al.) addresses the memory bandwidth bottleneck through tiling and recomputation, enabling attention on longer sequences without increasing asymptotic complexity — but the fundamental O(n²) compute still holds for dense attention. Sparse attention approximations reduce compute but involve tradeoffs in which interactions are modeled.
Transformers do not have memory in the recurrent sense. Each forward pass processes a fixed context window. Extending context requires either longer sequences (expensive) or external retrieval mechanisms. The architecture has no built-in mechanism to accumulate knowledge across multiple inference steps the way a stateful system would.
Training requires scale. Transformers outperform RNNs on most benchmarks, but they require significantly more data and compute to reach competitive performance. On small datasets, simpler architectures often match or exceed transformer performance. The sample efficiency of attention mechanisms is lower than their asymptotic capability suggests.
Positional encoding is not solved. Generalization to sequence lengths longer than those seen during training remains a research area. Architectures that perform well on 512-token sequences often degrade on 4096-token sequences unless specifically trained or adapted.
These limitations do not diminish the transformer’s practical dominance — they contextualize it. The architecture is not uniformly superior; it is particularly well-suited to tasks with large training sets, parallel compute, and moderate sequence lengths.
Frequently Asked Questions
Why did transformers replace RNNs?
RNNs process tokens sequentially, which prevents parallelization across the time dimension and creates gradient propagation difficulties over long sequences. Transformers compute attention across all positions simultaneously in a single matrix operation, enabling full GPU parallelism during training and more direct gradient flow between distant positions — making them faster to train and more effective at capturing long-range dependencies.
What is self-attention?
Self-attention is the mechanism by which each token in a sequence attends to all other tokens in the same sequence to build a contextualized representation. For each token, the model computes query, key, and value vectors, then derives attention weights via scaled dot-product between queries and keys. The output is a weighted sum of value vectors, capturing context from the full sequence in parallel.
What is multi-head attention?
Multi-head attention runs several attention operations simultaneously, each in a lower-dimensional subspace, using independently learned Q, K, and V projections. The per-head dimension is reduced so total compute stays comparable to single-head attention. The outputs are concatenated and projected back to the model dimension. This allows the model to attend along multiple relationship axes — syntactic, semantic, positional — within the same layer.
Do transformers need positional encoding?
Yes. Attention is permutation-equivariant: without positional information, the model cannot distinguish “the dog bit the man” from “the man bit the dog.” Positional encodings inject order information by adding position-dependent vectors to the token embeddings. The original paper used fixed sinusoidal encodings; modern architectures commonly use learned absolute embeddings or relative encoding schemes such as RoPE or ALiBi, which generalize better to sequence lengths not seen during training.
How many parameters does a transformer have?
Parameter count scales with model dimension, number of layers, and vocabulary size. The original base model from Vaswani et al. (2017) has approximately 65 million parameters. BERT-base is 110 million; GPT-2 ranges from 117 million to 1.5 billion depending on variant. The parameter count in the attention sublayers scales as O(d_model² × layers); the feedforward sublayers (typically 4× the model dimension) account for a significant fraction of the total.