For most of the past decade, convolutional neural networks were the unquestioned default for computer vision. ResNets, EfficientNets, and their relatives powered image classifiers, detectors, and segmentation systems with a reliability that made the architecture feel like a settled question.

Then, in late 2020, Dosovitskiy and colleagues published An Image Is Worth 16×16 Words, demonstrating that a nearly unmodified transformer — the architecture that had already reshaped NLP — could match or exceed CNN performance on ImageNet when trained at sufficient scale. The result was genuinely surprising, because transformers carry none of the spatial priors that CNNs were carefully designed around.

The five years since have produced a more nuanced picture. Vision transformers (ViTs) are not strictly better than CNNs, and CNNs are not obsolete. The right choice depends on your dataset size, compute budget, latency constraints, and task. This article walks through exactly why.

How Does a Vision Transformer Actually Work?

The core idea in ViT is elegant: treat an image as a sequence of fixed-size patches and feed that sequence to a standard transformer encoder.

Patch extraction and linear embedding. Given an image of shape H × W × C, the model first divides it into N non-overlapping patches of size P × P. For a 224 × 224 image with P = 16, this yields N = (224/16)² = 196 patches, each of shape 16 × 16 × 3 = 768 raw pixel values. Each patch is flattened and projected to a D-dimensional embedding vector through a single learned linear layer — effectively a learned dictionary that maps raw pixel blocks to a continuous representation space.

A learnable [CLS] token is prepended to the patch sequence (borrowing the convention from BERT), giving a sequence of length N + 1. Learned positional embeddings are added to each token so the model can use spatial position — without them, the transformer has no notion of where in the image each patch came from.

Transformer encoder. The embedded sequence is passed through L repeated blocks, each containing multi-head self-attention (MHSA) and a feed-forward network (FFN). In MHSA, every patch attends to every other patch. This is the key structural difference from CNNs: the receptive field is global from layer one. There is no hierarchy of local features that gradually expand into global context; the model can, in principle, relate a patch in the top-left corner directly to one in the bottom-right from the first attention layer.

After L blocks, the [CLS] token’s final representation is passed to a classification head (a simple MLP). For dense prediction tasks — segmentation, detection — different decoding strategies attach to the full patch token sequence rather than just the [CLS] token.

Understanding ViT is easier with a solid grounding in the underlying mechanism; if you haven’t already, the transformer architecture article covers self-attention, positional encoding, and encoder structure in depth.

What Inductive Biases Do CNNs Have That ViTs Lack?

This is the crux of the practical difference between the two architectures.

Locality. A convolutional filter operates on a small local region of the feature map — a 3 × 3 or 5 × 5 window. This reflects a real property of natural images: nearby pixels are more likely to be related than distant ones. The model does not have to learn this from scratch; it is built into the operation.

Translation equivariance. If a feature appears at position (x, y) in the input, the same filter will detect it at any shifted position (x + Δx, y + Δy), producing a spatially shifted activation. This means CNNs generalize across spatial locations by construction, with shared filter weights doing the work.

Hierarchical feature composition. Deep CNNs build representations progressively: edges and textures in early layers, parts in middle layers, objects in later layers. Downsampling (pooling or strided convolutions) creates spatial hierarchies that compress information in a way that mirrors multi-scale structure in visual scenes.

ViTs have none of these inductive biases by default. The self-attention operation treats every pair of patches as equally worth attending to. Positional information is injected through learned embeddings rather than baked into the computation. There is no inherent notion of locality, no translation equivariance, and no forced hierarchy.

This is simultaneously ViT’s weakness and its strength. The weakness: without these priors, the model must learn spatial relationships from data, which means it needs considerably more of it. The strength: without these priors, the model is not constrained by them. On very large datasets, ViTs can discover spatial structure that rigid convolutional architectures cannot represent.

Why Do Vision Transformers Need More Data — or Better Pretraining?

The data efficiency gap is well-documented. When trained from scratch on ImageNet-1k (~1.3M images), ViT-Base underperforms a comparably-sized ResNet. The gap closes substantially on ImageNet-21k (~14M images) and largely disappears when using JFT-300M (300M proprietary images). This scaling behavior is consistent with the theory: more data compensates for weaker inductive bias.

Several techniques bridge the gap at smaller scales:

Strong augmentation. DeiT (Data-efficient Image Transformers) showed that aggressive augmentation — Mixup, CutMix, RandAugment, repeated augmentation — combined with knowledge distillation from a CNN teacher brought ViT-Small to competitive ImageNet accuracy without large-scale pretraining. Augmentation effectively multiplies dataset size and forces the model to learn more invariant representations.

Self-supervised pretraining. Masked Autoencoder (MAE) pretraining, where roughly 75% of image patches are masked and the model reconstructs them, provides a powerful self-supervised signal. MAE-pretrained ViTs transfer well even from relatively modest fine-tuning datasets, partially because the reconstruction task forces the model to build rich spatial representations.

Supervised pretraining on large datasets. Using ImageNet-21k pretraining before fine-tuning is the most common practical strategy. It is significantly cheaper than full JFT-scale training while recovering most of the accuracy benefit.

CNNs, by contrast, often achieve strong performance with ImageNet-1k training alone, and smaller CNN variants (MobileNets, EfficientNet-B0) can be trained on domain-specific datasets of tens of thousands of images while remaining competitive. For practitioners working with limited labeled data and no access to large pretraining pipelines, CNNs frequently remain the more pragmatic choice.

How Do Compute and Latency Compare in Practice?

The theoretical complexity of self-attention is O(N²D) with respect to sequence length N and embedding dimension D, which grows quadratically with the number of patches. For a 224 × 224 image with P = 16, N = 196 — manageable. For higher resolutions (e.g., 512 × 512 with P = 16, N = 1024), the attention cost becomes substantially heavier.

CNNs scale more favorably with resolution in the pure arithmetic sense: convolution cost grows linearly with spatial dimensions rather than quadratically.

In practice, the comparison is complicated by hardware. Transformers are built from matrix multiplications that map extremely well to GPU and TPU tensor cores. Modern ViT implementations on recent GPUs often achieve throughput that theoretical complexity would not predict. Still, for latency-critical inference on edge hardware — microcontrollers, mobile chips, FPGAs — CNNs remain decisively faster. Operations like depthwise separable convolutions in MobileNet are highly optimized for these environments in a way that self-attention is not.

Throughput comparisons on server hardware are more architecture- and implementation-specific, and benchmarks should be run on the target hardware rather than taken from published tables.

What Are Hybrid Architectures, and Do They Help?

Recognizing that CNNs and transformers have complementary strengths, several hybrid designs have emerged.

Early convolutional stem. Many production ViT variants replace the initial patch embedding with a small convolutional stem — a few strided conv layers that produce patch-like tokens. This introduces local priors early in the pipeline, improves data efficiency, and tends to stabilize training. ConViT and CvT follow versions of this approach.

Interleaved conv and attention blocks. Architectures like CoAtNet (Convolution + Attention Networks) use convolutional layers in earlier stages and global attention in later stages, matching compute to the scale at which global context actually matters. Empirically, this design achieves strong accuracy-efficiency trade-offs.

Swin Transformer. Swin uses shifted window attention rather than global attention: each token attends only to patches within a local window, with windows shifting between layers to allow cross-window interaction. This restores locality and linear complexity while retaining the representational flexibility of attention. Swin Transformers have become a standard backbone for dense prediction tasks (detection, segmentation) where the quadratic cost of global attention at high resolution is prohibitive.

Hybrid approaches often outperform either pure architecture at moderate dataset sizes, and they are worth considering when neither pure ViT nor pure CNN fits the constraints cleanly.

When Should You Use a Vision Transformer vs a CNN?

There is no universal answer, but the following heuristics hold up well across practical settings.

Favor ViTs when: you have access to a large-scale pretrained checkpoint (ImageNet-21k or foundation model pretrained); your task benefits from long-range spatial relationships (e.g., whole-slide histopathology, satellite image analysis, document understanding); you are fine-tuning on a downstream task and have at least tens of thousands of labeled examples; you are building a multimodal system where sharing a transformer architecture across modalities simplifies the design.

Favor CNNs when: you are training from scratch on a limited dataset (under ~100k images without access to strong pretraining); latency on edge or mobile hardware is a hard constraint; your task has strong local structure (e.g., texture classification, low-resolution detection); you need a well-understood and heavily optimized deployment path on non-GPU hardware.

Consider hybrids when: you need dense prediction (detection, segmentation) at high resolution and cannot afford global attention; you want the data efficiency benefits of convolutional priors combined with the representational capacity of attention in later layers; you are building on Swin or ConvNeXt-style backbones that are already designed for this trade-off.

It is also worth noting that the gap between well-tuned CNNs and ViTs is smaller than popular narrative suggests. ConvNeXt, which modernized ResNet with training recipes and design choices from the ViT literature, matches or beats many ViT variants on standard benchmarks while retaining the deployment advantages of convolutions. The architecture choice matters, but so does the training pipeline.

The detection-transformer lineage that grew out of this same shift toward attention-based vision backbones is also the architectural foundation behind open-vocabulary detection models used as grounding infrastructure for AI agents — a different application of the same underlying representational shift discussed here.

Frequently Asked Questions

Do vision transformers need more data than CNNs?

Yes, generally. Without the locality and translation-equivariance priors built into convolutions, ViTs must learn spatial structure from data alone. This typically requires either large-scale pretraining (ImageNet-21k or larger) or strong augmentation strategies. CNNs often achieve competitive results with smaller labeled datasets trained from scratch, making them more practical when data is limited.

What is a patch embedding in a vision transformer?

A patch embedding is the process of dividing an image into fixed-size non-overlapping tiles — typically 16 × 16 pixels — flattening each tile into a vector, and projecting it to a fixed embedding dimension through a learned linear layer. These embedded patches, one per tile, become the input tokens to the transformer encoder. The embedding layer effectively replaces the first convolutional layer of a CNN.

Are vision transformers better than CNNs?

At large scale with strong pretraining, ViTs match or exceed CNNs on benchmark accuracy. At smaller scale or with limited data, well-tuned CNNs and hybrid architectures are often more accurate and more compute-efficient. Neither architecture strictly dominates — the right choice depends on dataset size, hardware, latency requirements, and whether a suitable pretrained ViT checkpoint is available for your domain.

Can you combine CNNs and transformers in a single model?

Yes, and this is common in practice. Hybrid architectures such as Swin Transformer (local window attention), CoAtNet, and CvT use convolutional layers for early feature extraction and attention in later stages. A convolutional stem before the patch embedding improves data efficiency and training stability in standard ViTs. These designs capture the local priors of CNNs where they matter most and the global modeling capacity of attention where spatial resolution is lower and context is more important.

What tasks are vision transformers particularly well-suited for?

ViTs excel where global context matters and pretraining data is abundant: large-scale image classification, multimodal vision-language tasks, medical imaging (whole-slide images), remote sensing, and document understanding. Dense prediction tasks like segmentation and detection are workable with windowed-attention variants (Swin) but require architectural adaptation. Tasks with strong local texture or edge cues, or those requiring deployment on constrained hardware, often still favor CNNs.