What is a Transformer Architecture?

Before transformers, sequence models like recurrent neural networks processed text one token at a time, passing along a compressed summary of everything seen so far, a bottleneck that made it hard to capture long-range relationships and slow to train because each step depended on the last. Self-attention instead lets the model compute, for every token, a weighted combination of every other token's representation in the sequence simultaneously, so a word at the end of a paragraph can directly attend to a relevant word at the beginning. Stacking many attention and feed-forward layers, combined with positional information about token order, produces the architecture underlying essentially every modern large language model. The practical significance is twofold: attention captures long-range dependencies far better than earlier architectures, and because tokens can be processed in parallel rather than strictly sequentially, transformers train efficiently on the massive datasets and hardware clusters that produced today's frontier models.