A ChatGPT Session
“The Transformer performs surprisingly well.” — Vaswani et al., Attention Is All You Need, 2017
What does “it” refer to in this sentence: “The trophy doesn’t fit in the suitcase because it is too big”? Like any English speaker, ChatGPT instantly knows it is the trophy. That wasn’t always true. No machine could work it out reliably until 2017, when eight researchers at Google published the paper that would finally crack it.
The paper was called “Attention Is All You Need.” Before it, models read a sentence one word at a time, carrying forward a single compressed memory of everything seen so far. That memory was a bottleneck, and detail from early words faded before the model reached the end of a long sentence. The mechanism it introduced, self-attention, did away with reading in sequence altogether. Every word looked directly at every other word, all at once, and the model learned for itself how much weight to place on each relationship.
Self-attention lets the model weigh “it” against both “trophy” and “suitcase” simultaneously, and lets the phrase “too big” pull that weight toward one or the other. Change “big” to “small” and the same mechanism flips the meaning. The model learned this from massive amounts of training data.
Self-attention was the core insight, but it created problems of its own. Attention alone has no sense of word order, so a positional encoding was added to track it. A single pass of attention could only capture one kind of relationship, so multiple heads ran in parallel, each free to specialise. Stacking dozens of these layers made training unstable, so residual connections and layer normalisation held it together. An encoder used all of this to read the input, and a decoder then wrote the output, one word at a time.
The results surprised even the people who built it. The Transformer beat every previous model on standard translation benchmarks. Even more remarkable, it did so using a fraction of the training time and compute of the current state of the art.
The architecture was published openly, and within three years it was put to a use far beyond machine translation. In 2020, OpenAI trained GPT-3, a model with 175 billion parameters, on a cluster of around 10,000 GPUs supplied by Microsoft, at an estimated cost of $4.6-10 million for a single run. It used the same self-attention mechanism from the original paper, applied at a scale nobody had attempted before, and it could write fluently and plausibly on almost any subject.
But fluent is not the same as useful. GPT-3 had only learned to predict the next likely word, and left alone it could invent facts with total confidence or drift into toxic language. Producing convincing text turned out to be the easy part. Getting it to do what a person actually wanted required human judgement, applied directly and repeatedly.
The fix was reinforcement learning from human feedback (RLHF), inspired by the same reinforcement learning that had taught machines to play backgammon and Go. OpenAI trained 40 labellers to rank and write example answers, producing tens of thousands of demonstrations and comparisons. A reward model learned their preferences, and GPT-3 was fine-tuned against it. The result was preferred over the raw model 78 per cent of the time.
This model reached developers through OpenAI’s API in January 2022. What turned it into a global phenomenon was not a better model. It was a simple chat interface, and the nerve to ship it as an admittedly imperfect “research preview” rather than wait for something more polished.
Machines had learned to read and, thanks to ChatGPT, everyone could talk to one. AI had finally come of age.
Further Reading