The previous article explained what some of the terms used to describe these models mean, and how they are used across various laws. This article answers a different question: where did foundation models come from, and why did this shift happen when it did?
The answer is not a single invention. It is a sequence of three steps, and each one follows the same pattern. Over time, developers specified less of the system directly, while the model at its core learned more from data during training. At each stage, systems that had previously been built separately also came to share more of the same components (Bommasani et al. 2021). Understanding that sequence explains much of why these models, and the systems built on them, are difficult to govern.
|
Stage |
What stopped being written down |
What became shared |
|---|---|---|
|
Machine learning, from the 1990s |
The method |
The learning algorithm |
|
Deep learning, from around 2010 |
The features |
The architecture |
|
Foundation models, from 2018 |
The functionality |
The model itself |
Before machine learning: Rules were specified by people
For much of the history of AI, building a system meant writing down how to solve the problem. A programmer or a domain expert specified the logic, and the machine followed it. Here, "rules" means the instructions, conditions and decision logic that determine how a system should respond to different inputs or situations. This approach works where the rules can be stated, and becomes difficult where they cannot.
Machine learning: The method is learned
From the 1990s, this changed with advances in machine learning. Instead of specifying how a task should be done, developers supplied examples and used a learning algorithm to infer the method from the data (Bommasani et al. 2021).
The consequence was that the "how" was no longer written by a person. It emerged from the training process. This also made systems more alike, because a wide range of applications could be built with the same general-purpose learning algorithm rather than with bespoke logic for each one.
A bottleneck remained. For more complex tasks, such as recognising objects in images or interpreting text, experts still had to identify and encode the characteristics the model should use. This process, known as feature engineering, involved writing domain-specific code to turn raw data into information a machine-learning model could work with (Bommasani et al. 2021). The model learned how to make predictions, but people still decided which characteristics of the data it should rely on.
Deep learning: The features are learned
Around 2010, deep neural networks began to regain prominence. This was driven by larger datasets, more computing power (particularly the availability of GPUs) and a greater willingness among researchers to attempt it (Bommasani et al. 2021).
This changed an important part of how AI systems were built. Instead of experts specifying in advance which features a model should look for, deep neural networks could be trained directly on raw inputs, such as the pixels in an image, and the relevant features emerged during training.
The significance became clear in 2012, when a deep neural network later known as AlexNet achieved a major improvement in performance on ImageNet, a large benchmark dataset for image recognition (Krizhevsky et al. 2012). Systems became more alike again: instead of a separate feature-engineering pipeline for each application, the same architecture could serve many (Bommasani et al. 2021).
Foundation models: Functionality is learned
By the end of 2018, another shift was under way, beginning in natural language processing. It was driven by two developments: transfer learning and scale (Bommasani et al. 2021).
Transfer learning is the idea that a model can learn useful patterns from one task and apply that knowledge to another. In practice, a model is pretrained on a large dataset and then adapted for specific tasks or applications. This approach is what makes foundation models possible.
Scale made these models increasingly capable, and it required three developments arriving together (Bommasani et al. 2021):
- Better hardware. Processors became faster and gained more memory, allowing much larger models to be trained.
- The Transformer architecture. Introduced in 2017, it could exploit that hardware by training in parallel, supporting far more expressive models than before (Vaswani et al. 2017).
- More training data. This provided the material from which these models could learn broad patterns, before being adapted for many different applications.
Self-supervised learning: Learning from unlabelled data
Transfer learning through pretraining was not new. Researchers had already used labelled datasets to pretrain image models for years. The challenge was that creating those labels required human time and expertise. As datasets grew larger, the cost and effort of labelling became a practical constraint on how much data could be used for pretraining (Bommasani et al. 2021).
Self-supervised learning reduced that constraint. Instead of requiring people to label each example, the training task is generated automatically from the data itself. BERT, for instance, was trained by hiding some words in a sentence and asking the model to predict them from the surrounding text (Devlin et al. 2018). The text provides both the training question and the expected answer, without requiring a person to label each example.
This significantly expanded the amount of data that could be used for training. Once training became less dependent on human labelling of individual examples, the main constraints became the availability of suitable data and the computing resources needed to process it.
It is important, however, to distinguish removing annotation from removing human judgement. Self-supervised learning reduced the need for people to label individual training examples. It did not remove human decisions from the process. Judgement remained important in choices about which data to collect, include, filter or exclude, and in later stages of training that shape how a model behaves.
The scaling continued
The shift did not end in 2018. According to Epoch AI, the computing power used to train notable and frontier AI models grew by roughly four to five times per year between 2010 and May 2024. Epoch also estimates that about two-thirds of the improvement in language-model performance over the past decade came from increases in scale (Sevilla and Roldán 2024). Epoch expects this trend to continue in the near future, absent new bottlenecks. The same driver that produced foundation models therefore remains at work, which is why the resources needed to build one have kept rising. A later article in this series examines compute in detail, including what it costs and who controls it.
What this history means for policy
Three points follow from this history, each of which becomes important later in this series.
First, more of the system's behaviour moved from code into training. The learning process first determined the method, then the features, and eventually much of the functionality. A system built this way cannot be understood simply by reading its code, because much of its behaviour has been learned rather than explicitly programmed. This is one of the reasons that auditing, explaining and assuring these systems is difficult.
Second, the systems became increasingly similar. Learning algorithms converged, model architectures converged and, eventually, many applications came to depend on the same small set of foundation models. This concentration can make development more efficient. It also means that weaknesses or failures in a widely used model can be reproduced across many systems. The next article examines this issue in more detail.
Third, scale helps explain where the barriers to developing advanced models now sit. Increasing capability has required increasingly large amounts of computing power, data, infrastructure and capital. As these requirements have grown, development has become concentrated among organisations with access to substantial resources. When later articles ask which models are being used in Africa, and who has the capacity to build them, part of the answer lies in this history.
The next article examines the two properties this development produced, emergence and homogenisation, and why they make foundation models different from many of the AI models that came before.
References
Bommasani, Rishi, Drew A. Hudson, Ehsan Adeli, et al. (2021), On the Opportunities and Risks of Foundation Models, arXiv:2108.07258, Stanford, CA: Center for Research on Foundation Models, Stanford University, accessed 13 September 2026.
Devlin, Jacob, Ming-Wei Chang, Kenton Lee, et al. (2018), BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv:1810.04805, accessed 13 September 2026.
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton (2012), ImageNet Classification with Deep Convolutional Neural Networks, in: Advances in Neural Information Processing Systems 25, Red Hook, NY: Curran Associates, accessed 13 September 2026.
Sevilla, Jaime, and Edu Roldán (2024), Training Compute of Frontier AI Models Grows by 4–5x per Year, 28 May, Epoch AI, accessed 13 September 2026.
Vaswani, Ashish, Noam Shazeer, Niki Parmar, et al. (2017), Attention Is All You Need, arXiv:1706.03762, accessed 13 September 2026.
