How Did We Get Here? From Machine Learning to Foundation Models

The previous article set out what some of the terms used when referring to models mean, and how they are used across various laws. This article answers a different question of where did foundation models come from, and why did this shift happen when it did?

The answer is not a single invention. It is a sequence of three steps, and each one follows the same pattern. Over time, developers specified less of the system directly, while more of it was learned from data during training. And at each stage, systems that had previously been built separately came to share more of the same components.[1] Understanding that sequence explains a good deal about why these systems are difficult to govern.

Before: Machine learning rules were specified by people

For much of the history of AI, building a system meant writing down how to solve the problem. A programmer or a domain expert specified the logic, and the machine followed it. This works where the rules can be stated, and becomes difficult where they cannot. Here, "rules" refers to the instructions, conditions and decision logic that determine how a system should respond to different inputs or situations.

Machine learning: The method is learned

From the 1990s this changed with advances in machine learning. Instead of specifying how a task should be done, developers supplied examples and used a learning algorithm to infer the method from the data.[1]

The consequence was that the "how" was no longer written by a person. It emerged from the training process. This also made systems more alike: a wide range of applications could be built with the same general-purpose learning algorithm, rather than with bespoke logic for each one.[1]

A bottleneck remained. For more complex tasks, such as recognising objects in images or interpreting text, experts still had to identify and encode the characteristics the system should use. This process, known as feature engineering, involved writing domain-specific code to turn raw data into information a machine-learning system could work with.[1] The system learned how to make predictions, but people still decided which characteristics of the data it should rely on.

Deep Learning: The features are learned

Around 2010, deep neural networks began to regain prominence, driven by larger datasets, more computing power — particularly the availability of GPUs — and a greater willingness to attempt it.[1]

This changed an important part of how AI systems were built. Instead of experts specifying in advance which features a system should look for, deep neural networks could be trained directly on raw inputs, such as the pixels in an image, and the relevant features emerged during training.[1]

The significance became clear in 2012, when a deep neural network called AlexNet achieved a major improvement in performance on ImageNet, a large benchmark dataset for image recognition.[2] Systems became more alike again: instead of a separate feature-engineering pipeline for each application, the same architecture could serve many.[1]

Foundation models: Functionality is learned

By the end of 2018, another shift was under way, driven by two developments: transfer learning and scale.[1]

Transfer learning is the idea that a model can learn useful patterns from one task and apply that knowledge to another. In practice, a model is pretrained on a large dataset and then adapted for specific tasks or applications. This approach is central to how foundation models are developed.[1]

Scale made these models increasingly capable, and it required several developments arriving together:[1]

• Better hardware. Processors became faster and gained more memory, allowing much larger models to be trained.

• The Transformer architecture, introduced in 2017, which could exploit that hardware by training in parallel, supporting far more expressive models than before.[3]

• More training data, providing the material from which these models could learn. This was the decisive change, shifting AI from systems trained for specific tasks towards models that learn broad patterns from large and diverse datasets and are later adapted for many different applications.

Self-supervised learning: Learning from unlabelled data

Transfer learning through pretraining was not new. Researchers had already used labelled datasets to pretrain image models for years. The challenge was that creating those labels required human time and expertise, and as datasets grew larger, the cost and effort of labelling became a practical constraint on how much data could be used for pretraining.[1]

Self-supervised learning reduced that constraint. Instead of requiring people to label each example, the training task is generated automatically from the data itself. BERT, for instance, was trained by hiding some words in a sentence and asking the model to predict them from the surrounding text.[4] The text provides both the training question and the expected answer, without requiring a person to label each example.

This significantly expanded the amount of data that could be used for training. Once training became less dependent on human labelling of individual examples, the main constraints became the availability of suitable data and the computing resources needed to process it.[1]

It is important, however, to distinguish removing annotation from removing human judgement. Self-supervised learning reduced the need for people to label individual training examples. It did not remove human decisions from the process. Judgement remained important in choices about which data to collect, include, filter or exclude, and in later stages of training that shape how a model behaves.

The scaling continued

The shift did not end in 2018. According to Epoch AI, the computing power used to train frontier AI models has grown at roughly four to five times per year since 2010, and its estimates suggest that around two-thirds of the improvement in language-model performance over the past decade has come from increases in scale rather than from new algorithmic ideas.[5] The same driver that produced foundation models is still operating, which is why the resources needed to build one have kept rising. A later article in this series examines compute in detail, including what it costs and who controls it.

What this history means for policy

Three points follow from this history, each of which becomes important later in this series.

1. First, more of the system's behaviour moved from code into training. The learning process first determined the method, then the features, and eventually much of the system's functionality. A system built this way cannot be understood simply by reading its code, because much of its behaviour has been learned rather than explicitly programmed. This is one of the reasons that auditing, explaining, and assuring these systems is difficult.

2. Second, the systems became increasingly similar. Learning algorithms converged, model architectures converged, and, eventually, many applications came to depend on the same small set of foundation models. This concentration can make development more efficient, but it also means that weaknesses or failures in a widely used model can be reproduced across many systems. The next article examines this issue in more detail.

3. Third, scale helps explain where the barriers to developing advanced models now sit. Increasing capability has required increasingly large amounts of computing power, data, infrastructure, and capital. As these requirements have grown, development has become concentrated among organisations with access to substantial resources. When later articles ask which models are being used in Africa, and who has the capacity to build them, part of the answer lies in this history.

The next article examines the two properties this development produced; emergence and homogenisation and why they make foundation models different from many AI systems that came before.

Sources

1. Bommasani, R., Hudson, D. A., Adeli, E., et al. (2021). On the Opportunities and Risks of Foundation Models. Center for Research on Foundation Models, Stanford Institute for Human-Centered Artificial Intelligence. arXiv:2108.07258. Available at https://arxiv.org/abs/2108.07258. Section 1 and section 1.1.

2. Krizhevsky, A., Sutskever, I. and Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems 25. Available at https://papers.nips.cc/paper_files/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html.

3. Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30. arXiv:1706.03762. Available at https://arxiv.org/abs/1706.03762.

4. Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805. Published in Proceedings of NAACL-HLT 2019. Available at https://arxiv.org/abs/1810.04805.

5. Epoch AI. Training compute of frontier AI models grows by 4-5x per year. Available at https://epoch.ai/publications/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year. Accessed on 15 September 2026