Introducing the Grounding Series: Understanding Foundation Models

This is the first article in the Africa Foundation Models Project's Grounding Series. Before examining what AI means for African policy, regulation and governance, the series steps back to the technology itself. It asks what foundation models are, what distinguishes them from the AI models that came before, and how their construction and deployment shape the questions that follow. This article begins with the term itself, where it came from, and why the argument about it reached statute.

In August 2021, more than a hundred researchers put a name to a shift already under way in AI: the foundation model (Bommasani et al. 2021). Researchers were building increasingly large models trained on vast and varied datasets, intended for adaptation to a wide range of uses. What was missing was a common way to describe the approach. The researchers proposed one.

Their definition was concise. A foundation model is a model trained on broad data, generally using self-supervision at scale, that can be adapted through methods such as fine-tuning to a wide range of downstream tasks (Bommasani et al. 2021). Three ideas sit within this definition, and each marks a departure from earlier practice: the breadth of the training data, the way the model learns from it, and its adaptability afterwards.

1. Broad data

A foundation model is not trained with a single task in mind. Its training data is deliberately broad, drawing on many sources and formats. These include text from the open web, code from public repositories, books, transcripts, images and their captions, and other publicly or commercially available material.

The aim is not a dataset that makes the model good at one job. It is to expose the model to enough different kinds of information, language and pattern that it develops capabilities useful across many tasks. Breadth, rather than fitness for a particular purpose, is the defining feature, and it determines what exists at the end of training: a general-purpose model rather than a finished tool.

2. Self-supervision at scale

The second part of the definition concerns how the model learns. For much of the history of machine learning, training relied on human-labelled examples. People identified which emails were spam, marked which X-rays showed a fracture, or classified whether a sentence expressed positive or negative sentiment. That approach remains useful where the task and the desired outcome are clearly defined. However, it requires substantial quantities of labelled data along with the time and expertise to produce it.

Self-supervised learning takes a different route. Rather than requiring each example to be labelled by a person, the model derives a training signal from the structure of the data itself. A word can be hidden from a sentence and the model is asked to predict what belongs there, with the surrounding text supplying the context from which it learns.

Since individual examples do not have to be labelled by hand, this can be applied across very large datasets. The scale of pretraining expands with the availability of suitable data and the computing resources to process it. It is important to note that human work does not disappear from the process. It moves into deciding what data is collected and filtered, and into a later stage of training that shapes how the model behaves. Subsequent articles discuss this further.

3. Adaptation, and the split it creates

Adaptation can take several forms. In fine-tuning, the model undergoes additional training on a smaller, task-specific dataset. Alternatively, the underlying model is left largely unchanged and instructions or examples are supplied as part of the input. This allows it to perform a new task without further training at all.

This is a significant departure from older practice, in which one model was generally built for one task. A sentiment classifier trained on labelled product reviews could classify sentiment and little else. Extracting clauses from contracts meant starting again with a new dataset, a new model and another round of labelling.

Foundation models split that process into two stages separated by cost, time and often institution. Pretraining produces the general model and consumes enormous quantities of data, computing power and money. Adaptation turns that model into a working application and is comparatively cheap. The foundation model is therefore not the finished application. It is the model from which many different applications are built. On its own, a model is a component. It becomes an AI system only once it is built into an application with an interface and other supporting parts, a distinction the European Union's AI Act also draws.

Both halves of the split were demonstrated within about two years. BERT, released by Google researchers in 2018, was pretrained on unlabelled text. It was then fine-tuned to state-of-the-art results across eleven language tasks by adding a single output layer (Devlin et al. 2018). GPT-3, published by OpenAI in 2020, went further. Its authors reported that for all tasks the model was applied "without any gradient updates or fine-tuning", with tasks specified through text instructions and a small number of demonstrations (Brown et al. 2020). Adaptation, at that point, no longer necessarily required training. It could require writing a prompt.

The asymmetry between the two stages matters well beyond the technical architecture. Very few organisations can perform the first stage. Very many can perform the second. African universities, hospitals, banks, courts and startups therefore encounter this technology largely at the adaptation layer. They build applications on models whose training data, design choices and failure modes were determined elsewhere. Subsequent articles look at some of the models being used on the continent.

Why the word "foundation"

The authors were explicit that the term was meant to fill a conceptual gap. Existing terminology, they argued, captured the technical dimension of these models but not the significance of the shift. Nor did it do so in a way accessible to readers outside machine learning (Bommasani et al. 2021). "Large language model" was too narrow, because the same approach was emerging in vision, audio, code and biology. "Pretrained model" described a method without saying what the method had changed.

A reflection published by the Stanford centre in October 2021 records how many alternatives were considered and rejected. Among them were base model, inframodel, platform model, polymodel and pluripotent model. After several weeks of debate, an internal tournament in which everyone in the centre weighed in settled on foundation model (Bommasani and Liang 2021).

The report's own explanation is that the word specifies the role these models play. A foundation model is itself incomplete, but serves as the common basis from which many task-specific models are built through adaptation. The authors also chose it to signal the importance of architectural stability. Their reasoning was that badly built foundations invite disaster, while well-executed ones provide reliable bedrock for what follows (Bommasani et al. 2021).

That metaphor carries a warning the authors made explicit. Writing in 2021, they said they did not fully understand the nature or quality of the foundation these models provide, and could not say whether it was trustworthy (Bommasani et al. 2021). A foundation is load-bearing, which means faults in it can affect everything built above. A later article will examine what follows when one model sits beneath many applications.

The dispute over the name

The term was contested almost immediately, from two directions. The first objection was that it overstates what these models are. Jitendra Malik of UC Berkeley accepted that large language models had real practical use. He rejected, however, the idea that they could serve as a foundation for AI more broadly. Because they lack sensorimotor grounding, they learn from text without direct connection to physical experience, and for this reason he called them "castles in the air" (Malik 2021). Writing in The Gradient, Gary Marcus and Ernest Davis made a related argument. In their view, relabelling pretrained language models does not make them the foundation that AI actually needs. That foundation would require reasoning, common sense and reliable world knowledge, which these models did not possess (Marcus and Davis 2021).

The second objection concerned what happens when a technology is renamed. Meredith Whittaker, then at the AI Now Institute, characterised the move as an attempt at erasure of the older term. She argued that a fresh label detaches a technology from the criticism accumulated around its predecessor and gives it a cleaner start in public discourse (Field 2021).

The authors' answer to both objections lies in the term itself. "Foundation" was chosen to signal incompleteness, not quality. Calling something a foundation does not assert that it is a good one (Bommasani and Liang 2021).

Why the argument matters outside the academy

A naming dispute among researchers would matter little had the term stayed in academic papers. It did not. In its June 2023 negotiating position on the AI Act, the European Parliament introduced "foundation model" as a distinct legal category. The final legislation dropped the term and regulates "general-purpose AI models" instead. In the United States, Executive Order 14110 of October 2023 defined a "dual-use foundation model". The definition referred to broad training data, self-supervision, a parameter count in at least the tens of billions and applicability across many contexts, together with a capability test tied to national security. That order was revoked in January 2025.

The result is not merely a disagreement about words. Different instruments use different terms for broadly overlapping sets of models, and apply different tests to decide what falls inside each category. For anyone drafting, applying or complying with AI rules, that creates a practical difficulty. What counts as a foundation model under one framework may not count under another. The boundary of the legal category is also not where the boundary of the technical one sits.

That is why the argument over a name matters. The words used to describe these models do not only organise technical ideas. They determine what is regulated, who falls within the rules, and where responsibility sits. The next article turns to that problem of scope.

References

Bommasani, Rishi, Drew A. Hudson, Ehsan Adeli, et al. (2021), On the Opportunities and Risks of Foundation Models, arXiv:2108.07258, Stanford, CA: Center for Research on Foundation Models, Stanford University, accessed 2 September 2026.

Bommasani, Rishi, and Percy Liang (2021), Reflections on Foundation Models, 18 October, Stanford, CA: Stanford Institute for Human-Centered Artificial Intelligence, accessed 2 September 2026.

Brown, Tom B., Benjamin Mann, Nick Ryder, et al. (2020), Language Models Are Few-Shot Learners, arXiv:2005.14165, accessed 2 September 2026.

Devlin, Jacob, Ming-Wei Chang, Kenton Lee, et al. (2018), BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv:1810.04805, accessed 2 September 2026.

Field, Hayden (2021), At Stanford's "Foundation Models" Workshop, Large Language Model Debate Resurfaces, Tech Brew, 30 August, accessed 2 September 2026.

Malik, Jitendra (2021), Response on "Foundation Models", 18 October, Stanford, CA: Center for Research on Foundation Models, accessed 2 September 2026.

Marcus, Gary, and Ernest Davis (2021), Has AI Found a New Foundation?, The Gradient, 11 September, accessed 2 September 2026.

Legal instruments

European Parliament (2023), Amendments Adopted by the European Parliament on 14 June 2023 on the Proposal for a Regulation … Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act), P9_TA(2023)0236, Brussels: European Parliament, accessed 2 September 2026.

European Union (2024), Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act), Official Journal of the European Union, L series, 12 July, accessed 2 September 2026.

United States (2023), Executive Order 14110 of October 30, 2023: Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, 88 Federal Register 75191, Washington, DC: Office of the Federal Register, accessed 2 September 2026.

United States (2025), Executive Order 14148 of January 20, 2025: Initial Rescissions of Harmful Executive Orders and Actions, 90 Federal Register 8237, Washington, DC: Office of the Federal Register, accessed 2 September 2026.