In the previous article, we traced the path from early machine learning to foundation models. That path produced two features that are particularly relevant to governance: emergence and homogenisation (Bommasani et al. 2021). Taken together, they explain why the governance question for foundation models is not the same as the governance question for conventional software.
1. Emergence: Capabilities that arise rather than being specified
Emergence means that some of a model's capabilities are not explicitly programmed by the developer. Instead, they arise as a by-product of training the model on a large quantity of data (Bommasani et al. 2021). As a result, a model may be used for purposes beyond those for which it was initially developed. GPT-3, for example, had 175 billion parameters. It could perform a wide range of tasks when given ordinary-language instructions, even though it had not been specifically trained for many of those tasks (Brown et al. 2020).
This creates a practical difficulty compared with conventional software, which usually has a specification. In conventional software, a developer decides what the system should do, records that decision and builds the system accordingly. If the system behaves unexpectedly, its conduct can be assessed against what it was designed to do.
Foundation models are documented through model cards, technical reports and evaluation results. Developers test them before release, so much of their behaviour is known and recorded. However, no complete specification defines everything the model is expected to do. Some behaviours are intentionally shaped during training and documented, while others are identified through testing and later use. The full range of what a model can do may therefore not be known when it reaches the market.
Do capabilities appear suddenly?
The idea of emergent abilities is disputed, but it is clear that some model capabilities may only be discovered after deployment.
In 2022, researchers identified tasks on which smaller models performed near chance while larger models performed substantially better. They called these emergent abilities: capabilities absent in smaller models that appear in larger ones, and that cannot be predicted simply by extrapolating from the performance of smaller models (Wei et al. 2022).
In 2023, Stanford researchers challenged this explanation. They argued that the apparent sudden improvement could result from how performance is measured, so that the choice of metric affects whether an ability appears to emerge suddenly. All-or-nothing measures can make gradual improvements look like sudden jumps, while measures that award partial credit show more gradual improvement in the same underlying model. On one widely used benchmark, more than 92 per cent of claimed emergent abilities appeared under just two such measures. In the cases the researchers re-tested, the apparent emergence largely disappeared when the measure was changed (Schaeffer et al. 2023).
The debate continues. For policymakers, however, the core point is that a foundation model's full capabilities may not be known before deployment. Regulation therefore cannot rely entirely on a developer's pre-release description of what a model can or cannot do. Some capabilities may only become apparent through further evaluation and real-world use.
How the law has responded: Continuing obligations rather than one-off certification
The EU AI Act provides a practical example of a regime built around this problem. Its obligations recognise that the risks and capabilities of general-purpose AI models may not be fully known before deployment.
Under Article 55, providers of general-purpose AI models with systemic risk must conduct and document model evaluations, including adversarial testing, to identify and reduce systemic risks. They must also:
- assess and mitigate risks that may arise from the model's development, placing on the market or use;
- report serious incidents to the AI Office without undue delay; and
- maintain adequate cybersecurity for the model and its physical infrastructure.
The Act therefore relies on ongoing evaluation, risk management, incident reporting and security measures, rather than treating compliance as a one-time assessment before release.
These are not prospective obligations. The general-purpose AI provisions have applied since 2 August 2025, and the Commission's power to enforce them through fines has applied since 2 August 2026. The relevant European technical standards are not yet in place, so Article 56 provides for codes of practice as an interim route to demonstrating compliance. The General-Purpose AI Code of Practice, prepared by independent experts, was published on 10 July 2025. It has three chapters. Transparency and Copyright apply to all providers of general-purpose AI models under Article 53, while Safety and Security applies only to providers of models with systemic risk under Article 55. Signing is voluntary, and the Commission and the Member States, through the AI Board, have confirmed the Code as an adequate way to demonstrate compliance.
2. Homogenisation: The same models beneath many applications
Homogenisation refers to the growing use of the same foundation models across different applications and sectors. This creates both benefits and risks. Improvements to a shared model can benefit many applications at once. Biases, security weaknesses, coverage gaps or other errors can equally spread across downstream systems, because models adapted from a foundation model inherit its defects (Bommasani et al. 2021).
The empirical position is more precise than the general concern suggests. A 2022 study built on earlier work on algorithmic monoculture (Kleinberg and Raghavan 2021). It examined what it termed outcome homogenisation: the extent to which particular individuals or groups experience negative outcomes from every system they encounter.
- Shared training data. Sharing training data reliably increased homogenisation. The effects were generally larger at the individual level than at the group level, so group-level analysis may not reveal systemic failures concentrated on particular individuals.
- Shared foundation models. For foundation models specifically, the results were mixed. The method used to adapt the model for the downstream task significantly influenced how much homogenisation occurred (Bommasani et al. 2022).
Adaptation is therefore an important point for assessing and managing this risk.
The legal significance of homogenisation is that widespread use of a common foundation model can turn a model-level problem into a systemic one. The AI Act recognises this in Article 3(65), which defines systemic risk by reference to effects that can be propagated at scale across the value chain. A risk originating in a foundation model can therefore spread through downstream systems and affect multiple users and sectors. This has implications for how risk is assessed, how responsibilities are allocated, and where obligations sit along the AI value chain.
Why these two features matter and the policy relevance
Emergence means that a model's full behaviour may not be known at release. Homogenisation means that many applications may depend on the same underlying model. Together they create a correlated risk, in which an unknown defect in one model can affect many systems at once. Several regulatory implications follow.
Application-level oversight may not be sufficient. Some characteristics that shape downstream risk are established during model development and training, often by a provider separate from the application developer. Assessing only the deployed application may therefore miss risks originating in the underlying model.
Group-level fairness assessments may not capture individual harms. Research suggests that negative outcomes can concentrate on the same individuals across different systems, even when group-level statistics appear acceptable. Regulators may therefore need to consider both group-level and individual-level effects.
Reliance on a small number of foundation models creates a resilience issue. A defect or limitation in a widely used model can propagate across applications and sectors. Model diversity therefore affects not only competition but also systemic resilience.
These issues are particularly relevant for jurisdictions that mainly host deployments rather than model development. Where model providers are based elsewhere, evaluations, adversarial testing and incident reporting may take place under foreign regulatory regimes and be reported to foreign authorities. This does not remove the value of those measures, but it creates a gap between model-level oversight and domestic oversight.
Foundation models therefore raise governance questions beyond whether an individual application is safe or compliant. Regulators also need to consider the underlying model, how it is adapted, who depends on it, and how risks can propagate across downstream uses.
This makes understanding which models are actually being used an important regulatory question and it is the subject of the next article.
References
Bommasani, Rishi, Drew A. Hudson, Ehsan Adeli, et al. (2021), On the Opportunities and Risks of Foundation Models, arXiv:2108.07258, Stanford, CA: Center for Research on Foundation Models, Stanford University, accessed 15 September 2026.
Bommasani, Rishi, Kathleen A. Creel, Ananya Kumar, et al. (2022), Picking on the Same Person: Does Algorithmic Monoculture Lead to Outcome Homogenization?, in: Advances in Neural Information Processing Systems 35, arXiv:2211.13972, accessed 15 September 2026.
Brown, Tom B., Benjamin Mann, Nick Ryder, et al. (2020), Language Models Are Few-Shot Learners, arXiv:2005.14165, accessed 15 September 2026.
Kleinberg, Jon, and Manish Raghavan (2021), Algorithmic Monoculture and Social Welfare, Proceedings of the National Academy of Sciences, 118, 22, e2018340118, accessed 15 September 2026.
Schaeffer, Rylan, Brando Miranda, and Sanmi Koyejo (2023), Are Emergent Abilities of Large Language Models a Mirage?, in: Advances in Neural Information Processing Systems 36, arXiv:2304.15004, accessed 15 September 2026.
Wei, Jason, Yi Tay, Rishi Bommasani, et al. (2022), Emergent Abilities of Large Language Models, Transactions on Machine Learning Research, arXiv:2206.07682, accessed 15 September 2026.
Legal instruments
European Commission (2025), The General-Purpose AI Code of Practice, 10 July, Brussels: European Commission, accessed 15 September 2026.
European Union (2024), Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act), Official Journal of the European Union, L series, 12 July, accessed 15 September 2026.
