Capability: What Foundation Models Can Do

The previous articles established what foundation models are, how they came about, and what distinguishes them from their predecessors. Attention now turns to what they can do and where those capabilities end, across two articles. This first article sets out the range of current capabilities and the care required when assessing claims about them. The next examines how these systems fail and the groups of people on whom failure falls unevenly.

Once trained, a foundation model can be applied to tasks it was not specifically developed for. This adaptability, together with training on broad data at scale, is one of its defining characteristics, as the earlier articles set out. The sections below describe the tasks these models currently handle, how a general model is narrowed to a particular use, what capability claims establish and what they do not, and what it means to describe a model as reasoning.

The range of current capabilities of foundation models

Language: The commonly known foundation models work on text. They draft and summarise documents, translate, answer questions, classify material, extract information from forms and sustain a conversation. One model can do all of this without retraining for each task. Before foundation models, each of these functions usually needed its own system, built and trained separately.

Code: Models trained on large quantities of source code write, explain and correct software. OpenAI’s Codex was an early demonstration that a language model could produce working programs from descriptions written in ordinary language (Chen et al. 2021). Programming is now among the most heavily measured capabilities, tested through benchmarks that ask models to resolve real issues taken from open-source projects rather than to complete textbook exercises (Jimenez et al. 2024).

Images: Models describe a photograph, read text in a scanned document and interpret a chart. Their reliability varies widely by task.

Audio: Speech recognition models transcribe and translate spoken language (Radford et al. 2022), and other models generate synthetic speech or music. Performance can differ sharply between languages.

Combined inputs: The most capable systems available now accept text, images and audio together. A user can submit a photograph of a completed form, ask a question about it by voice, and receive a written answer (OpenAI 2024).

Models that take action: A newer category is models that act rather than only produce text. They search the internet, run code, fill in forms and operate software on a user’s behalf. These systems are usually called agents. The International AI Safety Report 2026 records that the length of software tasks an agent can complete has been doubling roughly every seven months, and that if the trend holds, systems could reliably complete well-specified software engineering work taking a person several days by 2030. The report adds that it is unclear whether this will extend to other kinds of work (Bengio et al. 2026). Agents raise separate questions of governance, addressed in a later article.

As a general indication of where matters stand, the same report records that leading models now pass professional licensing examinations in medicine and law, and answer more than 80% of graduate-level science questions in some tests.

General-purpose and single-purpose systems

It helps to distinguish between two kinds of systems at the outset. A single-purpose system is built for one task. A credit-scoring model, or a spam filter, is trained on data chosen for that task and measured against it. Its scope is narrow and, in principle, precisely definable.

A general-purpose system, by contrast, is built without one task in mind. One organisation may use the same underlying model to summarise judgments in English, another to answer customer enquiries in Kiswahili, and a third to write promotional copy. This is what is called homogenisation, where many applications come to rest on a small number of shared models (Bommasani et al. 2021).

The distinction between a single-purpose and a general-purpose system matters for regulation. Rules for single-purpose systems typically assume the intended purpose is known while the system is being built, so risk can be assessed against it. With a general-purpose model, the developer often does not know how it will be used. The European Union’s AI Act addresses this by creating a separate category for general-purpose AI models, placing obligations on the model provider regardless of downstream use, alongside risk-based obligations that attach to specific applications (Regulation (EU) 2024/1689).

In practice, however, the boundary between the two is blurring at deployment. Many applications that appear single-purpose are built on a general model narrowed to one task, rather than trained for that task from the start. Some of the methods that make this possible include:

1. Fine-tuning, which continues training the general model on a smaller, task-specific dataset. Commonly the original weights are frozen and a small set of additional weights is trained alongside them, achieving much the same effect at a fraction of the cost.

2. System instructions, which fix the model’s role and state what it may and may not do, without further training. These shape behaviour rather than constrain it absolutely, and a sufficiently crafted input can override them.

3. Retrieval, which supplies the model with approved documents and directs it to base its answer on them. This is the method behind many legal and medical research products, and it reduces unsupported statements but does not completely eliminate them. A study of three commercial legal research tools found that each still produced hallucinated content between 17% and 33% of the time, although less often than a general-purpose chatbot (Magesh et al. 2024).

4. Filters, which sit around the model, also known as guardian models, and screen both what reaches it and what it returns. These are usually trained models themselves, with their own error rates.

Of the four, only fine-tuning changes how the model itself responds. The other three leave it as it is and control what goes in and what comes out. Either way, the general model’s characteristics continue to shape the narrowed application, which is why a foundation model’s inherited defects survive adaptation.

Instructions as the main interface for foundation models

With earlier AI systems, changing what a model did usually meant rebuilding it. With foundation models, behaviour is directed mainly through the instructions and material given as input, known as the prompt.

A prompt may be a single question, or it may be long and structured, holding instructions, background documents, worked examples and rules about format. The second kind has been shown to change results markedly. Including a few examples of a task done correctly often improves performance without altering the model, an effect demonstrated at scale with GPT-3 (Brown et al. 2020). Organisations that build products on foundation models also write detailed background instructions that the end user never sees, setting out what the model may and may not do.

This has two implications for policymakers. 

1. First, a deployed application’s behaviour depends heavily on its instruction design, which is often undocumented and can change without notice, so a service assessed in one quarter may behave differently the next even if the underlying model does not change. 

2. Second, because ordinary language is the interface, people without technical training can direct these systems. This lowers the barrier to use, and it also means that faulty instruction, or deliberate manipulation, can come from anyone with access.

Capability claims and their limits

Statements about what a model can do are made available through a number of forms:

1. Release announcements and technical reports, published by the developer when a model launches, usually including a table of benchmark results. OpenAI’s GPT-4 Technical Report, for example, reports the model’s results on professional and academic examinations alongside its stated limitations (OpenAI 2023).

2. Model and system cards, which set out a model’s intended uses, tested performance and known limitations. Researchers proposed the format in 2019 as a way of reporting performance “across different conditions, such as demographic and cultural groups” (Mitchell et al. 2019), and it is now standard practice. An example is the GPT-4o System Card (OpenAI 2024).

3. Public leaderboards, which rank systems against one another on shared tests. Arena, launched as Chatbot Arena in 2023, later known as LMArena and renamed in January 2026, ranks models on votes cast by members of the public who compare two anonymous responses to the same prompt (Chiang et al. 2024). Stanford’s Holistic Evaluation of Language Models publishes rankings across a wider set of standardised tasks (Liang et al. 2023).

4. Press coverage and sales material, which repeat figures drawn from the first three. Reports in July 2025 that a model had reached gold-medal standard at the International Mathematical Olympiad are an example. The finding originated in a single developer announcement and circulated widely from there.

Underlying all of them is the benchmark, a fixed set of test questions with known answers, allowing one system to be compared with another on the same material.

Two points about how these figures are produced and used matter:

1. The figures nearly always originate with the organisation that built the model, which also selects and runs the tests.

2. Institutions deciding whether to adopt a system generally rely on those figures in any event, because independent evaluation is costly and few organisations outside the largest laboratories carry it out.

Capability claims therefore carry considerable weight, often serving as the main evidence behind consequential choices, such as procurement decisions, policy assumptions or public expectations.

What a benchmark score establishes is real, and it is limited. It shows only that, on a stated date, a particular model version, tested under conditions the tester chose, answered a fixed set of questions correctly at a measured rate. This supports comparison between systems tested on the same material, but it does not establish that the model will perform the task for a given user in a given setting. Three features of benchmark testing explain why the score does not always translate into reliable performance in use:

1. Capability is uneven. The International AI Safety Report 2026 notes that performance “remains ‘jagged’, with leading systems still failing at some seemingly simple tasks” (Bengio et al. 2026). A field experiment with management consultants showed what this means in practice. Within the model’s competence, its help improved the quality of the work. Beyond that competence, consultants using it were more likely to reach wrong conclusions (Dell’Acqua et al. 2023). The model gives no indication of which tasks fall within its competence, and users have no reliable way of telling.

2. Results do not transfer across settings. A model’s ability to perform a task under test conditions does not establish that it will perform the same task reliably in a particular setting, population or language. That is the subject of the next article.

3. Tests become less informative over time, and a high score can be reached without the capability the test was designed to detect. The same report describes an evaluation gap, meaning that “pre-deployment test results are not always strongly predictive of real-world capabilities or risks”. Tests may be outdated, too narrow, or built from questions that already appear in the model’s training data. Once a test is widely adopted, leading systems score so near the maximum that the test no longer separates them. The report also records that models have become better at distinguishing test conditions from real deployment, and that they more often find loopholes which earn high scores without meeting the intended goal.

The questions to put to any capability claim are therefore how well the model does the task, how consistently, for which population, and on whose measurement.

Reasoning as one of the capabilities applied to foundation models

Some models are now marketed as reasoning models. They display a sequence of intermediate steps before giving an answer, and they score well on tests of mathematics, science and programming. When OpenAI introduced its o1 model in September 2024, it reported that the model solved 74% of problems on that year’s American Invitational Mathematics Examination, against 12% for its conversational predecessor, answered 77.3% of questions correctly on a benchmark of graduate-level science problems, and reached a competitive programming rating in the 89th percentile of human contestants (OpenAI 2024b).

These visible steps invite a particular inference, namely that the conclusion was reached by a route that can be followed, examined and relied upon.

Reasoning, however, is understood differently in legal and technical contexts, as each describes different aspects. The governance question then becomes whether the technical sense can carry the weight the legal sense places on it.

In the legal sense, a statement of reasons records how a conclusion was reached, and its value lies in being testable. It can be examined, challenged on appeal or review, and used to identify where the decision went wrong. A great deal of current administrative and data protection law rests on that assumption. For example, in Kenya, Article 47(2) of the Constitution gives a person whose right or fundamental freedom has been or is likely to be adversely affected by administrative action the right to be given written reasons for it, and the Fair Administrative Action Act 2015 gives effect to that right. Section 35 of Kenya’s Data Protection Act 2019 and section 71 of South Africa’s Protection of Personal Information Act each regulate decisions made solely by automated means. Under the General Data Protection Regulation, Article 22 restricts such decisions and Article 15(1)(h) entitles a data subject to meaningful information about the logic involved. Article 86 of the EU AI Act goes further for certain high-risk systems, giving a person affected by a decision a right to “clear and meaningful explanations of the role of the AI system in the decision-making procedure”.

Each of these provisions assumes that an explanation corresponds to the process that produced the decision.

The technical sense of reasoning refers to narrower aspects, namely the output format. It describes a model that generates a sequence of intermediate steps before answering, and the training methods that enable it to do so. The practice began as a prompting technique. In 2022, researchers found that asking a model to set out its working before answering, rather than answering immediately, improved its performance on arithmetic and logic problems (Wei et al. 2022). Developers then trained that behaviour into the models themselves, rewarding correct final answers so that longer workings, including the checking and revision of the model’s own output, emerged without being specified (DeepSeek-AI 2025). The steps are generated text, produced by the same operation as the answer they accompany.

The two forms of reasoning differ in important ways. The legal sense describes a relationship between an explanation and a decision. The technical sense describes an output format. Whether a model’s steps satisfy the legal sense is an empirical question, which is examined below.

Two questions arise about the technical kind of reasoning:

1. How reliable is the underlying capability, meaning how well it holds up under varied conditions?

2. Do the steps a model shows correspond to what actually produced its answer?

Researchers have studied both. On the first question, the evidence points both ways. Models solve problems that are unlikely to have appeared in their training data. In July 2025, Google DeepMind reported that an advanced version of its Gemini model reached gold-medal standard at the International Mathematical Olympiad, judged by the competition’s own coordinators and working from problems written in ordinary language (Google DeepMind 2025). At the same time, performance breaks down in ways that suggest reliance on familiar patterns. In a contrasting study, researchers at Apple generated variants of a standard school mathematics test by altering the numerical values in each question and found that every model tested performed worse. Adding a single clause that appears relevant but does not affect the answer reduced performance by up to 65% (Mirzadeh et al. 2025). Whether any of this amounts to reasoning in a deeper sense remains contested, one position being that a system trained only on the form of language cannot acquire meaning at all (Bender and Koller 2020). That argument can be set aside here, because the question that matters for a decision-maker is the second one.

On the second question, the evidence points more clearly in one direction. The steps a model shows often do not correspond to what produced its answer. In a 2023 study, researchers planted a bias in the prompt to steer the model towards a particular answer and found that the model then produced a convincing explanation that made no mention of it (Turpin et al. 2023). In 2025, Anthropic placed hints in prompts and found that, where a model used a hint, it often did not disclose having done so (Chen et al. 2025). The authors concluded that examining a model’s stated working is useful but cannot be relied on to show what the model was doing.

The technical sense of reasoning therefore does not deliver what the legal sense requires. Three points follow for those relying on these systems.

1. A model’s stated reasoning should be treated as an output. It is generated text, it may contain error, omission and after-the-fact justification, and it is not a verified record of how the answer was produced.

2. Correct steps do not guarantee a correct answer, and a correct answer does not validate the steps. Each requires checking on its own.

3. The label “reasoning model” describes a training method and a product category. It does not certify that any output meets a standard of reasoned decision-making.

These findings concern the status of a model’s stated reasoning, not the usefulness of the models themselves. On many well-defined problems they are measurably more accurate than earlier models, and inspecting the intermediate steps helps a human reviewer locate errors. The difficulty arises where the term “reasoning” is accepted in place of evidence, so that a system described as reasoning is assumed to produce explanations that can be relied upon.

Conclusion

As shown, capability is not a fixed property of a foundation model. A single model performs many functions, and the deploying organisation decides which function it performs in practice. Its behaviour within any given service depends on the instructions given. The evidence available to a prospective user comes from the developer and establishes only a narrow measure, namely the rate of correct answers to a fixed set of questions under conditions the developer selected.

Reasoning illustrates the wider difficulty. A term with a settled legal meaning has been applied to an output format, and the research on correspondence indicates that the two do not reliably coincide. Where a decision must be explained, a model’s stated workings will not necessarily discharge that obligation.

For anyone assessing these systems, the practical test is therefore how well the model performs the task, how consistently, for which population, and on whose measurement. The next article explains why those questions matter, by outlining how these systems fail and the groups of people on whom failure falls unevenly.

References

Bender, Emily M., and Alexander Koller (2020), Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5185–5198, Online: Association for Computational Linguistics, accessed 16 October 2026.

Bengio, Yoshua, Stephen Clare, Carina Prunkl, et al. (2026), International AI Safety Report 2026, London: UK Government, accessed 16 October 2026.

Bommasani, Rishi, Drew A. Hudson, Ehsan Adeli, et al. (2021), On the Opportunities and Risks of Foundation Models, arXiv preprint, 2108.07258, Stanford: Center for Research on Foundation Models, accessed 16 October 2026.

Brown, Tom B., Benjamin Mann, Nick Ryder, et al. (2020), Language Models are Few-Shot Learners, arXiv preprint, 2005.14165, accessed 16 October 2026.

Chen, Mark, Jerry Tworek, Heewoo Jun, et al. (2021), Evaluating Large Language Models Trained on Code, arXiv preprint, 2107.03374, accessed 16 October 2026.

Chen, Yanda, Joe Benton, Ansh Radhakrishnan, et al. (2025), Reasoning Models Don’t Always Say What They Think, arXiv preprint, 2505.05410, accessed 16 October 2026.

Chiang, Wei-Lin, Lianmin Zheng, Ying Sheng, et al. (2024), Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, arXiv preprint, 2403.04132, accessed 16 October 2026.

DeepSeek-AI, Daya Guo, Dejian Yang, et al. (2025), DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning, Nature, 645, accessed 16 October 2026.

Dell’Acqua, Fabrizio, Edward McFowland III, Ethan R. Mollick, et al. (2023), Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality, Harvard Business School Technology and Operations Management Unit Working Paper, 24-013, Boston: Harvard Business School, accessed 16 October 2026.

Google DeepMind (2025), Advanced Version of Gemini with Deep Think Officially Achieves Gold-Medal Standard at the International Mathematical Olympiad, London: Google DeepMind, accessed 16 October 2026.

Jimenez, Carlos E., John Yang, Alexander Wettig, et al. (2024), SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, arXiv preprint, 2310.06770, accessed 16 October 2026.

Liang, Percy, Rishi Bommasani, Tony Lee, et al. (2023), Holistic Evaluation of Language Models, arXiv preprint, 2211.09110, accessed 16 October 2026.

Magesh, Varun, Faiz Surani, Matthew Dahl, et al. (2024), Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, arXiv preprint, 2405.20362, accessed 16 October 2026.

Mirzadeh, Iman, Keivan Alizadeh, Hooman Shahrokhi, et al. (2025), GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models, arXiv preprint, 2410.05229, accessed 16 October 2026.

Mitchell, Margaret, Simone Wu, Andrew Zaldivar, et al. (2019), Model Cards for Model Reporting, in: Proceedings of the Conference on Fairness, Accountability, and Transparency, New York: Association for Computing Machinery, accessed 16 October 2026.

OpenAI (2023), GPT-4 Technical Report, arXiv preprint, 2303.08774, accessed 16 October 2026.

OpenAI (2024), GPT-4o System Card, arXiv preprint, 2410.21276, accessed 16 October 2026.

OpenAI (2024b), Learning to Reason with LLMs, San Francisco: OpenAI, accessed 16 October 2026.

Radford, Alec, Jong Wook Kim, Tao Xu, et al. (2022), Robust Speech Recognition via Large-Scale Weak Supervision, arXiv preprint, 2212.04356, accessed 16 October 2026.

Turpin, Miles, Julian Michael, Ethan Perez, and Samuel R. Bowman (2023), Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting, arXiv preprint, 2305.04388, accessed 16 October 2026.

Wei, Jason, Xuezhi Wang, Dale Schuurmans, et al. (2022), Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, arXiv preprint, 2201.11903, accessed 16 October 2026.

Legal instruments

Constitution of Kenya 2010, Article 47.

Fair Administrative Action Act 2015 (No. 4 of 2015), Kenya.

Data Protection Act 2019 (No. 24 of 2019), Kenya, section 35.

Protection of Personal Information Act 2013 (Act No. 4 of 2013), South Africa, section 71.

Regulation (EU) 2016/679 (General Data Protection Regulation), Articles 15 and 22.

Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 8