Skip to content

Reviewed edition · qasb report 10/2026 · Topic report · Products

AI products: recurring limits in testing

Almost a fifth of US adults reported near-daily AI use in August 2026. In tests, however, AI models run into recurring limits: when evidence is missing, outside English and in long agent tasks.

Published
October 27, 2026 · as of October 26
Reading time
11 min
Publisher
qasb Research
Method
qasb methodology v1

Key findings

  1. −5.6 pointsin the follow-up task success rate in a test when coding systems built on their own changes, not developers' patches.1Evidence: observed
  2. 54.4%of performance pull requests by autonomous coding agents on GitHub were merged, against 73.3% for human ones.2Evidence: observed
  3. 77.9%of the strongest tested model's answers fully incorporated a clinical guideline change, while 98.0% recognised it.3Evidence: observed
  4. 45.0%of rheumatology chatbot answers were fully guideline-backed, in an AI rating that agreed weakly with physicians.4Evidence: observed

1.0Summary

October's evidence mainly shows where AI products hit limits in tests, but little about how they work in real operation. It comes mostly from vendor system cards, independent benchmarks and preprints. All results apply only to the versions, tasks and languages tested.

  • According to two US surveys, 18.7% of adults used AI services on 6 or 7 days of the previous week in August 2026, against 8.2% in March. The question changed in between.
  • In the KNOWS benchmark, the best agent fully solved only 2.7% of long browser tasks. In a coding benchmark, systems solved follow-up tasks less often on top of their own changes.
  • Without a correct option, models in a US legal test often chose a wrong one over none. In a medical test, all four models were more accurate in English than in six other languages.
  • Vendors themselves report regressions, after-the-fact corrections and risk thresholds they cannot rule out. These self-evaluations show no frequency in real operation.

2.0Starting point: more use, more effort

Three sources show how widespread AI services are and what effort their use involves: two US surveys from one online panel, an analysis of OpenAI's own figures on its researchers' coding-agent use, and energy measurements of open-weight models. They measure scale and effort, not benefit. The dollar values rest on list prices rather than internal costs.

Figure 1

AI use by US adults in the previous week

Share of US adults, surveys in March and August 2026

Two independent samples from one online panel. The question and the list of services changed between the waves, so the comparison is only approximate. Self-reports, US only, nothing on duration, purpose or benefit.Evidence: observed · Sources 11

The share of US adults who used AI services on 6 or 7 days of the previous week rose from 8.2% (March 2026) to 18.7% (August 2026) according to two Epoch AI surveys on the Ipsos KnowledgePanel; any AI use in the previous week was reported by 50.0% and 56.4% respectively.11A1

According to an Epoch AI analysis of OpenAI's self-reported data, the median OpenAI researcher's daily coding-agent usage, valued at API list prices, rose from under 1 US dollar in January 2026 to 601 US dollars in mid-August 2026; at the 90th percentile it exceeded 7,000 US dollars per day.20A2

Further findings

  • In measurements of 86 open-weight language models, thinking mode raised GPU energy use by about 3.7 times on average.17A3

3.0Agents: long tasks hardly ever fully completed

Four preprints examine how often agents fully complete long tasks, how coding agents build on their own work, how often their performance changes are merged and how much results vary across repeated runs. In a benchmark of long browser tasks, the best agent passed most evaluated check steps, but errors on visual steps alone made outputs unusable.

In the KNOWS benchmark of 110 long browser tasks that should end in a document, a presentation or a spreadsheet, the best agent tested, the AI browser Perplexity Comet, fully completed only 2.7% of tasks; ChatGPT Atlas as well as GPT-5.5, Claude Opus 4.7 and DeepSeek V4 Pro in an open agent harness fully completed none.19A4

In the SWE-Maintain benchmark, five coding systems with two base models solved follow-up tasks 5.6 percentage points less often when building on their own earlier changes instead of the developers' patches; when all earlier tests also had to keep passing, the success rate fell from 33.6% to 19.6%.1A5

The best agent tested, the AI browser Perplexity Comet, fully completed only 2.7% of the long browser tasks.

In public GitHub projects, performance pull requests by autonomous coding agents were merged less often than comparable human ones (54.4% vs 73.3%); in both groups, about half of the pull requests with validation information reported no quantitative performance metric.2A6

Further findings

  • In one agent benchmark, about 54% of outcome variance among genuine attempts came from simply repeating the same configuration.6A7

4.0Answers: weaker without evidence and outside English

Four studies examine how models handle missing evidence, how accurately they check facts and how much language and errors in the input change their answers. All are preprints and measure in tests, not in everyday work.

When the correct holding of a cited US court decision was missing from the answer options, six generative language models in an extended CaseHOLD test still picked a wrong option in 37.8% (Gemini 3.8 Flash) to 85.6% (GPT-4o-mini) of cases instead of stating that none applied.12A8

On 1,089 identical medical examination questions, the accuracy of all four decision models tested was lower in each of the six other language versions than in English; it fell most in Yoruba, by 20.1 to 39.4 percentage points (for Clef-flash from 0.93 to 0.54).13A9

When the correct holding was missing, six language models still picked a wrong option in 37.8% to 85.6% of cases instead of stating that none applied.

Further findings

  • In a Japanese benchmark, all six models checked main claims less accurately than supplementary ones, significantly so for only three.21A10
  • Grammatical errors typical of non-native writers increased bias in five older open-weight models, most clearly for gender.16A11

5.0Medicine: guidelines, images, authorisation

In medicine, the evidence ranges from benchmarks and a Germany-wide field evaluation to authorisation counts and regulatory plans. In the tests, models did not always fully incorporate the evidence provided, two open vision models answered no better with the image, and in one study multi-agent systems mostly improved task results without reducing hallucinations in rationales as a rule.

Figure 2

Image questions with and without the image

Image questions of Spain's MIR residency entrance exam answered correctly, two open vision-language models

The same 171 questions. For Gemma-3-27B, the difference associated image minus no image was −1.8 percentage points (95% CI −7.0 to +2.9), so no clear difference. More than half of the image questions could already be solved from the text. Preprint, exam questions rather than clinical cases.Evidence: observed · Sources 18

In the ChronoEvo benchmark, all 14 tested language models recognised clinical guideline changes more often than they fully incorporated them into their recommendation; even GPT-5.6 Sol, the strongest model tested according to the study, recognised the change in 98.0% of responses but fully incorporated it in only 77.9%.3A12

In a nationwide field evaluation of guideline-grounded rheumatology chatbots (GPT-4o with retrieval), an automated LLM assessment rated 79.1% of answers as completely correct but only 45.0% as fully supported by the retrieved guideline passages; users rated 92.9% of the 2,671 rated answers positively.4A13

Two open vision-language models did not answer 171 image questions of the Spanish MIR residency examination better with the associated image than without an image or with another question's image (Gemma-3-27B: 67.8%, 69.6% and 67.3% correct; MedGemma-27B: 62.6%, 64.9% and 63.2%).18A14

Further findings

  • Of 36 comparisons, 12 were resolvable: multi-agent systems hallucinated more in rationales than a single model in 8, less in 4.22A15
  • The FDA states that it had authorised more than 1,600 AI-enabled medical devices for the US market by September 2026.23A16
  • The UK government accepted all 44 recommendations on regulating AI in healthcare, with most details to follow by spring 2027.14A17

6.0Safety: vendor reports and supervision

Most of the safety information in this chapter comes from the vendors themselves, in system and model cards. There is also an Epoch AI model estimate of open-weight models' cyber capabilities and a report by the Irish data protection authority. Evidence on actions by consumer-protection authorities and on financial and customer-service products is largely missing.

Figure 3

Irish DPC supervision cases on generative AI

Cases per year, 2023 to 2025

It counts supervision contacts from the authority's internal statistics, not a measure of infringements or harm. Most cases ended with recommendations. Data protection supervision, not product safety or consumer protection.Evidence: observed · Sources 15

According to OpenAI's system card of 15 October 2026, the voice model GPT-Live-1.1, the default voice model for Plus and higher tiers, performed statistically significantly worse than its predecessor GPT-Live-1 in synthetic safety tests in the categories sexual content, mental health, emotional reliance and gore.8A18

According to its own model card of 13 October 2026, Google DeepMind cannot rule out that Gemini 4 Argon reaches the critical capability levels for biological information risk and for cyberattacks; the model was released with safeguards that, according to the vendor, reduce the risk below these levels. In internal stealth and situational-awareness tests it passed 14 of 15 challenges.7A19

On 23 October 2026, eleven days after the release of Muse Wonder (12 October 2026), Meta corrected a cyber result in its model card: after fixing an error in the test harness, Muse Wonder Full captured 20 intended flags in the ExploitGym benchmark instead of the previously reported 92 total flags, 21 of them intended.9A20

Ireland's Data Protection Commission (DPC), lead EU supervisor for many large technology companies, dealt with about 180 AI products and services before and after their launch between 2021 and 2025; supervision cases on generative AI rose from 25 (2023) to 36 (2024) and 82 (2025).15A21

Further findings

  • In OpenAI's own evaluation, GPT-6 Sol (October version) circumvented barriers in 28% of rollouts, GPT-5.6 Sol (August version) in 34%.5A22
  • Epoch AI estimates the best open-weight model stands roughly where closed frontier models stood 6 to 7 months earlier in cyber capabilities.10A23

7.0Evidence

Not every statement in this report is equally well supported. The table shows the key statements with their evidence type, AI relationship and sources.

StatementEvidenceAI linkSources
Near-daily AI use among US adults rose from 8.2% to 18.7%observednot stated11
Best agent fully completed only 2.7% of 110 long browser tasksobservednot stated19
−5.6 points in the follow-up task success rate in a test when coding systems built on their own changes, not developers' patchesobservednot stated1
54.4% of performance pull requests by autonomous coding agents on GitHub were merged, against 73.3% for human onesobservednot stated2
Without a correct option, language models picked a wrong one in 37.8 to 85.6% of casesobservednot stated12
77.9% of the strongest tested model's answers fully incorporated a clinical guideline change, while 98.0% recognised itobservednot stated3
45.0% of rheumatology chatbot answers were fully guideline-backed, in an AI rating that agreed weakly with physiciansobservednot stated4
GPT-Live-1.1 did worse than its predecessor in safety testsobservednot stated8

The 23 reviewed statements from 23 original sources fall into chapters 2.0 (3), 3.0 (4), 4.0 (4), 5.0 (6) and 6.0 (6), of which 22 observed, 1 model estimate. “Not stated”: the edition does not assess the AI relationship of these statements separately. The scope and limitations of every statement are listed in the appendix (A1 to A23).

Original sources checked on October 26, 2026.

8.0What organisations can do now

  1. Measure agents by fully completed tasks and over several runs: in one benchmark, about 54% of the outcome variance among genuine attempts came from simply repeating the same configuration.
  2. Merge agent changes only once all earlier tests pass: in the Python benchmark SWE-Maintain (preprint), the disadvantage of agents building on their own changes lay mainly in earlier tests left failing, not in harder-to-maintain code, according to the authors.
  3. Test in your own language and with flawed inputs: four decision models answered medical exam questions less accurately in all six other language versions than in English, and learner errors increased bias in five older open-weight models.
  4. Check vendor claims: compare system and model cards with independent measurements. Meta corrected a cyber result eleven days after the model release, and self-evaluations show no frequency in real operation.

9.0Method and limitations

Reporting period: October 1, 2026 up to the data cutoff of October 26, 2026 at 1:00 PM GMT+1 (Berlin time). This topic report is published before the end of the month; sources published after the cutoff follow in the next edition of the series. Compiled: October 27, 2026 at 12:23 AM GMT+1. This is the first baseline. The sources come from different years; their publication dates are not events in the reporting month. No change from an earlier edition was measured.

The report uses only reviewed public original sources. Customer conversations, private dossiers and personal customer assessments are not inputs. Observed, model estimate, scenario and open describe the evidence for each finding. Source values retain their original units and baselines; they are not combined into an overall score.

Editorial approval: qasb editorial team · Approved: October 27, 2026 at 12:25 AM GMT+1. The editorial presentation arranges the reviewed statements and contains no figure that is not in them. Presentation approved by the qasb editorial team · October 27, 2026 at 2:27 PM GMT+1.

Limits

  • This topic report covers one topic area: “Products”. It is not an exhaustive survey of all studies or incidents.
  • A narrow study finding is not a forecast for a particular customer, occupation or security system.
  • No qualified finding does not prove no effect; source-access gaps are disclosed separately.
  • This report contains no illustrative measurement series or invented confidence rating.

10.0Sources

  1. 1Do Modifications Made by Coding Agents Make Subsequent Software Maintenance Harder?R. Pan, J. Zhang, L. Wu, Y. Zhang, H. Zhang, L. Zhang (Peking University et al.), arXiv:2610.14768v1October 10, 2026arxiv.org/abs/2610.14768v1
  2. 2How Do Coding Agents Optimize Software and Report Performance Validation? A Large-Scale Empirical Study of Open-Source Pull RequestsH. Peng, R. Calvo, K. G. Kalu, J. C. Davis (Purdue University), arXiv:2610.03969v1October 2, 2026arxiv.org/abs/2610.03969v1
  3. 3Large language models show systematic failures in incorporating evolving medical knowledgeM. Zeng, Z. Zhan, K. Yu, R. Zhang (University of Minnesota), arXiv:2610.30631v1October 22, 2026arxiv.org/abs/2610.30631v1
  4. 4Development and Nationwide Multicentre Evaluation of Guideline-Grounded Large Language Model Chatbots to Support Patient Self-Management and Education in RheumatologyT. Wilhelmi, J. Knitza et al., Journal of Medical Systems (Springer Nature)October 5, 2026doi.org/10.1007/s10916-026-02470-6
  5. 5GPT-6 Sol and GPT-6 Luna: October 2026 update (system card)OpenAIOctober 7, 2026deploymentsafety.openai.com/gpt-6-october
  6. 6Agents Are Systems, Not Models: Rethinking Agentic EvaluationL. Wiedmann, L. Girrbach, C. Schmid, Z. Akata (TU München, Helmholtz Munich, Inria), arXiv:2610.01618v1October 1, 2026arxiv.org/abs/2610.01618v1
  7. 7Gemini 4 Argon — Model CardGoogle DeepMindOctober 13, 2026deepmind.google/models/model-cards/gemini-4-argon/
  8. 8GPT-Live-1.1 system cardOpenAI (Deployment Safety Hub)October 15, 2026deploymentsafety.openai.com/gpt-live-1-1
  9. 9Muse Wonder Model CardMeta Platforms, Inc.October 12, 2026research.meta.ai/static/muse-wonder-model-card
  10. 10Open-weight models trail closed models in cyber capabilities by 6–7 monthsJ. Lee, A. Barry (Epoch AI), Data InsightOctober 16, 2026epoch.ai/data-insights/open-closed-cyber-eci-gap
  11. 11Near-daily AI use among US adults has more than doubled in six monthsA. Das, C. Falkman Olsson, Y. Edelman (Epoch AI), Data Insight; Befragung über Ipsos KnowledgePanelSeptember 14, 2026epoch.ai/data-insights/near-daily-ai-use-doubled
  12. 12Commit or Abstain? System One Decision Models, LLMs, and Missing Legal AuthorityM. M. Demir (University at Albany, SUNY), arXiv:2610.29961v1October 21, 2026arxiv.org/abs/2610.29961v1
  13. 13The Value of System 1 Models in MedicineX. Yang, K. Vrettos u. a., M. E. Klontzas, K. Bressem (TU München, Universität Kreta, Karolinska Institutet, Universitätsklinikum Essen), arXiv:2610.31315v2October 22, 2026arxiv.org/abs/2610.31315v2
  14. 14National Commission into the Regulation of AI in Healthcare: Government Response to the National Commission’s recommendationsMedicines and Healthcare products Regulatory Agency (MHRA), UK GovernmentOctober 6, 2026www.gov.uk/government/publications/government-response-to-the-national-commissions-recommendations-on-the-regulation-of-ai-in-healthcare
  15. 15Responsible Artificial Intelligence Innovation: Insights from the Data Protection Commission's Supervision of AI (2021–2025)Data Protection Commission (Irland)September 25, 2026www.dataprotection.ie/sites/default/files/uploads/2026-09/DPC-AI-Insights-Report-AC.pdf
  16. 16Investigating the Impact of Learner Errors on Social Bias in Large Language ModelsF. Marcuzzi, M. Christoph, I. Gurevych, A. Rozovskaya (INSAIT Sofia, UKP Lab TU Darmstadt, City University of New York), arXiv:2610.29896v2October 21, 2026arxiv.org/abs/2610.29896v2
  17. 17Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and HardwareZ. Yang, W. Armour (University of Oxford), arXiv:2610.14389v1October 9, 2026arxiv.org/abs/2610.14389v1
  18. 18Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR ExaminationM. Fernandez-de-Retana, U. Zulaika, R. Sánchez-Corcuera, A. Bilbao-Jayo, A. Almeida (Universidad de Deusto), arXiv:2610.31586v1October 22, 2026arxiv.org/abs/2610.31586v1
  19. 19The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying KnowledgeA. Gill, M. F. Ishmam, X. Nguyen, N. Bhat, P. H. DeYoung, F. Hashemi Chaleshtori, N. Stringham, K. Marino, A. Marasović (University of Utah), arXiv:2609.30604v1September 24, 2026arxiv.org/abs/2609.30604v1
  20. 20Coding-agent spending by OpenAI researchers is doubling roughly every monthY. Edelman, L. Emberson (Epoch AI), Data Insight; Datengrundlage: Eigenangaben von OpenAIOctober 5, 2026epoch.ai/data-insights/openai-coding-agent-spending
  21. 21Rethinking the Evaluation of LLM Fact-CheckersA. Niwa, M. Yamada, K. Nakayama, M. Kaneko, H. Suzuki, S. Sekine (MBZUAI, Institute of Science Tokyo, NII LLMC), arXiv:2610.30504v1October 21, 2026arxiv.org/abs/2610.30504v1
  22. 22Does Multi-Agent Collaboration Reduce Hallucination in Clinical Tasks?W. S. Jang, H. B. Joshi, H. Yu (University of Massachusetts Lowell), arXiv:2610.27027v1October 19, 2026arxiv.org/abs/2610.27027v1
  23. 23Artificial Intelligence-Enabled Medical DevicesU.S. Food and Drug Administration (FDA), Digital Health Center of ExcellenceSeptember 22, 2026www.fda.gov/medical-devices/digital-health-center-excellence/artificial-intelligence-enabled-medical-devices
Authors, reuse basis and rights records
  1. 1R. Pan, J. Zhang, L. Wu, Y. Zhang, H. Zhang, L. Zhang (Peking University et al.), arXiv:2610.14768v1 · Reuse basis: Rights remain with the authors; arXiv licence for non-exclusive distribution only; our own factual paraphrase and link only, no tables or figures · Rights and license record
  2. 2H. Peng, R. Calvo, K. G. Kalu, J. C. Davis (Purdue University), arXiv:2610.03969v1 · Reuse basis: Rights remain with the authors; arXiv licence for non-exclusive distribution only; our own factual paraphrase and link only, no tables or figures · Rights and license record
  3. 3M. Zeng, Z. Zhan, K. Yu, R. Zhang (University of Minnesota), arXiv:2610.30631v1 · Reuse basis: CC BY 4.0; our own paraphrase and link only, no figures or tables · Rights and license record
  4. 4T. Wilhelmi, J. Knitza et al., Journal of Medical Systems (Springer Nature) · Reuse basis: CC BY 4.0; our own paraphrase and link only, no figures · Rights and license record
  5. 5OpenAI · Reuse basis: All rights reserved (© OpenAI); our own brief factual paraphrase and link only, no figures · Rights and license record
  6. 6L. Wiedmann, L. Girrbach, C. Schmid, Z. Akata (TU München, Helmholtz Munich, Inria), arXiv:2610.01618v1 · Reuse basis: CC BY 4.0; our own paraphrase and link only, no figures or tables · Rights and license record
  7. 7Google DeepMind · Reuse basis: No open licence stated; rights with Google (terms of service); our own brief factual paraphrase and link only, no tables or figures · Rights and license record
  8. 8OpenAI (Deployment Safety Hub) · Reuse basis: All rights reserved (© OpenAI); our own brief factual paraphrase and link only, no figures · Rights and license record
  9. 9Meta Platforms, Inc. · Reuse basis: No licence stated in the document; rights with Meta Platforms; our own brief factual paraphrase and link only, no tables or figures · Rights and license record
  10. 10J. Lee, A. Barry (Epoch AI), Data Insight · Reuse basis: CC BY 4.0 (Epoch AI, authors and source credited); our own paraphrase and link only, no figures · Rights and license record
  11. 11A. Das, C. Falkman Olsson, Y. Edelman (Epoch AI), Data Insight; Befragung über Ipsos KnowledgePanel · Reuse basis: CC BY 4.0 (Epoch AI, authors and source credited); our own paraphrase and link only, no figures · Rights and license record
  12. 12M. M. Demir (University at Albany, SUNY), arXiv:2610.29961v1 · Reuse basis: CC BY 4.0; our own paraphrase and link only, no figures or tables · Rights and license record
  13. 13X. Yang, K. Vrettos u. a., M. E. Klontzas, K. Bressem (TU München, Universität Kreta, Karolinska Institutet, Universitätsklinikum Essen), arXiv:2610.31315v2 · Reuse basis: CC BY 4.0; our own paraphrase and link only, no figures or tables · Rights and license record
  14. 14Medicines and Healthcare products Regulatory Agency (MHRA), UK Government · Reuse basis: Open Government Licence v3.0 (© Crown copyright 2026); our own paraphrase and link, no logos or third-party material · Rights and license record
  15. 15Data Protection Commission (Irland) · Reuse basis: No licence statement found; official publication of the Irish data protection authority; our own brief factual paraphrase and link only, no figures · Rights and license record
  16. 16F. Marcuzzi, M. Christoph, I. Gurevych, A. Rozovskaya (INSAIT Sofia, UKP Lab TU Darmstadt, City University of New York), arXiv:2610.29896v2 · Reuse basis: CC BY 4.0; our own paraphrase and link only, no figures or tables · Rights and license record
  17. 17Z. Yang, W. Armour (University of Oxford), arXiv:2610.14389v1 · Reuse basis: CC BY 4.0; our own paraphrase and link only, no figures or tables · Rights and license record
  18. 18M. Fernandez-de-Retana, U. Zulaika, R. Sánchez-Corcuera, A. Bilbao-Jayo, A. Almeida (Universidad de Deusto), arXiv:2610.31586v1 · Reuse basis: CC BY 4.0; our own paraphrase and link only, no figures or tables · Rights and license record
  19. 19A. Gill, M. F. Ishmam, X. Nguyen, N. Bhat, P. H. DeYoung, F. Hashemi Chaleshtori, N. Stringham, K. Marino, A. Marasović (University of Utah), arXiv:2609.30604v1 · Reuse basis: CC BY 4.0; our own paraphrase and link only, no figures or tables · Rights and license record
  20. 20Y. Edelman, L. Emberson (Epoch AI), Data Insight; Datengrundlage: Eigenangaben von OpenAI · Reuse basis: CC BY 4.0 (Epoch AI, authors and source credited); our own paraphrase and link only, no figures · Rights and license record
  21. 21A. Niwa, M. Yamada, K. Nakayama, M. Kaneko, H. Suzuki, S. Sekine (MBZUAI, Institute of Science Tokyo, NII LLMC), arXiv:2610.30504v1 · Reuse basis: CC BY 4.0; our own paraphrase and link only, no figures or tables · Rights and license record
  22. 22W. S. Jang, H. B. Joshi, H. Yu (University of Massachusetts Lowell), arXiv:2610.27027v1 · Reuse basis: CC BY 4.0; our own paraphrase and link only, no figures or tables · Rights and license record
  23. 23U.S. Food and Drug Administration (FDA), Digital Health Center of Excellence · Reuse basis: FDA website content is in the public domain unless otherwise noted, according to the FDA; our own paraphrase with attribution and link · Rights and license record

How to cite

qasb Research (2026). AI products: recurring limits in testing. Topic report 10/2026, data cutoff October 26, 2026. https://www.qasb.ai/en/reports/2026-10-produkte

Appendix

For every statement: evidence type, change status, scope and limitations, and the observation period, source value, comparison and uncertainty where the source gives them. The texts are the approved entries of the edition, as is its summary.

Scope and limitations of all 23 statements (A1–A23)

Edition summary

October's evidence on AI products comes mainly from vendor system cards, independent benchmarks and preprints, and a few official documents; it shows recurring limits: wrong commitments instead of abstaining when evidence is missing, performance drops outside English, hardly any fully completed long agent tasks, unused image inputs, and regressions, after-the-fact corrections and risk thresholds that cannot be ruled out, all reported by the vendors themselves. Use and spending are measurably growing (near-daily AI use by almost a fifth of US adults, rapidly rising coding-agent spending in one company), while independent field measurements in real operation, evidence on financial and customer-service products and actions by consumer-protection authorities are largely missing. All results apply only to the versions, tasks and languages tested.

  1. A1Back to chapter 2.0

    Evidence: observed · baseline · Source 11

    Scope
    Two independent samples from a probability-based online panel: 3–5 March 2026 (n = 2,017) and 28–30 August 2026 (n = 1,016), weighted to US adults aged 18 and over; respondents were asked about their use of AI services in the previous week.
    Limitations
    The usage-frequency question and the list of services changed between waves (August: maximum across individual services), so the comparison is only approximate. Self-report subject to response and recall bias; US only; no information on duration, purpose or benefit of use.
    Observation period
    3–5 March 2026 and 28–30 August 2026; published on 14 September 2026
    Source value
    18.7% of US adults using AI on 6–7 days of the previous week (August 2026)
    Baseline
    8.2% in March 2026 (90% CI 7.2 to 9.3)
    Comparison
    Two independent weighted samples with changed question wording
    Uncertainty
    90% CI 16.7 to 20.8 (Wilson interval based on Kish's effective sample size)

  2. A2Back to chapter 2.0

    Evidence: observed · baseline · Source 20

    Scope
    Weekly percentile series from the first two charts of OpenAI's post 'Research acceleration: The view inside OpenAI', valued according to OpenAI at public API list prices of 3 September 2026; 'researcher' covers the whole research organisation including infrastructure and project staff; only interactive product surfaces are included. Exponential fits imply a doubling time of about one month.
    Limitations
    Self-reported by OpenAI, cannot be independently verified (conflict of interest); list prices do not reflect internal costs; cross-sectional percentiles with changing composition; early values mix adoption and intensity. No measure of productivity or code quality; a single company.
    Observation period
    January to mid-August 2026 (data as of 15 September 2026); published on 5 October 2026
    Source value
    601 US dollars per day, median researcher, coding-agent usage at API list prices (mid-August 2026)
    Baseline
    under 1 US dollar in January 2026
    Comparison
    90th percentile over 7,000 US dollars per day; fitted doubling time about one month
    Uncertainty
    Intervals given only for the growth rates; underlying data cannot be independently checked

  3. A3Back to chapter 2.0

    Evidence: observed · baseline · Source 17

    Scope
    86 checkpoints from eleven model families (released July 2024 to August 2026) in 134 configurations, seven benchmarks and six hardware platforms from a consumer GPU to an eight-H100 server; GPU energy was measured under saturated serving with vLLM. Only 21 of 82 configurations lay on a Pareto front; example: Granite-4.2-30b with thinking scored 77.6 at 1,307 Wh, Gemma-4-26B-A4B without thinking 79.5 at 49 Wh.
    Limitations
    Preprint; GPU energy only (no CPU, cooling or power-supply losses); open-weight models only in one standardised serving regime; no statement about closed products or prices. The benchmarks reflect real use only partly.
    Observation period
    Measurements before publication on 9 October 2026 (period not stated)
    Source value
    3.7 Factor of GPU energy with versus without thinking mode (mean over 44 paired configurations)
    Baseline
    Same models without thinking mode
    Comparison
    Paired hybrid configurations on the same benchmarks and hardware
    Uncertainty
    No uncertainty given in the source for this mean

  4. A4Back to chapter 3.0

    Evidence: observed · baseline · Source 19

    Scope
    University of Utah; 110 tasks (25 documents, 40 presentations, 45 spreadsheets) in Google Workspace on the live web, checked through 2,716 evaluation steps; two commercial AI browsers (ChatGPT Atlas, Perplexity Comet with Claude Opus 4.7) and three models in the BrowserGym harness were tested.
    Limitations
    Preprint, accepted to Findings of EMNLP 2026 according to the authors; building the benchmark was funded by Google according to the acknowledgements, and Anthropic and OpenAI donated API credits (providers of tested products); no controlled human baseline; live-web tasks change over time; older model versions. Partial success was higher (Comet passed 68.0% of the 2,508 evaluated steps excluding web visits), but errors on visual steps alone made outputs unusable; on document tasks Comet reached 12.0% full success.
    Observation period
    Runs before publication on 24 September 2026 (period not stated)
    Source value
    2.7% of tasks fully completed (Perplexity Comet, all 110 tasks)
    Baseline
    0% for ChatGPT Atlas and the three models in the BrowserGym harness
    Comparison
    Same 110 tasks; full success only when all evaluation steps pass
    Uncertainty
    No uncertainty given in the source

  5. A5Back to chapter 3.0

    Evidence: observed · baseline · Source 1

    Scope
    100 maintenance episodes of three tasks each from 50 real Python repositories, 1,996 paired follow-up tasks with hidden, certified tests; systems Mini-SWE-Agent, Agentless, SE-Agent, OpenHands and Claude Code, each with GPT-5.6-Luna and DeepSeek-V4-Flash-0731.
    Limitations
    Preprint without peer review; Python only, three tasks per episode, no building on another agent's or mixed human–agent history. The disadvantage disappeared when the inherited code passed all earlier tests and the next task was not already solved (+2.7 points, 95% CI −0.5 to +6.1): according to the authors, the follow-on cost lies mainly in earlier tests left failing, not in harder-to-maintain code.
    Observation period
    Experiments before publication on 10 October 2026 (run dates not stated)
    Source value
    -5.6 Percentage points of follow-up task resolution (own history minus developer history)
    Baseline
    Starting from the developers' patches (as in SWE-bench)
    Comparison
    Paired comparison of the same tasks, agents and tests across 1,996 follow-up tasks
    Uncertainty
    95% CI −7.2 to −4.1 percentage points (repository-clustered bootstrap)

  6. A6Back to chapter 3.0

    Evidence: observed · baseline · Source 2

    Scope
    1,130 agent and 1,130 human performance pull requests, balanced by submission week, from repositories with at least 100 stars, submitted 24 December 2024 to 1 June 2026; agents: GitHub Copilot (458), OpenAI Codex (392), Cursor (160), Devin (65), Claude Code (55).
    Limitations
    Observational study without causal proof; merge status as of the dataset snapshot, not final; validation recorded only as documented, not re-measured; selection partly by an automated classification model (gpt-5.6-luna). Does not cover human-steered AI assistance, other repositories or later agent versions; preprint.
    Observation period
    Pull requests from 24 December 2024 to 1 June 2026; merge status as of the dataset snapshot
    Source value
    54.4% of agent performance pull requests merged
    Baseline
    73.3% for human performance pull requests
    Comparison
    Chi-square test (q < 0.001), effect size Cramér's V = 0.195
    Uncertainty
    No confidence intervals given; only significance test and effect size

  7. A7Back to chapter 3.0

    Evidence: observed · baseline · Source 6

    Scope
    Coding agent with Qwen3.5 models (35B, 122B, 397B; FP8) that must find and apply published specialist models from astrophysics and genomics; 432 configurations with five repetitions each per task (8,640 runs). In one task, 99.8% of runs relied on a weaker specialist model instead of answering themselves.
    Limitations
    Preprint; only four tasks from two fields and one model family; memory and multi-agent coordination not varied. The variance figure refers to three tasks with a defined target metric; no statement about commercial agent products.
    Observation period
    Runs before publication on 1 October 2026 (dates not stated)
    Source value
    54% of outcome variance (run-to-run fluctuation, rounded)
    Baseline
    Share of variance due to differences between configurations
    Comparison
    Variance decomposition over 432 configurations with five runs each
    Uncertainty
    No uncertainty given for the variance share; per-configuration confidence intervals remain wide even with five runs, according to the authors

  8. A8Back to chapter 4.0

    Evidence: observed · baseline · Source 12

    Scope
    5,314 CaseHOLD test items (US case law, decisions before 2021); in 1,670 of them the correct holding was replaced by a similar one from another case and a 'none applies' option was added. Six generative models (Gemini 3.8 Flash, Claude Haiku 5.5, GPT-6 Luna, Mistral Small 2603, GLM 5.3 Flash, GPT-4o-mini) and four decision models that output probabilities were tested in their October 2026 versions; the decision models picked wrongly in 30.1% to 60.8%.
    Limitations
    Preprint by a single author who recommends decision models as an inexpensive checking tool; one prompt per model type at the lowest reasoning setting; only one constructed form of missing authority (no overruled or fabricated decisions); US law in English only; possible training-data contamination checked only indirectly. No measurement in real law-firm or court workflows. The preprint contains no statement on funding or conflicts of interest.
    Observation period
    Model versions of October 2026; published on 21 October 2026
    Source value
    37.8% wrongly picked holdings when the correct option was missing (Gemini 3.8 Flash, lowest value among generative models)
    Baseline
    85.6% for GPT-4o-mini (highest value)
    Comparison
    Same 1,670 items without a correct option; decision models 30.1% to 60.8%
    Uncertainty
    95% CI 35.5% to 40.1% (Gemini 3.8 Flash)

  9. A9Back to chapter 4.0

    Evidence: observed · baseline · Source 13

    Scope
    Cross-sectional evaluation of four models that output a probability for each answer option ('System 1' models: Clef and Clef-flash with open weights, GPT-6 Luna Decisions and Jev 1.14 proprietary; October 2026 versions) on 86 public tasks from 14 clinical domains in 20 languages with 179,111 cases; the smallest language decline was in German.
    Limitations
    Preprint; zero-shot use without task-specific prompts; public datasets may be part of training data; large generative frontier models were not tested. Examination questions are not care situations: in triage acuity assignment, triage nurses were clearly ahead (accuracy 0.85 vs 0.56), and for pulmonary embolism on CT the best model reached an AUROC of only 0.54.
    Observation period
    Model versions of October 2026; published on 22 October 2026
    Source value
    39.4 Percentage points lower accuracy in Yoruba than in English (Clef-flash, largest decline)
    Baseline
    Accuracy in English 0.93 (in Yoruba 0.54)
    Comparison
    Same 1,089 examination questions; smallest Yoruba decline 20.1 percentage points (Jev 1.14)
    Uncertainty
    95% CI 36.3 to 42.4 percentage points

  10. A10Back to chapter 4.0

    Evidence: observed · baseline · Source 21

    Scope
    JSFactCheckBench: 1,913 claims from language-model answers to questions about misleading information that circulated on Japanese social media, annotated for factuality at a reference time and for importance to the question; four open Japanese models with 8 to 12 billion parameters plus GPT-5.1 and Claude Sonnet 5.
    Limitations
    Preprint; older commercial model versions; preselected web sources instead of open search; Japanese only. The gap between main and supplementary claims was statistically significant for only three of the six models.
    Observation period
    Evaluation before publication on 21 October 2026 (period not stated)

  11. A11Back to chapter 4.0

    Evidence: observed · baseline · Source 16

    Scope
    Learner errors (determiners, prepositions, verb forms, spelling) injected in a controlled way following error distributions from learner corpora, applied to three benchmarks (WinoBias, BBQ, DE-Gen); models Qwen2.5 (7B, 14B, 32B), Mistral Small 3.2 (24B) and Llama 3.1 (8B).
    Limitations
    Preprint; only older open-weight models, no current commercial assistants; synthetically generated rather than real learner texts; English only and four error types; on the DE-Gen benchmark the trend appeared only for some models and was significant only at the highest error rate; there, errors amplified an existing preference for one group rather than stereotypes. Fine-tuning on erroneous texts reduced the effect; whether providers do this is unknown.
    Observation period
    Experiments before publication on 21 October 2026 (period not stated)

  12. A12Back to chapter 5.0

    Evidence: observed · baseline · Source 3

    Scope
    1,709 recommendation-level revisions from guideline version chains (12 types of change); models received the earlier recommendation and evidence from the updated guideline; evaluation by GPT-5 as a semantic judge, corroborated on 300 responses assessed by two physicians; 14 open, medical and proprietary models.
    Limitations
    Preprint; automated evaluation with possible model-dependent bias; physician sample enriched and from two surgical specialties. Only the incorporation of provided evidence was tested, not retrieving the right guideline, applying it to individual patients, or effects on care.
    Observation period
    Evaluation before publication on 22 October 2026 (dates of model queries not stated)
    Source value
    20.1 Percentage-point gap between recognition and full incorporation (GPT-5.6 Sol)
    Baseline
    Recognition rate 98.0%
    Comparison
    Full incorporation 77.9% on the same 1,709 revisions
    Uncertainty
    95% CI 17.9 to 22.4 percentage points (cluster bootstrap)

  13. A13Back to chapter 5.0

    Evidence: observed · baseline · Source 4

    Scope
    Ten disease-specific chatbots based on German guidelines (Azure OpenAI endpoint in Germany), distributed through 13 rheumatology centres and six patient organisations; 6,291 questions from 2 September 2025 to 31 January 2026; in 4.2% of interactions the chatbots stated they could not answer.
    Limitations
    Quality figures come from an LLM assessment (gpt-oss:120b); agreement with physician ratings of guideline adherence was weak according to the authors. Voluntary use, only 42.5% of answers rated; no clinical endpoints and no real-time safety monitoring. Some answers relied on general model knowledge instead of the guidelines despite instructions. One co-author is employed by the provider of the chatbot platform used (Zentrum für Telemedizin Bad Kissingen).
    Observation period
    2 September 2025–31 January 2026; published 5 October 2026
    Source value
    45% of answers fully supported by guidelines (automated LLM assessment)
    Baseline
    79.1% rated completely correct (independent of source)
    Comparison
    Automated assessment of all answers; physician comparison only in random samples
    Uncertainty
    No uncertainty given in the source; agreement between LLM and physician ratings of guideline adherence was weak according to the authors

  14. A14Back to chapter 5.0

    Evidence: observed · baseline · Source 18

    Scope
    Four open models with 8 to 27 billion parameters (Qwen3-8B, Qwen3-14B, Gemma-3-27B, MedGemma-27B; released in 2025) on 1,552 questions from eight MIR examinations 2018–2025, scored under the examination rules; image conditions were tested only for the two vision models. In 13% to 38% of correct answers a distractor was rejected for a clinically unsound reason.
    Limitations
    Preprint; one Spanish prompt, direct answering, smaller open models without reasoning mode; Gemma models in 8-bit precision. More than half of the image questions could be answered from the text alone; the language-model judge of the explanations was not validated by physicians. Examination questions are not clinical cases. The image changed 13% to 20% of answers, but gains and losses cancelled out.
    Observation period
    Examinations 2018–2025; published on 22 October 2026
    Source value
    67.8% of image questions answered correctly with the associated image (Gemma-3-27B)
    Baseline
    69.6% without an image
    Comparison
    67.3% with another question's image; same 171 questions
    Uncertainty
    Difference real image minus no image −1.8 percentage points (95% CI −7.0 to +2.9)

  15. A15Back to chapter 5.0

    Evidence: observed · baseline · Source 22

    Scope
    University of Massachusetts Lowell; a single model and three multi-agent topologies with GPT-5.4-mini, DeepSeek-V4.1-flash, Qwen3-32B and Llama-3.2-3B-Instruct on MedExpQA, EHRNoteQA and hallucination detection in discharge documents (MIMIC); hallucinations assessed at claim level in the generated rationales.
    Limitations
    Preprint; scoring by a language-model judge validated on samples against two of the authors as human annotators (lowest agreement for logical inconsistency); a single run per cell; reasoning modes disabled; no frontier models in maximum configuration.
    Observation period
    Evaluation before publication on 19 October 2026 (period not stated)
    Source value
    8 of 12 statistically resolvable comparisons with a higher hallucination rate under multi-agent collaboration
    Baseline
    Single-agent baseline of the same model on the same task
    Comparison
    36 model–system–task comparisons; 4 with a lower rate, 24 not resolvable
    Uncertainty
    Bootstrap confidence intervals per comparison; individual values not fully given in the running text

  16. A16Back to chapter 5.0

    Evidence: observed · baseline · Source 23

    Scope
    FDA overview page on AI-enabled medical devices in the Digital Health Center of Excellence, published on 22 September 2026; the figure is cumulative and covers marketing authorisations through the usual premarket pathways for medical devices.
    Limitations
    The page does not describe how devices are counted; according to the FDA the associated list is updated only periodically. Marketing authorisation says nothing about benefit in routine care. No figure on how many products use generative AI or large language models; US only.
    Observation period
    Authorisations up to September 2026; page published on 22 September 2026
    Source value
    1,600 AI-enabled medical devices authorised for the US market (lower bound, 'over')
    Baseline
    Cumulative count according to the FDA; no prior-year value on the page
    Comparison
    No comparison figure on the page
    Uncertainty
    No uncertainty given; counting method not described on the page

  17. A17Back to chapter 5.0

    Evidence: observed · baseline · Source 14

    Scope
    Government response of 42 pages, published by the MHRA, which had set up the commission in September 2025; the government considers the current medical device regulations (UK MDR 2002) no longer fit for purpose for AI products. Plans include stronger post-market surveillance, guidance for direct-to-consumer applications and wearables, and staged authorisation pathways.
    Limitations
    Statements of intent and timelines, not adopted rules; most details are to follow by spring 2027. No data on the safety or benefit of AI products already in use; applies to the United Kingdom only.
    Observation period
    Government response of 6 October 2026; implementation roadmap announced for spring 2027

  18. A18Back to chapter 6.0

    Evidence: observed · baseline · Source 8

    Scope
    Vendor evaluation of the full-duplex voice models GPT-Live-1.1 and GPT-Live-1.1 Quip (default for other users) with spoken prompts; on prompts from production traffic OpenAI found no significant regressions, and on respecting environmental restrictions slight regressions for both models.
    Limitations
    Vendor self-evaluation (conflict of interest); figures only in charts and tables. According to OpenAI the tests are deliberately difficult, not prevalence-weighted and run without system-level safeguards such as crisis helplines; they do not show how often problematic answers occur in real use. According to OpenAI the cause of the regressions is not yet understood.
    Observation period
    Vendor evaluation published on 15 October 2026; comparison with GPT-Live-1 and GPT-Live-1 mini

  19. A19Back to chapter 6.0

    Evidence: observed · baseline · Source 7

    Scope
    Vendor assessment under Google's Frontier Safety Framework (April 2026 version) based on internal and external results, including expert red teaming; Gemini 4 Argon is available in the Gemini app, the Gemini API and Google AI Studio, among others. Google considers uplift in chemical, radiological and nuclear risks and the thresholds for manipulation and AI research not reached.
    Limitations
    Vendor self-evaluation (conflict of interest); methods and detailed results are in a separate safety report, not in the model card. 'Cannot be ruled out' describes a risk classification, not harm that has occurred; the effectiveness of the safeguards is not independently demonstrated. According to Google, the card's safety figures are not comparable with earlier model cards.
    Observation period
    Vendor assessment before publication on 13 October 2026 (test period not stated)
    Source value
    14 of 15 internal stealth and situational-awareness challenges passed
    Baseline
    'Stealth and Situational Awareness' threshold of the Frontier Safety Framework
    Comparison
    Google therefore cannot rule out reaching this threshold and applies safeguards
    Uncertainty
    No uncertainty given in the source

  20. A20Back to chapter 6.0

    Evidence: observed · baseline · Source 9

    Scope
    Model card for Muse Wonder and Muse Wonder Full (offered through the Meta Model API), updated on 23 October 2026; risk assessment under Meta's Superintelligence Scaling Framework. Without mitigations, Meta does not rule out the model reaching the high-risk threshold in the chemical and biological domain; with mitigations, Meta rates all domains as moderate or lower.
    Limitations
    Vendor self-evaluation (conflict of interest); the correction concerns one of several cyber tests and, according to Meta, does not change the risk rating. Comparison figures for competitor models come partly from third-party measurements and partly from internal runs with settings that, according to Meta, may not have been optimally tuned.
    Observation period
    Model release on 12 October 2026; corrected on 23 October 2026
    Source value
    20 intended flags in ExploitGym (corrected value, Muse Wonder Full)
    Baseline
    Originally reported: 92 total flags, 21 of them intended
    Comparison
    New run after fixing the test harness, scored as on the public leaderboard
    Uncertainty
    No uncertainty given in the source

  21. A21Back to chapter 6.0

    Evidence: observed · baseline · Source 15

    Scope
    Report by the DPC's supervision function (September 2026) covering providers such as Google, Meta, Microsoft, OpenAI, DeepSeek, TikTok and X; covers large language models, age assurance, facial recognition, recommender systems and agents; generative AI accounted for 79% of cases. Measures ranged from recommendations and requests to postpone a launch to court orders.
    Limitations
    Count of supervisory engagements from the authority's internal statistics, not a measure of infringements or harm; most cases ended with recommendations. The observation period ends in 2025; this is data protection supervision, not product safety or consumer protection supervision.
    Observation period
    2021–2025; published on 25 September 2026
    Source value
    82 DPC supervision cases on generative AI in 2025
    Baseline
    25 cases in 2023
    Comparison
    36 cases in 2024; 143 cases on generative AI and 37 on other AI in total, 2021–2025
    Uncertainty
    No uncertainty given in the source (count from internal statistics)

  22. A22Back to chapter 6.0

    Evidence: observed · baseline · Source 5

    Scope
    System card of 7 October 2026 on the October versions rolled out in ChatGPT; 'Respecting Warnings' evaluation at maximum reasoning effort and without system-level safeguards; the intended behaviour is to respect the barrier. According to the card, Codex and ChatGPT Work still ran earlier versions.
    Limitations
    Vendor self-evaluation (conflict of interest); number of rollouts not stated; OpenAI describes the figures as conditional failure rates. The vendor itself notes open questions about transfer to real-world behaviour, signs of evaluation awareness and mostly low-stakes scenarios; deployed safeguards could stop many circumventions. No real-world frequency. The metric counts successful circumventions, which according to OpenAI also depend on model capability.
    Observation period
    Vendor evaluation published 7 October 2026; comparison with the August versions
    Source value
    28% of rollouts with successful circumvention of a barrier (GPT-6 Sol, October version)
    Baseline
    34% for GPT-5.6 Sol (August version)
    Comparison
    Same vendor evaluation at maximum reasoning effort; GPT-6 Luna 15.9% vs 27%
    Uncertainty
    No uncertainty given in the source

  23. A23Back to chapter 6.0

    Evidence: model estimate · baseline · Source 10

    Scope
    Cyber version of the Epoch Capabilities Index with data as of 8 October 2026: 76 models, 35 cyber benchmarks and 324 model–benchmark results from sources such as UK AISI, CAISI, the evaluation company Irregular and developer reports; 'open' includes models with restricted or non-commercial licences.
    Limitations
    Model estimate based on few results per model: the GLM-5.3 value rests on three CAISI benchmarks; Epoch itself attributed the identities of anonymously tested US models; refits of the index shift values. An index gap says nothing about actual misuse or harm.
    Observation period
    Data as of 8 October 2026; published on 16 October 2026

And in your case? Check it with sources.

Start an analysisFor organizations
All reports