Book Section
Magdalena Krysztoforska
The Measure of Generality
On the Scalar Dynamics of AI Model Evaluation Practices
The notion of ‘generality’ has featured prominently in recent discourse around AI, where the technology is increasingly presented as general-purpose, or as showcasing general intelligence. In order to interrogate the purported scientific basis for such claims and their rhetorical currency, this essay examines practices of model evaluation and benchmarking in AI through the critical lens of scale, highlighting numerous scalar conflations and distortions prevalent in AI research.
Keywords: artificial intelligence; machine learning; model evaluation; benchmark datasets; general-purpose technology; scaling laws; diagrams
| Title |
The Measure of Generality
|
| Subtitle |
On the Scalar Dynamics of AI Model Evaluation Practices
|
| Author(s) |
Magdalena Krysztoforska
|
| Identifier | |
| Description |
The notion of ‘generality’ has featured prominently in recent discourse around AI, where the technology is increasingly presented as general-purpose, or as showcasing general intelligence. In order to interrogate the purported scientific basis for such claims and their rhetorical currency, this essay examines practices of model evaluation and benchmarking in AI through the critical lens of scale, highlighting numerous scalar conflations and distortions prevalent in AI research.
|
| Is Part Of | |
| Place |
Berlin
|
| Publisher |
ICI Berlin Press
|
| Date |
November 24, 2026
|
| Subject |
artificial intelligence
machine learning
model evaluation
benchmark datasets
general-purpose technology
scaling laws
diagrams
|
| Rights |
© by the author(s)
Except for images or otherwise noted, this publication is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
|
| Language |
en-GB
|
| page start |
297
|
| page end |
330
|
| Source |
Scale: New Critical Essays, ed. by Christoph F. E. Holzhey and Marietta Kesting, Cultural Inquiry, 40 (ICI Berlin Press, 2026), pp. 297–330
|
Publication date: 24 November 2026
References
- ‘About’, OpenAI, 30 September 2025 <https://openai.com/about/> [accessed 25 January 2026]
- ‘About Google DeepMind’, Google DeepMind, January 2026 <https://deepmind.google/about/> [accessed 25 January 2026]
- ‘About Wordnet’, Wordnet, 2010 <https://wordnet.princeton.edu/citing-wordnet> [accessed 30 May 2026]
- ‘America’s AI Action Plan’, Executive Office of the President of the United States, July 2025 <https://www.whitehouse.gov/wp-content/uploads/2025/07/Americas-AI-Action-Plan.pdf> [accessed 25 January 2025]
- ‘General-Purpose AI Models in the AI Act — Questions & Answers’, European Commission, 9 September 2025 <https://digital-strategy.ec.europa.eu/en/faqs/general-purpose-ai-models-ai-act-questions-answers> [accessed 25 January 2026]
- ‘National AI Strategy’, HM Government, September 2021 <https://assets.publishing.service.gov.uk/media/614db4d1e90e077a2cbdf3c4/National_AI_Strategy_-_PDF_version.pdf> [accessed 10 July 2026]
- Agüera y Arcas, Blaise, and Peter Norvig, ‘Artificial General Intelligence Is Already Here’, Noema, 10 October 2023 <https://www.noemamag.com/artificial-general-intelligence-is-already-here/> [accessed 17 May 2026]
- Anderson, Philip W., ‘More Is Different: Broken Symmetry and the Nature of the Hierarchical Structure of Science’, Science, 177.4047 (1972), pp. 393–96 <https://doi.org/10.1126/science.177.4047.393>
- Anwar, Usman, and others, ‘Foundational Challenges in Assuring Alignment and Safety of Large Language Models’, arXiv (preprint), 2404.09932, 6 September 2024 <https://doi.org/10.48550/arXiv.2404.09932>
- Blili-Hamelin, Borhane, and others, ‘Stop Treating ‘AGI’ as the North-Star Goal of AI Research’, arXiv (preprint), 2502.03689, 7 July 2025 <https://doi.org/10.48550/arXiv.2502.03689>
- Bommasani, Rishi, and others, ‘On the Opportunities and Risks of Foundation Models’, arXiv, 12 July 2022 <https://doi.org/10.48550/arXiv.2108.07258>
- Bubeck, Sébastien, and others, ‘Sparks of Artificial General Intelligence: Early Experiments with GPT-4’, arXiv (preprint), 2303.12712, 6 September 2023 <https://doi.org/10.48550/arXiv.2303.12712>
- Burnell, Ryan, and others, ‘Rethink Reporting of Evaluation Results in AI’, Science, 380.6641 (2023), pp. 136–38 <https://doi.org/10.1126/science.adf6369>
- Center for AI Safety, Scale AI, and HLE Contributors Consortium, ‘A Benchmark of Expert-Level Academic Questions to Assess AI Capabilities’, Nature, 649 (2026), pp. 1139–46 <https://doi.org/10.1038/s41586-025-09962-4>
- Chang, Yupeng, and others, ‘A Survey on Evaluation of Large Language Models’, ACM Transactions on Intelligent Systems and Technology, 15.3 (2024), pp. 1–45 <https://doi.org/10.1145/3641289>
- Chatterji, Aaron, and others, ‘How People Use ChatGPT’, Working Paper 34255, National Bureau of Economic Research, September 2025 <https://doi.org/10.3386/w34255>
- Chung, Tsz Ting, and others, ‘DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models’, Findings of the Association for Computational Linguistics: EMNLP 2025 (Suzhou, China), November 2025, pp. 901–15 <https://doi.org/10.18653/v1/2025.findings-emnlp.47>
- Church, Kenneth Ward, and Valia Kordoni, ‘Emerging Trends: SOTA-Chasing’, Natural Language Engineering, 28.2 (2022), pp. 249–69 <https://doi.org/10.1017/S1351324922000043>
- Crawford, Kate, and Trevor Paglen, ‘Excavating AI: The Politics of Images in Machine Learning Training Sets’, AI & Society, 36 (2021), pp. 1105–16 <https://doi.org/10.1007/s00146-021-01162-8>
- Davidson, Tom, and others, ‘AI Capabilities Can Be Significantly Improved without Expensive Retraining’, arXiv (preprint), 2312.07413, 12 December 2023 <https://doi.org/10.48550/arXiv.2312.07413>
- Deng, Jia, and others, ‘ImageNet: A Large-Scale Hierarchical Image Database’, 2009 IEEE Conference on Computer Vision and Pattern Recognition, June 2009, pp. 248–55 <https://doi.org/10.1109/CVPR.2009.5206848>
- Denton, Emily, and others, ‘On the Genealogy of Machine Learning Datasets: A Critical History of ImageNet’, Big Data & Society, 8.2 (2021) <https://doi.org/10.1177/20539517211035955>
- DiCaglio, Joshua, Scale Theory: A Nondisciplinary Inquiry (University of Minnesota Press, 2021) <https://doi.org/10.5749/j.ctv22fqc19>
- Douglas Heaven, Will, ‘Rogue Superintelligence and Merging with Machines: Inside the Mind of OpenAI’s Chief Scientist’, MIT Technology Review, 26 October 2023 <https://www.technologyreview.com/2023/10/26/1082398/exclusive-ilya-sutskever-openais-chief-scientist-on-his-hopes-and-fears-for-the-future-of-ai/> [accessed 25 January 2026]
- Eloundou, Tyna, and others, ‘GPTs Are GPTs: Labor Market Impact Potential of LLMs’, Science, 384.6702 (2024), pp. 1306–08 <https://doi.org/10.1126/science.adj0998>
- European Parliament and Council of the European Union, Artificial Intelligence Act, no. 2024/1689, EUR-Lex, 12 July 2024 <http://data.europa.eu/eli/reg/2024/1689/oj> [accessed 25 January 2026]
- Fang, Alex, Simon Kornblith, and Ludwig Schmidt, ‘Does Progress on ImageNet Transfer to Real-World Datasets?’, Thirty-Seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS 2023), 26 September 2023 <https://openreview.net/forum?id=SS3CK3yx5Z> [accessed 31 May 2026]
- Fei-Fei, Li, ‘ImageNet: Crowdsourcing, Benchmarking & Other Cool Things’, Robotics Institute VASC Seminar, Carnegie Mellon University, March 2010 <https://image-net.org/static_files/papers/ImageNet_2010.pdf> [accessed 29 May 2026]
- Gray, Mary L., and Siddharth Suri, Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass (Houghton Mifflin Harcourt, 2019)
- Gubrud, Mark, ‘Nanotechnology and International Security’, Fifth Foresight Conference on Molecular Nanotechnology, 5–8 November 1997 <https://legacy.foresight.org/Conferences/MNT05/Papers/Gubrud/index.html> [accessed 10 July 2026]
- Hardt, Moritz, and Benjamin Recht, Patterns, Predictions, and Actions: Foundations of Machine Learning (Princeton University Press, 2022)
- Hendrycks, Dan, and others, ‘Measuring Mathematical Problem Solving with the MATH Dataset’, Proceedings of the Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks, 18 October 2021 <https://openreview.net/forum?id=7Bywt2mQsCe> [accessed 25 January 2026]
- Herod, Andrew, Scale (Routledge, 2010) <https://doi.org/10.4324/9780203641095>
- Horton, Zachary, The Cosmic Zoom: Scale, Knowledge, and Mediation (University of Chicago Press, 2021) <https://doi.org/10.7208/chicago/9780226742588.001.0001>
- Irani, Lilly, ‘The Cultural Work of Microwork’, New Media & Society, 17.5 (2015), pp. 720–39 <https://doi.org/10.1177/1461444813511926>
- Javaheripi, Mojan, and Sébastien Bubeck, ‘Phi-2: The Surprising Power of Small Language Models’, Microsoft Research Blog, 12 December 2023 <https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/> [accessed 25 January 2026]
- Jones, Katherine T., ‘Scale as Epistemology’, Political Geography, 17.1 (1998), pp. 25–28 <https://doi.org/10.1016/S0962-6298(97)00049-8>
- Kaplan, Jared, and others, ‘Scaling Laws for Neural Language Models’, arXiv (preprint), 2001.08361, 23 January 2020 <https://doi.org/10.48550/arXiv.2001.08361>
- Koch, Bernard, and others, ‘Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research’, Thirty-Fifth Conference on Neural Information Processing, 11 October 2021 <https://openreview.net/forum?id=zNQBIBKJRkd> [accessed 25 January 2026]
- Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton, ‘ImageNet Classification with Deep Convolutional Neural Networks’, in Advances in Neural Information Processing Systems 25 (NIPS 2012), 2012 <https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf> [accessed 10 July 2026]
- Leech, Gavin, and others, ‘Questionable Practices in Machine Learning’, arXiv (preprint), 2407.12220, 30 October 2024 <https://doi.org/10.48550/arXiv.2407.12220>
- Lewis, Martha, and Melanie Mitchell, ‘Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models’, Proceedings of the Annual Meeting of the Cognitive Science Society, 46 (2024) <https://escholarship.org/uc/item/58d9s666> [accessed 17 May 2026]
- Liang, Percy, and others, ‘Holistic Evaluation of Language Models’, arXiv (preprint), 2211.09110, 1 October 2023 <https://doi.org/10.48550/arXiv.2211.09110>
- Liao, Thomas, and others, ‘Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning’, 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Datasets and Benchmarks Track, 11 October 2021 <https://openreview.net/forum?id=mPducS1MsEK>
- Lones, Michael A., ‘Avoiding Common Machine Learning Pitfalls’, Patterns, 5.10 (2024), pp. 1–16 <https://doi.org/10.1016/j.patter.2024.101046>
- Luccioni, Sasha, and Kate Crawford, ‘The Nine Lives of ImageNet: A Sociotechnical Retrospective of a Foundation Dataset and the Limits of Automated Essentialism’, Journal of Data-Centric Machine Learning Research, 29 February 2024 <https://openreview.net/forum?id=jh0ck1bPGF> [accessed 20 January 2025]
- MacKinnon, Danny, ‘Reconstructing Scale: Towards a New Scalar Politics’, Progress in Human Geography, 35.1 (2011), pp. 21–36 <https://doi.org/10.1177/0309132510367841>
- Manoli, Aikaterina, and others, ‘Digital Companionship: Overlapping Uses of AI Companions and AI Assistants’, Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, 13 April 2026, pp. 1–25 <https://doi.org/10.1145/3772318.3791331>
- Martínez-Plumed, Fernando, and others, ‘Research Community Dynamics behind Popular AI Benchmarks’, Nature Machine Intelligence, 3.7 (2021), pp. 581–89 <https://doi.org/10.1038/s42256-021-00339-6>
- McCarthy, John, ‘Generality in Artificial Intelligence’, Communications of the ACM, 30.12 (1987), pp. 1030–35 <https://doi.org/10.1145/33447.33448>
- McCoy, R. Thomas, and others, ‘Embers of Autoregression Show How Large Language Models Are Shaped by the Problem They Are Trained to Solve’, Proceedings of the National Academy of Sciences, 121.41 (2024) <https://doi.org/10.1073/pnas.2322420121>
- McKendrick, Joe, ‘Why GPT Should Stand For “General Purpose Technology” For All’, Forbes, 8 August 2023 <https://www.forbes.com/sites/joemckendrick/2023/08/08/why-gpt-should-stand-for-general-purpose-technology-for-all/> [accessed 25 January 2026]
- McKenzie, Ian R., and others, ‘Inverse Scaling: When Bigger Isn’t Better’, Transactions on Machine Learning Research (TMLR 2023), 20 October 2023 <https://openreview.net/forum?id=DwgRm72GQF> [accessed 25 January 2026]
- Miceli, Milagros, and Julian Posada, ‘The Data-Production Dispositif’, Proceedings of the ACM on Human-Computer Interaction, 6.CSCW2 (2022), pp. 1–37 <https://doi.org/10.1145/3555561>
- Miceli, Milagros, Martin Schuessler, and Tianling Yang, ‘Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision’, Proceedings of the ACM on Human-Computer Interaction, 4.CSCW2 (2020), pp. 1–25 <https://doi.org/10.1145/3415186>
- Mitchell, Melanie, ‘Debates on the Nature of Artificial General Intelligence’, Science, 383.6689 (2024) <https://doi.org/10.1126/science.ado7069>
- Moore, Adam, ‘Rethinking Scale as a Geographical Category: From Analysis to Practice’, Progress in Human Geography, 32.2 (2008), pp. 203–25 <https://doi.org/10.1177/0309132507087647>
- Moore, Jared, and others, ‘Characterizing Delusional Spirals through Human-LLM Chat Logs’, arXiv (preprint), 2404.09932, 17 March 2026 <https://doi.org/10.48550/arXiv.2603.16567>
- Morris, Meredith Ringel, and others, ‘Position: Levels of AGI for Operationalizing Progress on the Path to AGI’, Proceedings of the 41st International Conference on Machine Learning (ICML 2024), 2024 <https://proceedings.mlr.press/v235/morris24b.html> [accessed 31 May 2026]
- Narayanan, Arvind, and Sayash Kapoor, AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference (Princeton University Press, 2024) <https://doi.org/10.1515/9780691249643>
- Newell, Allen, John Clifford Shaw, and Herbert A. Simon, ‘Report on a General Problem-Solving Program’, Proceedings of the International Conference on Information Processing (Carnegie Mellon University), 9 February 1959 <https://home.mis.u-picardie.fr/~furst/docs/Newell_Simon_General_Problem_Solving_1959.pdf> [accessed 10 July 2026]
- Newell, Allen, John Clifford Shaw, and Herbert A. Simon, ‘Preliminary Description of General Problem Solving Program- I (GPS-I)’, CIP Working Paper #7, 29 December 1957 <http://iiif.library.cmu.edu/file/Simon_box00008_fld00526_bdl0001_doc0002/Simon_box00008_fld00526_bdl0001_doc0002.pdf> [accessed 10 July 2026]
- Ong, Keane, and others, ‘Human Behavior Atlas: Benchmarking Unified Psychological and Social Behavior Understanding’, The Fourteenth International Conference on Learning Representations (ICLR 2026), 26 January 2026 <https://openreview.net/forum?id=ZKE23BBvlQ> [accessed 1 June 2026]
- OpenAI and others, ‘GPT-4 Technical Report’, arXiv (preprint), 2303.08774, 15 March 2023 <https://doi.org/10.48550/arXiv.2303.08774>
- Ott, Simon, and others, ‘Mapping Global Dynamics of Benchmark Creation and Saturation in Artificial Intelligence’, Nature Communications, 13.6793 (2022) <https://doi.org/10.1038/s41467-022-34591-0>
- Ouyang, Siru, and others, ‘The Shifted and the Overlooked: A Task-Oriented Investigation of User-GPT Interactions’, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 2375–93 <https://doi.org/10.18653/v1/2023.emnlp-main.146>
- Raji, Inioluwa Deborah, and others, ‘AI and the Everything in the Whole Wide World Benchmark’, 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Datasets and Benchmarks Track, 2021 <https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/084b6fbb10729ed4da8c3d3f5a3ae7c9-Abstract-round2.html> [accessed 10 July 2026]
- Rogers, Anna, and Alexandra Sasha Luccioni, ‘Position: Key Claims in LLM Research Have a Long Tail of Footnotes’, ICML ’24: Proceedings of the 41st International Conference on Machine Learning, 21 July 2024 <https://dl.acm.org/doi/10.5555/3692070.3693805>
- Sap, Maarten, and others, ‘Social IQa: Commonsense Reasoning about Social Interactions’, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 4462–72 <https://doi.org/10.18653/v1/D19-1454>
- Saxon, Michael, and others, ‘Benchmarks as Microscopes: A Call for Model Metrology’, First Conference on Language Modeling (COLM 2024), 10 July 2024 <https://openreview.net/forum?id=bttKwCZDkm¬eId=Yfwy2d4fiT> [accessed 25 January 2026]
- Schaeffer, Rylan, Brando Miranda, and Sanmi Koyejo, ‘Are Emergent Abilities of Large Language Models a Mirage?’, 37th Conference on Neural Information Processing Systems (NeurIPS 2023), 23 June 2023 <https://openreview.net/forum?id=JRdN9GcI52>
- Scheuerman, Morgan Klaus, Alex Hanna, and Remi Denton, ‘Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development’, Proceedings of the ACM on Human-Computer Interaction, 5.CSCW2 (2021), pp. 1–37 <https://doi.org/10.1145/3476058>
- Sejnowski, Terrence J., The Deep Learning Revolution (MIT Press, 2018) <https://doi.org/10.7551/mitpress/11474.001.0001>
- Song, Peiyang, Pengrui Han, and Noah Goodman, ‘A Survey on Large Language Model Reasoning Failures’, 2nd AI for Math Workshop @ ICML 2025, 9 July 2025 <https://openreview.net/forum?id=hsgMn4KBFG> [accessed 25 January 2026]
- Srivastava, Aarohi, and others, ‘Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models’, Transactions on Machine Learning Research (TMLR 2023), 11 May 2023 <https://openreview.net/forum?id=uyTL5Bvosj> [accessed 25 January 2026]
- Srivastava, Aarohi, and others, BIG-Bench (2023) <https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks> [accessed 25 January 2026]
- Steyerl, Hito, ‘Mean Images’, New Left Review, April 2023, nos. 140/141, pp. 82–97 <https://doi.org/10.64590/uhm>
- Sutton, Rich, ‘The Bitter Lesson’, Incomplete Ideas, 13 March 2019 <http://www.incompleteideas.net/IncIdeas/BitterLesson.html> [accessed 25 January 2026]
- Tay, Yi, and others, ‘Scale Efficiently: Insights from Pretraining and Finetuning Transformers’, International Conference on Learning Representations, 28 January 2022 <https://openreview.net/forum?id=f2OYVDyfIB> [accessed 25 January 2026]
- Tsing, Anna Lowenhaupt, ‘The Global Situation’, Cultural Anthropology, 15.3 (2000), pp. 327–60 <https://doi.org/10.1525/can.2000.15.3.327>
- Tyrangiel, Josh, ‘Sam Altman on ChatGPT’s First Two Years, Elon Musk and AI Under Trump’, Bloomberg, 5 January 2025 <https://www.bloomberg.com/features/2025-sam-altman-interview/> [accessed 25 January 2026]
- Wang, Alex, and others, ‘GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding’, arXiv (preprint), 1804.07461, 22 February 2019 <https://doi.org/10.48550/arXiv.1804.07461>
- Wang, Pei, Ben Goertzel, and Stan Franklin, ‘Front Matter’, Proceedings of the 2008 Conference on Artificial General Intelligence 2008: Proceedings of the First AGI Conference, 20 June 2008, pp. i–x <https://dl.acm.org/doi/10.5555/1566174.1566175>
- Wei, Jason, and others, ‘Emergent Abilities of Large Language Models’, Transactions on Machine Learning Research (TMLR 2022), 31 August 2022 <https://openreview.net/forum?id=yzkSU5zdwD> [accessed 25 January 2026]
- Whittaker, Meredith, ‘The Steep Cost of Capture’, Interactions, 28.6 (2021), pp. 50–55 <https://doi.org/10.1145/3488666>
- Xu, Jialiang, Yifan Mai, and Percy Liang, ‘HELM Capabilities: Evaluating LMs Capability by Capability’, CRFM: Centre for Research on Foundation Models, 20 March 2025 <https://crfm.stanford.edu/2025/03/20/helm-capabilities.html> [accessed 17 May 2026]
- Zao-Sanders, Marc, ‘How People Are Really Using Gen AI in 2025’, Harvard Business Review, 9 April 2025 <https://hbr.org/2025/04/how-people-are-really-using-gen-ai-in-2025> [accessed 19 August 2025]
Cite as:
Magdalena Krysztoforska, ‘The Measure of Generality: On the Scalar Dynamics of AI Model Evaluation Practices’, in Scale: New Critical Essays, ed. by Christoph F. E. Holzhey and Marietta Kesting, Cultural Inquiry, 40 (ICI Berlin Press, 2026), pp. 297-330 <https://doi.org/10.37050/ci-40_mk>