Oukhay, Issam, Begoug, Mahi, Chouchen, Moataz et Ouni, Ali.
2026.
« When AI code doesn't stick: An empirical study on reverted changes introduced by AI coding agents ».
In MSR '26: Proceedings of the 23rd International Conference on Mining Software Repositories (Rio de Janeiro, Brazil, Apr. 13-14, 2026)
847–851.
Association for Computing Machinery, Inc.
Prévisualisation |
PDF
Ouni-A-2026-34498.pdf - Version publiée Licence d'utilisation : Creative Commons CC BY. Télécharger (5MB) | Prévisualisation |
Résumé
Agentic AI systems are increasingly integrated into software development workflows, contributing code alongside human developers. However, some AI-authored changes are later reverted, reflecting situations where agent-generated contributions are judged unsuitable after integration. This paper presents a large-scale empirical study of reverted changes introduced by AI coding agents to better understand the causes behind their rejection. We analyze 33,580 agentic pull requests comprising 86,315 commits authored by five major AI coding agents: Claude, Copilot, Cursor, Devin, and OpenAI Codex. Our results show that 2.66% of agentic pull requests contain at least one reverting commit, with substantial variation across agents, ranging from 0.7% for OpenAI Codex to 7.6% for GitHub Copilot indicating notable differences in code generation reliability. Through a manual analysis of 500 reverting commits, we derive a taxonomy comprising eight categories and 25 themes that explain why agent-generated code is reverted. The most common causes are unintended side effects (22.33%), overengineering (22.13%), functional incorrectness (17.71%), and dependency management problems (12.47%). Overall, our findings indicate that AI coding agents struggle primarily with scope management and contextual understanding, rather than purely functional defects. This study provides actionable guidance for practitioners, informs the design of human-AI collaboration workflows, and highlights priority areas for improving agentic code generation systems.
| Type de document: | Compte rendu de conférence |
|---|---|
| Chercheur(-euse): | Chercheur(-euse) Ouni, Ali |
| Affiliation: | Génie logiciel et des technologies de l'information |
| Date de dépôt: | 02 oct. 2026 18:42 |
| Dernière modification: | 02 oct. 2026 22:27 |
| URI: | https://espace2.etsmtl.ca/id/eprint/34498 |
Actions (Authentification requise)
 |
Dernière vérification avant le dépôt |