What Science tells us about the Productivity Effect of Generative AI in Software Development
Generative AI tools, such as GitHub Copilot, are transforming the way we develop software. These AI-powered tools are designed to provide developers with intelligent code recommendations, helping them complete tasks more efficiently. Recent research has shed light on the productivity effect of these tools in software engineering.
Empirical Evidence from Recent Studies
Peng et al. (2023) conducted a study with 95 participants from Upwork, measuring the impact of GitHub Copilot on developer productivity. They found a 55% reduction in completion time with a higher success rate for participants using the AI tool. The effect was even more significant for older age groups, those with less experience, and those working more development hours per day.
Dakhel et al. (2022) compared GitHub Copilot’s solutions to those provided by junior developers and students. They concluded that although Copilot’s solutions were more advanced, expert developers were still required to detect and filter out buggy or non-optimal solutions. Therefore, Copilot can be an asset when used by expert developers but may turn into a liability when used by inexperienced programmers.
In a study by Mastropaolo et al. (2023), researchers found that the effectiveness of AI-supported programming is heavily dependent on how well developers describe the code components they are looking for. They observed that semantically equivalent but different method descriptions resulted in different code recommendations in about 46% of cases. This highlights the importance of providing a proper code description when using AI-based recommenders to synthesize code.
Finnie-Ansley et al. (2023) found that OpenAI’s Codex performed better than most students on code writing questions from both CS1 and CS2 tests. The AI scored in the top quartile of test marks when compared with students, showing its potential to improve developer productivity.
Reflections and Open Questions
From a German perspective I find it particular interesting, that it seems like older employees might profit the most (because some experience seems needed and the GitHub studies shows the effect increasing with age).
Finally, however, it has to be remarked that none of these studies tells us very much about the effect on productivity that you can expect in any large scale software organisation. All studies so far have looked at well encapsulated problems and only coding itself, not debugging, not requirements engineering, not infrastructure engineering, not meetings and alignment. If anyone knows of studies that shed light on this, I would love to hear! Also if you know of researchers that would like to start answering these questions.
This article was initially published on LinkedIn