Converging AI and Human Roles
Tech & Engineering

Do Coding Agents Make Developers Faster? What the Trials Show

Controlled trials of AI coding tools range from a 19% slowdown for experienced open-source developers to 26% more completed tasks at three large employers. Tool generation, task, measure and who ran the study explain much of the gap.

SWE-bench Verified, share of tasks solved in Epoch AI's independent runs, by model (release month)
GPT-4o (Nov 2024)31.0%
Claude 3.7 Sonnet (Feb 2025)61.0%
GPT-5, high (Aug 2025)73.6%
Gemini 3 Pro preview (Nov 2025)72.9%
GPT-5.5 pre-release, xhigh (Apr 2026)80.6%
Claude Opus 4.7, max (Apr 2026)83.5%
Source: Epoch AI benchmark data (mean score), runs on 484 of the benchmark's 500 tasks; Epoch upgraded its setup in February 2026. Retrieved 2026-10-13. Epoch rates the benchmark "Flawed" (contamination and broken tests). Vendor-reported scores are not shown.
Chart: share of SWE-bench Verified tasks (real GitHub issues from 12 Python projects) that six models solved in Epoch AI's own runs. It measures what a model can fix unaided, not how fast developers work.

Role scorecard

Software Developers SOC 15-1252.00

  • Core tasks AI does in production1 of 6
  • Most mature evidenceIn production
  • Cost$19 per seat a month (Copilot Business list price)
  • Benefit26.08% more tasks completed (3 field RCTs)

As of · Tasks: O*NET 31.01

Core tasks of Software Developers and what AI does in each, as of 2026-10-13
Core task (O*NET) What AI or robots do Extent Evidence grade Example case Source
Analyze user needs and software requirements to determine feasibility of design within time and cost constraints No deployed case of AI judging feasibility, time or cost. GitHub says its cloud agent can research a repository to "confirm assumptions before planning" Not yet Vendor claim GitHub Copilot cloud agent documentation, 2026 2
Develop or direct software system testing or validation procedures, programming, or documentation Generates extra unit tests for existing human-written test suites and filters out any that fail to build, fail to pass or add no coverage; engineers decide which to land Assists In production TestGen-LLM at Meta's Instagram and Facebook test-a-thons; 73% of its recommendations accepted for production (Meta-authored report, 2024) 3
Modify existing software to correct errors, adapt it to new hardware, or upgrade interfaces and improve performance Unattended agents write complete fixes, such as repairs for flaky tests flagged by CI, and open pull requests with no human-written code; another engineer reviews before merge Does it In production Minions at Stripe, more than 1,000 merged pull requests a week (operator-stated, February 2026); Google's AI-assisted code migrations, 80% of changed code AI-authored in one case 4, 5
Prepare reports or correspondence concerning project specifications, activities, or status Drafts pull-request summaries for reviewers; GitHub tells users to review the generated summary carefully Not yet Vendor claim GitHub Copilot pull-request summaries documentation, 2026 6
Analyze information to determine, recommend, and plan installation of a new system or modification of an existing system Reads a codebase and proposes a change plan, making no edits until the developer approves it Not yet Vendor claim Claude Code plan mode and GitHub Copilot cloud agent plans, vendor documentation, 2026 7, 2
Design, develop and modify software systems, using scientific analysis and mathematical models to predict and measure outcomes and consequences of design Generates much of the new code from developers' instructions; engineers direct, review and approve it Assists In production Claude Code and GitHub Copilot CLI across Microsoft engineering, since January 2026; at Google, 75% of new code AI-generated and approved by engineers (company-stated, April 2026) 8, 9

Extent and grade follow our evidence grades: "Does it" needs a production case where AI performs the task on its own; "Assists" means AI helps a person do it; "Not yet" means only prototypes, vendor claims or nothing. Six of the 11 core tasks O*NET lists are shown; the five left out (conferring with analysts and engineers, consulting customers or other departments, conferring with project managers, determining system performance standards, and storing and manipulating data for analysis) had no case in the sources below. "Does it" for correcting errors rests on Stripe's statement that its agents' merged pull requests contain no human-written code; an engineer still starts each run, and another engineer reviews the pull request, as for human-written code. New development counts as "Assists" because engineers direct and approve the code.

Costs and benefits

GitHub Copilot list price$19 USD per granted seat per month (Business) or $39 (Enterprise); usage beyond the pooled AI credits is charged at $0.01 USD per credit (vendor list price, checked 2026-10-13)10
Claude Code list priceIncluded in Claude Pro at $20 a month billed monthly, or $17 a month billed annually (vendor list price, checked 2026-10-13)11
Token spend at scale"Can run into millions of dollars annually" (Microsoft researchers, on company-wide agent rollouts)8
Completed tasks+26.08% a week (SE 10.3%), three field randomized trials, 4,867 developers (two Microsoft Research co-authors; MIT GenAI Initiative funding for two others)12
Merged pull requests+24.0% per engineer per day (95% CI +14.5% to +33.7%), Microsoft's observational study of its own 2026 rollout8
Pull-request frequencyUp 76% (operator-stated, Spotify, June 2026)13

Seen at work

Spotify's chief architect on how its engineers and its background coding agent, Honk, use Claude across the company's codebase, at Code with Claude 2026. Video: Claude (Anthropic's channel; vendor video of the operator's talk).

As of · Randomized trials of AI coding tools with software developers; results as each study states them

Randomized trials of AI coding tools with software developers, as of 2026-10-13
Study Year Design Participants (n) Setting Tool Outcome Result as stated Authors' affiliation / funder Source
METR, early 2025Becker et al. 2025 Randomized trial, per task 16 experienced developers, 246 tasks Real work Cursor Pro with Claude 3.5/3.7 Sonnet (Feb–Jun 2025) Task completion time, in mature open-source projects +19% longer with AI allowed (95% CI +2% to +39%) METR, independent nonprofit; developers paid $150 an hour 14, 15
METR, late 2025 follow-upBecker et al. 2026 Randomized trial, per task 10 returning and 47 new developers Real work Latest AI tools, including agents, from Aug 2025 Task completion time, open-source projects Time change −18% for returning developers (95% CI −38% to +9%) and −4% for new ones (−15% to +9%); METR calls the data unreliable METR, independent nonprofit; developers paid $50 an hour 15
GitHub Copilot experimentPeng et al. 2023 Randomized controlled experiment 95 freelance developers recruited Lab task GitHub Copilot, before general availability (May–Jun 2022) Time to write an HTTP server in JavaScript 55.8% faster (95% CI 21% to 89%) Microsoft Research, GitHub and MIT 16
Three field experimentsCui et al. 2026 Randomized field experiments (3, pooled) 4,867 developers Real work GitHub Copilot code completions (experiments from Sep 2022) Weekly completed tasks at Microsoft, Accenture and an anonymous Fortune 100 company +26.08% (SE 10.3%); results vary across the three experiments Princeton, MIT, Wharton and Microsoft Research; MIT GenAI Initiative funding for two authors; trials run by the companies 12
Google enterprise trialParadis et al. 2024 Randomized controlled trial 96 Google engineers Lab task Google's internal AI code completion, smart paste and natural-language-to-code (summer 2024) Time on one enterprise-grade task About 21% less time, with a large confidence interval Google employees 17
Skill formation trialShen and Tamkin 2026 Randomized controlled trial 52 mostly junior engineers Lab task AI assistant in a coding platform's sidebar Learning a new Python library (Trio): task time and a quiz Quiz score 17% lower with AI; speed gain not statistically significant Anthropic 18

The outcomes differ (minutes on one task, tasks per week, time on tasks developers chose, quiz scores), so the rows can't be compared with each other or ranked. Signs are as each study reports them: METR reports the change in completion time, so +19% means slower and −18% means faster; the others report time saved or output gained. "Year" is the year of the publication cited.

As of · Observational studies of coding agents, kept apart from the randomized trials above

Observational studies of coding agents, as of 2026-10-13
Study Year Design Sample Tool Outcome Result as stated Authors' affiliation Source
Microsoft's 2026 rolloutMurphy-Hill et al. 2026 Synthetic control (CausalImpact) plus within-person analysis Tens of thousands of Microsoft engineers, Jan 5–Apr 29, 2026 Claude Code, GitHub Copilot CLI Merged pull requests per engineer per day +24.0% (95% CI +14.5% to +33.7%); no statistically distinguishable fade over four months Microsoft employees 8
Agents in open-source repositoriesAgarwal et al. 2026 Staggered difference-in-differences with matched controls 401 agent-first and 117 AI-editor-first repositories, with matched controls Coding agents (first agent-generated pull request) Commits, lines added, static-analysis warnings, code complexity Large, front-loaded velocity gains only where agents were the first AI tool; warnings up about 18% and cognitive complexity about 39% in both groups Carnegie Mellon University 19

In both studies adopters chose to use the tools, so the comparisons can be biased in ways a randomized trial avoids; the Microsoft authors report a placebo test that their estimate passed.

Four trials, four answers

A coding agent is AI software that can read a codebase, run commands and write or change code toward a goal a developer sets, rather than only suggesting the next line. Whether these tools, and the code assistants before them, make software developers faster has been tested in a handful of randomized trials, and the answers disagree. In the strictest test on experienced developers, METR, an independent nonprofit research group, randomly allowed or banned AI on each of 246 real tasks that 16 developers chose from mature open-source projects they had worked on for five years on average. With AI allowed (mostly Cursor Pro with Claude 3.5 and 3.7 Sonnet, between February and June 2025), tasks took 19% longer.14

GitHub's 2022 experiment pointed the other way. Of 95 freelance developers recruited on Upwork to build an HTTP server in JavaScript, those given GitHub Copilot finished 55.8% faster; the authors worked at Microsoft Research, GitHub and MIT.16 Field experiments that Microsoft, Accenture and an anonymous Fortune 100 company ran in the ordinary course of business, pooled across 4,867 developers, found 26.08% more completed tasks a week among developers given Copilot's code completions.12 A Google trial of 96 of its own engineers, using three internal AI features on one enterprise-grade task, put the time saving at about 21%, with what its Google authors call a large confidence interval.17

Why the results diverge

The studies differ on almost everything that matters. They tested different generations of tools: Copilot's 2022 code completions, Google's 2024 internal features, and the early-2025 editor-and-chat tools in METR's study. They measured different things: minutes on one lab task, tasks completed per week, or time on tasks developers picked themselves. And they studied different people. The Google authors note that about 45% of the Upwork sample in GitHub's trial were self-employed,17 while METR's developers knew their projects intimately.14

Experience keeps coming up. GitHub's experiment found that less experienced developers benefited most,16 and the field experiments found that less experienced developers both adopted the tool more and gained more.12 METR examined 20 properties of its setting that might explain the slowdown, including project size and quality standards and developers' prior AI experience, and concluded the effect was unlikely to be mainly an artifact of its design.14 Self-reports were a poor guide: before starting, METR's developers forecast that AI would cut their time by 24%, and afterwards they estimated it had cut it by 20%.14

Agents broke the old study design

METR began a larger follow-up in August 2025 with 10 returning and 47 new developers. In February 2026 it said the results were an unreliable signal. More developers declined to take part because they did not want to work without AI, 30% to 50% of those surveyed said they held back tasks they didn't want to do without it, and time records were unreliable for people running several agents at once.15 The raw estimates point toward speedup, a time change of −18% for returning developers and −4% for new ones, but both confidence intervals include zero, and METR calls the data only very weak evidence while saying the true effect may be larger among the developers and tasks that dropped out.15

A later METR note adds a measurement problem. When AI makes some tasks cheaper, people shift their time toward them, so a speedup measured on the old mix of tasks understates the value gained, and one measured on the new mix overstates it.20

What operators report from production

With randomization harder, the newest evidence comes from operators measuring their own rollouts. Microsoft researchers studied tens of thousands of the company's engineers after two command-line coding agents, Claude Code and GitHub Copilot CLI, took off internally from January 5, 2026. Against a synthetic control (a modelled estimate of what adopters would have done without the tools), adopters merged 24.0% more pull requests per engineer per day, with no statistically distinguishable fade over four months.8 The authors caution that adopters chose themselves and that merged pull requests are an imperfect proxy that rewards small, frequent changes and can miss quality costs.8

Company statements go further, but they are not measurements. Google's chief executive said in October 2024 that more than a quarter of all new code at Google was generated by AI, then reviewed and accepted by engineers,21 and in April 2026 that the share had reached 75%, up from 50% the previous fall.9 Spotify says more than 99% of its engineers use AI coding tools every week, 94% report that AI has made them more productive, and pull-request frequency is up 76%.13 Stripe says its in-house unattended agents, called minions, produce more than a thousand merged pull requests a week that are reviewed by humans but contain no human-written code.4 These figures are company-stated, and none is counted against a common standard, so they can't be compared with each other or with the trials.

Speed is not the only outcome

A Carnegie Mellon study of open-source repositories found that adopting coding agents brought large, front-loaded gains in commits and lines added only where agents were a project's first AI tool; repositories already using AI editors saw small or short-lived gains. In both groups, static-analysis warnings rose by about 18% and cognitive complexity, a measure of how hard code is to follow, by about 39%.19 Google Cloud's 2025 DORA survey of nearly 5,000 technology professionals found that 90% use AI at work and more than 80% believe it has raised their productivity, yet 30% report little or no trust in AI-generated code, and AI adoption still showed a negative relationship with software delivery stability.22 In one of Google's AI-assisted migrations, engineers usually had to revert at least some changes the model made.5

Skills are a cost under study too. In an Anthropic randomized trial, 52 mostly junior engineers learning an unfamiliar Python library scored 17% lower on a quiz afterwards when they had an AI assistant, and the speed gain did not reach statistical significance. Participants who used the assistant to ask for explanations retained more.18

What benchmarks can and can't show

A benchmark measures what a model can do on fixed tasks, not how fast a person works. SWE-bench Verified asks a model to resolve real GitHub issues from 12 open-source Python repositories. Epoch AI, which runs it independently on the 484 of its 500 tasks that work on Epoch's infrastructure, lists scores ranging from 31.0% for GPT-4o (released November 2024) to 83.5% for Claude Opus 4.7 at maximum effort (April 2026) in its current data; it significantly upgraded its setup in February 2026.23 Epoch now rates the benchmark "Flawed". It cites an OpenAI audit, released February 23, 2026, which found that 59.4% of the 27.6% of tasks it examined had tests that rejected functionally correct fixes, and OpenAI's conclusion that gains on the benchmark increasingly reflect how much a model saw it in training.24 Model makers' own published scores have used different task counts (477 for GPT-5 and 489 for Claude 3.7 Sonnet, by Epoch's account), one reason vendor-reported scores and independent runs are never mixed here.23

Who stands behind each result

Most large positive results come from tool makers or the companies that deploy them. GitHub and Microsoft Research staff wrote the 2023 experiment; two Microsoft Research economists co-wrote the field-experiment paper, whose trials the companies ran themselves; Google engineers ran Google's trial; Microsoft researchers measured Microsoft's rollout; and Anthropic, which makes Claude Code, ran the skills trial. METR and the Carnegie Mellon team are independent of the tools they studied. Usage data carries the same caveat: Anthropic's first Economic Index found that 37.2% of conversations on Claude.ai fell in computer and mathematical occupations, covering tasks such as software modification and debugging. That measures observed use of one product, not time saved.25

The role behind the numbers

O*NET, the US occupational database, lists 11 core tasks for software developers. The scorecard above shows AI in production in three of the six tasks it covers, and doing one of them, correcting errors in existing code, without a person writing the change, at Stripe.14 Bureau of Labor Statistics data shown on O*NET count 1,717,800 software developers in 2025 with a median annual wage of $135,980, and BLS's 2025–2035 projections class the occupation's growth as much faster than average.1 None of the studies above measured employment; they measured the output of developers who kept working. This page describes what the evidence shows; it isn't advice about anyone's career or about which tool to buy.

Sources

  1. O*NET OnLine (O*NET 31.0; wages and employment from the Bureau of Labor Statistics), 15-1252.00 Software Developers: tasks, wages and employment, retrieved 2026-10-13.
  2. GitHub Docs, Research, plan, and iterate on code changes with Copilot cloud agent (vendor documentation), retrieved 2026-10-13.
  3. Alshahwan N, et al. (Meta), arXiv, Automated Unit Test Improvement using Large Language Models at Meta, February 14, 2024.
  4. Stripe, Minions: Stripe's one-shot, end-to-end coding agents, February 9, 2026.
  5. Nikolov S, et al. (Google), arXiv, How is Google using AI for internal code migrations?, January 12, 2025.
  6. GitHub Docs, Creating a pull request summary with GitHub Copilot (vendor documentation), retrieved 2026-10-13.
  7. Anthropic, Claude Code Docs, Common workflows: plan before editing (vendor documentation), retrieved 2026-10-13.
  8. Murphy-Hill E, Butler J, Savelieva A (Microsoft), arXiv, Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI, July 1, 2026.
  9. Google, Cloud Next '26: Momentum and innovation at Google scale (CEO remarks), April 22, 2026.
  10. GitHub Docs, Plans for GitHub Copilot (vendor pricing), retrieved 2026-10-13.
  11. Anthropic, Claude pricing (vendor pricing), retrieved 2026-10-13.
  12. Cui ZK, Demirer M, Jaffe S, Musolff L, Peng S, Salz T, Management Science, The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers, February 27, 2026 (abstract and funding note via Crossref; experiment details from the authors' working-paper draft at MIT Economics).
  13. Spotify Engineering, Code with Claude: Coding is no longer the constraint, June 3, 2026.
  14. Becker J, Rush N, Barnes E, Rein D (METR), arXiv, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, July 12, 2025.
  15. METR, We are Changing our Developer Productivity Experiment Design, February 24, 2026.
  16. Peng S, Kalliamvakou E, Cihon P, Demirer M, arXiv, The Impact of AI on Developer Productivity: Evidence from GitHub Copilot, February 13, 2023.
  17. Paradis E, et al. (Google), arXiv, How much does AI impact development speed? An enterprise-based randomized controlled trial, October 16, 2024 (revised November 11, 2024).
  18. Anthropic, How AI assistance impacts the formation of coding skills, January 29, 2026 (paper: Shen JH, Tamkin A, How AI Impacts Skill Formation, arXiv).
  19. Agarwal S, He H, Vasilescu B (Carnegie Mellon University), arXiv, AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development, January 27, 2026 (v2; MSR '26).
  20. METR, Task Substitution and Uplift, May 8, 2026.
  21. Google, Q3 earnings call: CEO's remarks, October 29, 2024.
  22. Google Cloud, Announcing the 2025 DORA Report, September 23, 2025.
  23. Epoch AI, SWE-bench Verified (methodology) and benchmark data, retrieved 2026-10-13.
  24. Epoch AI, SWE-bench Verified: Benchmark review, September 3, 2026 (summarizing OpenAI's audit of February 23, 2026).
  25. Anthropic, The Anthropic Economic Index, February 10, 2025.