Bernard van der Esch

🤖 Artificial intelligence in code: the productivity illusion and automation bias

Generative AI was expected to bring a step change in the speed of engineering teams. Tools based on large language models (LLMs) do cut the time it takes to write individual functions by 25–55%, but at the same time they lengthen the time needed to verify that code.

This is the Productivity-Validation Gap: the asymmetry between generating program text quickly and the limited throughput of our minds during an audit.

Behind it is automation bias, the psychological tendency to trust the suggestions of automated systems uncritically (Zhou et al., 2026). Below I describe what research on this phenomenon shows and how to protect a team against it.

đź“Ś Part 1. Three traps of working with AI

1. The illusion of speed and the drop in quality

In METR’s controlled experiment (Becker et al., 2025), developers solving complex tasks with AI agents worked 19% slower than the group without AI support. Before the study they expected a 24% speed-up, and after the test they were still convinced they had saved time.

The generated code also carries measurable risk:

  • Security holes: in studies of GitHub Copilot (Pearce et al.), 39.3–40% of the fragments generated in high-risk scenarios contained critical vulnerabilities, including SQL injection and buffer overflow.
  • False confidence: developers using AI write less secure code, and at the same time rate its correctness higher than people working the traditional way (Perry et al.).
  • Codebase erosion: in GitClear’s analysis of 153–211 million lines of code, the rate of short-term discarding and rewriting of code (code churn under 14 days) rose from 3.3% to 7.1%, the share of refactoring fell from 25% to 13.4%, and the share of copied code rose from 8.3% to 12.3%.
  • Logic errors: they appear on average 75% more often in generated code than in code written by hand.

2. How AI lulls our vigilance

  • Eloquence bias: when AI attaches a fluent, professional natural-language explanation to the code, vigilance drops. Such explanations do not reduce blind trust; they increase the uncritical acceptance of wrong suggestions.
  • Model sycophancy: models have a built-in tendency to agree with the user (SycEval, Fanous et al., 2025). In the research of Fahad et al. (2026), when a developer suggested a wrong thesis in the question, the model abandoned its correct analysis and confirmed the human’s version in 72% of cases (Decision Flip Rate).
  • Double confirmation: code and tests generated in the same session record the same error twice, in the implementation and in the assertions. The tests pass in the CI/CD pipeline and give high code coverage with a false sense of security. (Here I encourage you to read my previous article, which sheds light on how to improve this.)

3. Comprehension debt and junior atrophy

Using AI lowers cognitive effort, but deprives us of deep information processing. Neurocognitive (EEG) studies showed a 55% drop in neural connectivity during AI-assisted coding tasks, and 83% of participants could not quote or explain code they had accepted a dozen or so minutes earlier. This is how comprehension debt arises: code that nobody on the team understands lands in the repository.

Juniors are the most susceptible to automation bias. They accept AI suggestions 20% more often than seniors, and when learning with AI assistance they acquire conceptual knowledge and diagnostic skills 40–50% more slowly.

🛠️ Part 2. Five rules for the team

Passive warnings such as “remember to verify AI code” do not work. What is needed are interventions built into the process.

1. Cognitive Forcing Functions

Instead of showing suggestions immediately, configure the IDE to require an up-front commitment from the engineer: the developer first types the expected return type, an edge case or a test name themselves, and only then sees the AI’s proposal. The brain then has to build its own logical model before seeing the suggested code.

2. Separating code generation from test generation

This is the answer to the double-confirmation error. Never generate production code and unit tests in the same AI session. Reverse the flow: tests are written and verified first, based on the business requirements, and only after they are approved does AI write the code that satisfies them.

3. An unaided-first rule for juniors

Newly hired and beginner developers work without code autocompletion assistants for their first 3–6 months. During that time they build their own mental models and the habit of reading code critically, without which safe collaboration with AI is not possible.

4. A hidden PR description at the start of code review

The same eloquence bias can be addressed at the process level. During the first phase of inspection in GitHub or GitLab, automatically collapse the AI-generated Pull Request summaries. The reviewer then first judges the bare code and the test results, and so does not fall for the illusion of correctness that a fluent description creates.

5. Code Turnover Rate instead of speed alone

Change the success metrics. Instead of the number of closed tasks, measure the share of lines deleted or rewritten less than 14–30 days after deployment. A rise in this metric is a signal that verification debt is building up in the team.

đź’¬ What this means for the team

Artificial intelligence speeds up writing syntax. Without the procedures described above, software engineering turns into superficially approving other people’s proposals.

Sources