What Is Devin Actually Doing When It Codes? Analyzing the Boundaries of AI Agent Capabilities

AI coding agent Devin once promised that "AI can do the work," not just write code. But independent reports show that its output still requires human review. Behind this lies a huge gap between benchmark tests and real projects, and it also foreshadows profound changes in the work of junior developers.

Devin's Promise: From "Writing Code" to "Doing Work"

In 2024, Cognition's Devin was positioned as an "AI software engineer" that can plan tasks, use a terminal, edit files, run tests, and fix its own work. This shift in claim is significant: AI is no longer just assisting in generating code snippets, but attempting to "complete work" end to end.

This positioning distinguishes Devin from traditional code completion tools. It attempts to simulate the complete workflow of a human developer, from understanding requirements to delivering runnable code. However, this "agent" model also brings higher expectations and more complex evaluation challenges.

The key difference between AI coding agents and traditional code completion or conversational assistants lies in autonomy and tool use. Agents typically have the ability to plan tasks, call terminals, read and write files, run tests, observe execution results, and iteratively correct themselves, attempting to autonomously advance tasks over a longer time span. This model shifts the evaluation focus from "whether the generated code snippet is correct" to "whether the end-to-end task is completed," and therefore places higher demands on error recovery, context management, and environment interaction capabilities.

The Limitations of SWE-bench: Benchmarks ≠ Real Projects

Devin's excellent performance on SWE-bench made the promise of "AI doing work" quantifiable. SWE-bench contains a large number of real issues from GitHub, requiring AI to generate patches and pass tests. High scores seem to prove Devin's capabilities.

But the problem is that SWE-bench issues are clearly defined and closed. Each problem has a clear description and verifiable test cases. In real software development, however, requirements are often vague, tests may already be broken, and codebases may be in disrepair. Benchmark completion does not equal project ownership—this is precisely the core question facing Devin.

SWE-bench is a code benchmark proposed by Princeton University and other institutions. It is constructed by extracting real issues and their corresponding fix PRs from several mainstream Python open-source projects on GitHub, and using the test changes in the fix PRs as the basis for automatic judgment. The model needs to generate a patch based on the issue description so that it passes the newly added tests without breaking the original tests. This design makes the evaluation closer to real development tasks, but because the issues themselves have already been clarified by maintainers and the test cases explicitly exist, the task boundaries are relatively closed, making it difficult to reflect scenarios with vague requirements, missing tests, or aging codebases.

Real-World Challenges: Vague Requirements and Aging Code

Independent evaluation reports point out that Devin's output still requires extensive human review. In real projects, developers often face vague requests such as "this feature doesn't feel right, take a look and change it," or test suites that are largely red due to dependency conflicts.

AI agents are good at optimizing within clear boundaries, but real software engineering requires a great deal of tacit knowledge: understanding business context, weighing technical debt, and communicating with stakeholders. These are precisely the shortcomings of current AI agents. Devin can run tests, but it is difficult for it to judge "whether this test itself is reasonable"; it can edit files, but it is difficult for it to understand "why this code was written this way."

The concept of tacit knowledge was proposed by the philosopher Michael Polanyi. It refers to knowledge that is difficult to fully convey through language and can only be acquired through practice and immersion in context. In software engineering, it manifests as an understanding of business background, historical decisions, team conventions, and the evolution of code. AI agents can obtain explicit information from code and tests, but it is difficult for them to automatically acquire this tacit knowledge. Therefore, when facing vague requests such as "this feature doesn't feel right," they often lack a basis for judgment and need human developers to supplement the context.

The Impact on Junior Developers: Reshaping the Content of Work

The gap exposed by Devin may reshape the work of junior developers. If AI can handle clearly defined issues, then junior developers will no longer mainly rely on "fixing bugs" to become familiar with the codebase. Their work may shift toward reviewing AI-generated code, transforming vague requirements into clear tasks, and maintaining the health of test suites.

This is not a bad thing. Junior developers can gain exposure to higher-level skills such as system design and requirements analysis more quickly. But it also requires them to have stronger code review skills and a better understanding of the business. The value of simply being able to write code is declining, while the ability to judge whether code is good or bad and to define problems is increasing.

Conclusion: The Boundaries of Agents and the Value of Humans

Devin represents an important step for AI coding agents, but it also clearly delineates the boundaries of current technology. No matter how high the benchmark score is, it cannot replace an understanding of the complexity of real projects. For developers and teams, taking a rational view of the capability radius of AI agents and positioning them as "efficient executors" rather than "project owners" may be a more pragmatic choice.

One-line Summary

Devin's promise to go from writing code to doing work exposes the gap between benchmarks and real projects.

Paragraph Summary

The article focuses on the AI coding agent Devin, pointing out that its positioning has been upgraded from "assisting with writing code" to "completing work end to end," and that it has achieved quantifiably high scores on the SWE-bench benchmark. However, independent evaluations show that Devin's output still requires extensive human review. The core reason is that SWE-bench issues are clearly defined and tests are verifiable, while real projects often involve vague requirements, broken tests, and legacy code, requiring tacit knowledge such as business context and trade-offs around technical debt. The article argues that this gap may reshape the work of junior developers: the proportion of bug fixing will decline, while the value of code review, requirements transformation, and test maintenance will rise. The conclusion is that benchmark completion does not equal project ownership, and AI agents should be positioned as efficient executors rather than project owners.