Jan 23, 2025Unverified

Devin billed $500/month in ACU credits while failing roughly 85% of assigned tasks

Approved by person, Oct 12, 2026. How entries are approved

Showing full summaries.

Devin, marketed as the first AI software engineer, was rolled out to the general public in December 2024 at $500 per month. Researchers from Answer.AI tested it and completed only three out of 20 tasks successfully. Engineers Hamel Husain, Isaac Flath, and Johno Whitaker reported that tasks that seemed straightforward often took days rather than hours and that Devin had a concerning tendency to press forward with tasks that were not actually possible. Three data scientists found they only completed three of 20 tasks. The Register summarized the independent reviews under the headline that the tool touted as the first AI software engineer is bad at its job.

What this means for youAt work, paid AI coding tools can cost a lot while delivering little, so a team should test them on its own real tasks before subscribing. Before paying for any AI agent, ask for a trial on your own work and check how often it actually finishes tasks.

Sources

The Registerhttps://www.theregister.com/2025/01/23/ai_developer_devin_poor_reviews/

Incident records from the AI Agent Incident Database by sipi.bot (https://sipi.bot/incidents/), licensed CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).

Source data license: CC BY 4.0. Attribution to sipi.bot is required.

Covered by snapshot 45

da93f03b406f409940d0a6ef47b526607f82ebf457d98f69299b88aceede576a

This entry is record number 2417. It has not been edited since it was approved; the hash above would not match if it had. How to verify