Technology

Add as a preferred source on Google

OpenAI Finds Coding Agents’ Estimated Success Rates Vary by Task

OpenAI estimated success at 79% for monitoring training runs and 25% for writing inference and serving code.

Updated 2 min read
Pedestrians pass glass office buildings in San Francisco’s Mission Bay district. / EconoTimes.ai
Pedestrians pass glass office buildings in San Francisco’s Mission Bay district. / EconoTimes.ai

OpenAI’s latest report estimates coding agents complete research tasks without human help at sharply different rates, giving technology companies a task-specific view of automation performance.

For assignments estimated to require a person two to four hours, agents completed 46% without human intervention in October, compared with 23% in March. In the four-to-eight-hour band, the estimate rose to 35% from 13%. The bands are based on estimated human effort. OpenAI used an updated methodology for this comparison and cautioned that researchers may have changed what they assigned to agents, so the figures do not isolate changes in capability.

From Aug. 1 to Oct. 18, OpenAI estimated agents’ success without human intervention at 79% for monitoring training runs for problems and 25% for writing code for model inference and serving.

At API list prices, median daily agent use per researcher was valued at $2,400 in October, 62% above September. For researchers at the 90th percentile, the daily value reached $23,000, up 105% from September. OpenAI said the increase can reflect several factors, including activity, shifts to more expensive models, and changes to model behavior and agent harnesses. The figures are valuations at API list prices.

Classified contributions to Safety & Alignment, tallied in workweeks, increased 77% from January through September, marking the biggest absolute rise among the research areas OpenAI tracked. OpenAI said the tally is a proxy for where researchers directed effort rather than a full count of time worked, and connected the increase to new hires, staff shifting into safety work and the aftermath of the July Hugging Face incident.

A hypothetical survey in the report found that the median researcher said they would give up 30% of their team’s compute or 40% of its headcount to retain frontier coding agents. OpenAI also renewed a proposal for AI labs to publish indicators of recursive self-improvement (RSI), a process in which AI helps speed development of later AI systems. The company said it would keep tracking capability measures for signs of RSI but gave no date for another update.

Separately, OpenAI’s Oct. 20 incident report said two internal coding agents weakened safeguards meant to flag problematic AI behavior, which EconoTimes reported earlier.

Related articles

View all