AI agents’ 80%-reliable task length runs about 5x shorter than their 50% horizon

Scatter plot of METR 50% and 80% success time horizons in human-expert hours by model release date, 2024-05 to 2026-04

Across 19 models released between 2024-05-13 and 2026-04-07 in METR’s Time Horizon 1.1 data, the task length a model completes with 80% predicted success is a median 5.3x shorter (range 4.0x to 10.3x) than the length it completes with 50% success. The most recent point, an early Claude Mythos Preview, reaches 17.4 human-expert hours at 50% but only 3.1 hours at 80%, and METR cautions that estimates above 16 hours are unreliable. The gap matters because attackers can tolerate frequent failure: in the AI-agent hacking campaign covered in today’s SI News brief, Reuters reports that the agents attempted more than 1,200 sites and compromised more than 100, while businesses deploying agents generally need the reliable end of the curve.

Sources: METR, Task-Completion Time Horizons of Frontier AI Models (Time Horizon 1.1 data file; p50 and p80 horizon estimates and release dates) · Reuters (republished by RTÉ) · NorForge, Inside Cairn
Original story: SI News: Agents that hack, Anthropic’s public-benefit fund, Disney’s Character.AI hire
Note: METR’s horizon measures how long the tasks take human experts, not how long an agent runs. Points are point estimates; METR’s confidence intervals are wide (Claude Mythos Preview at 50%: 8.5 to 55 hours).

Posted by on • 0 Comments

Hybrid reasoning cut estimated decoding cost by 63%–73%

Hybrid reasoning cut estimated decoding cost by 63%–73%

In the authors’ matched evaluation, estimated generation cost per million questions fell from $359 to $98 at 4B parameters, $439 to $121 at 8B, and $470 to $176 at 14B. That is a 62.6%–72.7% reduction, although the estimate assumes $2.50 per H100-hour and 10% utilization and excludes prompt prefill, context-dependent attention, and external tool execution. The result adds cost detail to SI News’s report on Hybrid Reasoning Agents, showing that their shorter decoded trajectories could materially reduce inference expense even after latent steps are charged as full backbone passes.

Original SI News story: 1. Anthropic 5% shares into public benefit fund. 2. MiniMax model M3.1 Flash with 1M context. 3. Stanford + UIUC + Together AI on "Hybrid Reasoning Agents".

Data sources

Categories: AI

Posted by on • 0 Comments