Why AI progress has not slowed

My attempt at Dwarkesh’s first question

Dwarkesh’s prompt:

A couple years ago, there was this idea that AI progress might slow down as we make further progress into the RL regime. 1. Because as horizon lengths increase, the AI needs to do many days’ worth of work before we can even see if it did it right, so if we’re still in a naive policy gradient world, the reward signal / FLOP goes down, and 2. We’d crossed through many OOMs of RL compute from GPT 4 to o1 to o3, and it would not be feasible to replicate that many OOMs increase in compute immediately again. But AI progress seems to have been fast nonetheless - even potentially speeding up if rumors about Spud or Mythos are to be believed. What gives? What did that previous intuition pump that motivated longer timelines miss? Feel free to deny premise of question.

Whence recent AI progress?

Bottom Line Up Front: The RL pessimism argument was mathematically sound but the longer timelines conclusion some drew from it was not. First, pretraining was paused rather than exhausted, and second, each new verifiable RL environment resets the scaling curve.

If projections from last year’s RL pessimism had been right, we should be seeing a slowdown in AI progress. But METR’s time-horizon graph continues to double every seven months. Mythos Preview saturates the 16-hour ceiling of what their task suite can currently measure.

Source

There were two parts to RL pessimism. First, RL is information-inefficient per FLOP, and the inefficiency gets worse on long-horizon tasks where the reward arrives many steps after the action. Second, RL training compute scales much worse than pretraining or inference compute, and we had already burned through many OOMs of it from GPT-4 through o1 to o3, so further OOMs would not come cheap.

Toby Ord made the second argument in October. There are three compute axes: pretraining, RL training, and inference. Plotting o1 and o3 performance on AIME against RL training compute, Ord showed that RL scales much worse than inference. A 100x increase in RL training moves AIME from 33 to 66 percent; a 100x in inference moves it from 20 to 80. Getting from 20 to 80 via RL training requires roughly 10,000x more compute. Combined with Jones (2021), which estimated that 1000x inference scaling is roughly equivalent to 100x pretraining scaling, Ord concluded that a single GPT-level capability jump from RL alone would require around a million times more total RL compute than has already been spent.

From o1 to o3 the RL compute spend was small relative to the pretraining bill, so each subsequent 10x added almost nothing in absolute terms. If pretraining is 100 units of compute and RL is 1, then scaling RL by 10x takes the training bill from 101 to 110, only 1.1x. But once RL compute equals pretraining (100 each, 200 total), another 10x of RL takes the bill to 1100, a 5.5x increase. Ord pointed to the Grok 4 launch chart (at 8:50; white is pretraining compute and orange is RL compute) as the first public sign a frontier lab has hit that crossover, with the rest of the frontier labs probably not far behind.

Source

This analysis was sound, but two mechanisms explain why progress has continued unabated.

The first is that pretraining was paused, not exhausted. When GPT-4.5 dropped in February 2025, many observers reacted as though pretraining had hit a wall. Trevor Chow saw otherwise. Meta’s Llama 3 family was a near-clean controlled comparison, with eight open-weight models ranging from 1B to 405B parameters released within an eight-month window, so compute-efficiency improvements between them were minimal and pretraining compute was essentially the only variable. Chow fit a scaling curve relating pretraining compute to benchmark performance across those eight points, and GPT-4.5 lands on the predicted curve or slightly beats it.

Source

Pretraining was working as predicted—we just could not see it because the field had spent two years extracting capability gains from cheaper interventions in mid-training and post-training instead. Chow predicted pretraining would return once RL had to be done across many domains (expensive) and once CoT inference costs grew enough that a smarter base model needing fewer CoT tokens became the cheaper option.

Mythos and Spud were trained on the current generation of frontier chips, including Nvidia Blackwell GPUs, the generation after the Hopper GPUs that trained most pre-2026 frontier models. Anthropic recently signed a deal for over 220,000 GPUs at xAI’s Colossus 1, on top of multi-gigawatt deals with Amazon, Google, and Microsoft. Pretraining will keep delivering gains as this compute comes online.

The second mechanism, which answers both halves of the pessimism, is targeted RL on narrow verifiable domains. Frontier labs have been running this since o1, picking a domain with a clean grader (math, code, SWE-bench), building an environment, and running RLVR until performance plateaus.

On the OOMs problem: each new domain enters its scaling curve at the bottom, so the early 10x of RL compute in that domain adds little to the total bill, the same way the o1-to-o3 RL spend did. Meta’s ScaleRL paper showed that RL compute and performance follow predictable sigmoidal curves. Rewards rise quickly at first, then plateau after thousands of training steps. The curves are useful for choosing training configurations but far from the deep predictive power of pretraining scaling laws.

Source

On the signal-per-FLOP problem: long-horizon agentic training no longer relies on a single end-of-trajectory reward signal. Sub-step verifiers chop long horizons into chains of short ones, and process reward models score each intermediate step to densify the signal further. The information-inefficiency problem still exists for tasks without clean intermediate signals, and long-horizon RL is far from solved in general. It probably will be, eventually. For verifiable agentic work, the long-horizon problem already got engineered around.

Mythos’s system card fits this pattern, with the biggest jumps on cyber and software engineering and substantial gains across the rest. Cyber is where the narrative around the current frontier has centered—Mythos was held back from public release on cyber-risk grounds.

Source

Where did the cyber gains come from? Anthropic’s position is that they were not trained for, but “emerged as a downstream consequence of general improvements in code, reasoning, and autonomy.”

UK AISI supports this position in its evaluation of Spud (GPT-5.5). AISI tested the base model (not the access-controlled GPT-5.5-cyber variant) and found it matched or slightly exceeded Mythos on advanced cyber tasks and on the multi-step corporate network attack simulation. AISI writes that “cyber-offensive skill is emerging as a byproduct of more general improvements in long-horizon autonomy, reasoning, and coding.”

Source

Beren Millidge has a different read. Base models have huge latent capacity sitting around, and RLVR is now strong enough to elicit almost any capability. As Millidge puts it, cyberattack RL is “almost the perfect RLVR task”: success is easy to verify, practice targets are abundant, and agentic coding infrastructure already exists. By his account, the cyber spike on Mythos is not an emergent property of scale but the predictable result of Anthropic running RLVR on cyberattack environments. He notes that Anthropic was unusually well-positioned for cyber, having already built cyber harnesses and SFT data for safety evaluations.

I tend to agree with Millidge here. To be clear, I take Anthropic at their word that they did not train Mythos on cyberattack environments directly. But cyber capability falls out of code plus agency, and Anthropic has invested heavily in agentic coding environments, so a cyber spike is roughly the expected result even without targeting it. Further, AISI finding similar elevation in Spud is not evidence for emergence over targeted training. Frontier labs run similar RLVR programs over similar environments, and converge on similar capabilities. If that is correct, we can forecast some future capability spikes from which environments are being built on the market and by the labs. Epoch AI’s January survey of the RL environments market finds that coding is the largest category and enterprise workflows are a major growth area.

Each new verifiable environment opens a fresh scaling curve that starts cheap. Labs will continue building and acquiring new environments, while incoming compute will deliver further generations of broad pretraining gains.

Thanks to Ryan Hauser and Rudolf Laine for feedback.