Pretraining in the Agentic Era Cannot Rely on NLL Alone: Notes on MAI, Beam, ML4, and Nemotron

In the era of long-horizon agents, pretraining can no longer rely on static NLL / BPB alone. Continued pretraining, or midtraining, should not just mean switching to clean static data for annealing and long-context training.

If you want a model to emit multi-turn interaction and tool-use patterns, those patterns need to exist in the data first. The core role of CPT is to act as a bridge: inject those behavioral priors early, raise the pre-RL starting point as high as possible, and pair lightweight RL with data feedback for fast iteration.

5 released new technical reports. Each team includes a strong group of former Google DeepMind researchers.

8T parameters and 50T tokens, yet the report still relies exclusively on static NLL / BPB. 1 Pro, and two questions the reports leave open.