Nathan Lambert
Researcher at the Allen Institute for AI (AI2) and author of the Interconnects blog. His work focuses on open-source large language models, reinforcement learning, and AI policy research.
Timeline (last 90 days)
Nathan Lambert says there are rumors that Meta will release a new open-weight model again.
Nathan Lambert believes that Chinese companies releasing open-source models is a pragmatic commercial choice aimed at entering the global conversation and serving paid demand in the US market, rather than government intervention or altruism.
Nathan Lambert believes China is in no way inferior to the US in AI talent, and that the real shortcoming is compute; he gives the example that an ordinary OpenAI researcher can call on roughly 5,000 H100s of compute at any time.
Nathan Lambert believes there is some basis for viewing distillation as a primary means by which Chinese open-source models catch up, but that this is oversimplified; he notes that post-training has now shifted to being mainly about reinforcement learning environments, and that simply using the strongest model to generate completions for SFT and watching scores rise is hard to reproduce in the open literature.
Nathan Lambert points out that China has a large number of API proxies and model proxies with enormous token usage, and there are even cases such as Kimi offering Claude services to users within its product, which makes him keep error bars on the conclusion that "distillation is not a major factor."
Nathan Lambert believes that regulatory action against distillation does more harm than good, that terms of service are not even a signed contract, and that the only thing that can be done is to revoke access.
Nathan Lambert proposes that open and closed models operate on different exponential timescales: closed-source labs are highly integrated and can be first into high-value markets (such as coding), with revenue going from zero to the hundred-billion-dollar level; the open-source ecosystem develops more slowly due to poor information flow and a lack of economies of scale.
Nathan Lambert's new lab, Trillium, is positioned as a nonprofit open lab whose core work is building open-source RL environments for high-leverage tasks and publicly releasing the tooling and trajectories for RSI (recursive self-improvement).
Nathan Lambert takes the TMax work as an example: using frontier models to generate terminal tasks, using multiple prompts to generate diversity, and using model rollouts to discover and eliminate buggy data—the whole process requires an investment of thousands to tens of thousands of dollars.
Nathan Lambert believes the science of pretraining itself is not that high a barrier, and that one does not need to spend a billion dollars to get a very good base model; many startups fork and train models based on open-source tools such as Megatron.
12 more timeline events
All Facts (20)
Nathan Lambert's new lab, Trillium, is positioned as a nonprofit open lab whose core work is building open-source RL environments for high-leverage tasks and publicly releasing the tooling and trajectories for RSI (recursive self-improvement).
50%UnverifiedNathan Lambert believes the science of pretraining itself is not that high a barrier, and that one does not need to spend a billion dollars to get a very good base model; many startups fork and train models based on open-source tools such as Megatron.
50%UnverifiedNathan Lambert believes there is some basis for viewing distillation as a primary means by which Chinese open-source models catch up, but that this is oversimplified; he notes that post-training has now shifted to being mainly about reinforcement learning environments, and that simply using the strongest model to generate completions for SFT and watching scores rise is hard to reproduce in the open literature.
50%UnverifiedNathan Lambert points out that China has a large number of API proxies and model proxies with enormous token usage, and there are even cases such as Kimi offering Claude services to users within its product, which makes him keep error bars on the conclusion that "distillation is not a major factor."
50%UnverifiedNathan Lambert believes that regulatory action against distillation does more harm than good, that terms of service are not even a signed contract, and that the only thing that can be done is to revoke access.
50%UnverifiedNathan Lambert proposes that open and closed models operate on different exponential timescales: closed-source labs are highly integrated and can be first into high-value markets (such as coding), with revenue going from zero to the hundred-billion-dollar level; the open-source ecosystem develops more slowly due to poor information flow and a lack of economies of scale.
50%UnverifiedNathan Lambert takes the TMax work as an example: using frontier models to generate terminal tasks, using multiple prompts to generate diversity, and using model rollouts to discover and eliminate buggy data—the whole process requires an investment of thousands to tens of thousands of dollars.
50%UnverifiedNathan Lambert believes that closed-source frontier labs such as OpenAI, Anthropic, and Google hold clear advantages in three resources—compute, data, and talent—and that the real difference lies in organizational capability.
50%UnverifiedNathan Lambert says Meta has returned to a positive trajectory after a complete organizational reset.
50%UnverifiedNathan Lambert believes OpenAI kicked off the reasoning model revolution, and that the inference scaling law curve was OpenAI's peak moment; Anthropic focused on coding capability for a long time and was ultimately rewarded by the breakout of Claude Code.
50%UnverifiedNathan Lambert believes that Chinese companies releasing open-source models is a pragmatic commercial choice aimed at entering the global conversation and serving paid demand in the US market, rather than government intervention or altruism.
50%UnverifiedNathan Lambert believes China is in no way inferior to the US in AI talent, and that the real shortcoming is compute; he gives the example that an ordinary OpenAI researcher can call on roughly 5,000 H100s of compute at any time.
50%UnverifiedOpen-source model researcher Nathan Lambert said of MuseSpark Open that before this, US models still lagged behind Chinese models, and MuseSpark Open has at least halved the gap.
50%UnverifiedNathan Lambert of Interconnects pointed out that obtaining high-quality, diverse fine-tuning prompts is something open-source projects find hard to replicate.
50%UnverifiedNathan Lambert is the author of Interconnects and co-founder of Trillium Labs.
50%UnverifiedInterconnects is a technical newsletter operated by Nathan Lambert, primarily focusing on topics such as reinforcement learning from human feedback (RLHF), model alignment, and the open AI ecosystem.
50%UnverifiedNathan Lambert previously worked on the alignment team at Hugging Face.
50%UnverifiedNathan Lambert is an AI2 researcher and author of the Interconnects blog.
50%UnverifiedNathan Lambert says there are rumors that Meta will release a new open-weight model again.
50%UnverifiedNathan Lambert submitted written testimony to the U.S. Congress on the global power dynamics of open-source models.
50%