OrderGPT
A discrete data generative model that learns generation order, built on Any-Order GPT. It learns task-level generation order through attention patterns and is applicable to modalities such as language, human pose, and tabular data.
Timeline (last 90 days)
The OrderGPT paper proposes that a "single adapted order" between left-to-right and arbitrary order is a neglected middle ground, and presents a method that lets the model learn the generation order itself.
OrderGPT is built on Any-Order GPT, which supports prediction in arbitrary order but requires the order to be explicitly given.
OrderGPT's training mechanism includes extracting dependency signals from random prefixes, aggregating them into a dependency graph and deriving priorities, and using a lightweight OrderMLP to predict the order.
OrderGPT explores the generation order early in training and freezes the learned order later for subsequent autoregressive training and generation.
In language modeling, OrderGPT adaptively recovers a dependency structure close to left-to-right.
On human pose data, OrderGPT learns an anatomically organized decomposition, generating joints according to human body structure rather than an arbitrary coordinate order.
On tabular data, OrderGPT discovers decompositions that place statistically and semantically related fields near each other, while improving downstream modeling metrics across all domains.
OrderGPT's code is open-sourced on GitHub (Chenhe-yang/OrderGPT).
All Facts (8)
The OrderGPT paper proposes that a "single adapted order" between left-to-right and arbitrary order is a neglected middle ground, and presents a method that lets the model learn the generation order itself.
50%UnverifiedOrderGPT is built on Any-Order GPT, which supports prediction in arbitrary order but requires the order to be explicitly given.
50%UnverifiedOrderGPT's training mechanism includes extracting dependency signals from random prefixes, aggregating them into a dependency graph and deriving priorities, and using a lightweight OrderMLP to predict the order.
50%UnverifiedOrderGPT explores the generation order early in training and freezes the learned order later for subsequent autoregressive training and generation.
50%UnverifiedIn language modeling, OrderGPT adaptively recovers a dependency structure close to left-to-right.
50%UnverifiedOn human pose data, OrderGPT learns an anatomically organized decomposition, generating joints according to human body structure rather than an arbitrary coordinate order.
50%UnverifiedOn tabular data, OrderGPT discovers decompositions that place statistically and semantically related fields near each other, while improving downstream modeling metrics across all domains.
50%UnverifiedOrderGPT's code is open-sourced on GitHub (Chenhe-yang/OrderGPT).
50%