Training Text-to-Image Models Without a VAE: A New Direction in Diffusion Architecture

Why Text-to-Image Models Have Long Relied on VAEs

The dominant text-to-image diffusion models today are almost universally built on a shared technical foundation: the Variational Autoencoder (VAE). Whether it's the Stable Diffusion family or other open-source diffusion models, they broadly adopt a Latent Diffusion architecture — first compressing high-resolution images into a low-dimensional latent space using a VAE, then performing diffusion and denoising within that compressed space, and finally decoding back to pixel images through the VAE decoder.

The core motivation behind this design is to reduce computational cost. Running diffusion directly in raw pixel space (e.g., 512×512×3) is enormously expensive. By compressing images into latent space (e.g., 64×64×4), the amount of data the diffusion model needs to process drops dramatically, significantly improving both training and inference efficiency. In this sense, the VAE has been one of the key components that made high-resolution diffusion models practical in the first place.

VAEs (Variational Autoencoders) were introduced by Kingma and Welling in 2013. The core idea is to use an encoder to map input data onto the parameters of a probability distribution (typically Gaussian), sample a latent vector from that distribution, and then reconstruct the original data via a decoder. Compared to ordinary autoencoders, a VAE's latent space is continuous and interpolatable, making it a natural fit for diffusion models. In the Latent Diffusion Model (LDM — the predecessor to Stable Diffusion), the VAE typically applies an 8× compression ratio (in linear dimensions), meaning a 512×512 image is compressed to a 64×64 latent representation, reducing computation by roughly 64×. This compression efficiency is the fundamental reason LDMs can run on consumer-grade GPUs. It's worth noting that the VAE used in Stable Diffusion was specifically optimized with perceptual loss and adversarial training, achieving reconstruction quality far beyond early standard VAEs — though it still suffers from inherent issues like high-frequency detail loss and color shift.

What Removing the VAE Actually Means

Research discussed on Hacker News has proposed a noteworthy direction: training text-to-image models without using a VAE. This may sound like simply swapping out one module, but it actually involves redesigning the entire generation pipeline.

The VAE is efficient, but not without costs. It introduces additional training complexity: a separate encoder-decoder must be trained or loaded, and the VAE's own reconstruction quality becomes a hard ceiling for the entire system — if the VAE discards certain details from an image, no matter how powerful the diffusion model is, that information cannot be recovered in the final output. Furthermore, the latent space is not perfectly semantically aligned, and the distributional properties of latent representations can introduce subtle biases into diffusion training.

Bypassing the VAE would intuitively eliminate the information loss and architectural coupling introduced by these intermediate steps, making the model more end-to-end. But the real challenge is: without latent space compression, how do you control computational costs and ensure generation quality at high resolutions? That is the core question such research must answer.

Potential Technical Approaches

Looking at the broader trajectory of generative model development in recent years, moving away from VAEs is not an isolated attempt. Several related directions have already been explored for simplifying or replacing the latent space component:

• Direct pixel-space diffusion: Performing diffusion directly on raw pixels, paired with a cascaded structure that generates from low resolution up to high resolution in stages, balancing computational cost.
• Lightweight tokenizers as VAE replacements: Using other forms of image discretization or compression to reduce dependence on a specific autoencoder's reconstruction quality.
• End-to-end unified architectures: A single network that handles both representation learning and generation, avoiding the error accumulation that comes with staged training.

Each of these approaches involves trade-offs. The most direct benefit of dropping the VAE is architectural simplicity and a potentially higher quality ceiling, while the primary risk is that training and inference compute demands may increase. The value of this line of research lies in verifying whether a new efficiency equilibrium can be found — one that doesn't sacrifice (and may even improve) generation quality.

Cascaded Diffusion Models are a pixel-space diffusion approach proposed by Google Research in 2021, with Imagen as a flagship example. The basic idea is to first generate a low-resolution image (e.g., 64×64) with a base model, then progressively upscale it to the final resolution (e.g., 1024×1024) through multiple super-resolution diffusion models, distributing the computational burden across stages. This architecture operates entirely in pixel space with no VAE dependency, though it correspondingly increases total inference steps and model parameter counts. The "lightweight tokenizer" direction draws inspiration from visual autoregressive models (such as VQVAE and VQGAN), encoding images as discrete token sequences through vector quantization. This allows generative models to operate in token space rather than continuous latent space, further decoupling representation learning from generative modeling — and laying the groundwork for unified multimodal architectures that can handle both text and image tokens simultaneously.

Potential Impact on the Open-Source Ecosystem

For developers and the research community, the exploration of VAE-free approaches has real practical significance. The current VAE-based pipeline is quite mature, but it has also created a kind of path dependency — a large ecosystem of fine-tuning tools, LoRA, ControlNet, and more are all built around specific latent spaces. If a VAE-free training paradigm with comparable quality were to emerge, it could open up a more flexible design space for models over the long term.

That said, it's worth keeping perspective: the discussion on Hacker News currently has relatively limited traction (11 upvotes, 7 comments), suggesting this is still in early exploration and has yet to form industry-wide consensus or large-scale validation. Research like this is more about opening up possibilities for future architectural evolution than immediately replacing existing solutions.

Conclusion

The VAE has played a behind-the-scenes starring role in text-to-image models for years, but any component that has been uncritically accepted as a default in a technology stack is worth reexamining. The push toward VAE-free approaches is, at its core, asking a deeper question: is the current latent diffusion paradigm truly the optimal solution, or is it an engineering compromise shaped by a particular historical moment? The answer will likely require more experiments and community validation to become clear — but the act of asking the question has already opened a window onto the next wave of generative model evolution.

Note: This article is based on a thread from Hacker News and is extended with background analysis from the field. The discussion of technical approaches represents interpretive context drawn from domain knowledge. For specific implementation details, please refer to the original papers.