UNet

Reverse generation learns a denoising network that starts from pure noise, gradually predicts and removes noise, and ultimately generates clear samples.

The core insight of Prompt-to-Prompt is that the attention maps of the cross-attention layers in diffusion models directly control the spatial layout of the image.

The core training objective of DDPM is often simplified to the mean squared error of predicting the noise ε.

The original diffusion model paper used UNet as the noise prediction network and incorporated attention mechanisms.

In diffusion models, the input to UNet is the noisy image X_t, and the output is a predicted noise map of the same size as the input; the time step t is embedded via sinusoidal positional encoding and injected into each residual block.

UNet was originally used for medical image segmentation and has a symmetric U-shaped structure comprising an encoder, a decoder, and skip connections.

The diffusion model uses UNet as its main structure, including downsampling, convolutional layers, upsampling and feature concatenation.

The main model (checkpoint) is a complete weight package containing UNet, CLIP and VAE that determines the base art style and capabilities; the text encoder converts prompts into conditioning vectors; LoRA is a small low-rank adaptation incremental weight; and VAE encodes and decodes between latent space and pixel space.

The UNet part of Stable Diffusion has about 860 million parameters.

Skip connections pass the high-resolution features from each level of the encoder directly to the corresponding level of the decoder, compensating for the spatial details lost during downsampling, and significantly improving the accuracy of segmentation edges for fine structures such as tooth contours and alveolar bone boundaries.

In the encoder-decoder architecture, the encoder gradually reduces the spatial resolution of feature maps through convolution and pooling operations to extract high-level semantic information, and the decoder restores low-resolution features to the original image size through upsampling to output pixel-wise segmentation results.

InstantFusion's shared interface makes cross-architecture online policy distillation possible, allowing a teacher model (e.g., a large DiT) and a student model (e.g., a lightweight UNet) to exchange intermediate states in a shared latent space.

DiT replaces convolutional layers with Transformer blocks, outperforming earlier diffusion models that used UNet architectures in high-resolution generation.

The weights of image generation models are typically divided into multiple components such as a text encoder, a diffusion backbone network (UNet or DiT), and a decoder (VAE).

In a semantic segmentation task, the baseline model UNet achieved an original mIoU score of 0.817.

Compared to the original weight averaging, the final solution improved mean mIoU by approximately three points, but certain stacked combinations actually decreased the score, meaning more is not always better.

In the first round of optimization, the changes proposed by the Agent focused on directional convolutions, deep supervision, and training strategies, improving mIoU from 0.817 to 0.82.

Traditional ControlNet requires copying and fine-tuning all weights of the UNet encoder, resulting in a size comparable to the base model and relatively high deployment costs.

The model is built on flow matching with a spatiotemporal UNet architecture, using velocity prediction and 50-step Euler sampling.

Diffusion model backbone networks (UNet/DiT) have large parameter counts and can generally tolerate more aggressive quantization, while retaining higher precision (such as INT8 or even FP16) for the text encoder as a semantic entry point is usually more cost-effective.