HAD-VLM: Using Diffusion Models to Solve the Numerical Discretization Problem in Medical Vision-Language Models

Quantitative measurement of medical images has long relied on manual annotation by physicians, and the rise of vision-language models (VLMs) has made it possible to "complete measurements using natural language instructions." But there is a fundamental contradiction here: measurement results are continuous values, while the text output of mainstream VLMs consists of discrete tokens. Forcing continuous values into a discrete vocabulary makes precision loss almost inevitable. A study from an rss source proposes HAD-VLM, attempting to bridge this gap with a hybrid autoregressive-diffusion architecture.

Why Discrete Tokens Cannot Predict Continuous Measurements

The typical approach of existing medical VLMs is to generate measurement values as a string of text tokens. For example, for interventricular septal thickness of 1.36 mm, the model outputs "1" "." "3" "6" in sequence. The problem is that the token classification loss does not directly optimize numerical error, and the model's ability to distinguish between 1.36 and 1.37 is diluted by the vocabulary structure. More troublesome is that measurement tasks often require simultaneously providing anatomical endpoint coordinates and lengths. These quantities lack continuity constraints in discrete space, making outputs where coordinates and lengths contradict each other prone to occur.

The paper categorizes this as "the mismatch between continuous measurement targets and discrete token prediction." This judgment hits the nail on the head: it is not that the model is not large enough, but that the form of the output representation is misaligned with the essence of the task.

Vision-language models (VLMs) are a class of multimodal models that process both images and text, typically consisting of a visual encoder that maps images into feature sequences, which are then passed to a large language model to generate text in an autoregressive manner. Medical VLMs build on this by fine-tuning with medical image-text pairs as instructions, enabling them to answer diagnostic questions or execute measurement instructions. Their output layer is a categorical distribution over a discrete vocabulary, with each token sampled from a fixed vocabulary, making them naturally suited for generating text but not for expressing high-precision real numbers.

Autoregressive generation refers to the model predicting the next token one at a time, with each step depending on the previously generated context. This mechanism ensures language fluency and instruction-following ability, but each numerical value is split into individual characters predicted one by one, with no numerical continuity constraint between characters, and the model cannot directly output a real number through a single forward computation.

HAD-VLM's Hybrid Architecture Design

The core idea of HAD-VLM is to simultaneously retain two pathways in a shared backbone: autoregressive text generation and diffusion-based continuous prediction. On the visual side, visual prompt tuning is used to adapt the visual encoder to the special textures and noise distribution of ultrasound images; on the measurement side, a diffusion process generates a five-component representation—caliper endpoint coordinates plus measurement length.

The ingenuity of this design lies in the division of labor: the language interaction part continues to use autoregression, ensuring the flexibility of prompt conditions and that medical question-answering capability is not compromised; the numerical part is handed to diffusion, directly optimized in continuous space. The two share the backbone, avoiding the semantic fragmentation that would result from two separate models operating independently.

Coordinate-Length Consistency Post-Processing

What diffusion outputs is a raw estimate. The paper further proposes a coordinate-length consistency mechanism: the distance calculated from endpoint coordinates should be cross-validated with the predicted length, combined with ensemble prediction for refinement. This step specifically targets the self-contradiction problem common in discrete methods, using geometric constraints to pull numerical values back into a reasonable range.

Diffusion models are a class of generative models that recover target samples from Gaussian noise through gradual denoising. Unlike autoregressive token-by-token generation, diffusion operates in continuous space and outputs real-valued vectors, so it can directly optimize numerical error. In HAD-VLM, the diffusion head is used to generate a five-dimensional continuous vector (four endpoint coordinates plus one length), and its training objective is to minimize the continuous loss between predicted values and ground-truth measurements, rather than token classification loss.

Visual prompt tuning is a parameter-efficient visual adaptation method: most of the pretrained visual encoder is frozen, and only a small number of learnable prompt vectors are inserted into the input sequence, allowing the model to adapt to the texture and noise distribution of new modalities (such as ultrasound) with minimal parameters. Compared with full fine-tuning, it reduces the risk of overfitting and makes it easier to serve both the language and measurement pathways in a shared backbone.

Experimental Results: Error Reduced by 20-30%

In the four echocardiographic measurement tasks on EchoNet-LVH, the mean absolute errors of HAD-VLM were: IVSd 1.36 mm, LVIDd 2.77 mm, LVIDs 3.05 mm, and LVPWd 1.16 mm. Compared with the Vicuna autoregressive baseline, the error was reduced by 21.4%–37.6%, and it outperformed fine-tuned Qwen3-VL and direct MLP regression on all four tasks.

Cross-modal validation is also noteworthy: in the X-ray vertebral measurement task, the mean absolute error dropped from 11.4 pixels to 6.7 pixels, nearly halving. This shows that the benefits of the continuous prediction pathway are not limited to ultrasound but are generalizable to anatomical measurement tasks. Meanwhile, medical question-answering performance was preserved on the evaluation datasets, addressing the concern of whether "introducing diffusion would sacrifice language capability."

Mean absolute error (MAE) is a commonly used metric for measuring continuous measurement precision, calculated as the average of the absolute deviations between predicted values and true values, with units consistent with the measured quantity (such as millimeters or pixels); smaller values indicate higher precision. This paper uses MAE as the primary evaluation criterion because it directly reflects numerical error, unlike classification accuracy which depends on discrete thresholds.

EchoNet-LVH is a public echocardiographic dataset containing annotations related to left ventricular hypertrophy, commonly used to evaluate cardiac structure measurement algorithms. The four tasks in the paper correspond to interventricular septal thickness (IVSd), left ventricular internal dimension at end-diastole (LVIDd), left ventricular internal dimension at end-systole (LVIDs), and left ventricular posterior wall thickness (LVPWd), all of which are key indicators for clinical assessment of left ventricular structure and function.

Some Sober Reflections on Medical VLMs

The value of this work goes beyond a few error numbers. It points to a more general problem: when VLMs move from "describing images" to "quantifying images," the representational bottleneck of discrete tokens will recur. The diffusion head is one solution, but not the only one—regression heads, continuous embeddings, and mixture of experts are all potential directions. The significance of HAD-VLM lies in providing a reproducible architectural paradigm and demonstrating that continuous prediction and language generation can coexist in the same backbone.

It should be noted that the experiments are concentrated on two types of tasks—cardiac ultrasound and vertebral measurement—and sample size and clinical diversity remain limited; the coordinate-length consistency mechanism relies on the specific annotation form of caliper endpoints, and may require redesign when generalized to free-form lesion measurement. Introducing diffusion prediction into VLMs is a directional step, but there is still room for validation before clinical-grade robustness is achieved.

For developers focused on medical AI, perhaps the most practical insight from this paper is: when building quantitative VLMs, first ask whether the output representation matches the essence of the task, then decide how many parameters to stack.

One-line Summary

HAD-VLM uses a hybrid autoregressive-plus-diffusion architecture to enable medical VLMs to directly output continuous measurement values, reducing error by 20-30%.

Paragraph Summary

Quantitative measurement of medical images has long relied on manual annotation, while mainstream vision-language models (VLMs) output text as discrete tokens, making it difficult to precisely express continuous values; token classification loss also does not directly optimize numerical error, leading to contradictions between coordinates and lengths. A study from an rss source proposes HAD-VLM, which simultaneously retains two pathways in a shared backbone: autoregressive text generation and diffusion-based continuous prediction. On the visual side, visual prompt tuning adapts to ultrasound images; on the measurement side, a diffusion process generates a five-component representation of endpoint coordinates plus length, with a coordinate-length consistency mechanism and ensemble prediction for post-processing. On four echocardiographic tasks in EchoNet-LVH, mean absolute error was reduced by 21.4%–37.6% compared with the Vicuna baseline, X-ray vertebral measurement error dropped from 11.4 pixels to 6.7 pixels, and medical question-answering capability was preserved.