Multimodal Large Language Model

Multimodal large models convert images into high-dimensional vector representations through a Vision Encoder, which are then fused with the text comprehension capabilities of the language model.

Current large models processing video do not understand it frame by frame; instead, they use intelligent sampling to extract key frames from the video (typically 1–4 frames per second) as image inputs for analysis.

MiMo-V2.6 covers multiple modalities including text, image, and speech, making it an all-modality model.

The architectural core of a multimodal large model must include a large language model (LLM) as its backbone.

If the input is audio, the Vision Encoder can be replaced with a Speech Encoder to encode and align the data before passing it to the large language model.

The image-understanding capabilities of modern multimodal large models have greatly eroded CAPTCHA's assumption that it is "easy for humans, hard for machines," and many CAPTCHAs can be solved with high accuracy.

Current multimodal architectures inherently cause a break in the "perception–reasoning" information channel: visual features are compressed into a limited number of tokens before entering the language model, and fine-grained spatial relations and information for distinguishing similar objects are easily lost.

The rule order in the prompt affects the output of the multimodal large model, which is related to the model's attention mechanism and context processing; content appearing earlier typically receives higher attention weight, and semantic overlap or conflict between rules changes the disambiguation path.

The study proposes a training-free six-class wafer map defect classification framework that leverages a frozen large multimodal model and achieves zero-shot classification through connected-component highlighting and rule-ordering prompts.

The vision encoders of multimodal large models are usually frozen or only lightly adapted, limiting their domain adaptation capability.

Current MLLMs tend to rely on semantic patterns in training data rather than flexibly adapting to visual uncertainty.

The study uses two datasets: SilCC (which modulates naming difficulty through progressively complex abstract shape compositions) and PixCC (which increases recognition difficulty by adding random visual noise) to compare the associative creativity of humans and multimodal large models.

Compared with multimodal large models, the associative descriptions produced by humans have greater semantic distance, higher entropy, and more novel expressions.

In the PixCC dataset, humans showed a strategy shift: from holistic semantic association to describing low-level visual attributes (such as color and texture), whereas MLLMs tended to adhere to semantic-level association.

Humans still significantly outperform multimodal large models in associative creativity, especially when facing visual stimuli that are difficult to name, where humans can produce more diverse and more original descriptions.

When evidence spans multiple images, the models' integration ability declines markedly.

In the MMEnterprise-Bench experiments, the research team evaluated 10 closed-source and open-source multimodal large models.

The reviewer of an image generation agent relies on multimodal large models or dedicated scoring models, with common quantitative dimensions including compositional soundness, clarity, style consistency, and prompt adherence.

In interpretability research, visual attribution refers to methods that attribute model outputs to specific image regions or visual features, with common approaches including gradient saliency, attention rollout, and perturbation contrast.

The OpenVLHarness team conducted controlled experiments on 22 benchmarks and found that simply increasing the inference budget of the base model yields limited gains.