No Retraining Needed: Autonomous Evolution Doubles VLM Chart Analysis Accuracy

Why Can't Open-Source Vision-Language Models Read Scientific Charts?

Scientific chart analysis is a common task in data analysis—humans can easily grasp the trends behind bar charts, line graphs, and scatter plots, but open-source vision-language models (VLMs) perform poorly on this type of question answering. A new paper points out a neglected fact: a large number of errors stem not from insufficient model capability, but from an imprecise reasoning pipeline. The researchers posed a natural question: without changing a single model parameter, how much accuracy can be recovered?

Their answer: quite a lot. Using an autonomous evolution method to search for reasoning programs and surrounding code, three VLMs all improved on four scientific chart benchmarks, with exact match rates rising from an average of 16.6% to 41.3%.

Method: Freeze the Model, Evolve the "Shell"

Iterative Search Starting from a Baseline Program

The researchers borrowed the autonomous evolution approach used in AlphaEvolve. They start with a baseline reasoning program; in each round, an LLM acts as a judge, shown the current program's code along with some failure cases on training problems, and asked to propose an improved "evolution action." Candidate programs are kept in a small program pool ranked by training accuracy, and the next round of editing is performed on the highest-scoring program in the pool.

The key to this process is that the VLM under evaluation remains completely frozen—all optimization happens at the level of the reasoning program and surrounding code. This means no gradient updates, no additional training data, and no modification of model weights.

AlphaEvolve is an evolutionary code search framework proposed by DeepMind in 2025. Its core idea is to have a large language model act as a "mutation operator": given a program to optimize (such as a matrix multiplication kernel or a scheduling algorithm), the LLM generates modification suggestions based on evaluation feedback; candidate programs are actually executed and scored against the target metric, and high-scoring individuals enter the population and continue to participate in the next round of mutation and crossover. Compared with traditional genetic programming, AlphaEvolve does not require manually designing the search space; instead, it leverages the LLM's code understanding to perform semantic-level edits. This article borrows this closed loop of "LLM proposal + automatic evaluation + population selection," except that the object being optimized is changed from an algorithm kernel to the VLM's reasoning shell, and the evaluation signal is changed from runtime to exact match rate on the training set.

Why Is Exact Match So Important

The research began with an observation: many errors stem from "imprecise answers." The model may understand the gist of the chart, but the answer it gives differs from the reference answer in format or detail. Therefore, the paper uses the proportion of answers that exactly match the reference answer on descriptive questions as its metric. Although strict, this metric directly reflects the influence of the reasoning pipeline on output standardization.

Exact Match (EM) is a strict evaluation metric commonly used in question answering and information extraction tasks: an answer is counted as correct only when the model output is completely identical to the reference answer after normalization; any extra embellishment, unit difference, or formatting deviation is judged wrong. The opposite is loose matching (such as containment, F1 score, or LLM-judge scoring), which is more tolerant of answers that are semantically similar but differently worded. The advantage of exact match is that it is reproducible and requires no human judgment; the disadvantage is that it is extremely sensitive to output standardization. Precisely for this reason, it is well suited to amplifying the variable of "whether the reasoning pipeline is precise": with the same frozen model, if the prompt and post-processing are well designed, outputs will more closely match the reference answer format and EM will rise; otherwise, many points will be lost.

Vision-language models (VLMs) refer to multimodal large models that accept both image and text inputs and generate text outputs; typical examples include GPT-4V, LLaVA, Qwen-VL, and others. Their architecture usually consists of a visual encoder (such as ViT) that splits the image into patches and encodes them as visual tokens, which are then aligned with the language model's token sequence through a projection layer or cross-attention, and finally a language decoder autoregressively generates the answer. Open-source VLMs refer to versions with publicly available weights that can be deployed locally. In scientific chart question answering, a VLM needs to complete multiple subtasks such as OCR, legend recognition, axis reading, numerical comparison, and arithmetic reasoning; a deviation in any step may cause the final answer to mismatch the reference answer—which also explains why refined design of the reasoning pipeline can yield significant gains.

Experimental Results: Average Accuracy Jumps Sharply

The study conducted experiments on three VLMs and four popular scientific chart benchmarks. Accuracy improved in all experimental settings:

• Average accuracy rose from 16.6% for the starting program to 41.3% after evolution
• The largest single improvement reached 55.2 percentage points
• As a side benefit, the evolved programs were 3.8 times faster than the starting program

The reason for the speedup is intuitive: short answers are easier to score and faster to generate. The evolution process tends to produce more concise and standardized outputs, which both improves exact match rates and reduces inference latency.

You may not have noticed that these improvements were achieved with the model completely unchanged. The paper's conclusion is clear: a substantial portion of observed errors originate in the reasoning pipeline itself, and optimizing the pipeline can improve a "frozen" VLM's performance without retraining.

Implications: The Reasoning Pipeline Is an Underestimated Optimization Space

The value of this work lies not only in the numerical improvements, but also in redrawing the boundaries of the problem. When a VLM performs poorly on a specific task, the default reaction is often "the model isn't strong enough; it needs fine-tuning or a larger model." But this study shows that before committing training resources, examining the design of the reasoning program—prompt structure, output format constraints, post-processing logic—may yield gains of a comparable order of magnitude.

Autonomous evolution plays the role of automated search here: it turns "how to write a good reasoning program" from manual trial and error into an iterable optimization process. For application scenarios that need to handle scientific charts, financial report data, or technical documents, this path offers a low-cost, reusable approach to improvement.

Of course, the method has its boundaries: the evolution process relies on failure cases on training problems as feedback signals, and the size of the program pool and the quality of the LLM judge affect search effectiveness. The paper does not disclose how these factors transfer across different domains, which remains a direction worth watching.

One-line Summary

Freezing VLM weights and optimizing only the reasoning program via autonomous evolution raises exact match rates on scientific chart question answering from 16.6% to 41.3%.

Paragraph Summary

The article introduces a study on the scientific chart question answering capabilities of open-source vision-language models. The researchers found that many of the model's errors stem not from insufficient knowledge, but from imprecise outputs of the reasoning pipeline. To address this, they borrowed AlphaEvolve's autonomous evolution approach to iteratively improve reasoning programs and surrounding code using an LLM judge based on failure cases from the training set, all without modifying VLM weights at all. Across three VLMs and four scientific chart benchmarks, exact match rates rose from an average of 16.6% to 41.3%, with a maximum single improvement of 55.2 percentage points, and the evolved programs were also 3.8 times faster. The article notes that this redraws the boundaries of the problem: before committing training resources, optimizing prompt structure, output constraints, and post-processing logic may be an underestimated space for improvement.