Why Is GEPA Merge Unstable? Mechanism-Level Diagnosis Reveals Configuration Sensitivity and Behavioral Probes

Title

Why Is GEPA Merge Unstable? Mechanism-Level Diagnosis Reveals Configuration Sensitivity and Behavioral Probes

Meta Description

GEPA merge operator unstable across models? Mechanism-level diagnosis finds configuration sensitivity is the main cause, static features cannot predict merge decisions, behavioral probes can avoid high-risk failures but do not surpass NoMerge.

Article Content

GEPA (Generalized Evolutionary Prompt Adaptation) is one of the strongest reflective prompt optimizers for language model programs today, but its system-aware merge operator is extremely unstable across models: the paper has observed that merging can improve the average performance of gpt-4.1-mini yet degrade that of Qwen3-8B, and this asymmetry has remained unexplained. A recent mechanism-level diagnostic study delves into this issue and, through systematic experiments across multiple datasets and models, provides an answer that differs from intuition.

Core Finding: Merge Instability Stems from Configuration Sensitivity, Not Operator Defects

The study investigates five datasets and two models (gpt-4.1-mini and Qwen3-8B), scanning nine configurations of merge timing and merge operator. The results show that when configured appropriately, merging can simultaneously improve both models' performance. This counterintuitive conclusion directly challenges the assumption that "the merge operator has inherent defects"—the instability is more a manifestation of configuration sensitivity than a problem with the operator itself.

According to the original paper authors' description in the rss source, the average improvement of merging for gpt-4.1-mini contrasts sharply with the degradation for Qwen3-8B, but this had not been explained before. The new diagnostic work reproduces this phenomenon through configuration scanning and demonstrates that changing merge timing or operator selection is sufficient to reverse the outcome.

This means that in practice, if merging leads to performance degradation, the first troubleshooting direction should be the configuration combination, not abandoning the merge mechanism.

GEPA and reflective prompt optimization: GEPA (Generalized Evolutionary Prompt Adaptation) is a prompt optimization framework based on evolutionary strategies. Its core loop is to have the language model "reflect" on the current prompt's performance on training samples, generate improved candidate prompts, and then decide whether to retain them through evaluation. Unlike traditional methods that rely on gradients or random mutations, GEPA leverages the model's own reasoning ability for directed modifications, hence called a "reflective" optimizer.

Merge operator: In evolutionary prompt optimization, the merge operator is responsible for integrating two or more well-performing candidate prompts into a new prompt, typically by having the model compare the pros and cons of different prompts and synthesize their strengths. The original intent of merging is to accelerate convergence and improve final performance, but its effect highly depends on the complementarity of the merged prompts and the model's ability to follow the synthesized instructions.

Static Feature Gating Fails: Prompt-Level Features Cannot Predict Merge Decisions

The study further examines whether prompt-level features can predict GEPA's merge-accept decisions. The result is no—static-feature gating is explicitly ruled out. In other words, it is impossible to judge solely from the prompt's textual features (such as length, perplexity, semantic complexity, etc.) whether merging will bring positive gains.

This finding has direct methodological implications: any approach that attempts to pre-screen merge operations through simple rules or lightweight classifiers will be difficult to make work reliably. The benefit of merging depends on the dynamic interaction between model and task, not on static prompt attributes.

Static feature gating: Refers to using static attributes of the prompt text (such as length, perplexity, vocabulary diversity, semantic complexity, or embedding vectors) to train a lightweight classifier or set rules before executing the merge operation, in order to predict whether merging will improve performance and decide whether to accept the merge accordingly. This method is low-cost and highly interpretable, but its premise is that a stable mapping exists between merge benefits and the prompt's static features.

Prompt-level features: As opposed to task-level or model-level features, prompt-level features only describe the textual form of the prompt itself, without involving the model's internal state or task data distribution. Diagnostic research shows that merge benefits depend on the dynamic interaction between model and task, so prompt-level features alone are insufficient to reliably predict merge decisions.

Behavioral Probe: A Feasible Risk-Avoidance Approach

As a proof of concept, the researchers built a behavior-based probe based on the above diagnosis. This probe does not predict whether merging is "beneficial," but instead identifies high-risk tasks and abstains on them, thereby avoiding substantial merge failures across multiple random seed runs.

However, the paper also candidly points out that the probe does not stably surpass the NoMerge baseline. In other words, it reduces worst-case risk but has not yet brought significant improvement in average performance. This leaves clear room for future work—how to design more precise abstention strategies, or combine configuration search to further unleash the potential of merging.

Behavioral probe: Unlike static feature gating, a behavioral probe does not directly predict whether merging is "beneficial," but instead identifies high-risk tasks by observing changes in the model's behavior before and after merging (such as output distribution shifts, confidence fluctuations, response consistency on specific task samples, etc.), and actively abstains on these tasks, i.e., skips the merge operation. Its design goal is to minimize worst-case performance loss rather than maximize average gains.

NoMerge baseline: Refers to a prompt optimization process that does not perform any merge operations at all, relying only on reflective mutation and selection. It is the reference standard for evaluating the net benefit of the merge mechanism. Although the behavioral probe can avoid substantial failures, it has not yet stably surpassed this baseline, indicating that its abstention strategy is still conservative or not precise enough.

Implications for Prompt Optimization Practitioners

This diagnostic work conveys several actionable signals:

1. Do not reject merging because of a single negative result: The nine-configuration scan shows that adjusting merge timing or operator can change the conclusion. It is recommended to perform a small-scale configuration search on a fixed dataset before deciding whether to enable merging.
2. Abandon static feature pre-screening: Prompt-level features cannot predict merge decisions, and simple rule-based gating is unreliable.
3. Prioritize risk control: If merge failures are unacceptable, a behavioral-probe-style abstention mechanism is currently a feasible compromise, although it does not guarantee performance gains.
4. Focus on the configuration space: Merge instability is more likely a configuration problem; systematically exploring timing and operator combinations is more direct than tuning the prompt itself.

The code is open-sourced as GEPA-MergeProbe (https://github.com/ChongZZZZ/GEPA_MERGE), available for reproduction and extension.

Configuration scan: Refers to systematically enumerating combinations of merge timing (e.g., which generation to merge at, number of merge candidates) and merge operator (e.g., different synthesis strategies) on a fixed dataset and model, and evaluating performance under each configuration. The nine-configuration scan is a small-scale, low-cost experimental design aimed at revealing sensitive dimensions in the configuration space, helping practitioners quickly locate merge configurations suitable for their own scenarios, rather than relying on default settings or single experiment results.

Summary

The cross-model instability of the GEPA merge operator is not incurable. Mechanism-level diagnosis shows that configuration sensitivity is the main cause, static feature gating is infeasible, and behavioral probes can avoid high-risk failures but have not yet surpassed NoMerge. For teams using GEPA, prioritizing merge configuration scanning and introducing risk abstention mechanisms is currently the most pragmatic path.

One-line Summary

GEPA merge operator's cross-model instability is mainly due to configuration sensitivity rather than operator defects; static feature gating fails; behavioral probes can avoid risk but do not surpass NoMerge.

Paragraph Summary

A mechanism-level diagnostic study investigated the causes of GEPA merge operator's unstable cross-model performance. Through nine configuration scans on five datasets and two models, the study found that when configured appropriately, merging can simultaneously improve the performance of gpt-4.1-mini and Qwen3-8B, indicating that instability stems from configuration sensitivity rather than inherent operator defects. The study also examined whether prompt-level static features can predict merge-accept decisions, concluding they cannot, and static feature gating was ruled out. As a proof of concept, the researchers built a behavioral probe that avoids substantial merge failures by abstaining on high-risk tasks, but the probe does not stably surpass the NoMerge baseline. Practical recommendations include prioritizing configuration search of merge timing and operator, abandoning static feature pre-screening, and introducing risk abstention mechanisms.