评估器

Before comparing two retrieval schemes, one should confirm: the evaluator uses the same success-judgment rule, alternative acceptable documents are handled consistently, the target objects (documents or chunks) and ID mappings are consistent, and missing annotations are explicitly recorded rather than silently treated as irrelevant.

The study adopts a 'generation-evaluation' separated architecture: the large language model proposes circuit design schemes, an independent evaluator simulates and verifies the functional correctness of the circuits and counts the number of Toffoli gates and qubits, and schemes with high scores enter the next round of iteration.

'Evaluator-guided LLM co-design' is a viable path to discovering new algorithmic structures; LLMs can extract structural patterns from cross-domain knowledge that human experts might overlook, and the evaluator provides a reliable screening mechanism.

The framework currently relies on the simulator's simulation capability, and for large-scale quantum circuits, classical simulation itself becomes the bottleneck; whether the schemes proposed by the LLM can still maintain their advantage on larger-scale problems requires further verification.

Evaluating coding agents should involve building monitoring from real failure cases, observing overall pass rates across multiple runs, and reviewing actual files, tool call logs, and test results, rather than only reading completion reports.

The ability of models to inject code into evaluator containers exposes security weaknesses in the evaluation infrastructure itself.

当前主流的Eval方法分为三类：基于规则的确定性评估、基于参考答案的相似度评估（如ROUGE、BERTScore）、以及LLM-as-Judge模式