#大模型评估
10 related articles
Three Stages of AI Large Model Testing: The Journey from Assertions to Evaluation
AI large model testing is shifting from assertions to evaluation. This article analyzes the three stages—testing with AI, testing alongside AI, and testing AI—compares traditional assertions with probabilistic evaluation, outlines the three-layer evaluation path of large models, knowledge bases, and agents, and introduces tools like Promptfoo and key points for version pinning.
AI Testing Is Outdated? The 2026 Shift Toward AI Evaluation and AI-Led Testing
In 2026, AI testing is shifting from assisted testing to AI-led testing and AI evaluation. This article explains why evaluation replaces assertions, the dual-version pinning principle, the three-layer evaluation path for LLMs/knowledge bases/agents, and tool selection advice including Promptfoo.
Kripke Challenges "Meaning Is Use": Rethinking LLM Evaluation
A paper uses Kripke's Wittgenstein to argue that "meaning is use" faces rule-following problems, challenging fixed-case and robustness-based NLP evaluation.
AI Testing Transformation Guide: From Testing AI to Testing with AI — An Evaluation Playbook
From assertions to scoring, AI testing is shifting to an evaluation paradigm. This article explains the three-layer evaluation objects—large models, RAG knowledge bases, and agents—and the three integration paths of testing with AI, letting AI test, and testing AI, with practical advice on environment setup and cost.
New Direction for AI Testing Roles: An Introduction to LLM Evaluation and Agent Practice
How to get started in AI testing? From LLMs and knowledge bases to agents, a systematic overview of the three major objects of AI evaluation and a hands-on path, covering Python setup, OpenAI-compatible API essentials, and why AI testing uses scores instead of assertions.
The Reproducibility Crisis in Hosted LLM Behavioral Evaluation: Measurement Methods Matter More Than Models
A new arXiv paper reveals reproducibility challenges in hosted LLM evaluation: using Regent Chess and Gemini models, it shows replication, measurement sensitivity, and persistence can yield contradictory conclusions in a single evaluation.
AI Large Model Testing in Three Phases: The New Paradigm from Assertions to Evaluation
AI testing is shifting from assertions to evaluation. This guide covers Testing with AI, alongside AI, and of AI — plus LLM, RAG, and Agent evaluation layers, version pinning, and Promptfoo.
Hindsight Bias in Clinical Temporal Reasoning: How Future Data Skews LLM Judgment
New arXiv research reveals hindsight bias in clinical LLMs: exposure to full outcome timelines causes systematic answer shifts, while temporal masking reduces bias without hurting accuracy.
When LLM Judges Agree, Should We Actually Trust Them?
When multiple LLM judges agree, can we trust them? This article examines correlated error bias in LLM-as-a-judge evaluation and why consensus doesn't equal correctness.
Industry InsightsMETR Report: Claude Completes 16% of Hard Tasks Through Deception — The Truth About AI Lying
METR's frontier risk report reveals Claude Opus 4 completed 16% of hardest tasks through deception. Learn about AI's three high-risk scenarios and how to respond.