Test Set
A fully isolated subset of data used to evaluate a model's true performance on unseen data. It serves as the final standard for measuring a model's generalization ability.
Timeline (last 90 days)
Each sample in the test set needs a clear expected output or scoring criterion, which can be exact match, key information inclusion, or scoring by another LLM according to a rubric (LLM-as-judge).
An initial 50-100 high-quality samples can expose most regression problems.
The first step in evaluation is to collect a representative set of input-output pairs covering core functional paths, common edge cases, and known historical failure cases.
The test set should come from real anonymized messages, including Hinglish and speech-to-text transcripts, asserting that every price-related message triggers an escalation; this test suite must be rerun every time the classifier or prompt is changed.
If both models pass every case, then no matter how large the test set is, it provides no information about the difference between them.
Test sets in RAG scenarios usually need to cover three types of cases: normal questions (to verify basic capability), edge-case questions (such as multi-hop reasoning and questions requiring synthesis across documents), and adversarial questions (such as cases where the retrieval source itself lacks the answer, or distracting documents are present); the latter two types are most likely to expose defects in the coordination between retrieval and generation.
Training a vision model requires dividing the data into a training set, a validation set, and a test set: the training set is used to update model parameters, the validation set is used to tune hyperparameters and select the model version, and the test set is used to evaluate the model's true performance on unseen data.
When evaluating a model, data is usually split into a training set and a test set, with the test set used to measure the true level of generalization.
The correct practice is to complete all hyperparameter selection on an independent validation set, run the final evaluation on the test set only once, and report the results regardless of outcome.
On a small test set of only one hundred scenarios, scores can swing by several percentage points between different runs, and judgment should be made on the full dataset.
All Facts (10)
If both models pass every case, then no matter how large the test set is, it provides no information about the difference between them.
60%UnverifiedEach sample in the test set needs a clear expected output or scoring criterion, which can be exact match, key information inclusion, or scoring by another LLM according to a rubric (LLM-as-judge).
50%UnverifiedThe first step in evaluation is to collect a representative set of input-output pairs covering core functional paths, common edge cases, and known historical failure cases.
50%UnverifiedAn initial 50-100 high-quality samples can expose most regression problems.
50%UnverifiedThe test set should come from real anonymized messages, including Hinglish and speech-to-text transcripts, asserting that every price-related message triggers an escalation; this test suite must be rerun every time the classifier or prompt is changed.
50%UnverifiedTest sets in RAG scenarios usually need to cover three types of cases: normal questions (to verify basic capability), edge-case questions (such as multi-hop reasoning and questions requiring synthesis across documents), and adversarial questions (such as cases where the retrieval source itself lacks the answer, or distracting documents are present); the latter two types are most likely to expose defects in the coordination between retrieval and generation.
50%UnverifiedTraining a vision model requires dividing the data into a training set, a validation set, and a test set: the training set is used to update model parameters, the validation set is used to tune hyperparameters and select the model version, and the test set is used to evaluate the model's true performance on unseen data.
50%UnverifiedWhen evaluating a model, data is usually split into a training set and a test set, with the test set used to measure the true level of generalization.
50%UnverifiedThe correct practice is to complete all hyperparameter selection on an independent validation set, run the final evaluation on the test set only once, and report the results regardless of outcome.
50%UnverifiedOn a small test set of only one hundred scenarios, scores can swing by several percentage points between different runs, and judgment should be made on the full dataset.
50%