Generalization
The ability of a machine learning model to make correct judgments on data it has not seen before. It is a core metric for evaluating model performance.
Core Facts
When a model's training data contains benchmark test questions or highly similar content, the model may achieve high scores through memorization rather than genuine understanding.
The quality and diversity of training data directly affects the reliability of model outputs, and training data containing large amounts of low-quality, contradictory, or false information can cause models to learn incorrect patterns.
If evaluation samples (including near-duplicate samples) appear in the training data, scores reflect memorization ability rather than genuine skill.
Timeline (last 90 days)
The monitoring effective rank can serve as a metric for detecting whether a model has entered the generalization phase, but one cannot expect to "force" grokking by enforcing low rank.
Artificially suppressing rank does not accelerate generalization and may in fact hinder grokking.
Networks that only memorize without generalizing show a much smaller degree of effective-rank compression.
Approximation is not the same as generalization: the existence of a wide network that can fit the training data and whether that network can perform well on new data are two different things.
The key to determining whether a new relation can generalize is not whether the fact can be recalled, but whether its shared structure is easy to read out linearly in the pretrained representation.
The low consistency between the feedback score and the held-out score is essentially a concrete manifestation of overfitting and the generalization gap in the scenario of agent self-improvement.
When evaluating a model, data is usually split into a training set and a test set, with the test set used to measure the true level of generalization.
If evaluation samples (including near-duplicate samples) appear in the training data, scores reflect memorization ability rather than genuine skill.
The corresponding core concepts in AI are generalization and out-of-distribution reasoning; a model performing well within its training distribution does not guarantee it can handle novel problems not encountered during training.
Overfitting refers to the phenomenon where a model performs extremely well on the training set but generalizes poorly to new data, and this risk is especially prominent in pharmacy datasets with relatively small sample sizes.
1 more timeline events
All Facts (12)
When a model's training data contains benchmark test questions or highly similar content, the model may achieve high scores through memorization rather than genuine understanding.
80%VerifiedThe quality and diversity of training data directly affects the reliability of model outputs, and training data containing large amounts of low-quality, contradictory, or false information can cause models to learn incorrect patterns.
70%VerifiedIf evaluation samples (including near-duplicate samples) appear in the training data, scores reflect memorization ability rather than genuine skill.
65%UnverifiedThe monitoring effective rank can serve as a metric for detecting whether a model has entered the generalization phase, but one cannot expect to "force" grokking by enforcing low rank.
50%UnverifiedNetworks that only memorize without generalizing show a much smaller degree of effective-rank compression.
50%UnverifiedArtificially suppressing rank does not accelerate generalization and may in fact hinder grokking.
50%UnverifiedApproximation is not the same as generalization: the existence of a wide network that can fit the training data and whether that network can perform well on new data are two different things.
50%UnverifiedThe key to determining whether a new relation can generalize is not whether the fact can be recalled, but whether its shared structure is easy to read out linearly in the pretrained representation.
50%UnverifiedThe low consistency between the feedback score and the held-out score is essentially a concrete manifestation of overfitting and the generalization gap in the scenario of agent self-improvement.
50%UnverifiedWhen evaluating a model, data is usually split into a training set and a test set, with the test set used to measure the true level of generalization.
50%UnverifiedThe corresponding core concepts in AI are generalization and out-of-distribution reasoning; a model performing well within its training distribution does not guarantee it can handle novel problems not encountered during training.
50%UnverifiedOverfitting refers to the phenomenon where a model performs extremely well on the training set but generalizes poorly to new data, and this risk is especially prominent in pharmacy datasets with relatively small sample sizes.
50%