n-gram
A technique that splits text into segments of n consecutive tokens, used to detect near-duplicate content and commonly applied in data contamination detection.
Timeline (last 90 days)
Contamination detection is performed on each instruction set using two checks: the problem text is normalized by taking the first 80 characters of the HumanEval docstring and compared; the reference answer, after removing comments and docstrings, is checked for whether more than half of the 13-token n-grams match.
The choice of n in n-gram matching is a trade-off: too small an n (such as 3) produces many false positives due to common phrases; too large an n (such as 50) is overly sensitive to reasonable code rewrites and prone to missed detections; 13 is an empirically折中 choice in code scenarios.
The model places n-gram lookup tables at layers 1 and 14, with 24 hashes each indexing approximately 16 million rows, totaling 196 billion parameters, exceeding one-third of the model's weights.
Approximately 200 GB of n-gram tables are stored in regular system memory rather than on the GPU, with background transfer prefetching the rows about to be used.
A Qwen Flash model released by Alibaba at the end of August also places 51 billion n-gram parameters in system memory for execution.
Blindly chasing n-gram overlap drags entailment precision down from 0.522 to 0.266.
All Facts (6)
Contamination detection is performed on each instruction set using two checks: the problem text is normalized by taking the first 80 characters of the HumanEval docstring and compared; the reference answer, after removing comments and docstrings, is checked for whether more than half of the 13-token n-grams match.
50%UnverifiedThe choice of n in n-gram matching is a trade-off: too small an n (such as 3) produces many false positives due to common phrases; too large an n (such as 50) is overly sensitive to reasonable code rewrites and prone to missed detections; 13 is an empirically折中 choice in code scenarios.
50%UnverifiedThe model places n-gram lookup tables at layers 1 and 14, with 24 hashes each indexing approximately 16 million rows, totaling 196 billion parameters, exceeding one-third of the model's weights.
50%UnverifiedApproximately 200 GB of n-gram tables are stored in regular system memory rather than on the GPU, with background transfer prefetching the rows about to be used.
50%UnverifiedA Qwen Flash model released by Alibaba at the end of August also places 51 billion n-gram parameters in system memory for execution.
50%UnverifiedBlindly chasing n-gram overlap drags entailment precision down from 0.522 to 0.266.
50%