top_k
A sampling parameter in the decoding stage of large language models that directly truncates to the k candidates with the highest probabilities, controlling the degree of concentration in the output.
Timeline (last 90 days)
top_k directly truncates to the k most probable candidate tokens, and the smaller k is, the more concentrated the output.
When temperature, top_p and top_k are used in combination, their effects are hard to predict, and this is the main source of output style drift for the same prompt under different parameters.
After the removal of sampling parameters, the optimization some teams used of limiting the candidate range with top_k to reduce inference cost is no longer available, and they need to turn to caching, batching or tiered model routing to control costs.
All Facts (3)
top_k directly truncates to the k most probable candidate tokens, and the smaller k is, the more concentrated the output.
50%UnverifiedWhen temperature, top_p and top_k are used in combination, their effects are hard to predict, and this is the main source of output style drift for the same prompt under different parameters.
50%UnverifiedAfter the removal of sampling parameters, the optimization some teams used of limiting the candidate range with top_k to reduce inference cost is no longer available, and they need to turn to caching, batching or tiered model routing to control costs.
50%