Concept分词器fertility
Fertility
A metric measuring tokenizer efficiency, defined as the average number of subword tokens into which each word is split; a higher fertility indicates lower tokenization efficiency.
A metric measuring tokenizer efficiency, defined as the average number of subword tokens into which each word is split; a higher fertility indicates lower tokenization efficiency.