Unigram
A subword tokenization algorithm that assumes a vocabulary composed of independent subword units and selects an optimal tokenization scheme through a probabilistic model. It is commonly used in the SentencePiece tool.
Core Facts
Timeline (last 90 days)
Subword tokenization cuts words into smaller segments, common algorithms include BPE, WordPiece and Unigram, and SentencePiece is one language-agnostic engineering implementation among them.
The segmentation decisions of most practical tokenizers are localized, with natural independent boundaries between adjacent segments, and finding and exploiting these boundaries enables incremental tokenization without disabling the reference implementation.
The paper compares LCT with BPE, Unigram, and parity-aware BPE under the setting of 104 languages and a 200K vocabulary.
The optimization objectives of the mainstream tokenization algorithms BPE and Unigram are essentially both compression: minimizing the corpus encoding length under a limited vocabulary.
Compared with BPE, Unigram, and parity-aware BPE, LCT improves the aggregate score on four multilingual downstream benchmarks by 1.48, 1.83, and 2.00 points, respectively.
The SentencePiece toolkit includes a complete implementation of Unigram language model tokenization.
The choice among the three tokenization algorithms involves different trade-offs among vocabulary size, rare-word coverage, and cross-lingual generalization ability.
The Unigram language model tokenization method was proposed by Kudo in 2018, starting from a large vocabulary and progressively pruning it to retain the set of subwords that minimizes the likelihood loss on the corpus.
All Facts (8)
Subword tokenization cuts words into smaller segments, common algorithms include BPE, WordPiece and Unigram, and SentencePiece is one language-agnostic engineering implementation among them.
65%UnverifiedThe segmentation decisions of most practical tokenizers are localized, with natural independent boundaries between adjacent segments, and finding and exploiting these boundaries enables incremental tokenization without disabling the reference implementation.
50%UnverifiedThe paper compares LCT with BPE, Unigram, and parity-aware BPE under the setting of 104 languages and a 200K vocabulary.
50%UnverifiedThe optimization objectives of the mainstream tokenization algorithms BPE and Unigram are essentially both compression: minimizing the corpus encoding length under a limited vocabulary.
50%UnverifiedCompared with BPE, Unigram, and parity-aware BPE, LCT improves the aggregate score on four multilingual downstream benchmarks by 1.48, 1.83, and 2.00 points, respectively.
50%UnverifiedThe SentencePiece toolkit includes a complete implementation of Unigram language model tokenization.
50%UnverifiedThe choice among the three tokenization algorithms involves different trade-offs among vocabulary size, rare-word coverage, and cross-lingual generalization ability.
50%UnverifiedThe Unigram language model tokenization method was proposed by Kudo in 2018, starting from a large vocabulary and progressively pruning it to retain the set of subwords that minimizes the likelihood loss on the corpus.
50%