GoiBERT
A compact pre-trained encoder for Vietnamese that uses two-stage knowledge distillation and processes raw text directly without external tokenization, available in 26M and 49M versions.
Timeline (last 90 days)
GoiBERT comes in two compact versions, 26M and 49M.
GoiBERT processes raw text directly and requires no tokenization preprocessing.
GoiBERT employs two-stage knowledge distillation: the first stage teaches general language representations from a Vietnamese encoder teacher with a shared vocabulary, and the second stage provides task-level guidance from a larger teacher model via soft labels.
The second-stage soft-label generation process does not require manual annotation of the teacher's own training data.
The paper evaluates GoiBERT on eight benchmarks, covering text classification, aspect-level sentiment analysis, natural language inference, and named entity recognition.
GoiBERT's compression is nearly lossless on sentiment analysis and text classification tasks, but there is still noticeable accuracy loss on natural language inference and named entity recognition tasks.
For natural language inference and named entity recognition tasks, the accuracy-cost trade-off needs to be re-evaluated when selecting a model.
All Facts (7)
GoiBERT comes in two compact versions, 26M and 49M.
50%UnverifiedGoiBERT processes raw text directly and requires no tokenization preprocessing.
50%UnverifiedGoiBERT employs two-stage knowledge distillation: the first stage teaches general language representations from a Vietnamese encoder teacher with a shared vocabulary, and the second stage provides task-level guidance from a larger teacher model via soft labels.
50%UnverifiedThe second-stage soft-label generation process does not require manual annotation of the teacher's own training data.
50%UnverifiedThe paper evaluates GoiBERT on eight benchmarks, covering text classification, aspect-level sentiment analysis, natural language inference, and named entity recognition.
50%UnverifiedGoiBERT's compression is nearly lossless on sentiment analysis and text classification tasks, but there is still noticeable accuracy loss on natural language inference and named entity recognition tasks.
50%UnverifiedFor natural language inference and named entity recognition tasks, the accuracy-cost trade-off needs to be re-evaluated when selecting a model.
50%