Bengali
A low-resource language with more than 250 million speakers and limited annotated resources, often used as a representative language in low-resource NLP research.
Timeline (last 90 days)
Transferring from a distant language such as English to Bengali coreference resolution degrades performance.
Bengali has more than 250 million speakers.
Transferring from Hindi to Bengali coreference resolution did not yield consistent gains across models.
Many FLORES sentences are not in Unicode NFC normalized form: 609 of 1,012 Bengali sentences, 573 Punjabi sentences, and 93 Hindi sentences; failing to normalize first causes token count drift.
The trained model achieves a per-token negative log-likelihood (NLL) of 0.93, compared to 12.60 for a randomly initialized twin model of the same architecture.
The final trained model has approximately 400 million (0.4B) parameters, targets Bengali as the primary language, and uses a corpus of approximately 2 billion tokens.
Under naive byte-level tokenization, Bengali is compressed to an average of approximately 1.4 characters per token, compared to approximately 3.9 for English, causing the linguistic balance of the corpus to be inverted.
Bengali is the world's seventh-largest native-speaker language with approximately 230 million speakers, and belongs to the Indo-Iranian branch of the Indo-European language family.
Bengali exhibits a significant divide between the formal written language (Sadhu Bhasha) and the colloquial written language (Cholito Bhasha).
All Facts (9)
Transferring from a distant language such as English to Bengali coreference resolution degrades performance.
50%UnverifiedBengali has more than 250 million speakers.
50%UnverifiedTransferring from Hindi to Bengali coreference resolution did not yield consistent gains across models.
50%UnverifiedMany FLORES sentences are not in Unicode NFC normalized form: 609 of 1,012 Bengali sentences, 573 Punjabi sentences, and 93 Hindi sentences; failing to normalize first causes token count drift.
50%UnverifiedThe trained model achieves a per-token negative log-likelihood (NLL) of 0.93, compared to 12.60 for a randomly initialized twin model of the same architecture.
50%UnverifiedThe final trained model has approximately 400 million (0.4B) parameters, targets Bengali as the primary language, and uses a corpus of approximately 2 billion tokens.
50%UnverifiedUnder naive byte-level tokenization, Bengali is compressed to an average of approximately 1.4 characters per token, compared to approximately 3.9 for English, causing the linguistic balance of the corpus to be inverted.
50%UnverifiedBengali is the world's seventh-largest native-speaker language with approximately 230 million speakers, and belongs to the Indo-Iranian branch of the Indo-European language family.
50%UnverifiedBengali exhibits a significant divide between the formal written language (Sadhu Bhasha) and the colloquial written language (Cholito Bhasha).
50%