Punjabi Song Toxic Content Dataset PGT-FG Released

Content moderation on music platforms has always been a headache—text, audio, and visuals are intertwined, and toxic content often hides in lyrical metaphors and the emotional charge of melodies. For low-resource languages, this challenge is compounded by another layer: scarce annotated data and underdeveloped language toolchains. Punjabi finds itself in exactly this situation.

A recently released academic resource attempts to fill this gap. Researchers have built the Punjabi Geet Toxicity Dataset (PGT-FG), a fine-grained multimodal toxicity annotation dataset for Punjabi music content.

How the Dataset Was Built

From 620 Songs to 2,396 Segments

The data comes from publicly available Punjabi songs on YouTube. The research team initially collected 620 songs, successfully downloaded 465, and segmented them into 3,402 preliminary clips (covering 450 songs). After quality control and the annotation process, 2,396 independent segments from 392 songs were retained.

The segmentation step is worth noting. Song-level toxicity judgments are too coarse—a song's chorus may be clean while its verses contain violent or gang-related references. Segment-level annotation is what enables the training of more fine-grained detection models.

Segment-level annotation refers to dividing a longer piece of media content into shorter segments and assigning labels to each segment individually, rather than giving a single overall label to the entire song. In music toxicity detection, this approach captures uneven label distributions within a song—for example, a clean chorus paired with violent references in the verses. Segment-level labels also multiply the number of training samples and more closely mirror how real content moderation works: cutting content into time windows and making judgments segment by segment. The trade-off is that the choice of segment boundaries affects label quality, requiring unified segmentation rules and ensuring each segment is relatively semantically complete.

Annotation Engineering with Four Native Speakers

Annotation was carried out by four native Punjabi speakers, producing a total of 2,870 annotation records. Of these, 158 segments were covered by all four annotators and used to compute inter-annotator agreement.

Each annotation contains a rich set of information:

• Binary toxicity label
• Eight fine-grained toxicity categories
• Severity level
• Sentiment orientation
• Modality information
• Uncertainty flag

Fleiss' kappa was used to measure agreement on the shared subset, and final labels for each segment were determined by majority vote.

Fleiss' kappa is a statistical measure of inter-annotator agreement for scenarios where three or more annotators label the same set of samples by category. It is computed based on the proportion of annotators assigning each category to each sample, correcting for the possibility of chance agreement, and typically ranges from -1 to 1 (in practice, mostly 0 to 1). Common rule-of-thumb benchmarks are: below 0.20 slight agreement, 0.21–0.40 fair agreement, 0.41–0.60 moderate agreement, 0.61–0.80 substantial agreement, and above 0.81 almost perfect agreement. The 0.6521 reported here falls within the substantial agreement range, indicating good but not perfect agreement among annotators—consistent with the inherent subjectivity of toxicity judgments.

What the Data Reveals

Of the 2,396 segments, 705 (29.42%) were labeled toxic and 1,691 (70.58%) non-toxic. A toxicity rate approaching one-third shows that toxicity in music content is far from a marginal phenomenon.

Among the fine-grained categories, violence appears most frequently—539 toxic segments (76.45%) involve violent content, followed by gang references (227, 32.20%) and drug references (133, 18.87%). The overwhelming share of the violence category echoes the long-standing violent narrative tradition in certain genres of the Punjabi music industry.

In terms of agreement, the kappa value for the overall toxicity label is 0.6521, falling in the "substantial agreement" range. For a toxicity annotation task involving subjective judgment, this figure is reasonably robust, but it also indicates that roughly one-third of borderline cases were disputed among annotators—which is precisely why the dataset includes an uncertainty flag.

Multimodal toxicity signals means that toxicity is not only expressed in lyric text but may also be conveyed through audio (e.g., tone, shouting, specific sound effects) and video (e.g., gestures, scenes, on-screen text). For music content, the text modality can capture explicit insults or violent vocabulary, the audio modality can reflect emotional intensity and aggressive tone, and the video modality may contain visual cues such as gang gestures. The goal of multimodal fusion models is to jointly model features from different modalities to compensate for cases missed by any single modality. PGT-FG annotates modality information for each segment, enabling researchers to analyze which modality the toxicity signal primarily comes from and to design fusion strategies accordingly.

What Problem Does This Dataset Solve

Toxicity detection resources for Punjabi have long been scarce, and multimodal resources targeting music content are virtually nonexistent. The value of PGT-FG manifests at several levels.

Fine-grained annotation enables models to go beyond answering "is it toxic?" and distinguish specific categories such as violence, gangs, and drugs—which has practical significance for tiered moderation actions, since gang references and violent threats clearly warrant different handling strategies. Segment-level granularity is closer to real-world moderation scenarios.

The multimodal design accounts for the particularities of music content. Beyond lyric text, audio and video information also carry toxicity signals, and the dataset provides an annotation foundation for subsequent multimodal fusion research.

The dataset is publicly available on Zenodo (DOI: 10.5281/zenodo.23040156), released as one final record per segment. For copyright reasons, original lyrics, lyric excerpts, and copyrighted audio/video content are not redistributed with the dataset.

Limitations and Observations

Copyright restrictions create a practical problem: researchers receive annotations and reference information rather than the original media files. Reproducing full multimodal experiments requires obtaining the audio/video materials independently, which raises the barrier to use.

With only four annotators and a shared annotation subset of just 158 segments, the statistical power of the agreement assessment is limited. Some of the eight toxicity categories have small sample sizes (e.g., only 133 drug references), and class imbalance may affect model performance on minority categories.

Another direction worth attention is cross-script transfer. Punjabi spans India and Pakistan, where different writing systems are used (Gurmukhi and Shahmukhi). All data sources are publicly available YouTube songs, and dialectal and register coverage is not described in detail in the paper.

For teams working on content moderation in low-resource languages, PGT-FG offers a usable starting point—particularly its segment-level, multi-label annotation framework, which can be adapted to other languages and music scenarios.

Gurmukhi and Shahmukhi are the two main writing systems used for Punjabi. Gurmukhi is primarily used in the Indian state of Punjab and is its official script; Shahmukhi is mainly used in the Punjab region of Pakistan and derives from the Perso-Arabic script system. Although both record the same language, their character sets, writing directions, and orthographic conventions differ, meaning that natural language processing tools (such as tokenizers and pretrained language models) often need separate adaptation. For toxicity detection, if training data covers only one writing system, model performance may drop noticeably when transferred to the other—a language-variant issue that resource building for low-resource languages needs to keep in mind.