[KongchangAI]
· 2 min read· 1,073 words

AraCrypto-ML: A Multi-label Corpus of Arabic Cryptocurrency Discourse

AraCrypto-ML: A Multi-label Corpus of Arabic Cryptocurrency Discourse

AraCrypto-ML builds the first multi-label corpus of Arabic Facebook cryptocurrency posts, revealing function-engagement associations.

This article introduces AraCrypto-ML, a multi-label corpus of 10,000 Arabic Facebook cryptocurrency posts from 2014 to 2024. Posts are annotated into six communicative functions: market commentary, education/popular science, promotion/marketing, questioning/warning, religious framing, and other/unclear. Annotation was performed by the Fanar-2-27B model under a prompt-only protocol, achieving a macro-averaged kappa of 0.74 against human gold standards—higher than the 0.62 between two human annotators—with Rogan-Gladen calibration to correct bias. Analysis reveals significant differences in engagement across post functions: education/popular science and promotion/marketing posts receive about 70% "Love" reactions, while questioning/warning posts have the highest "Haha" reaction share (about 39%). The corpus provides a valuable resource for computational linguistics, social media research, and financial sociology, while demonstrating the potential of large language models in multi-label annotation for low-resource languages.

Arabic cryptocurrency discussions on social media exhibit complex communicative heterogeneity: a single post may simultaneously contain price reporting, protocol explanations, investment promotion, scam warnings, or even references to Islamic law. However, no annotated resource has previously captured this functional diversity. AraCrypto-ML fills this gap—a multi-label corpus of 10,000 Arabic Facebook cryptocurrency posts.

AraCrypto-ML paper source screenshot

Corpus Construction and Annotation Framework

Data Source and Collection

The research team collected Arabic Facebook cryptocurrency-related posts from 2,898 public pages between 2014 and 2024, totaling 10,000 posts. These posts were filtered using a set of relevant keywords, covering a wide range of discussion scenarios to ensure corpus representativeness.

Six Communicative Function Labels

Each post was annotated into one of the following six categories (under a multi-label protocol, a post may belong to multiple categories):

  • Market Commentary: price analysis, market predictions, etc.
  • Education/Popular Science: explaining blockchain protocols, technical principles, etc.
  • Promotion/Marketing: promoting investment projects, referral invitations, etc.
  • Questioning/Warning: warning about scams, expressing doubt, etc.
  • Religious Framing: discussing cryptocurrency compliance with reference to Islamic law
  • Other/Unclear: posts that cannot be assigned to the above categories

Annotation was performed by the Fanar-2-27B model under a prompt-only multi-label protocol, avoiding the reliance of traditional supervised annotation on large amounts of manual labeling.

Annotation Quality and Calibration

Model-Human Annotation Agreement

The research team sampled 1,000 posts for manual verification. Using adjudicated gold-standard labels as the benchmark, the model achieved a macro-averaged kappa of 0.74. You may not have noticed that the macro-averaged kappa between the two human annotators was only 0.62, lower than the agreement between the model and either annotator (0.70 and 0.69, respectively). This indicates that the model's annotation consistency even surpasses that of ordinary human annotators, while also reflecting the inherently subjective nature of the task.

Macro-averaged Cohen's kappa is a statistical metric for measuring inter-annotator agreement, typically ranging from -1 to 1: 0 indicates agreement equivalent to random annotation, and 1 indicates complete agreement. It is calculated by first computing Cohen's kappa for each category separately, then taking the arithmetic mean across all categories. Unlike simple accuracy, kappa deducts the probability of random agreement, making it more suitable for multi-label scenarios with imbalanced category distributions. In natural language processing annotation tasks, kappa above 0.6 is generally considered acceptable agreement, and above 0.8 is regarded as high agreement. In this paper, the model's 0.74 against the gold standard falls in the "good" range, while the 0.62 between human annotators is only at the lower bound of "acceptable"—this contrast is precisely the key quantitative basis for the authors' argument for the feasibility of model annotation.

Rogan-Gladen Calibration

To correct annotation bias, the study employed the Rogan-Gladen calibration method to statistically adjust the model annotation results, further improving label reliability. This step is particularly important for researchers using the corpus for downstream analysis.

Rogan-Gladen calibration originates from prevalence estimation methods in epidemiology, used to correct proportion estimates when misclassification bias exists. Its core idea is: if the classifier's sensitivity (true positive rate) and specificity (true negative rate) are known, and the misclassification pattern is assumed stable, the observed positive proportion can be inverted to estimate the true positive proportion. In this paper's context, the model-annotated labels are not perfect ground truth; directly using them to calculate category proportions would produce systematic bias. Using sensitivity and specificity estimated from the manual verification set, the estimated proportion of each functional category can be corrected. The method's assumptions include that classifier errors are independent of sample features and that sensitivity and specificity remain stable across the entire corpus. If the corpus has strong internal heterogeneity, this assumption may be weakened, so the authors also caution that calibrated results should be interpreted carefully.

Differences in Engagement Patterns Across Post Functions

The value of the corpus lies not only in the annotation itself but also in revealing the association between different communicative functions and user engagement. Analysis found that engagement responses differ significantly by post function:

  • In Education/Popular Science and Promotion/Marketing posts, "Love" reactions account for about 70% of reactions per post, indicating that users show relatively high positive sentiment toward such content.
  • Questioning/Warning posts are markedly different: "Love" reactions account for only about 33%, while "Haha" reactions are highest at about 39%. This may reflect that users hold a somewhat mocking or resigned attitude toward warning content, or that the warnings themselves contain some dark humor.

These patterns suggest that emotional expression in cryptocurrency discourse is closely tied to content function, and simple sentiment analysis that ignores the functional dimension may be misleading.

Research Significance and Limitations

AraCrypto-ML is the first multi-label functional corpus of Arabic Facebook cryptocurrency discourse, providing a valuable resource for computational linguistics, social media research, and financial sociology. Its annotation protocol demonstrates the potential of large language models in low-resource languages and multi-label scenarios, especially when model agreement exceeds that of human annotators—the effectiveness of prompt engineering warrants further exploration.

Of course, the study has limitations: it covers only the Facebook platform, and keyword filtering may miss some marginal discussions; the sample size of religious framing posts is relatively small, which may affect the stability of related analyses. Additionally, while Rogan-Gladen calibration can correct bias, it relies on certain statistical assumptions and requires careful interpretation in practical application.

Overall, AraCrypto-ML lays the foundation for understanding the complex communicative behavior of Arabic cryptocurrency communities, and future research can build on it to explore causal relationships between functional labels and information dissemination, user participation, and more.

One-line Summary

AraCrypto-ML constructs the first multi-label corpus of Arabic Facebook cryptocurrency posts, revealing function-engagement associations.

Paragraph Summary

This article introduces AraCrypto-ML, a multi-label corpus of 10,000 Arabic Facebook cryptocurrency posts spanning 2014 to 2024. Posts are annotated into six communicative functions: market commentary, education/popular science, promotion/marketing, questioning/warning, religious framing, and other/unclear. Annotation was performed by the Fanar-2-27B model under a prompt-only protocol, with a macro-averaged kappa of 0.74 against human gold standards—higher than the agreement between two human annotators (0.62)—and Rogan-Gladen calibration was used to correct bias. Analysis found significant differences in engagement responses across post functions: education/popular science and promotion/marketing posts received about 70% "Love" reactions, while questioning/warning posts had the highest "Haha" reaction share (about 39%). The corpus provides a valuable resource for computational linguistics, social media research, and financial sociology, while demonstrating the potential of large language models in multi-label annotation for low-resource languages.

#AraCrypto-ML#Arabic cryptocurrency corpus#multi-label annotation#Facebook discourse#communicative functions
Share:

Related articles