LLM Tackles Nepali Tweet NER Challenge: Surpasses PLM on Hard Cases

Named Entity Recognition (NER) on social media text in low-resource languages has always been a thorny task. Nepali tweets are an especially tough case—informal spelling, code-mixing, frequent abbreviations, and emojis cause even traditional pre-trained language models (PLMs) to stumble. But a recent analysis points to an interesting strategy: using a large language model (LLM) specifically to handle the "hard cases" that PLMs can't crack, which can actually improve overall performance.

When PLMs Meet Tough Tweets

The research team used a Nepali tweet dataset and leveraged the token probabilities output by a fine-tuned PLM to split the test set into "easy" and "hard" portions—samples with low probabilities where the model hesitated were classified as hard cases. This effectively performed a "difficulty stratification" of the data, making subsequent evaluation more targeted.

On the full test set and the easy set, the LLM indeed underperformed the PLM, consistent with prior observations: for structured prediction tasks like NER, pre-trained models tend to be more precise. But the real highlight is the hard-case set.

Token Probabilities and Confidence: In sequence labeling tasks, the model outputs a probability distribution for each token, representing its confidence in each label class. When a fine-tuned PLM is tested, if a token's predicted probability is low (e.g., the maximum probability falls below a threshold), it indicates the model is uncertain about that position; such samples often contain ambiguous boundaries, type ambiguities, or unseen language patterns. Using token probabilities for difficulty stratification is an active-learning-style evaluation method that requires no additional annotation and can effectively identify the model's knowledge blind spots.

Prompt Engineering: Helping the LLM Triumph on Hard Cases

The researchers experimented with 16 different prompt configurations for the LLM on the hard test set. The key finding: selecting examples based on semantic similarity (few-shot example retrieval) and increasing the number of examples boosted the F1 score by nearly 10 points over the baseline configuration. This is no small improvement—on a metric-sensitive task like NER, a nearly 10-point gain means a large number of entity boundaries and types are now correctly identified.

Semantic similarity retrieval works because hard cases often exhibit unique language patterns or domain vocabulary; similar examples can guide the LLM to better understand the current context. Increasing the number of examples gives the model more "pattern references," compensating for the LLM's lack of fine-tuning on the specific task.

Although the LLM still lagged behind the PLM on the full set and the easy set, it successfully surpassed the PLM on the hard-case set. This means the two models are complementary: the PLM excels at regular patterns, while the LLM can handle edge cases.

Few-shot Example Retrieval: In in-context learning, the choice of examples significantly affects LLM performance. Randomly selected examples may introduce irrelevant patterns, whereas retrieving examples most similar to the current input based on semantic similarity (e.g., computing cosine similarity using sentence embeddings) helps the model more accurately capture task boundaries and label semantics. For hard cases, similar examples provide targeted pattern references, effectively dynamically constructing a small "training set" that alleviates the LLM's lack of task-specific fine-tuning.

Model Combination: 1+1>2

The study further combined the predictions of both models, resulting in improved overall scores. This result suggests a practical engineering approach: rather than insisting on replacing the PLM with an LLM, let them divide the work. For social media NER in low-resource languages, first using the PLM to quickly process most easy samples, then focusing the LLM's firepower on low-confidence hard cases, may be a more cost-effective solution.

This hybrid strategy also alleviates concerns about LLM inference costs—only hard cases require LLM calls, greatly reducing the volume of calls.

Implications for Low-Resource NER

Although this analysis focuses on Nepali tweets, its methodology is universally applicable. Social media data in low-resource languages generally suffers from high noise and scarce annotation; PLM confidence stratification and LLM prompt optimization can be transferred to other languages and domains.

You may not have noticed that the choice of prompt configuration is not trivial: semantic similarity retrieval and the number of examples are two key levers, and simply piling on random examples yields limited gains. For practitioners, this means that when deploying LLMs to handle hard cases, it's worth investing effort in designing example retrieval strategies.

From a broader perspective, the role of LLMs in NER tasks is becoming clearer: it is not a replacement for PLMs but a complementary tool for handling long-tail hard cases. In scenarios with limited resources and expensive annotation, this combinatorial approach may be more pragmatic than pursuing a single "strongest model."