Fikiria: Improving Swahili Mathematical Reasoning with Reinforcement Learning

Fikiria uses SFT+GRPO to improve Swahili math reasoning, with automatic metrics and human preferences diverging.
Fikiria proposes a two-stage approach for low-resource language math reasoning: supervised fine-tuning on ~2,300 Swahili GSM8K problems translated by GPT-4.1 and filtered by SSA-COMET, followed by GRPO reinforcement learning with a reward combining a format gate and a weighted sum of answer accuracy and ChrF++ reasoning-chain similarity. On AfriMGSM, Fikiria-AfriqueQwen-14B significantly outperforms the GPT-4o baseline, which often loses points for format non-compliance. Yet in small-scale human evaluation, GPT-4o outputs were preferred, and ChrF++ only measures surface text fidelity, not reasoning-step validity. The study shows the combined strategy is actionable for low-resource languages, but a gap remains between automatic metrics and human judgment.
The Mathematical Reasoning Predicament of Low-Resource Languages
Large language models excel at mathematical reasoning tasks in English, but their performance often drops sharply when faced with low-resource languages such as Swahili. This is not merely a matter of vocabulary coverage—it also concerns whether the model can organize a complete reasoning chain in the target language. A recent study proposed Fikiria, a two-stage fine-tuning and reinforcement learning approach that attempts to enable models not only to answer correctly but also to "write out the problem-solving process" in Swahili.

Two-Stage Training: From Supervised Fine-Tuning to GRPO
Fikiria's training pipeline consists of two steps. The first stage is supervised fine-tuning (SFT), which builds the model's basic language and problem-solving abilities on Swahili math problems. The second stage employs GRPO (Group Relative Policy Optimization) for reinforcement learning, with its core innovation lying in the design of the reward function.
This reward is the product of two parts: a format gate, and a weighted sum of answer accuracy and the ChrF++ similarity between the Swahili reasoning chain and the reference chain. The format gate ensures that outputs conform to the required answer format, while ChrF++ similarity measures how closely the model-generated reasoning process approximates the reference reasoning chain at the surface-text level. This design makes the training objective attend to both the final answer and the linguistic fidelity of the reasoning trajectory, rather than fixating solely on answer correctness.
GRPO (Group Relative Policy Optimization) is a policy optimization algorithm in reinforcement learning designed specifically for language model fine-tuning. Unlike traditional PPO, which requires an independent value network to estimate the baseline, GRPO samples a group of outputs for the same prompt and directly normalizes advantages using the mean and standard deviation of rewards within the group, thereby eliminating the value model and reducing GPU memory overhead. Its "relative" nature is reflected in the fact that each output's advantage is determined by its performance relative to other outputs in the same group, rather than by an absolute reward value. This makes training insensitive to reward scale, suitable for sparse or composite reward scenarios such as answer accuracy and format compliance, and convenient for conducting multiple rounds of reinforcement learning on 14B-level models under limited compute.
ChrF++ is a machine translation evaluation metric based on character n-grams, which can be regarded as an extended version of chrF. It computes both precision and recall of character-level n-grams (typically n=1–6) and incorporates a weighting of word-level n-grams, ultimately combining them in the form of an F-score. Compared with BLEU, which relies on word-level matching, ChrF++ is more robust for morphologically rich, agglutinative languages (such as Swahili) and can capture partial semantic similarity arising from affix changes. However, it is essentially still a surface string-overlap measure and does not judge logical equivalence: text with similar wording but flawed reasoning may also receive a high score, which is precisely the root of the reward design's limitation described in the paper.
Data Construction and Evaluation Results
The research team used GPT-4.1 to translate GSM8K and constructed approximately 2,300 Swahili math problems, filtering quality with SSA-COMET. On the AfriMGSM Swahili benchmark, Fikiria-AfriqueQwen-14B achieved results significantly better than baseline models.
Under the same fixed prompts and strict answer-format scoring, GPT-4o performed notably worse, partly because it frequently failed to satisfy the required answer format. According to the IrokoBench report, GPT-4o can reach a level close to Fikiria when using the best per-model prompts, but this number is not directly comparable to the results of this study because the prompting strategies and scoring conditions differ.
SSA-COMET is a translation quality evaluation model based on the COMET framework, trained for Sub-Saharan African languages. COMET-style metrics do not rely on a single reference translation; instead, they use a multilingual pre-trained encoder to jointly encode the source sentence, translation, and reference sentence, and regress a human quality score. SSA-COMET is further fine-tuned on parallel corpora of African languages, making it more sensitive to translation errors in low-resource languages such as Swahili. In this study, it was used to filter GPT-4.1's machine translation outputs, with the aim of removing problems with semantic drift or numerical errors and ensuring that the training data is trustworthy in both answers and reasoning chains, rather than merely performing surface-level cleaning.
AfriMGSM is a math word problem benchmark for African languages, expanded by translating mathematical datasets such as the English GSM8K. It covers multiple low-resource languages including Swahili, Yoruba, and Hausa, and provides reference answers and reasoning steps. It is used to test whether a model can complete multi-step arithmetic reasoning in the target language, not just language understanding. Compared with the original GSM8K, AfriMGSM's evaluation pays more attention to whether the translated mathematical expressions preserve the original meaning, so it is often used together with strict answer formats and manual verification, and is one of the main public benchmarks for evaluating mathematical reasoning ability in low-resource languages.
The Gap Revealed by Human Evaluation
Interestingly, in a small-scale human evaluation in which the co-authors served as annotators, GPT-4o's outputs were generally preferred. This creates an intriguing tension with the automatic metric results: Fikiria has the edge in format compliance and surface similarity, but human readers may place more value on the naturalness and readability of the reasoning.
Another key limitation is that ChrF++ measures surface text fidelity, not the validity of individual reasoning steps. The model may generate reasoning that is similar in wording to the reference chain but logically flawed. This means the reward function encourages "writing it like" the reference but cannot fully guarantee "thinking it right."
Implications for AI in Low-Resource Languages
Fikiria's value lies in demonstrating a viable path: using machine translation to build training data, using surface similarity as a proxy signal for reasoning quality, and then injecting format and content constraints into the model through reinforcement learning. For low-resource languages lacking high-quality human annotations, this combined strategy is practically actionable.
However, the divergence in human evaluation preferences reminds us that a gap remains between automatic metrics and human judgment. Future improvements may include finer-grained verification of reasoning steps and incorporating human feedback into reward design, rather than relying solely on the surface similarity to reference chains.
One-line Summary
Fikiria uses SFT+GRPO to improve Swahili mathematical reasoning, with automatic metrics and human preferences diverging.
Paragraph Summary
Fikiria proposes a two-stage approach to improve mathematical reasoning in low-resource languages: first, supervised fine-tuning on approximately 2,300 Swahili GSM8K problems translated by GPT-4.1 and filtered by SSA-COMET; then reinforcement learning with GRPO, where the reward is a format gate multiplied by a weighted sum of answer accuracy and ChrF++ reasoning-chain similarity. On AfriMGSM, Fikiria-AfriqueQwen-14B significantly outperforms the GPT-4o baseline, which often loses points due to format non-compliance. However, in a small-scale human evaluation, GPT-4o outputs were preferred, and ChrF++ only measures surface text fidelity and cannot verify the validity of reasoning steps. The study shows that this combined strategy is actionable for low-resource languages, but a gap remains between automatic metrics and human judgment.
Related articles

Gemini 4 Neon Released: A Balance Between Frontier Intelligence and Efficient Task Execution
Google DeepMind releases Gemini 4 Neon, focused on frontier intelligence and fast end-to-end task execution. This article breaks down its benchmark performance in coding, enterprise analysis, scientific reasoning, and cybersecurity, its four thinking levels, pricing, and safety alignment data.

Is AI the Culprit Behind Global Outages? The Trust Crisis Behind the NSA and FAA Failures
Fourth major failure in five days hits NSA classified network. Microsoft blames a software bug, FAA denies its new AI system, NSA says routine maintenance. Senator Warner suggests AI accelerates vulnerability discovery, leading to rushed fixes — while 39% of Americans already view AI negatively, exposing a deepening trust crisis.

Robot Developer Builds Custom B-rep Kernel with 3,487 Tests
A robot developer builds a custom B-rep geometry kernel with 3,487 tests, exploring why robotics needs dedicated CAD tools and the barriers to building one.