CARAT

The paper "CARAT: Do Materials LLMs Reason or Recite?" raises the question of whether large language models for materials science are reasoning or reciting.

Traditional accuracy metrics cannot distinguish whether a model is reasoning from structure or directly copying answers already printed in the input.

The methodological core of CARAT is controlled variables: fixing the questions and ground-truth answers while comparing across eight matched views.

CARAT introduces GraphSpace as its core representation framework, organizing information such as atomic positions, bonding relationships, and symmetry of crystal structures in the form of structured graphs.

CARAT employs matched fine-tuning, answer masking, evidence injection, paired reasoning, and a mechanism that allows the model to abstain from answering.

On the hardest category of questions, grounded views outperformed views that simply fed formula inputs by 17.3 points.

A shortcut rule that skips the relational chain and reads the list directly can answer four out of seven augmented question families correctly.

The researchers iteratively rebuilt the benchmark until all eleven shortcuts were suppressed to near-random-guess levels.

The frozen model gives the same answer in 95.6% of paired cases when links are redirected elsewhere, indicating that the model is merely reciting the relationship rather than using it.

After matched supervised training, model performance improved to 99.8%, while removing the relational links caused accuracy to plummet to 23.4%, even below the best shortcut's 27.0%.

High accuracy in LLM evaluations may be a product of information leakage rather than genuine reasoning ability, a problem that is especially pronounced in structured scientific data such as CIF files, which simultaneously contain multiple attribute fields.