faithfulness

Faithfulness refers to whether the reasoning process in a model's output truly reflects its internal computational logic, in contrast with Plausibility.

For a question asking which cognitive biases the papers involve, one generation honestly said the papers did not name specific biases and relevancy gave it 0; another generation confidently listed a bias that does not exist in the corpus and relevancy instead gave it 0.58, with only faithfulness catching the fabrication (0.17).

The author gives reference thresholds for comparing two single runs: about 0.15 for faithfulness, about 0.08 for relevancy, about 0.09 for recall, and about 0.10 for precision (at the 18-question scale, and about 40% wider on a 9-question subset); only mean differences exceeding the threshold are worth taking seriously.

Switching the judge model moved recall by 0.31 and faithfulness by 0.26, three times and nearly twice their respective thresholds, showing that the judge model has a greater impact than the k parameter.

For recall and precision, re-running the judge on saved results suffices and even saves generation calls; for faithfulness and relevancy, answers must be regenerated, because re-judging alone captures only about 15% of the variance in faithfulness, making it look more than twice as stable as it actually is.

For a question asking which paper won the best paper award, the correct answer is 'none'; all three generations answered correctly, but faithfulness scored them 1.00, 0.33 and 0.00 respectively, because a negated statement itself cannot be verified against the chunks.

In RAG evaluation, about 85% of the run-to-run variance in the faithfulness metric comes from the generated answer, and almost all of the variance in answer relevancy comes from the answer; whereas 100% of the variance in context recall and context precision comes from the judge, because RAGAS does not read the generated answer when computing these two metrics, only comparing the retrieved chunks against the question and the reference answer.

When only re-judging the same batch of saved answers, the standard deviation of the 18-question mean ranged from 0.021 (faithfulness) to 0.035 (precision); after regenerating answers, the standard deviation of faithfulness rose to 0.054, and relevancy rose from 0.002 to 0.028.

In the faithfulness metric, the number of claims the judge splits from the same answer varies (for example, one answer was split into 5, 10 and 11 claims across three runs), causing the denominator to change and thus the score to fluctuate; precision and recall consist of yes/no judgments, with 5 of 148 chunk judgments flipping across the three runs and 3 of 22 reference-statement judgments flipping; since 14 of the 18 reference answers are a single sentence, recall on those questions is either 0 or 1.

General LLM assessment metrics (such as helpfulness, toxicity, faithfulness) are domain-agnostic and cannot detect business factual errors (such as a difference between 24 hours and 48 hours in fare rules).

Recall@k and MRR are commonly used to evaluate retrieval quality, while evaluating generation typically looks at the consistency between answers and sources, i.e., faithfulness.

Many current RAG evaluation frameworks (such as RAGAS) have listed faithfulness as a core metric, but in practical engineering evaluations this step is often simplified or skipped.

Answer accuracy and retrieval faithfulness are two mutually independent evaluation dimensions; faithfulness evaluation requires verifying whether each key claim in the answer can be directly traced back to the retrieved documents.