DocXEval: Systematic Evaluation of Explainability for Multimodal Document Understanding

Multimodal Transformer models continue to set new performance records on document understanding tasks, but their black-box nature remains the biggest obstacle to deployment in high-stakes scenarios. When documents such as medical reports, financial contracts, and legal filings are misread by a model, the consequences go far beyond a single failed API call. Explainability research has accumulated substantial work in vision and language tasks, yet XAI efforts targeting multimodal document understanding have long been absent. DocXEval is the first systematic evaluation framework proposed to fill this gap.

Why Document Understanding XAI Needs Its Own Framework

Document understanding models process three modalities simultaneously—text, layout, and images—which makes their explainability fundamentally different from pure vision or pure text tasks. The key information on an invoice may be determined jointly by textual content, spatial position, and a stamp image; explanation methods must decompose the contribution of each modality rather than produce a single generic heatmap.

Most existing XAI methods are designed for a single modality, and directly transferring them to document scenarios loses layout information. DocXEval's approach is to adapt existing explanation methods and evaluation metrics to the multimodal document context, and to introduce efficient multi-target adapted versions for tasks such as semantic entity recognition.

Multimodal document understanding tasks typically include semantic entity recognition (e.g., extracting amounts and dates from invoices), document classification, and document visual question answering. Unlike pure text or pure image tasks, the inputs to these tasks simultaneously contain character content, the coordinates of characters on the page, and the rendered page image; the model must fuse three heterogeneous sources of information to make a prediction. Therefore, explanation methods must not only indicate which words or regions are important, but also distinguish whether the information comes from textual semantics, spatial layout, or visual appearance—this constitutes the fundamental challenge that sets document XAI apart from general XAI.

Evaluation Scale: 13 Methods, 8 Datasets, 17 Metrics

According to the paper, the research team used DocXEval to systematically evaluate 13 XAI methods, covering 3 types of document understanding tasks, 8 datasets, 4 models, and 17 evaluation metrics. This scale of combination is the first of its kind in the document XAI field; the goal is not to select a single best method but to establish a reusable benchmark.

A core finding of the evaluation is that only a few methods maintain consistent performance across all evaluation criteria, with Occlusion achieving the best overall performance on both faithfulness and interpretability. This conclusion has direct reference value for practitioners—when a team needs to quickly add explanation capabilities to a document model, Occlusion is a relatively safe starting point.

Occlusion is a class of perturbation-based post-hoc explanation methods: it systematically occludes different parts of the input (e.g., text segments, image regions, or layout blocks), observes changes in the model's output probabilities, and uses the magnitude of change as a measure of the importance of that part. In document scenarios, occlusion can be applied separately to text, images, and layout coordinates, making it naturally suited to decomposing multimodal contributions. Its advantages are that it is model-agnostic and conceptually intuitive; its drawbacks are that computational cost grows with the number of occlusion units and that it is sensitive to the choice of occlusion granularity.

Meta-Evaluation Reveals Credibility Differences Among Metrics

Another piece of work in DocXEval is meta-evaluation, i.e., examining whether the evaluation metrics themselves are reliable. The study finds moderate-to-strong agreement among several faithfulness metrics and complexity metrics, indicating that these metrics measure similar things; at the same time, it identifies some metrics that produce unreliable results.

The warning for XAI research is clear: if the evaluation metrics themselves are unreliable, method rankings derived from them may be distorted. Future research should prioritize metrics that have been validated through meta-evaluation and show high agreement with other metrics when selecting evaluation metrics.

Meta-evaluation refers to re-evaluating the evaluation metrics themselves, examining whether they are stable, consistent, and truly reflect explanation quality. In the XAI field, a common practice is to compute rank correlations (such as Spearman coefficients) between different metrics; if multiple metrics intended to measure the same construct (e.g., faithfulness) are highly consistent, they are considered credible; if a metric contradicts other metrics or is sensitive to perturbations, its conclusions are questionable. DocXEval's meta-evaluation uses exactly this kind of inter-metric agreement analysis to screen out unreliable metrics and prevent method rankings from being misled by noise.

Insights into Model Behavior from Dataset-Level Diagnostic Analysis

Beyond method rankings, DocXEval provides several key insights into model behavior through multiple dataset-level diagnostic analyses: differences in the contribution of each modality across tasks and labels, spatial attribution patterns around target entities, and potential dataset biases.

The value of these analyses lies in advancing explainability from "explaining a single prediction" to "understanding the model's overall behavior." For example, spatial attribution patterns can reveal whether the model over-relies on layout cues around entities, while the discovery of dataset biases reminds researchers to guard against shortcut learning at the data construction stage.

Spatial attribution patterns refer to the distribution of importance that explanation methods assign to spatial regions around a target entity (e.g., adjacent text blocks, table lines, stamp positions). By aggregating spatial attributions across a large number of samples, one can determine whether the model over-relies on positional shortcuts—for example, predicting an amount as the total simply because it appears in the upper-right corner of the page, rather than genuinely understanding the semantics. Such diagnostics elevate single-sample explanations to dataset-level behavioral profiles, helping to uncover systematic biases and shortcut learning and to guide data augmentation or model regularization.

Implications for Practitioners

DocXEval serves both as a benchmark and as a practical guide. For researchers, it provides a unified evaluation protocol and publicly available code (released on GitHub), facilitating reproduction and cross-comparison; for engineering teams, it offers empirical evidence for method selection and highlights pitfalls in choosing evaluation metrics. For document understanding models to enter high-stakes industries, this kind of systematic evaluation is indispensable infrastructure.

Frequently Asked Questions

Which document understanding tasks does DocXEval support?

The paper covers 3 types of document understanding tasks, including typical tasks such as semantic entity recognition, and provides multi-target adapted versions of explanation methods and metrics for these tasks.

Why does Occlusion perform best?

The paper's conclusion is that Occlusion is overall optimal on faithfulness and interpretability, but the specific mechanisms require reference to the original paper's comparative analysis of each method.

Is the code open source?

Yes, the code is publicly available at https://github.com/saifullah3396/docxeval/ to support reproducible research.

One-line Summary

DocXEval is the first systematic evaluation framework for explainability in multimodal document understanding, featuring 13 methods and 17 metrics.

Paragraph Summary

DocXEval is the first systematic evaluation framework for explainability (XAI) in multimodal document understanding, aiming to fill the gap in XAI evaluation for document scenarios. It adapts to the three modalities of text, layout, and image, and systematically evaluates 13 XAI methods across 3 task types, 8 datasets, 4 models, and 17 metrics. A core finding is that Occlusion achieves the best overall performance on faithfulness and interpretability. Meta-evaluation reveals moderate-to-strong agreement among some metrics while also identifying unreliable ones. Dataset-level diagnostic analysis further reveals differences in modality contributions, spatial attribution patterns, and potential dataset biases. The code is open source and can serve as a research benchmark and a guide for engineering method selection.