GRAC: A New Graph-Based Retrieval-Augmented Framework for Surgical Scene Captioning

Surgical records are an important basis for surgical training, performance evaluation, and procedural review, but extracting useful information from them often relies on extensive manual review and clinical expertise. How to enable machines to automatically generate accurate, clinically meaningful surgical scene descriptions has long been a challenge at the intersection of medical imaging and language. The recently proposed GRAC (Graph-Based Retrieval-Augmented Captioning) framework attempts to represent surgical scenes as graphs and combine retrieval-augmented generation to improve description quality.

Background: Why Surgical Scene Captioning Is Difficult

Traditional surgical scene captioning methods typically map images directly to text, ignoring the complex relationships among instruments, actions, and anatomical structures during surgery. In a surgical scene, the same instrument may perform multiple actions, and the same action may act on different tissues. This combinatorial explosion makes it difficult for purely end-to-end models to accurately capture factual details.

The core idea of GRAC is to model surgical scenes as graphs: nodes represent instruments, actions, and anatomical structures, and edges represent the interactive relationships among them. This structured representation explicitly preserves the key elements of surgical activity, providing a reliable foundation for subsequent retrieval and generation.

CIDEr-D is a widely used automatic evaluation metric for image captioning tasks, with the full name Consensus-based Image Description Evaluation; its D version introduces a penalty for repeated fragments. It measures semantic consistency by computing the n-gram cosine similarity between generated descriptions and multiple human reference descriptions, and by weighting and combining scores of different n-gram orders. A higher CIDEr-D score indicates that the generated text is closer to the reference text in vocabulary and semantics. Note that this metric focuses on linguistic similarity and does not directly reflect whether the description is medically factually accurate. Therefore, studies usually also employ factual consistency metrics (such as triplet accuracy) for comprehensive evaluation.

SGT here refers to Surgical Graph Transformer, a baseline method that models surgical scenes as graphs and uses the Transformer architecture for relational reasoning. It propagates information among instrument, action, and anatomical structure nodes through graph attention mechanisms, thereby capturing structured dependencies in surgical activity. Unlike GRAC, SGT mainly relies on knowledge stored in model parameters for generation and does not introduce an external retrieval mechanism to provide traceable reference examples. Therefore, on the factual consistency task that requires accurately reconstructing instrument-action-target triplets, SGT performs significantly worse than GRAC, which incorporates retrieval augmentation. This also constitutes an important reference for the comparative experiments in the paper.

The GRAC Framework: Graph Representation + Retrieval Augmentation

The GRAC workflow consists of two steps. Given an input scene graph, the model first retrieves relevant "graph-description" paired samples from the training data. It then compares the instrument-tissue interactions in the retrieved samples with those in the input scene, uses these comparative features to adjust the retrieved description representations, and finally generates a textual description for the current scene.

This design allows the model to draw on the linguistic expressions of similar historical scenes while correcting for subtle differences in the current scene. Compared with generative models that rely solely on parametric memory, the retrieval mechanism provides traceable references for descriptions, helping to reduce factual errors.

Retrieval-Augmented Generation (RAG) is a technical paradigm that combines external knowledge retrieval with generative models. Its core idea is: before generating the target text, first retrieve samples or documents relevant to the input from an external knowledge base, and then feed the retrieval results as additional context to the generative model. The advantage of this approach is that the model does not need to compress all knowledge into its parameters, can leverage traceable external evidence to reduce factual errors, and when knowledge is updated, only the retrieval library needs to be updated without retraining the model. In GRAC, the retrieval library consists of "graph-description" pairs in the training data. The retrieval results are corrected by comparative features and then used to generate descriptions of the current surgical scene, representing a concrete implementation of the RAG idea in a medical multimodal setting.

Experimental Results: Significant Improvements Across Multiple Metrics

On the EndoVis-18 benchmark, GRAC achieves a CIDEr-D score of 5.88, compared with 4.23 for SGT and 3.36 for the strongest traditional Transformer baseline. On the stricter factual consistency metric, the instrument-action-target triplet set accuracy reaches 52.6%, compared with 32.7% for SGT and 26.2% for the strongest traditional baseline.

On the TORS dataset, GRAC also achieves a CIDEr-D of 5.06, while the strongest baseline reaches 3.80. These results show that the combination of structured scene representation and retrieved examples not only improves linguistic fluency but also demonstrates a clear advantage in factual agreement.

Instrument-action-target triplet set accuracy is a factual consistency evaluation metric for surgical scene descriptions. It parses a description into several (instrument, action, target) triplets, such as (grasper, grasp, gallbladder), then compares them with a human-annotated set of standard triplets and calculates the proportion of exact matches. This metric does not care whether the linguistic expression is fluent; it only focuses on whether the surgical facts stated in the description are correct and complete. Compared with n-gram similarity-based metrics such as CIDEr-D, triplet accuracy better reflects whether the model truly understands the interactive relationships in a surgical scene. Therefore, in surgical scene captioning research, it is regarded as a stricter and more clinically meaningful evaluation method.

Significance and Limitations

The value of GRAC lies in moving surgical scene captioning from "black-box generation" toward "evidence-based generation." By capturing recurring patterns in surgical activity through graph structures and introducing prior knowledge via a retrieval mechanism, the descriptions generated by the model are more contextually relevant.

However, the effectiveness of this framework depends heavily on the quality and coverage of the retrieval library, and there may be few useful examples for rare surgical scenes. In addition, the graph construction process itself requires accurate instrument and tissue recognition, and the end-to-end practicality still needs further validation.

Overall, GRAC provides an interpretable and extensible direction for surgical scene understanding, with potential future applications in automating surgical records, intraoperative decision support, and medical education.

One-line Summary

GRAC represents surgical scenes as graphs and combines retrieval-augmented generation to significantly improve the factual accuracy of descriptions.

Paragraph Summary

GRAC is a new framework for surgical scene captioning. It models surgical scenes as graphs, where nodes represent instruments, actions, and anatomical structures, and edges represent interactive relationships, and on this basis introduces retrieval-augmented generation. Given an input scene graph, the model first retrieves similar graph-description pairs from the training data, then corrects the retrieved description representations by comparing instrument-tissue interaction features, and finally generates text for the current scene. On the EndoVis-18 benchmark, GRAC achieves a CIDEr-D score of 5.88 and an instrument-action-target triplet accuracy of 52.6%, both substantially surpassing SGT and traditional Transformer baselines; on the TORS dataset, CIDEr-D reaches 5.06, validating cross-dataset generalization. The value of this method lies in moving surgical captioning from black-box generation toward evidence-based generation, but its effectiveness depends on the coverage of the retrieval library and may be limited in rare scenes.