KierstonBench: A Benchmark for LLM Criminal Investigation Capabilities

Large language models are being considered for record collection and analysis in criminal investigations and mass surveillance, but their capabilities and accuracy in such work are currently unmeasurable. Real deployments lack the ground truth needed to score model conclusions, and no benchmark evaluates these capabilities in a simulated environment. To address this, researchers propose Kierston—a city-scale simulation engine—and KierstonBench—a benchmark that tests these capabilities against known simulated events.

Kierston: How the City-Scale Simulation Engine Is Built

Kierston simulates approximately 125,000 residents of Allentown, Pennsylvania, generating their daily activities along with the telecommunications, financial, administrative, and license plate records collected by surveillance and investigation systems. These records derive from shared behavioral timelines and are distributed across multiple sources representing different providers.

The cleverness of this design lies in the fact that all records are derived from the same behavioral timeline, ensuring cross-source data consistency while simulating how data is fragmented across different providers in reality. Models conduct investigations through a restricted record interface, linking observations across sources without access to the underlying simulation. This both guarantees the benchmark's scorability and preserves the challenge of incomplete information inherent in real investigations.

Ground truth in machine learning evaluation refers to reference data with known correct answers. In criminal investigation scenarios, ground truth means we know exactly who the perpetrator is, who is connected to whom, and how events unfolded. In real deployments, investigative conclusions often cannot be verified—cases may go unsolved, evidence may be missing, and the truth may never come to light, making it impossible to score models with real data. Kierston generates resident behavioral timelines through simulation, so that every event, every record, and every personal relationship has a definite generative origin, thereby providing verifiable ground truth for each task. This is the fundamental prerequisite that enables the benchmark to quantitatively evaluate model capabilities.

The restricted record interface is the limited channel through which models interact with the simulated environment in Kierston. Models cannot directly access the underlying simulation data or the complete database; they can only query records from specific sources—telecommunications, financial, administrative, license plates—on demand, similar to how records are retrieved in real investigations. This design simulates the information barriers investigators face in reality: data is fragmented across different providers and requires cross-source linking and reasoning. At the same time, the interface restricts the scope of information available to models, ensuring that what is evaluated is the model's investigative reasoning ability rather than its ability to directly read simulation data, thereby guaranteeing the benchmark's scorability and fairness.

KierstonBench: 32 Task Families Covering Four Core Capabilities

KierstonBench contains 32 task families covering record linking, daily trajectory reconstruction, event investigation, and deceptive statement assessment. These tasks directly correspond to core capabilities in criminal investigation and surveillance scenarios:

• Record linking: Associating the same person or event across different sources;
• Daily trajectory reconstruction: Reconstructing an individual's daily activity patterns from scattered records;
• Event investigation: Integrating multi-source evidence around a specific event;
• Deceptive statement assessment: Identifying false or misleading statements.

The benchmark uses known simulated events as the basis for scoring—an advantage that real deployments cannot provide, since in real scenarios the correctness of investigative conclusions often cannot be verified.

Performance of Sixteen Models: Most Are Not Up to the Task

In evaluations of sixteen language models, most were unable to conduct record investigations and submitted identification conclusions not supported by the records. This means their answers lacked evidentiary support and could be misleading in real criminal investigation scenarios.

A few frontier models solved most tasks and chose to abstain when evidence was insufficient—the ability to know when not to answer is equally important. However, even these models still fell into common deception and assumption traps in some cases, and no model could replace an investigator.

This result reveals the realistic positioning of large models in criminal investigation and surveillance tasks: they can serve as auxiliary tools, but there is still a clear gap before they can independently take on investigative work. For agencies considering deploying such systems, KierstonBench provides a quantifiable evaluation method to help determine the reliability boundaries of models in specific tasks.

Abstention in evaluation refers to a model choosing not to give a conclusion when evidence is insufficient or uncertain, rather than forcing a guess. In criminal investigation and surveillance scenarios, abstention is a critical safety behavior: erroneous identification conclusions can lead to misjudgments or violate the rights of innocent people. KierstonBench treats abstention as an evaluable behavioral dimension, requiring models to explicitly state that they cannot make a judgment when records do not support a conclusion. A few frontier models demonstrated this ability, showing they can recognize evidentiary boundaries; but even these models still fall into deception traps, indicating that abstention behavior has not yet been systematically and reliably trained. This dimension offers a new perspective for evaluating model reliability.

Why This Benchmark Is Worth Attention

The demand for AI in criminal investigation and surveillance is real, but the lack of quantifiable capability evaluation has been a bottleneck. The combination of Kierston and KierstonBench provides, for the first time, a repeatable, scorable testing framework for these capabilities in a simulated environment. Its city-scale size and cross-source record design bring it closer to the complexity of real investigations than simplified tests.

For the AI research community, this benchmark also raises a thought-provoking question: can the ability to abstain when evidence is insufficient and to draw conclusions when evidence is sufficient be systematically trained and improved? KierstonBench's 32 task families provide an experimental foundation for answering this question.

One-line Summary

Researchers introduce the Kierston simulation engine and KierstonBench benchmark to evaluate LLM investigative capabilities in criminal investigation and surveillance.

Paragraph Summary

Applications of large language models in criminal investigation and surveillance lack quantifiable capability evaluation. To address this, researchers built Kierston—a city-scale engine simulating approximately 125,000 residents of Allentown, Pennsylvania, generating multi-source records including telecommunications, financial, administrative, and license plate data, while ensuring cross-source data consistency. Based on this, KierstonBench contains 32 task families covering record linking, trajectory reconstruction, event investigation, and deceptive statement assessment. Testing of 16 models shows that most cannot conduct record investigations and submit identification conclusions lacking evidentiary support; a few frontier models can solve most tasks and abstain when evidence is insufficient, but still fall into deception traps, and none can replace an investigator. The benchmark fills the gap left by real deployments' lack of ground truth, providing a repeatable, scorable testing framework for evaluating the reliability boundaries of models in criminal investigation and surveillance.