Composable Causality: A Benchmark Toolkit for Time-Series Causal Discovery and Intervention Effects

Research on time-series causal discovery and intervention effect estimation has long relied on synthetic data, yet real-world settings rarely come with validated causal structures. Existing tools are either fixed datasets that cannot be modified after release, or script generators where each script hardcodes a single combination of mechanisms, confounding, and missingness. Neither supports controlled experiments that change a single property (such as sparsity, noise distribution, or missingness mechanism) while holding other properties constant. Counterfactual evaluation has the same gap, since existing simulators are mostly built per paper, hardcode mechanisms, or only generate observational data.

According to the DTCD (Datasets and Toolkit for Causal Discovery) project, this library is designed to address these problems. It is a data generation library that produces multivariate time series with ground-truth contemporaneous and lagged adjacency matrices. Its core design treats topology, mechanisms, noise, observation masks, and intervention policies as independent components that are combined purely through configuration. This means researchers can systematically vary one dimension while controlling the rest, enabling genuinely controlled experiments.

From Fixed Datasets to Composable Generation

In traditional approaches, a researcher who wants to test an algorithm's robustness to missing values often has to find a dataset that happens to contain missing values, or manually modify the generation script. This is not only hard to reproduce but also prone to introducing unintended variables. DTCD's componentized approach makes each property a pluggable module: topology type, causal mechanism, noise distribution, missingness mechanism, and intervention policy are selected via a configuration file, while everything else stays fixed.

This design directly supports ablation studies. For example, when evaluating a causal discovery algorithm, one can fix the mechanism and noise and vary only sparsity to observe how performance changes with graph complexity. Likewise, in intervention effect estimation, one can fix the data-generating process and vary only the intervention policy, thereby isolating the estimator's sensitivity to intervention design.

Counterfactual twin is a construct used in causal inference for individual-level intervention effect evaluation: for the same individual, a "factual" trajectory (unintervened) and a "counterfactual" trajectory (with intervention) are generated separately, identical in all conditions except the intervention. DTCD achieves this by caching the random numbers of the factual rollout and re-injecting them into the intervention rollout, so that both trajectories share the same random noise and the difference is attributable purely to the intervention. This numerically exactly recovers individual intervention effects, avoiding the problem where random noise differences between independently generated trajectories drown out the intervention signal.

Paired Counterfactuals and Numerical Precision

Counterfactual evaluation is central to causal inference, but synthetic data usually provide only observational trajectories. DTCD generates paired counterfactual twin data by caching the random realization in the factual rollout and re-injecting that realization in the intervention rollout. The paper states that this approach can recover individual intervention effects to numerical precision. For research that needs to evaluate individual-level intervention effect estimators, this provides a reproducible benchmarking means.

The significance of the paired design is that it eliminates confounding from randomness. In traditional simulation, if factual and counterfactual trajectories are generated independently, random noise differences drown out the intervention signal. Through caching and re-injection, the two rollouts share the same random seed realization, so the difference arises solely from the intervention itself, making the computation of individual intervention effects exact.

Observation mask is a boolean structure in time-series data generation used to mark whether each variable at each time point is observed. It allows missingness mechanisms (such as missing completely at random, missing at random, and missing not at random) to be configured independently of the data-generating process. DTCD treats the observation mask as an independent component, allowing researchers to change only the missingness pattern while keeping topology, mechanism, and noise fixed, thereby systematically assessing the impact of missing values on causal discovery and intervention effect estimation.

Benchmarks Reveal Failure Modes Hidden by Aggregate Metrics

DTCD comes with a reference benchmark covering eight discovery algorithms across eighty-one configurations and four intervention effect estimators across eighteen configurations. The benchmark results expose failure modes that aggregate metrics tend to hide.

One finding is that standard discovery methods cannot handle missing values, regardless of whether the missingness mechanism is informative. This means their reliability is questionable in missing-data scenarios common in real-world data. Another finding concerns the marginal variance shortcut (varsortability): some algorithms may exploit the marginal variance ordering of variables to infer causal direction rather than genuinely learning causal structure. DTCD standardizes measurements and removes this shortcut, making the benchmark better reflect an algorithm's true ability to identify causal structure.

These failure modes are often averaged away in a single aggregate score, but for practical applications they may determine whether a method is usable. DTCD's configurable benchmark lets researchers pinpoint an algorithm's weak spots under specific conditions.

Varsortability refers to the phenomenon where the marginal variance ordering of variables in the data happens to coincide with the causal order. Some causal discovery algorithms may exploit this statistical shortcut to infer direction rather than genuinely learning causal structure. In real data, variance ordering and causal order are not necessarily consistent, so algorithms that rely on this shortcut may fail out of distribution. DTCD standardizes measurements and removes this shortcut, making the benchmark better reflect an algorithm's true ability to identify causal structure.

Implications for Causal Research Practice

DTCD's value lies in transforming data generation from "one-off scripts" into a "configurable experimentation platform." Researchers can design factorial experiments to systematically explore algorithm behavior under different data properties, rather than relying on a handful of fixed datasets. This is especially important for the field of time-series causal discovery, where real data lack ground truth and synthetic data are the primary means of evaluation.

The project code and data are open-sourced on GitHub (hferdous/Composable_Causality). For researchers working on causal discovery, intervention effect estimation, or time-series analysis, this tool offers a more controllable way to build benchmarks. Its componentized design also lowers the barrier to reproduction and extension, helping to drive more rigorous empirical comparisons.

However, the tool itself still relies on the assumptions of synthetic data. Whether the configured mechanisms and noise distributions cover the complexity of real scenarios must be judged by users for their specific problems. The failure modes revealed by the benchmark should be treated as warnings rather than final verdicts on algorithmic capability.