TY - GEN
T1 - CausalProfiler: Generating Synthetic Benchmarks for Rigorous and Transparent Evaluation of Causal Machine Learning
AU - Panayiotou, Panayiotis
AU - Poinsot, Audrey
AU - Leite, Alessandro
AU - Chesneau, Nicolas
AU - Schoenauer, Marc
AU - Şimşek, Özgür
PY - 2026/6/1
Y1 - 2026/6/1
N2 - Causal machine learning aims to answer "what if" questions using machine learning algorithms, making it a promising tool for high-stakes decision-making. Yet, empirical evaluation practices remain limited. Existing benchmarks often rely on a handful of hand-crafted or semi-synthetic datasets, leading to brittle, non-generalizable conclusions. To bridge this gap, we introduce CausalProfiler, a synthetic benchmark generator for causal machine learning methods. Based on a set of explicit design choices about the class of causal models, queries, and data considered, CausalProfiler randomly samples causal models, data, queries, and ground truths constituting the synthetic causal benchmarks. In this way, causal machine learning can be rigorously and transparently evaluated under a variety of conditions. This work offers the first random generator of synthetic causal benchmarks with coverage guarantees and transparent assumptions operating on the three levels of causal reasoning: observation, intervention, and counterfactual. We demonstrate its utility by evaluating several state-of-the-art methods under diverse conditions and assumptions, both in and out of the identification regime, illustrating the types of analyses and insights CausalProfiler enables.
AB - Causal machine learning aims to answer "what if" questions using machine learning algorithms, making it a promising tool for high-stakes decision-making. Yet, empirical evaluation practices remain limited. Existing benchmarks often rely on a handful of hand-crafted or semi-synthetic datasets, leading to brittle, non-generalizable conclusions. To bridge this gap, we introduce CausalProfiler, a synthetic benchmark generator for causal machine learning methods. Based on a set of explicit design choices about the class of causal models, queries, and data considered, CausalProfiler randomly samples causal models, data, queries, and ground truths constituting the synthetic causal benchmarks. In this way, causal machine learning can be rigorously and transparently evaluated under a variety of conditions. This work offers the first random generator of synthetic causal benchmarks with coverage guarantees and transparent assumptions operating on the three levels of causal reasoning: observation, intervention, and counterfactual. We demonstrate its utility by evaluating several state-of-the-art methods under diverse conditions and assumptions, both in and out of the identification regime, illustrating the types of analyses and insights CausalProfiler enables.
M3 - Chapter in a published conference proceeding
VL - 306
T3 - Proceedings of Machine Learning Research
SP - 96055
EP - 96110
BT - Proceedings of the 43rd International Conference on Machine Learning
A2 - Zhang, Tong
A2 - Dudik, Miroslav
A2 - Jaggi, Martin
A2 - Agarwal, Alekh
A2 - Li, Sharon
A2 - Schuurmans, Dale
A2 - Zhu, Jerry
A2 - Berkenkamp, Felix
A2 - Dong, Hanze
A2 - Bietti, Alberto
PB - PMLR
ER -