Authors: Nate Breznau, Eike Mark Rinke, Alexander Wuttke, … Marcel Neunhoeffer, … et al. (185 researchers)
Published in: Royal Society Open Science, 12(3), 241038 (2025)
DOI: 10.1098/rsos.241038
Abstract
This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a ’transparent group’ receiving original study and code or ‘opaque group’ receiving only a method and results description and no code.
Key Findings
- The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%)
- Second-decimal place exact numerical reproductions were less common (76.9% and 48.1% respectively)
- Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations
- When curating mistakes, only the transparent group was reliably successful
Significance
The findings imply a need for transparency, but also institutional checks and less subjective difficulty for researchers ‘doing reproduction’, implying a need for better training.
Connection to Previous Work
This paper is a follow-up to “Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty” (PNAS, 2022), extending the analysis to focus specifically on computational reproducibility aspects of the multi-analyst framework.