Authors: Nate Breznau, Eike Mark Rinke, Alexander Wuttke, … Marcel Neunhoeffer, … et al. (166+ researchers)
Published in: Proceedings of the National Academy of Sciences (PNAS), 2022
Abstract
This study explores how researchers’ analytical choices affect conclusions drawn from the same data. We coordinated 73 research teams to independently analyze identical data and test the same hypothesis about immigration and public support for social policies. Despite using the same data, teams reached meaningfully different conclusions due to variations in their analytical approaches—a phenomenon known as “researcher degrees of freedom.”
The results reveal a hidden universe of uncertainty in empirical research: the variation in results across teams was substantial, demonstrating that analytical flexibility can lead to divergent findings even when researchers have access to identical information.
Key Findings
- 73 teams analyzed the same dataset with the same research question
- Results varied significantly across teams due to different analytical choices
- This variation reveals hidden uncertainty often not captured in single studies
- Findings have important implications for research transparency and replication
Significance
This large-scale collaboration demonstrates the importance of:
- Pre-registration and transparent reporting of analytical choices
- Multi-analyst approaches to assess robustness of findings
- Acknowledging uncertainty in empirical research conclusions