Dear Colleagues and Friends,
When scientific findings lose credibility, as is exemplified by the current challenge facing the Behavioral Science community, they also lose their usefulness to society. While many aspects of scientific research affect credibility, reproducibility and replicability of research findings are critical. Recent examinations of empirical studies in the social sciences, however, paint a concerning picture of non-reproducibility and non-replicability, posing a potential threat to the integrity of academic pursuits. Various factors contribute to the replicability challenges of social science research, including publication bias, undisclosed analysis, p-hacking (the exploitation of data analysis to make a preferred hypothesis appear significant), and outright fraud.
Addressing these issues requires sharing data and code so that published results can be reliably reproduced. Additionally, it is important to understand the extent to which published results can be replicated. In 2019, Management Science, as a premier business and management journal, introduced a Data and Code Disclosure Policy (for short, 2019 Policy) with the goal of ensuring the availability of the material necessary to replicate the research published in the journal. Recently, the journal undertook two projects aimed at assessing reproducibility and replicability of published results.
The Management Science Reproducibility Project
Fišar et al. (2023) aim to establish the level of reproducibility of Management Science articles before and after the new Data and Code Disclosure Policy was established. To assess the reproducibility of results, a team of five academics collaborated to undertake the Management Science Reproducibility Project. The team includes Miloš Fišar, Masaryk University; Ben Greiner and Christoph Huber, Vienna University of Business and Economics; Elena Katok, University of Texas at Dallas; and Ali Ozkes, SKEMA Business School. The team employed a crowd-science approach to this project, enlisting the expertise of 733 volunteers from the various journal sub-fields to establish the Management Science Reproducibility Collaboration. The collaboration attempted to reproduce published results of nearly 500 articles in Management Science that were published before and after the introduction of the 2019 Policy. These "reproducibility reviewers" invested over 6,500 collective hours to replicate results using the replication material provided by the respective authors.
Figure 1 from Fišar et al. (2023)
Articles subject to the 2019 Policy demonstrated an impressive 95.3% reproducibility rate (fully or largely reproduced) when reviewers had access to necessary data and could meet the hard- and software requirements of the analysis. However, approximately 29% of articles could not be reproduced because datasets were unavailable either because they were proprietary or only accessible through a subscription. Reproducibility rates varied by discipline primarily because different disciplines employ different methodologies, e.g., laboratory experiments and computation/simulation studies were more reproducible than studies that rely on field data. By contrast, before the 2019 Policy was implemented, provision of data and code was voluntary, and the vast majority of articles (88%) did not include those material. Of those that did, about 55% could be reproduced, even though this is a highly biased sample. See Figure above for details.
The Management Science Replicability Project
Reproducibility does not imply replicability but serves as an important foundation for evaluating replicability. Earlier this year, in another Management Science initiative, Davis et al. (2023) focused on the replicability of behavioral experiments in operations management, to firmly establish (or not) key findings of highly impactful experimental papers through high-powered replication across multiple locations. In that study, eight researchers from five universities attempted to replicate key results from ten experimental Management Science papers published since 2000. The faculty involved include Andrew Davis, Cornell University; Blair Flicker, University of South Carolina; Kyle Hyndman and Elena Katok, University of Texas at Dallas; Samantha Keppler and Stephen Leider, University of Michigan; and Xiaoyang Long and Jordan Tong, University of Wisconsin.
The team collected survey results from the community, asking participants to vote for the papers they would like to see replicated. The papers were in the following five areas: inventory management, supply chain contracts, queueing, forecasting, and sourcing. The team selected two papers with the highest number of votes from each category, for a total of 10 papers. Each paper was replicated at two different sites. Of the ten papers, six were fully replicated (at two locations), two were partially replicated (at one of the locations), and two failed to replicate (at either location).
Summary
The implications of Fišar et al. (2023) and Davis et al. (2023) speak to the larger academic community, beyond Management Science. Improving the reproducibility of social science research in the face of recent challenges requires authors to provide essential material that allow others to reproduce their results. In essence, the Fišar et al. (2023) study provides a roadmap for enhanced scientific reproducibility, offering insights that extend beyond the realm of Management Science, shaping the future of rigorous and trustworthy scientific inquiry. The Davis et al. (2023) study provides an example of how to conduct rigorous large-scale replications. Going beyond behavioral operations management, and beyond the Management Science journal, the study speaks to the importance of encouraging efforts that either replicate or fail to replicate published results.
Happy Holidays and Best Wishes for the New Year!
David Simchi-Levi
Editor-in-Chief
Management Science
References:
Andrew M. Davis, Blair Flicker, Kyle Hyndman, Elena Katok, Samantha Keppler, Stephen Leider, Xiaoyang Long, Jordan D. Tong (2023) A Replication Study of Operations Management Experiments in Management Science. Management Science 69(9):4977-4991.
Miloš Fišar, Ben Greiner, Christoph Huber, Elena Katok, Ali I. Ozkes (2023) Reproducibility in Management Science. Management Science, Reproducibility in Management Science | Management Science (informs.org).
------------------------------
David Simchi-Levi
Professor of Engineering Systems
Massachusetts Institute of Technology
Cambridge MA
------------------------------