ASA Develops Reproducible Research Recommendations

As statisticians play prominent roles in producing reproducible research and more publications and scientific agencies place greater emphasis on research being reproduced, the ASA has developed a guidance document that provides funding agencies with suggestions for how they can help support reproducible research. In showcasing the work of notable statisticians and the value of statistics to science, the document also raises the profile and enhances the voice of statisticians.

The document, “Recommendations to Funding Agencies for Supporting Reproducible Research,” begins by differentiating between two common—and often misused—terms: reproducibility and replicability. It then lists clearly defined principles that state reproducibility should follow “best practices,” should not be thought of as a binary state, and is the only effective guarantee in a published study. Observations of several barriers that exist in making research reproducible are also addressed, as are other challenges noting the availability of infrastructure resources and lack of funding models for data and code to be made public.

Some of the recommendations, which are specific to a given funding agency, include the following:

• Develop funding mechanisms to support small-scale software development and data products by researchers in domain areas, rather than software developers
• Fund work that includes an aim to reproduce and/or replicate previous work when that previous work is sufficiently important
• Provide support for the development of appropriate courses
• Consider including a code management plan as part of the current data management plan section of grant proposals
• Require reviewers of a grant proposal to make a specific assessment of the data management plan
• Provide best practices for what a researcher should do for their study to be seen as reasonably reproducible

ASA Science Policy Director Steve Pierson has been working with contributors Karl Broman, Mine Çetinkaya-Rundel, Amy Nussbaum, Christopher Paciorek, Roger Peng, Daniel Turk, and Hadley Wickham to produce the document. “This guidance will help amplify the statistical aspects, be helpful to ongoing reproducible research discussions by funding agencies and others, and propel efforts to improve science by encouraging more robust and reliable science,” noted Pierson.