Reproducibility PRINCIPLES: Join the discussion
The ACM’s base position is that, “an experimental result is not fully established unless it can be independently reproduced.” Reproducibility refers to the ability of a different team to arrive at the same scientific results using the same experimental setup. “For computational experiments, this means that an independent group can obtain the same result using the author’s own artifacts.”
This leaves many questions to be sorted out. Over the next few months, we will explore related issues such as principles, solutions, training, publishing, and preservation in the context of reproducibility.
We invite the community to join us for these essential conversations.
| PRINCIPLES | We will explicate ACM principles with respect to reproducibility. |
| SOLUTIONS | The current state of solutions and tools that support reproducibility. |
| TRAINING | How and where is scientific reproducibility taught? |
| PUBLISHING | Journals’ and conferences’ approaches to computational reproducibility. |
| PRESERVATION | Reproducibility in the long term requires curation and preservation. |
We begin with fundamental principles related to reproducibility. What are the principles we can agree on? What still needs to be clarified? To start the conversation, we identify a few key issues:
- Open source: What are the SIGs’ position on open-source solutions vs. proprietary solutions that support reproducibility? Are there promising approaches currently being tested or implemented, for example, around licensing?
- Ethics: What are the stated ethical guidelines that matter for reproducibility? e.g., algorithmic bias, unfair data practices.
- Research integrity: How does/should reproducibility touch on these issues? e.g., fraud, abuse, questionable research practices, etc.
- Labor: Who is responsible for the work of reproducibility? How should that work be rewarded?
Join us March 25, 2021 at 12 PM (EDT)!
Conversation will be moderated by the P-WG leaders: Limor Peer , Daniel Oberski , Vicky Rampin
Important information:
- Conversation: March 25, 2021, 12 -1 PM EDT (5-6 PM CET) [registration link]
- Collaborative notes: (google doc)
- Resources: The ACM has led significant efforts toward reproducibility (see summary)
What to expect: This is an interactive session. We are interested in “taking the pulse” and hearing your views about these issues. We’ll take a similar approach for each of the issues: An initial blog post describing the issue, community feedback via open meeting and online conversation, and proposed recommendations to advance the issue.
—
About this blog: P-WG is extending an invitation to the community to engage in conversation about topics related to the reproducibility of computational research via the blog, “Taking the Pulse.” In a series of open community meetings, the P-WG will explore issues related to reproducibility and document the views of different communities, culminating in recommendations informed by current practices. The P-WG will hold a forum for conversation about these topics over the course of several months starting in March 2021. Each month we will focus on a particular topic related to reproducibility: principles, solutions, training, publishing, and preservation.