SC’16 introduced the first reproducibility efforts for the SC conference series technical papers program. This initial effort followed the strategy of Result and Artifact Review and Badging, and asked for volunteers to complete an appendix to their paper that described the details of their software environment and computational experiments to the extent that an independent person could replicate their results. In SC’16, 9 authors submitted appendices.
The format changed slightly with SC’17, with the introduction of a second appendix describing how computational results were obtained. In SC’17, more than half of the accepted papers had submitted a reproducibility appendix.
For ISAV 2018, we adopt the following approach following the model of SC17:
Artifact Description Appendix: We will use the format of one of the SC18 appendix (with minor revisions) for ISAV 2018 submissions. Authors will provide the completed appendix (at most 2 pages), along with their submission.
Notes:
The format is available [see attached .tex template]
Q. Is the Artifact Description appendix required in order to submit a paper to ISAV 2018?
A. No. These appendices are not required. If you do not submit any appendix, it will not disqualify your submission. At the same time, if two papers are otherwise comparable in quality, the existence and quality of appendices can be a factor in ranking one paper over another.
Q. Do I need to make my software open source in order to complete the Artifacts Description appendix?
A. No. It is not required that you make any changes to your computing environment in order to complete the appendix. The Artifacts Description appendix is meant to provide information about the computing environment you used to produce your results, reducing barriers for future replication of your results. However, in order to be eligible for the ACM Artifacts Available badge (see below), your software must be downloadable by anyone without restriction.
Q. Who will review my appendices?
A. The Artifact Description and Computational Results Analysis appendices will be submitted at the same time as your paper and will be reviewed as part of the standard review process by the same reviewers who handle the rest of your paper.
Q. Does the Artifacts Description appendix really impact scientific reproducibility?
A. The Artifacts Description appendix is simply a description of the computing environment used to produce the results in a paper. By itself, this appendix does not directly improve scientific reproducibility. However, if this artifact is done well, it can be used by scientists x(including the authors at a later date) to more easily replicate and build upon the results in the paper. Therefore, the Artifacts Description appendix can reduce barriers and costs of replicating published results. It is an important first step toward full scientific reproducibility.
Q. What are my responsibilities when reviewing a paper that has an Artifacts Description appendix?
A. Reviewers are expected to read any appendix that is submitted with a paper. The information in these appendices is often included in any well-written paper. By collecting this information in a uniform way, we hope that the review process will be more effective without requiring a significant new burden on reviewers.
Q. Must I confirm that links in an appendix work?
A. Confirming links in an appendix are not required of every reviewer. We are asking for volunteers from the review committee who would be willing to validate links provided in appendices.
Q. Must I evaluate the code and results associated with a Computational Results Analysis appendix?
A. Code and results evaluations associated with a Computational Results Analysis appendix are not required of every reviewer. We are asking for volunteers from the review committee who would be willing to evaluate code and results associated with these appendices.
Q. How may I use appendices as part of the review process?
A. The absence or poor quality of an appendix cannot be used to categorically disqualify a paper. Furthermore, inaccessibility of artifacts such as source code cannot be used to categorically disqualify a paper. However, the presence and quality of an appendix can be used as part of the review process. In particular, if two papers have similar ranking based on the manuscripts, the availability and quality of appendices can provide additional evaluation criteria.