Artifact Evaluation Track

ICSA 2021 will be the first ICSA featuring an Artifact Evaluation Track. An artifact evaluation track aims to review and promote the research artifacts of accepted papers.

Artifacts can be software systems, scripts, or datasets. High quality artifacts of published research papers are a foundation for the research results to be reproduced by other researchers and are thus a desirable part of the publication itself.

The artifact evaluation system is based on the upcoming NISO Recommended Practice on Reproducibility Badging and Definitions, which is supported by our publisher, IEEE, and the evaluation criteria are also inspired by ACM’s artifact review and badging system as well as the criteria used by ICSE 2021.

Who can submit?

Authors of the papers accepted to the following ICSA 2021 and 2020 tracks are invited to submit an artifact for evaluation for the Research Object Reviewed (ROR) – reusable badge and the Open Research Object (ORO) badge: Technical Track, Journal-First Track, Software Architecture in Practice Track, and New and Emerging Ideas Track. All authors of papers related to the topics mentioned in the call for papers of the ICSA technical track invited to submit studies for the Results Reproduced (ROR-R) and Results Replicated (RER) badges.

In addition, authors of any prior software architecture research are invited to submit an artifact for evaluation as Results Reproduced (ROR-R) and Results Replicated (RER).

Please note that we require one author of each artifact submission to peer-review 2-3 other artifacts!

Evaluation Criteria and Badges

Evaluation criteria for badges and additional information have been taken from ICSE 2021, which are based on the ACM policy on Artifact Review and Badging Version 1.1, and from the upcoming NISO Recommended Practice on Reproducibility Badging and Definitions, which is supported by our publisher, IEEE.

The ICSA artifact evaluation track uses a single-blind review process. Artifacts will be evaluated using the criteria in the last row of the table below. The goal of this track is to encourage reusable research products. Hence, no functional badges will be awarded.

Functional	Research Object Reviewed (ROR) – Reusable	Open Research Object (ORO)	results Reproduced (ROR-R)	results replicated (RER)
No Badge
Artifacts documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation	Functional + very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. In particular, norms and standards of the research community for artifacts of this type are strictly adhered to.	Functional + placed on a publicly accessible archival repository. A DOI or link to this repository along with a unique identifier for the object is provided. (this matches the ACM “Available” badge)	ROR + ORO + main results of the paper have been obtained in a subsequent study by a person or team other than the authors, using, in part, artifacts provided by the author.	ROR + ORO + the main results of the paper have been independently obtained in a subsequent study by a person or team other than the authors, without the use of author-supplied artifacts.

Functional

Research Object Reviewed (ROR) – Reusable

Open Research Object (ORO)

results Reproduced (ROR-R)

results replicated (RER)

No Badge

Artifacts documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation

Functional + very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. In particular, norms and standards of the research community for artifacts of this type are strictly adhered to.

Functional + placed on a publicly accessible archival repository. A DOI or link to this repository along with a unique identifier for the object is provided.

(this matches the ACM “Available” badge)

ROR + ORO + main results of the paper have been obtained in a subsequent study by a person or team other than the authors, using, in part, artifacts provided by the author.

ROR + ORO + the main results of the paper have been independently obtained in a subsequent study by a person or team other than the authors, without the use of author-supplied artifacts.

Papers with such badges contain reusable products that other researchers can use to bootstrap their own research. Experience shows that such papers earn increased citations and greater prestige in the research community. Artifacts of interest include (but are not limited to) the following:

Software, which are implementations of systems or algorithms potentially useful in other studies.
Data repositories, which are data (e.g., logging data, system traces, survey raw data) that can be used for multiple software engineering approaches.
Frameworks, which are tools and services illustrating new approaches to software engineering that could be used by other researchers in different contexts.

This list is not exhaustive, so the authors are asked to email the chairs before submitting if their proposed artifact is not on this list. Further information on data sharing principles and approaches are further introduced along an introduction of the general notion of open science in the book chapter Open Science in Software Engineering by Méndez, Graziotin, Wagner, and Seibold: https://arxiv.org/abs/1904.06499.

General information and further suggested reading about artifact evaluation can be viewed in a public draft standard by Baldassarre, Ernst, Hermann and Menzies.

The reviewers for the artifacts will be a combination of artifact authors and ICSA program committee members.

The best artifact selected by the reviewers will be awarded the best artifact award.

For accepted ICSA 2020 papers, we plan to integrate the batch on the paper in the official IEEE proceedings. For 2020 papers, this may not be possible.

Important Dates

Artifact Evaluation Registration Deadline: February 2, 2021 (mandatory)

Artifact Evaluation Submissions Deadline: February 5, 2021

Artifact Evaluation Notification: February 26, 2021

Submission and Review

The submission and review criteria are mainly taken from ICSE 2021, with some extensions.

IMPORTANT NOTE: different badges have different submission procedures. See below.

For Results Reproduced (ROR-R) and Results Replicated (RER) Badges

For Results Reproduced (ROR-R) and Results Replicated (RER) badges, authors will need to offer appropriate documentation that their artifacts have reached that stage.

By February 02, 2021 register your research artifact at the artifact track at the ICSA EasyChair site by submitting a one page (max) abstract in PDF format describing your artifact.

The abstract should include the paper title, the purpose of the research artifact, the badge(s) you are claiming, and the technology skills assumed by the reviewer evaluating the artifact. Please also mention if running your artifact requires specific Operating Systems or other environments.

TITLE: A (Partial)? (Replication|Reproduction) of XYZ . Please add the term partial to your title if only some of the original work could be replicated/reproduced.
WHO: name the original authors (and paper) and the authors that performed the replication/reproduction. Include contact information and mark one author as the corresponding author.
IMPORTANT : include also a web link to a publically available URL directory containing (a) the original paper (that is being reproduced) and (b) any subsequent paper(s)/documents/reports that do the reproduction.
WHAT: describe the “thing” being replicated/reproduced;
WHY: clearly state why that “thing” is interesting/important;
HOW: say how it was done first;
WHERE: describe the replication/reproduction. If the replication/reproduction was only partial, then explain what parts could be achieved or had to be missed.
DISCUSSION: What aspects of this “thing” made it easier/harder to replicate/reproduce. What are the lessons learned from this work that would enable more replication/reproduction in the future for other kinds of tasks or other kinds of research.

Two reviewers will review each abstract, possibly reaching out to the authors of the abstract or original paper. Abstracts will be ranked as follows.

If the reviewers do not find sufficient substantive evidence for replication/reproduction, the abstract will be rejected.
Any abstract that is judged to be unnecessarily critical of prior work will be rejected (*).
The remaining abstracts will be sorted according to (a) how interesting they are to the community and (b) their correctness.
The top ranked abstracts will be invited to give lightning talks.

(*) Our goal is to foster a positive environment that supports and rewards researchers for conducting replications and reproductions. To that end, we require that all abstracts and presentations pay due respect to the work they are reproducing/replicating. Criticism of prior work is acceptable only as part of a balanced and substantive discussion of prior accomplishments.

In addition, all instructions listed below for Research Object Reviewed (ROR) – reusable badge and the Open Research Object (ORO) badge also have to be followed for the Results Reproduced (ROR-R) and Results Replicated (RER) badges.

For Research Object Reviewed (ROR) – reusable and Open Research Object (ORO) Badges

Only authors of papers accepted to the following ICSA 2021 and 2020 tracks are invited to submit an artifact for evaluation for the Research Object Reviewed (ROR) – reusable badge and the Open Research Object (ORO) badge: Technical Track, Journal-First Track, Software Architecture in Practice Track, and New and Emerging Ideas Track.

By February 02, 2021 register your research artifact at the artifact track at the ICSA EasyChair site by submitting a one page (max) abstract in PDF format describing your artifact.

For the Research Object Reviewed (ROR) – reusable badge and the Open Research Object (ORO) badge, authors must offer “download information” showing how reviewers can access and execute (if appropriate) their artifact.

Authors must perform the following steps to submit an artifact:

Preparing the artifact
Making the artifact publicly available (by using repositories granting public access)
Documenting the artifact
Submitting the artifact
Clarification period

1. Preparing the Artifact

There are two options depending on the nature of the artifacts: Installation Package or Simple Package. In both cases, the configuration and installation for the artifact should take less than 30 minutes. Otherwise, the artifact is unlikely to be endorsed simply because the committee will not have sufficient time to evaluate it.

Installation Package. If the artifact consists of a tool or software system, then the authors need to prepare an installation package so that the tool can be installed and run in the evaluator’s environment. Provide enough associated instruction, code, and data such that some CS person with a reasonable knowledge of scripting, build tools, etc. could install, build, and run the code. If the artifact contains or requires the use of a special tool or any other non-trivial piece of software the authors must provide a virtual machine in the Open Virtual Appliance (OVA) format (e.g., using Oracle Virtual Box or VMware), or a Docker container image with a working environment containing the artifact and all the necessary tools. Alternatively, authors can provide a link to a webservice that allows reviewers to execute the artifact (e.g., using Jupyter notebooks).

We expect that the artifacts have been vetted on a clean machine before submission.

Simple Package. If the artifact only contains documents which can be used with a simple text editor, a PDF viewer, or some other common tool (e.g., a spreadsheet program in its basic configuration) the authors can just save all documents in a single package file (zip or tar.gz).

2. Making the Artifact Available

The authors need to make the packaged artifact (installation package or simple package) available so that the Evaluation Committee can access it. We suggest a link to a public repository (e.g., GitHub) or to a single archive file in a widely available archive format.

If the authors are aiming for the badges Open Research Object (ORO) and beyond, the artifact needs to be publicly accessible in an archival repository that guarantees long-time storage. We suggest to use Zenodo, figshare, or similar free services that provide Digital Object Identifiers (DOIs).

In other cases, the artifacts do not necessarily have to be publicly accessible for the review process. In this case, the authors are asked to provide a private link or a password-protected link. In any case, we encourage the authors to use permanent repositories dedidated at data sharing where no registration is necessary for those accessing the artifacts (e.g., please avoid using services such as GoogleDrive).

3. Documenting the Artifact

The authors need to write and submit a documentation explaining how to obtain the artifact package, how to unpack the artifact, how to get started, and how to use the artifacts in more detail. The artifact submission must only describe the technicalities of the artifacts and uses of the artifact that are not already described in the paper.

The submission should contain the following documents (in plain text or pdf format) in a zip archive:

A README main file describing what the artifact does and where it can be obtained (with hidden links and access password if necessary). Also, there should be a clear step-by-step description how to reproduce the results presented in the paper (i.e. tables, figures and other reported results), including links to all necessary scripts and input data.
If the results presented in the paper take more than 1 hour on a standard laptop to be reproduced, authors are expected to additionally provide simplified examples with runtimes of less than one hour.
If the results in the paper have been generated with an artifact using data that underlies non-disclosure agreements, authors should provide artifical replacement data that allows reviewers to assess the artifact independently of that data.
A STATUS file stating what kind of badge(s) the authors are applying for as well as the reasons why the authors believe that the artifact deserves that badge(s).
A LICENSE file describing the distribution rights. Note that to score Open Research Object (ORO) or higher, then that license needs to reflect some form of open source license.
An INSTALL file with step-by-step installation instructions (if any). These instructions should include notes illustrating a very basic usage example or a method to test the installation. This could be, for instance, on what output to expect that confirms that the code is installed and working; and the code is doing something interesting and useful,
A copy of the accepted paper in pdf format.

Note that for the badge Research Object Reviewed (ROR) – reusable, extensive documentation of the artifact is needed that allows other researchers to repurpose the tool. For software artifacts, this requires to include an overview of the artifact source code and its architecture.

4. Submitting the Artifact

By February 02, 2021 register your research artifact at the ICSA EasyChair site. Please use the submission form fields as follows:

The title field should contain the name of the original paper to which the artifact belongs, preceeded or followed by the name of the artifact, if any.
The abstract should describe research artifact so that reviewers can bid for it, including, at the end teh required OS (if any) and used virtualization technique.

The remaining submission fields are explained at the submission site.

By February 05, 2021 complete your artifact submission by uploading the zip archive containing the documentation and providing the link to the software.

Before the actual evaluation, reviewers will check the integrity of the artifact and look for any possible setup problems that may prevent it from being properly evaluated (e.g., corrupted or missing files, VM won’t start, immediate crashes on the simplest example, etc.).

5. Clarification period

During the review, the reviewers may contact the authors to request clarifications on the basic installation and start-up procedures, to resolve simple installation problems or other questions. More details on this procedure can be found below in Section “Communication during Clarification Period”.

Reviewer Guidelines

Artifact bidding

Please bid on artifacts during the bidding phase using Easychair. Please consider the platform and OS in information provided by the authors when making your bids.

Artifact review

Please assess the artifacts based on the criteria for the badges the authors have applied for (see table above). If in doubt about how to assess a property, please use the Easychair discussion (after submitting a draft review or even an almost empty review) to contact your fellow reviewers as well as the track chairs to discuss. Additionally, if you notice any potential ethicial or legal issues such as unfit licencing (too restrictive? too permissive?), institutional review of human subjects research (the IRB certificate should form part of the submission), data provenance (is the dataset biased?), and privacy protection (can individual information be obtained?), please ask the authors to clarify and additionally contact the track chairs.

If you need clarifications on the basic installation and start-up procedures, to resolve simple installation problems or other questions, please use the communication channel described below to contact the authors.

Please try to install the artifacts soon after assignment but before February 13, so that the authors have time to respond to potential problems.

Clarification period

We will establish a communication channel between authors and reviewers of an artifact in which reviewers stay autonomous (see Section “Communication during Clarification Period” below.)

In this channel, please feel free to also share a draft of your review so that authors can comment on it. When doing so, please do not use any formatting, as the final reviews shall be transferred into Easychair, which only supports plain text.

Feel free to add an early draft of your review to Easychair as well and revise it later, so that you can use Easychair to privately discuss with other reviewers as well, even before coming up with a recommendation yourself.

Final artifact review

Please enter your final review into Easychair and comment on whether the artifact shall receive the badges the authors have applied for. If the authors have applied for multiple badges, please comment on these individually. A possible outcome of your reviews may be that an artifact only receives a subset of the badges the authors applied for.

Communication during Clarification Period

For each submission, an anonymous Google Docs file will be set up for each artifact by the artifact evaluation chairs. Reviewers will receive a link to this document so that they can edit it anonymously. Authors will be invited with their e-mail address, so their names will be visible in their edits. Authors are expected to set up a notification for this document for their artifact to get informed about reviewer questions.

Authors will be invited to document during artifact bidding and reviewers will receive the link to it after artifact assignment.

Given the short review time available, the authors are expected to respond within a 48-hour period. Authors may update their research artifact after submission only for changes requested by reviewers in the clarification period.

Reviewers are requested to consider at least one update to the artifact or to the usage instructions by the authors, but are invited to consider as many updates and replies by the authors as needed to clarify an issue.

In the document, please follow the following conventions

Please start each contribution you make to the document with an identifier in square brackets (authors: your name, reviewers: the “label” Google provides (such as “anonymous squirrel”) or a label of your choice that has not been used by others for this artifact, e.g. “Reviewer 1”)
When replying to a contribution, please edit right below it and indent your reply. Subsequent replies can stay on that same level of indentation. Please also include your identifier in square brackets for every reply.
authors submitting an open source repository link, are expected to give a tag to time-stamp the submissions.

If authors or reviewers are not allowed to use Google Docs because of constraints of their organization, please inform the artifact track chairs as soon as possible, so that alternatives can be found.

Artifact Evaluation Committee

Allen, Claudine, University of The West Indies, Mona, Jamaica
Baresi, Luciano, Politecnico di Milano, Italy
Diaz-Pace, J. Andres, ISISTAN Research Institute, UNICEN University, Argentina
Ernst, Neil, University of Victoria, Canada
Liang, Peng, Wuhan University, China
Maia, Marcelo, UFU, Brazil
Muccini, Henry, University of L’Aquila, Italy
Männistö, Tomi, University of Helsinki, Finland
Nakagawa, Elisa Yumi, University of S‹o Paulo, Brazil
Navarro, Elena, University of Castilla-La Mancha, Spain
Oquendo, Flavio, IRISA (UMR CNRS) – Univ. Bretagne-Sud (UBS), France, France
Perez, Jennifer, UPM, Spain
Tamburri, Damian Andrew, Eindhoven University of Technology, Netherlands
Trubiani, Catia, Gran Sasso Science Institute, Italy
Velasco-Elizondo, Perla, Autonomous University of Zacatecas, Mexico
von Flach, Christina, UFBA, Brazil
Zdun, Uwe, University of Vienna, Austria
Challita, Stephanie, INRIA, France
Gilson, Fabian, University of Canterbury, New Zealand
Solis, Carlos, Acuris Global, UK
Gerostathopoulos, Ilias, Vrije Universiteit Amsterdam, Netherlands
Ullah, Faheem, University of Adelaide, Australia

In case of questions, please do not hesitate to contact the chairs: Anne Koziolek, Marie Platenius-Mohr