ICSA Technical Track Guidelines for Reviewers
The following instructions cover the main aspects of the creation of reviews as part of the ICSA 2018 Technical Track evaluation process. We provide these instructions to create a common ground for decision making, and to support reviewers in delivering high-quality evaluation of the submissions.
Evaluation
All technical track submissions will be reviewed using a double-blind review process described under the Instructions for Authors. Eligible submissions will be reviewed by at least three members of the Program Committee (PC). The final decisions discussion will happen online and will be moderated by PC co-chairs.
Submission Quality Criteria
Technical track submissions shall be evaluated based on the following criteria (inspired from criteria used also by ICSE 2018):
- Relevance: The submissions need to respond to ICSA 2018 Technical track Call for papers. We especially welcome submissions responding to the theme of the conference, which is “Breaking Down Barriers”.
- Soundness: Are all claimed contributions supported by the rigorous application of appropriate research methods? The claims should be scoped to what can be supported, and limitations should be discussed.
- Significance: Are contributions evaluated for their importance and impact with respect to the existing body of knowledge? The authors are expected to explicitly argue for the relevance and usefulness of the research and discuss the novelty of the claimed contributions through a comparison with pertinent related work.
- Novelty: Is there sufficient originality in the contribution, and is it clearly and correctly explained with respect to the state of the art?
- Replicability: Is there sufficient information in the paper for the results to be independently replicated? The evaluation of submissions will take into account the extent to which sufficient information is available to support the full or partial independent replication of the claimed findings.
- Presentation Quality: Are results clearly presented? Submissions are expected to meet high standards of presentation, including adequate use of the English language, absence of major ambiguity, clearly readable figures and tables, and respect of the formatting instructions provided in ICSA 2018 Technical track Call for papers.
Review Quality Criteria
ICSA not only emphasizes high quality of accepted papers, but also high quality of reviews. All reviews are therefore expected to meet the following review quality criteria, which will be used when rating the quality of reviews as an input for the Best Reviewer Award.
Find below the review quality evaluation template:
1. Aspect “Helpfulness for Authors”
Helptext: This aspect is solely interested in how well the review aids the authors for improving their work and/or writing.
Facet “Is constructive”
- The review offers suggestions or ideas for how to make the actual improvements for all or nearly all points of substantial criticism.
- For much, but not nearly all criticism.
- Only rarely or never.
Facet “Provides evidence”
- The review provides good justification and explanation for all or nearly all points of substantial criticism.
- For much, but not nearly all criticism.
- Only rarely or never.
Facet “Uses a friendly tone”
- The review uses a friendly/polite tone throughout and phrases all its criticism gently.
- With some exceptions.
- Very often not.
Facet “Cites related work”
- The review provides actionable citations whenever relevant.
- With some exceptions.
- Not at all.
2. Aspect “Helpfulness for Decision”
Helptext: This aspect is solely interested in how well the PC chairs and/or the co-reviewers of the same article will be aided in making a good, balanced acceptance/rejection decision.
Facet “Is thorough in breadth”
- The review discusses all aspects of the article worth discussing, all across the paper.
- The review selectively discusses only a modest fraction of the relevant aspects.
- The review discusses hardly anything of relevance.
Facet “Criticism is specific”
- The review explains where exactly a criticism applies.
- It is only often or usually specific.
- Only rarely or never.
Facet “Criticism is justified”
- For its criticism, the review always provides reasoning or argumentation why a particular point is problematic and how: incomplete, illogical, inappropriate for the purpose, etc.
- The justification is missing for at least one important point of criticism.
- The justification is frequently missing.
Facet “Provides criticism and praise”
- The review mentions several non-trivial points of both criticism and of praise.
- Does so for either criticism or praise, but not both.
- Provides little substance at all.
Review Quality Checker Interface
Other Guidelines
Review Authorship
We ask PC members to personally author each review. Reviewers can ask students and/or colleagues for feedback (and indicate them as sub-reviewers), but their input should be integrated into the PC member’s own review.
Empty reviews
Submitting an empty or near-empty review will void a reviewer’s eligibility to review the paper in question, requiring the assignment of a different reviewer to the paper. We ask reviewers to be extremely careful not to submit an empty review.
Ethical issues
Any evidence of professional misconduct related to a submission or its evaluation (plagiarism, concurrent submissions, conflict of interest, etc.) should be reported directly to the PC co-chairs.