The Meta-Review Process for Paper Submissions

SIGCSE has been progressively improving our reviewing process in response to community requests, e.g., [1,2,3]. At the same time, the number of submissions to SIGCSE has risen over time. (Last year, there were 2 Program Chairs, 324 paper submissions, and over 1500 reviews.)

To continue the process of improvement, to provide authors with more information on how the Program Chairs use reviews to analyze the paper and how authors can use reviews constructively in case of resubmission, and to mitigate the increasing burden on the Program Committee chairs, we have established a prototype paper reviewing process for SIGCSE 2009. This page explains how the new process affects each of the constituencies.

Authors

Authors submit papers for double-blind review. The goal of the anonymized version is to, as much as possible, allow the author(s) of the paper an unbiased review. The authors choose the level of anonymity in the paper. At the lowest level, they may choose to use no anonymization; at the highest level, they may remove all references to the authors (including author's names and affiliation plus identifying information within the body of the paper such as websites or related publications). After reviewing is complete, authors will see their reviews as in previous years.

New This Year: Authors will also receive meta-reviews that synthesize the existing reviews, clarifying how the program committee interprets the disparate information from individual reviews.

Reviewers

At least four reviewers review each paper. Reviewers are drawn from the SIGCSE reviewer pool with preference for those whose expertise match the paper's topic areas.

A review must objectively, accurately, and clearly assess a paper's suitability for publication at SIGCSE, founded in the reviewer's disciplinary expertise and on the basis of the written paper's originality, technical soundness, contribution to computer science education, and clarity of presentation.

New This Year: Reviews that do not meet this standard may be deleted. For example, an unacceptable review might:

be incoherent, unreadable, or irrelevant to the paper;
focus on the paper's topic area or presumed authors at the expense of assessment of the paper itself; or
provide no justification for its numeric ratings. (Even in "obvious" cases, reviewers should briefly justify ratings.)

Reviewers whose reviews are consistently unacceptable may be permanently disbarred from SIGCSE reviewing.

Note: A difference in rating or opinion with other reviewers or with Program Committee members will never be cause for deletion of a review.

Associate Program Chairs

New This Year: One of the six Associate Program Chairs will synthesize each paper's reviews into a "meta-review" to improve accuracy, quality, and effectiveness of the selection process.

Associate Program Chairs were selected by the Program Chairs for their disciplinary expertise. Associate Program Chairs read the papers' reviews and abstracts though not, in most cases, the paper itself.

Associate Program Chairs will:

Delete, in the rare cases described above, unacceptable reviews.
Usually, summarize the case for and against accepting a paper based on its abstract, reviews, and possibly its contents.
Sometimes, summarize only the case for accepting if reviewers' ratings are consistently high, or analogously, only the case against accepting when ratings are consistently low.

In rare cases (e.g., when the reviews are deemed irreconcilably disparate or when too few reviewers claim familiarity with the area), read the paper fully and provide a "normal" review, in addition to the meta-review.
Rate the paper on the basis of existing reviews. Associate Program Chairs are charged with making strong recommendations; thus, their ratings will often be 1s or 6s. They will briefly justify the rating in the meta-review.

Program Chairs

The Program Chairs' duty, quoted from SIGCSE's Program Chair Responsibility Policy, is "to create a good, well balanced program at the SIGCSE Technical Symposium and the ITiCSE conference. The program should provide both short and long term benefits to the conference attendees and the wider community."

Within the framework described here, that means ensuring overall quality in the reviewing process and assembling an excellent program that reflects reviewers' and Associate Program Chairs' decisions with respect to the overall composition, exigencies, and special focus of the conference.

Program Chairs make all final decisions on accepting or rejecting papers on the basis of the papers, their reviews, their meta-reviews (new this year), and a global effort to compose an excellent program.

A Program Chair will read every meta-review of every submitted paper. In many cases, Program Chairs will also consult abstracts, reviews, or the papers themselves.

Finally, the Program Chairs will report to the community --- in the proceedings or at the business meeting --- on the aggregate outcome of the process and its potential for revision or extension in coming years.

References

Program Chair Responsibility Policy, http://sigcse.org/conferences/ProgramChair.shtml.
Dan Joyce, Bruce Klein, Rene McCauley. Report on improving SIGCSE/ITiCSE program creation process, SIGCSE Bulletin, 39(4) (December 2007), p. 128-129.
Amruth Kumar, Patricia A. Joseph, Paul Wagner, and Michael Goldweber. It is Time to Review the SIGCSE Reviewing Process. In Proceedings of the Thirty-Eighth SIGCSE Technical Symposium on Computer Science Education, page 591, Covington, Kentucky, USA, March 2007.

Questions? Please contact:

Steve Wolfman and Gary Lewandowski
SIGCSE 2009 Program Chairs

sigcse09-programchairs@cs.xu.edu