Student Report on IMC 2018 Shadow PC

The 2018 IMC Shadow PC has been the second incarnation of IMC Shadow PCs. In contrast to the previous and first 2017 iteration, this year two Shadow PCs were formed. One Shadow PC had an in-person PC meeting, whereas the other Shadow PC held their PC meeting via a conference call. I participated in the in-person meeting. There also is a CCR report by the Shadow PC organisers comparing both approaches.

Preparation

After having successfully applied for the Shadow PC, all Shadow reviewers were asked to do some read-up on review ethics and to explicitly confirm to review ethically. This especially entailed to not review papers from conflicted authors and not to share any information on submitted papers outside the PC. The Shadow PC took a rather broad stance here, any paper that could potentially constitute a conflict, should not be reviewed by the conflicted reviewer. Reviewers were also encouraged to read “How to Read A Paper” by S. Keshav, “You Must Be Joking… Is Change” by M. Faloutsos and “Ethics of Peer Review: A Guide for Manuscript Reviewers” by S. Rockwell on how to read papers properly and how to review constructively and ethically.

1st Review Round

After having gone through these bootstrapping steps, we were given access to submitted papers on a dedicated HotCRP instance. The first step was to assign reviewers to papers. As a prerequisite, reviewers were asked to check submissions for possible author conflicts. IMC 2018 was a single-blind submission, reviewers were thus able to check for conflicts. In a double-blind submission setting, this will be different. After having declared and cleared conflicts, reviewers were able to bid on papers. Bidding followed a two-step approach. HotCRP calculated an automated bidding score, based on a papers topics and the topic interests declared by the reviewers. But we were also able to override bidding scores manually, to put higher scores on papers we were keen to review, and low scores on papers we did not want to review too much. I tried to use this Shadow PC as an opportunity to widen my horizon. In this phase, we could already see the abstract of the papers as well as the full papers (given there were no conflicts). I did bid on papers that were close to my own research, because I felt comfortable to review them. But I did also bid on papers that I found interesting but that were out of my core scope of research. I felt being (voluntarily) obliged to review such papers would be an excellent opportunity to expand my knowledge beyond my immediate comfort zone. After the bidding phase, I was assigned seven papers for which I had roughly two weeks time to provide a review. For the first two reviews phases, we were only able to see other reviews after we had submitted our own review for a paper.

The Shadow PC was only formed after the paper submission deadline for the main conference, which implied that the schedule for the Shadow PC was a bit tighter than for the real PC. The Shadow PC chairs thus decided based on the round one reviews which papers proceeded to round two. I would assume that in a real PC there is a bit more discussion between reviewers which papers should proceed to round two. In this Shadow PC case, the chairs seem to have done promotions to round two mostly based on the overall review merit. For this specific case this means that papers for which not all reviews were submitted on time or some reviewers had a very negative opinion did not move on. I would assume that for a real PC if only some reviewers have a negative opinion while others have a positive, they will discus first before a decision is made. Similarly in a case where reviews are missing I assume a real PC has more flexibility to wait or request further reviews. However, since the Shadow PC’s main purpose is to educate future reviewers (and not to provide comprehensive reviews for all submissions), I think this decision by the Shadow PC chairs was well justified. And in their defence, all reviewers were explicitly given the opportunity to ask for further papers to proceed to the second round.

2nd Review Round

The second round in general followed the structure of the first round. I was given about ten days of time to review another three papers. In contrast to the first round, there was an explicit discussion phase after the round two reviews. The aim of the discussion round in general was (if possible) to converge and align different reviews. This does not imply for all reviewers to agree on the same final verdict, but to discuss and align factual differences. This entailed discussing different understandings of papers, for example as to whether a methodology or argument was sound or flawed. The idea was that all reviewers can develop a crisp understanding of the technical details, possibilities but also limitations of a paper, which would later on allow them to make an informed decision regarding acceptance or rejection. This constituted an excellent opportunity to also learn from more experienced reviewers in the specific topic of a paper. In my experience all discussions I had were open and friendly, (Shadow) reviewers were honestly interested in understanding the topic better without a personal judgement of being right or wrong. This discussion took place online via HotCRP (almost) until the in-person Shadow PC meeting. However, in cases were no consensus could be reached, decisions and discussion were deferred to the in-person meeting. Based on the review and discussion outcomes, the Shadow PC chairs decided which papers should proceed into the actual Shadow PC meeting.

In-Person Shadow PC Meeting

The Shadow PC meeting took place on 23 July 2018 at FU Berlin and was kindly organised by Matthias Wählisch and his group. We had a room for the full day with tables laid out in a U-shape, making discussions really easy and pleasant. We had a longer break for lunch and a few shorter coffee and tea breaks during the day (If I remember correctly, it is now almost three month after the meeting that I am putting down these words.) The chairs starting the meeting with outlining the agenda and procedure for the day and some general stats and remarks. In contrast to the previous review rounds, the chairs now had opened all papers and all reviews for everyone, so that everybody could see all reviews to be easier able to participate in discussions. However, this did not extend to conflicted papers. The Shadow PC kept paper confidentiality against conflicts by sending conflicted reviewers out of the room for the duration of the discussion of a paper conflict.

After that we started with a calibration phase of papers that received mostly positive or mostly negative reviews, to allow all reviewers to have an idea what the expected range of papers would be. In general, for each paper a dedicated reviewer was tasked to provide a brief summary of the paper and the main points of all reviews, also pointing out whether reviewers mostly agreed or on which points their opinions differed. After this, other reviewers of the papers were asked to amend the summary were they felt need for it, before the paper went into the real discussion phase. In this phase, it was not only the reviewers of the paper but the whole Shadow PC that took part in the discussion. Of course, for questions about specific details of the paper only those who had read it could answer. But when it came to judgements of for example merit, impact and perspective, also in relation to other works, quite often other PC members chimed in (basing their judgement on the summaries of the reviewers who had read the actual paper). After the discussion, one of three possible outcomes for each paper was decided on. A paper could be accepted, rejected, or put aside to be revisited later on. Some of these were preliminarily tagged as “accept-if-room”. Interestingly, the length and intensity of discussion did not necessarily correlate with the outcome. Some papers were accepted or rejected without much discussion, whereas for some (but only a few), the full PC voted after the discussion phase. Over the course of the day, we so decided on the (hypothetical fate) of 28 papers. We were lucky that the other (remote) Shadow PC had their meeting on the previous day, so we could conclude our PC meeting by comparing our outcomes for some papers with their outcomes and discussion phase. After the meeting, those who did not go home the same day went out for a small IMC Shadow PC social. (I should mention that there also was a social on the previous evening, but I did not arrive on time to make it, so no experience report on this one from me.)

My personal opinion

Prior to this IMC 2018 Shadow PC meeting, I did also participate in the 2017 IMC Shadow PC as well as in EuroSys and TMA Shadow PCs. I enjoyed all Shadow PCs and found them very valuable. I enjoyed forming my own opinion on papers, and then being able to contrast and discuss this opinion with other reviewers, quite often leading to revelations on how differently a topic can be perceived, and also how much your own opinion might change after being challenged by others.

In general, there are two different parts which I think make a Shadow PC participation valuable. For one, being forced to read a paper to provide a review, stimulated me to read significantly more carefully, paying more attention to detail and also making sure to extract enough facts from it to justify having an opinion on it. I found the change of role from a PhD student paper writer and submitter to reviewer quite revealing. During the reviews I have done, I really learned to appreciate clearly and explicitly written papers, that make an effort to make sure every reader can follow the inner logic of it.

But even more than writing reviews, I enjoyed being given the opportunity to discuss my findings with the others reviewers. I found it really interesting and sometimes eye-opening how differently the same assortment of words can be perceived and how careful you sometimes have to be to precisely express your opinion and point of view. After all, I really enjoyed the open and honest discussion I had in my Shadow PC participations. I believe that these experiences in the future will allow me to express my opinions more nuanced, cohesive and less ambiguous. Besides, an in-person Shadow PC is a really good opportunity to get to know fellow PhD students, that quite likely are working on topics closely related to your own research. I also learned a lot about how different the environment and processes for PhD students in different countries or even universities are. Recommendations for future participants If you (even remotely) consider to participate, I can only strongly recommend you to apply and do it. Every time I participated it was a very valuable experience for me. If you are doubting whether you are knowledgeable enough, apply anyway. On most papers I have reviewed I have not been an expert, but I have learnt a lot about measurement topics by reviewing papers on different topics and also with completely different methodologies. In my opinion a Shadow PC is not about being an expert, but about having a justified opinion and being willing to present, challenge, discuss and if necessary change it. It also is an excellent opportunity to get to know fellow PhD students working on related topics, that you will quite likely meet again at the next conference.

The only requirement I would like to very strongly emphasise is commitment. Shadow PCs often happen after the real conference paper submission, yet ideally their results should come in no later than the real PC. They thus often operate on a limited timescale. In addition, doing the first reviews properly takes time, to carefully read a paper, reflect about the content and if necessary read additional material, especially if the topic is not within your core expertise. If you apply for a Shadow PC, please be fully committed and budget enough time to provide high quality reviews in due time. A Shadow PC is not only an opportunity for sharpening your individual profile, a Shadow PC predominantly is a group experience. Overall high group commitment will make the experience way more valuable for every participant. There simply is not much to discuss if only one or two reviewers have provided thorough reviews

I would like to use this opportunity to thank Matthias Wählisch, Gareth Tyson and Aruna Balasubramanian for organising this very educational experience and I would also like to thank Marcin Nawrocki for proofreading this text and his very valuable suggestions.