22 Mar, 2014

Whether you are a novice Python programmer, or a guru with several solid projects under your belt the EuroPython conference is the event you want to attend. This year I decided to submit my talk proposal, and as other proposal authors got invited to participate in peer review of submitted talks. As the first review round is complete, I wanted to share my experience with you (+ some stats on currently accepted/rejected proposals).

Couple of months ago, I decided that it is a good idea to participate in this year’s EuroPython not only as an attendee, but as a speaker as well. So, I submitted my proposal — and then got an email from Martin v. Löwis, which he sent to all proposal authors. There Prof. Löwis invited us to participate in talk selection process as reviewers.

Now, let me elaborate a bit on selection criteria and how we as reviewers could influence which talk gets selected and which doesn’t. Basically, we were to read proposal description and write a short review, along with a numeric grade:

• -1 := strong reject
• -0 := weak reject
• +0 := weak accept
• +1 := strong accept

Then guaranteed acceptance and rejection rules are simple:

• Papers with at least one +1 vote and no - votes are in
• Papers with at least one -1 vote and no + votes are out

Obviously, it is getting more interesting once we get to the cases which have both + and - grades (of various intensity).

As suggested by Prof. Löwis during this round we should achieve following goals:

• Make sure that each proposal gets at least 3 reviews
• Once the first review phase is over we will have number of proposals that are definitely accepted/rejected
• Depending on number of accepted proposals, we will define further procedure to decide on those mixed cases

This Wednesday we have completed the first review iteration, and here is some interesting statistics:

 Total # of proposals 297 Max. # of reviews per proposal 20 Min. # of reviews per proposal 3 Total # of definitely accepted proposals 86 Total # of definitely rejected proposals 17 The best review grade 8 +1s The worst review grade 19 -1s and 1 -0 Total # of proposal I reviewed 17 Review grades I assigned 4 -1s, 1 -0, 5 +0s, 7 +1s

Now, after we identified 86 accepted and 17 rejected proposals, the following categorization was proposed for the next round:

Category Number of Cases
Likely accept (no -1, some -0, some +0, some +1) 45
Likely reject (some -1, some -0, some +0, no +1) 22
No strong opinion (no -1, some -0, some +0, no +1) 4
Mixed Cases, +1 and -1 (some -1, some -0, some +0, some +1) 123

So, since we do not have enough accepted talks yet, we need another round of reviews. During this second round, all those reviewers who provided feedback for a talk that is not definitely accepted/rejected, will get another chance to assess the validity of their original review and change their numeric grade if necessary — either into a strong acceptance or into a strong rejection. This should make it easier to decide whether talk is to be rejected/accepted, plus in case of Likely Accept category (no -1s, some -0s, some +0s, some +1s), we can end up with even more definitely accepted talks too.

It will be actually quite interesting to see what comes out of this second round, and how many more talks get accepted. Next week we should have the round finished. If there still will be space for more talks i.e. we still will not have enough definitely accepted talks, then EuroPython team will manually review the proposals.

So far, I think that the review process is well-thought and therefore is really very helpful. Hopefully, it reduces amount of manual work required even further after the second iteration. Should Prof. Löwis share more stats, I will write another post on the final outcome.

Now, you are probably interested how well my own proposal did.. It turned out quite well for me, as my proposal got 5 reviews, all of which were +1. So, while it is not yet officially announced, it seems I made it into the list of speakers for the EuroPython 2014! Will wait for the official announcement to celebrate though :)

If you haven’t already, go and grab your ticket, and see you all on EuroPython 2014!