www-apachecon-discuss mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Eckart de Castilho <...@apache.org>
Subject Re: Some feedback on the new review mechanism
Date Thu, 29 Sep 2016 07:43:11 GMT
I second Jan's and Chris's opinions. 

This type of crowd-sourcing approach IMHO requires a huge number of reviewers to be successful
and it would also require individual reviewers to get less of a load and more feeling of doing
something useful. Even reducing the pairs per session to like 10 or 25 would probably help.
People could do it quickly during lunch break or on the bus home, etc.

Also a button like "I don't want to see this submission never ever again!" would be nice -
for it to be excluded from future pairs.


-- Richard

> On 29.09.2016, at 09:13, Jan Willem Janssen <janwillem.janssen@luminis.eu> wrote:
>> On 29 Sep 2016, at 08:48, Christofer Dutz <christofer.dutz@c-ware.de> wrote:
>> I just wanted to also take the opportunity to give some feedback on the modified
review process:
>> 1. Seeing that I would have to do 30000 decisions sort of turned me off right away
(It's sort of ... "yeah let me help" and then getting a huge pile of work dumped on my desk)
>> 2. With that huge amount of possible work I could see only little progress for quite
some time put into it) ... 30000 decisions would require reading of 60000 applications. If
I assume 30 seconds per application that's about 500 hours which is about 20 days without
doing anything else. I sort of quit at about 400 decisions.
>> 3. I noticed for myself that at first you start reading the applications carefully
but that accuracy goes down very fast as soon as you get a lot of the talks you reviewed earlier
... unfortunately even if you only think you read it before. I noticed me not reading some
similar looking applications and voting for one thinking it's the other. Don't know if this
is desirable.
>> I liked the simple interface however. So how about dropping the Deathmatch approach
and just displaying one application, and let the user select how much he likes it (ok ...
this is just the way the old version worked, but as I said, I liked the UI ... just clicking
once) ... eventually the user could also add tags to the application and suggest tracks.
> I share this as well: given the large amount of proposals, the decision making for all
permutations is
> simply too much. Also, due to the relatively small amount of reviewers and the “real”
randomisation, I
> think there’s a large bias in the final decisions: I’ve come across the same “match”
several times,
> which implies that the one talk that lost the battle has a negative bias.
> I’ve tried to do as many “battles” as possible, but got only up to about 1000 before
I was fed up and
> no longer could spend time on it due to other obligations. I’m not sure if I’ve seen
all proposals
> (probably not), which is a pity, in my opinion...

View raw message