The goal of the study

Distribution of worker skill level across questions has the potential to bias the outcomes of the larger crowdsourcing task. The reason is that software faults can be overlooked if they are covered by questions* that were mostly answered by lower skill workers.

*Each question covers a certain number of source code lines. Questions have the following format:“Do you believe the source code between lines 35 and 45 is related to the described failure?”

Therefore, my goal is to investigate how worker skill level is distributed across questions. Worker skill level was measured by the following three attributes.

  • Test score: workers received a score (zero to five) after performing a computer programming test used to filter out workers without enough programming knowledge. Workers who scored 2 or less were not allowed to participate.
  • Years of programming experience: Worker quality can also be measured by years of programming experience, which is an information self-declared by workers befote they take any test.

  • Profession: worker profession is also an indicator of quality. Professions consisted of professional programmers, hobbyists, graduate students, undergraduate students, and others.

    • Focus of the current analysis

    The current analysis focuses only on the worker profession.

    Actual analysis

    Charts show that workers are not equally distributed across questions, for instance, up to 10 questions received 7 out of 20 answers from undergraduate students (Fig.4). Workers seem rather normally distributed. Since undergraduates have the lowest level of answer accuracy, these questions might have been overlooked.

    While filtering workers by profession, I will have to bear this bias in mind. One way to cope with it is to run a Monte Carlo simulation on top of filtering. This way, I can minimize the effect of the existing bias on the population.