In a nutshell

Jigme Wangdi is our highest-ranked applicant for the 2019 ITF award.

Raw results

Since we had three reviewers for each applicant, the simplest way to rank applicants is to add up their scores across the 1-5 scales used in the review process. Jigme Wangdi ranks the highest.

Applicants ranked by summed reviewer responses. These data simply assign Likert-type responses values of 1 - 5 based on strength of disagreement/agreement.

Applicants ranked by summed reviewer responses. These data simply assign Likert-type responses values of 1 - 5 based on strength of disagreement/agreement.

Response models

Anticipating unequal reviews among applicants, I repurposed a model I developed to calculate an Agreement Index for Likert-style data based on simulated multinomial effect sizes. Summed model scores also ranked Jigme Wangdi as the highest applicant. Natnael Demelash is excluded from these analyses; they were the only applicant for whom a reviewer determined the application should not be reviewed and this broke my code.

Summarized modeled results

Summarized results of response models for each applicant. Individual scores, with confidence intervals, for each question by each applicant displayed below.

Summarized results of response models for each applicant. Individual scores, with confidence intervals, for each question by each applicant displayed below.

Individual modeled results

Note that not only does Jigme Wangdi have the highest ranking, but reviewer agreement was most consistent as depicted by narrow ranges of 95% confidence intervals. I’ve considered sending individual results to each applicant to given them feedback on improvement should they want to reapply in the future.

Results of response models for individual applicants, organized by question category (colors). Points represent mean weighted effect sizes of simulated multinomial distributions, bars depict 95% confidence intervals from 1000 simulations per question per applicant. Likert data in these models assign negative values to disagree responses and positive values to agree responses, with ambivalence scoring zero.

Results of response models for individual applicants, organized by question category (colors). Points represent mean weighted effect sizes of simulated multinomial distributions, bars depict 95% confidence intervals from 1000 simulations per question per applicant. Likert data in these models assign negative values to disagree responses and positive values to agree responses, with ambivalence scoring zero.