3.1 Peer Review Algorithm

The peer-review process is divided into mandatory and discretionary reviews, which are queued differently in the system.

This process aims to encourage automatic scoring based on submission quality. Should a submission be redirected for Moderation, the following triggers are built into the system to notify discrepancies and/or suspicious behaviour:

• Inconsistent scoring after ten reviews
• Submission not reviewed in the peer review process for an extended period.

The following improvements were made to encourage good and fair behaviour from teachers and learners.

• Notifications warning users of potential bad behaviour i.e, submitting answers too quickly, taking less than 60sec to review submissions.
• Introduction of remark requests, credibility weighting, and patterned behaviour.

Unlike previous challenges, the teacher’s score held equal weight to the learner’s score and thus did not override previous learner scores within the system.

• Types of inappropriate submissions:
• Plagiarism
• Irrelevant & repeated answers
• Inappropriate and explicit links included in submissions.

Regular system checks were implemented at the end of each new challenge week, with additional random checks and adjustments of scores by the systems managers.

A detailed exposition of the algorithm can be found here:

• https://sites.google.com/jasiri.org/wavumbuziplaybook/home/product/how-it-works-pseudocode
• https://sites.google.com/jasiri.org/wavumbuziplaybook/home/product/how-scores-finalise

4.1 1. Wild fluctuation in credibility weights

Previous analyses have shown that users’ CWs were incredibly volatile, swinging aggressively up and down over time. Red flag: CW cannot possibly be a good indicator of the degree to which we trust in someone’s reviews if it fluctuates so wildly for most users.

So, it was concluded that volatility in CW is undesirable, and we ought to figure out how to configure CW such that:

• It converges to some semblance of true credibility for each user;
• Provides a stable sense of the trust to assign to a reviewer - even if evolving.

4.2 2. Credibility weight only going down and not up

During Kenya 2021 Challenge (August), it was raised that the CW was mainly going down and not up for most users.

It is, however, essential to note that there are cases where the CW does not change because the student did an OK review where he gets half points increase. Nonetheless, it is always essential to investigate

4.3 3. Consistency in how CWs are updated relative to # of reviews performed

In a previous iteration, it was reported that 16% of users (~280) who made reviews have more credibility update entries than the number of reviews performed.

4.4 4. Moderation is pulling a lot of scores to 0.

In a previous iteration, it was reported that Moderation seem to pull a lot of scores to 0 as shown in the figure below

5.1 Univariate analysis

Descriptive measures under the univariate setting will be applied to summarize central tendencies and distribution for metric variables.

• Categorical variables such as gender and nationality will be summarized using frequencies and percentages.
• Categorical variables derived from continuous variables will be categorized using acceptable limits derived from literature
• Continuous variables such as age, CW, and scores will be summarized using mean and corresponding SD - assuming Gaussian distribution.
• Continuous variables that are skewed will be summarized as median and the corresponding interquartile range (IQR).

5.2 Bivariate setting

• Association between categorical variables will be assessed using Pearson’s Chi-Square test
• Association between categorical and continuous variables will be done using a two-sample t-test, assuming Gaussian distribution for continuous variables.
• Association between categorical and continuous variables which are not normally distributed will be conducted using a two-sample Wilcoxon rank-sum test.
• Kruskal Wallis test (one-way ANOVA on ranks) will be used if the levels of the categorical variable are greater than 2.

5.3 Regression Modeling

Where applicable:

• For continuous response, => Linear regression model will be fitted. Effect modifiers and potential confounders will be adjusted. Regression coefficients and the corresponding 95% confidence interval will be reported.
• For binary response => association between multiple risk factors and the response, the variable will be conducted by fitting multiple logistic regression models. Effect modifiers and potential confounders will be adjusted. Odds ratios and the corresponding 95% confidence interval will be reported.

6.1 Submission

Variable type: logical

Variable type: numeric

Variable type: POSIXct

6.2 Credibility Weighting History

Variable type: character

Variable type: logical

Variable type: numeric

Variable type: POSIXct

6.3 Submission Score History

Variable type: character

Variable type: logical

Variable type: numeric

Variable type: POSIXct

6.4 Users

Trancated due to large table

Variable type: character

Variable type: logical

Variable type: numeric

7.1 Study Cohort

7.2 Credibility Analysis

Remember: credibility is the extent to which we trust each reviewer’s suggestions

7.2.2 CW Volatility assessment

Comparing this with changes in CW from the previous iteration (depicted in Fig.4 below):

• There is more trust in how the credibility scores are updated
• The updates do not fluctuate so wildly for most users, as seen in the previous report
• For most users, CW updates are smoother and predictable compared to previous iterations
• Volatility in CW (which is an undesirable characteristic) has significantly reduced

Questions to investigate

• Why do we see a significant change in CW only during the first 15 days and the last 20 days of the challenge?

7.3 Consistency Analysis

the extent to which they agree with one another [Consistency]