This presentation summarizes an article from our research discussion assignment #4:

Sanjay Krishnan, Jay Patel, Michael J. Franklin, Ken Goldberg, “Social Influence Bias in Recommender Systems: A Methodology for Learning, Analyzing, and Mitigating Bias in Ratings”

The article addresses:

• Social influence bias in user-item ratings
• Measuring and quantifying the effect
• Developing a model to mitigate (i.e., reverse) the effect
• Illustrating with the California Report Card recommender system

• What is it?

  • “The tendency for individuals to conform to the ‘norm’ in a community”

  • Example: a buyer’s rating of a product on Amazon (or a movie on Netflix) may be influenced by the average rating displayed

• Why does it matter?

  • User ratings may be biased, i.e., ratings may not accurately reflect the user’s true opinion

  • User ratings may be closer to the mean / median of all ratings and less diverse than the community’s range of evaluation

• Online survey relating to California’s performance on various political issues. E.g., “Please grade the State of California on Implementation of the Affordable Care Act”

• Participants are asked to assign letter grades (A+ through F, a range of 13 ratings) on each issue

• After the initial ratings are entered, participants are shown the median ratings (from all users up to that point) for each issue, and are allowed to change their ratings

• Data was collected from 1/18/2014 to 4/20/2014, including 1,575 participants and 9,390 ratings

• Over the observation period:

  • 556 out of 1,575 participants (35%) changed one or more ratings after seeing the median ratings
  • 862 out of 9,390 ratings (9%) were adjusted

• In aggregate, ratings that were changed were 19.3% closer to the median value (lower dispersion), on average, than ratings that were not changed

• Ratings that were changed would have to be adjusted by 2/3 of a letter grade, on average across all issues, in order for the variability about the median to be equivalent to that for the ratings that were not changed

• Correction model to estimate the initial (true) ratings as a function of the final ratings and the median ratings

  ==> this model can be used to correct (i.e., remove bias from) the final ratings and infer the initial ratings

• Prediction model to estimate the final (potentially adjusted) ratings as a function of the initial ratings and the median ratings

  ==> this model can be used to study the social bias effect, e.g., drivers of the effect, potential asymmetry of initial ratings below / above the median

• Social influence bias can be substantial when users have access to ratings from other users (individually or in aggregate)

• This bias may weaken the effectiveness of collaborative filtering techniques that rely on similarity measures of user-item ratings

• The study suggests that the bias effect can be quantified and mitigated using machine learning methods