Understanding Bias

Response, Nonresponse and Undercoverage

Response Bias

  • Occurs in a study when a participant provides a response that does not reflect their true thoughts, feelings, or behavior.

  • Respondent bias is comprised of any error in a study that is a result of participants' inability or unwillingness to provide accurate or honest answers to a survey.

  • Anything that influences responses.

Examples

Leading questions

  • Wording of the question or the way that questions are asked

Social desirability

  • Answering what appears to be the "morally right" answer
    • e.g. woman surveys a man on his attitudes to domestic violence

Ways to avoid Response bias



  • Write questions that are clear, precise, and relatively short
  • Do not use "loaded" or "leading" question ("Don't you agree ...")
  • Avoid double-barreled questions (questions that combine more than one issue)
  • Avoid double negatives
  • Etc

Nonresponse bias



  • Individuals chosen for the sample are unwilling or unable to participate in the survey.

  • In other words, occurs when some respondents included in the sample do not respond.

  • It occurs after the sampling step of survey (don't confuse with failure to cover target population by sampling frame)

Examples of Nonresponse bias


Asking for sensitive information

Survey measuring tax payment compliance. Citizens who do not properly follow tax laws will be the most uncomfortable filling out this survey and be more likely to refuse.

Invitation Issues

A large portion of young adults and business sector workers answer the majority of their emails through their smartphones. If the survey invite is provided through an email that doesn't render well on mobile devices, response rates in smartphone users will drop dramatically.

Ways to avoid Nonresponse bias



  • Avoid Rushed or Short Data Collection Periods.
  • Send Reminders to Potential Respondents
  • Ensure Confidentiality
  • Use Incentives

Undercoverage bias



  • Occurs when some members of the population are inadequately represented in the sample

  • In other words, some groups in the population are left out of the process of choosing the sample.
  • Proportion of a segment of the population is lower in a sample than it is in the population.
  • Notice that the response rate can be high but it could still have nonresponse bias.
  • Often a problem with convenience samples

Example of undercoverage bias



A commonly-cited example of undercoverage is the poll taken by the Literary Digest in 1936 that indicated that Landon would win an election against Roosevelt by a large margin when, in fact, it was Roosevelt who won by a large margin. A common explanation is that poorer people were undercovered because they were less likely to have telephones and that this group was more likely to support Roosevelt.

Source: http://onlinestatbook.com/2/research_design/sampling.html

Question 01

Question from http://stattrek.com/.

Which of the following statements are true?

I. Random sampling is a good way to reduce response bias. II. To guard against bias from undercoverage, use a convenience sample. III. To guard against nonresponse bias, use a mail-in survey.

  1. I only.
  2. II only.
  3. III only.
  4. None of the above.

Go back to the slides and try to remember what the bias means and how you could avoid them. Does the options presented make sense?

Random sampling provides strong protection against bias from undercoverage bias and voluntary response bias; but it is not effective against response bias. A convenience sample does not protect against undercoverage bias; in fact, it sometimes causes undercoverage bias. And finally, using a mail-in survey does not prevent nonresponse bias. In fact, mail-in surveys are quite vulnerable to nonresponse bias.

Source: http://stattrek.com/

Question 02

All of the following are examples of a sample that suffers from an undercoverage bias except:

  1. An estimate of an election outcome based on a sample of voters' names taken from automobile registrations.
  2. A poll on Missouri dentist's attitudes, based on questionnaires sent to a registry of attendees at the largest dental convention in the state.
  3. A random sample of 5,000 U.S. adults with survey data collected via e-mail.
  4. A simple random sample of licensed drivers, taken from driver-licensing records.

In which of them does the sample frame is equal to the population?

A simple random sample of licensed drivers, taken from driver-licensing records. In this example, the sampling frame covers the entire population: licensed drivers.

Question 03

A telephone research agency has been contracted to collect survey information on buying habits of teenagers, aged 16 and 17. The agency conducts phone calls between 3 P.M. and 4:30 P.M. each weekday. The survey finds that the majority or teenagers aged 16 and 17 have access to a credit card. Which of the following is the most likely problem with this analysis?

  1. This is a voluntary response sample, and only affluent youths will answer the survey.

  2. The respondents were not randomly selected.

  3. The survey questions are leading and influence the way affluent youths respond.

  4. The sample suffers from undercoverage due to the hours when polling took place.

  5. The respondents were being untruthful.

Are the students voluntaring themselves to participate in the survey? Did the questions explained how they selected the phones to call? What about the question asked?

The sample suffers from undercoverage. Many teenagers aren't at home during those hours, and many of those who are at home are likely to share traits that could bias the survey results.