Rating versus ranking: How diverse response options can reveal more about people’s beliefs and opinions

Introduction

Likert scales are often used for studying attitudes and beliefs. However, participants in studies often endorse any (and all) statements aligning with their attitudes, regardless of the relative importance of these statements (Holford et al., 2023). For instance, people with negative vaccine attitudes (NVAs) will usually endorse statements that vaccines are unsafe, ineffective, and unnecessary. This has led some to conclude that people with NVAs have a “monological belief system” (Holford et al., 2023). In other words, people with NVAs may hold a set of beliefs that reinforce each other instead of resting on unique sets of evidence.

This “monological belief system” interpretation would be consistent with the “attitude roots” perspective (Hornsey et al., 2018). In this framework, people hold beliefs not based on evidence per se, but ultimately based on various phobias and biases. According to the “attitude roots” framework, people with NVAs may ultimately dislike being told what to do or have a phobia of needles. They rationalize these responses with post hoc evidence. Thus, when they respond to questions on a survey, they endorse reasons that align with their rationalizations rather than “admit” they are ultimately they are just resistant to directives or anxious about needles.

It is also possible, however, that using Likert ratings, as a method, lacks the ability to measure subtle differences in attitdues, beliefs, and priorities. In other words, perhaps people can exhibit differences in beliefs about vaccine safety, efficacy, and neccessity, if they are “asked” a certain way. In the following two batches of data, we use Likert ratings and ranking data on the same items (e.g., “vaccines are unsafe”, “vaccines are ineffective”) to determine whether this is a possibility.

Research questions

This project seeks to address the following research questions:

1. Does ranking data help increase measurement acuity (relative to Likert items alone) for vaccine attitudes?
2. Are there sociodemographic variables linked with distinct priorities in vaccine attitude topics?
3. How well will the attitude roots framework hold in light of the present data?

Study 1

Method

Only people with NVAs were recruited for these surveys. The first item on the survey read, “This survey is designed to understand people who have some degree of vaccine hesitancy. This could range between “having concerns” to “outright refusal” and everything in between. In other words, we want to hear from people that have some degree of negative attitudes towards at least one type of vaccine. If you do not fit this description, please do not complete this survey. PLEASE ONLY CONTINUE WITH THIS SURVEY IF YOU FIT THE DESCRIPTIONS ABOVE.”

242 participants recruited from Prolific (\(M_{age}\) = 34.96, 56.33% male, 77.85% white They were given 7 statements potentially justifying NVAs (see Table below). Participants rated their agreement with these statements and ranked them. On a 5-point scale ranging from 1 = “strongly disagree” and 5 = “Strongly agree”.

Participants were also allowed to “veto” statements, removing them from the list of items to be rated or ranked. They were also allowed to create their own statements, just in case the list of items provided was not sufficiently exhaustive in their case.

Results

Reasons kept

Since participants were allowed to “veto” reasons. It is useful to examine which reasons were kept (i.e., not vetoed by participants). The table below shows each of the seven statements alongside the percentage of participants who kept these statements in their set for rating and ranking.

The table above helps elucidate the relative importance of some reasons over others. More importantly, we need to examine the extent to which individuals’ Likert ratings reflect these patterns. In other words, we need to determine whether these “veto” behaviors lend any new information the Likert items themselves did not. (I’m betting they would have at least rated a reason lower if they also vetoed it from their ranking set.)

How useful are the ranking data compared to Likert data alone?

The figure below shows the number of unique ratings for each participant. In other words, if a participant gave a 5 (“strongly agree”) to all 7 NVA reasons, they would have only 1 unique rating. If at least one of their Likert ratings were different from the rest, then they had 2 unique ratings, and so on.

In 45% of cases, Likert data would have been insufficient to differentiate the relative importance statements because they gave the same rating to all 7 items. There were also a substantial number of cases where they only gave 2 unique ratings. Overall, these results suggest that rankings have the potential to serve as “tie breakers”.

Highest and lowest ranked items.

The most consistently high-ranked NVA justifications were “distrust” and “not enough known”. The lowest ranked justifications were “pain” and “religious / moral beliefs”.

Conservatives retained the most diverse set of reasons. Liberals tended to rank pain from injection higher than others.

Novel reasons

Participants were allowed to supply their own reasons, if they felt the provided reasons were not exhaustive enough. However, as is evident in the following list, not every participant understood these directions. Participants did not always understand the scope of certain reasons provided. For instance, we intended for “There are too many risks associated with vaccines” to encompass “side effects”. We also observed there were reasons supplied by participants that our provided items were not exhausted enough. For instance, there were many practical concerns (e.g., time, expenses, insurance) that were not included in the provided reasons. It was also evident that people assumed we were referring to the COVID-19 vaccines or clearly exhibited beliefs specific to those vaccines versus older ones.

• “I have a phobia of needles”
• “parents were anti-vax and the things they told me made me afraid”
• “side effects”
• “I am concerned about having an allergic reaction to the ingredients”
• “Bad reaction to vaccine”
• “Some of the”vaccines” are not vaccines.”
• “I’m allergic to an ingredient in the vaccine”
• “The COVID vaccine is different in type and was rushed through research”
• “The new MRNA vaccines have questionable efficacy and safety.”
• “Too many vaccines are given together, especially to children, and immunity response may be problematic.”
• “No long-term effects are known about current vaccines”
• “Vaccine causing me to feel sick after getting it, even if for a short time.”
• “I just don’t want to take it.”
• “Doctor recommended I DON’T get the vaccine”
• “Potential reproductive harm”
• “I’m afraid of any side effects from vaccines”
• “People lie about whether vaccines are effective or not.”
• “I feel upset that people were forced to take the Covid vaccine.”
• “The seemingly ineffective nature of the COVID vaccines increased my concerns about vaccines.”
• “Pharmaceutical companies seem to want to make fast dollar rather than make sure their vaccines are safe and effective.”
• “problems later in life from an untested vaccine.”
• “concered about side effects both short and long term”
• “It might make me immune to vaccines and will eventually not work anymore”
• “some vaccines are not as effective as claimed”
• “Vaccines are designed so you have to continually get boosters.”
• “Vaccines are only temporary.”
• “side effects”
• “recovery time”
• “They are not safe and not effective. Mother Nature’s immune system does a much better job”
• “I have gotten sick from each vaccine I have ever had.”
• “I still got infected, after the vaccine, and it was just as bad as if I didn’t receive the vaccine.”
• “The newest vaccines have not existed long enough to know their true efficacy.”
• “Not a large enough of a sample size was tested before approval.”
• “They could possible cause Autism”
• “I choose to live an all natural darwanistic life, vaccines do not fall into that philsophy.”
• “afraid of side effects”
• “afraid of long term changes to dna”
• “I have gotten the illness after I recieved the vaccine.”
• “I do not like to be forced to take a vaccine to be allowed to go into work.”
• “The cost of the vaccine.”
• “Time needed to go get vaccine.”
• “Concerned my medical insurance does not cover cost of vaccine”
• “I do not trust the safety of the mRNA technology.”
• “The medical community has done a lot of damage to their credibility.”
• “I have heard of stories of people having bad side effects from the vaccines.”
• “The vaccine technology is too new.”
• “I am fairly healthy, for diseases that are not that bad, like the Flu or Covid, I don’t mind just getting sick instead”
• “Certain vaccines are not tested well enough for long-term effects before being distributed.”
• “There may be side effects such as autism”
• “I’m not in an at-risk group that needs the vaccine.”
• “I’m not a frontline worker.”
• “There are aborted fetuses in the vaccines.”
• “Many people have died mysteriously from vaccines like COVID for example.”

We also noticed some trends that mirror our (anecdotal) conversations with relatives. For instance, we have found that people’s doctors told them not to receive a COVID-19 vaccine if they are healthy, not in an at-risk group, etc. This wouldn’t be the first time individual physicians give advice that is contrary to established guidelines. In the next study, we included questions asking about their personal physician’s recommendations as well as their beliefs about the necessity of vaccines when they view themselves as healthy, not in an at-risk group, etc. There were quite a few additional reasons people might have for vaccine hesitancy found in the free response data that we had not considered. However, not all of these would be able to fit into a short set of mutually exclusive, mutually exhaustive items. Thus, some of them were included in a separate section of Likert items only.

Study 2

Based on the results of Study 1, we believed it was important to specify which vaccine(s) we were referring to and to revise the items being rated and ranked. Specifically, we sought to clarify the scope of each item. For instance, we wanted to make it clearer that “risks” included “side effects”. We also added a large number of additional questions based on participants open response data from the previous study

Method

226 participants recruited from Prolific (\(M_{age}\) = 38.28, 58.41% male, 64.60% white. Participants were shown a revised list of 6 statements justifying NVAs (see Table below). They gave Likert ratings of agreement and later ranked them in terms of relative importance. Based on open-ended questions from Study 1, we repeated the same procedure separately for COVID-19 and influenza vaccines.

Results

Participants showed far more variability in their Likert ratings (see Figure below).

We performed a Multidimensional Preference Analysis (MPA). Like traditional multidimensional scaling analyses, MPAs create an N-dimensional space where individual rankings and the items being ranked can be plotted. This helps researchers learn how similar or dissimilar different people are (or the things they are ranking).

Factor analyses

We performed a series of exploratory factor analyses (EFAs) on the Likert data. We sought to determine whether any socio-demographic variables correlated with emergine factor structures.

COVID-19 vaccines.

For the COVID-19 items, a parallel analysis suggested a 3 factor solution (See Table below).

Factor 1 mostly reflects themes of distrust and religion/morality. Factor 2 mostly reflects perceived risks and the belief that not enough is known about the vaccines. Factor 3 mostly reflects concerns about pain as well as practical considerations.

We performed a series of exploratory regression analyses, using estimated factor scores for each participant along these three factors as dependent variables in each model.

People who were more conservative and had lower formal educational attainment were higher on the distrust/religion/morality factor. Model \(R^2 = .15\).

People who were more conservative and who had less formal educational attainment were also higher on the second factor, which reflects perceived risks and beliefs that not enough is known about the COVID-19 vaccines. Model \(R^2 = .14\).

None of the socio-demographic variables from the previous two analyses were significantly related to the third factor, which reflected concerns about pain and other practical issues. Model \(R^2 = .02\).

Flu vaccines

Unlike the COVID-19 data, a parallel analysis on the flue vaccine ratings suggested a 2 factor solution (see Table below).

The first factor reflects distrust and religious/moral reasons, like with the COVID-19 data, but also more broadly reflect risk perceptions. There are substantial cross loadings for practical considerations and perceived risks. Factor 2 appears to reflect concerns about pain but not uniquely so. Overall, the factor structure suggested by the parallel analysis for the flu vaccine data are a lot less “clean” in their structure. Nonetheless, we will refer to the two factors as “general negativity” and “pain”, respectively.

People who identified as White and those who were more conservative were higher on the “general negativity” factor. Model \(R^2 = .11\).

None of the socio-demographic variables were significantly related to the second factor, which mostly reflected concerns about pain. Model \(R^2 = .04\)

Extra Likert items

We also included a number of unique Likert items in Study 2, based on the results of Study 1.

“Natural immunity is better than vaccine-induced immunity.”

likert.labels=c("Strongly disagree","Somewhat disagree","Neither...         ","Somewhat agree","Strongly agree")
barplot(table(dat[,59]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,59]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 59] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.9823 -0.6637  0.1382  0.7978  2.4594 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    2.462170   0.387957   6.346 1.08e-09 ***
## dat$age        0.007981   0.006162   1.295   0.1965    
## dat$is.male    0.243111   0.151210   1.608   0.1092    
## dat$is.white  -0.145027   0.156467  -0.927   0.3549    
## dat$politics   0.369368   0.069172   5.340 2.14e-07 ***
## dat$education -0.024904   0.055434  -0.449   0.6537    
## dat$income    -0.093124   0.051097  -1.822   0.0696 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.135 on 242 degrees of freedom
##   (4 observations deleted due to missingness)
## Multiple R-squared:  0.1349, Adjusted R-squared:  0.1135 
## F-statistic: 6.291 on 6 and 242 DF,  p-value: 3.665e-06

“Some ‘vaccines’ are not really vaccines (i.e., some of the COVID-19 “vaccines” shouldn’t be called “vaccines”).

barplot(table(dat[,60]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,60]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 60] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.64725 -0.76593  0.09228  0.68802  2.98766 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    2.470319   0.383129   6.448 6.15e-10 ***
## dat$age        0.002249   0.006086   0.370 0.712073    
## dat$is.male    0.089397   0.149689   0.597 0.550920    
## dat$is.white  -0.230624   0.154631  -1.491 0.137152    
## dat$politics   0.459134   0.068278   6.724 1.27e-10 ***
## dat$education -0.184764   0.054894  -3.366 0.000888 ***
## dat$income     0.048548   0.050716   0.957 0.339404    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.12 on 241 degrees of freedom
##   (5 observations deleted due to missingness)
## Multiple R-squared:  0.1999, Adjusted R-squared:   0.18 
## F-statistic: 10.04 on 6 and 241 DF,  p-value: 6.769e-10

My opinions towards the COVID-19 vaccines are completely separate from other vaccines (e.g., influenza, MMR).

barplot(table(dat[,61]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,61]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 61] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.9821 -1.0865  0.3308  0.9584  2.0413 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    3.549150   0.435286   8.154 1.94e-14 ***
## dat$age        0.001399   0.006902   0.203  0.83958    
## dat$is.male   -0.014167   0.169445  -0.084  0.93344    
## dat$is.white  -0.261224   0.175486  -1.489  0.13791    
## dat$politics   0.155521   0.077934   1.996  0.04711 *  
## dat$education -0.181496   0.062510  -2.903  0.00403 ** 
## dat$income     0.112490   0.057414   1.959  0.05123 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.27 on 241 degrees of freedom
##   (5 observations deleted due to missingness)
## Multiple R-squared:  0.05672,    Adjusted R-squared:  0.03323 
## F-statistic: 2.415 on 6 and 241 DF,  p-value: 0.02761

“mRNA vaccines are unsafe”

barplot(table(dat[,62]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,62]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 62] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.1728 -1.1203 -0.2367  1.1835  2.8257 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    2.637406   0.468681   5.627 5.05e-08 ***
## dat$age       -0.004097   0.007444  -0.550  0.58258    
## dat$is.male   -0.062795   0.182673  -0.344  0.73133    
## dat$is.white  -0.306073   0.189023  -1.619  0.10670    
## dat$politics   0.007870   0.083564   0.094  0.92505    
## dat$education  0.196479   0.066968   2.934  0.00367 ** 
## dat$income    -0.122624   0.061729  -1.986  0.04811 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.371 on 242 degrees of freedom
##   (4 observations deleted due to missingness)
## Multiple R-squared:  0.05622,    Adjusted R-squared:  0.03282 
## F-statistic: 2.403 on 6 and 242 DF,  p-value: 0.02835

“mRNA-based vaccines can alter people’s DNA”

barplot(table(dat[,63]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,63]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 63] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.4724 -0.4344 -0.0715  0.5756  2.3338 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    2.511854   0.310467   8.091 2.92e-14 ***
## dat$age        0.005497   0.004932   1.115   0.2661    
## dat$is.male   -0.003699   0.121300  -0.030   0.9757    
## dat$is.white  -0.233848   0.125304  -1.866   0.0632 .  
## dat$politics   0.282125   0.055329   5.099 6.91e-07 ***
## dat$education -0.097783   0.044483  -2.198   0.0289 *  
## dat$income     0.026292   0.041098   0.640   0.5229    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.9077 on 241 degrees of freedom
##   (5 observations deleted due to missingness)
## Multiple R-squared:  0.1217, Adjusted R-squared:  0.09985 
## F-statistic: 5.567 on 6 and 241 DF,  p-value: 2.003e-05

“Some COVID-19 vaccines can cause reproductive harm (e.g., lower sperm count, lower fertility).”

barplot(table(dat[,64]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,64]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 64] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.61780 -0.62177  0.07746  0.67457  2.51904 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    2.646651   0.345732   7.655 4.59e-13 ***
## dat$age        0.002013   0.005491   0.367  0.71426    
## dat$is.male   -0.038768   0.134752  -0.288  0.77383    
## dat$is.white  -0.455374   0.139437  -3.266  0.00125 ** 
## dat$politics   0.341192   0.061643   5.535 8.08e-08 ***
## dat$education -0.160778   0.049401  -3.255  0.00130 ** 
## dat$income     0.038610   0.045536   0.848  0.39733    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.011 on 242 degrees of freedom
##   (4 observations deleted due to missingness)
## Multiple R-squared:  0.1752, Adjusted R-squared:  0.1548 
## F-statistic: 8.569 on 6 and 242 DF,  p-value: 1.871e-08

“Some COIVD-19 vaccines can cause cardiovascular issues.”

barplot(table(dat[,65]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,65]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 65] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.82719 -0.58944 -0.02048  0.72900  2.60433 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    2.759169   0.369665   7.464 1.51e-12 ***
## dat$age        0.003812   0.005872   0.649   0.5168    
## dat$is.male   -0.105262   0.144080  -0.731   0.4657    
## dat$is.white  -0.228077   0.149089  -1.530   0.1274    
## dat$politics   0.292436   0.065910   4.437 1.39e-05 ***
## dat$education -0.134137   0.052820  -2.539   0.0117 *  
## dat$income     0.065946   0.048688   1.354   0.1769    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.081 on 242 degrees of freedom
##   (4 observations deleted due to missingness)
## Multiple R-squared:  0.1034, Adjusted R-squared:  0.08116 
## F-statistic: 4.651 on 6 and 242 DF,  p-value: 0.0001701

“If you take too many vaccines, future vaccines won’t be as effective for me.”

barplot(table(dat[,66]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,66]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 66] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.7792 -0.6468  0.1059  0.7655  2.0201 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    3.115043   0.341394   9.124  < 2e-16 ***
## dat$age        0.010802   0.005429   1.990 0.047764 *  
## dat$is.male   -0.025169   0.133105  -0.189 0.850179    
## dat$is.white  -0.165640   0.138014  -1.200 0.231254    
## dat$politics   0.263920   0.060857   4.337 2.13e-05 ***
## dat$education -0.175728   0.049052  -3.582 0.000412 ***
## dat$income     0.042471   0.045210   0.939 0.348458    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.9983 on 241 degrees of freedom
##   (5 observations deleted due to missingness)
## Multiple R-squared:  0.1245, Adjusted R-squared:  0.1027 
## F-statistic: 5.714 on 6 and 241 DF,  p-value: 1.419e-05

“Taking vaccines lowers the body’s natural immunity.”

barplot(table(dat[,67]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,67]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 67] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.20750 -0.79014 -0.01844  0.86470  2.82402 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    2.736169   0.365577   7.485 1.33e-12 ***
## dat$age        0.004730   0.005807   0.815 0.416072    
## dat$is.male    0.060939   0.142487   0.428 0.669264    
## dat$is.white  -0.230101   0.147441  -1.561 0.119917    
## dat$politics   0.221617   0.065181   3.400 0.000788 ***
## dat$education -0.080348   0.052236  -1.538 0.125313    
## dat$income    -0.102046   0.048150  -2.119 0.035079 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.069 on 242 degrees of freedom
##   (4 observations deleted due to missingness)
## Multiple R-squared:  0.1023, Adjusted R-squared:  0.08001 
## F-statistic: 4.595 on 6 and 242 DF,  p-value: 0.0001939

“The body’s natural immunity doesn’t work as well when you receive vaccines.”

barplot(table(dat[,68]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,68]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 68] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.90649 -0.93965  0.02595  0.83972  2.79967 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    2.600041   0.401011   6.484 4.99e-10 ***
## dat$age        0.006650   0.006369   1.044   0.2975    
## dat$is.male    0.063195   0.156298   0.404   0.6863    
## dat$is.white  -0.403252   0.161732  -2.493   0.0133 *  
## dat$politics   0.368900   0.071499   5.160 5.16e-07 ***
## dat$education -0.120835   0.057299  -2.109   0.0360 *  
## dat$income    -0.095970   0.052817  -1.817   0.0704 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.173 on 242 degrees of freedom
##   (4 observations deleted due to missingness)
## Multiple R-squared:  0.1664, Adjusted R-squared:  0.1458 
## F-statistic: 8.054 on 6 and 242 DF,  p-value: 6.119e-08

“Vaccines are only temporary”.

barplot(table(dat[,69]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,69]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 69] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.7584 -0.8975  0.0293  0.8561  2.7154 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    3.017801   0.393815   7.663 4.37e-13 ***
## dat$age        0.002662   0.006255   0.426  0.67075    
## dat$is.male    0.113042   0.153493   0.736  0.46216    
## dat$is.white  -0.321039   0.158829  -2.021  0.04435 *  
## dat$politics   0.251636   0.070216   3.584  0.00041 ***
## dat$education -0.151931   0.056271  -2.700  0.00742 ** 
## dat$income    -0.061670   0.051869  -1.189  0.23562    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.152 on 242 degrees of freedom
##   (4 observations deleted due to missingness)
## Multiple R-squared:   0.12,  Adjusted R-squared:  0.09816 
## F-statistic: 5.499 on 6 and 242 DF,  p-value: 2.341e-05

“The COVID-19 vaccines were designed to continually require boosters.”

barplot(table(dat[,70]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,70]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 70] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.8646 -0.5475  0.2676  0.5480  1.7969 
## 
## Coefficients:
##                 Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    3.6359850  0.3487946  10.424   <2e-16 ***
## dat$age       -0.0020928  0.0055406  -0.378   0.7060    
## dat$is.male   -0.0002993  0.1362744  -0.002   0.9982    
## dat$is.white   0.2008430  0.1407733   1.427   0.1550    
## dat$politics  -0.0034885  0.0621590  -0.056   0.9553    
## dat$education  0.0659531  0.0499749   1.320   0.1882    
## dat$income    -0.1195276  0.0461712  -2.589   0.0102 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.02 on 241 degrees of freedom
##   (5 observations deleted due to missingness)
## Multiple R-squared:  0.03076,    Adjusted R-squared:  0.006628 
## F-statistic: 1.275 on 6 and 241 DF,  p-value: 0.2696

“There is no need to take a COVID-19 vaccine unless your are a frontline worker or part of an at-risk group.”

barplot(table(dat[,71]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,71]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 71] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.1328 -0.6922  0.1782  0.7503  2.0231 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    3.491437   0.345492  10.106  < 2e-16 ***
## dat$age        0.005569   0.005488   1.015  0.31126    
## dat$is.male   -0.238079   0.134984  -1.764  0.07904 .  
## dat$is.white   0.045397   0.139440   0.326  0.74503    
## dat$politics   0.215941   0.061570   3.507  0.00054 ***
## dat$education -0.093469   0.049502  -1.888  0.06020 .  
## dat$income    -0.039953   0.045734  -0.874  0.38321    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.01 on 241 degrees of freedom
##   (5 observations deleted due to missingness)
## Multiple R-squared:  0.09498,    Adjusted R-squared:  0.07245 
## F-statistic: 4.216 on 6 and 241 DF,  p-value: 0.0004698

“People who are healhty don’t need to take COVID-19 vaccines.”

barplot(table(dat[,72]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,72]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 72] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.57818 -0.91620  0.02322  0.95221  2.46502 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    2.093311   0.425584   4.919 1.61e-06 ***
## dat$age        0.003835   0.006674   0.575  0.56605    
## dat$is.male    0.161942   0.163689   0.989  0.32350    
## dat$is.white  -0.232440   0.169332  -1.373  0.17113    
## dat$politics   0.218587   0.075346   2.901  0.00406 ** 
## dat$education -0.029560   0.059911  -0.493  0.62218    
## dat$income     0.041025   0.055196   0.743  0.45805    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.226 on 241 degrees of freedom
##   (5 observations deleted due to missingness)
## Multiple R-squared:  0.04436,    Adjusted R-squared:  0.02056 
## F-statistic: 1.864 on 6 and 241 DF,  p-value: 0.08763

“People who are healthy don’t need to take flu vaccines.”

barplot(table(dat[,73]),col="red4",names.arg=likert.labels,las=2,cex.names=.55)

summary(lm(dat[,73]~
             dat$age+dat$is.male+dat$is.white+dat$politics
           +dat$education+dat$income))

## 
## Call:
## lm(formula = dat[, 73] ~ dat$age + dat$is.male + dat$is.white + 
##     dat$politics + dat$education + dat$income)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.87363 -0.93172 -0.03117  1.04284  2.55326 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    1.713065   0.421825   4.061  6.6e-05 ***
## dat$age        0.004332   0.006700   0.647    0.519    
## dat$is.male    0.200177   0.164410   1.218    0.225    
## dat$is.white  -0.149811   0.170126  -0.881    0.379    
## dat$politics   0.439827   0.075210   5.848  1.6e-08 ***
## dat$education -0.089913   0.060273  -1.492    0.137    
## dat$income     0.052708   0.055558   0.949    0.344    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.234 on 242 degrees of freedom
##   (4 observations deleted due to missingness)
## Multiple R-squared:  0.1372, Adjusted R-squared:  0.1158 
## F-statistic: 6.415 on 6 and 242 DF,  p-value: 2.744e-06

General discussion

People might exhibit a ”monological belief system” depending on the specificity of the questions posed to them. We observed in Study 1 that, when people are asked about “vaccines” (no specific vaccine), they exhibit strong homogeneity in their Likert ratings for statements justifying NVAs.

It is unclear, however, whether the increased diversity in responses between Study 1 and Study 2 was due to increased specificity of the questions or because of the rephrasing of the focal items that were both rated and ranked.

When creating items for a ranking task, it becomes important to have heterogeneity (dissimilarities) between items. With Likert ratings, though, it is often beneficial to have some degree of homogeneity (similarities) between items.

Further analyses need to be done in order to further clarify the extent to which ranking data adds useful information, alongside Likert items.