Pilot Results

Average Time to Complete Survey

ds_content <- rio::import("assets/data/ds_content.rds", trust=T)

Total complete observations: \(n = 11\).

Content Validity Ratio (CVR)

The Content Validity Ratio (CVR) was developed by Lawshe (1975) to provide a quantitative measure of content validity. The CVR provides an index that represents the proportion of agreement among experts on the essentialness of an item.

Calculation: To calculate CVR, each item is rated by a panel of experts who are knowledgeable about the construct being measured. The experts rate the items on a scale such as “essential,” “useful, but not essential,” or “not necessary.”

Formula: \[CVR = \frac{n_e - \frac{N}{2}}{\frac{N}{2}}\]

Where:

• \(n_e\) is the number of panelists indicating “essential;” and,
  
• \(N\) is the total number of panelists.

CVR Guidelines^⚠️

1. A positive \(CVR\) value indicates that more than half of the experts rated the item as “essential.” The closer the \(CVR\) is to \(1\), the higher the agreement among experts that the item is essential.
   
2. A \(CVR\) of \(0\) indicates that exactly half the panelists rated the item as “essential.”
3. A negative \(CVR\) value indicates that less than half of the panelists rated the item as “essential.”

CVR Guidelines

Minimum Values of Content Validity Ratio. Reprinted from Lawshe (1975).

According to Lawshe (1975), for an item to be retained, the \(CVR\) must be significant at a certain level. The level of significance depends on the number of panelists. For example, with \(5\) panelists, the \(CVR\) must be \(0.99\) or higher to be significant at the \(0.05\) level. With \(10\) panelists, the \(CVR\) must be \(0.62\) or higher, and with \(20\) panelists, the \(CVR\) must be \(0.42\) or higher.

CVR Guidelines

CVR

CVR results

Content Validity Index (CVI)

The Content Validity Index (CVI) quantifies content validity. It’s calculated based on expert ratings of the relevance of individual items and the overall scale.

Item-level CVI (I-CVI)

This is calculated for each individual item.

I-CVI Formula: \[I-CVI = \frac{n_{r}}{N}\]

Where: - \(n_{r}\) is the number of experts who rated the item as quite or highly relevant; and,
- \(N\) is the total number of experts.

Scale-level CVI (S-CVI)

This is calculated for the entire scale or instrument.

S-CVI Formulas

There are two types of S-CVI: Average CVI (S-CVI/Ave) and Universal Agreement CVI (S-CVI/UA).

\[S-CVI/Ave = \frac{\sum_{i=1}^{k} I-CVI_i}{k}\]

\[S-CVI/UA = \frac{n_{I-CVI=1}}{k}\]

Where: - \(I-CVI_i\) is the I-CVI for item \(i\),
- \(k\) is the number of items, and,
- \(n_{I-CVI=1}\) is the number of items with an I-CVI of \(1\).

CVI Guidelines

Lynn (1986) suggests that an I-CVI of \(0.78\) or higher is acceptable for at least nine experts. While Yusoff (2019) summarizes a bit more detailed guidelines from various authors.

CVI

CVI Calculation

i_cvi <- function(x) {
  n <- length(x)
  n_relevant <- sum(x >= 3)  # Count ratings 3 and 4 as relevant
  return(n_relevant / n)
}
i_cvi_values <- apply(ds_content[,cvi_items], 2, i_cvi)

# Compute S-CVI
s_cvi_ave <- mean(i_cvi_values, na.rm = TRUE)
s_cvi_ua <- sum(i_cvi_values == 1, na.rm = TRUE) / length(i_cvi_values)

CVI Results