Computational Psychomet [ comment ] ics Applied to Enginee [ comment ] ing

.title[
# Computational Psychomet<svg viewBox="0 0 581 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;"> [ comment ] <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z" /></svg>ics Applied to Enginee<svg viewBox="0 0 581 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;"> [ comment ] <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z" /></svg>ing
]
.subtitle[
## <br>Validity and Its Evidence
]
.author[
### Jo<svg viewBox="0 0 581 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;"> [ comment ] <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z" /></svg>ge Sinval
]
.date[
### 2026-08-12
]

---

# Validity
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Validity

At the turn of the 20<sup>th</sup> century, the growth of educational and psychological measurement in the U.S.A. raised quality concerns. The APA established committees in 1895 and 1906 for standardization, but early efforts were ineffective (Newton and Shaw, 2013). Later, the Standardization Committee aimed to create consensus on assessing test superiority, providing definitions for key terms and outlining processes for determining validity and reliability:

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

Two of the most important types of problem in measurement are those connected with the determination of what a test measures, and of how consistently it measures. The first should be called the problem of **validity**, the second, the problem of **reliability**.

---

# Validity

## What is it?

.can-edit.key-validity[
- thing one
- thing two
- ...
]

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment.

---

# Validity

## What is it?

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

Validity refers to the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests.

.tr[
📖 (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 2014, p. 11)
]]

---

# Validity

## Unitary Concept

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

Although different sources and mixes of evidence are required to support
particular score-based inferences, validity is a unitary concept in the sense that score meaning as embodied in construct validity underlies all score-based inferences.

---

# Validity

A test or measure is neither valid nor invalid:

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

Hence, what is to be validated is not the test or observation device as such but the inferences derived from test scores or other indicators &mdash; inferences about score meaning or interpretation and about the implications for action that the interpretation entails.

---

# Validity

Few studies have properly addressed validity, and many have been criticized for their lack of rigor.

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

Validity has long been one of the major deities in the pantheon of the psychometrician. It is universally praised, but the good works done in its name are remarkably few.

---

# Validity

## The necessity of evidence

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

It never makes sense to employ a tool without supporting validity evidence&mdash;regardless of how cheap, how low the adverse impact, or how easy to administer

---

# Validity

## Standards for talking and thinking about validity

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

1. Thou shalt not refer to 'the validity of the test' (TVOTT), that is, as though validity were a property of tests.  
2. Thou shalt (not) use validity modifier labels (VMLs), that is, terms like content validity and predictive validity (there is a not in parentheses because it was promoted by the first three editions yet rejected by the fourth and fifth).

---

# Validity

## Standards for talking and thinking about validity

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[
Validity is not a property of the test. Rather, it is a property of the proposed interpretations and uses of the test scores. Interpretations and uses that make sense and are supported by appropriate evidence are considered to have high validity (or for short, to be valid), and interpretations or uses that are not adequately supported, or worse, are contradicted by the available evidence are taken to have low validity (or for short, to be invalid). The scores generated by a given test can be given different interpretations, and some of these interpretations may be more plausible than others.

---

# Validity

## Standards for talking and thinking about validity

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[
The sloppy language of 'the test is reliable' or 'the test is valid' should be avoided; instead, 'measurement reliability', 'test score reliability', 'validity of test scores or test use' are better choices. Sloppy language use in research context may corrupt our understanding about the nature of measurement reliability and validity, and may lead to unhealthy research practice, such as making little or no effort in evaluating or reporting the reliability of our measurement in a research study."

---

# Validity

## Standards for talking and thinking about validity

Validity refers to the characteristics of measurement &mdash; that is, the characteristics of the scores obtained from administering a test to a specific group, rather than the characteristics of a test itself (Fan, 2013).

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[
...a test is not 'reliable' or 'unreliable.' Rather, reliability is a property of the scores on a test for a particular group of examinees. Thus, potential test users need to determine whether reliability reported in test manuals are based on samples similar in composition and variability to the group for whom the test will be used.

---

# Validity

## Standards for talking and thinking about validity

Reliability does not refer to the instrument itself:

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[
Technically, reliability refers to the consistency of the results obtained, not to the instrument itself.

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[
reliability refers to the results obtained with an evaluation instrument and not to the instrument itself… it is more appropriate to speak of the reliability of the 'test scores' or of the 'measurement' than of the 'test' or the 'instrument.'

---

# Validity

## Standards for talking and thinking about validity

As previously mentioned, validity refers to the appropriate use of test scores in a given measurement situation but not to the test or instrument itself (American Educational Research Association American Psychological Association et al., 2014; Fan, 2013).

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

If a researcher consciously or unconsciously assumes that reliability (validity) is a characteristic of a test or instrument itself, then reliability information from the test norming sample (as reported in the test technical manual), or those reported in other studies, would be applicable in his/her own study; as a result, there would be no, or less, need to evaluate measurement reliability and to report reliability estimates for his/her own data.

---

# Validity

## Standards for talking and thinking about validity

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

We sometimes speak of the 'validity of a test' for the sake of convenience, but it is more correct to speak of the validity of the interpretation and use to be made of the results

---
# Validity

## The "Unqualified" Phrase: The Conflict with Standards

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

"The 2014 edition of the Standards reinforces this perspective, explicitly stating that
referring to “the validity of the test” is incorrect and unqualified. Nevertheless ... it is not uncommon to see language testing scholars use phrases such as “the validity of the test” or “test validity” in academic publications."

---

# Validity

## The "Unqualified" Phrase: How often is the phrase used?

An analysis of 449 research articles in *Language Testing* and *Language Assessment Quarterly* revealed:

* **26.5% (119 articles)** included the phrase "validity of + test".
* The frequency was similar across journals: **27.3%** in LAQ and **25.7%** in LT.

---

# Validity

## Author Awareness: Shorthand vs. Ignorance

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

We hypothesize that convenience is the main driver behind the use of this expression... "test" may be a shorthand for saying "inference about a test taker based on a score."

* **26.9%** of the authors who used the "incorrect" phrase *also* explicitly discussed validity in terms of score interpretation and use elsewhere in their text.  
* This suggests that use of the phrase often co-exists with an awareness of the Messickian definition (Messick, 1989), rather than indicating a lack of knowledge.

---

# Validity

## Evolution of the concept in the _Standards_

<div class="figure" style="text-align: center">
<img src="assets/img/standards_evolution.png" alt="The Evolution of the Concept of Validity in the &lt;i&gt;Standards&lt;/i&gt;. Reprinted from Sireci and Padilla (2014)." width="45%" />
<p class="caption">The Evolution of the Concept of Validity in the <i>Standards</i>. Reprinted from Sireci and Padilla (2014).</p>
</div>

---

# Validity

## Evolution of the concept in the _Standards_

The concept of validity has been evident in literature since the early 20<sup>th</sup> century and has evolved over time. Initially, validity was seen as a static property measured by a single statistic, typically the correlation of test scores with a certain criterion. However, modern validity theory views validity more about the appropriateness of inferences and uses made from assessment results, including the consequences of the usage of test scores (Shaw and Crisp, 2011).

]

.pull-right-1[
<div class="figure" style="text-align: center">
<img src="assets/img/standards_evolution_ii.png" alt="Timeline of the evolution of validity theory Tracing. Reprinted from Shaw and Crisp (2011)." width="65%" />
<p class="caption">Timeline of the evolution of validity theory Tracing. Reprinted from Shaw and Crisp (2011).</p>
</div>

]

---

# Validity

## The new _Standards_

|Evidence Source|Definition|Examples|
|---|---|---|
|Content| The "relationship between a test’s content and the construct it is intended to measure." Refers to themes, wording, and format of items on an assessment instrument. Includes analyses by experts regarding how adequately items represent the content domain. Also includes development strategies to ensure appropriate content representation|Surveying experienced teachers regarding the adequacy and representativeness of proposed instrument items. Choosing items previously utilized in similar settings. Developing instruments based on established educational theories.|
|Response process|Analyses of responses, including the actions, strategies, and thought processes of individual respondents or observers. Differences in response processes may reveal sources of variance that are irrelevant to the construct being measured. Also includes instrument security, scoring, and reporting of results.|Interviewing and studying learners regarding factors that influence the ratings they assign to teachers. Analyzing varying response patterns among different categories/levels of learners.|
|Internal structure|The degree to which individual items within the instrument fit the underlying constructs. Items measuring a unidimensional construct should be homogeneous, while items measuring complex constructs should not. Most often reported as measures of internal consistency reliability and factor analysis.|Using factor analysis to determine the dimensional structure of an instrument’s scores, and determining the reliability of scores. Studying differential functioning of items among a homogeneous group of evaluators.|
|Relations to other variables|The relationship between scores and other variables relevant to the construct being measured. Relationships may be positive (convergent or predictive) or negative (divergent or discriminant) depending on the constructs being measured.|How well do teachers’ assessment scores predict learners’ performance on high-stakes examinations, or their choice of a medical specialty? Do scores correlate with other measures of the same construct? Can results of an evaluation be generalized from one setting to another, similar setting?|
|Consequences|Assessments are intended to have some desired effect (e.g., improve teaching performance), but they also have unintended effects. Evaluating such consequences can support or challenge the validity of score interpretations.|Do equally qualified teachers’ performances on clinical teaching assessments correlate with factors that are not being measured, such as medical subspecialty, gender, or ethnicity?|

]

Validity: Sources of Evidence, Definitions, and Examples. Reprinted from Beckman, Cook, and Mandrekar (2005).

---

# Validity

## Five sources of validity evidence

<div class="figure" style="text-align: center">
<img src="assets/img/Standards.jpg" alt="The Standards of Educational and Psychological Testing(American Educational Research Association American Psychological Association et al., 2014). Freely available &lt;a href="https://www.testingstandards.net/uploads/7/6/6/4/76643089/9780935302356.pdf"&gt;here&lt;/a&gt;." width="28%" />
<p class="caption">The Standards of Educational and Psychological Testing(American Educational Research Association American Psychological Association et al., 2014). Freely available <a href="https://www.testingstandards.net/uploads/7/6/6/4/76643089/9780935302356.pdf">here</a>.</p>
</div>

---

# Evidence Sources
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Evidence Sources

## Based on Content

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

**Content** &mdash; relationship between test content and the construct of interest; theory; hypothesis about content; independent assessment of match between content sampled and domain of interest; solid, scientific, quantitative evidence.

???
Partially based on [this](https://researchmethod.net/content-validity/) page.

---

# Evidence Sources

## Based on Content: A parabole

### The blind men and the elephant. A hindoo fable

**I**.

It was six men of Indostan  
To learning much inclined,  
Who went to see the Elephant  
(Though all of them were blind),  
That each by observation  
Might satisfy his mind.

]

**II**.

The _First_ approached the Elephant,  
And happening to fall  
Against his broad and sturdy side,  
At once began to bawl:  
"God bless me!—but the Elephant  
Is very like a wall!"

]

The _Second_, feeling of the tusk,  
Cried: "Ho!—what have we here  
So very round and smooth and sharp?  
To me 't is mighty clear  
This wonder of an Elephant  
Is very like a spear!"

]

---

# Evidence Sources

## Based on Content: A parabole

### The blind men and the elephant. A hindoo fable

**IV**.

The _Third_ approached the animal,  
And happening to take  
The squirming trunk within his hands,  
Thus boldly up and spake:  
"I see," quoth he, "the Elephant  
Is very like a snake!"

]

**V**.

The _Fourth_ reached out his eager hand,  
And felt about the knee.  
"What most this wondrous beast is like  
Is mighty plain," quoth he;  
"'T is clear enough the Elephant  
Is very like a tree!"

]

**VI**.

The _Fifth_, who chanced to touch the ear,  
Said: "E'en the blindest man  
Can tell what this resembles most;  
Deny the fact who can,  
This marvel of an Elephant  
Is very like a fan!"

]

---

# Evidence Sources

## Based on Content: A parabole

### The blind men and the elephant. A hindoo fable

**VII**.

The _Sixth_ no sooner had begun  
About the beast to grope,  
Than, seizing on the swinging tail  
That fell within his scope,  
"I see," quoth he, "the Elephant  
Is very like a rope!"

]

**VIII**.

And so these men of Indostan  
Disputed loud and long,  
Each in his own opinion  
Exceeding stiff and strong,  
Though each was partly in the right,  
And all were in the wrong!

]

**Moral**.

So, oft in theologic wars  
The disputants, I ween,  
Rail on in utter ignorance  
Of what each other mean,  
_And prate about an Elephant_
_Not one of them has seen!_

(Saxe, 1873, pp. 135-136)

]

---

# Evidence Sources

## Based on Content: A parabole

### The blind men and the elephant

<br>

<div class="figure" style="text-align: center">
<img src="assets/img/blind_men_and_elephant.jpg" alt="The blind men and the elephant. Reprinted from Woods (1916, p. 14)." width="41%" />
<p class="caption">The blind men and the elephant. Reprinted from Woods (1916, p. 14).</p>
</div>

---

# Evidence Sources

## Based on Content

Examples (Downing and Yudkowsky, 2009, p. 30):

• Examination blueprint  
  • Representativeness of test blueprint to achievement domain  
  • Test specifications  
  • Match of item content to test specifications  
  • Representativeness of items to domain  
  • Logical/empirical relationship of content tested to achievement domain  
  • Quality of test questions  
  • Item writer qualifications  
  • Sensitivity review

---

# Evidence Sources

## Based on Content: Content Validity Ratio (CVR)

The Content Validity Ratio (CVR) was developed by Lawshe (1975) to provide a quantitative measure of content validity. The CVR provides an index that represents the proportion of agreement among experts on the essentialness of an item.

**Calculation**: To calculate CVR, each item is rated by a panel of experts who are knowledgeable about the construct being measured. The experts rate the items on a scale such as "essential," "useful, but not essential," or "not necessary."

**Formula:** The CVR is then calculated for each item using the formula:

`$$CVR = \frac{n_e - \frac{N}{2}}{\frac{N}{2}}$$`

Where:

- `$n_e$` is the number of panelists indicating "essential;" and,  
  - `$N$` is the total number of panelists.

---

# Evidence Sources

## Based on Content: CVR Guidelines<sup>⚠️</sup>

1. A positive `$CVR$` value indicates that more than half of the experts rated the item as "essential." The closer the `$CVR$` is to `$1$`, the higher the agreement among experts that the item is essential.  
  2. A `$CVR$` of `$0$` indicates that exactly half the panelists rated the item as "essential." 
  3. A negative `$CVR$` value indicates that less than half of the panelists rated the item as "essential."

.footnote[<sup>⚠️</sup>A high CVR doesn't necessarily mean the item is good, it just means there's a high agreement among experts on its essentialness. Other aspects such as item difficulty, and discrimination should also be considered when evaluating the quality of a test item.]

---

# Evidence Sources

## Based on Content: CVR Guidelines

<div class="figure" style="text-align: center">
<img src="assets/img/Lawshe1975.png" alt="Minimum Values of Content Validity Ratio. Reprinted from ." width="41%" />
<p class="caption">Minimum Values of Content Validity Ratio. Reprinted from .</p>
</div>

According to , for an item to be retained, the `$CVR$` must be significant at a certain level. The level of significance depends on the number of panelists. For example, with `$5$` panelists, the CVR must be `$0.99$` or higher to be significant at the `$0.05$` level. With `$10$` panelists, the `$CVR$` must be `$0.62$` or higher, and with `$20$` panelists, the `$CVR$` must be `$0.42$` or higher.

---

# Evidence Sources

## Based on Content: CVR Guidelines

A more recent proposal (Ayre and Scally, 2014).

]

<div class="figure" style="text-align: center">
<img src="assets/img/cvr_critical_ayre2014.png" alt="Critical Values of Content Validity Ratio. Reprinted from ." width="56%" />
<p class="caption">Critical Values of Content Validity Ratio. Reprinted from .</p>
</div>

]

---

# Evidence Sources

## Based on Content: Example CVR

> Let's assume you have collected data from five experts who rated the relevance of four items on a scale from 1 to 3 (3 &mdash; "Not necessary," 2 &mdash; "Useful, but not essential," 1 &mdash; "Essential").

``` r
ds <- data.frame(
  item1 = c(3, 1, 2, 2, 1),
  item2 = c(1, 2, 2, 1, 1),
  item3 = c(1, 1, 1, 1, 1),
  item4 = c(2, 2, 3, 3, 3))

psychometric::CVratio(NTOTAL = lengths(ds),
                      NESSENTIAL = colSums(ds == 1))
```

```
## item1 item2 item3 item4 
##  -0.2   0.2   1.0  -1.0
```

]

---

# Evidence Sources

## Based on Content: Content Validity Index (CVI)

The Content Validity Index (CVI) quantifies content validity. It's calculated based on expert ratings of the relevance of individual items and the overall scale.

**Item-level CVI (I-CVI)**

This is calculated for each individual item.

**I-CVI Formula**

The I-CVI is calculated using the formula:

`$$I-CVI = \frac{n_{r}}{N}$$`

Where,  
  - `$n_{r}$` is the number of experts who rated the item as quite or highly relevant, and,  
  - `$N$` is the total number of experts.

---

# Evidence Sources

## Based on Content: CVI

**Scale-level CVI (S-CVI)**

This is calculated for the entire scale or instrument.

**S-CVI Formulas**

There are two types of S-CVI: Average CVI (S-CVI/Ave) and Universal Agreement CVI (S-CVI/UA).

`$$S-CVI/Ave = \frac{\sum_{i=1}^{k} I-CVI_i}{k}$$`

`$$S-CVI/UA = \frac{n_{I-CVI=1}}{k}$$`

Where,  
  - `$I-CVI_i$` is the I-CVI for item `$i$`,  
  - `$k$` is the number of items, and,  
  - `$n_{I-CVI=1}$` is the number of items with an I-CVI of `$1$`.

---

# Evidence Sources

## Based on Content: CVI Guidelines

Lynn (1986) suggests that an I-CVI of `$0.78$` or higher is acceptable for at least nine experts. While Yusoff (2019) summarizes a bit more detailed guidelines from various authors.

|Number of experts|Acceptable `$CVI$` values|Source of recommendation|
|---|---|---|---|
| `$2$` experts| `$CVI \geq.80$`|Davis (1992)|
| `$3-5$` experts| `$CVI = 1.00$`|Polit and Beck (2006); Polit, Beck, and Owen (2007)|
| `$\geq 6$` experts| `$CVI \geq .83$`|Polit and Beck (2006); Polit Beck et al. (2007)|
| `$6-8$` experts| `$CVI \geq .83$`|Lynn (1986)|
| `$\geq9$` experts| `$CVI \geq .78$`|Lynn (1986)|

Recommended values of `$CVI$`. Adapted from Yusoff (2019).

---

# Evidence Sources

## Based on Content: Example CVI

> Let's say you have a set of four items that are rated by five experts on a 4-point scale (1 &mdash; "Not relevant," 2 &mdash; "Somewhat relevant," 3 &mdash; "Quite relevant," 4 &mdash; "Highly relevant").

``` r
ds <- data.frame(
  item1 = c(4, 3, 4, 2, 4),
  item2 = c(3, 2, 4, 3, 2),
  item3 = c(2, 2, 3, 2, 1),
  item4 = c(4, 4, 4, 3, 4))
# Compute I-CVI without missing data
i_cvi <- function(x) {
  n <- length(x)
  n_relevant <- sum(x >= 3)  # Count ratings 3 and 4 as relevant
  return(n_relevant / n)}
i_cvi_values <- apply(ds, 2, i_cvi)
# Compute S-CVI
s_cvi_ave <- mean(i_cvi_values)
s_cvi_ua <- sum(i_cvi_values == 1) / length(i_cvi_values)
i_cvi_values
s_cvi_ave
s_cvi_ua
```

---

# Evidence Sources

## Based on Content: Example CVI

```
## item1 item2 item3 item4 
##   0.8   0.6   0.2   1.0
```

```
## [1] 0.65
```

```
## [1] 0.25
```

]

---

# Evidence Sources

## Based on Content: Practical Example

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

de
Melo, Duval, and Isler (2023) conducted a survey at the University of São Paulo's main campus, where most participants `$(83.1\%)$` stated they benefit from free circular bus service. The questionnaire included questions about socio-economic profiles, travel behavior, and attitudes towards the bus service. Participants were asked to express their level of agreement with positive statements about the bus service, such as its cost-effectiveness, safety, and low incidence of assaults. The responses were measured on an ordinal scale with options ranging from "Strongly Disagree" to "Strongly Agree". The data from this survey was used in a study published at the National Association of Research and Teaching in Transport congress. Part of the study's data was didactically adapted for this example.

]

---

# Evidence Sources

## Based on Content: Practical Example

Participants were asked to express their level of agreement with the following statements on a 5-point Likert scale ranging from "Strongly Disagree" to "Strongly Agree".

"For me, using the circular bus lines to access the USP Campus in São Paulo is..."

|Item|Strongly Disagree<br>1|Disagree<br>2|Neutral<br>3|Agree<br>4|Strongly Agree<br>5|
|---|---|---|---|---|---|
|...a healthy option| | | | | |
|...a quick option| | | | | |
|...a comfortable option| | | | | |
|...a safe option against thefts and robberies| | | | | |
|...an interesting option| | | | | |
|...a punctual option| | | | | |
|...a cheap option| | | | | |
|...an unfeasible option| | | | | |
|...a pleasant option| | | | | |
|...a safe option in relation to traffic accidents| | | | | |
|...a low-polluting option| | | | | |
|...an easy and practical option| | | | | |

]

---

# Evidence Sources

## Based on Content: Practical Example

The items content will be analyzed by a panel of transportation engineering experts (the students in our class). The experts are asked to rate the relevance of each item to the construct being measured ([link](https://rstudio-cld.d.incd.pt:8443/index.php/168145)).
<center>
<embed src="https://rstudio-cld.d.incd.pt:8443/index.php/168145" width=1000px height=400px />
</center>

---

# Evidence Sources

## Based on Content: Practical Example

Instructions to the experts (regarding `$CVR$`):

The following items have been developed for a psychometric instrument intended to measure attitudes regarding the use of circular bus lines to access the USP Campus in São Paulo. Keeping in mind the purpose of the said instrument, you are asked to rate each of the 12 items presented below using the following response scale: 1 &mdash; "Essential"; 2 &mdash; "Useful, but not essential"; 3 &mdash; "Not necessary". The instrument to be presented to the participants (users of the public bus transportation system that operates between the Butantã metro station in São Paulo and the CUASO units of the University of São Paulo) is shown next, with the following instructions:

_Factors that make up the attitude (possible answers 'Strongly Disagree', 'Disagree', 'Indifferent', 'Agree' or 'Strongly Agree')._

|Item<br>**_For me, using the circular bus lines to access the USP Campus in São Paulo is..._**|Essential<br>1|Useful, but Not Essential<br>2|Not Necessary<br>3|
|---|---|---|---|
|...a healthy option| | | | 
|...a quick option| | | |
|...a comfortable option| | | |
|...a safe option against thefts and robberies| | | |
|...an interesting option| | | |
|...a punctual option| | | |
|...a cheap option| | | |
|...an unfeasible option| | | |
|...a pleasant option| | | |
|...a safe option in relation to traffic accidents| | | |
|...a low-polluting option| | | |
|...an easy and practical option| | | |

]

---

# Evidence Sources

## Based on Content: Practical Example

Instructions to the experts (regarding `$CVI$`):

The following items have been developed for a psychometric instrument that intends to measure attitudes related to the use of circular bus lines to access the USP Campus in São Paulo.  Considering the aim of the instrument in question, you are asked to rate each of the 12 items below using the following response scale: 1 &mdash; "Not relevant," 2 &mdash; "Slightly relevant," 3 &mdash;Quite relevant," 4 &mdash; "Highly relevant". The instrument to be presented to the participants (users of the public bus transportation system that operates between the Butantã metro station in São Paulo and the CUASO units of the University of São Paulo) is shown next, with the following instructions:

_Factors that make up the attitude (possible answers 'Strongly Disagree', 'Disagree', 'Indifferent', 'Agree' or 'Strongly Agree')._

|Item<br>**_For me, using the circular bus lines to access the USP Campus in São Paulo is..._**|Not Relevant<br>1|Slightly relevant<br>2|Quite Relevant<br>3|Highly Relevant<br>4|
|---|---|---|---|---|
|...a healthy option| | | | |
|...a quick option| | | | |
|...a comfortable option| | | | |
|...a safe option against thefts and robberies| | | | |
|...an interesting option| | | | |
|...a punctual option| | | | |
|...a cheap option| | | | |
|...an unfeasible option| | | | |
|...a pleasant option| | | | |
|...a safe option in relation to traffic accidents| | | | |
|...a low-polluting option| | | | |
|...an easy and practical option| | | | |

]

---

# Su<svg viewBox="0 0 581 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;" xmlns="http://www.w3.org/2000/svg">  <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"></path></svg>vey <svg viewBox="0 0 581 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;" xmlns="http://www.w3.org/2000/svg">  <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"></path></svg>esults

Read or survey data directly from LimeSurvey (LimeSurvey GmbH, 2024):

<div class="pre-name">limesurvey.R</div>

``` r
survey_id <- 168145
pacman::p_load(limer)
options(lime_api = Sys.getenv("lime_api"),
        lime_password = Sys.getenv("lime_password"),
        lime_username = Sys.getenv("lime_username"))
get_session_key()
ds <- get_responses(iSurveyID= survey_id,
                    sCompletionStatus='all',
                    sResponseType='short')
#select the variables for cvr (the ones that start wiht "cvr" and "cvi")
item_att_cvr <- colnames(ds)[grep("^cvr", colnames(ds))]
item_att_cvi <- colnames(ds)[grep("^cvi", colnames(ds))]
ds[ds == ""] <- NA #if "" the answer should replaced by NA
ds <- ds[complete.cases(ds[,c(item_att_cvi,item_att_cvr)]),
         c(item_att_cvi,item_att_cvr)] #remove rows with NA
```

```
## [1] "jU7ycisEk_Wd5mz38bg0HyoYDjfOA2GI"
```

---

<div class="pre-name">example-cvr.R</div>

``` r
kableExtra::kbl(head(ds[,item_att_cvr]), caption = "Items' CVR: First 6 rows")

#count number of NA per columns
n_total <- nrow(ds[,item_att_cvr]) - colSums(is.na(ds[,item_att_cvr]))

psychometric::CVratio(NTOTAL = n_total,
                      NESSENTIAL = colSums(ds[,item_att_cvr] == 1, #1 = Essential
                                           na.rm = TRUE)) |> round(2)
```

---

<table>
<caption>Items' CVR: First 6 rows</caption>
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:right;"> cvr.item1. </th>
   <th style="text-align:right;"> cvr.item2. </th>
   <th style="text-align:right;"> cvr.item3. </th>
   <th style="text-align:right;"> cvr.item4. </th>
   <th style="text-align:right;"> cvr.item5. </th>
   <th style="text-align:right;"> cvr.item6. </th>
   <th style="text-align:right;"> cvr.item7. </th>
   <th style="text-align:right;"> cvr.item8. </th>
   <th style="text-align:right;"> cvr.item9. </th>
   <th style="text-align:right;"> cvr.item10. </th>
   <th style="text-align:right;"> cvr.item11. </th>
   <th style="text-align:right;"> cvr.item12. </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 5 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 6 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 7 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
</tbody>
</table>

```
##  cvr.item1.  cvr.item2.  cvr.item3.  cvr.item4.  cvr.item5.  cvr.item6. 
##       -0.57        1.00        0.43        0.71       -0.86        0.71 
##  cvr.item7.  cvr.item8.  cvr.item9. cvr.item10. cvr.item11. cvr.item12. 
##        0.43       -0.14       -0.29        0.57       -0.14        0.43
```

---

<div class="pre-name">example-cvi.R</div>

``` r

kableExtra::kbl(head(ds[,item_att_cvi]), caption = "Items' CVI: First 6 rows")
i_cvi <- function(x) {
  n <- length(x)
  n_relevant <- sum(x >= 3)  # Count ratings 3 and 4 as relevant
  return(n_relevant / n)}
i_cvi_values <- apply(ds[,item_att_cvi], 2, i_cvi)
# Compute S-CVI
s_cvi_ave <- mean(i_cvi_values, na.rm = TRUE)
s_cvi_ua <- sum(i_cvi_values == 1, na.rm = TRUE) / length(i_cvi_values)
print(c("I-CVI: ", round(i_cvi_values,2)))
print(c("S-CVI/Ave: ",round(s_cvi_ave,2)))
print(c("S-CVI/UA: ",round(s_cvi_ua,2)))
```

---

<table>
<caption>Items' CVI: First 6 rows</caption>
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:right;"> cvi.item1. </th>
   <th style="text-align:right;"> cvi.item2. </th>
   <th style="text-align:right;"> cvi.item3. </th>
   <th style="text-align:right;"> cvi.item4. </th>
   <th style="text-align:right;"> cvi.item5. </th>
   <th style="text-align:right;"> cvi.item6. </th>
   <th style="text-align:right;"> cvi.item7. </th>
   <th style="text-align:right;"> cvi.item8. </th>
   <th style="text-align:right;"> cvi.item9. </th>
   <th style="text-align:right;"> cvi.item10. </th>
   <th style="text-align:right;"> cvi.item11. </th>
   <th style="text-align:right;"> cvi.item12. </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 4 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 5 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 6 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 7 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
</tbody>
</table>

```
##              cvi.item1.  cvi.item2.  cvi.item3.  cvi.item4.  cvi.item5. 
##   "I-CVI: "      "0.29"         "1"      "0.79"      "0.86"      "0.07" 
##  cvi.item6.  cvi.item7.  cvi.item8.  cvi.item9. cvi.item10. cvi.item11. 
##      "0.93"      "0.86"      "0.71"      "0.36"      "0.86"      "0.64" 
## cvi.item12. 
##      "0.71" 
## [1] "S-CVI/Ave: " "0.67"       
## [1] "S-CVI/UA: " "0.08"
```

---

# Evidence Sources

## Based on Content

What can be done after?

**Item Selection and Revision**

Based on the CVR scores and expert feedback, identify items that demonstrate high content validity. Items with low CVI scores may need revision or removal. Consider the experts’ comments and suggestions to refine and improve the wording, clarity, or coverage of the items.

**Pilot Testing**

Administer the revised measurement instrument to a small sample of respondents to gather data. Analyze the data to assess the appropriateness and relevance of the items. Examine response patterns, item means, and variability to ensure that the items effectively capture the intended content domain.

---

# Evidence Sources

## Based on Response Processes

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

**Response Process** &mdash; analysis of individual responses to stimuli; debriefing of examinees; process studies aimed at understanding what is measured and the soundness of intended score interpretations; quality assurance and quality control of assessment data.

---

# Evidence Sources

## Based on Response Processes

Examples (Downing and Yudkowsky, 2009, p. 30):

• Respondent format familiarity  
  • Quality control of electronic scanning/scoring  
  • Key validation of preliminary scores  
  • Accuracy in combining different item format scores  
  • Quality control/accuracy of final scores/marks/ grades  
  • Accuracy of applying pass–fail decision rules to scores  
  • Quality control of score reporting to respondents/organization  
  • Understandable/accurate descriptions/interpretations of scores for respondents

---

# Evidence Sources

## Based on Response Processes

Tourangeau and Rasinski (1988) suggest a four-step process:

• Interpretation of the question  
  • Retrieval of relevant information from memory  
  • Judgment formation  
  • Response execution  
  .transp[• Response editing (some include this 5<sup>th</sup> step)]

---

# Evidence Sources

## Based on Response Processes: Example

``` r
# Create a hypothetical dataset for transport satisfaction
ds <- data.frame(commuter_id = 1:500,
sat_with_punctuality = sample(1:5, 500, T, c(.1, .2, .3, .2, .2)), #5:very satisfied, 1:very dissatisfied
sat_with_comfort = sample(1:5, 500, T, c(.2, .3, .2, .2, .1)),#5:very satisfied, 1:very dissatisfied
sat_with_cost = sample(1:5, 500, T, c(.05, .15, .2, .3, .3)),#5:very satisfied, 1: very dissatisfied
will_to_recommend = sample(1:5, 500, replace = T),#5: highly recommend, 1:would not recommend
overall_rating = sample(c("good", "poor"), 500, replace = TRUE))#binary overall rating from commuter focus groups
# Key validation of preliminary scores
# Assuming a score above 3 in all categories is considered 'good'
ds$rating_check <- ifelse(ds$sat_with_punctuality > 3 &
                          ds$sat_with_comfort > 3 &
                          ds$sat_with_cost > 3 &
                          ds$will_to_recommend > 3, "good", "poor")
# Accuracy in combining different item format scores
# We can create a composite satisfaction score
ds$composite_satisfaction <- ds$sat_with_punctuality +
                             ds$sat_with_comfort +
                             ds$sat_with_cost +
                             ds$will_to_recommend
```

---

# Evidence Sources

## Based on Response Processes: Example

``` r
# Quality control/accuracy of final scores/marks/grades
# Check if the "good" or "poor" status matches with the actual scores
ds$accuracy_check <- ds$overall_rating == ds$rating_check

# Accuracy of applying decision rules to scores
accuracy <- mean(ds$accuracy_check)
print(paste("Accuracy of decision rule:", accuracy))
```

```
## [1] "Accuracy of decision rule: 0.528"
```

---

# Evidence Sources

## Based on Response Processes: Example

Other examples of response process evidence include:

• Think-aloud protocols  
  • Cognitive interviews  
  • Debriefing interviews  
  • Observations of examinees  
  • Observations of raters  
  • Observations of test administrators  
  • Observations of scorers  
  • Observations of item writers  
  • Observations of item reviewers

---

# Evidence Sources

## Based on Internal Structure

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

**Internal Structure** &mdash; data internal to assessments such as: reliability or reproducibility of scores; inter-item correlations; statistical characteristics of items; statistical analysis of item option function; factor studies of dimensionality; Differential Item Functioning (DIF) studies

---

# Evidence Sources

## Based on Internal Structure

Examples (Downing and Yudkowsky, 2009, p. 30):

• Item analysis data  
  • Item difficulty/discrimination  
  • Item Characteristic Curve (ICC)<sup>⚠️</sup> or Test Characteristic Curve (TCC)  
  • Inter-item correlations  
  • Item-total correlations  
  • Score scale reliability  
  • Standard errors of measurement `$(SEM)$`  
  • Subscore/subscale analyses  
  • Generalizability  
  • Dimensionality  
  • Item factor analysis  
  • Differential Item Functioning (DIF)  
  • Psychometric model

.footnote[<sup>⚠️</sup> not to be confused with the `\$ICC\$` used in Intraclass Correlation Coefficient.]

---

# Evidence Sources

## Based on Relations to other Variables

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

**Relations to Other Variables** &mdash; data external to assessments such as: correlations of assessment variable(s) to external, independent measures; hypothesis and theory driven investigations; correlational research based on previous studies, literature:  
  a. **Convergent and discriminant evidence**: relationships between similar and different measures  
  b. **Test-criterion evidence**: relationships between test and criterion measure(s)  
  c. **Validity generalization**: can the validity evidence be generalized?  Evidence that the validity studies may generalize to other settings.

---

# Evidence Sources

## Based on Relations to other Variables

### Convergent and Discriminant Evidence

**Convergent evidence**: intended to assess the same or similar constructs provide convergent evidence.

**Discriminant evidence**<sup>⚠️</sup>: relationships between test scores and measures purportedly of different constructs.

.footnote[<sup>⚠️</sup>
_Discriminant validity_ refers to how unrelated a measure is to constructs it shouldn't be associated with. However, many mistakenly use the term to describe a test's ability to differentiate between groups, which is actually a sign of convergent validity. The correct term for a test's ability to distinguish between groups is "discriminative validity" (Lilienfeld, Pydych, Lynn, Latzman, and Waldman, 2017).]

---

# Evidence Sources

## Based on Relations to other Variables

### Test-Criterion Evidence

**Concurrent**: A concurrent study obtains test scores and criterion<sup>💡</sup> information at about the same time.

**Predictive**: A predictive study indicates the strength of the relationship between test scores and criterion scores that are obtained at a later time.

.footnote[<sup>💡</sup>A _criterion variable_ is a measure of some attribute or outcome that is operationally distinct from the test. Thus, the test is not a measure of a criterion, but rather is a measure hypothesized as a potential predictor of that targeted criterion. (American Educational Research Association American Psychological Association et al., 2014).
]

---

# Evidence Sources

## Based on Relations to other Variables

### Validity Generalization

Validity generalization<sup>⚠️</sup> refers to how well study findings can be applied across different populations, settings, or times (American Educational Research Association American Psychological Association et al., 2014). Thus it is is a process of evaluating a test’s validity coefficients across a large set of studies (Schmidt, 1988).

Although single local validation studies can be imprecise, a well-conducted study with an adequate sample size may provide sufficient evidence to support or reject test use in new contexts. Comparing the value of local studies versus meta-analyses is crucial for robust conclusions.

.footnote[<sup>⚠️</sup>_External validity_ concerns the generalizability of results to other settings, while ecological validity, as a subtype, specifically examines whether a study design reflects natural settings (Lilienfeld Pydych et al., 2017; Andrade, 2018). A study can have high ecological validity yet still lack external validity if the findings don't apply to broader situations.

]

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

"Scientifically speaking, to 'make clear what something is' means to set forth the laws in which it occurs. We shall refer to the interlocking system of laws which constitute a theory as a _nomological network_." The laws in a nomological network may relate (a) observable properties or quantities to each other; or (b) theoretical constructs to observables; or (c) different theoretical constructs to one another. These “laws” may be statistical or deterministic

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network

Cronbach and Meehl (1955) defined a nomological network as the interlocking system of hypotheses, principles, and laws linking the constructs that constitute any theory.

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

A nomological network encompasses the theoretical constructs being measured, how the concept is going to be measured, and the specification of the interrelationships between the theoretical and empirical planes.

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

**nomological network** a conceptual network: a broadly integrative theoretical framework that identifies the key constructs associated with a phenomenon of interest and the associations among those constructs. For example, psychopathy is a complex notion involving a significant nomological network of knowledge and speculations about components, causes, correlates, and consequences as well as their interrelationships and means of measurement or evaluation.

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

Is the latent trait measured by the test different from the trait(s) measured by other tests?  
Is the latent trait measured related to other traits of interest by a statistical or psychometric theory?  
Are there expected differences in the latent trait for different manifest groups?  
Does the latent trait change over time?

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network

<div class="figure" style="text-align: center">
<img src="assets/img/nomological_network.svg" alt="The nomological network: 1:1 correspondence between theoretical constructs and measurements (which is not always the case). Reprinted from Masterson and Rainer
Jr. (2009)." width="26%" />
<p class="caption">The nomological network: 1:1 correspondence between theoretical constructs and measurements (which is not always the case). Reprinted from Masterson and Rainer
Jr. (2009).</p>
</div>

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network: Transport Satisfaction

To analyze a measure of **Transport Satisfaction (TSAT)**, we must examine its "laws" of interaction within the broader engineering system. We postulate a network where satisfaction is not an isolated metric but a predictable outcome of operational variables.

**Theoretical Predictions (The Network "Laws"):**

1.  **Efficiency Drivers:** We expect strong positive correlations with **Reliability (REL)** and **Punctuality (PUN)**. If the system is efficient, satisfaction *must* rise.
2.  **Comfort Factors:** We predict a negative correlation with **Crowding (CROW)** (as passenger density increases, satisfaction decreases) and a positive correlation with **Seat Availability (SEAT)**.
3.  **Service Quality:** **Cleanliness (CLEAN)** and **Information (INFO)** should act as "delighters," showing positive associations.
4.  **Cost Sensitivity:** We anticipate a weak negative relationship with **Cost (COST)**, assuming the user base is price-sensitive but prioritizes reliability.

If our data (e.g., from the *Punctuality* sensors or *Automated Passenger Counters*) contradicts these correlations, either our measure of "Satisfaction" is flawed, or our understanding of the passenger experience is incorrect.

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network: An example

<div class="figure">
<div class="grViz html-widget html-fill-item" id="htmlwidget-8367e1f57a78be76c316" style="width:100%;height:396px;"></div>
<script type="application/json" data-for="htmlwidget-8367e1f57a78be76c316">{"x":{"diagram":"\n  digraph G {\n    # Graph attributes\n    graph [bgcolor=transparent, layout = \"neato\", overlap = false]\n    \n    # Node attributes\n    node [fontname = \"Helvetica\", style = filled]\n    \n    # --- Central Construct ---\n    node [shape = circle, fixedsize = true, width = 2.2, height = 2.2, fillcolor = \"#4D8DC9\", fontcolor = white, fontsize = 16]\n    TSAT [label = \"Transport\nSatisfaction\n(TSAT)\", pos = \"0,0!\"]\n\n    # --- External Operational Variables ---\n    node [shape = circle, fixedsize = true, width = 1.6, height = 1.6, fillcolor = \"#f4f5f6\", fontcolor = \"#2f4858\", fontsize = 12]\n    \n    # Positive Correlations (Greenish zones)\n    REL   [label = \"Reliability\n(REL)\", pos = \"0, 3.5!\"]\n    PUN   [label = \"Punctuality\n(PUN)\", pos = \"2.5, 2.5!\"]\n    CLEAN [label = \"Cleanliness\n(CLEAN)\", pos = \"3.5, 0!\"]\n    SEAT  [label = \"Seat\nAvail.\n(SEAT)\", pos = \"2.5, -2.5!\"]\n    INFO  [label = \"Info\n(INFO)\", pos = \"1.5, -3.5!\"]\n    \n    # Negative Correlations (Reddish zones)\n    CROW  [label = \"Crowding\n(CROW)\", pos = \"-2.5, 2.5!\"]\n    COST  [label = \"Fare Cost\n(COST)\", pos = \"-3.5, 0!\"]\n    NOISE [label = \"Noise\n(NOISE)\", pos = \"-2.5, -2.5!\"]\n\n    # --- Edges (Relationships) ---\n    edge [penwidth = 2]\n\n    # Strong Positive (Thick Green)\n    TSAT -> REL   [dir = both, color = \"#44bc96\", label = \"+++\"]\n    TSAT -> PUN   [dir = both, color = \"#44bc96\", label = \"++\"]\n    TSAT -> CLEAN [dir = both, color = \"#44bc96\", label = \"++\"]\n    \n    # Moderate Positive (Green)\n    TSAT -> SEAT  [dir = both, color = \"#44bc96\", label = \"+\"]\n    TSAT -> INFO  [dir = both, color = \"#44bc96\", label = \"+\"]\n\n    # Negative (Red)\n    TSAT -> CROW  [dir = both, color = \"#d33f49\", label = \"--\"]\n    TSAT -> COST  [dir = both, color = \"#d33f49\", label = \"-\"]\n    TSAT -> NOISE [dir = both, color = \"#d33f49\", label = \"-\"]\n  }\n","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>
<p class="caption">Nomological network of transport satisfaction.</p>
</div>

---

# Evidence Sources

## Based on Relations to other Variables: Nomological Network

</center>
Nomological network of transport satisfaction (the snail plot can be seen in Mussel (2013)).

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network

**Considering a construct's place in a broader nomological network offers far more insight than pairwise comparisons.** Congruence assessment considers similarities and differences between pairs of variables relative to a network of other variables, not in isolation (Franke, Sarstedt, and Danks, 2021). High congruence, like high correlations:

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[
...challenge the proponent of distinct constructs to locate or create the circumstances under which the variables in question are distinguishable...

---

# Evidence Sources

## Based on Relations to other Variables

Examples (Downing and Yudkowsky, 2009, p. 30):

• Correlation with other relevant variables  
  • Convergent correlations—internal/external  
  • Similar tests  
  • Divergent correlations—internal/external  
  • Dissimilar measures  
  • Test-criterion correlations  
  • Generalizability of evidence

---

# Evidence Sources

## Based on Relations to other Variables

There are &mdash; at least &mdash; four main approaches to evaluate convergent and discriminant evidence, each differing in complexity, statistical rigor, age, and the need for explicit predictions (Furr, 2022). These approaches vary in their conceptual and statistical complexity, with some being well-established and others more recent innovations. Despite these differences, they all serve as valuable tools for assessing the validity evidence of measures:

1. **Focused Associations**: Associations between test scores and particularly important criteria.

2. **Sets of Correlations**: Associations between test scores and a wide range of relevant variables (usually evaluated subjectively via "eyeballing" the entire set of associations).

3. **Multitrait–Multimethod Matrices (MTMMM)** (Campbell and Fiske, 1959): Assocations between multiple constructs (traits), each of which is measured by multiple methods.

4. **Quantifying Construct Validity** (Westen and Rosenthal, 2003): Association between a _predicted_ pattern of convergent and discriminant correlations and the _observed_ pattern of correlations.

---

# Evidence Sources

## Based on Relations to other Variables

### Focused Associations

This type emphasizes the relevance of specific criterion variables to evaluate the validity of a test. It involves examining the correlation between test scores and key outcomes related to the construct being measured. For instance, a standardized college admission (e.g., INEM, SAT) test's validity evidence is closely tied to its correlation with college performance indicators, such as GPA. Strong correlations with these crucial variables support the test's validity, while weak correlations raise concerns about its effectiveness.

---

# Evidence Sources

## Based on Relations to other Variables: Focus Associations

Incremental explained variance of transport satisfaction relative to other predictors in the context of transport frequency prediction (the Venn plot can be seen in Mussel (2013)).

</center>

---

# Evidence Sources

## Based on Relations to other Variables

### Sets of Correlations

This approach involves examining a broader range of criterion variables to assess the validity evidence. Researchers compute correlations between the scores of the psychometric instrument of interest and various other measures to evaluate convergent and discriminant evidence. For example, a new measure of burnout might show expected associations with both similar and distinct psychological constructs, thereby confirming its validity through a comprehensive correlation analysis.

---

# Evidence Sources

## Based on Relations to other Variables

### Sets of Correlations: A Set of Radial Plots

<div class="figure">
<img src="assets/img/radial_cor_plot.svg" alt="Stets of correlations for empathyDuong, Hall, and Schwartz (2020)." width="40%" />
<p class="caption">Stets of correlations for empathyDuong, Hall, and Schwartz (2020).</p>
</div>

</center>

---

# Evidence Sources

## Based on Relations to other Variables

### Multitrait–Multimethod Matrices (MTMMM) (Campbell and Fiske, 1959)

This method provides a structured way to evaluate convergent and discriminant evidence It involves measuring multiple traits using various methods (self-report, peer ratings, etc.), allowing researchers to discern whether observed correlations are due to actual psychological constructs or influenced by the methods used. The MTMTM helps clarify the relationships among traits and the effects of different measurement techniques.

Four possibilities exist:

1. **Heterotrait–heteromethod** correlations: different constructs, and different methods (weakest correlations expected)
  2. **Heterotrait–monomethod** correlations: different constructs,and the same method (moderate correlations expected?).  
  3. **Monotrait–heteromethod** correlations: same construct, and different methods (moderate correlations expected?).  
  4. **Monotrait–monomethod** correlations: same construct, and same method (strongest correlations expected). Test-retest reliability is a special case of this type of correlation.

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV)

Westen and Rosenthal (2003) proposed a method called QCV. This procedure enables researchers to systematically assess the degree of 'fit' between:

a) their theoretical predictions regarding convergent and discriminant correlations; and,  
  (b) the actual correlations observed in their data.

It seeks to answer one key question:

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[
Does this measure predict an array of other measures in a way predicted by theory?
.tr[(Westen and Rosenthal, 2003, p. 609)
]]

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV)

The QCV process consists of three main steps:

1. **Elicit the predicted pattern of validity correlations**: Based on the literature formulate hypotheses about the expected relationships between the construct of interest and other relevant variables.  
  2. **Estimate the actual pattern of correlations**: Collect data and calculate the correlations between the construct of interest and other variables.  
  3. **Compare the predicted pattern with the actual pattern**: Evaluate the degree of match between the predicted and observed correlations to assess the validity evidence.

<div class="figure">
<div class="grViz html-widget html-fill-item" id="htmlwidget-59ae6712ff7a2d8e057b" style="width:100%;height:216px;"></div>
<script type="application/json" data-for="htmlwidget-59ae6712ff7a2d8e057b">{"x":{"diagram":"\ndigraph validity_comparison {\ngraph [bgcolor=transparent];  // Set background to transparent\n\n  # Graph attributes\n  graph [layout = dot, rankdir = LR]\n\n  # Node definitions with styles\n  node [shape = rectangle, style = filled, color = lightblue]\n  L [label = \"Review literature\"]\n  A [label = \"Elicit the predicted pattern\nof validity correlations\"]\n  B [label = \"Estimate\nactual pattern of correlations\"]\n  C [label = \"Compare predicted pattern\nwith actual pattern\"]\n\n  # Result node with different styling\n  node [shape = oval, style = filled, color = lightgrey]\n  D [label = \"Degree of match\"]\n\n  # Edges to connect nodes\n  L -> A [label = \"Formulate predictions\"]\n  A -> B [label = \"Gather data\"]\n  B -> C [label = \"Compare correlations\"]\n  C -> D [label = \"Evaluate match\"]\n\n  # Styling for edges\n  edge [color = darkgreen, penwidth = 2]\n}\n","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>
<p class="caption">The steps in the QCV process.</p>
</div>

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV) (Westen and Rosenthal, 2003)

QCV emphasizes that the theoretical basis of a construct guides the study and interpretation of validity evidence. However, interpretations of validity correlations have often relied on subjective assessments. For instance, researchers may 'eyeball' correlations and make subjective judgments about how well they align with theoretical expectations. Such reliance on subjective evaluations can lead to inconsistencies among researchers.

The QCV procedure is designed to provide a more precise and objective quantitative estimate of support from the overall pattern of evidence (Furr, 2022).

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV) (Westen and Rosenthal, 2003)

A relatively straightforward effect size is `$r_{alerting-CV}$` `$(r_a)$` \eqref{zalerting} the correlation between hypothesized correlations (or equivalently, `$\lambda$` values) and `$z$` transformed actual correlations (\$Z_r\$ values):

`$\lambda = r -\bar{r}$`

`$Z_r =\frac{1}{2}log_e\left(\frac{1+r}{1-r}\right)$`

\begin{align}
r_{ \lambda ,Z_r}=r_a \label{zalerting}
\end{align}

Large `$r_{alerting-CV}$` values ostensibly support a test's convergent and discriminant evidence (in terms of the intended construct).

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV) (Westen and Rosenthal, 2003)

Another effect size is the `$r_{contrast-CV}$`, it also includes the actual and hypothesized correlations, but it includes uses more information. First, it is necessary to calculate the "_remarkableness_ of size of contrast" \eqref{remarkableness} (Westen and Rosenthal, 2003). `$r_x$` is the median of the intercorrelations among the criterion measures.  
As an unstandardized and unfamiliar effect size, 'remarkableness' is not typically reported; however, for a given set of `$\lambda$` weights, larger positive values reflect greater correspondence between hypothesized and actual correlations (with a zero value representing no systematic correspondence) (Furr and Heuckeroth, 2019).

\begin{align}
remarkableness = \frac{\sum\left(\lambda Z_r\right)}{\sqrt{(\sum\lambda^2)\left(1-r_x\right)\left(\frac{1-\bar{r^2}\left(\frac{1-r_x}{2\left(1-\bar{r^2}\right)}\right)}{1-\bar{r^2}}\right)}} \label{remarkableness}
\end{align}

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV) (Westen and Rosenthal, 2003)

`$Remarkableness$` can be used to obtain an inferential test, where `$N$` is the number of participants on which the actual correlations are based \eqref{zcontrast}.

\begin{align}  
Z_{contrast} = remarkableness \sqrt{N-3} \label{zcontrast}  
\end{align}

Using this value, researchers obtain a `$p-value$` (one-sided, based on the standard normal distribution). A significant `$p-value$` as indicates that the 'prediction about the magnitude of correlations could not likely have been obtained by chance' Westen and Rosenthal (2003, p. 612).

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV) (Westen and Rosenthal, 2003)

Since `$remarkableness$` is an unstandardized effect size, Westen and Rosenthal (2003) provide `$r_{contrast-CV}$` as a standardized effect size on a familiar metric. To obtain `$r_{contrast-CV}$` \eqref{rcontrast} researchers identify the `$\mathcal{t}$` value associated with the `$p-value$` `$(df = N − 2)$`.<sup>💡</sup> Note that this `$\mathcal{t}$` value is not used for an inferential test; rather it is simply a step through which researchers move from a `$p-value$` to a standardized effect size.

\begin{align}
r_{contrast-CV} = \sqrt{\frac{t^2}{t^2+df}} \label{rcontrast}
\end{align}

``` r
n_size <- 90; p_value <- .0000000000002762477 #example
qt(p = p_value, df = n_size-2, lower.tail = F)
```

```
## [1] 8.451081
```

]

---

# Evidence Sources

## Based on Relations to other Variables

### An example using the `qcv` package

In <svg viewBox="0 0 581 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;" xmlns="http://www.w3.org/2000/svg">  <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"></path></svg> the [`qcv`]([https://cran.r-project.org/web/packages/qcv/index.html](https://cran.r-project.org/web/packages/qcv/index.html)) package (Furr and Heuckeroth, 2019) provides a straightforward way to implement the QCV procedure. Here's an example with hypothetical data on transport satisfaction.

`n` &mdash; sample size

`actr` &mdash; vector of actual validity correlations

`predr` &mdash; vector of predicted validity correlations (in same order as actr)

`medr` &mdash; median intercorrelation among criterion variables

<div class="pre-name">example-qcv.R</div>

``` r
# Names: PUN, COM, CROW, COST, FREQ, SAFE, SEC, ACC, CLEAN, INFO, NOISE, REL
actrTransport  <- c(.46, .13, -.24, -.03, .12, .03, .39, .06, .51, .08, .24, .66)
predrTransport <- c(.58, .24, -.04, .06, -.04, .18, .36, .08, .64, .56, .36, .56)
library(qcv)
qcv(n=90, actr=actrTransport, predr=predrTransport, medr=.075)
```

???

Example by "hand" using the formulae

``` r
hip_cor <- c(.64,.58, .56, .56, .36, .36, .24, .18, .08, .06, -.04, -.04)

lambdas <- scale(hip_cor, center = T, scale = F)

act_cor <- c(.51,.46,.66,.08,.39,.24,.13,.03,.06,-.03,-.24,.12)

zs <- (1/2)*log((1+act_cor)/(1-act_cor))

lamb_zs <- lambdas*zs

#r_alerting_CV
#$\lambda = r -\bar{r};Z_r =\frac{1}{2}log_e\left(\frac{1+r}{1-r}\right); r_{alerting-CV}=r_{\lambda, Z_r}$
(r_alerting_CV <- cor(lambdas, zs))
```

```
##           [,1]
## [1,] 0.7887882
```

``` r
#r_contrast_CV

r_x <- .075 #median inter-correlation value among the convergent/divergent measures (not including the measure of interest) in this case 12 variables (raw data not provided)

#$remarkableness = \frac{\sum\left(\lambda Z_r\right)}{\sqrt{(\sum\lambda^2)\left(1-r_x\right)\left(\frac{1-\bar{r^2}\left(\frac{1-r_x}{2\left(1-\bar{r^2}\right)}\right)}{1-\bar{r^2}}\right)}}$

remarkableness <- sum(lambdas*zs)/sqrt(sum(lambdas^2)*(1-r_x)*((1-mean(act_cor^2)*((1-r_x)/(2*(1-mean(act_cor^2)))))/(1-mean(act_cor^2))))

#z_contrast
#$Z_contrast = remarkableness \sqrt{N-3}$

n_size <- 90
z_contrast <- remarkableness*sqrt(n_size-3)

#obtaining the one-sided p-value
(p_value <- pnorm(z_contrast, lower.tail = F))
```

```
## [1] 0.0000000000002762477
```

``` r
#obtaining the t-value based on p-value and degrees of freedom
(t_value <- qt(p = p_value, df = n_size-2, lower.tail = F))
```

```
## [1] 8.451081
```

``` r
#obtaining the effect size (r_{contrast-cv})
#$r_{contrast-CV} = \sqrt{\frac{t^2}{t^2+df}}$
(r_contrast_CV <- sqrt((t_value^2)/(t_value^2 + n_size-2)))
```

```
## [1] 0.6693292
```

---

# Evidence Sources

## Based on Relations to other Variables

### An example using the `qcv` package

```
## ************  QCV output  ******************************* 
## ********************************************************* 
## 
## ralerting-CV: 0.7888 
## rcontrast-CV: 0.6693         95% CI = 0.5366  to  0.7697 
## zcontrast:    7.2117 
## p:            0.000000000000276247689097745 
## N:            90 
## k:            12 
## szr:          0.2795 
## medr:         0.075 
## rbarsq:       0.1015 
## t:            8.4511 
## remark:       0.7732
```

The data shows substantial correspondence between our theory of the construct and its empirical correlates, thus good convergent and discriminant evidence.

---

# Evidence Sources

## Based on Relations to other Variables

### QCV using SEM

- **QCV Focus**: Targets a small subset of the correlation matrix between a focal test and criterion tests, emphasizing the evaluation of convergent and discriminant evidence.  
  - **Structural Equation Modeling (SEM)**: Offers a broader, flexible framework for analyzing entire correlation matrices, providing fit indices and inferential tests similar to QCV.
  - **SEM Advantages**:  
    - Widely recognized methods and indices.  
    - Expanded analytical possibilities, such as multi-group evaluation of convergent and discriminant validity.  
  - **Potential Integration**: QCV analyses could be subsumed into SEM, potentially eliminating the need for `$r_{alerting-CV}$` and `$r_{contrast-CV}$` tests.

---

# Evidence Sources

## Based on Relations to other Variables

### QCV using SEM

- **Model Specification**: Use SEM software (e.g., R, M<i>plus</i>) to compare a hypothesized correlation matrix (convergent/discriminant validity) with the actual observed matrix.  
    - **Focal Test’s Correlations**: Constrain these to hypothesized values to assess validity.  
  - **Criterion Tests’ Correlations**: Constrain these to observed values to isolate errors in focal test hypotheses.  
  - **Fit Indices**: The model will yield fit indices like `$TLI$`, `$CFI$`, `$SRMR$`, `$RMSEA$`, and `$\chi^2$`, which reflect how well the hypothesized matrix fits the observed data.

---

# Evidence Sources

## Based on Relations to other Variables

### QCV using SEM

``` mplus
TITLE: QCV-like procedure via Mplus (Transport Satisfaction Data);

DATA: 
  FILE IS /data/ubuntu/Disciplines/cpae/assets/data/qcvexample4mplus.dat;
  TYPE IS INDIVIDUAL;

VARIABLE:
  ! Rename the variables from the raw file to these new names
  NAMES = TSAT PUN COM CROW COST FREQ SAFE SEC ACC CLEAN INFO NOISE REL;
  USEVARIABLES = TSAT PUN COM CROW COST FREQ SAFE SEC ACC CLEAN INFO NOISE REL;

DEFINE:
  STANDARDIZE TSAT PUN COM CROW COST FREQ SAFE SEC ACC CLEAN INFO NOISE REL;

ANALYSIS:
  TYPE = GENERAL;
  ESTIMATOR = ML;

MODEL:
  !predicted corrs bw focal test (TSAT) and criterion variables
  TSAT WITH PUN@.58;
  TSAT WITH COM@.24;
  TSAT WITH CROW@-.04;
  TSAT WITH COST@.06;
  TSAT WITH FREQ@-.04;
  TSAT WITH SAFE@0.18;
  TSAT WITH SEC@0.36;
  TSAT WITH ACC@0.08;
  TSAT WITH CLEAN@0.64;
  TSAT WITH INFO; !@.56; !that this constraint was freed due to non-convergence
  TSAT WITH NOISE@0.36;
  TSAT WITH REL@0.56;

! Actual correlations among criterion variables
  PUN WITH COM@-.02;
  PUN WITH CROW@-.42;
  PUN WITH COST@-.02;
  PUN WITH FREQ@0.13;
  PUN WITH SAFE@0.08;
  PUN WITH SEC@0.53;
  PUN WITH ACC@0.11;
  PUN WITH CLEAN@0.34;
  PUN WITH INFO@-0.07;
  PUN WITH NOISE@-0.08;
  PUN WITH REL@0.50;
  
  COM WITH CROW@.14;
  COM WITH COST@.32;
  COM WITH FREQ@0.31;
  COM WITH SAFE@0.22;
  COM WITH SEC@-0.01;
  COM WITH ACC@0.22;
  COM WITH CLEAN@-0.16;
  COM WITH INFO@0.40;
  COM WITH NOISE@-0.01;
  COM WITH REL@-0.08;
  
  CROW WITH COST@.07;
  CROW WITH FREQ@0.03;
  CROW WITH SAFE@-0.12;
  CROW WITH SEC@-0.46;
  CROW WITH ACC@0.02;
  CROW WITH CLEAN@-0.17;
  CROW WITH INFO@0.10;
  CROW WITH NOISE@0.17;
  CROW WITH REL@-0.30;
  
  COST WITH FREQ@0.52;
  COST WITH SAFE@0.34;
  COST WITH SEC@0.05;
  COST WITH ACC@0.26;
  COST WITH CLEAN@-0.46;
  COST WITH INFO@0.28;
  COST WITH NOISE@-0.44;
  COST WITH REL@-0.12;
  
  FREQ WITH SAFE@0.21;
  FREQ WITH SEC@0.22;
  FREQ WITH ACC@0.46;
  FREQ WITH CLEAN@-0.12;
  FREQ WITH INFO@0.15;
  FREQ WITH NOISE@-0.16;
  FREQ WITH REL@-0.06;
  
  SAFE WITH SEC@0.34;
  SAFE WITH ACC@0.40;
  SAFE WITH CLEAN@-0.17;
  SAFE WITH INFO@0.56;
  SAFE WITH NOISE@-0.04;
  SAFE WITH REL@0.10;
  
  SEC WITH ACC@0.45;
  SEC WITH CLEAN@0.26;
  SEC WITH INFO@0.03;
  SEC WITH NOISE@0.00;
  SEC WITH REL@0.39;
  
  ACC WITH CLEAN@-0.02;
  ACC WITH INFO@0.26;
  ACC WITH NOISE@-0.19;
  ACC WITH REL@-0.06;
  
  CLEAN WITH INFO@-0.14;
  CLEAN WITH NOISE@0.55;
  CLEAN WITH REL@0.60;
  
  INFO WITH NOISE@-0.09;
  INFO WITH REL@0.03;
  
  NOISE WITH REL@0.35;

OUTPUT: RES SAMPSTAT MODINDICES;
```

]

---

# Evidence Sources

## Based on Relations to other Variables

### QCV using SEM

```
## Mplus VERSION 8.11 (Linux)
## MUTHEN & MUTHEN
## 08/12/2026   6:26 PM
## 
## INPUT INSTRUCTIONS
## 
##   TITLE: QCV-like procedure via Mplus (Transport Satisfaction Data);
## 
##   DATA:
##     FILE IS /data/ubuntu/Disciplines/cpae/assets/data/qcvexample4mplus.dat;
##     TYPE IS INDIVIDUAL;
## 
##   VARIABLE:
##     ! Rename the variables from the raw file to these new names
##     NAMES = TSAT PUN COM CROW COST FREQ SAFE SEC ACC CLEAN INFO NOISE REL;
##     USEVARIABLES = TSAT PUN COM CROW COST FREQ SAFE SEC ACC CLEAN INFO NOISE REL;
## 
##   DEFINE:
##     STANDARDIZE TSAT PUN COM CROW COST FREQ SAFE SEC ACC CLEAN INFO NOISE REL;
## 
##   ANALYSIS:
##     TYPE = GENERAL;
##     ESTIMATOR = ML;
## 
##   MODEL:
##     !predicted corrs bw focal test (TSAT) and criterion variables
##     TSAT WITH PUN@.58;
##     TSAT WITH COM@.24;
##     TSAT WITH CROW@-.04;
##     TSAT WITH COST@.06;
##     TSAT WITH FREQ@-.04;
##     TSAT WITH SAFE@0.18;
##     TSAT WITH SEC@0.36;
##     TSAT WITH ACC@0.08;
##     TSAT WITH CLEAN@0.64;
##     TSAT WITH INFO; !@.56; !that this constraint was freed due to non-convergence
##     TSAT WITH NOISE@0.36;
##     TSAT WITH REL@0.56;
## 
##     ! Actual correlations among criterion variables
##     PUN WITH COM@-.02;
##     PUN WITH CROW@-.42;
##     PUN WITH COST@-.02;
##     PUN WITH FREQ@0.13;
##     PUN WITH SAFE@0.08;
##     PUN WITH SEC@0.53;
##     PUN WITH ACC@0.11;
##     PUN WITH CLEAN@0.34;
##     PUN WITH INFO@-0.07;
##     PUN WITH NOISE@-0.08;
##     PUN WITH REL@0.50;
## 
##     COM WITH CROW@.14;
##     COM WITH COST@.32;
##     COM WITH FREQ@0.31;
##     COM WITH SAFE@0.22;
##     COM WITH SEC@-0.01;
##     COM WITH ACC@0.22;
##     COM WITH CLEAN@-0.16;
##     COM WITH INFO@0.40;
##     COM WITH NOISE@-0.01;
##     COM WITH REL@-0.08;
## 
##     CROW WITH COST@.07;
##     CROW WITH FREQ@0.03;
##     CROW WITH SAFE@-0.12;
##     CROW WITH SEC@-0.46;
##     CROW WITH ACC@0.02;
##     CROW WITH CLEAN@-0.17;
##     CROW WITH INFO@0.10;
##     CROW WITH NOISE@0.17;
##     CROW WITH REL@-0.30;
## 
##     COST WITH FREQ@0.52;
##     COST WITH SAFE@0.34;
##     COST WITH SEC@0.05;
##     COST WITH ACC@0.26;
##     COST WITH CLEAN@-0.46;
##     COST WITH INFO@0.28;
##     COST WITH NOISE@-0.44;
##     COST WITH REL@-0.12;
## 
##     FREQ WITH SAFE@0.21;
##     FREQ WITH SEC@0.22;
##     FREQ WITH ACC@0.46;
##     FREQ WITH CLEAN@-0.12;
##     FREQ WITH INFO@0.15;
##     FREQ WITH NOISE@-0.16;
##     FREQ WITH REL@-0.06;
## 
##     SAFE WITH SEC@0.34;
##     SAFE WITH ACC@0.40;
##     SAFE WITH CLEAN@-0.17;
##     SAFE WITH INFO@0.56;
##     SAFE WITH NOISE@-0.04;
##     SAFE WITH REL@0.10;
## 
##     SEC WITH ACC@0.45;
##     SEC WITH CLEAN@0.26;
##     SEC WITH INFO@0.03;
##     SEC WITH NOISE@0.00;
##     SEC WITH REL@0.39;
## 
##     ACC WITH CLEAN@-0.02;
##     ACC WITH INFO@0.26;
##     ACC WITH NOISE@-0.19;
##     ACC WITH REL@-0.06;
## 
##     CLEAN WITH INFO@-0.14;
##     CLEAN WITH NOISE@0.55;
##     CLEAN WITH REL@0.60;
## 
##     INFO WITH NOISE@-0.09;
##     INFO WITH REL@0.03;
## 
##     NOISE WITH REL@0.35;
## 
##   OUTPUT: RES SAMPSTAT MODINDICES;
## 
## 
## 
## INPUT READING TERMINATED NORMALLY
## 
## 
## 
## QCV-like procedure via Mplus (Transport Satisfaction Data);
## 
## SUMMARY OF ANALYSIS
## 
## Number of groups                                                 1
## Number of observations                                          90
## 
## Number of dependent variables                                   13
## Number of independent variables                                  0
## Number of continuous latent variables                            0
## 
## Observed dependent variables
## 
##   Continuous
##    TSAT        PUN         COM         CROW        COST        FREQ
##    SAFE        SEC         ACC         CLEAN       INFO        NOISE
##    REL
## 
## 
## Estimator                                                       ML
## Information matrix                                        OBSERVED
## Maximum number of iterations                                  1000
## Convergence criterion                                    0.500D-04
## Maximum number of steepest descent iterations                   20
## 
## Input data file(s)
##   /data/ubuntu/Disciplines/cpae/assets/data/qcvexample4mplus.dat
## 
## Input data format  FREE
## 
## 
## SAMPLE STATISTICS
## 
## 
##      SAMPLE STATISTICS
## 
## 
##            Means
##               TSAT          PUN           COM           CROW          COST
##               ________      ________      ________      ________      ________
##                 0.000         0.000         0.000         0.000         0.000
## 
## 
##            Means
##               FREQ          SAFE          SEC           ACC           CLEAN
##               ________      ________      ________      ________      ________
##                 0.000         0.000         0.000         0.000         0.000
## 
## 
##            Means
##               INFO          NOISE         REL
##               ________      ________      ________
##                 0.000         0.000         0.000
## 
## 
##            Covariances
##               TSAT          PUN           COM           CROW          COST
##               ________      ________      ________      ________      ________
##  TSAT           1.000
##  PUN            0.464         1.000
##  COM            0.148         0.016         1.000
##  CROW          -0.273        -0.477         0.141         1.000
##  COST           0.016         0.050         0.431        -0.028         1.000
##  FREQ           0.139         0.273         0.285        -0.100         0.470
##  SAFE          -0.039         0.058         0.320        -0.112         0.338
##  SEC            0.327         0.523         0.106        -0.521         0.139
##  ACC            0.007         0.168         0.135         0.057         0.255
##  CLEAN          0.459         0.364        -0.301        -0.198        -0.414
##  INFO          -0.017        -0.092         0.424         0.058         0.255
##  NOISE          0.142        -0.162        -0.175         0.220        -0.415
##  REL            0.608         0.408        -0.119        -0.183         0.018
## 
## 
##            Covariances
##               FREQ          SAFE          SEC           ACC           CLEAN
##               ________      ________      ________      ________      ________
##  FREQ           1.000
##  SAFE           0.095         1.000
##  SEC            0.307         0.332         1.000
##  ACC            0.443         0.292         0.384         1.000
##  CLEAN         -0.056        -0.206         0.171         0.077         1.000
##  INFO           0.118         0.553         0.080         0.200        -0.207
##  NOISE         -0.204        -0.044        -0.168        -0.321         0.435
##  REL            0.021         0.085         0.280        -0.005         0.550
## 
## 
##            Covariances
##               INFO          NOISE         REL
##               ________      ________      ________
##  INFO           1.000
##  NOISE         -0.101         1.000
##  REL           -0.055         0.358         1.000
## 
## 
##            Correlations
##               TSAT          PUN           COM           CROW          COST
##               ________      ________      ________      ________      ________
##  TSAT           1.000
##  PUN            0.464         1.000
##  COM            0.148         0.016         1.000
##  CROW          -0.273        -0.477         0.141         1.000
##  COST           0.016         0.050         0.431        -0.028         1.000
##  FREQ           0.139         0.273         0.285        -0.100         0.470
##  SAFE          -0.039         0.058         0.320        -0.112         0.338
##  SEC            0.327         0.523         0.106        -0.521         0.139
##  ACC            0.007         0.168         0.135         0.057         0.255
##  CLEAN          0.459         0.364        -0.301        -0.198        -0.414
##  INFO          -0.017        -0.092         0.424         0.058         0.255
##  NOISE          0.142        -0.162        -0.175         0.220        -0.415
##  REL            0.608         0.408        -0.119        -0.183         0.018
## 
## 
##            Correlations
##               FREQ          SAFE          SEC           ACC           CLEAN
##               ________      ________      ________      ________      ________
##  FREQ           1.000
##  SAFE           0.095         1.000
##  SEC            0.307         0.332         1.000
##  ACC            0.443         0.292         0.384         1.000
##  CLEAN         -0.056        -0.206         0.171         0.077         1.000
##  INFO           0.118         0.553         0.080         0.200        -0.207
##  NOISE         -0.204        -0.044        -0.168        -0.321         0.435
##  REL            0.021         0.085         0.280        -0.005         0.550
## 
## 
##            Correlations
##               INFO          NOISE         REL
##               ________      ________      ________
##  INFO           1.000
##  NOISE         -0.101         1.000
##  REL           -0.055         0.358         1.000
## 
## 
## UNIVARIATE SAMPLE STATISTICS
## 
## 
##      UNIVARIATE HIGHER-ORDER MOMENT DESCRIPTIVE STATISTICS
## 
##          Variable/         Mean/     Skewness/   Minimum/ % with                Percentiles
##         Sample Size      Variance    Kurtosis    Maximum  Min/Max      20%/60%    40%/80%    Median
## 
##      TSAT                  0.000       0.259      -2.540    1.11%      -0.880     -0.376     -0.181
##               90.000       1.000      -0.205       2.519    1.11%       0.258      0.781
##      PUN                   0.000       0.055      -2.200    1.11%      -0.755     -0.327     -0.067
##               90.000       1.000      -0.416       2.208    1.11%       0.220      0.963
##      COM                   0.000      -0.002      -2.383    1.11%      -0.922     -0.365     -0.009
##               90.000       1.000      -0.706       2.187    1.11%       0.148      1.044
##      CROW                  0.000       0.142      -2.230    1.11%      -0.957     -0.255     -0.118
##               90.000       1.000      -0.702       2.195    1.11%       0.146      0.921
##      COST                  0.000      -0.264      -2.828    1.11%      -1.040     -0.204      0.166
##               90.000       1.000      -0.394       2.108    1.11%       0.297      0.815
##      FREQ                  0.000       0.180      -2.280    1.11%      -0.936     -0.237     -0.037
##               90.000       1.000      -0.233       2.363    1.11%       0.260      0.716
##      SAFE                  0.000      -0.233      -2.772    1.11%      -0.769     -0.161      0.072
##               90.000       1.000       0.220       2.396    1.11%       0.284      0.816
##      SEC                   0.000       0.274      -2.220    1.11%      -0.807     -0.281     -0.079
##               90.000       1.000       0.197       2.552    1.11%       0.181      0.868
##      ACC                   0.000      -0.323      -2.735    1.11%      -0.837     -0.188      0.070
##               90.000       1.000      -0.011       2.692    1.11%       0.366      0.790
##      CLEAN                 0.000       0.081      -2.575    1.11%      -0.914     -0.235      0.043
##               90.000       1.000       0.021       2.787    1.11%       0.234      0.752
##      INFO                  0.000      -0.253      -2.183    1.11%      -0.879     -0.206     -0.031
##               90.000       1.000      -0.551       1.899    1.11%       0.257      0.883
##      NOISE                 0.000       0.216      -2.056    1.11%      -0.914     -0.318     -0.129
##               90.000       1.000      -0.371       2.743    1.11%       0.101      0.937
##      REL                   0.000       0.117      -2.389    1.11%      -0.887     -0.244      0.077
##               90.000       1.000       0.311       3.109    1.11%       0.285      0.842
## 
## 
##      NO CONVERGENCE.  SERIOUS PROBLEMS IN ITERATIONS.
##      ESTIMATED COVARIANCE MATRIX NON-INVERTIBLE.
##      CHECK YOUR STARTING VALUES.
## 
## 
## 
## MODEL RESULTS
## 
##                     Estimate
## 
##  TSAT     WITH
##     PUN                0.580
##     COM                0.240
##     CROW              -0.040
*##     COST               0.060
*##     FREQ              -0.040
*##     SAFE               0.180
*##     SEC                0.360
*##     ACC                0.080
*##     CLEAN              0.640
*##     INFO               0.000
*##     NOISE              0.360
*##     REL                0.560
*## 
*##  PUN      WITH
*##     COM               -0.020
*##     CROW              -0.420
*##     COST              -0.020
*##     FREQ               0.130
*##     SAFE               0.080
*##     SEC                0.530
*##     ACC                0.110
*##     CLEAN              0.340
*##     INFO              -0.070
*##     NOISE             -0.080
*##     REL                0.500
*## 
*##  COM      WITH
*##     CROW               0.140
*##     COST               0.320
##     FREQ               0.310
##     SAFE               0.220
##     SEC               -0.010
##     ACC                0.220
##     CLEAN             -0.160
##     INFO               0.400
##     NOISE             -0.010
##     REL               -0.080
## 
##  CROW     WITH
##     COST               0.070
##     FREQ               0.030
##     SAFE              -0.120
##     SEC               -0.460
##     ACC                0.020
##     CLEAN             -0.170
##     INFO               0.100
##     NOISE              0.170
##     REL               -0.300
## 
##  COST     WITH
##     FREQ               0.520
##     SAFE               0.340
##     SEC                0.050
##     ACC                0.260
##     CLEAN             -0.460
##     INFO               0.280
##     NOISE             -0.440
##     REL               -0.120
## 
##  FREQ     WITH
##     SAFE               0.210
##     SEC                0.220
##     ACC                0.460
##     CLEAN             -0.120
##     INFO               0.150
##     NOISE             -0.160
##     REL               -0.060
## 
##  SAFE     WITH
##     SEC                0.340
##     ACC                0.400
##     CLEAN             -0.170
##     INFO               0.560
##     NOISE             -0.040
##     REL                0.100
## 
##  SEC      WITH
##     ACC                0.450
##     CLEAN              0.260
##     INFO               0.030
##     NOISE              0.000
##     REL                0.390
## 
##  ACC      WITH
##     CLEAN             -0.020
##     INFO               0.260
##     NOISE             -0.190
##     REL               -0.060
## 
##  CLEAN    WITH
##     INFO              -0.140
##     NOISE              0.550
##     REL                0.600
## 
##  INFO     WITH
##     NOISE             -0.090
##     REL                0.030
## 
##  NOISE    WITH
##     REL                0.350
## 
##  Means
##     TSAT               0.000
##     PUN                0.000
##     COM                0.000
##     CROW               0.000
##     COST               0.000
##     FREQ               0.000
##     SAFE               0.000
##     SEC                0.000
##     ACC                0.000
##     CLEAN              0.000
##     INFO               0.000
##     NOISE              0.000
##     REL                0.000
## 
##  Variances
##     TSAT               0.500
##     PUN                0.500
##     COM                0.500
##     CROW               0.500
##     COST               0.500
##     FREQ               0.500
##     SAFE               0.500
##     SEC                0.500
##     ACC                0.500
##     CLEAN              0.500
##     INFO               0.500
##     NOISE              0.500
##     REL                0.500
## 
## 
## MODEL COMMAND WITH FINAL ESTIMATES USED AS STARTING VALUES
## 
##      tsat WITH pun@0.58000;
##      tsat WITH com@0.24000;
##      tsat WITH crow@-0.04000;
##      tsat WITH cost@0.06000;
##      tsat WITH freq@-0.04000;
##      tsat WITH safe@0.18000;
##      tsat WITH sec@0.36000;
##      tsat WITH acc@0.08000;
##      tsat WITH clean@0.64000;
##      tsat WITH info*0;
##      tsat WITH noise@0.36000;
##      tsat WITH rel@0.56000;
##      pun WITH com@-0.02000;
##      pun WITH crow@-0.42000;
##      pun WITH cost@-0.02000;
##      pun WITH freq@0.13000;
##      pun WITH safe@0.08000;
##      pun WITH sec@0.53000;
##      pun WITH acc@0.11000;
##      pun WITH clean@0.34000;
##      pun WITH info@-0.07000;
##      pun WITH noise@-0.08000;
##      pun WITH rel@0.50000;
##      com WITH crow@0.14000;
##      com WITH cost@0.32000;
##      com WITH freq@0.31000;
##      com WITH safe@0.22000;
##      com WITH sec@-0.01000;
##      com WITH acc@0.22000;
##      com WITH clean@-0.16000;
##      com WITH info@0.40000;
##      com WITH noise@-0.01000;
##      com WITH rel@-0.08000;
##      crow WITH cost@0.07000;
##      crow WITH freq@0.03000;
##      crow WITH safe@-0.12000;
##      crow WITH sec@-0.46000;
##      crow WITH acc@0.02000;
##      crow WITH clean@-0.17000;
##      crow WITH info@0.10000;
##      crow WITH noise@0.17000;
##      crow WITH rel@-0.30000;
##      cost WITH freq@0.52000;
##      cost WITH safe@0.34000;
##      cost WITH sec@0.05000;
##      cost WITH acc@0.26000;
##      cost WITH clean@-0.46000;
##      cost WITH info@0.28000;
##      cost WITH noise@-0.44000;
##      cost WITH rel@-0.12000;
##      freq WITH safe@0.21000;
##      freq WITH sec@0.22000;
##      freq WITH acc@0.46000;
##      freq WITH clean@-0.12000;
##      freq WITH info@0.15000;
##      freq WITH noise@-0.16000;
##      freq WITH rel@-0.06000;
##      safe WITH sec@0.34000;
##      safe WITH acc@0.40000;
##      safe WITH clean@-0.17000;
##      safe WITH info@0.56000;
##      safe WITH noise@-0.04000;
##      safe WITH rel@0.10000;
##      sec WITH acc@0.45000;
##      sec WITH clean@0.26000;
##      sec WITH info@0.03000;
##      sec WITH noise@0;
##      sec WITH rel@0.39000;
##      acc WITH clean@-0.02000;
##      acc WITH info@0.26000;
##      acc WITH noise@-0.19000;
##      acc WITH rel@-0.06000;
##      clean WITH info@-0.14000;
##      clean WITH noise@0.55000;
##      clean WITH rel@0.60000;
##      info WITH noise@-0.09000;
##      info WITH rel@0.03000;
##      noise WITH rel@0.35000;
## 
##      [ tsat*0 ];
##      [ pun*0 ];
##      [ com*0 ];
##      [ crow*0 ];
##      [ cost*0 ];
##      [ freq*0 ];
##      [ safe*0 ];
##      [ sec*0 ];
##      [ acc*0 ];
##      [ clean*0 ];
##      [ info*0 ];
##      [ noise*0 ];
##      [ rel*0 ];
## 
##      tsat*0.50000;
##      pun*0.50000;
##      com*0.50000;
##      crow*0.50000;
##      cost*0.50000;
##      freq*0.50000;
##      safe*0.50000;
##      sec*0.50000;
##      acc*0.50000;
##      clean*0.50000;
##      info*0.50000;
##      noise*0.50000;
##      rel*0.50000;
## 
## 
## 
##      Beginning Time:  18:26:43
##         Ending Time:  18:26:44
##        Elapsed Time:  00:00:01
## 
## 
## 
## MUTHEN & MUTHEN
## 3463 Stoner Ave.
## Los Angeles, CA  90066
## 
## Tel: (310) 391-9971
## Fax: (310) 391-8971
## Web: www.StatModel.com
## Support: Support@StatModel.com
## 
## Copyright (c) 1998-2024 Muthen & Muthen
```

]

---

# Evidence Sources

## Based on Consequences of Testing

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

**Evidence Based on Consequences of Testing** &mdash; intended and unintended consequences of test use; differential consequences of test use; impact of assessment on respondents, instructors, organizations, society; impact of assessments on curriculum; cost/benefit analysis with respect to tradeoff between work/instructional time and assessment time

---

# Evidence Sources

## Based on Consequences of Testing

Examples (Downing and Yudkowsky, 2009, p. 30):

• Impact of test scores/results on respondents/society  
  • Consequences on learners/future learning  
  • Positive consequences outweigh unintended negative consequences?  
  • Reasonableness of method of establishing pass–fail (cut) score  
  • Pass/fail consequences  
  • P/F Decision reliability—Classification accuracy  
  • Conditional Standard Error of Measurement at pass score `$(CSEM)$` (Huebner and Skar, 2021)  
  • False positives/negatives  
  • Instructional/learner consequences

---

# References    
American Educational Research Association, American Psychological
Association, et al. (2014). _Standards for Educational and
Psychological Testing_. Washington, DC: American Psychological
Association. ISBN: 9780935302356.

Andrade, C. (2018). "Internal, external, and ecological validity in
research design, conduct, and evaluation". In: _Indian Journal of
Psychological Medicine_ 40 (5), pp. 498-499. ISSN: 0253-7176. DOI:
[10.4103/IJPSYM.IJPSYM_334_18](https://doi.org/10.4103%2FIJPSYM.IJPSYM_334_18).
URL:
[https://journals.sagepub.com/doi/10.4103/IJPSYM.IJPSYM_334_18](https://journals.sagepub.com/doi/10.4103/IJPSYM.IJPSYM_334_18).

Ayre, C. and A. J. Scally (2014). "Critical values for Lawshe's Content
Validity Ratio". In: _Measurement and Evaluation in Counseling and
Development_ 47.1, pp. 79-86. ISSN: 0748-1756. DOI:
[10.1177/0748175613513808](https://doi.org/10.1177%2F0748175613513808).
URL:
[https://www.tandfonline.com/doi/full/10.1177/0748175613513808](https://www.tandfonline.com/doi/full/10.1177/0748175613513808).

Beckman, T. J., D. A. Cook, et al. (2005). "What is the validity
evidence for assessments of clinical teaching?" In: _Journal of General
Internal Medicine_ 20 (12), pp. 1159-1164. ISSN: 08848734. DOI:
[10.1111/j.1525-1497.2005.0258.x](https://doi.org/10.1111%2Fj.1525-1497.2005.0258.x).

Buckingham, B. R., W. A. McCall, et al. (1921). "Report of the
Standardization Committee". In: _Journal of Educational Research_ 4.1,
pp. 78-80. ISSN: 00220671, 19400675. URL:
[http://www.jstor.org/stable/27524508](http://www.jstor.org/stable/27524508).

---
# References    
Campbell, D. T. and D. W. Fiske (1959). "Convergent and discriminant
validation by the multitrait-multimethod matrix". In: _Psychological
Bulletin_ 56.2, pp. 81-105. ISSN: 0033-2909. DOI:
[10.1037/h0046016](https://doi.org/10.1037%2Fh0046016). URL:
[http://content.apa.org/journals/bul/56/2/81](http://content.apa.org/journals/bul/56/2/81).

Crocker, L. and J. Algina (2008). _Introduction to classical and modern
test theory_. Mason, OH: Cencage Learning. ISBN: 978-0-495-39591-1.

Cronbach, L. J. and P. E. Meehl (1955). "Construct validity in
psychological tests." In: _Psychological Bulletin_ 52 (4), pp. 281-302.
ISSN: 0033-2909. DOI:
[10.1037/h0040957](https://doi.org/10.1037%2Fh0040957). URL:
[http://www.ncbi.nlm.nih.gov/pubmed/19586159
http://content.apa.org/journals/bul/52/4/281](http://www.ncbi.nlm.nih.gov/pubmed/19586159
http://content.apa.org/journals/bul/52/4/281).

Davis, L. L. (1992). "Instrument review: Getting the most from a panel
of experts". In: _Applied Nursing Research_ 5.4, pp. 194-197. ISSN:
08971897. DOI:
[10.1016/S0897-1897(05)80008-4](https://doi.org/10.1016%2FS0897-1897%2805%2980008-4).
URL:
[https://linkinghub.elsevier.com/retrieve/pii/S0897189705800084](https://linkinghub.elsevier.com/retrieve/pii/S0897189705800084).

Downing, S. M. and R. Yudkowsky, ed. (2009). _Assessment in health
professions education_. New York, NY: Routledge. ISBN: 9780203880135.
DOI: [10.4324/9780203880135](https://doi.org/10.4324%2F9780203880135).
URL:
[https://www.taylorfrancis.com/books/9780203880135](https://www.taylorfrancis.com/books/9780203880135).

---
# References    
Duong, F., J. A. Hall, et al. (2020). "Who are empathic people? It
depends on your definition of empathy". In: _Poster presented at SPSP
2020_. URL: [osf.io/dme3w](osf.io/dme3w).

Ebel, R. L. (1961). "Must all tests be valid?". In: _American
Psychologist_ 16.10, pp. 640-647. ISSN:
1935-990X(Electronic);0003-066X(Print). DOI:
[10.1037/h0045478](https://doi.org/10.1037%2Fh0045478).

Fan, X. (2013). "“The test is reliable”; “the test is valid”: Language
use, unconscious assumptions, and education research practice". In:
_The Asia-Pacific Education Researcher_ 22 (2), pp. 217-218. ISSN:
0119-5646. DOI:
[10.1007/s40299-012-0036-y](https://doi.org/10.1007%2Fs40299-012-0036-y).
URL:
[http://link.springer.com/10.1007/s40299-012-0036-y](http://link.springer.com/10.1007/s40299-012-0036-y).

Franke, G. R., M. Sarstedt, et al. (2021). "Assessing measure
congruence in nomological networks". In: _Journal of Business Research_
130, pp. 318-334. ISSN: 01482963. DOI:
[10.1016/j.jbusres.2021.03.003](https://doi.org/10.1016%2Fj.jbusres.2021.03.003).
URL:
[https://linkinghub.elsevier.com/retrieve/pii/S0148296321001569](https://linkinghub.elsevier.com/retrieve/pii/S0148296321001569).

Furr, R. M. (2022). _Psychometrics: An introduction_. 4th ed. Thousand
Oaks, CA: SAGE. ISBN: 978-1-0718-2409-2.

---
# References    
Furr, R. M. and S. Heuckeroth (2019). "The “quantifying construct
validity” procedure: Its role, value, interpretations, and
computation". In: _Assessment_ 26.4, pp. 555-566. ISSN: 1073-1911. DOI:
[10.1177/1073191118820638](https://doi.org/10.1177%2F1073191118820638).
URL:
[http://journals.sagepub.com/doi/10.1177/1073191118820638](http://journals.sagepub.com/doi/10.1177/1073191118820638).

Gronlund, N. E. and R. L. Linn (1990). _Measurement and evaluation in
teaching_. 6th ed. New York, NY: Macmillan. ISBN: 0023481110.

Huebner, A. and G. B. Skar (2021). "Conditional standard error of
measurement: Classical test theory, generalizability theory and
many-facet Rasch measurement with applications to writing assessment".
In: _Practical Assessment, Research & Evaluation_ 26.1, pp. 1-20. DOI:
[10.7275/vzmm-0z68](https://doi.org/10.7275%2Fvzmm-0z68). URL:
[https://openpublishing.library.umass.edu/pare/article/id/1573/](https://openpublishing.library.umass.edu/pare/article/id/1573/).

Kane, M. T. (2013). "Validating the interpretations and uses of test
scores". In: _Journal of Educational Measurement_ 50.1, pp. 1-73. ISSN:
00220655. DOI:
[10.1111/jedm.12000](https://doi.org/10.1111%2Fjedm.12000). URL:
[http://doi.wiley.com/10.1111/jedm.12000](http://doi.wiley.com/10.1111/jedm.12000).

Kim, H. and S. Baghestani (2025). "How many language testing
publications use “the unqualified phrase ‘the validity of the test’”?"
In: _Language Teaching Research Quarterly_ 51 (Special Issue), pp.
273-283. ISSN: 26676753. DOI:
[10.32038/ltrq.2025.51.05](https://doi.org/10.32038%2Fltrq.2025.51.05).

---
# References    
Lilienfeld, S. O., A. L. Pydych, et al. (2017). "50 differences that
make a difference: A compendium of frequently confused term pairs in
psychology". In: _Frontiers in Education_ 2, pp. 1-11. ISSN: 2504-284X.
DOI:
[10.3389/feduc.2017.00037](https://doi.org/10.3389%2Ffeduc.2017.00037).
URL:
[http://journal.frontiersin.org/article/10.3389/feduc.2017.00037/full](http://journal.frontiersin.org/article/10.3389/feduc.2017.00037/full).

LimeSurvey GmbH (2024). _LimeSurvey: An open source survey tool
(version 6.2.5) [Computer software]_. Hamburg. URL:
[http://www.limesurvey.org](http://www.limesurvey.org).

Lissitz, R. W. and K. Samuelsen (2007). "A suggested change in
terminology and emphasis regarding validity and education". In:
_Educational Researcher_ 36.8, pp. 437-448. ISSN: 0013-189X. DOI:
[10.3102/0013189X07311286](https://doi.org/10.3102%2F0013189X07311286).
URL:
[http://journals.sagepub.com/doi/10.3102/0013189X07311286](http://journals.sagepub.com/doi/10.3102/0013189X07311286).

Lynn, M. R. (1986). "Determination and quantification of content
validity". In: _Nursing Research_ 35.6, pp. 382-386. ISSN: 0029-6562.
DOI:
[10.1097/00006199-198611000-00017](https://doi.org/10.1097%2F00006199-198611000-00017).
URL:
[http://journals.lww.com/00006199-198611000-00017](http://journals.lww.com/00006199-198611000-00017).

Masterson, M. J. and R. K. Rainer Jr. (2009). "The nomological network
and the research continuum". In: _Encyclopedia of information science
and technology_. Ed. by M. Khosrow-Pour. 2nd ed. Vol. 1-8. IGI Global,
pp. 2827-2833. DOI:
[10.4018/978-1-60566-026-4.ch452](https://doi.org/10.4018%2F978-1-60566-026-4.ch452).
URL:
[http://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-60566-026-4.ch452](http://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-60566-026-4.ch452).

---
# References    
Melo, L. E. A. de, V. R. Duval, et al. (2023). "Método para comparação
entre inconsistência cognitiva, satisfação e intenção no uso de
transporte público". In: _Anais do Congresso de Pesquisa e Ensino em
Transportes_. URL:
[https://proceedings.science/p/174678?lang=pt-br](https://proceedings.science/p/174678?lang=pt-br).

Messick, S. (1989). "Validity". In: _Educational measurement_. Ed. by
R. L. Linn. 3rd ed. New York, NY: American Council on
Education/Macmillan. Chap. 1, pp. 13-103.

Messick, S. (2000). "Consequences of test interpretation and use: The
fusion of validity and values in psychological assessment". In:
_Problems and Solutions in Human Assessment_ , pp. 3-20. URL:
[http://link.springer.com/chapter/10.1007/978-1-4615-4397-8\_1](http://link.springer.com/chapter/10.1007/978-1-4615-4397-8\_1).

Miller, D., R. L. Linn, et al. (2009). _Measurement and assessment in
teaching_. 10th ed. Upper Saddle River, NJ: Pearson Education. ISBN:
9780137126712; 0137126719. DOI: [LK -
https://worldcat.org/title/769655999](https://doi.org/LK%20-%20https%3A%2F%2Fworldcat.org%2Ftitle%2F769655999).

Mussel, P. (2013). "Introducing the construct curiosity for predicting
job performance". In: _Journal of Organizational Behavior_ 34.4, pp.
453-472. ISSN: 08943796. DOI:
[10.1002/job.1809](https://doi.org/10.1002%2Fjob.1809). URL:
[http://doi.wiley.com/10.1002/job.1809](http://doi.wiley.com/10.1002/job.1809).

---
# References    
Newton, P. E. and S. D. Shaw (2013). "Standards for talking and
thinking about validity.". In: _Psychological Methods_ 18.3, pp.
301-319. ISSN: 1939-1463. DOI:
[10.1037/a0032969](https://doi.org/10.1037%2Fa0032969). URL:
[http://doi.apa.org/getdoi.cfm?doi=10.1037/a0032969](http://doi.apa.org/getdoi.cfm?doi=10.1037/a0032969).

Polit, D. F. and C. T. Beck (2006). "The content validity index: Are
you sure you know what's being reported? Critique and recommendations".
In: _Research in Nursing & Health_ 29.5, pp. 489-497. ISSN: 0160-6891.
DOI: [10.1002/nur.20147](https://doi.org/10.1002%2Fnur.20147). eprint:
NIHMS150003. URL: [http://doi.wiley.com/10.1002/nur.20270
http://doi.wiley.com/10.1002/nur.20147](http://doi.wiley.com/10.1002/nur.20270
http://doi.wiley.com/10.1002/nur.20147).

Polit, D. F., C. T. Beck, et al. (2007). "Is the CVI an acceptable
indicator of content validity? Appraisal and recommendations". In:
_Research in Nursing & Health_ 30.4, pp. 459-467. ISSN: 01606891. DOI:
[10.1002/nur.20199](https://doi.org/10.1002%2Fnur.20199). URL:
[https://onlinelibrary.wiley.com/doi/10.1002/nur.20199](https://onlinelibrary.wiley.com/doi/10.1002/nur.20199).

Ryan, A. M. and N. T. Tippins (2004a). "Attracting and selecting: What
psychological research tells us". In: _Human Resource Management_ 43
(4), pp. 305-318. ISSN: 0090-4848. DOI:
[10.1002/hrm.20026](https://doi.org/10.1002%2Fhrm.20026). URL:
[https://onlinelibrary.wiley.com/doi/10.1002/hrm.20026](https://onlinelibrary.wiley.com/doi/10.1002/hrm.20026).

Saxe, J. G. (1873). _The poems of John Godfrey Saxe_. James R. Osgood
and Co. URL:
[https://lccn.loc.gov/30003160](https://lccn.loc.gov/30003160).

---
# References    
Schmidt, F. L. (1988). "Validity generalization and the future of
criterion-related validity". In: _Test validity_. Ed. by H. Wainer and
H. I. Braun. Hillsdale, NJ: Routledge. Chap. 11, pp. 173-189. DOI:
[10.4324/9780203056905-22](https://doi.org/10.4324%2F9780203056905-22).
URL:
[https://www.taylorfrancis.com/chapters/edit/10.4324/9780203056905-15](https://www.taylorfrancis.com/chapters/edit/10.4324/9780203056905-15).

Shaw, S. and V. Crisp (2011). "Tracing the evolution of validity in
educational measurement: Past issues and contemporary challenges". In:
_Research Matters_ (11), pp. 14-17. URL:
[https://www.cambridgeassessment.org.uk/our-research/all-published-resources/research-matters/rm-11/](https://www.cambridgeassessment.org.uk/our-research/all-published-resources/research-matters/rm-11/).

Sireci, S. G. and J. Padilla (2014). "Validating assessments:
Introduction to the special section". In: _Psicothema_ 26 (1), pp.
97-99. ISSN: 1886-144X. DOI:
[10.7334/psicothema2013.255](https://doi.org/10.7334%2Fpsicothema2013.255).
URL:
[http://www.ncbi.nlm.nih.gov/pubmed/24444736](http://www.ncbi.nlm.nih.gov/pubmed/24444736).

Tourangeau, R. and K. A. Rasinski (1988). "Cognitive processes
underlying context effects in attitude measurement.". In:
_Psychological Bulletin_ 103.3, pp. 299-314. ISSN: 1939-1455. DOI:
[10.1037/0033-2909.103.3.299](https://doi.org/10.1037%2F0033-2909.103.3.299).
URL:
[http://doi.apa.org/getdoi.cfm?doi=10.1037/0033-2909.103.3.299](http://doi.apa.org/getdoi.cfm?doi=10.1037/0033-2909.103.3.299).

VandenBos, G. R., ed. (2015). _APA dictionary of psychology_. 2nd ed.
Washington, DC: American Psychological Association. ISBN:
1-4338-1944-9. DOI:
[10.1037/14646-000](https://doi.org/10.1037%2F14646-000). eprint:
arXiv:1011.1669v3. URL:
[http://content.apa.org/books/14646-000](http://content.apa.org/books/14646-000).

---
class: center, bottom, inverse

# More info

Slides created with the <svg viewBox="0 0 581 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;" xmlns="http://www.w3.org/2000/svg">  <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"></path></svg> package [`xaringan`](https://github.com/yihui/xaringan).

Practice is the best strategy for learning.

_In God we trust, all others bring data_

Edwards Deming

THE END

---
class: center, bottom, inverse

![:scale 50%](assets/gif/the_end.gif)