Psychometrics Applied to Organizational and Work Psychology

.title[
# Psychometrics Applied to Organizational and Work Psychology
]
.subtitle[
## <br>Validity and Its Evidence
]
.author[
### Jorge Sinval
]
.date[
### 2024-11-15
]

---

# Validity
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Validity

At the turn of the 20<sup>th</sup> century, the growth of educational and psychological measurement in the U.S.A. raised quality concerns. The APA established committees in 1895 and 1906 for standardization, but early efforts were ineffective (Newton and Shaw, 2013). Later, the Standardization Committee aimed to create consensus on assessing test superiority, providing definitions for key terms and outlining processes for determining validity and reliability:

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

Two of the most important types of problem in measurement are those connected with the determination of what a test measures, and of how consistently it measures. The first should be called the problem of **validity**, the second, the problem of **reliability**.

---

# Validity

## What is it?

.can-edit.key-validity[
- thing one
- thing two
- ...
]

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment.

---

# Validity

## What is it?

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

Validity refers to the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests.

.tr[
📖 (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 2014, p. 11)
]]

---

# Validity

## Unitary Concept

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

Although different sources and mixes of evidence are required to support
particular score-based inferences, validity is a unitary concept in the sense that score meaning as embodied in construct validity underlies all score-based inferences.

---

# Validity

A test or measure is neither valid nor invalid:

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

Hence, what is to be validated is not the test or observation device as such but the inferences derived from test scores or other indicators &mdash; inferences about score meaning or interpretation and about the implications for action that the interpretation entails.

---

# Validity

## Standards for talking and thinking about validity

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

1. Thou shalt not refer to 'the validity of the test' (TVOTT), that is, as though validity were a property of tests.  
2. Thou shalt (not) use validity modifier labels (VMLs), that is, terms like content validity and predictive validity (there is a not in parentheses because it was promoted by the first three editions yet rejected by the fourth and fifth).

---

# Validity

## Standards for talking and thinking about validity

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[
Validity is not a property of the test. Rather, it is a property of the proposed interpretations and uses of the test scores. Interpretations and uses that make sense and are supported by appropriate evidence are considered to have high validity (or for short, to be valid), and interpretations or uses that are not adequately supported, or worse, are contradicted by the available evidence are taken to have low validity (or for short, to be invalid). The scores generated by a given test can be given different interpretations, and some of these interpretations may be more plausible than others.

---

# Validity

## Standards for talking and thinking about validity

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[
The sloppy language of 'the test is reliable' or 'the test is valid' should be avoided; instead, 'measurement reliability', 'test score reliability', 'validity of test scores or test use' are better choices. Sloppy language use in research context may corrupt our understanding about the nature of measurement reliability and validity, and may lead to unhealthy research practice, such as making little or no effort in evaluating or reporting the reliability of our measurement in a research study."

---

# Validity

## Standards for talking and thinking about validity

Validity refers to the characteristics of measurement &mdash; that is, the characteristics of the scores obtained from administering a test to a specific group, rather than the characteristics of a test itself (Fan, 2013).

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[
...a test is not 'reliable' or 'unreliable.' Rather, reliability is a property of the scores on a test for a particular group of examinees. Thus, potential test users need to determine whether reliability reported in test manuals are based on samples similar in composition and variability to the group for whom the test will be used.

---

# Validity

## Standards for talking and thinking about validity

Reliability does not refer to the instrument itself:

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[
Technically, reliability refers to the consistency of the results obtained, not to the instrument itself.

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[
reliability refers to the results obtained with an evaluation instrument and not to the instrument itself… it is more appropriate to speak of the reliability of the 'test scores' or of the 'measurement' than of the 'test' or the 'instrument.'

---

# Validity

## Standards for talking and thinking about validity

As previously mentioned, validity refers to the appropriate use of test scores in a given measurement situation but not to the test or instrument itself (American Educational Research Association American Psychological Association et al., 2014; Fan, 2013).

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

If a researcher consciously or unconsciously assumes that reliability (validity) is a characteristic of a test or instrument itself, then reliability information from the test norming sample (as reported in the test technical manual), or those reported in other studies, would be applicable in his/her own study; as a result, there would be no, or less, need to evaluate measurement reliability and to report reliability estimates for his/her own data.

---

# Validity

## Standards for talking and thinking about validity

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

We sometimes speak of the 'validity of a test' for the sake of convenience, but it is more correct to speak of the validity of the interpretation and use to be made of the results

---

# Validity

## Evolution of the concept in the _Standards_

<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#assets/img/standards_evolution.png" alt="The Evolution of the Concept of Validity in the &lt;i&gt;Standards&lt;/i&gt;. Reprinted from Sireci and Padilla (2014)." width="45%" />
<p class="caption">The Evolution of the Concept of Validity in the <i>Standards</i>. Reprinted from Sireci and Padilla (2014).</p>
</div>

---

# Validity

## Evolution of the concept in the _Standards_

The concept of validity has been evident in literature since the early 20<sup>th</sup> century and has evolved over time. Initially, validity was seen as a static property measured by a single statistic, typically the correlation of test scores with a certain criterion. However, modern validity theory views validity more about the appropriateness of inferences and uses made from assessment results, including the consequences of the usage of test scores (Shaw and Crisp, 2011).

]

.pull-right-1[
<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#assets/img/standards_evolution_ii.png" alt="Timeline of the evolution of validity theory Tracing. Reprinted from Shaw and Crisp (2011)." width="65%" />
<p class="caption">Timeline of the evolution of validity theory Tracing. Reprinted from Shaw and Crisp (2011).</p>
</div>

]

---

# Validity

## The new _Standards_

|Evidence Source|Definition|Examples|
|---|---|---|
|Content| The "relationship between a test’s content and the construct it is intended to measure." Refers to themes, wording, and format of items on an assessment instrument. Includes analyses by experts regarding how adequately items represent the content domain. Also includes development strategies to ensure appropriate content representation|Surveying experienced teachers regarding the adequacy and representativeness of proposed instrument items. Choosing items previously utilized in similar settings. Developing instruments based on established educational theories.|
|Response process|Analyses of responses, including the actions, strategies, and thought processes of individual respondents or observers. Differences in response processes may reveal sources of variance that are irrelevant to the construct being measured. Also includes instrument security, scoring, and reporting of results.|Interviewing and studying learners regarding factors that influence the ratings they assign to teachers. Analyzing varying response patterns among different categories/levels of learners.|
|Internal structure|The degree to which individual items within the instrument fit the underlying constructs. Items measuring a unidimensional construct should be homogenous, while items measuring complex constructs should not. Most often reported as measures of internal consistency reliability and factor analysis.|Using factor analysis to determine the dimensional structure of an instrument’s scores, and determining the reliability of scores. Studying differential functioning of items among a homogenous group of evaluators.|
|Relations to other variables|The relationship between scores and other variables relevant to the construct being measured. Relationships may be positive (convergent or predictive) or negative (divergent or discriminant) depending on the constructs being measured.|How well do teachers’ assessment scores predict learners’ performance on high-stakes examinations, or their choice of a medical specialty? Do scores correlate with other measures of the same construct? Can results of an evaluation be generalized from one setting to another, similar setting?|
|Consequences|Assessments are intended to have some desired effect (e.g., improve teaching performance), but they also have unintended effects. Evaluating such consequences can support or challenge the validity of score interpretations.|Do equally qualified teachers’ performances on clinical teaching assessments correlate with factors that are not being measured, such as medical subspecialty, gender, or ethnicity?|

]

Validity: Sources of Evidence, Definitions, and Examples. Reprinted from Beckman, Cook, and Mandrekar (2005).

---

# Validity

## Five sources of validity evidence

<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#assets/img/Standards.jpg" alt="The Standards of Educational and Psychological Testing(American Educational Research Association American Psychological Association et al., 2014). Freely available &lt;a href="https://www.testingstandards.net/uploads/7/6/6/4/76643089/9780935302356.pdf"&gt;here&lt;/a&gt;." width="28%" />
<p class="caption">The Standards of Educational and Psychological Testing(American Educational Research Association American Psychological Association et al., 2014). Freely available <a href="https://www.testingstandards.net/uploads/7/6/6/4/76643089/9780935302356.pdf">here</a>.</p>
</div>

---

# Evidence Sources
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Evidence Sources

## Based on Content

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

**Content** &mdash; relationship between test content and the construct of interest; theory; hypothesis about content; independent assessment of match between content sampled and domain of interest; solid, scientific, quantitative evidence.

???
Partially based on [this](https://researchmethod.net/content-validity/) page.

---

# Evidence Sources

## Based on Content: A parabole

### The blind men and the elephant. A hindoo fable

**I**.

It was six men of Indostan  
To learning much inclined,  
Who went to see the Elephant  
(Though all of them were blind),  
That each by observation  
Might satisfy his mind.

]

**II**.

The _First_ approached the Elephant,  
And happening to fall  
Against his broad and sturdy side,  
At once began to bawl:  
"God bless me!—but the Elephant  
Is very like a wall!"

]

The _Second_, feeling of the tusk,  
Cried: "Ho!—what have we here  
So very round and smooth and sharp?  
To me 't is mighty clear  
This wonder of an Elephant  
Is very like a spear!"

]

---

# Evidence Sources

## Based on Content: A parabole

### The blind men and the elephant. A hindoo fable

**IV**.

The _Third_ approached the animal,  
And happening to take  
The squirming trunk within his hands,  
Thus boldly up and spake:  
"I see," quoth he, "the Elephant  
Is very like a snake!"

]

**V**.

The _Fourth_ reached out his eager hand,  
And felt about the knee.  
"What most this wondrous beast is like  
Is mighty plain," quoth he;  
"'T is clear enough the Elephant  
Is very like a tree!"

]

**VI**.

The _Fifth_, who chanced to touch the ear,  
Said: "E'en the blindest man  
Can tell what this resembles most;  
Deny the fact who can,  
This marvel of an Elephant  
Is very like a fan!"

]

---

# Evidence Sources

## Based on Content: A parabole

### The blind men and the elephant. A hindoo fable

**VII**.

The _Sixth_ no sooner had begun  
About the beast to grope,  
Than, seizing on the swinging tail  
That fell within his scope,  
"I see," quoth he, "the Elephant  
Is very like a rope!"

]

**VIII**.

And so these men of Indostan  
Disputed loud and long,  
Each in his own opinion  
Exceeding stiff and strong,  
Though each was partly in the right,  
And all were in the wrong!

]

**Moral**.

So, oft in theologic wars  
The disputants, I ween,  
Rail on in utter ignorance  
Of what each other mean,  
_And prate about an Elephant_
_Not one of them has seen!_

(Saxe, 1873, pp. 135-136)

]

---

# Evidence Sources

## Based on Content: A parabole

### The blind men and the elephant

<br>

<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#assets/img/blind_men_and_elephant.jpg" alt="The blind men and the elephant. Reprinted from Woods (1916, p. 14)." width="41%" />
<p class="caption">The blind men and the elephant. Reprinted from Woods (1916, p. 14).</p>
</div>

---

# Evidence Sources

## Based on Content

Examples (Downing and Yudkowsky, 2009, p. 30):

• Examination blueprint  
  • Representativeness of test blueprint to achievement domain  
  • Test specifications  
  • Match of item content to test specifications  
  • Representativeness of items to domain  
  • Logical/empirical relationship of content tested to achievement domain  
  • Quality of test questions  
  • Item writer qualifications  
  • Sensitivity review

---

# Evidence Sources

## Based on Content: Content Validity Ratio (CVR)

The Content Validity Ratio (CVR) was developed by Lawshe (1975) to provide a quantitative measure of content validity. The CVR provides an index that represents the proportion of agreement among experts on the essentialness of an item.

**Calculation**: To calculate CVR, each item is rated by a panel of experts who are knowledgeable about the construct being measured. The experts rate the items on a scale such as "essential," "useful, but not essential," or "not necessary."

**Formula:** The CVR is then calculated for each item using the formula:

`$$CVR = \frac{Ne - \frac{N}{2}}{\frac{N}{2}}$$`

Where:

- `$n_e$` is the number of panelists indicating "essential;" and,  
  - `$N$` is the total number of panelists.

---

# Evidence Sources

## Based on Content: CVR Guidelines<sup>⚠️</sup>

1. A positive `$CVR$` value indicates that more than half of the experts rated the item as "essential." The closer the `$CVR$` is to `$1$`, the higher the agreement among experts that the item is essential.  
  2. A `$CVR$` of `$0$` indicates that exactly half the panelists rated the item as "essential." 
  3. A negative `$CVR$` value indicates that less than half of the panelists rated the item as "essential."

.footnote[<sup>⚠️</sup>A high CVR doesn't necessarily mean the item is good, it just means there's a high agreement among experts on its essentialness. Other aspects such as item difficulty, and discrimination should also be considered when evaluating the quality of a test item.]

---

# Evidence Sources

## Based on Content: CVR Guidelines

<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#assets/img/cvr_min_val_lawshe1975.png" alt="Minimum Values of Content Validity Ratio. Reprinted from Lawshe (1975)." width="41%" />
<p class="caption">Minimum Values of Content Validity Ratio. Reprinted from Lawshe (1975).</p>
</div>

According to Lawshe (1975), for an item to be retained, the `$CVR$` must be significant at a certain level. The level of significance depends on the number of panelists. For example, with `$5$` panelists, the CVR must be `$0.99$` or higher to be significant at the `$0.05$` level. With `$10$` panelists, the `$CVR$` must be `$0.62$` or higher, and with `$20$` panelists, the `$CVR$` must be `$0.42$` or higher.

---

# Evidence Sources

## Based on Content: CVR Guidelines

A more recent proposal (Ayre and Scally, 2014).

]

<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#assets/img/cvr_critical_ayre2014.png" alt="Critical Values of Content Validity Ratio. Reprinted from Ayre and Scally (2014)." width="56%" />
<p class="caption">Critical Values of Content Validity Ratio. Reprinted from Ayre and Scally (2014).</p>
</div>

]

---

# Evidence Sources

## Based on Content: Example CVR

> Let's assume you have collected data from five experts who rated the relevance of four items on a scale from 1 to 3 (3 &mdash; "Not necessary," 2 &mdash; "Useful, but not essential," 1 &mdash; "Essential").

``` r
ds <- data.frame(
  item1 = c(3, 1, 2, 2, 1),
  item2 = c(1, 2, 2, 1, 1),
  item3 = c(1, 1, 1, 1, 1),
  item4 = c(2, 2, 3, 3, 3))

psychometric::CVratio(NTOTAL = lengths(ds),
                      NESSENTIAL = colSums(ds == 1))
```

```
## item1 item2 item3 item4 
##  -0.2   0.2   1.0  -1.0
```

]

---

# Evidence Sources

## Based on Content: Content Validity Index (CVI)

The Content Validity Index (CVI) quantifies content validity. It's calculated based on expert ratings of the relevance of individual items and the overall scale.

**Item-level CVI (I-CVI)**

This is calculated for each individual item.

**I-CVI Formula**

The I-CVI is calculated using the formula:

`$$I-CVI = \frac{n_{r}}{N}$$`

Where,  
  - `$n_{r}$` is the number of experts who rated the item as quite or highly relevant, and,  
  - `$N$` is the total number of experts.

---

# Evidence Sources

## Based on Content: CVI

**Scale-level CVI (S-CVI)**

This is calculated for the entire scale or instrument.

**S-CVI Formulas**

There are two types of S-CVI: Average CVI (S-CVI/Ave) and Universal Agreement CVI (S-CVI/UA).

`$$S-CVI/Ave = \frac{\sum_{i=1}^{k} I-CVI_i}{k}$$`

`$$S-CVI/UA = \frac{n_{I-CVI=1}}{k}$$`

Where,  
  - `$I-CVI_i$` is the I-CVI for item `$i$`,  
  - `$k$` is the number of items, and,  
  - `$n_{I-CVI=1}$` is the number of items with an I-CVI of `$1$`.

---

# Evidence Sources

## Based on Content: CVI Guidelines

Lynn (1986) suggests that an I-CVI of `$0.78$` or higher is acceptable for at least nine experts. While Yusoff (2019) summarizes a bit more detailed guidelines from various authors.

|Number of experts|Acceptable `$CVI$` values|Source of recommendation|
|---|---|---|---|
| `$2$` experts| `$CVI \geq.80$`|Davis (1992)|
| `$3-5$` experts| `$CVI = 1.00$`|Polit and Beck (2006); Polit, Beck, and Owen (2007)|
| `$\geq 6$` experts| `$CVI \geq .83$`|Polit and Beck (2006); Polit Beck et al. (2007)|
| `$6-8$` experts| `$CVI \geq .83$`|Lynn (1986)|
| `$\geq9$` experts| `$CVI \geq .78$`|Lynn (1986)|

Recommended values of `$CVI$`. Adapted from Yusoff (2019).

---

# Evidence Sources

## Based on Content: Example CVI

> Let's say you have a set of four items that are rated by five experts on a 4-point scale (1 &mdash; "Not relevant," 2 &mdash; "Somewhat relevant," 3 &mdash; "Quite relevant," 4 &mdash; "Highly relevant").

``` r
ds <- data.frame(
  item1 = c(4, 3, 4, 2, 4),
  item2 = c(3, 2, 4, 3, 2),
  item3 = c(2, 2, 3, 2, 1),
  item4 = c(4, 4, 4, 3, 4))
# Compute I-CVI
i_cvi <- function(x) {
  n <- length(x)
  n_relevant <- sum(x >= 3)  # Count ratings 3 and 4 as relevant
  return(n_relevant / n)}
i_cvi_values <- apply(ds, 2, i_cvi)
# Compute S-CVI
s_cvi_ave <- mean(i_cvi_values)
s_cvi_ua <- sum(i_cvi_values == 1) / length(i_cvi_values)
i_cvi_values
s_cvi_ave
s_cvi_ua
```

---

# Evidence Sources

## Based on Content: Example CVI

```
## item1 item2 item3 item4 
##   0.8   0.6   0.2   1.0
```

```
## [1] 0.65
```

```
## [1] 0.25
```

]

---

# Evidence Sources

## Based on Content: Practical Example

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

A team of researchers conducted a workplace survey at a multinational corporation based in São Paulo, where the majority of participants (78.6%) indicated that they benefit from flexible working hours. The survey included questions about the employees' job satisfaction, work-life balance, and productivity levels. Participants were asked to rate their agreement with positive statements about the flexible work schedule, such as its impact on reducing stress, improving work efficiency, and fostering a healthier work-life balance. The responses were measured on a Likert scale ranging from 'Strongly Disagree' to 'Strongly Agree.'

]

---

# Evidence Sources

## Based on Content: Practical Example

Participants were asked to express their level of agreement with the following statements on a 5-point Likert scale ranging from "Strongly Disagree" to "Strongly Agree".

_For me, having flexible working hours at my job is..._

| Item                                                                                     | Strongly Disagree<br>1 | Disagree<br>2 | Neutral<br>3 | Agree<br>4 | Strongly Agree<br>5 |
|------------------------------------------------------------------------------------------|------------------------|---------------|--------------|------------|---------------------|
| ...beneficial to my mental and physical health                                            |                        |               |              |            |                     |
| ...an efficient way to manage my time                                                     |                        |               |              |            |                     |
| ...a comfortable way to balance work and personal life                                    |                        |               |              |            |                     |
| ...a safe option in terms of workplace security                                           |                        |               |              |            |                     |
| ...a motivating factor in my job                                                          |                        |               |              |            |                     |
| ...reliable in terms of meeting deadlines                                                 |                        |               |              |            |                     |
| ...a cost-effective option for my daily commute                                           |                        |               |              |            |                     |
| ...an unrealistic approach to productivity                                                |                        |               |              |            |                     |
| ...a pleasant and stress-reducing work arrangement                                        |                        |               |              |            |                     |
| ...a safer way to avoid job-related accidents due to fatigue                              |                        |               |              |            |                     |
| ...an environmentally friendly alternative due to reduced commuting                       |                        |               |              |            |                     |
| ...a practical and easy way to improve my work performance                                |                        |               |              |            |                     |

]

---

# Evidence Sources

## Based on Content: Practical Example

The items content will be analyzed by a panel of work and organizational psychology experts (the students in our class). The experts are asked to rate the relevance of each item to the construct being measured ([link](https://rstudio-cld.ncg.ingrid.pt:8443/index.php/545815)).
<center>
<embed src="https://rstudio-cld.ncg.ingrid.pt:8443/index.php/545815" width=1000px height=400px />
</center>

---

# Evidence Sources

## Based on Content: Practical Example

Instructions to the experts (regarding `$CVR$`):

The following items have been developed for a psychometric instrument intended to measure attitudes regarding the use of flexible working hours in a company. Keeping in mind the purpose of this instrument, you are asked to rate each of the 12 items presented below using the following response scale: 1 &mdash; "Essential"; 2 &mdash; "Useful, but not essential"; 3 &mdash; "Not necessary." The instrument to be presented to the participants (employees working under a flexible schedule) is shown next, with the following instructions:

_Factors that make up the attitude (possible answers 'Strongly Disagree,' 'Disagree,' 'Indifferent,' 'Agree,' or 'Strongly Agree')._

| Item<br>**_For me, having flexible working hours at my job is..._**                        | Essential<br>1 | Useful, but Not Essential<br>2 | Not Necessary<br>3 |
  |--------------------------------------------------------------------------------------------|----------------|--------------------------------|--------------------|
  | ...beneficial to my mental and physical health                                             |                |               |              |            |                     |
  | ...an efficient way to manage my time                                                      |                |               |              |            |                     |
  | ...a comfortable way to balance work and personal life                                    |                        |               |              |            |                     |
  | ...a safe option in terms of workplace security                                           |                        |               |              |            |                     |
  | ...a motivating factor in my job                                                          |                        |               |              |            |                     |
  | ...reliable in terms of meeting deadlines                                                 |                        |               |              |            |                     |
  | ...a cost-effective option for my daily commute                                           |                        |               |              |            |                     |
  | ...an unrealistic approach to productivity                                                |                        |               |              |            |                     |
  | ...a pleasant and stress-reducing work arrangement                                        |                        |               |              |            |                     |
  | ...a safer way to avoid job-related accidents due to fatigue                              |                        |               |              |            |                     |
  | ...an environmentally friendly alternative due to reduced commuting                       |                        |               |              |            |                     |
  | ...a practical and easy way to improve my work performance                                |                        |               |              |            |                     |

]

---

# Evidence Sources

## Based on Content: Practical Example

Instructions to the experts (regarding `$CVI$`):

The following items have been developed for a psychometric instrument that intends to measure attitudes related to the use of flexible working hours in a company. Considering the aim of the instrument in question, you are asked to rate each of the 12 items below using the following response scale: 1 &mdash; "Not relevant," 2 &mdash; "Slightly relevant," 3 &mdash; "Quite relevant," 4 &mdash; "Highly relevant." The instrument to be presented to the participants (employees working under a flexible schedule) is shown next, with the following instructions:

_Factors that make up the attitude (possible answers 'Strongly Disagree', 'Disagree', 'Indifferent', 'Agree' or 'Strongly Agree')._

| Item<br>**_For me, having flexible working hours at my job is..._**                       |Not Relevant<br>1|Slightly relevant<br>2|Quite Relevant<br>3|Highly Relevant<br>4|
|---|---|---|---|---|
| ...beneficial to my mental and physical health                                            |                        |               |              |            |                     |
| ...an efficient way to manage my time                                                     |                        |               |              |            |                     |
| ...a comfortable way to balance work and personal life                                    |                        |               |              |            |                     |
| ...a safe option in terms of workplace security                                           |                        |               |              |            |                     |
| ...a motivating factor in my job                                                          |                        |               |              |            |                     |
| ...reliable in terms of meeting deadlines                                                 |                        |               |              |            |                     |
| ...a cost-effective option for my daily commute                                           |                        |               |              |            |                     |
| ...an unrealistic approach to productivity                                                |                        |               |              |            |                     |
| ...a pleasant and stress-reducing work arrangement                                        |                        |               |              |            |                     |
| ...a safer way to avoid job-related accidents due to fatigue                              |                        |               |              |            |                     |
| ...an environmentally friendly alternative due to reduced commuting                       |                        |               |              |            |                     |
| ...a practical and easy way to improve my work performance                                |                        |               |              |            |                     |

]

---

# Su<svg viewBox="0 0 581 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;" xmlns="http://www.w3.org/2000/svg">  <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"></path></svg>vey <svg viewBox="0 0 581 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;" xmlns="http://www.w3.org/2000/svg">  <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"></path></svg>esults

Read or survey data directly from LimeSurvey (LimeSurvey GmbH, 2024):

<div class="pre-name">limesurvey.R</div>

``` r
survey_id <- 545815
pacman::p_load(limer)
options(lime_api = Sys.getenv("lime_api"),
        lime_password = Sys.getenv("lime_password"),
        lime_username = Sys.getenv("lime_username"))
get_session_key()
ds <- get_responses(iSurveyID= survey_id,
                    sCompletionStatus='all',
                    sResponseType='short')
#select the variables for cvr (the ones that start wiht "cvr" and "cvi")
item_att_cvr <- colnames(ds)[grep("^cvr", colnames(ds))]
item_att_cvi <- colnames(ds)[grep("^cvi", colnames(ds))]
ds[ds == ""] <- NA #if "" the answer should replaced by NA
ds <- janitor::remove_empty(ds[,c(item_att_cvi,item_att_cvr)], which = "rows") #remove rows with NA
```

```
## [1] "iQM4Oas37nVgLtFUZi~oOBPq2ATKLYxI"
```

---

<div class="pre-name">example-cvr.R</div>

``` r
knitr::kable(ds[,item_att_cvr], caption = "Items' CVR")

#count number of NA per columns
n_total <- nrow(ds[,item_att_cvr]) - colSums(is.na(ds[,item_att_cvr]))

psychometric::CVratio(NTOTAL = n_total,
                      NESSENTIAL = colSums(ds[,item_att_cvr] == 1, #1 = Essential
                                           na.rm = TRUE))
```

---

<table>
<caption>Items' CVR</caption>
 <thead>
  <tr>
   <th style="text-align:right;"> cvr.1. </th>
   <th style="text-align:right;"> cvr.2. </th>
   <th style="text-align:right;"> cvr.3. </th>
   <th style="text-align:right;"> cvr.4. </th>
   <th style="text-align:right;"> cvr.5. </th>
   <th style="text-align:right;"> cvr.6. </th>
   <th style="text-align:right;"> cvr.7. </th>
   <th style="text-align:right;"> cvr.8. </th>
   <th style="text-align:right;"> cvr.9. </th>
   <th style="text-align:right;"> cvr.10. </th>
   <th style="text-align:right;"> cvr.11. </th>
   <th style="text-align:right;"> cvr.12. </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
</tbody>
</table>

```
##     cvr.1.     cvr.2.     cvr.3.     cvr.4.     cvr.5.     cvr.6.     cvr.7. 
##  0.3333333  0.5000000  0.6666667 -0.1666667  0.6666667  0.0000000  0.0000000 
##     cvr.8.     cvr.9.    cvr.10.    cvr.11.    cvr.12. 
##  0.0000000  1.0000000 -0.3333333 -0.5000000  0.3333333
```

]

---

<div class="pre-name">example-cvi.R</div>

``` r
knitr::kable(ds[,item_att_cvi], caption = "Items' CVI")
i_cvi <- function(x) {
  n <- length(x)
  n_relevant <- sum(x >= 3)  # Count ratings 3 and 4 as relevant
  return(n_relevant / n)}
i_cvi_values <- apply(ds[,item_att_cvi], 2, i_cvi)
# Compute S-CVI
s_cvi_ave <- mean(i_cvi_values)
s_cvi_ua <- sum(i_cvi_values == 1) / length(i_cvi_values)
print(i_cvi_values)
print(s_cvi_ave)
print(s_cvi_ua)
```

---

<table>
<caption>Items' CVI</caption>
 <thead>
  <tr>
   <th style="text-align:right;"> cvi.i1. </th>
   <th style="text-align:right;"> cvi.i2. </th>
   <th style="text-align:right;"> cvi.i3. </th>
   <th style="text-align:right;"> cvi.i4. </th>
   <th style="text-align:right;"> cvi.i5. </th>
   <th style="text-align:right;"> cvi.i6. </th>
   <th style="text-align:right;"> cvi.i7. </th>
   <th style="text-align:right;"> cvi.i8. </th>
   <th style="text-align:right;"> cvi.i9. </th>
   <th style="text-align:right;"> cvi.i10. </th>
   <th style="text-align:right;"> cvi.i11. </th>
   <th style="text-align:right;"> cvi.i12. </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
</tbody>
</table>

```
##   cvi.i1.   cvi.i2.   cvi.i3.   cvi.i4.   cvi.i5.   cvi.i6.   cvi.i7.   cvi.i8. 
## 0.8333333 0.9166667 0.9166667 0.4166667 0.8333333 0.5833333 0.5000000 0.5000000 
##   cvi.i9.  cvi.i10.  cvi.i11.  cvi.i12. 
## 0.9166667 0.6666667 0.5000000 0.7500000 
## [1] 0.6944444
## [1] 0
```

]

---

# Evidence Sources

## Based on Content

What can be done after?

**Item Selection and Revision**

Based on the CVR scores and expert feedback, identify items that demonstrate high content validity. Items with low CVI scores may need revision or removal. Consider the experts’ comments and suggestions to refine and improve the wording, clarity, or coverage of the items.

**Pilot Testing**

Administer the revised measurement instrument to a small sample of respondents to gather data. Analyze the data to assess the appropriateness and relevance of the items. Examine response patterns, item means, and variability to ensure that the items effectively capture the intended content domain.

---

# Evidence Sources

## Based on Response Processes

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

**Response Process** &mdash; analysis of individual responses to stimuli; debriefing of examinees; process studies aimed at understanding what is measured and the soundness of intended score interpretations; quality assurance and quality control of assessment data.

---

# Evidence Sources

## Based on Response Processes

Examples (Downing and Yudkowsky, 2009, p. 30):

• Respondent format familiarity  
  • Quality control of electronic scanning/scoring  
  • Key validation of preliminary scores  
  • Accuracy in combining different item format scores  
  • Quality control/accuracy of final scores/marks/ grades  
  • Accuracy of applying pass–fail decision rules to scores  
  • Quality control of score reporting to respondents/organization  
  • Understandable/accurate descriptions/interpretations of scores for respondents

---

# Evidence Sources

## Based on Response Processes

Tourangeau and Rasinski (1988) suggest a four-step process:

• Interpretation of the question  
  • Retrieval of relevant information from memory  
  • Judgment formation  
  • Response execution  
  .transp[• Response editing (some include this 5<sup>th</sup> step)]

---

# Evidence Sources

## Based on Response Processes: Example

``` r
ds <- data.frame(
  employee_id = 1:500,
  engagement_level = sample(1:5,500,T, c(.1,.2,.2,.2,.3)),           # 5: very engaged, 1: very disengaged
  motivation_level = sample(1:5,500,T, c(.1,.1,.2,.3,.3)),           # 5: very motivated, 1: not motivated
  satisfaction_with_roles = sample(1:5,500,T, c(.05,.15,.2,.2,.4)),  # 5: very satisfied, 1: very dissatisfied
  willingness_to_recommend = sample(1:5, 500, replace = TRUE),       # 5: highly recommend, 1: would not recommend
  pass_fail = sample(c("pass", "fail"), 500, replace = TRUE)         # Assume we have a pass/fail criteria for engagement based on HR individual interviews
)

# Key validation of preliminary scores
# Assuming a score above 4 in all categories is considered as 'pass'
ds$pass_fail_check <- ifelse(ds$engagement_level > 4 & ds$motivation_level > 4 & 
ds$satisfaction_with_roles > 4 & ds$willingness_to_recommend > 4, "pass", "fail")

# Accuracy in combining different item format scores
# In this case, we combine the scores into a single measurement
ds$combined_score <- ds$engagement_level + ds$motivation_level + 
                     ds$satisfaction_with_roles + ds$willingness_to_recommend
```

---

# Evidence Sources

## Based on Response Processes: Example

``` r
# Quality control/accuracy of final scores/marks/grades
# Check if the "pass" or "fail" status matches with the actual scores

ds$accuracy_check <- ds$pass_fail == ds$pass_fail_check

# Accuracy of applying pass-fail decision rules to scores
accuracy <- mean(ds$accuracy_check)
```

---

# Evidence Sources

## Based on Response Processes: Example

Other examples of response process evidence include:

• Think-aloud protocols  
  • Cognitive interviews  
  • Debriefing interviews  
  • Observations of examinees  
  • Observations of raters  
  • Observations of test administrators  
  • Observations of scorers  
  • Observations of item writers  
  • Observations of item reviewers

---

# Evidence Sources

## Based on Internal Structure

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

**Internal Structure** &mdash; data internal to assessments such as: reliability or reproducibility of scores; inter-item correlations; statistical characteristics of items; statistical analysis of item option function; factor studies of dimensionality; Differential Item Functioning (DIF) studies

---

# Evidence Sources

## Based on Internal Structure

Examples (Downing and Yudkowsky, 2009, p. 30):

• Item analysis data  
  • Item difficulty/discrimination  
  • Item Characteristic Curve (ICC)<sup>⚠️</sup> or Test Characteristic Curve (TCC)  
  • Inter-item correlations  
  • Item-total correlations  
  • Score scale reliability  
  • Standard errors of measurement `$(SEM)$`  
  • Subscore/subscale analyses  
  • Generalizability  
  • Dimensionality  
  • Item factor analysis  
  • Differential Item Functioning (DIF)  
  • Psychometric model

.footnote[<sup>⚠️</sup> not to be confused with the `\$ICC\$` used in Intraclass Correlation Coefficient.]

---

# Evidence Sources

## Based on Relations to other Variables

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

**Relations to Other Variables** &mdash; data external to assessments such as: correlations of assessment variable(s) to external, independent measures; hypothesis and theory driven investigations; correlational research based on previous studies, literature:  
  a. **Convergent and discriminant evidence**: relationships between similar and different measures  
  b. **Test-criterion evidence**: relationships between test and criterion measure(s)  
  c. **Validity generalization**: can the validity evidence be generalized?  Evidence that the validity studies may generalize to other settings.

---

# Evidence Sources

## Based on Relations to other Variables

### Convergent and Discriminant Evidence

**Convergent evidence**: intended to assess the same or similar constructs provide convergent evidence.

**Discriminant evidence**<sup>⚠️</sup>: relationships between test scores and measures purportedly of different constructs.

.footnote[<sup>⚠️</sup>
_Discriminant validity_ refers to how unrelated a measure is to constructs it shouldn't be associated with. However, many mistakenly use the term to describe a test's ability to differentiate between groups, which is actually a sign of convergent validity. The correct term for a test's ability to distinguish between groups is "discriminative validity" (Lilienfeld, Pydych, Lynn, Latzman, and Waldman, 2017).]

---

# Evidence Sources

## Based on Relations to other Variables

### Test-Criterion Evidence

**Concurrent**: A concurrent study obtains test scores and criterion<sup>💡</sup> information at about the same time.

**Predictive**: A predictive study indicates the strength of the relationship between test scores and criterion scores that are obtained at a later time.

.footnote[<sup>💡</sup>A _criterion variable_ is a measure of some attribute or outcome that is operationally distinct from the test. Thus, the test is not a measure of a criterion, but rather is a measure hypothesized as a potential predictor of that targeted criterion. (American Educational Research Association American Psychological Association et al., 2014).
]

---

# Evidence Sources

## Based on Relations to other Variables

### Validity Generalization

Validity generalization<sup>⚠️</sup> refers to how well study findings can be applied across different populations, settings, or times (American Educational Research Association American Psychological Association et al., 2014). Thus it is is a process of evaluating a test’s validity coefficients across a large set of studies (Schmidt, 1988).

Although single local validation studies can be imprecise, a well-conducted study with an adequate sample size may provide sufficient evidence to support or reject test use in new contexts. Comparing the value of local studies versus meta-analyses is crucial for robust conclusions.

.footnote[<sup>⚠️</sup>_External validity_ concerns the generalizability of results to other settings, while ecological validity, as a subtype, specifically examines whether a study design reflects natural settings (Lilienfeld Pydych et al., 2017; Andrade, 2018). A study can have high ecological validity yet still lack external validity if the findings don't apply to broader situations.

]

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

"Scientifically speaking, to 'make clear what something is' means to set forth the laws in which it occurs. We shall refer to the interlocking system of laws which constitute a theory as a _nomological network_." The laws in a nomological network may relate (a) observable properties or quantities to each other; or (b) theoretical constructs to observables; or (c) different theoretical constructs to one another. These “laws” may be statistical or deterministic

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network

Cronbach and Meehl (1955) defined a nomological network as the interlocking system of hypotheses, principles, and laws linking the constructs that constitute any theory.

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

A nomological network encompasses the theoretical constructs being measured, how the concept is going to be measured, and the specification of the interrelationships between the theoretical and empirical planes.

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

**nomological network** a conceptual network: a broadly integrative theoretical framework that identifies the key constructs associated with a phenomenon of interest and the associations among those constructs. For example, psychopathy is a complex notion involving a significant nomological network of knowledge and speculations about components, causes, correlates, and consequences as well as their interrelationships and means of measurement or evaluation.

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

Is the latent trait measured by the test different from the trait(s) measured by other tests?  
Is the latent trait measured related to other traits of interest by a statistical or psychometric theory?  
Are there expected differences in the latent trait for different manifest groups?  
Does the latent trait change over time?

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network

<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#assets/img/nomological_network.svg" alt="The nomological network: 1:1 correspondence between theoretical constructs and measurements (which is not always the case). Reprinted from Masterson and Jr. (2009)." width="26%" />
<p class="caption">The nomological network: 1:1 correspondence between theoretical constructs and measurements (which is not always the case). Reprinted from Masterson and Jr. (2009).</p>
</div>

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network: An example

<div class="figure">
<div class="grViz html-widget html-fill-item" id="htmlwidget-a12e3094d171b3b4416e" style="width:100%;height:396px;"></div>
<script type="application/json" data-for="htmlwidget-a12e3094d171b3b4416e">{"x":{"diagram":"\n  digraph G {\n    graph [bgcolor=transparent];  // Set background to transparent\n    node [shape = circle, fixedsize = true, width = 2, height = 2, style = filled, fillcolor = lightblue]\n    JobSatisfaction [label = \"Job\nSatisfaction\", pos = \"0,0!\"]\n\n    node [shape = circle, fixedsize = true, width = 2, height = 2, style = filled, fillcolor = lightgrey]\n    WorkLifeBalance [label = \"Work-Life\nBalance\", pos = \"-2,2!\"]\n    RoleAmbiguity [label = \"Role\nAmbiguity\", pos = \"-2,-2!\"]\n    Autonomy [label = \"Autonomy\", pos = \"2,2!\"]\n    Burnout [label = \"Burnout\", pos = \"2,-2!\"]\n    OrgCommitment [label = \"Organizational\nCommitment\", pos = \"-3,0!\"]\n    SupervisorSupport [label = \"Supervisor\nSupport\", pos = \"3,0!\"]\n    OrgJustice [label = \"Organizational\nJustice\", pos = \"0,3!\"]\n    JobInsecurity [label = \"Job\nInsecurity\", pos = \"0,-3!\"]\n\n    WorkLifeBalance -> JobSatisfaction [dir = both, label = \"+\", fontsize = 12]\n    RoleAmbiguity -> JobSatisfaction [dir = both, label = \"-\", fontsize = 12]\n    Autonomy -> JobSatisfaction [dir = both, label = \"+\", fontsize = 12]\n    Burnout -> JobSatisfaction [dir = both, label = \"-\", fontsize = 12]\n    OrgCommitment -> JobSatisfaction [dir = both, label = \"+\", fontsize = 12]\n    SupervisorSupport -> JobSatisfaction [dir = both, label = \"+\", fontsize = 12]\n    OrgJustice -> JobSatisfaction [dir = both, label = \"+\", fontsize = 12]\n    JobInsecurity -> JobSatisfaction [dir = both, label = \"-\", fontsize = 12]\n\n    graph [layout = \"neato\", overlap = false]\n  }\n","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>
<p class="caption">Nomological network of job satisfaction.</p>
</div>

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network: The Snail Plot

<div class="figure">
<img src="data:image/png;base64,#assets/img/snail_plot.svg" alt="Nomological network of job curiosityMussel (2013)." width="40%" />
<p class="caption">Nomological network of job curiosityMussel (2013).</p>
</div>

</center>

Constructs with higher correlation with curiosity are displayed closer to the center and constructs with lower correlation further outside.

---

# Evidence Sources

## Based on Relations to other Variables

### The Nomological Network

**Considering a construct's place in a broader nomological network offers far more insight than pairwise comparisons.** Congruence assessment considers similarities and differences between pairs of variables relative to a network of other variables, not in isolation (Franke, Sarstedt, and Danks, 2021). High congruence, like high correlations:

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[
...challenge the proponent of distinct constructs to locate or create the circumstances under which the variables in question are distinguishable...

---

# Evidence Sources

## Based on Relations to other Variables

Examples (Downing and Yudkowsky, 2009, p. 30):

• Correlation with other relevant variables  
  • Convergent correlations—internal/external  
  • Similar tests  
  • Divergent correlations—internal/external  
  • Dissimilar measures  
  • Test-criterion correlations  
  • Generalizability of evidence

---

# Evidence Sources

## Based on Relations to other Variables

There are &mdash; at least &mdash; four main approaches to evaluate convergent and discriminant evidence, each differing in complexity, statistical rigor, age, and the need for explicit predictions (Furr, 2022). These approaches vary in their conceptual and statistical complexity, with some being well-established and others more recent innovations. Despite these differences, they all serve as valuable tools for assessing the validity evidence of measures:

1. **Focused Associations**: Associations between test scores and particularly important criteria.

2. **Sets of Correlations**: Associations between test scores and a wide range of relevant variables (usually evaluated subjectively via "eyeballing" the entire set of associations).

3. **Multitrait–Multimethod Matrices (MTMMM)** (Campbell and Fiske, 1959): Assocations between multiple constructs (traits), each of which is measured by multiple methods.

4. **Quantifying Construct Validity** (Westen and Rosenthal, 2003): Association between a _predicted_ pattern of convergent and discriminant correlations and the _observed_ pattern of correlations.

---

# Evidence Sources

## Based on Relations to other Variables

### Focused Associations

This type emphasizes the relevance of specific criterion variables to evaluate the validity of a test. It involves examining the correlation between test scores and key outcomes related to the construct being measured. For instance, a standardized college admission (e.g., INEM, SAT) test's validity evidence is closely tied to its correlation with college performance indicators, such as GPA. Strong correlations with these crucial variables support the test's validity, while weak correlations raise concerns about its effectiveness.

---

# Evidence Sources

## Based on Relations to other Variables

## Focus Associations: A Venn Diagram

<div class="figure">
<img src="data:image/png;base64,#assets/img/venn_diagramm.svg" alt="Incremental explained variance of curiosity over 12 cognitive and non-cognitive predictors (dark gray) and vice versa (light gray) Mussel (2013)." width="30%" />
<p class="caption">Incremental explained variance of curiosity over 12 cognitive and non-cognitive predictors (dark gray) and vice versa (light gray) Mussel (2013).</p>
</div>

</center>

Some refer to this as _incremental validity_.<sup>🧙‍♂️</sup>

.footnote[
<sup>🧙‍♂️</sup> Incremental validity is the extent to which a test adds to the prediction of a criterion variable beyond what is predicted by other measures.
]

---

# Evidence Sources

## Based on Relations to other Variables

### Sets of Correlations

This approach involves examining a broader range of criterion variables to assess the validity evidence. Researchers compute correlations between the scores of the psychometric instrument of interest and various other measures to evaluate convergent and discriminant evidence. For example, a new measure of burnout might show expected associations with both similar and distinct psychological constructs, thereby confirming its validity through a comprehensive correlation analysis.

---

# Evidence Sources

## Based on Relations to other Variables

### Sets of Correlations: A Set of Radial Plots

<div class="figure">
<img src="data:image/png;base64,#assets/img/radial_cor_plot.svg" alt="Stets of correlations for empathyDuong, Hall, and Schwartz (2020)." width="40%" />
<p class="caption">Stets of correlations for empathyDuong, Hall, and Schwartz (2020).</p>
</div>

</center>

---

# Evidence Sources

## Based on Relations to other Variables

### Multitrait–Multimethod Matrices (MTMMM) (Campbell and Fiske, 1959)

This method provides a structured way to evaluate convergent and discriminant evidence It involves measuring multiple traits using various methods (self-report, peer ratings, etc.), allowing researchers to discern whether observed correlations are due to actual psychological constructs or influenced by the methods used. The MTMTM helps clarify the relationships among traits and the effects of different measurement techniques.

Four possibilities exist:

1. **Heterotrait–heteromethod** correlations: different constructs, and different methods (weakest correlations expected)
  2. **Heterotrait–monomethod** correlations: different constructs,and the same method (moderate correlations expected?).  
  3. **Monotrait–heteromethod** correlations: same construct, and different methods (moderate correlations expected?).  
  4. **Monotrait–monomethod** correlations: same construct, and same method (strongest correlations expected). Test-retest reliability is a special case of this type of correlation.

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV)

Westen and Rosenthal (2003) proposed a method called QCV. This procedure enables researchers to systematically assess the degree of 'fit' between:

a) their theoretical predictions regarding convergent and discriminant correlations; and,  
  (b) the actual correlations observed in their data.

It seeks to answer one key question:

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[
Does this measure predict an array of other measures in a way predicted by theory?
.tr[(Westen and Rosenthal, 2003, p. 609)
]]

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV)

The QCV process consists of three main steps:

1. **Elicit the predicted pattern of validity correlations**: Based on the literature formulate hypotheses about the expected relationships between the construct of interest and other relevant variables.  
  2. **Estimate the actual pattern of correlations**: Collect data and calculate the correlations between the construct of interest and other variables.  
  3. **Compare the predicted pattern with the actual pattern**: Evaluate the degree of match between the predicted and observed correlations to assess the validity evidence.

<div class="figure">
<div class="grViz html-widget html-fill-item" id="htmlwidget-395105e5a8422300f840" style="width:100%;height:216px;"></div>
<script type="application/json" data-for="htmlwidget-395105e5a8422300f840">{"x":{"diagram":"\ndigraph validity_comparison {\ngraph [bgcolor=transparent];  // Set background to transparent\n\n  # Graph attributes\n  graph [layout = dot, rankdir = LR]\n\n  # Node definitions with styles\n  node [shape = rectangle, style = filled, color = lightblue]\n  L [label = \"Review literature\"]\n  A [label = \"Elicit the predicted pattern\nof validity correlations\"]\n  B [label = \"Estimate\nactual pattern of correlations\"]\n  C [label = \"Compare predicted pattern\nwith actual pattern\"]\n\n  # Result node with different styling\n  node [shape = oval, style = filled, color = lightgrey]\n  D [label = \"Degree of match\"]\n\n  # Edges to connect nodes\n  L -> A [label = \"Formulate predictions\"]\n  A -> B [label = \"Gather data\"]\n  B -> C [label = \"Compare correlations\"]\n  C -> D [label = \"Evaluate match\"]\n\n  # Styling for edges\n  edge [color = darkgreen, penwidth = 2]\n}\n","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>
<p class="caption">The steps in the QCV process.</p>
</div>

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV) (Westen and Rosenthal, 2003)

QCV emphasizes that the theoretical basis of a construct guides the study and interpretation of validity evidence. However, interpretations of validity correlations have often relied on subjective assessments. For instance, researchers may 'eyeball' correlations and make subjective judgments about how well they align with theoretical expectations. Such reliance on subjective evaluations can lead to inconsistencies among researchers.

The QCV procedure is designed to provide a more precise and objective quantitative estimate of support from the overall pattern of evidence (Furr, 2022).

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV) (Westen and Rosenthal, 2003)

A relatively straightforward effect size is `$r_{alerting-CV}$` `$(r_a)$` \eqref{zalerting} the correlation between hypothesized correlations (or equivalently, `$\lambda$` values) and `$z$` transformed actual correlations (\$Z_r\$ values):

`$\lambda = r -\bar{r}$`

`$Z_r =\frac{1}{2}log_e\left(\frac{1+r}{1-r}\right)$`

\begin{align}
r_{ \lambda ,Z_r}=r_a \label{zalerting}
\end{align}

Large `$r_{alerting-CV}$` values ostensibly support a test's convergent and discriminant evidence (in terms of the intended construct).

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV) (Westen and Rosenthal, 2003)

Another effect size is the `$r_{contrast-CV}$`, it also includes the actual and hypothesized correlations, but it includes uses more information. First, it is necessary to calculate the "_remarkableness_ of size of contrast" \eqref{remarkableness} (Westen and Rosenthal, 2003). `$r_x$` is the median of the intercorrelations among the criterion measures.  
As an unstandardized and unfamiliar effect size, 'remarkableness' is not typically reported; however, for a given set of `$\lambda$` weights, larger positive values reflect greater correspondence between hypothesized and actual correlations (with a zero value representing no systematic correspondence) (Furr and Heuckeroth, 2019).

\begin{align}
remarkableness = \frac{\sum\left(\lambda Z_r\right)}{\sqrt{(\sum\lambda^2)\left(1-r_x\right)\left(\frac{1-\bar{r^2}\left(\frac{1-r_x}{2\left(1-\bar{r^2}\right)}\right)}{1-\bar{r^2}}\right)}} \label{remarkableness}
\end{align}

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV) (Westen and Rosenthal, 2003)

`$Remarkableness$` can be used to obtain an inferential test, where `$N$` is the number of participants on which the actual correlations are based \eqref{zcontrast}.

\begin{align}  
Z_{contrast} = remarkableness \sqrt{N-3} \label{zcontrast}  
\end{align}

Using this value, researchers obtain a `$p-value$` (one-sided, based on the standard normal distribution). A significant `$p-value$` as indicates that the 'prediction about the magnitude of correlations could not likely have been obtained by chance' Westen and Rosenthal (2003, p. 612).

---

# Evidence Sources

## Based on Relations to other Variables

### Quantifying Construct Validity (QCV) (Westen and Rosenthal, 2003)

Since `$remarkableness$` is an unstandardized effect size, Westen and Rosenthal (2003) provide `$r_{contrast-CV}$` as a standardized effect size on a familiar metric. To obtain `$r_{contrast-CV}$` \eqref{rcontrast} researchers identify the `$\mathcal{t}$` value associated with the `$p-value$` `$(df = N − 2)$`.<sup>💡</sup> Note that this `$\mathcal{t}$` value is not used for an inferential test; rather it is simply a step through which researchers move from a `$p-value$` to a standardized effect size.

\begin{align}
r_{contrast-CV} = \sqrt{\frac{t^2}{t^2+df}} \label{rcontrast}
\end{align}

``` r
n_size <- 90; p_value <- .0000000000002762477 #example
qt(p = p_value, df = n_size-2, lower.tail = F)
```

```
## [1] 8.451081
```

]

---

# Evidence Sources

## Based on Relations to other Variables

### An example using the `qcv` package

In <svg viewBox="0 0 581 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;" xmlns="http://www.w3.org/2000/svg">  <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"></path></svg> the [`qcv`](https://cran.r-project.org/web/packages/qcv/index.html) package (Furr and Heuckeroth, 2019) provides a straightforward way to implement the QCV procedure.

`n` &mdash; sample size

`actr` &mdash; vector of actual validity correlations

`predr` &mdash; vector of predicted validity correlations (in same order as actr)

`medr` &mdash; median intercorrelation among criterion variables

<div class="pre-name">example-qcv.R</div>

``` r
actrIM  <- c(.46, .13, -.24, -.03, .12, .03, .39, .06, .51, .08, .24, .66)
predrIM <- c(.58, .24, -.04, .06, -.04, .18, .36, .08, .64, .56, .36, .56)

library(qcv)
qcv(n=90, actr=actrIM, predr=predrIM, medr=.075)
```

???

Example by "hand" using the formulae

``` r
hip_cor <- c(.64,.58, .56, .56, .36, .36, .24, .18, .08, .06, -.04, -.04)

lambdas <- scale(hip_cor, center = T, scale = F)

act_cor <- c(.51,.46,.66,.08,.39,.24,.13,.03,.06,-.03,-.24,.12)

zs <- (1/2)*log((1+act_cor)/(1-act_cor))

lamb_zs <- lambdas*zs

#r_alerting_CV
#$\lambda = r -\bar{r};Z_r =\frac{1}{2}log_e\left(\frac{1+r}{1-r}\right); r_{alerting-CV}=r_{\lambda, Z_r}$
(r_alerting_CV <- cor(lambdas, zs))
```

```
##           [,1]
## [1,] 0.7887882
```

``` r
#r_contrast_CV

r_x <- .075 #median inter-correlation value among the convergent/divergent measures (not including the measure of interest) in this case 12 variables (raw data not provided)

#$remarkableness = \frac{\sum\left(\lambda Z_r\right)}{\sqrt{(\sum\lambda^2)\left(1-r_x\right)\left(\frac{1-\bar{r^2}\left(\frac{1-r_x}{2\left(1-\bar{r^2}\right)}\right)}{1-\bar{r^2}}\right)}}$

remarkableness <- sum(lambdas*zs)/sqrt(sum(lambdas^2)*(1-r_x)*((1-mean(act_cor^2)*((1-r_x)/(2*(1-mean(act_cor^2)))))/(1-mean(act_cor^2))))

#z_contrast
#$Z_contrast = remarkableness \sqrt{N-3}$

n_size <- 90
z_contrast <- remarkableness*sqrt(n_size-3)

#obtaining the one-sided p-value
(p_value <- pnorm(z_contrast, lower.tail = F))
```

```
## [1] 0.0000000000002762477
```

``` r
#obtaining the t-value based on p-value and degrees of freedom
(t_value <- qt(p = p_value, df = n_size-2, lower.tail = F))
```

```
## [1] 8.451081
```

``` r
#obtaining the effect size (r_{contrast-cv})
#$r_{contrast-CV} = \sqrt{\frac{t^2}{t^2+df}}$
(r_contrast_CV <- sqrt((t_value^2)/(t_value^2 + n_size-2)))
```

```
## [1] 0.6693292
```

---

# Evidence Sources

## Based on Relations to other Variables

### An example using the `qcv` package

```
## ************  QCV output  ******************************* 
## ********************************************************* 
## 
## ralerting-CV: 0.7888 
## rcontrast-CV: 0.6693         95% CI = 0.5366  to  0.7697 
## zcontrast:    7.2117 
## p:            0.000000000000276247689097745 
## N:            90 
## k:            12 
## szr:          0.2795 
## medr:         0.075 
## rbarsq:       0.1015 
## t:            8.4511 
## remark:       0.7732
```

The data shows substantial correspondence between our theory of the construct and its empirical correlates, thus good convergent and discriminant evidence.

---

# Evidence Sources

## Based on Relations to other Variables

### QCV using SEM

- **QCV Focus**: Targets a small subset of the correlation matrix between a focal test and criterion tests, emphasizing the evaluation of convergent and discriminant evidence.  
  - **Structural Equation Modeling (SEM)**: Offers a broader, flexible framework for analyzing entire correlation matrices, providing fit indices and inferential tests similar to QCV.
  - **SEM Advantages**:  
    - Widely recognized methods and indices.  
    - Expanded analytical possibilities, such as multi-group evaluation of convergent and discriminant validity.  
  - **Potential Integration**: QCV analyses could be subsumed into SEM, potentially eliminating the need for `$r_{alerting-CV}$` and `$r_{contrast-CV}$` tests.

---

# Evidence Sources

## Based on Relations to other Variables

### QCV using SEM

- **Model Specification**: Use SEM software (e.g., R, M<i>plus</i>) to compare a hypothesized correlation matrix (convergent/discriminant validity) with the actual observed matrix.  
    - **Focal Test’s Correlations**: Constrain these to hypothesized values to assess validity.  
  - **Criterion Tests’ Correlations**: Constrain these to observed values to isolate errors in focal test hypotheses.  
  - **Fit Indices**: The model will yield fit indices like `$TLI$`, `$CFI$`, `$SRMR$`, `$RMSEA$`, and `$\chi^2$`, which reflect how well the hypothesized matrix fits the observed data.

---

# Evidence Sources

## Based on Relations to other Variables

### QCV using SEM

``` mplus
TITLE: QCV-like procedure via Mplus (as applied to IM data);
DATA: FILE is /home/ubuntu/Disciplines/PAOWP/assets/data/qcvexample4mplus.dat;
VARIABLE:
  NAMES = imscale dep mach dis res se ext agr comp psc sm anx nfb;
  USEVARIABLES = imscale dep mach dis res se ext agr comp psc sm anx nfb ;
DEFINE:
  STANDARDIZE imscale dep mach dis res se ext agr comp psc sm anx nfb;
ANALYSIS:
  TYPE = GENERAL;
  ESTIMATOR = ML;
MODEL:
! predicted corrs bw focal test and criterion variables (key to QCV)
imscale WITH dep@.58;
imscale WITH mach@.24;
imscale WITH dis@-.04;
imscale WITH res@.06;
imscale WITH se@-.04;
imscale WITH ext@0.18;
imscale WITH agr@0.36;
imscale WITH comp@0.08;
imscale WITH psc@0.64;
imscale WITH sm; !@.56; !that this constraint was freed due to non-convergence
imscale WITH anx@0.36;
imscale WITH nfb@0.56;
!Actual correlations among criterion variables
dep WITH mach@-.02;
dep WITH dis@-.42;
dep WITH res@-.02;
dep WITH se@0.13;
dep WITH ext@0.08;
dep WITH agr@0.53;
dep WITH comp@0.11;
dep WITH psc@0.34;
dep WITH sm@-0.07;
dep WITH anx@-0.08;
dep WITH nfb@0.50;
mach WITH dis@.14;
mach WITH res@.32;
mach WITH se@0.31;
mach WITH ext@0.22;
mach WITH agr@-0.01;
mach WITH comp@0.22;
mach WITH psc@-0.16;
mach WITH sm@0.40;
mach WITH anx@-0.01;
mach WITH nfb@-0.08;
dis WITH res@.07;
dis WITH se@0.03;
dis WITH ext@-0.12;
dis WITH agr@-0.46;
dis WITH comp@0.02;
dis WITH psc@-0.17;
dis WITH sm@0.10;
dis WITH anx@0.17;
dis WITH nfb@-0.30;
res WITH se@0.52;
res WITH ext@0.34;
res WITH agr@0.05;
res WITH comp@0.26;
res WITH psc@-0.46;
res WITH sm@0.28;
res WITH anx@-0.44;
res WITH nfb@-0.12;
se WITH ext@0.21;
se WITH agr@0.22;
se WITH comp@0.46;
se WITH psc@-0.12;
se WITH sm@0.15;
se WITH anx@-0.16;
se WITH nfb@-0.06;
ext WITH agr@0.34;
ext WITH comp@0.40;
ext WITH psc@-0.17;
ext WITH sm@0.56;
ext WITH anx@-0.04;
ext WITH nfb@0.10;
agr WITH comp@0.45;
agr WITH psc@0.26;
agr WITH sm@0.03;
agr WITH anx@0.00;
agr WITH nfb@0.39;
comp WITH psc@-0.02;
comp WITH sm@0.26;
comp WITH anx@-0.19;
comp WITH nfb@-0.06;
psc WITH sm@-0.14;
psc WITH anx@0.55;
psc WITH nfb@0.60;
sm WITH anx@-0.09;
sm WITH nfb@0.03;
anx WITH nfb@0.35;
! fix variances to 1.0
imscale@1;
dep@1;
mach@1;
dis@1;
res@1;
se@1;
ext@1;
agr@1;
comp@1;
psc@1;
sm@1;
anx@1;
nfb@1;
OUTPUT: RES SAMPSTAT MODINDICES;
```

]

---

# Evidence Sources

## Based on Relations to other Variables

### QCV using SEM

```
## Mplus VERSION 8.11 (Linux)
## MUTHEN & MUTHEN
## 11/15/2024   5:10 AM
## 
## INPUT INSTRUCTIONS
## 
##   TITLE: QCV-like procedure via Mplus (as applied to IM data);
##   DATA: FILE is /home/ubuntu/Disciplines/PAOWP/assets/data/qcvexample4mplus.dat;
##   VARIABLE:
##     NAMES = imscale dep mach dis res se ext agr comp psc sm anx nfb;
##     USEVARIABLES = imscale dep mach dis res se ext agr comp psc sm anx nfb ;
##   DEFINE:
##     STANDARDIZE imscale dep mach dis res se ext agr comp psc sm anx nfb;
##   ANALYSIS:
##     TYPE = GENERAL;
##     ESTIMATOR = ML;
##   MODEL:
##   ! predicted corrs bw focal test and criterion variables (key to QCV)
##   imscale WITH dep@.58;
##   imscale WITH mach@.24;
##   imscale WITH dis@-.04;
##   imscale WITH res@.06;
##   imscale WITH se@-.04;
##   imscale WITH ext@0.18;
##   imscale WITH agr@0.36;
##   imscale WITH comp@0.08;
##   imscale WITH psc@0.64;
##   imscale WITH sm; !@.56; !that this constraint was freed due to non-convergence
##   imscale WITH anx@0.36;
##   imscale WITH nfb@0.56;
##   !Actual correlations among criterion variables
##   dep WITH mach@-.02;
##   dep WITH dis@-.42;
##   dep WITH res@-.02;
##   dep WITH se@0.13;
##   dep WITH ext@0.08;
##   dep WITH agr@0.53;
##   dep WITH comp@0.11;
##   dep WITH psc@0.34;
##   dep WITH sm@-0.07;
##   dep WITH anx@-0.08;
##   dep WITH nfb@0.50;
##   mach WITH dis@.14;
##   mach WITH res@.32;
##   mach WITH se@0.31;
##   mach WITH ext@0.22;
##   mach WITH agr@-0.01;
##   mach WITH comp@0.22;
##   mach WITH psc@-0.16;
##   mach WITH sm@0.40;
##   mach WITH anx@-0.01;
##   mach WITH nfb@-0.08;
##   dis WITH res@.07;
##   dis WITH se@0.03;
##   dis WITH ext@-0.12;
##   dis WITH agr@-0.46;
##   dis WITH comp@0.02;
##   dis WITH psc@-0.17;
##   dis WITH sm@0.10;
##   dis WITH anx@0.17;
##   dis WITH nfb@-0.30;
##   res WITH se@0.52;
##   res WITH ext@0.34;
##   res WITH agr@0.05;
##   res WITH comp@0.26;
##   res WITH psc@-0.46;
##   res WITH sm@0.28;
##   res WITH anx@-0.44;
##   res WITH nfb@-0.12;
##   se WITH ext@0.21;
##   se WITH agr@0.22;
##   se WITH comp@0.46;
##   se WITH psc@-0.12;
##   se WITH sm@0.15;
##   se WITH anx@-0.16;
##   se WITH nfb@-0.06;
##   ext WITH agr@0.34;
##   ext WITH comp@0.40;
##   ext WITH psc@-0.17;
##   ext WITH sm@0.56;
##   ext WITH anx@-0.04;
##   ext WITH nfb@0.10;
##   agr WITH comp@0.45;
##   agr WITH psc@0.26;
##   agr WITH sm@0.03;
##   agr WITH anx@0.00;
##   agr WITH nfb@0.39;
##   comp WITH psc@-0.02;
##   comp WITH sm@0.26;
##   comp WITH anx@-0.19;
##   comp WITH nfb@-0.06;
##   psc WITH sm@-0.14;
##   psc WITH anx@0.55;
##   psc WITH nfb@0.60;
##   sm WITH anx@-0.09;
##   sm WITH nfb@0.03;
##   anx WITH nfb@0.35;
##   ! fix variances to 1.0
##   imscale@1;
##   dep@1;
##   mach@1;
##   dis@1;
##   res@1;
##   se@1;
##   ext@1;
##   agr@1;
##   comp@1;
##   psc@1;
##   sm@1;
##   anx@1;
##   nfb@1;
##   OUTPUT: RES SAMPSTAT MODINDICES;
## 
## 
## 
## INPUT READING TERMINATED NORMALLY
## 
## 
## 
## QCV-like procedure via Mplus (as applied to IM data);
## 
## SUMMARY OF ANALYSIS
## 
## Number of groups                                                 1
## Number of observations                                          90
## 
## Number of dependent variables                                   13
## Number of independent variables                                  0
## Number of continuous latent variables                            0
## 
## Observed dependent variables
## 
##   Continuous
##    IMSCALE     DEP         MACH        DIS         RES         SE
##    EXT         AGR         COMP        PSC         SM          ANX
##    NFB
## 
## 
## Estimator                                                       ML
## Information matrix                                        OBSERVED
## Maximum number of iterations                                  1000
## Convergence criterion                                    0.500D-04
## Maximum number of steepest descent iterations                   20
## 
## Input data file(s)
##   /home/ubuntu/Disciplines/PAOWP/assets/data/qcvexample4mplus.dat
## 
## Input data format  FREE
## 
## 
## SAMPLE STATISTICS
## 
## 
##      SAMPLE STATISTICS
## 
## 
##            Means
##               IMSCALE       DEP           MACH          DIS           RES
##               ________      ________      ________      ________      ________
##                 0.000         0.000         0.000         0.000         0.000
## 
## 
##            Means
##               SE            EXT           AGR           COMP          PSC
##               ________      ________      ________      ________      ________
##                 0.000         0.000         0.000         0.000         0.000
## 
## 
##            Means
##               SM            ANX           NFB
##               ________      ________      ________
##                 0.000         0.000         0.000
## 
## 
##            Covariances
##               IMSCALE       DEP           MACH          DIS           RES
##               ________      ________      ________      ________      ________
##  IMSCALE        1.000
##  DEP            0.457         1.000
##  MACH           0.127        -0.025         1.000
##  DIS           -0.243        -0.419         0.143         1.000
##  RES           -0.035        -0.017         0.322         0.074         1.000
##  SE             0.120         0.131         0.312         0.033         0.516
##  EXT            0.029         0.076         0.221        -0.120         0.339
##  AGR            0.391         0.530        -0.015        -0.459         0.051
##  COMP           0.060         0.107         0.224         0.017         0.255
##  PSC            0.507         0.337        -0.158        -0.166        -0.457
##  SM             0.084        -0.072         0.396         0.096         0.282
##  ANX            0.236        -0.080        -0.006         0.174        -0.442
##  NFB            0.661         0.498        -0.080        -0.297        -0.124
## 
## 
##            Covariances
##               SE            EXT           AGR           COMP          PSC
##               ________      ________      ________      ________      ________
##  SE             1.000
##  EXT            0.212         1.000
##  AGR            0.223         0.345         1.000
##  COMP           0.461         0.399         0.452         1.000
##  PSC           -0.115        -0.166         0.262        -0.022         1.000
##  SM             0.149         0.557         0.025         0.258        -0.137
##  ANX           -0.158        -0.044        -0.002        -0.195         0.546
##  NFB           -0.065         0.098         0.394        -0.059         0.603
## 
## 
##            Covariances
##               SM            ANX           NFB
##               ________      ________      ________
##  SM             1.000
##  ANX           -0.089         1.000
##  NFB            0.026         0.351         1.000
## 
## 
##            Correlations
##               IMSCALE       DEP           MACH          DIS           RES
##               ________      ________      ________      ________      ________
##  IMSCALE        1.000
##  DEP            0.457         1.000
##  MACH           0.127        -0.025         1.000
##  DIS           -0.243        -0.419         0.143         1.000
##  RES           -0.035        -0.017         0.322         0.074         1.000
##  SE             0.120         0.131         0.312         0.033         0.516
##  EXT            0.029         0.076         0.221        -0.120         0.339
##  AGR            0.391         0.530        -0.015        -0.459         0.051
##  COMP           0.060         0.107         0.224         0.017         0.255
##  PSC            0.507         0.337        -0.158        -0.166        -0.457
##  SM             0.084        -0.072         0.396         0.096         0.282
##  ANX            0.236        -0.080        -0.006         0.174        -0.442
##  NFB            0.661         0.498        -0.080        -0.297        -0.124
## 
## 
##            Correlations
##               SE            EXT           AGR           COMP          PSC
##               ________      ________      ________      ________      ________
##  SE             1.000
##  EXT            0.212         1.000
##  AGR            0.223         0.345         1.000
##  COMP           0.461         0.399         0.452         1.000
##  PSC           -0.115        -0.166         0.262        -0.022         1.000
##  SM             0.149         0.557         0.025         0.258        -0.137
##  ANX           -0.158        -0.044        -0.002        -0.195         0.546
##  NFB           -0.065         0.098         0.394        -0.059         0.603
## 
## 
##            Correlations
##               SM            ANX           NFB
##               ________      ________      ________
##  SM             1.000
##  ANX           -0.089         1.000
##  NFB            0.026         0.351         1.000
## 
## 
## UNIVARIATE SAMPLE STATISTICS
## 
## 
##      UNIVARIATE HIGHER-ORDER MOMENT DESCRIPTIVE STATISTICS
## 
##          Variable/         Mean/     Skewness/   Minimum/ % with                Percentiles
##         Sample Size      Variance    Kurtosis    Maximum  Min/Max      20%/60%    40%/80%    Median
## 
##      IMSCALE               0.000       0.316      -1.903    2.22%      -0.747     -0.362     -0.169
##               90.000       1.000      -0.527       2.335    2.22%       0.216      0.987
##      DEP                   0.000       0.251      -2.192    2.22%      -0.599     -0.333     -0.068
##               90.000       1.000       0.080       2.587    1.11%       0.198      0.729
##      MACH                  0.000      -0.406      -2.682    2.22%      -0.871     -0.095      0.164
##               90.000       1.000      -0.005       2.233    1.11%       0.423      0.681
##      DIS                   0.000       0.308      -2.613    1.11%      -0.881     -0.361     -0.015
##               90.000       1.000       0.364       2.931    1.11%       0.159      0.678
##      RES                   0.000      -0.102      -2.397    1.11%      -0.955     -0.235      0.005
##               90.000       1.000      -0.297       2.167    2.22%       0.246      0.726
##      SE                    0.000      -0.222      -3.759    1.11%      -0.782     -0.241      0.030
##               90.000       1.000       1.626       2.466    2.22%       0.301      0.571
##      EXT                   0.000      -0.166      -2.035    3.33%      -0.854     -0.329      0.064
##               90.000       1.000      -0.871       1.902    2.22%       0.429      0.852
##      AGR                   0.000      -0.608      -3.212    1.11%      -0.689     -0.148      0.032
##               90.000       1.000       0.323       1.654    2.22%       0.212      0.933
##      COMP                  0.000       0.125      -2.849    1.11%      -0.934     -0.215      0.024
##               90.000       1.000       0.361       2.418    2.22%       0.024      0.742
##      PSC                   0.000       0.469      -2.095    1.11%      -0.848     -0.225     -0.069
##               90.000       1.000       0.033       2.736    2.22%       0.087      0.866
##      SM                    0.000       0.290      -2.499    1.11%      -0.854     -0.306     -0.032
##               90.000       1.000       0.322       2.572    2.22%       0.105      0.653
##      ANX                   0.000       0.421      -2.415    1.11%      -0.781     -0.336     -0.187
##               90.000       1.000       0.041       2.487    1.11%       0.110      0.704
##      NFB                   0.000       0.103      -2.266    1.11%      -0.801     -0.215     -0.068
##               90.000       1.000      -0.176       2.423    1.11%       0.225      0.811
## 
## 
## THE MODEL ESTIMATION TERMINATED NORMALLY
## 
## 
## 
## MODEL FIT INFORMATION
## 
## Number of Free Parameters                       14
## 
## Loglikelihood
## 
##           H0 Value                       -1731.127
##           H1 Value                       -1414.455
## 
## Information Criteria
## 
##           Akaike (AIC)                    3490.254
##           Bayesian (BIC)                  3525.252
##           Sample-Size Adjusted BIC        3481.067
##             (n* = (n + 2) / 24)
## 
*## Chi-Square Test of Model Fit
*## 
*##           Value                            633.345
*##           Degrees of Freedom                    90
*##           P-Value                           0.0000
*## 
*## RMSEA (Root Mean Square Error Of Approximation)
*## 
*##           Estimate                           0.259
*##           90 Percent C.I.                    0.240  0.278
*##           Probability RMSEA <= .05           0.000
*## 
*## CFI/TLI
*## 
*##           CFI                                0.000
*##           TLI                                0.000
*## 
*## Chi-Square Test of Model Fit for the Baseline Model
*## 
*##           Value                            491.407
*##           Degrees of Freedom                    78
*##           P-Value                           0.0000
*## 
*## SRMR (Standardized Root Mean Square Residual)
*## 
*##           Value                              0.041
## 
## 
## 
## MODEL RESULTS
## 
##                                                     Two-Tailed
##                     Estimate       S.E.  Est./S.E.    P-Value
## 
##  IMSCALE  WITH
##     DEP                0.580      0.000    999.000    999.000
##     MACH               0.240      0.000    999.000    999.000
##     DIS               -0.040      0.000    999.000    999.000
##     RES                0.060      0.000    999.000    999.000
##     SE                -0.040      0.000    999.000    999.000
##     EXT                0.180      0.000    999.000    999.000
##     AGR                0.360      0.000    999.000    999.000
##     COMP               0.080      0.000    999.000    999.000
##     PSC                0.640      0.000    999.000    999.000
##     SM                 0.135      0.008     17.859      0.000
##     ANX                0.360      0.000    999.000    999.000
##     NFB                0.560      0.000    999.000    999.000
## 
##  DEP      WITH
##     MACH              -0.020      0.000    999.000    999.000
##     DIS               -0.420      0.000    999.000    999.000
##     RES               -0.020      0.000    999.000    999.000
##     SE                 0.130      0.000    999.000    999.000
##     EXT                0.080      0.000    999.000    999.000
##     AGR                0.530      0.000    999.000    999.000
##     COMP               0.110      0.000    999.000    999.000
##     PSC                0.340      0.000    999.000    999.000
##     SM                -0.070      0.000    999.000    999.000
##     ANX               -0.080      0.000    999.000    999.000
##     NFB                0.500      0.000    999.000    999.000
## 
##  MACH     WITH
##     DIS                0.140      0.000    999.000    999.000
##     RES                0.320      0.000    999.000    999.000
##     SE                 0.310      0.000    999.000    999.000
##     EXT                0.220      0.000    999.000    999.000
##     AGR               -0.010      0.000    999.000    999.000
##     COMP               0.220      0.000    999.000    999.000
##     PSC               -0.160      0.000    999.000    999.000
##     SM                 0.400      0.000    999.000    999.000
##     ANX               -0.010      0.000    999.000    999.000
##     NFB               -0.080      0.000    999.000    999.000
## 
##  DIS      WITH
##     RES                0.070      0.000    999.000    999.000
##     SE                 0.030      0.000    999.000    999.000
##     EXT               -0.120      0.000    999.000    999.000
##     AGR               -0.460      0.000    999.000    999.000
##     COMP               0.020      0.000    999.000    999.000
##     PSC               -0.170      0.000    999.000    999.000
##     SM                 0.100      0.000    999.000    999.000
##     ANX                0.170      0.000    999.000    999.000
##     NFB               -0.300      0.000    999.000    999.000
## 
##  RES      WITH
##     SE                 0.520      0.000    999.000    999.000
##     EXT                0.340      0.000    999.000    999.000
##     AGR                0.050      0.000    999.000    999.000
##     COMP               0.260      0.000    999.000    999.000
##     PSC               -0.460      0.000    999.000    999.000
##     SM                 0.280      0.000    999.000    999.000
##     ANX               -0.440      0.000    999.000    999.000
##     NFB               -0.120      0.000    999.000    999.000
## 
##  SE       WITH
##     EXT                0.210      0.000    999.000    999.000
##     AGR                0.220      0.000    999.000    999.000
##     COMP               0.460      0.000    999.000    999.000
##     PSC               -0.120      0.000    999.000    999.000
##     SM                 0.150      0.000    999.000    999.000
##     ANX               -0.160      0.000    999.000    999.000
##     NFB               -0.060      0.000    999.000    999.000
## 
##  EXT      WITH
##     AGR                0.340      0.000    999.000    999.000
##     COMP               0.400      0.000    999.000    999.000
##     PSC               -0.170      0.000    999.000    999.000
##     SM                 0.560      0.000    999.000    999.000
##     ANX               -0.040      0.000    999.000    999.000
##     NFB                0.100      0.000    999.000    999.000
## 
##  AGR      WITH
##     COMP               0.450      0.000    999.000    999.000
##     PSC                0.260      0.000    999.000    999.000
##     SM                 0.030      0.000    999.000    999.000
##     ANX                0.000      0.000    999.000    999.000
##     NFB                0.390      0.000    999.000    999.000
## 
##  COMP     WITH
##     PSC               -0.020      0.000    999.000    999.000
##     SM                 0.260      0.000    999.000    999.000
##     ANX               -0.190      0.000    999.000    999.000
##     NFB               -0.060      0.000    999.000    999.000
## 
##  PSC      WITH
##     SM                -0.140      0.000    999.000    999.000
##     ANX                0.550      0.000    999.000    999.000
##     NFB                0.600      0.000    999.000    999.000
## 
##  SM       WITH
##     ANX               -0.090      0.000    999.000    999.000
##     NFB                0.030      0.000    999.000    999.000
## 
##  ANX      WITH
##     NFB                0.350      0.000    999.000    999.000
## 
##  Means
##     IMSCALE            0.000      0.105      0.000      1.000
##     DEP                0.000      0.105      0.000      1.000
##     MACH               0.000      0.105      0.000      1.000
##     DIS                0.000      0.105      0.000      1.000
##     RES                0.000      0.105      0.000      1.000
##     SE                 0.000      0.105      0.000      1.000
##     EXT                0.000      0.105      0.000      1.000
##     AGR                0.000      0.105      0.000      1.000
##     COMP               0.000      0.105      0.000      1.000
##     PSC                0.000      0.105      0.000      1.000
##     SM                 0.000      0.105      0.000      1.000
##     ANX                0.000      0.105      0.000      1.000
##     NFB                0.000      0.105      0.000      1.000
## 
##  Variances
##     IMSCALE            1.000      0.000    999.000    999.000
##     DEP                1.000      0.000    999.000    999.000
##     MACH               1.000      0.000    999.000    999.000
##     DIS                1.000      0.000    999.000    999.000
##     RES                1.000      0.000    999.000    999.000
##     SE                 1.000      0.000    999.000    999.000
##     EXT                1.000      0.000    999.000    999.000
##     AGR                1.000      0.000    999.000    999.000
##     COMP               1.000      0.000    999.000    999.000
##     PSC                1.000      0.000    999.000    999.000
##     SM                 1.000      0.000    999.000    999.000
##     ANX                1.000      0.000    999.000    999.000
##     NFB                1.000      0.000    999.000    999.000
## 
## 
## QUALITY OF NUMERICAL RESULTS
## 
##      Condition Number for the Information Matrix              0.158E-02
##        (ratio of smallest to largest eigenvalue)
## 
## 
## RESIDUAL OUTPUT
## 
## 
##      ESTIMATED MODEL AND RESIDUALS (OBSERVED - ESTIMATED)
## 
## 
##            Model Estimated Means
##               IMSCALE       DEP           MACH          DIS           RES
##               ________      ________      ________      ________      ________
##                 0.000         0.000         0.000         0.000         0.000
## 
## 
##            Model Estimated Means
##               SE            EXT           AGR           COMP          PSC
##               ________      ________      ________      ________      ________
##                 0.000         0.000         0.000         0.000         0.000
## 
## 
##            Model Estimated Means
##               SM            ANX           NFB
##               ________      ________      ________
##                 0.000         0.000         0.000
## 
## 
##            Residuals for Means
##               IMSCALE       DEP           MACH          DIS           RES
##               ________      ________      ________      ________      ________
##                 0.000         0.000         0.000         0.000         0.000
## 
## 
##            Residuals for Means
##               SE            EXT           AGR           COMP          PSC
##               ________      ________      ________      ________      ________
##                 0.000         0.000         0.000         0.000         0.000
## 
## 
##            Residuals for Means
##               SM            ANX           NFB
##               ________      ________      ________
##                 0.000         0.000         0.000
## 
## 
##            Standardized Residuals (z-scores) for Means
##               IMSCALE       DEP           MACH          DIS           RES
##               ________      ________      ________      ________      ________
##                 0.000         0.000         0.000         0.000         0.000
## 
## 
##            Standardized Residuals (z-scores) for Means
##               SE            EXT           AGR           COMP          PSC
##               ________      ________      ________      ________      ________
##                 0.000         0.000         0.000         0.000         0.000
## 
## 
##            Standardized Residuals (z-scores) for Means
##               SM            ANX           NFB
##               ________      ________      ________
##                 0.000         0.000         0.000
## 
## 
##            Normalized Residuals for Means
##               IMSCALE       DEP           MACH          DIS           RES
##               ________      ________      ________      ________      ________
##                 0.000         0.000         0.000         0.000         0.000
## 
## 
##            Normalized Residuals for Means
##               SE            EXT           AGR           COMP          PSC
##               ________      ________      ________      ________      ________
##                 0.000         0.000         0.000         0.000         0.000
## 
## 
##            Normalized Residuals for Means
##               SM            ANX           NFB
##               ________      ________      ________
##                 0.000         0.000         0.000
## 
## 
##            Model Estimated Covariances
##               IMSCALE       DEP           MACH          DIS           RES
##               ________      ________      ________      ________      ________
##  IMSCALE        1.000
##  DEP            0.580         1.000
##  MACH           0.240        -0.020         1.000
##  DIS           -0.040        -0.420         0.140         1.000
##  RES            0.060        -0.020         0.320         0.070         1.000
##  SE            -0.040         0.130         0.310         0.030         0.520
##  EXT            0.180         0.080         0.220        -0.120         0.340
##  AGR            0.360         0.530        -0.010        -0.460         0.050
##  COMP           0.080         0.110         0.220         0.020         0.260
##  PSC            0.640         0.340        -0.160        -0.170        -0.460
##  SM             0.135        -0.070         0.400         0.100         0.280
##  ANX            0.360        -0.080        -0.010         0.170        -0.440
##  NFB            0.560         0.500        -0.080        -0.300        -0.120
## 
## 
##            Model Estimated Covariances
##               SE            EXT           AGR           COMP          PSC
##               ________      ________      ________      ________      ________
##  SE             1.000
##  EXT            0.210         1.000
##  AGR            0.220         0.340         1.000
##  COMP           0.460         0.400         0.450         1.000
##  PSC           -0.120        -0.170         0.260        -0.020         1.000
##  SM             0.150         0.560         0.030         0.260        -0.140
##  ANX           -0.160        -0.040         0.000        -0.190         0.550
##  NFB           -0.060         0.100         0.390        -0.060         0.600
## 
## 
##            Model Estimated Covariances
##               SM            ANX           NFB
##               ________      ________      ________
##  SM             1.000
##  ANX           -0.090         1.000
##  NFB            0.030         0.350         1.000
## 
## 
##            Model Estimated Correlations
##               IMSCALE       DEP           MACH          DIS           RES
##               ________      ________      ________      ________      ________
##  IMSCALE        1.000
##  DEP            0.580         1.000
##  MACH           0.240        -0.020         1.000
##  DIS           -0.040        -0.420         0.140         1.000
##  RES            0.060        -0.020         0.320         0.070         1.000
##  SE            -0.040         0.130         0.310         0.030         0.520
##  EXT            0.180         0.080         0.220        -0.120         0.340
##  AGR            0.360         0.530        -0.010        -0.460         0.050
##  COMP           0.080         0.110         0.220         0.020         0.260
##  PSC            0.640         0.340        -0.160        -0.170        -0.460
##  SM             0.135        -0.070         0.400         0.100         0.280
##  ANX            0.360        -0.080        -0.010         0.170        -0.440
##  NFB            0.560         0.500        -0.080        -0.300        -0.120
## 
## 
##            Model Estimated Correlations
##               SE            EXT           AGR           COMP          PSC
##               ________      ________      ________      ________      ________
##  SE             1.000
##  EXT            0.210         1.000
##  AGR            0.220         0.340         1.000
##  COMP           0.460         0.400         0.450         1.000
##  PSC           -0.120        -0.170         0.260        -0.020         1.000
##  SM             0.150         0.560         0.030         0.260        -0.140
##  ANX           -0.160        -0.040         0.000        -0.190         0.550
##  NFB           -0.060         0.100         0.390        -0.060         0.600
## 
## 
##            Model Estimated Correlations
##               SM            ANX           NFB
##               ________      ________      ________
##  SM             1.000
##  ANX           -0.090         1.000
##  NFB            0.030         0.350         1.000
## 
## 
##            Residuals for Covariances
##               IMSCALE       DEP           MACH          DIS           RES
##               ________      ________      ________      ________      ________
##  IMSCALE        0.000
##  DEP           -0.123         0.000
##  MACH          -0.113        -0.005         0.000
##  DIS           -0.203         0.001         0.003         0.000
##  RES           -0.095         0.003         0.002         0.004         0.000
##  SE             0.160         0.001         0.002         0.003        -0.004
##  EXT           -0.151        -0.004         0.001         0.000        -0.001
##  AGR            0.031         0.000        -0.005         0.001         0.001
##  COMP          -0.020        -0.003         0.004        -0.003        -0.005
##  PSC           -0.133        -0.003         0.002         0.004         0.003
##  SM            -0.051        -0.002        -0.004        -0.004         0.002
##  ANX           -0.124         0.000         0.004         0.004        -0.002
##  NFB            0.101        -0.002         0.000         0.003        -0.004
## 
## 
##            Residuals for Covariances
##               SE            EXT           AGR           COMP          PSC
##               ________      ________      ________      ________      ________
##  SE             0.000
##  EXT            0.002         0.000
##  AGR            0.003         0.005         0.000
##  COMP           0.001        -0.001         0.002         0.000
##  PSC            0.005         0.004         0.002        -0.002         0.000
##  SM            -0.001        -0.003        -0.005        -0.002         0.003
##  ANX            0.002        -0.004        -0.002        -0.005        -0.004
##  NFB           -0.005        -0.002         0.004         0.001         0.003
## 
## 
##            Residuals for Covariances
##               SM            ANX           NFB
##               ________      ________      ________
##  SM             0.000
##  ANX            0.001         0.000
##  NFB           -0.004         0.001         0.000
## 
## 
##            Residuals for Correlations
##               IMSCALE       DEP           MACH          DIS           RES
##               ________      ________      ________      ________      ________
##  IMSCALE        0.000
##  DEP           -0.123         0.000
##  MACH          -0.113        -0.005         0.000
##  DIS           -0.203         0.001         0.003         0.000
##  RES           -0.095         0.003         0.002         0.004         0.000
##  SE             0.160         0.001         0.002         0.003        -0.004
##  EXT           -0.151        -0.004         0.001         0.000        -0.001
##  AGR            0.031         0.000        -0.005         0.001         0.001
##  COMP          -0.020        -0.003         0.004        -0.003        -0.005
##  PSC           -0.133        -0.003         0.002         0.004         0.003
##  SM            -0.051        -0.002        -0.004        -0.004         0.002
##  ANX           -0.124         0.000         0.004         0.004        -0.002
##  NFB            0.101        -0.002         0.000         0.003        -0.004
## 
## 
##            Residuals for Correlations
##               SE            EXT           AGR           COMP          PSC
##               ________      ________      ________      ________      ________
##  SE             0.000
##  EXT            0.002         0.000
##  AGR            0.003         0.005         0.000
##  COMP           0.001        -0.001         0.002         0.000
##  PSC            0.005         0.004         0.002        -0.002         0.000
##  SM            -0.001        -0.003        -0.005        -0.002         0.003
##  ANX            0.002        -0.004        -0.002        -0.005        -0.004
##  NFB           -0.005        -0.002         0.004         0.001         0.003
## 
## 
##            Residuals for Correlations
##               SM            ANX           NFB
##               ________      ________      ________
##  SM             0.000
##  ANX            0.001         0.000
##  NFB           -0.004         0.001         0.000
## 
## 
##            Standardized Residuals (z-scores) for Covariances
##               IMSCALE       DEP           MACH          DIS           RES
##               ________      ________      ________      ________      ________
##  IMSCALE        0.000
##  DEP           -1.064         0.000
##  MACH          -1.065        -0.045         0.000
##  DIS           -1.868         0.011         0.027         0.000
##  RES           -0.897         0.025         0.020         0.035         0.000
##  SE             1.510         0.005         0.022         0.030        -0.032
##  EXT           -1.430        -0.043         0.007         0.003        -0.005
##  AGR            0.275         0.000        -0.043         0.010         0.012
##  COMP          -0.185        -0.030         0.036        -0.025        -0.042
##  PSC           -1.129        -0.029         0.019         0.041         0.025
##  SM            -0.487        -0.016        -0.035        -0.039         0.017
##  ANX           -1.142         0.004         0.035         0.034        -0.016
##  NFB            0.802        -0.014         0.005         0.026        -0.034
## 
## 
##            Standardized Residuals (z-scores) for Covariances
##               SE            EXT           AGR           COMP          PSC
##               ________      ________      ________      ________      ________
##  SE             0.000
##  EXT            0.014         0.000
##  AGR            0.031         0.048         0.000
##  COMP           0.006        -0.007         0.016         0.000
##  PSC            0.047         0.039         0.023        -0.015         0.000
##  SM            -0.010        -0.021        -0.046        -0.020         0.028
##  ANX            0.017        -0.040        -0.016        -0.043        -0.033
##  NFB           -0.044        -0.015         0.039         0.013         0.025
## 
## 
##            Standardized Residuals (z-scores) for Covariances
##               SM            ANX           NFB
##               ________      ________      ________
##  SM             0.000
##  ANX            0.010         0.000
##  NFB           -0.042         0.007         0.000
## 
## 
##            Normalized Residuals for Covariances
##               IMSCALE       DEP           MACH          DIS           RES
##               ________      ________      ________      ________      ________
##  IMSCALE        0.000
##  DEP           -1.064         0.000
##  MACH          -1.065        -0.045         0.000
##  DIS           -1.868         0.011         0.027         0.000
##  RES           -0.897         0.025         0.020         0.035         0.000
##  SE             1.510         0.005         0.022         0.030        -0.032
##  EXT           -1.430        -0.043         0.007         0.003        -0.005
##  AGR            0.275         0.000        -0.043         0.010         0.012
##  COMP          -0.185        -0.030         0.036        -0.025        -0.042
##  PSC           -1.129        -0.029         0.019         0.041         0.025
##  SM            -0.486        -0.016        -0.035        -0.039         0.017
##  ANX           -1.142         0.004         0.035         0.034        -0.016
##  NFB            0.802        -0.014         0.005         0.026        -0.034
## 
## 
##            Normalized Residuals for Covariances
##               SE            EXT           AGR           COMP          PSC
##               ________      ________      ________      ________      ________
##  SE             0.000
##  EXT            0.014         0.000
##  AGR            0.031         0.048         0.000
##  COMP           0.006        -0.007         0.016         0.000
##  PSC            0.047         0.039         0.023        -0.015         0.000
##  SM            -0.010        -0.021        -0.046        -0.020         0.028
##  ANX            0.017        -0.040        -0.016        -0.043        -0.033
##  NFB           -0.044        -0.015         0.039         0.013         0.025
## 
## 
##            Normalized Residuals for Covariances
##               SM            ANX           NFB
##               ________      ________      ________
##  SM             0.000
##  ANX            0.010         0.000
##  NFB           -0.042         0.007         0.000
## 
## 
## MODEL MODIFICATION INDICES
## 
## NOTE:  Modification indices for direct effects of observed dependent variables
## regressed on covariates may not be included.  To include these, request
## MODINDICES (ALL).
## 
## Minimum M.I. value for printing the modification index    10.000
## 
##                                    M.I.     E.P.C.  Std E.P.C.  StdYX E.P.C.
## 
## WITH Statements
## 
## DEP      WITH IMSCALE           3048.870    -0.629     -0.629       -0.629
## MACH     WITH IMSCALE           1913.267    -0.853     -0.853       -0.853
## MACH     WITH DEP               1546.882     1.214      1.214        1.214
## DIS      WITH IMSCALE           1785.081    -0.803     -0.803       -0.803
## DIS      WITH DEP               1334.650     1.064      1.064        1.064
## DIS      WITH MACH               761.327     1.303      1.303        1.303
## RES      WITH IMSCALE           2987.673    -0.609     -0.609       -0.609
## RES      WITH DEP               2396.054     0.862      0.862        0.862
## RES      WITH MACH              1545.764     1.197      1.197        1.197
## RES      WITH DIS               1401.204     1.101      1.101        1.101
## SE       WITH IMSCALE           2810.893     0.756      0.756        0.756
## SE       WITH DEP               2154.608    -1.027     -1.027       -1.027
## SE       WITH MACH              1253.385    -1.284     -1.284       -1.284
## SE       WITH DIS               1208.464    -1.261     -1.261       -1.261
## SE       WITH RES               1819.149    -0.857     -0.857       -0.857
## EXT      WITH IMSCALE           1049.085    -0.727     -0.727       -0.727
## EXT      WITH DEP                673.130     0.837      0.837        0.837
## EXT      WITH MACH               344.349     0.908      0.908        0.908
## EXT      WITH DIS                283.551     0.765      0.765        0.765
## EXT      WITH RES                680.042     0.833      0.833        0.833
## EXT      WITH SE                 578.669    -0.943     -0.943       -0.943
## AGR      WITH IMSCALE            190.258    -0.278     -0.278       -0.278
## AGR      WITH DEP                142.969     0.363      0.363        0.363
## AGR      WITH MACH                68.113     0.378      0.378        0.378
## AGR      WITH DIS                 57.203     0.316      0.316        0.316
## AGR      WITH RES                136.618     0.342      0.342        0.342
## AGR      WITH SE                 119.426    -0.396     -0.396       -0.396
## AGR      WITH EXT                 28.185     0.231      0.231        0.231
## COMP     WITH IMSCALE            104.610     0.224      0.224        0.224
## COMP     WITH DEP                 75.625    -0.286     -0.286       -0.286
## COMP     WITH MACH                36.530    -0.301     -0.301       -0.301
## COMP     WITH DIS                 32.419    -0.266     -0.266       -0.266
## COMP     WITH RES                 74.281    -0.275     -0.275       -0.275
## COMP     WITH SE                  62.625     0.310      0.310        0.310
## COMP     WITH EXT                 13.711    -0.169     -0.169       -0.169
## PSC      WITH IMSCALE           3151.931    -0.487     -0.487       -0.487
## PSC      WITH DEP               2818.155     0.769      0.769        0.769
## PSC      WITH MACH              1555.607     0.913      0.913        0.913
## PSC      WITH DIS               1419.650     0.844      0.844        0.844
## PSC      WITH RES               2400.161     0.646      0.646        0.646
## PSC      WITH SE                2374.141    -0.847     -0.847       -0.847
## PSC      WITH EXT                688.005     0.634      0.634        0.634
## PSC      WITH AGR                151.934     0.288      0.288        0.288
## PSC      WITH COMP                79.466    -0.222     -0.222       -0.222
## ANX      WITH IMSCALE            982.086    -0.579     -0.579       -0.579
## ANX      WITH DEP                640.768     0.664      0.664        0.664
## ANX      WITH MACH               384.845     0.863      0.863        0.863
## ANX      WITH DIS                346.052     0.789      0.789        0.789
## ANX      WITH RES                604.061     0.622      0.622        0.622
## ANX      WITH SE                 585.359    -0.797     -0.797       -0.797
## ANX      WITH EXT                131.642     0.473      0.473        0.473
## ANX      WITH AGR                 31.234     0.228      0.228        0.228
## ANX      WITH COMP                15.871    -0.171     -0.171       -0.171
## ANX      WITH PSC                830.863     0.650      0.650        0.650
## NFB      WITH IMSCALE            871.773     0.593      0.593        0.593
## NFB      WITH DEP                540.017    -0.665     -0.665       -0.665
## NFB      WITH MACH               311.721    -0.819     -0.819       -0.819
## NFB      WITH DIS                272.881    -0.743     -0.743       -0.743
## NFB      WITH RES                529.755    -0.646     -0.646       -0.646
## NFB      WITH SE                 476.215     0.779      0.779        0.779
## NFB      WITH EXT                100.676    -0.438     -0.438       -0.438
## NFB      WITH AGR                 26.238    -0.217     -0.217       -0.217
## NFB      WITH COMP                13.600     0.169      0.169        0.169
## NFB      WITH PSC                567.573    -0.522     -0.522       -0.522
## NFB      WITH ANX                124.989    -0.441     -0.441       -0.441
## 
## Variances/Residual Variances
## 
## IMSCALE                         3362.674     0.792      0.792        0.792
## DEP                             1740.398     1.267      1.267        1.267
## MACH                             462.309     1.563      1.563        1.563
## DIS                              364.443     1.269      1.269        1.269
## RES                             1602.242     1.139      1.139        1.139
## SE                              1239.733     1.548      1.548        1.548
## EXT                               58.279     0.492      0.492        0.492
## PSC                             2025.152     0.825      0.825        0.825
## ANX                               86.421     0.509      0.509        0.509
## NFB                               43.894     0.404      0.404        0.404
## 
## 
##      Beginning Time:  05:10:04
##         Ending Time:  05:10:04
##        Elapsed Time:  00:00:00
## 
## 
## 
## MUTHEN & MUTHEN
## 3463 Stoner Ave.
## Los Angeles, CA  90066
## 
## Tel: (310) 391-9971
## Fax: (310) 391-8971
## Web: www.StatModel.com
## Support: Support@StatModel.com
## 
## Copyright (c) 1998-2024 Muthen & Muthen
```

]

---

# Evidence Sources

## Based on Consequences of Testing

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[

**Evidence Based on Consequences of Testing** &mdash; intended and unintended consequences of test use; differential consequences of test use; impact of assessment on respondents, instructors, organizations, society; impact of assessments on curriculum; cost/benefit analysis with respect to tradeoff between work/instructional time and assessment time

---

# Evidence Sources

## Based on Consequences of Testing

Examples (Downing and Yudkowsky, 2009, p. 30):

• Impact of test scores/results on respondents/society  
  • Consequences on learners/future learning  
  • Positive consequences outweigh unintended negative consequences?  
  • Reasonableness of method of establishing pass–fail (cut) score  
  • Pass/fail consequences  
  • P/F Decision reliability—Classification accuracy  
  • Conditional Standard Error of Measurement at pass score `$(CSEM)$` (Huebner and Skar, 2021)  
  • False positives/negatives  
  • Instructional/learner consequences

---

# References    
American Educational Research Association, American Psychological
Association, et al. (2014). _Standards for Educational and
Psychological Testing_. Washington, DC: American Psychological
Association. ISBN: 9780935302356.

Andrade, C. (2018). "Internal, external, and ecological validity in
research design, conduct, and evaluation". In: _Indian Journal of
Psychological Medicine_ 40 (5), pp. 498-499. ISSN: 0253-7176. DOI:
[10.4103/IJPSYM.IJPSYM_334_18](https://doi.org/10.4103%2FIJPSYM.IJPSYM_334_18).
URL:
[https://journals.sagepub.com/doi/10.4103/IJPSYM.IJPSYM_334_18](https://journals.sagepub.com/doi/10.4103/IJPSYM.IJPSYM_334_18).

Ayre, C. and A. J. Scally (2014). "Critical values for Lawshe's Content
Validity Ratio". In: _Measurement and Evaluation in Counseling and
Development_ 47.1, pp. 79-86. ISSN: 0748-1756. DOI:
[10.1177/0748175613513808](https://doi.org/10.1177%2F0748175613513808).
URL:
[https://www.tandfonline.com/doi/full/10.1177/0748175613513808](https://www.tandfonline.com/doi/full/10.1177/0748175613513808).

Beckman, T. J., D. A. Cook, et al. (2005). "What is the validity
evidence for assessments of clinical teaching?" In: _Journal of General
Internal Medicine_ 20 (12), pp. 1159-1164. ISSN: 08848734. DOI:
[10.1111/j.1525-1497.2005.0258.x](https://doi.org/10.1111%2Fj.1525-1497.2005.0258.x).

Buckingham, B. R., W. A. McCall, et al. (1921). "Report of the
Standardization Committee". In: _Journal of Educational Research_ 4.1,
pp. 78-80. ISSN: 00220671, 19400675. URL:
[http://www.jstor.org/stable/27524508](http://www.jstor.org/stable/27524508).

---
# References    
Campbell, D. T. and D. W. Fiske (1959). "Convergent and discriminant
validation by the multitrait-multimethod matrix". In: _Psychological
Bulletin_ 56.2, pp. 81-105. ISSN: 0033-2909. DOI:
[10.1037/h0046016](https://doi.org/10.1037%2Fh0046016). URL:
[http://content.apa.org/journals/bul/56/2/81](http://content.apa.org/journals/bul/56/2/81).

Crocker, L. and J. Algina (2008). _Introduction to classical and modern
test theory_. Mason, OH: Cencage Learning. ISBN: 978-0-495-39591-1.

Cronbach, L. J. and P. E. Meehl (1955). "Construct validity in
psychological tests." In: _Psychological Bulletin_ 52 (4), pp. 281-302.
ISSN: 0033-2909. DOI:
[10.1037/h0040957](https://doi.org/10.1037%2Fh0040957). URL:
[http://www.ncbi.nlm.nih.gov/pubmed/19586159
http://content.apa.org/journals/bul/52/4/281](http://www.ncbi.nlm.nih.gov/pubmed/19586159
http://content.apa.org/journals/bul/52/4/281).

Davis, L. L. (1992). "Instrument review: Getting the most from a panel
of experts". In: _Applied Nursing Research_ 5.4, pp. 194-197. ISSN:
08971897. DOI:
[10.1016/S0897-1897(05)80008-4](https://doi.org/10.1016%2FS0897-1897%2805%2980008-4).
URL:
[https://linkinghub.elsevier.com/retrieve/pii/S0897189705800084](https://linkinghub.elsevier.com/retrieve/pii/S0897189705800084).

Downing, S. M. and R. Yudkowsky, ed. (2009). _Assessment in health
professions education_. New York, NY: Routledge. ISBN: 9780203880135.
DOI: [10.4324/9780203880135](https://doi.org/10.4324%2F9780203880135).
URL:
[https://www.taylorfrancis.com/books/9780203880135](https://www.taylorfrancis.com/books/9780203880135).

---
# References    
Duong, F., J. A. Hall, et al. (2020). "Who are empathic people? It
depends on your definition of empathy". In: _Poster presented at SPSP
2020_. URL: [osf.io/dme3w](osf.io/dme3w).

Fan, X. (2013). "“The test is reliable”; “the test is valid”: Language
use, unconscious assumptions, and education research practice". In:
_The Asia-Pacific Education Researcher_ 22 (2), pp. 217-218. ISSN:
0119-5646. DOI:
[10.1007/s40299-012-0036-y](https://doi.org/10.1007%2Fs40299-012-0036-y).
URL:
[http://link.springer.com/10.1007/s40299-012-0036-y](http://link.springer.com/10.1007/s40299-012-0036-y).

Franke, G. R., M. Sarstedt, et al. (2021). "Assessing measure
congruence in nomological networks". In: _Journal of Business Research_
130, pp. 318-334. ISSN: 01482963. DOI:
[10.1016/j.jbusres.2021.03.003](https://doi.org/10.1016%2Fj.jbusres.2021.03.003).
URL:
[https://linkinghub.elsevier.com/retrieve/pii/S0148296321001569](https://linkinghub.elsevier.com/retrieve/pii/S0148296321001569).

Furr, R. M. (2022). _Psychometrics: An introduction_. 4th ed. Thousand
Oaks, CA: SAGE. ISBN: 978-1-0718-2409-2.

Furr, R. M. and S. Heuckeroth (2019). "The “quantifying construct
validity” procedure: Its role, value, interpretations, and
computation". In: _Assessment_ 26.4, pp. 555-566. ISSN: 1073-1911. DOI:
[10.1177/1073191118820638](https://doi.org/10.1177%2F1073191118820638).
URL:
[http://journals.sagepub.com/doi/10.1177/1073191118820638](http://journals.sagepub.com/doi/10.1177/1073191118820638).

---
# References    
Gronlund, N. E. and R. L. Linn (1990). _Measurement and evaluation in
teaching_. 6th ed. New York, NY: Macmillan. ISBN: 0023481110.

Huebner, A. and G. B. Skar (2021). "Conditional standard error of
measurement: Classical test theory, generalizability theory and
many-facet Rasch measurement with applications to writing assessment".
In: _Practical Assessment, Research & Evaluation_ 26.1, pp. 1-20. DOI:
[10.7275/vzmm-0z68](https://doi.org/10.7275%2Fvzmm-0z68). URL:
[https://openpublishing.library.umass.edu/pare/article/id/1573/](https://openpublishing.library.umass.edu/pare/article/id/1573/).

Kane, M. T. (2013). "Validating the interpretations and uses of test
scores". In: _Journal of Educational Measurement_ 50.1, pp. 1-73. ISSN:
00220655. DOI:
[10.1111/jedm.12000](https://doi.org/10.1111%2Fjedm.12000). URL:
[http://doi.wiley.com/10.1111/jedm.12000](http://doi.wiley.com/10.1111/jedm.12000).

Lawshe, C. H. (1975). "A quantitative approach to content validity".
In: _Personnel Psychology_ 28.4, pp. 563-575. ISSN: 0031-5826. DOI:
[10.1111/j.1744-6570.1975.tb01393.x](https://doi.org/10.1111%2Fj.1744-6570.1975.tb01393.x).
URL:
[http://doi.wiley.com/10.1111/j.1744-6570.1975.tb01393.x](http://doi.wiley.com/10.1111/j.1744-6570.1975.tb01393.x).

Lilienfeld, S. O., A. L. Pydych, et al. (2017). "50 differences that
make a difference: A compendium of frequently confused term pairs in
psychology". In: _Frontiers in Education_ 2, pp. 1-11. ISSN: 2504-284X.
DOI:
[10.3389/feduc.2017.00037](https://doi.org/10.3389%2Ffeduc.2017.00037).
URL:
[http://journal.frontiersin.org/article/10.3389/feduc.2017.00037/full](http://journal.frontiersin.org/article/10.3389/feduc.2017.00037/full).

---
# References    
LimeSurvey GmbH (2024). _LimeSurvey: An open source survey tool
(version 6.2.5) [Computer software]_. Hamburg. URL:
[http://www.limesurvey.org](http://www.limesurvey.org).

Lissitz, R. W. and K. Samuelsen (2007). "A suggested change in
terminology and emphasis regarding validity and education". In:
_Educational Researcher_ 36.8, pp. 437-448. ISSN: 0013-189X. DOI:
[10.3102/0013189X07311286](https://doi.org/10.3102%2F0013189X07311286).
URL:
[http://journals.sagepub.com/doi/10.3102/0013189X07311286](http://journals.sagepub.com/doi/10.3102/0013189X07311286).

Lynn, M. R. (1986). "Determination and quantification of content
validity". In: _Nursing Research_ 35.6, pp. 382-386. ISSN: 0029-6562.
DOI:
[10.1097/00006199-198611000-00017](https://doi.org/10.1097%2F00006199-198611000-00017).
URL:
[http://journals.lww.com/00006199-198611000-00017](http://journals.lww.com/00006199-198611000-00017).

Masterson, M. J. and R. K. R. Jr. (2009). "The nomological network and
the research continuum". In: Ed. by M. Khosrow-Pour. 2nd ed. Vol. 1-8.
IGI Global, pp. 2827-2833. DOI:
[10.4018/978-1-60566-026-4.ch452](https://doi.org/10.4018%2F978-1-60566-026-4.ch452).
URL:
[http://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-60566-026-4.ch452](http://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/978-1-60566-026-4.ch452).

Messick, S. (1989). "Validity". In: _Educational measurement_. Ed. by
R. L. Linn. 3rd ed. New York, NY: American Council on
Education/Macmillan. Chap. 1, pp. 13-103.

---
# References    
Messick, S. (2000). "Consequences of test interpretation and use: The
fusion of validity and values in psychological assessment". In:
_Problems and Solutions in Human Assessment_ , pp. 3-20. URL:
[http://link.springer.com/chapter/10.1007/978-1-4615-4397-8\_1](http://link.springer.com/chapter/10.1007/978-1-4615-4397-8\_1).

Miller, D., R. L. Linn, et al. (2009). _Measurement and assessment in
teaching_. 10th ed. Upper Saddle River, NJ: Pearson Education. ISBN:
9780137126712; 0137126719. DOI: [LK -
https://worldcat.org/title/769655999](https://doi.org/LK%20-%20https%3A%2F%2Fworldcat.org%2Ftitle%2F769655999).

Mussel, P. (2013). "Introducing the construct curiosity for predicting
job performance". In: _Journal of Organizational Behavior_ 34.4, pp.
453-472. ISSN: 08943796. DOI:
[10.1002/job.1809](https://doi.org/10.1002%2Fjob.1809). URL:
[http://doi.wiley.com/10.1002/job.1809](http://doi.wiley.com/10.1002/job.1809).

Newton, P. E. and S. D. Shaw (2013). "Standards for talking and
thinking about validity.". In: _Psychological Methods_ 18.3, pp.
301-319. ISSN: 1939-1463. DOI:
[10.1037/a0032969](https://doi.org/10.1037%2Fa0032969). URL:
[http://doi.apa.org/getdoi.cfm?doi=10.1037/a0032969](http://doi.apa.org/getdoi.cfm?doi=10.1037/a0032969).

Polit, D. F. and C. T. Beck (2006). "The content validity index: Are
you sure you know what's being reported? Critique and recommendations".
In: _Research in Nursing & Health_ 29.5, pp. 489-497. ISSN: 0160-6891.
DOI: [10.1002/nur.20147](https://doi.org/10.1002%2Fnur.20147). eprint:
NIHMS150003. URL: [http://doi.wiley.com/10.1002/nur.20270
http://doi.wiley.com/10.1002/nur.20147](http://doi.wiley.com/10.1002/nur.20270
http://doi.wiley.com/10.1002/nur.20147).

---
# References    
Polit, D. F., C. T. Beck, et al. (2007). "Is the CVI an acceptable
indicator of content validity? Appraisal and recommendations". In:
_Research in Nursing & Health_ 30.4, pp. 459-467. ISSN: 01606891. DOI:
[10.1002/nur.20199](https://doi.org/10.1002%2Fnur.20199). URL:
[https://onlinelibrary.wiley.com/doi/10.1002/nur.20199](https://onlinelibrary.wiley.com/doi/10.1002/nur.20199).

Saxe, J. G. (1873). _The poems of John Godfrey Saxe_. James R. Osgood
and Co. URL:
[https://lccn.loc.gov/30003160](https://lccn.loc.gov/30003160).

Schmidt, F. L. (1988). "Validity generalization and the future of
criterion-related validity". In: _Test validity_. Ed. by H. Wainer and
H. I. Braun. Hillsdale, NJ: Routledge. Chap. 11, pp. 173-189. DOI:
[10.4324/9780203056905-22](https://doi.org/10.4324%2F9780203056905-22).
URL:
[https://www.taylorfrancis.com/chapters/edit/10.4324/9780203056905-15](https://www.taylorfrancis.com/chapters/edit/10.4324/9780203056905-15).

Shaw, S. and V. Crisp (2011). "Tracing the evolution of validity in
educational measurement: Past issues and contemporary challenges". In:
_Research Matters_ (11), pp. 14-17. URL:
[https://www.cambridgeassessment.org.uk/our-research/all-published-resources/research-matters/rm-11/](https://www.cambridgeassessment.org.uk/our-research/all-published-resources/research-matters/rm-11/).

Sireci, S. G. and J. Padilla (2014). "Validating assessments:
Introduction to the special section". In: _Psicothema_ 26 (1), pp.
97-99. ISSN: 1886-144X. DOI:
[10.7334/psicothema2013.255](https://doi.org/10.7334%2Fpsicothema2013.255).
URL:
[http://www.ncbi.nlm.nih.gov/pubmed/24444736](http://www.ncbi.nlm.nih.gov/pubmed/24444736).

---
# References    
Tourangeau, R. and K. A. Rasinski (1988). "Cognitive processes
underlying context effects in attitude measurement.". In:
_Psychological Bulletin_ 103.3, pp. 299-314. ISSN: 1939-1455. DOI:
[10.1037/0033-2909.103.3.299](https://doi.org/10.1037%2F0033-2909.103.3.299).
URL:
[http://doi.apa.org/getdoi.cfm?doi=10.1037/0033-2909.103.3.299](http://doi.apa.org/getdoi.cfm?doi=10.1037/0033-2909.103.3.299).

VandenBos, G. R., ed. (2015a). _APA dictionary of psychology_. 2nd ed.
Washington, DC: American Psychological Association. ISBN:
1-4338-1944-9. DOI:
[10.1037/14646-000](https://doi.org/10.1037%2F14646-000). eprint:
arXiv:1011.1669v3. URL:
[http://content.apa.org/books/14646-000](http://content.apa.org/books/14646-000).

Westen, D. and R. Rosenthal (2003). "Quantifying construct validity:
Two simple measures.". In: _Journal of Personality and Social
Psychology_ 84.3, pp. 608-618. ISSN: 1939-1315. DOI:
[10.1037/0022-3514.84.3.608](https://doi.org/10.1037%2F0022-3514.84.3.608).
URL:
[http://doi.apa.org/getdoi.cfm?doi=10.1037/0022-3514.84.3.608](http://doi.apa.org/getdoi.cfm?doi=10.1037/0022-3514.84.3.608).

Woods, S. L. (1916). _World stories for children_. Ainsworth and
Company. URL:
[https://books.google.com.sg/books?id=Pe4AAAAAYAAJ](https://books.google.com.sg/books?id=Pe4AAAAAYAAJ).

Worthen, B. R., L. R. White, et al. (1999). _Measurement and assessment
in schools_. 2nd ed. New York, NY: Longman. ISBN: 080131660X;
9780801316609. DOI: [LK -
https://worldcat.org/title/39291881](https://doi.org/LK%20-%20https%3A%2F%2Fworldcat.org%2Ftitle%2F39291881).

---
class: center, bottom, inverse

# More info

Slides created with the <svg viewBox="0 0 581 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;" xmlns="http://www.w3.org/2000/svg">  <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"></path></svg> package [`xaringan`](https://github.com/yihui/xaringan).

Practice is the best strategy for learning.

_In God we trust, all others bring data_

Edwards Deming

THE END

---
class: center, bottom, inverse

![:scale 50%](data:image/png;base64,#assets/gif/the_end.gif)