Exploring the Depths of Psychometrics:

.title[
# Exploring the Depths of Psychometrics:
]
.subtitle[
## <br> From Classical Theories to Modern Applications
]
.author[
### Jo<svg viewBox="0 0 581 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;"> [ comment ] <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z" /></svg>ge Sinval
]
.date[
### 2025-03-24
]

---

# Measurement
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

.kbd {
    display: inline-block;
    padding: .2em .5em;
    font-size: 0.75em;
    line-height: 1.75;
    color: #555;
    vertical-align: middle;
    background-color: #fcfcfc;
    border: solid 1px #ccc;
    border-bottom-color: #bbb;
    border-radius: 3px;
    box-shadow: inset 0 -1px 0 #bbb
}

</style>

---

# Measurement

## What is the importance of measurement?

.can-edit.key-measurement[
- thing one
- thing two
- ...
]

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[
Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.

---

# Measurement

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt0[
In physical science a first essential step in the direction of learning any subject is to find principles of numerical reckoning and methods for practicably measuring some quality connected with it. I often say that when you can measure what you are speaking about and express it in numbers you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the stage of science, whatever the matter may be.

---
class: inverse, center, middle

# Psychometrics
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Psychometrics

## What is it?

.can-edit.key-psychometrics[
- thing one
- thing two
- ...
]

## <center>A dictionary of psychology</center>

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[
The construction and application of psychological tests; mental measurement. [From Greek psyche mind + metron a measure + -ikos of, relating to, or resembling]

---

# Psychometrics

## In the beginning... Another path...

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt0[

Psychometry, the science of the measurement of the duration, and of mental processes. An occult power of divining the secret properties fo things by mere contact.

---

# Psychometrics

## Evolution

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt0[
Psychometrics is defined in _Chambers Twentieth Century Dictionary_ as the ‘branch of psychology dealing with measurable factors’, but also as the ‘occult power of defining the properties of things by mere contact’. While it is the first of these definitions that we shall be dealing with in this book, there have been times in recent years when the second might have seemed more accurate as a description of current practice, particularly in debates about intelligence. It is impossible to consider the development of modern-day psychometrics without looking at the substantial influence of the intelligence testing movement in the late nineteenth and early twentieth centuries. However, the origins of the subject go back long before then.

---

# Psychometrics

## Biased approach?

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt1[
The quantitative imperative is the view that studying something scientifically means measuring it. Measurement is thought to be a necessary part of science and non-quantitative methods are thought to be pre-scientific. This imperative is motivated by the idea that all attributes are fundamentally quantitative, an idea originating with the pre-Socratic Pythagoreans (Huffman, 1999). What is not all that widely recognized amongst psychologists is just how deeply this Pythagorean idea permeates Western culture.

---

# Psychometrics

## <center>APA dictionary of statistics and research methods</center>

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt2[
The branch of psychology concerned with the quantification and measurement of human attributes, behavior, performance, and the like, as well as with the design, analysis, and improvement of the tests, questionnaires, and so on used in such measurement. Also called psychometric psychology; psychometry.

---

# Psychometrics

## <center>APA dictionary of psychology</center>

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt2[
Psychometrics: The study of the measurements of psychological characteristics such as abilities, aptitudes, achievement, personality traits and knowledge.

It shares the same definition of the _APA dictionary of statistics and research methods_ (Zedeck, 2014, p. 279)

---

# Psychometrics

## <center>The Cambridge dictionary of statistics</center>

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[
The study of the measurements of psychological characteristics such as abilities, aptitudes, achievement, personality traits and knowledge

---

# Psychometrics

> The character which shapes our conduct is a definite and durable 'something', and therefore... it is reasonable to attempt to measure it. (Galton, 1884)

> The history of science is the history of measurement (J. M. Cattell, 1893)

> Whatever exists at all exists in some amount. To know it thoroughly involves knowing its quantity as well as its quality (E. L. Thorndike, 1918)

---

# Psychometrics

> We hardly recognize a subject as scientic if measurement is not one of its tools (Boring, 1929)

> There is yet another [method] so vital that, if lacking it, any study is thought... not be scientic in the full sense of the word. This further an crucial method is that of measurement. (Spearman, 1937)

> One's knowledge of science begins when he can measure what he is speaking about and express in numbers (Eysenck, 1973)

---

# Psychometrics

## Critics?

.can-edit.key-critics[
...
]

In the past:

* Eugenics
  * Racism
  * ...

---

# Psychometrics

## History

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt0[
Group tests of intelligence entered widespread use following the First World War, during which the Army alpha and beta tests had been introduced and were subsequently applied to millions within the USA as part of the conscription process. A committee under the chairmanship of Robert Yerkes, president of the American Psychological Association, and including Terman, devised these tests to the following criteria: adaptability to group use, correlation with the Binet scales, measurement of a wide range of ability, objectivity and rapidity of scoring, unfavourableness to malingering and cheating, independence of school training, minimum of writing and economy of time. In seven working days they constructed ten subtests with enough items for ten different forms. These were piloted on 500 subjects from a broad sampling of backgrounds, including schools for the retarded, a psychopathic hospital, recruits, officer trainees and high-school students. The entire process was complete in less than 6 months. Before 1940, these tests, and others based on them, were widely used as part of eugenicist programmes in both the USA and Europe, leading to the sterilisation of people with low IQ scores and restrictions on their movements between states and countries.

---

# Psychometrics

"A picture is worth a thousand words"

<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#assets/img/jones2006.jpg" alt="Some of the important figures in the early history of psychometrics on an approximate timeline. This figure was extracted from  Jones and Thissen (2006)" width="60%" />
<p class="caption">Some of the important figures in the early history of psychometrics on an approximate timeline. This figure was extracted from  Jones and Thissen (2006)</p>
</div>

---

# Psychometrician
<br>
<center>
<iframe title="New York Times Video - Embed Player" width="720" height="481.5" frameborder="0" scrolling="no" allowfullscreen="true" marginheight="0" marginwidth="0" id="nyt_video_player" src="https://www.nytimes.com/video/players/offsite/index.html?videoId=1194817099099"></iframe>
</center>

---

# Latent variables
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Latent variables

## What are latent variables?

.can-edit.key-latent[
- thing one
- thing two
- ...
]

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[
The many, as we say, are seen but not known, and the ideas are known but not seen.

---

# Latent variables

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt3[
Mathematically, latent variable models specify a generalized regression function that can be written as `$f(\mathbb{E}(\boldsymbol{X}))=g(\boldsymbol{\Theta})$`, where `$f$` is a link function, `$\mathbb{E}$` is the expectation operator, `$\boldsymbol{X}$` denotes a matrix of observed variables, `$\boldsymbol{\Theta}$` is a latent structure, and `$g$` is some function that relates the latent structure to the observed variables.
.tr[
📚 (Borsboom, 2008, p. 26)
]]

---

# Latent variables

.pull-left[
<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#assets/img/reflective.png" alt="Reflective model. This figure was extracted from  Beaujean (2014)" width="60%" />
<p class="caption">Reflective model. This figure was extracted from  Beaujean (2014)</p>
</div>
]

.pull-right[
<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#assets/img/formative.png" alt="Formative model. This figure was extracted from  Beaujean (2014)" width="60%" />
<p class="caption">Formative model. This figure was extracted from  Beaujean (2014)</p>
</div>
]

.footnote[<sup>⚠️</sup> If we assume the latent construct is the ‘true’ score, using a sum or single item will likely not capture the relationship between the construct and the target very well, yet these are far more common the approach in practice.]

---

# Latent variables

Three traditions:

- Test-Score tradition (i.e., Classical Test Theory<sup>🤓</sup> [CTT]);  
  - Scaling tradition (i.e., Item Response Theory [IRT]); and,  
  - Structural tradition (i.e., Structural equation models).

---

# Latent variables

<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#assets/img/measurement_theories.png" alt="Traditions of measurement. This figure was extracted from  Engelhard and Wind (2021)" width="70%" />
<p class="caption">Traditions of measurement. This figure was extracted from  Engelhard and Wind (2021)</p>
</div>

---

# Latent variables

Measures with more items (longer) are more reliable than their conterparts

Comparing scores from different measures can be only doen when the test forms/mesures are parallel

Item properties depend on a representative sample
]

Measures with lesser items (shorter) can be more reliable than their counterparts

Item responses of different measures can be compared as long as they are measuring  the same latent trait.

Item properties do not depend on a representative sample

]

---

# Latent variables

.pull-left[CTT  
Position on the latent trait continuum is derived from comparing the test score with score of reference group

All items on the measure must have the same response categories
]

.pull-right[IRT  
Position on the latent trait continuum are derived by comparing the distance between item on the ability scale.

Items on measure can have different response categories.

]

---

# CTT
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# CTT

## Readings

.can-edit.key-measurement[
- thing one
- thing two
- ...
]

---

# CTT

## Theory and Assumptions

- Classical Test Theory (CTT) – also known as weak or true-score test theory

- Called classic relative to Item Response Theory (IRT) which is a more modern approach
  
  - CTT describes a set of psychometric procedures used to test items and scales reliability, difficulty, discrimination, etc.

.footnote[In the context of CTT, a psychometric instrument is said to have evidence of reliability if the error in the true score `$(\tau)$` is minimal.

]

---

# CTT

- CTT analyses are the are most widely used form of analyses.
  
  - The statistics can be computed by readily available statistical packages
  
  - CTT Analyses are performed on the test as a whole rather than on the item and although item statistics can be generated, they apply only to that group of students on that collection of items

---

# CTT

• Assumes that every person has a true score on an item or a scale if we can only measure it directly without error

• CTT analyses assumes that a person’s test score is comprised of their “true” score plus some measurement error.

• This is the common true score model:

\begin{align}
X = \tau + \varepsilon \label{truescore}
\end{align}

An observed test-score of a person is the sum of that persons true score and an error of measurement

<center>
<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#assets/img/ctt.gif" alt="Graphical representation of the CTT. This figure was extracted from &lt;a href="https://conjointly.com/kb/true-score-theory/"&gt;https://conjointly.com/kb/true-score-theory/&lt;/a&gt;" width="30%" />
<p class="caption">Graphical representation of the CTT. This figure was extracted from <a href="https://conjointly.com/kb/true-score-theory/">https://conjointly.com/kb/true-score-theory/</a></p>
</div>

---

# CTT

- Based on the expected values of each component for each person we can see that \eqref{expectation}:

\begin{align}
\mathbb{E}(X_i)=\tau_i \label{expectation}
\end{align}

The expected value of observed scores is the true score

`$\varepsilon_i=X_i-\tau_i$`

`$\mathbb{E}(X_i-\tau_i)=\mathbb{E}(X_i)-\mathbb{E}(\tau_i)=\tau_i-\tau_i=0$`

`$\varepsilon$` and `$X$` are random variables, `$\tau$` is constant

However this is theoretical and **not done at the individual level**.

---

# CTT

\begin{align}
\rho_{ \varepsilon,\tau}=0 \label{corr1}
\end{align}

The error of measurement on a test and the true scores on that test are uncorrelated

\begin{align}
\rho_{ \varepsilon_1,\varepsilon_2}=0 \label{corr2}
\end{align}

Error scores on two different tests are uncorrelated

\begin{align}
\rho_{ \varepsilon_1,\tau_2}=0 \label{corr3}
\end{align}

The error of measurement on a test and the true scores on all other tests are uncorrelated

If two tests have observed scores `$X$` and `$X^\prime$` that satisfy assumptions \eqref{truescore} to \eqref{corr3}, and if, for every population of examinees, `$\tau = \tau^\prime$` and `$\sigma^2_\varepsilon=\sigma^2_{\varepsilon^\prime}$`, then the tests are called **parallel tests**. In other words, parallel tests have the same true scores and error variances.

If two tests have observed scores `$X_1$` and `$X_2$` that satisfy assumptions \eqref{truescore} to \eqref{corr3}, and if, for every population of examinees, `$\tau_1=\tau_2+c$`, where `$c$` is a constant, then the tests are called essentially `$\tau-\text{equivalent tests}$`. To put it differently, **essentially τ-equivalent tests** have true scores that differ by a constant.

---

# CTT

If we assume that people are randomly selected then `$\tau$` becomes a random variable as well (as seen in equation \ref{truescore})

Therefore, in CTT we assume that `$\varepsilon$` (i.e., the error)):  
  
  * Has a mean of zero (i.e., `$\mu = 0$`)
  * Is normally distributed (i.e., `$\mathcal{N}(0,\sigma)$`)
  * Uncorrelated with true score (i.e., `$\rho_{\varepsilon, \tau}=0$` see equation \eqref{corr1})

---

# CTT

.center[
![](data:image/png;base64,#exploring_the_depths_of_psychometrics_files/figure-html/true-score-chunk-1.png)
]

]

.center[
![](data:image/png;base64,#exploring_the_depths_of_psychometrics_files/figure-html/observed-score-chunk-1.png)
]

]

---

# CTT

Measurement error `$(\varepsilon)$` around a `$\tau$` can be large or small, for example `$X_1$`, `$X_2$`, and `$X_3$`.

.center[
![:scale 42%](data:image/png;base64,#exploring_the_depths_of_psychometrics_files/figure-html/error-amount-chunk-1.png)
]

---

# CTT

## Domain Sampling Theory<sup>💡</sup>

• Another Central Component of CTT

• Another way of thinking about populations and samples

• Domain &mdash; Population or universe of all possible items measuring a single concept or trait (theoretically infinite)

• Test &mdash; a sample of items from that universe

.footnote[<sup>💡</sup>Assumes that the items that have been selected for any one test are just a sample of items from an infinite domain of potential items. Domain sampling is the most common CTT used for practical purposes.]

---

# CTT

## Domain Sampling Theory

• A person’s true score would be obtained by having them respond to all items in the "universe" of items

• We only see responses to the sample of items on the test

• So, reliability is the proportion of variance in the "universe" explained by the test variance

---

# CTT

## Domain Sampling Theory

• A universe is made up of a (possibly infinitely) large number of items

• So, as tests get longer they represent the domain better, therefore longer tests should have higher reliability

• Also, if we take multiple random samples from the population we can have a distribution of sample scores that represent the population

---

# CTT

## Domain Sampling Theory

• Each random sample from the universe would be "randomly parallel" to each other

• Unbiased estimate of reliability:

\begin{align}
r_{1, \tau }=\sqrt{\bar{r}_1 ,_j} \label{reliability}
\end{align}

* `$r_{1,\tau} =$` correlation between test and true score  
  
  * `$\bar r_{1,j} =$` average correlation between the test and all other randomly parallel tests

---

# CTT

## Reliability

• Reliability is theoretically the correlation between a `$X$` (test-score) and a `$\tau$` (the true score), squared

• Essentially the proportion of `$X$` that is `$\tau$`

`$$\rho^2_{X,\tau}=\frac{\sigma^2_{\tau}}{\sigma^2_{X}}=\frac{\sigma^2_{\tau}}{\sigma^2_{\tau}+\sigma^2_{\varepsilon}} \label{reliabilityfrac}$$`

• This can’t be measured directly so we use other methods to estimate

---

# CTT

## Reliability

• Reliability can be viewed as a measure of consistency or how well as test "holds together"

• Reliability is measured on a scale of `$0-1$`. The greater the number the higher the reliability<sup>⚠️</sup>.

.footnote[<sup>⚠️</sup>Values very close to `\$1\$` can be seem as indicative of redundancy between the items.]

---

# CTT

## Reliability

The approach to estimating reliability depends on:

* Estimation of "true" score  
  * Source of measurement error

Types of reliability:

* Test-retest  
  * Parallel Forms  
  * Split-half  
  * Internal Consistency

---

# CTT

## Test-Retest Reliability

• Evaluates the error associated with administering a test at two different times.  
  
  • _Time Sampling Error_  
  
  • How-To:  
    • Apply the psychometric instrument at Time 1 `$(X_1)$`  
    • Apply the psychometric instrument at Time 2 `$(X_2)$`  
    • Calculate `$r_{X_1,X_2}$` for the two scores

• Easy to do; one test does it all.

---

# CTT

## Test-Retest Reliability

• Assume 2 administrations `$X_1$` and `$X_2$`:

`$$\varepsilon_{X_{1,i}}  = \varepsilon_{X_{2,i}} ~~~~~~ \sigma^2_{\varepsilon_{1,i}}=\sigma^2_{\varepsilon_{2,i}}
\therefore \rho_{X_1,X_2}=\frac{\sigma_{X_1,X_2}}{\sigma_{X_1}\sigma_{X_2}}=\frac{\sigma^2_{\tau}}{\sigma^2_{X}}=\rho_{X,\tau}$$`

• The correlation between the 2 administrations is the reliability

---

# CTT

## Test-Retest Reliability

• Sources of error:

* random fluctuations in performance  
  * uncontrolled testing conditions  
  * extreme changes in weather  
  * sudden noises/chronic noise  
  * other distractions

• internal factors:

* illness, fatigue, emotional strain, worry  
  * recent experiences

---

# CTT

## Test-Retest Reliability

Generally used to evaluate constant traits:

* Intelligence, personality

Not appropriate for qualities that change rapidly over time:

* Mood, hunger  
  
Problem: Carryover Effects (Exposure to the test at time #1 influences scores on the test at time #2)

Only a problem when the effects are random.

If everybody goes up 5pts, you still have the same variability

---

# CTT

## Test-Retest Reliability

• Practice effects

* Type of carryover effect  
  * Some skills improve with practice  
  * Manual dexterity, ingenuity or creativity  
  * Practice effects may not benefit everybody in the same way.

Carryover & Practice effects more of a problem with short inter-test intervals (ITI).
  
But, longer ITI’s have other problems:

* developmental change, maturation, exposure to historical events

---

# CTT

## Parallel Forms Reliability

Evaluates the error associated with selecting a particular set of items.

_Item Sampling Error_

How To:

* Develop a large pool of items (i.e. Domain) of varying difficulty.  
  * Choose equal distributions of difficult / easy items to produce multiple forms of the same test.  
  * Give both forms close in time.  
  * Calculate `$r$` for the two administrations.

---

# CTT

## Parallel Forms Reliability

Also known as _Alternative Forms_ or _Equivalent Forms_

Can give parallel forms at different points in time; produces error estimates of time and item sampling.

One of the most rigorous assessments of reliability currently in use.

Infrequently used in practice – too expensive to develop two tests.

---

# CTT

## Parallel Forms Reliability

Assume 2 parallel tests `$X$` and `$X^\prime$`:

`$$\varepsilon(X_i)=\varepsilon(X_i^\prime) ~~~~~~ \sigma^2_{E_i}=\sigma^2_{E^\prime_i}$$`

`$$\therefore \rho_{XX^\prime} = \frac{\sigma_{XX^\prime}}{\sigma_{X}\sigma_{X^\prime}}= \frac{\sigma^2_{\tau}}{\sigma^2_{X}}=\rho_{XT}$$`

• The correlation between the 2 parallel forms is the reliability

---

# CTT

## Split Half Reliability

What if we treat halves of one test as parallel forms? (Single test as whole domain)

That’s what a split-half reliability does

This is testing for _Internal Consistency_

* Scores on one half of a test are correlated with scores on the second half of a test
  
Big question: “How to split?”:

* First half vs. last half  
  * Odd vs Even  
  * Create item groups called testlets

---

# CTT

## Split Half Reliability

How to:

* Compute scores for two halves of single test, calculate `$r$`.
  
Problem:

* Considering the domain sampling theory what’s wrong with this approach?
  * A `$20$` item test cut in half, is two `$10-$`item tests, what does that do to the reliability?  
  * If only we could correct for that…

---

# CTT

## Spearman-Brown Formula

Estimates the reliability for the entire test based on the split-half

Can also be used to estimate the affect changing the number of items on a test has on the reliability

`$r^\ast = \frac{j(r)}{1+(j-1)r}$`

Where `$r^\ast$` is the estimated reliability, `$r$` is the correlation between the halves, `$j$` is the new length proportional to the old length

---

# CTT

## Spearman-Brown Formula

For a split-half it would be:

`$$r^\ast=\frac{2(r)}{(1+r)}$$`

Since the full length of the test is twice the length of each half

---

# CTT

## Spearman-Brown Formula

**Example 1:** a 30-item test with a split-half reliability of `$.65$`

`$$r^\ast=\frac{2(.65)}{(1+.65)}=.79$$`

• The `$.79$` is a much better reliability than the `$.65$`

---

# CTT

## Spearman-Brown Formula

**Example 2**: a 30-item test with a test retest reliability of `$.65$` is lengthened to `$90$` items

`$$r^\ast=\frac{3(.65)}{1+(3-1).65}=\frac{1.95}{2.3}=.85$$`

**Example 3**: a 30 item test with a test re-test reliability of .65 is cut to 15 items

`$$r^\ast=\frac{.5(.65)}{1+(.5-1).65}=\frac{.325}{.675}=.48$$`

---

# CTT

## Detour 1: Variance Sum Law

Often multiple items are combined in order to create a composite score

The variance of the composite is a combination of the variances and covariances of the items creating it

General Variance Sum Law states that if `$X$` and `$Y$` are random variables:

`$$\sigma^2_{X \pm Y}=\sigma^2_{X}+\sigma^2_{Y}\pm2\sigma^2_{XY}$$`

---

# CTT

## Detour 1: Variance Sum Law

Given multiple variables we can create a variance/covariance matrix

For 3 items:

<table class="table table-striped" style="color: black; width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;">  </th>
   <th style="text-align:left;"> $X_1$ </th>
   <th style="text-align:left;"> $X_2$ </th>
   <th style="text-align:left;"> $X_3$ </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> $X_1$ </td>
   <td style="text-align:left;"> $\sigma^2_1$ </td>
   <td style="text-align:left;"> $\sigma_{12}$ </td>
   <td style="text-align:left;"> $\sigma_{13}$ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> $X_2$ </td>
   <td style="text-align:left;"> $\sigma_{21}$ </td>
   <td style="text-align:left;"> $\sigma^2_{2}$ </td>
   <td style="text-align:left;"> $\sigma_{23}$ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> $X_3$ </td>
   <td style="text-align:left;"> $\sigma_{31}$ </td>
   <td style="text-align:left;"> $\sigma_{32}$ </td>
   <td style="text-align:left;"> $\sigma^2_3$ </td>
  </tr>
</tbody>
</table>

---

# CTT

## Detour 1: Variance Sum Law
  
Example Variables `$X$`, `$Y$` and `$Z$`

Covariance Matrix

By the variance sum law the composite variance would be:

`$$\sigma^2_{X+Y+Z}=\sigma^2_{Total}=\sigma^2_{X}+\sigma^2_{Y}+\sigma^2_{Z}+2\sigma_{XY}+2\sigma_{XZ}+2\sigma_{YZ}$$`

---

# CTT

## Detour 1: Variance Sum Law
  
  
<table class="table table-striped" style="color: black; width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;">  </th>
   <th style="text-align:right;"> $X$ </th>
   <th style="text-align:right;"> $Y$ </th>
   <th style="text-align:right;"> $Z$ </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> $X$ </td>
   <td style="text-align:right;"> 55.83 </td>
   <td style="text-align:right;"> 29.52 </td>
   <td style="text-align:right;"> 30.33 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> $Y$ </td>
   <td style="text-align:right;"> 29.52 </td>
   <td style="text-align:right;"> 17.49 </td>
   <td style="text-align:right;"> 16.15 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> $Z$ </td>
   <td style="text-align:right;"> 30.33 </td>
   <td style="text-align:right;"> 16.15 </td>
   <td style="text-align:right;"> 29.06 </td>
  </tr>
</tbody>
</table>

By the variance sum law the composite variance would be:
  
`$S^2_{total}=55.83+17.49+29.06+2\times29.52+2\times30.33+2\times16.15=254.38$`

---

# CTT

## Internal Consistency Reliability

• If items are measuring the same construct they should elicit similar if not identical responses

• Coefficient OR Cronbach’s Alpha is a widely used measure of internal consistency for continuous data

• Knowing the a composite is a sum of the variances and covariances of a measure we can assess consistency by how much covariance exists between the items relative to the total variance

---

# CTT

## Internal Consistency Reliability

• Coefficient Alpha is defined as:

`$$\alpha = \frac{k}{k-1}\left(\frac{\sum S_{ij}}{S^2_{Total}}\right)$$`

• `$S^2_{Total}$` is the composite variance (if items were summed)

• `$S_{ij}$` is covariance between the `$i^{th}$` and `$j^{th}$` items where `$i \neq j$`

• `$k$` is the number of items

---

# CTT

## Internal Consistency Reliability

• Using the same continuous items `$X$`, `$Y$` and `$Z$`

• The covariance matrix is:

• The total variance is `$254.38$`

• The sum of all the covariances is `$152$`

`$$\alpha = \frac{k}{k-1}\left(\frac{\sum S_{ij}}{S^2_{Total}}\right)= \frac{3}{3-1}\left(\frac{152}{254.38}\right)=0.8962969$$`

---

# CTT

## Internal Consistency Reliability

• Coefficient Alpha can also be defined as:

`$$\alpha=\frac{k}{k-1}\left(\frac{S^2_{Total}-\sum S^2_i}{S^2_{Total}}\right)$$`

• `$S^2_{Total}$` is the composite variance (if items were summed)  
• `$S^2_{i}$` is variance for each item  
• `$k$` is the number of items

---

# CTT

## Internal Consistency Reliability

• Using the same continuous items `$X$`, `$Y$` and `$Z$`  
• The covariance matrix is:  
  
<table class="table table-striped" style="color: black; width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;">  </th>
   <th style="text-align:right;"> $X$ </th>
   <th style="text-align:right;"> $Y$ </th>
   <th style="text-align:right;"> $Z$ </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> $X$ </td>
   <td style="text-align:right;"> 55.83 </td>
   <td style="text-align:right;"> 29.52 </td>
   <td style="text-align:right;"> 30.33 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> $Y$ </td>
   <td style="text-align:right;"> 29.52 </td>
   <td style="text-align:right;"> 17.49 </td>
   <td style="text-align:right;"> 16.15 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> $Z$ </td>
   <td style="text-align:right;"> 30.33 </td>
   <td style="text-align:right;"> 16.15 </td>
   <td style="text-align:right;"> 29.06 </td>
  </tr>
</tbody>
</table>
  
• The total variance is `$254.38$`  
• The sum of all the variances is `$102.38$`

`$\alpha=\frac{k}{k-1}\left(\frac{S^2_{Total}-\sum S^2_i}{S^2_{Total}}\right)=\frac{3}{3-1}\left(\frac{254.38-102.38}{254.38}\right)=0.8962969$`

---

# CTT

## Internal Consistency Reliability: Example

<div class="pre-name">internal_consistency.R</div>

``` r
#download data
ds <- readr::read_csv('https://ndownloader.figshare.com/files/22299075')
#the function to be used "ufs"
ufs::scaleStructure(dat = ds,
items = c("SIJS1","SIJS2", "SIJS3","SIJS4", "SIJS5"))
```
  
  + `dat` &mdash; set the dataset.
  + `items` &mdash; sets the for which the reliability (internal consistency) estimates should be computed.

---

# CTT

## Internal Consistency Reliability: Example

.scroll-output[
<img src="data:image/png;base64,#exploring_the_depths_of_psychometrics_files/figure-html/unnamed-chunk-12-1.png" width="100%" /><img src="data:image/png;base64,#exploring_the_depths_of_psychometrics_files/figure-html/unnamed-chunk-12-2.png" width="100%" />

### Scale structure

#### Information about this scale

<table>
 <thead>
  <tr>
*   <th style="text-align:left;">   </th>
   <th style="text-align:left;">  </th>
  </tr>
* </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Dataframe: </td>
   <td style="text-align:left;"> ds </td>
  </tr>
*  <tr>
   <td style="text-align:left;"> Items: </td>
   <td style="text-align:left;"> SIJS1, SIJS2, SIJS3, SIJS4 &amp; SIJS5 </td>
*  </tr>
  <tr>
   <td style="text-align:left;"> Observations: </td>
   <td style="text-align:left;"> 1171 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Positive correlations: </td>
   <td style="text-align:left;"> 10 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Number of correlations: </td>
   <td style="text-align:left;"> 10 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Percentage positive correlations: </td>
   <td style="text-align:left;"> 100 </td>
  </tr>
</tbody>
</table>

#### Estimates assuming interval level

<table>
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:right;">  </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Omega (total): </td>
   <td style="text-align:right;"> 0.85 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Omega (hierarchical): </td>
   <td style="text-align:right;"> 0.81 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Revelle's Omega (total): </td>
   <td style="text-align:right;"> 0.88 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Greatest Lower Bound (GLB): </td>
   <td style="text-align:right;"> 0.87 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Coefficient H: </td>
   <td style="text-align:right;"> 0.89 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Coefficient Alpha: </td>
   <td style="text-align:right;"> 0.84 </td>
  </tr>
</tbody>
</table>

##### Confidence intervals

<table>
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:left;">  </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Omega (total): </td>
   <td style="text-align:left;"> [0.83; 0.86] </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Coefficient Alpha: </td>
   <td style="text-align:left;"> [0.83; 0.85] </td>
  </tr>
</tbody>
</table>

#### Estimates assuming ordinal level

<table>
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:right;">  </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Ordinal Omega (total): </td>
   <td style="text-align:right;"> 0.88 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Ordinal Omega (hierarch.): </td>
   <td style="text-align:right;"> 0.88 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Ordinal Coefficient Alpha: </td>
   <td style="text-align:right;"> 0.88 </td>
  </tr>
</tbody>
</table>

##### Confidence intervals

<table>
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:left;">  </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Ordinal Omega (total): </td>
   <td style="text-align:left;"> [0.87; 0.89] </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Ordinal Coefficient Alpha: </td>
   <td style="text-align:left;"> [0.86; 0.89] </td>
  </tr>
</tbody>
</table>

Note: the normal point estimate and confidence interval for omega are based on the procedure suggested by Dunn, Baguley & Brunsden (2013) using the MBESS function ci.reliability, whereas the psych package point estimate was suggested in Revelle & Zinbarg (2008). See the help ('?ufs::scaleStructure') for more information.

</div>
<div style="display:block;clear:both;" class="scale-structure-end"></div>
]

---

# CTT

## Internal Consistency Reliability

• Coefficient Alpha is considered a lower-bound estimate of the reliability of continuous items  
• It was developed by Cronbach (1951) in the 50’s but is based on an earlier formula by Kuder and Richardson (1937) that tackled internal consistency for dichotomous ("Yes"/"No", "Right"/"Wrong") items

---

# CTT

## Detour 2: Dichotomous Items

• If `$Y$` is a dichotomous item:  
  + `$p$` &mdash; proportion of successes OR items answer correctly  
  + `$q$` &mdash; proportion of failures OR items answer incorrectly  
  + `$\bar Y=p$` &mdash; observed proportion of successes 
  
  + `$S^2_Y = pq$`

---

# CTT

## Internal Consistency Reliability

•  Kuder and Richardson (1937) developed the `$KR_{20}$` that is defined as:

`$$KR_{20}=\alpha=\frac{k}{k-1}\left(\frac{S^2_{Total}-\sum pq}{S^2_{Total}}\right)$$`

• Where `$pq$` is the variance for each dichotomous item  
• The `$KR_{21}$` is a quick and dirty estimate of the `$KR_{20}$`

---

# CTT

## Reliability of Observations

• What if you’re not using a test but instead observing individual’s behaviors as a psychological assessment tool?

• How can we tell if the judge’s (assessor’s) are reliable?

---

# CTT

## Reliability of Observations

• Typically a set of criteria are established for judging the behavior and the judge is trained on the criteria

• Then to establish the reliability of both the set of criteria and the judge, multiple judges rate the same series of behaviors

• The correlation between the judges is the typical measure of reliability

• But, couldn’t they agree by accident? Especially on dichotomous or ordinal scales?

---

# CTT

## Reliability of Observations

• `$\kappa$` (kappa) is a measure of inter-rater reliability that controls for chance agreement

• Values range from `$-1$` (less agreement than expected by chance) to `$1$` (perfect agreement)

• `$\kappa \geq .75$` &mdash; excellent  
  • `$.40 \leq \kappa < .75$` &mdash; fair to good  
  • `$\kappa <.40$` &mdash; poor

---

# CTT

## Standard Error of Measurement

• So far the standard error of measurement was approached as the error associated with trying to estimate a true score from a specific test  
• This error can come from many sources  
• We can calculate it’s size by:

`$$S_{measurement} = S\sqrt{1-r}$$`

• `$S$` is the standard deviation  
• `$r$` is reliability

---

# CTT

## Standard Error of Measurement

• Using the same continuous items `$X$`, `$Y$` and `$Z$`  
• The total variance is 254.38  
• `$s = \sqrt{254.38} = 15.9492947$`  
• `$\alpha = 0.8962969$`

`$$s_{measurement}=15.9492947\times \sqrt{1-0.8962969}=5.1361464$$`

---
# CTT

## The Prophecy Formula

• How much reliability do we want?

• Typically we want values above `$.80$`

• What if we don’t have them?

• The Spearman-Brown can be algebraically manipulated to achieve

`$$j=\frac{r_d\left(1-r_o\right)}{r_o\left(1-r_d\right)}$$`

• `$j$` &mdash; # of tests at the current length

• `$r_d$` &mdash; desired reliability

• `$r_o$` &mdash; observed reliability

---

# CTT

## The Prophecy Formula

• Using the same continuous items `$X$`, `$Y$` and `$Z$`

• `$\alpha = 15.9492947\times \sqrt{1-0.8962969}$`

• What if we want a .95 reliability?

`$$j=\frac{r_d\left(1-r_o\right)}{r_o\left(1-r_d\right)}=\frac{.95\left(1-0.8962969\right)}{0.8962969\left(1-.95\right)}=\frac{0.098518}{0.0448148}=2.1983333$$`

• We need a test that is `$2.2$` times longer than the original  
• Nearly `$7$` items to achieve .95 reliability

---

# CTT

## Attenuation

• Correlations are typically sought at the true score level but the presence of measurement error can cloud (attenuate) the size the relationship  
• We can correct the size of a correlation for the low reliability of the items.  
• Called the Correction for Attenuation

---

# CTT

## Attenuation

• Correction for attenuation is calculated as:

`$$\hat r_{12}=\frac{r_{12}}{\sqrt{r_{11}r_{22}}}$$`

• `$\hat r_{12}$` &mdash; corrected correlation  
• `$\sqrt{r_{12}}$` &mdash;  uncorrected correlation  
• `$\sqrt{r_{11}}$` and `$\sqrt{r_{22}}$` &mdash; the reliabilities of the tests

---

# CTT

## Attenuation

• For example `$X$` and `$Y$` are correlated at `$.45$`, `$X$` has a reliability of `$.8$` and `$Y$` has a reliability of `$.6$`, the corrected correlation is

`$$\hat r_{12}=\frac{r_{12}}{\sqrt{r_{11}r_{22}}}=\frac{.45}{\sqrt{.8\times.6}}=\frac{.45}{\sqrt{.48}}=.65$$`

---
class: inverse, center, middle

# Item Response Theory
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Item Response Theory

.can-edit.key-measurement[
- thing one
- thing two
- ...
]

---

# Item Response Theory (IRT)

In a world with and more big and naturally-occurring data, IRT offers a few promises:

1. Understand and leverage item variability

2. More precise measures of latent constructs

3. More information with fewer data points

---

# Wordbank example

Wordbank (wordbank.stanford.edu) provides open source data from over 80k MacArthur-Bates Communicative Development Inventory (MB-CDI) administrations.

---

# Warm up: Answer with a partner

1. Who is the highest ability person? Who is the lowest ability person?
2. Which item is the hardest? Which is the easiest?
3. Which item is the best? Which is the worst?
4. Who has a higher ability between person D and person I?				
5. Estimate the probability of person G getting item 2 correct.

<small>

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> person </th>
   <th style="text-align:right;"> item 1 </th>
   <th style="text-align:right;"> item 2 </th>
   <th style="text-align:right;"> item 3 </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> A </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> B </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> C </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> D </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> E </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> F </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> G </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> NA </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> H </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> I </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> J </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
</tbody>
</table>

</small>

---

# What is measurement?

1. You're interested in a latent construct (extroversion, job satisfaction, math ability, etc.)

2. You measure that latent construct by giving people items (psychometric instrument)

3. You do some science with that measurement

---

# Relevant questions

1. Is this a good psychometric instrument? Are some items better than others?

2. Does this psychometric instrument measure the latent construct I care about?

3. Is this psychometric instrument fair?

4. How do we get from responses to the items to the measure of latent trait?

---

# How do I get from responses to the latent trait?

<small>

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> child </th>
   <th style="text-align:right;"> mommy </th>
   <th style="text-align:right;"> yesterday </th>
   <th style="text-align:right;"> trash </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> A </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> B </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> C </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> D </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> E </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> F </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> G </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> NA </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> H </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> I </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> J </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
</tbody>
</table>

</small>

---

# The sum score

1. What assumptions does it make?

2. What are its limitations?

---

# The sum score

## Assumptions

1. Items are equally difficult

2. Items are equally related to the latent construct

3. 1 on all items is positively related to the construct

## Limitations

1. How do I handle missing data?

2. How do I make predictions?

3. How do I make an adaptive psychometric instruments?

---

# Item Response Theory (IRT) to the rescue!

A parametric framework for item response data

Each person `$p$` has an ability `$\theta_p$`

Each item `$i$` has an easiness `$b_i$`

These combine to give the probability of correct response

---

# The logistic function

We use the logistic `$\sigma(x) = \dfrac{\exp(x)}{1 + \exp(x)}$` function to map the sum of ability and easiness to probability of correct response

---

# Looking at easiness

---

# Question: Probability of responses

1. Calculate P(correct, correct, incorrect | ability = 0)

2. Calculate P(correct, correct, incorrect | ability = 1)

---

# Answer: Probability of responses

``` r
logistic <- function(x) {exp(x) / (1 + exp(x))}
```

1. Calculate P(correct, correct, incorrect | ability = 0)

``` r
logistic(2 + 0) * logistic(0 + 0) * (1 - logistic(-2 + 0))
```

```
## [1] 0.3879017
```

2. Calculate P(correct, correct, incorrect | ability = 1)

``` r
logistic(2 + 1) * logistic(0 + 1) * (1 - logistic(-2 + 1))
```

```
## [1] 0.5091
```

---

# Who uses IRT?

Basically any measurement that happens in education:

- International large-scale assessments (ILSAS) as PISA, TIMSS, PIRLS...

- National tests (Enem)

- Professional Orders

- Big Organizations in their R&S process

Very common in other fields as well:

- Psychology

- Health

- Economics

---

# IRT in practice

We'll show the power of IRT with the Wordbank data [(wordbank.stanford.edu](wordbank.stanford.edu))

<table>
<caption>Word comprehension</caption>
 <thead>
  <tr>
   <th style="text-align:left;"> sex </th>
   <th style="text-align:right;"> age </th>
   <th style="text-align:right;"> yum yum </th>
   <th style="text-align:right;"> bee </th>
   <th style="text-align:right;"> cockadoodledoo </th>
   <th style="text-align:right;"> buy </th>
   <th style="text-align:right;"> camping </th>
   <th style="text-align:right;"> moo </th>
   <th style="text-align:right;"> ouch </th>
   <th style="text-align:right;"> aunt </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Female </td>
   <td style="text-align:right;"> 27 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Female </td>
   <td style="text-align:right;"> 21 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Female </td>
   <td style="text-align:right;"> 26 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Male </td>
   <td style="text-align:right;"> 27 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Female </td>
   <td style="text-align:right;"> 19 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Female </td>
   <td style="text-align:right;"> 30 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
</tbody>
<tfoot>
<tr>
<td style = 'padding: 0; border:0;' colspan='100%'><sup>a</sup> Note. Age in months.</td>
</tr>
</tfoot>
</table>

---

# Items

---

# Children

---

# Fit item parameters

## code

``` r
irt_model_rasch <- 
  mirt(
    data = english_words %>% select(-sex, -age),
    model = 1,
    itemtype = "Rasch",
    verbose = FALSE
  )
```

---

## Item curves

``` r
plot(irt_model_rasch, type = 'trace')
```

---

# Ability Estimates

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> sex </th>
   <th style="text-align:right;"> age </th>
   <th style="text-align:right;"> sum_score </th>
   <th style="text-align:right;"> theta_rasch </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Female </td>
   <td style="text-align:right;"> 27 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> -0.8383636 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Female </td>
   <td style="text-align:right;"> 21 </td>
   <td style="text-align:right;"> 6 </td>
   <td style="text-align:right;"> 1.3984447 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Female </td>
   <td style="text-align:right;"> 26 </td>
   <td style="text-align:right;"> 4 </td>
   <td style="text-align:right;"> -0.1224431 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Male </td>
   <td style="text-align:right;"> 27 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> -0.8383636 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Female </td>
   <td style="text-align:right;"> 19 </td>
   <td style="text-align:right;"> 3 </td>
   <td style="text-align:right;"> -0.8383636 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Female </td>
   <td style="text-align:right;"> 30 </td>
   <td style="text-align:right;"> 7 </td>
   <td style="text-align:right;"> 2.3200869 </td>
  </tr>
</tbody>
</table>

---

# Ability estimates by sex

``` r
abilities %>%  
  ggplot(aes(x = theta_rasch)) +
  geom_histogram() +
  facet_wrap(~ sex, ncol = 1)
```

---

# Wait a second

---

# Moving from Rasch to 2PL

## Rasch

Each person has ability `$\theta_p$`. Each item has easiness `$b_i$`.

`$P(y_{pi} = 1 | \theta_p, b_i) = \sigma(\theta_p + b_i)$`

where

`$\sigma(x) = \dfrac{\exp(x)}{1 + \exp(x)}$`

## 2PL

Each person has ability `$\theta_p$`. Each item has easiness `$b_i$` and discrimination `$a_i$`.

`$P(y_{pi} = 1 | \theta_p, b_i, a_i) = \sigma(a_i \cdot \theta_p + b_i)$`

---

# Discrimination

The discrimination `$a_i$` describes the strength of the relationship between the item and ability

---

# Question: Weighting

Which of the outcomes is more likely for a person with ability `$\theta_p = 2$`? (The easiness of each item is 0).

<table>
 <thead>
  <tr>
   <th style="text-align:right;"> item discrimination </th>
   <th style="text-align:left;"> outcome 1 </th>
   <th style="text-align:left;"> outcome 2 </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:right;"> 0.5 </td>
   <td style="text-align:left;"> correct </td>
   <td style="text-align:left;"> correct </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 1.0 </td>
   <td style="text-align:left;"> incorrect </td>
   <td style="text-align:left;"> correct </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 2.0 </td>
   <td style="text-align:left;"> incorrect </td>
   <td style="text-align:left;"> correct </td>
  </tr>
  <tr>
   <td style="text-align:right;"> 3.0 </td>
   <td style="text-align:left;"> correct </td>
   <td style="text-align:left;"> incorrect </td>
  </tr>
</tbody>
</table>

---

# Answer: Weighting

Which of the outcomes is more likely for a person with ability `$\theta_p = 2$`? (The easiness of each item is 0).

Outcome 1

``` r
logistic(0.5 * 2 + 0) * 
  (1 - logistic(1 * 2 + 0)) * 
  (1 - logistic(2 * 2 + 0)) * 
  logistic(3 * 2 + 0)
```

```
## [1] 0.00156352
```

Outcome 2

``` r
logistic(0.5 * 2 + 0) * 
  logistic(1 * 2 + 0) * 
  logistic(2 * 2 + 0) * 
  (1 - logistic(3 * 2 + 0))
```

```
## [1] 0.00156352
```

---

# Fit 2PL model

## code

``` r
irt_model_2pl <-
  mirt(
    data = english_words %>% select(-sex, -age),
    model = 1,
    itemtype = "2PL",
    verbose = FALSE
  )
```

---

# Item curves

---

# 2PL item parameters

---

# 2PL abilities

---

# Why stop at 2 item parameters?

## 2PL

Each person has ability `$\theta_p$`. Each item has easiness `$b_i$` and discrimination `$a_i$`.

`$P(y_{pi} = 1 | \theta_p, b_i) = \sigma(a_i \cdot \theta_p + b_i)$`

## What might a 3rd item parameter do?

## 3PL

Each person has ability `$\theta_p$`. Each item has easiness `$b_i$`, discrimination `$a_i$`, and guessability `$g_i$`.

`$P(y_{pi} = 1 | \theta_p, a_i, b_i, g_i) = g_i + (1 - g_i) \cdot \sigma(a_i \cdot \theta_p + b_i)$`

---

# Intuition behind each of the 3 parameters

- Easiness is horizontal translation

- Discrimination is slope

- Guessability is starting point at ability negative infinity

---

# Fit 3PL model

## code

``` r
irt_model_3pl <-
  mirt(
    data = english_words %>% select(-sex, -age),
    model = 1,
    itemtype = "3PL",
    verbose = FALSE
  )
```

---

# Item curves

---

# 3PL item parameters

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> item </th>
   <th style="text-align:right;"> a1 </th>
   <th style="text-align:right;"> b </th>
   <th style="text-align:right;"> g </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> yum yum </td>
   <td style="text-align:right;"> 1.33 </td>
   <td style="text-align:right;"> 1.21 </td>
   <td style="text-align:right;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> bee </td>
   <td style="text-align:right;"> 3.34 </td>
   <td style="text-align:right;"> 0.85 </td>
   <td style="text-align:right;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> cockadoodledoo </td>
   <td style="text-align:right;"> 2.18 </td>
   <td style="text-align:right;"> -0.56 </td>
   <td style="text-align:right;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> buy </td>
   <td style="text-align:right;"> 3.04 </td>
   <td style="text-align:right;"> -1.97 </td>
   <td style="text-align:right;"> 0.01 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> camping </td>
   <td style="text-align:right;"> 2.35 </td>
   <td style="text-align:right;"> -3.28 </td>
   <td style="text-align:right;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> moo </td>
   <td style="text-align:right;"> 3.05 </td>
   <td style="text-align:right;"> 2.19 </td>
   <td style="text-align:right;"> 0.24 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> ouch </td>
   <td style="text-align:right;"> 1.90 </td>
   <td style="text-align:right;"> 1.75 </td>
   <td style="text-align:right;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> aunt </td>
   <td style="text-align:right;"> 2.81 </td>
   <td style="text-align:right;"> -1.11 </td>
   <td style="text-align:right;"> 0.04 </td>
  </tr>
</tbody>
</table>

---

# 3PL abilities - compare to 2PL

---

# 3PL abilities - compare to sum score

---

# Comparing sexes

---

# Comparing ages

---

# Differential item functioning (DIF)

---

# Polytymous item response theory

---

# Multidimensional models

---

# Summary

Item response theory (IRT) provides a parametric framework for people responding to items (which can be broadly defined!).

It has a few specific advantages:

- Putting participants and item on the same scale

- Understanding items through item parameters

- Better measurement of the latent construct

- Better understanding of the relationship between the latent construct and the items

- Handling of missing data

- Ability to make predictions

- More complicated things like equating, testing for bias, comparisons with other models etc.

---

# Learning more

- Most popular way to estimate is the mirt R package written by Phil Chalmers

- Phil Chalmers has some good workshop materials on [his GitHub](https://github.com/philchalmers/mirt/wiki)

- Mike Frank reccommends the Embretson & Reise book [Item Response Theory for Psychologists](https://www.amazon.com/Response-Theory-Psychologists-Multivariate-Applications/dp/0805828192)

- Great resources on Bayesian Item Response Theory with at education-stan.github.io

- Denny Borsboom article [The attack of the psychometricians](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2779444/) is fantastic (and Mike Frank wrote it [a love letter](http://babieslearninglanguage.blogspot.com/2019/11/letter-of-recommendation-attack-of.html))

???
- [Exercise](https://github.com/stenhaug/irt-basics/blob/master/exercise.Rmd) associated with this presentation

---

# References

Beaujean, A. A. (2014). _Latent variable modeling using R: A
step-by-step guide_. New York, NY: Routledge. ISBN: 978-1-84872-698-7.

Borsboom, D. (2008). "Latent variable theory". In: _Measurement:
Interdisciplinary Research & Perspective_ 6.1-2, pp. 25-53. ISSN:
1536-6367. DOI:
[10.1080/15366360802035497](https://doi.org/10.1080%2F15366360802035497).
URL:
[http://www.tandfonline.com/doi/abs/10.1080/15366360802035497](http://www.tandfonline.com/doi/abs/10.1080/15366360802035497).

Colman, A. M. (2015). _A dictionary of psychology_. 4th ed. Oxford
University Press. ISBN: 9780199657681. DOI:
[10.1093/acref/9780199657681.001.0001](https://doi.org/10.1093%2Facref%2F9780199657681.001.0001).
URL:
[http://www.oxfordreference.com/view/10.1093/acref/9780199657681.001.0001/acref-9780199657681](http://www.oxfordreference.com/view/10.1093/acref/9780199657681.001.0001/acref-9780199657681).

Cronbach, L. J. (1951). "Coefficient alpha and the internal structure
of tests". In: _Psychometrika_ 16.3, pp. 297-334. ISSN: 0033-3123. DOI:
[10.1007/BF02310555](https://doi.org/10.1007%2FBF02310555). URL:
[http://link.springer.com/10.1007/BF02310555](http://link.springer.com/10.1007/BF02310555).

Davidson, T. and J. L. Geddie, ed. (1901). _Chambers's twentieth
century dictionary_. London: W. & R. Chambers.

---

# References

Engelhard, G. and S. A. Wind (2021). "A history of Rasch measurement
theory". In: _The history of educational measurement: Key advancements
in theory, policy, and practice_. Ed. by B. E. Clauser and M. B. Bunch.
New York, NY: Routledge, pp. 343-360. DOI:
[10.4324/9780367815318-15](https://doi.org/10.4324%2F9780367815318-15).
URL:
[https://www.taylorfrancis.com/books/9781000402391/chapters/10.4324/9780367815318-15](https://www.taylorfrancis.com/books/9781000402391/chapters/10.4324/9780367815318-15).

Everitt, B. S. and A. Skrondal (2010). _The Cambridge dictionary of
statistics_. 4th ed. New York, NY: Cambridge University Press. ISBN:
9788578110796. DOI:
[10.1017/CBO9781107415324.004](https://doi.org/10.1017%2FCBO9781107415324.004).
eprint: arXiv:1011.1669v3.

Harrington, H. J. (1987). _The improvement process: How America's
leading companies improve quality_. New York, NY: McGraw-Hill. ISBN:
D-D7-DEb7SM-S.

Huffman, C. A. (1999). "The Pythagorean tradition". In: _The Cambridge
companion to early Greek philosophy_. Ed. by A. A. Long. Cambridge
University Press, pp. 66-87. DOI:
[10.1017/CCOL0521441226.004](https://doi.org/10.1017%2FCCOL0521441226.004).
URL:
[https://www.cambridge.org/core/product/identifier/CBO9781139000734A007/type/book\_part](https://www.cambridge.org/core/product/identifier/CBO9781139000734A007/type/book\_part).

Jones, L. V. and D. Thissen (2006). "A history and overview of
psychometrics". In: _Handbook of Statistics_. Ed. by C. Rao and S.
Sinharay. Vol. 26. Amsterdam: Elsevier, pp. 1-27. ISBN: 9780444521033.
DOI:
[10.1016/S0169-7161(06)26001-2](https://doi.org/10.1016%2FS0169-7161%2806%2926001-2).
URL:
[http://www.stat.cmu.edu/~brian/905-2009/all-papers/jones-thissen-2007.pdf](http://www.stat.cmu.edu/~brian/905-2009/all-papers/jones-thissen-2007.pdf).

---
class: center, bottom, inverse

# More info

Slides created with the <svg viewBox="0 0 581 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;">  [ comment ]  <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"></path></svg> package [`xaringan`](https://github.com/yihui/xaringan).

Practice is the best strategy for learning.

_In God we trust, all others bring data_

Edwards Deming

THE END

---
class: center, bottom, inverse

![:scale 50%](data:image/png;base64,#assets/gif/the_end.gif)