Psychometrics Applied to Organizational and Work Psychology

.title[
# Psychometrics Applied to Organizational and Work Psychology
]
.subtitle[
## <br>Adapting Existing Instruments
]
.author[
### Jorge Sinval
]
.date[
### 2025-03-07
]

---

# .white[Adapting Existing Instruments]
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

.white {
  color: #FFFFFF;
}

.red {
  color: #FF0000;
}

.green {
  color: #00FF00;
}

.kbd {
    display: inline-block;
    padding: .2em .5em;
    font-size: 0.75em;
    line-height: 1.75;
    color: #555;
    vertical-align: middle;
    background-color: #fcfcfc;
    border: solid 1px #ccc;
    border-bottom-color: #bbb;
    border-radius: 3px;
    box-shadow: inset 0 -1px 0 #bbb
}

</style>

---

# .white[Current Practices]
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Current Practices

## An example

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt0[
Soixante-sept articles publiés entre 1989 et 2019 dans la Revue canadienne des sciences du comportement et la Revue québécoise de psychologie ont été sélectionnés pour mener cette recherche.  
...  
De façon générale, les résultats indiquent une certaine diversité dans les pratiques adoptées par les chercheurs francophones pour assurer la traduction et la validation d’un instrument de mesure. Plus particulièrement, **seul un faible nombre d’études semble avoir procédé à un prétest, évalué les compétences en matière de bilinguisme des traducteurs, évalué la validité de contenu, ou encore établi des normes.**

---

# .white[Guidelines]
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Guidelines: A list

1. Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 2014)
  
--

2. Guidance for Industry &mdash; Patient-Reported Outcomes Measures: Use in Medical Product Development to Support Labeling Claims (FDA, 2009)  
  
--

3. Consensus-Based Standards for the Selection of Health Measurement Instruments (COSMIN; Mokkink, Terwee, Patrick, Alonso, Stratford, Knol, Bouter, and de
Vet, 2010)  
  
--

4. Evaluating the Measurement of Patient-Reported Outcomes  (EMPRO; Valderas, Ferrer, Mendívil, Garin, Rajmil, Herdman, and Alonso, 2008)  
--

5. Principles for the Validation and Use of Personnel Selection Procedures (SIOP, 2018)  
  
--

6. Test Reviewing for the Mental Measurement Yearbook at the Buros Center for Testing (Carlson, Geisinger, and Jonson, 2017)  
  
--

7. European Federation of Psychologists’ Association’s (EFPA) review model (Evers, Hagemeister, Høstmælingen, Lindley, Muñiz, and Sjöberg, 2013)

---

# .white[ITC Guidelines for Translating and Adapting Tests (2<sup>nd</sup> ed.)]
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# ITC Guidelines

## The International Test Commission (ITC)

The International Test Commission (ITC) is an association of national psychological associations, test commissions, publishers and other organizations committed to promoting effective testing and assessment policies and to the proper development, evaluation and uses of educational and psychological instruments.

## ITC Guidelines for Translating and Adapting Tests (2<sup>nd</sup> ed.)

The 2<sup>nd</sup> ed. of the ITC Guidelines for Translating and Adapting Tests (International Test Commission, 2018) are designed to provide a framework for the appropriate translation and adaptation of psychometric instruments. They cover a range of areas and intend to ensure that the resulting translations and adaptations of tests are equivalent to the original versions, and are valid and reliable.

The guidelines are intended to help professionals involved in the process of translating and adapting tests to ensure that they are carried out appropriately and ethically. They are not intended to replace professional judgement, but rather to support it.

---

# ITC Guidelines

Key areas addressed in the guidelines include:

I. **Pre-conditions for Tests Adaptation**: This section emphasizes the importance of understanding the construct being measured, the target population, and the cultural and linguistic characteristics of the population. It also discusses the need for a justification for test adaptation, such as the absence of an existing suitable test in the target language or cultural context.

II. **Test Development Guidelines**: This part covers the initial steps of test adaptation, which include translation, review and back-translation. It highlights the need for multiple independent translations, consensus meetings to resolve discrepancies, and back-translation to check for semantic and conceptual equivalence.

III. **Confirmation Guidelines**: These guidelines focus on the importance of pretesting the adapted test with a sample of the target population. The aim is to check the comprehensibility and relevance of the test items, and to collect preliminary data on the test's psychometric properties.

---

# ITC Guidelines

IV. **Administration Guidelines**: This section provides instructions on how to administer the adapted test. It emphasizes the need to consider cultural differences that may affect the administration of the test, such as differences in test-taking behavior or attitudes towards testing. Instructions should be clear and standardized to ensure equivalence across different administrations.

V. **Scoring and Interpretations Guidelines**: These guidelines discuss how to score the adapted test and interpret the results. It emphasizes the need for equivalence in scoring procedures between the original and adapted tests, and the need for caution when comparing scores across different cultural or linguistic groups.

VI. **Documentation Guidelines**: This final section covers the necessary documentation of the test adaptation process. It emphasizes the need to provide a detailed and transparent account of all steps taken in the adaptation process, including any difficulties encountered and how they were resolved. This information is crucial for evaluating the validity and reliability of the adapted test.

---

# .white[Category I: Pre-Condition Guidelines]
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Pre-Condition Guidelines

**Step 1**

Obtain permission from the holder of the intellectual property.

---

# Pre-Condition Guidelines

Assess if the degree of overlap in the definition and content of the construct measured by the instrument is enough for the intended use.
]

.pull-right[
<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#assets/img/misfit_definition_items.jpeg" alt="Misfit between construct definition and items' content. Reprinted from I-O Psych Memes (2024)." width="50%" />
<p class="caption">Misfit between construct definition and items' content. Reprinted from I-O Psych Memes (2024).</p>
</div>
]

---

# Pre-Condition Guidelines

**Step 3**

Minimize the influence of any linguistic and cultural differences which are irrelevant for the intended use.

---

# .white[Category II: Test Development Guidelines]
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Test Development Guidelines

**Step 1**

Consider linguistic, psychological, and cultural differences through the selection of experienced experts.

**Step 2**

Use appropriate translation guidelines.

**Step 3**

Obtain evidence that the content of the items and the test instructions have congruent meaning for the intended populations.

**Step 4**

Obtain evidence that the score categories, items format, rating scales, modes of administration and test use conventions are suitable for the intended populations.

**Step 5**

Gather pilot data to conduct a small-scale validity evidence assessment.

---

# .white[Category III: Confirmation Guidelines]
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Confirmation Guidelines

**Step 1**

Select a relevant sample both in terms of representativeness as in terms of size.

**Step 2**

Provide statistical evidence about construct, method and item equivalence between source and target versions.

**Step 3**

Provide evidence that support the norms and validity evidence of the instrument.

**Step 4**

Use adequate data analysis and equating design when linking scores from different versions.

---

# .white[Category IV: Administration Guidelines]
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Administration Guidelines

**Step 1**

Prepare administration instructions and materials to diminish language- and culture-related problems caused by response modes and application processes that affect the validity of the inferences drawn from the obtained scores.

**Step 2**

Establish the necessary conditions to apply the instrument in the intended populations.

---

# .white[Category V: Scoring and Interpretation Guidelines]
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Scoring and Interpretation Guidelines

**Step 1**

Conduct the interpretation of group differences referring to all meaningful available information.

**Step 2**

Establish comparisons between test scores across populations when the level of measurement invariance allows it.

---

# .white[Category VI: Documentation Guidelines]
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Documentation Guidelines

**Step 1**

Supply documentation that accounts for any change to the original version and evidence that supports equivalence of the scores across populations.

**Step 2**

Develop documentation for users of the adapted measure that supports good practices in the context of the intended population.

---

# Applying the ITC Guidelines

To implement these guidelines effectively, Hernández, Hidalgo, Hambleton, and Gómez-Benito (2020) provided a structured list with evaluative criteria. These criteria are classified as "Not applicable", "Not acceptable", "Acceptable", and "Excellent", and cover all phases of instrument adaptation.

---

# Pre-Condition

ITC guidelines with proposed criteria: The evaluative checklist Hernández Hidalgo et al. (2020)

]

---

# Test Development

.font60[
|ITC guidelines|Assessment criteria| Not applicable `\(^{\ast}\)`|Not acceptable| Acceptable| Excellent|
|:-------------|:-------------------|:-----------------------------|:------------|:----------|:--------|
|TD1: Ensure that the translation and adaptation processes consider linguistic, psychological, and cultural differences in the intended populations through the choice of experts with relevant expertise.|TD1-1: Form a multidisciplinary team composed of: a) professional translators who are proficient in the source and target languages (if different languages are involved) and have knowledge of the cultures involved, b) experts in the construct to be measured, c) experts in the cultures involved, and d) experts in test construction. In some cases, the same team member may be an expert in more than one of these aspects, for example, the languages and cultures, the construct and cultures, etc.|||||
|TD2: Use appropriate judgmental designs and procedures to maximize the suitability of the test adaptation in the intended populations.|TD2-1: Use recommended translation designs and justify the choice. Forward, backward or simultaneous translations may be used, depending on the purpose of the adaptation, the scope of the project, the number of cultures involved, and whether or not it will be necessary to compare scores of individuals from different cultures. Independent translators (at least two professionals or two teams) are involved in forward and backward translations. If a test is intended to be used cross-culturally from its inception, use simultaneous / concurrent development of multiple language versions of the test from the outset.<br>TD2-2: Have several translators that work independently and form a committee of experts to review and compare the proposed translations in order to compile judgmental reviews, resolve possible discrepancies, and produce a consensus version.|||||
|TD3: Provide evidence that the test instructions and item content have similar meaning for all intended populations.|TD3-1: Ensure that the instructions are clear and comprehensible, using terms that are familiar to the target population.<br>TD3-2: Ensure that the item content is clear and expressed with similar levels of commonality and difficulty in the source and target cultures. Linguistic elements that could hinder the understanding of the translated version, such as words with different meaning, double negations, etc., should be avoided. Non-linguistic elements (such as images and pictures) must be contextualized for considering the target population.|||||
  
`\(^{\ast}\)` The requirements for concluding that the level of accomplishment is **acceptable** or **excellent** are described in [this document](https://www.cop.es/pdf/ITC-guidelines-for-test-adaptation-CRITERION-CHECKLIST.pdf). If one or more criteria are not applicable this must be explicitly justified.

]

---

# Test Development

]

---

# Confirmation

.font60[
|ITC guidelines|Assessment criteria| Not applicable `\(^{\ast}\)`|Not acceptable| Acceptable| Excellent|
|:-------------|:-------------------|:-----------------------------|:------------|:----------|:--------|
|C1: Select sample with characteristics that are relevant for the intended use of the test and of suffi cient size and relevance for the empirical analyses.|C1-1: Ensure that the target sample is large enough to carry out the necessary statistical analyses and to adequately represent the population.<br>C1-2: When the focus of interest is on cross-cultural comparisons, ensure that the source and target samples are comparable for all relevant variables except for language and/or cultural background.|||||
|C2: Provide relevant statistical evidence about the construct equivalence, method equivalence, and item equivalence for all intended populations.|C2-1: When there is interest in comparing the source and target populations, use statistical procedures to ensure that construct equivalence holds across populations.<br>C2-2: When there is interest in comparing the source and target populations, check for method equivalence (instrument characteristics, administration process, and sample characteristics).<br>C2-3: When there is interest in comparing the source and target populations, assess DIF between the cultural groups to be compared using statistical procedures appropriate to the item format, sample size, and test dimensionality.<br>C2-4: In the event that DIF is detected at meaningful levels, carry out analyses to understand the reasons for the DIF (e.g., linguistic or method effects) across cultures.|||||
|C3: Provide evidence supporting the norms, reliability and validity of the adapted version of the test in the intended populations.|C3-1: Ensure that the type of reliability indicators reported is adequate for the type of test, using adequate statistical analysis and sample sizes. The obtained values must be satisfactory and the standard error of measurement must be reported.<br>C3-2: Provide validity evidence consistent with the intended use of the test scores, using adequate statistical analysis and sample sizes.<br>C3-3: Ensure and verify that the norms are adequate for interpreting the test scores of the target population.|||||
|C4: Use an appropriate equating design and data analysis procedures when linking score scales from different language versions of a test.|C4-1: When cross-cultural/cross-lingual assessment is the objective, and comparability of scores across groups is necessary but some items are functioning differentially, use appropriate linking designs and data analysis procedures before comparison.|||||
  
`\(^{\ast}\)` The requirements for concluding that the level of accomplishment is **acceptable** or **excellent** are described in [this document](https://www.cop.es/pdf/ITC-guidelines-for-test-adaptation-CRITERION-CHECKLIST.pdf). If one or more criteria are not applicable this must be explicitly justified.

]

---

# Administration

`\(^{\ast}\)` The requirements for concluding that the level of accomplishment is **acceptable** or **excellent** are described in [this document](https://www.cop.es/pdf/ITC-guidelines-for-test-adaptation-CRITERION-CHECKLIST.pdf). If one or more criteria are not applicable this must be explicitly justified.

]

---

# Score Scales and Interpretation

.font60[
|ITC guidelines|Assessment criteria| Not applicable `\(^{\ast}\)`|Not acceptable| Acceptable| Excellent|
|:-------------|:-------------------|:-----------------------------|:------------|:----------|:--------|
|SSI1: Interpret any group score differences with reference to all relevant available information.|SSI1-1: When score comparisons are justifi ed on the basis of measurement invariance analysis, consider a number of interpretations of cross-cultural differences, taking into account the information that has been systematized and documented in PC3-1 (G3, C4) regarding cultural and linguistic distance. To understand the differences in the observed scores, the role of these variables (e.g., religiosity, individualism, different response tendencies) should be considered.|||||
|SSI2: Only compare scores across populations when the level of invariance has been established on the scale on which scores are reported.|SSI2-1: To compare individual scores of people belonging to different cultures, and/ or mean scores across cultures, ensure that measurement equivalence (a.k.a. lack of DIF) is assessed and supported, at least for a meaningful number of items.|||||

]

---

# Documentation

]

---

# Adapt or Create Instruments?

## Concluding Remarks

- The decision to adapt or build a new instrument depends on the researcher and associated costs.  
- The adaptation of psychometric instruments not only benefits researchers but also professionals.  
- Adaptation allows for a more comprehensive tool choice based on adaptation studies and validity evidence.

---

# .white[Recommendations to Increase Response Rate]
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

---

# Recommendations

## Increase Response

.can-edit.key-measurement[
- thing one
- thing two
- ...
]

---

# Recommendations

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt0[
We found substantial heterogeneity among trial results in half of the strategies. The odds of response were increased by more than a half using non-monetary incentives `\((1.72;\)` `\(95\%~\)` `\(CI ~1.09~\)` to `\(~2.72\)`; heterogeneity `\(P < 0.00001,\)` `\(I^2 = 95\%)\)`, shorter e-questionnaires `\((1.73;\)` `\(1.40~\)` to `\(~2.13;\)` `\(P = 0.08,\)` `\(I^2 = 68\%)\)`, including a statement that others had responded `\((1.52;\)` `\(95\%~\)` `\(CI\)` `\(~1.36~\)` to `\(~1.70)\)`, and a more interesting topic `\((1.85;\)` `\(95\%~\)` `\(CI\)` `\(~1.52~\)` to `\(~2.26)\)`. The odds of response increased by a third using a lottery with immediate notification of results `\((1.37;\)` `\(95\%~\)` `\(CI\)` `\(~1.13~\)` to `\(~1.65)\)`, an offer of survey results `\((1.36;\)` `\(95\%~\)` `\(CI\)` `\(~1.15~\)` to `\(~1.61)\)`, and using a white background `\((1.31;\)` `\(95\%~\)` `\(CI\)` `\(~1.10~\)` to `\(~1.56)\)`. The odds of response were also increased with personalised e-questionnaires `\((1.24;\)` `\(95\%~\)` `\(CI\)` `\(~1.17~\)` to `\(~1.32;\)` `\(P = 0.07,\)` `\(I^2 = 41\%)\)`, using a simple header `\((1.23;\)` `\(95\%~\)` `\(CI\)` `\(~1.03~\)` to `\(~1.48)\)`, using textual representation of response categories `\((1.19;\)` `\(95\%~\)` `\(CI\)` `\(~1.05~\)` to `\(~1.36)\)`, and giving a deadline `\((1.18;\)` `\(95\%~\)` `\(CI\)` `\(~1.03~\)` to `\(~1.34)\)`. The odds of response tripled when a picture was included in an e-mail `\((3.05;\)` `\(95\%~ CI\)` `\(~1.84~\)` to `\(~5.06;\)` `\(P = 0.27,\)` `\(I^2 = 19\%)\)`. The odds of response were reduced when "Survey" was mentioned in the e-mail subject line `\((0.81;\)` `\(95\%~\)` `\(CI\)` `\(~0.67~\)` to `\(~0.97;\)` `\(P = 0.33,\)` `\(I^2 = 0\%)\)`, and when the e-mail included a male signature `\((0.55;\)` `\(95\%~\)` `\(CI\)` `\(~0.38\)` to `\(0.80;\)` `\(P = 0.96,\)` `\(I^2 = 0\%)\)`.

---

# References    
American Educational Research Association, American Psychological
Association, et al. (2014). _Standards for Educational and
Psychological Testing_. Washington, DC: American Psychological
Association. ISBN: 9780935302356.

Carlson, J. F., K. F. Geisinger, et al., ed. (2017). _The twentieth
mental measurements yearbook_. Lincoln, NE: The Buros Center for
Testing. ISBN: 978-0-910674-66-9.

Edwards, P. J., I. Roberts, et al. (2009). "Methods to increase
response to postal and electronic questionnaires". In: _Cochrane
Database of Systematic Reviews_ 2010.1. ISSN: 14651858. DOI:
[10.1002/14651858.MR000008.pub4](https://doi.org/10.1002%2F14651858.MR000008.pub4).
URL:
[http://doi.wiley.com/10.1002/14651858.MR000008.pub4](http://doi.wiley.com/10.1002/14651858.MR000008.pub4).

Evers, A., C. Hagemeister, et al. (2013). _EFPA Review model for the
description and evaluation of psychological and educational tests_.
EFPA. URL:
[https://www.efpa.eu/sites/default/files/2023-06/110c_EFPA_BOA_TEST_REVIEW_MODEL_version426.pdf](https://www.efpa.eu/sites/default/files/2023-06/110c_EFPA_BOA_TEST_REVIEW_MODEL_version426.pdf).

FDA (2009). _Guidance for industry patient-reported outcome measures:
Use in medical product development to support labeling claims_. URL:
[https://www.fda.gov/media/77832/download](https://www.fda.gov/media/77832/download).

---
# References    
Hernández, A., M. D. Hidalgo, et al. (2020). "International test
commission guidelines for test adaptation: A criterion checklist". In:
_Psicothema_ 32.3, pp. 390-398. ISSN: 1886144X. DOI:
[10.7334/psicothema2019.306](https://doi.org/10.7334%2Fpsicothema2019.306).

I-O Psych Memes (2024). _It's disappointing when I like the description
of a construct but then I hate the survey items used to measure [Image
attached] [Post]_. URL:
[https://www.linkedin.com/posts/iopsychmemes_psychometrics-ohpsych-iopsych-activity-7218222820919586816-Za9r](https://www.linkedin.com/posts/iopsychmemes_psychometrics-ohpsych-iopsych-activity-7218222820919586816-Za9r).

International Test Commission (2018). "ITC Guidelines for Translating
and Adapting Tests (second edition)". In: _International Journal of
Testing_ 18.2, pp. 101-134. ISSN: 1530-5058. DOI:
[10.1080/15305058.2017.1398166](https://doi.org/10.1080%2F15305058.2017.1398166).
URL:
[https://www.tandfonline.com/doi/full/10.1080/15305058.2017.1398166](https://www.tandfonline.com/doi/full/10.1080/15305058.2017.1398166).

Lauzier, M., K. Côté, et al. (2021). "La validation transculturelle
d'instruments de mesure en psychologie : Un portrait des pratiques
utilisées dans les travaux publiés entre 1989 et 2019 [Cross-cultural
validation of measuring instruments in psychology: A portrait of the
practices used in t". In: _Canadian Psychology / Psychologie
canadienne_, pp. 1-18. ISSN: 1878-7304. DOI:
[10.1037/cap0000302](https://doi.org/10.1037%2Fcap0000302). URL:
[http://doi.apa.org/getdoi.cfm?doi=10.1037/cap0000302](http://doi.apa.org/getdoi.cfm?doi=10.1037/cap0000302).

Mokkink, L. B., C. B. Terwee, et al. (2010). "The COSMIN study reached
international consensus on taxonomy, terminology, and definitions of
measurement properties for health-related patient-reported outcomes".
In: _Journal of Clinical Epidemiology_ 63.7, pp. 737-745. ISSN:
08954356. DOI:
[10.1016/j.jclinepi.2010.02.006](https://doi.org/10.1016%2Fj.jclinepi.2010.02.006).
URL:
[http://linkinghub.elsevier.com/retrieve/pii/S0895435610000909](http://linkinghub.elsevier.com/retrieve/pii/S0895435610000909).

---
class: center, bottom, inverse

# More info

Slides created with the <svg viewBox="0 0 581 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;">  [ comment ]  <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"></path></svg> package [`xaringan`](https://github.com/yihui/xaringan).

Practice is the best strategy for learning.

_In God we trust, all others bring data_

Edwards Deming

THE END

---
class: center, bottom, inverse

![:scale 50%](data:image/png;base64,#assets/gif/the_end.gif)