Computational Psychomet [ comment ] ics Applied to Enginee [ comment ] ing

.title[
# Computational Psychomet<svg viewBox="0 0 581 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;"> [ comment ] <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z" /></svg>ics Applied to Enginee<svg viewBox="0 0 581 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;"> [ comment ] <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z" /></svg>ing
]
.subtitle[
## <br>Developing New Instruments
]
.author[
### Jo<svg viewBox="0 0 581 512" xmlns="http://www.w3.org/2000/svg" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#384CB7;"> [ comment ] <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z" /></svg>ge Sinval
]
.date[
### 2025-11-21
]

---

# .white[Create or Adapt?]
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=800px></html>

.white {
  color: #FFFFFF;
}

.red {
  color: #FF0000;
}

.green {
  color: #00FF00;
}

.kbd {
    display: inline-block;
    padding: .2em .5em;
    font-size: 0.75em;
    line-height: 1.75;
    color: #555;
    vertical-align: middle;
    background-color: #fcfcfc;
    border: solid 1px #ccc;
    border-bottom-color: #bbb;
    border-radius: 3px;
    box-shadow: inset 0 -1px 0 #bbb
}

</style>

---

# Create or Adapt?

## Create

.can-edit.key-validity[
- advantages
- disadvantages
- ...
]

## Adapt

.can-edit.key-validity[
- advantages
- disadvantages
- ...
]

---

# Create or Adapt?

## Too many psychometric instruments?

Yes...

]

<div class="figure" style="text-align: center">
<img src="assets/img/constructs_redundancy.jpeg" alt="See Banks, Gooty, Ross, Williams, and Harrington (2018)." width="50%" />
<p class="caption">See Banks, Gooty, Ross, Williams, and Harrington (2018).</p>
</div>

]

---

# Create or Adapt?

## Too many psychometric instruments?

Yes...

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

A large proportion of measures are only used once or twice and the widespread lack of agreement about their implementation threatens research credibility. The <strong>S</strong>tandardisation <strong>O</strong>f <strong>BE</strong>havior <strong>R</strong>esearch (SOBER) guidelines that specifically address issues of flexibility and norming in measurement.

]]

---

# Create or Adapt?

## Too many new measures?

<div class="figure" style="text-align: center">
<img src="assets/img/Elson2023.png" alt="Measurement proliferation in psychology. Reprinted from Elson Hussey et al. (2023)." width="50%" />
<p class="caption">Measurement proliferation in psychology. Reprinted from Elson Hussey et al. (2023).</p>
</div>

---

# Create or Adapt?

## What to Report?

|Policy|Compliance by authors|Enforcement by reviewers and editors|
|---|---|---|
|Demonstrate nonredundancy|When creating a new measure for a primary study, provide evidence of nonredundancy (with other measures and/or constructs) or incremental validity in a separate sample, e.g. through latent variable-based analyses of associations with a large selection of scales as opposed to simple correlations.|Require validation and norming in independent data, or an explanation why this is not necessary. Reject studies that use novel/ad-hoc measures without providing validity evidence from independent data.|
|Demonstrate protocol adherence|When following a measurement procedure published elsewhere, cite relevant protocols, and demonstrate you adhered to them (e.g., by sharing study materials).|Check claims of protocol adherence by comparing materials and cited sources. Journals dedicate resources to this task.|
|Justify Modifications|Prove that any deviation or modification to an existing measure is either meaningful (e.g. to address non-invariance of a measure between samples) or irrelevant, e.g. by providing validity evidence in a separate sample. Document when deviations happened, and if possible assess the robustness of conclusions.|Discourage authors from modifying existing scales by, for instance, dropping “poorly performing” items if the authors cannot show how these deviations affect the measure out-of-sample. Journals incentivise methodological research primarily providing validity evidence for commonly used measures rather than answering substantive research questions.|
|Preregistration and Registered Reports|Determine procedural and scoring details ahead of time, reporting all deviations as they potentially weaken the strength of conclusions. Analytical and statistical decisions should be preregistered with code rather than a narrative description.|Require authors to provide rationales for decision making in measurement specifically. Compare preregistrations (and any reported deviations) with what was actually employed and reported.|
|Comprehensive Reporting|Report all of the items, stimuli, instructions, procedural parameters or other measurement characteristics used in a study or generated during the development process.|Check for comprehensive data and materials beyond what is reported in the manuscript.|
|Facilitate Research Synthesis|Report standard deviations and means (regardless of whether data are shared), do not exclusively report effect sizes relative to the in-sample variation (so-called standardised effect sizes like Cohen’s `\(d\)`, correlations, `\(r^2\)`).|Insist on complete descriptive statistics to make rigorous meta-analysis feasible. Suggest effect sizes be standardised by test norms instead of in-sample variation.|

]

The SOBER policies for psychological journals; how authors comply with them, and how they should be enforced by editors and reviewers. Reprinted from Elson Hussey et al. (2023). For a more detail see Supplementary Table 1 (<a href="https://static-content.springer.com/esm/art%3A10.1038%2Fs44271-023-00026-9/MediaObjects/44271_2023_26_MOESM1_ESM.pdf">here</a>).

---

# Create or Adapt?

## Too many psychometric instruments?

No...

.bg-washed-green.b--dark-green.ba.bw2.br3.shadow-5.ph4.mt4[

Measurement proliferation is an important part of theory development and that standardization comes at a cost to sensitive treatments of contextual characteristics of a given sample.

]]

---

# Developing New Instruments

## What to Report?

Questionable measurement practices, including poor reporting (Flake and Fried, 2020), are a serious problem.

|Question| Information to report|
|---|---|
|1. What is your construct?| Define the construct<br>Describe theories and research supporting the construct|
|2. Why and how did you select your measure?|Justify the measure selection<br>Report existing validity evidence|
|3. What measure did you use to operationalize the construct?|Describe the measure and administration procedure<br>Match the measure to the construct|
|4. How did you quantify your measure?|Describe response coding and transformation<br>Report the items or stimuli included in each score<br>Describe the calculation of scores<br>Describe all conducted (e.g., psychometric) analyses|
|5. Did you modify the scale? And if so, how and why?|Describe any modifications<br>Indicate if modifications occurred before or after data collection<br>Provide justification for modifications|
|6. Did you create a measure on the fly?|Justify why you did not use an existing measure<br>Report all measurement details for the new measure<br>Describe all available validity evidence; if there is no evidence, report that|

]

Six Questions to Promote Transparent Reporting of Measurement Practices. Reprinted from Flake and Fried (2020).

---

# Questionable Practices

]

<div class="figure" style="text-align: center">
<img src="assets/img/meme_items.jpeg" alt="Adding items from different instruments." width="65%" />
<p class="caption">Adding items from different instruments.</p>
</div>

]

---

# Choice of Instrument

Before embarking on the development of a new psychometric instrument for research or practice, it is advisable to search for an instrument that has already been developed and meets the desired requirements (Gall, Gall, and Borg, 2014).

---

# Importance of Instrument Choice

The appropriate choice of measurement instrument is essential, as they have important value in making decisions about individuals, organizations, and institutions.

---

# Strategy for Choosing Instruments

Nitko (2015) review and evaluation strategy for choosing instruments includes six steps:

Step 1: Clarify the Purpose

Step 2: Use the Local Context

Step 3: Study Professional Reviews

Step 4: Obtain Copies of the Instrument  
  
--

Step 5: Summarize Strengths and Weaknesses

Step 6: Reach a Conclusion

---

# Step 1: Clarify the Purpose

Define the objectives and specific needs for which the instrument will be used, considering the target population and intended outcomes.

---

# Step 2: Use the Local Context

Consider how the instrument will be used in the specific context and identify the needs, challenges, and resources available.

---

# Step 3: Study Professional Reviews

Examine the analyses from information sources on psychometric instruments such as the Mental Measurements Yearbook or the [Test Collection website](https://www.ets.org/test-collection.html) of the Educational Testing Service, scientific articles, technical manuals, and user guides.

---

# Step 4: Obtain Copies of the Instrument

Review the instrument materials, including test items, administration and scoring manuals, and any supplemental materials.

---

# Step 5: Summarize Strengths and Weaknesses

Based on the review and evaluation, identify the strengths and weaknesses of the instrument in relation to the specific context and needs.

---

# Step 6: Reach a Conclusion

Make a final decision on the appropriateness of the instrument for the desired purposes and prepare a justification for the choice based on the evidence gathered during the evaluation process.

---

# Development of New Psychometric Instruments

When creating an instrument there should be a strong enough underlying reason to justify the construction of a new instrument (Moreau and Wiebels, 2022).

---

# What does the test measure?

Them: What is that test actually measuring?<br><br>The researcher:<br>
.center[

(Aish [@aish_caliperce], 2024)

]

---

# Guidelines for Developing New Instruments

Several guidelines exist to develop new psychometric instruments. Some examples:

## Bandalos (2018)

1 &mdash; State the purpose of the scale  
  2 &mdash; Identify and define the domain  
  3 &mdash; Determine whether a measure already exists  
  4 &mdash; Determine the item format  
  5 &mdash; Write out the testing objectives  
  6 &mdash; Create the initial item pool  
  7 &mdash; Conduct the initial item review  
  8 &mdash; Conduct preliminary Item tryouts  
  9 &mdash; Conduct a large-scale field test of items  
  10 &mdash; Prepare guidelines for administration

---

# Guidelines for Developing New Instruments

## Boateng, Neilands, Frongillo, Melgar-Quiñonez, and Young (2018)

1 &mdash; identification of the domain(s) and item generation  
2 &mdash; consideration of content validity  
3 &mdash; pre-testing questions  
4 &mdash; sampling and survey administration  
5 &mdash; item reduction  
6 &mdash; extraction of latent factors  
7 &mdash; tests of dimensionality  
8 &mdash; tests of reliability  
9 &mdash; tests of validity

Let's see it in more detail...

---