Measurement

Joe Ripberger

Research Process

Levels of Measurement

  1. Nominal scales exhibit no order among categories
  2. Ordinal scales exhibit order among the categories, but distance is indeterminable
  3. Interval and ratio scales have order and determinable distances

Levels of Measurement

  • Variables with ordinal or nominal scales are sometimes called ________ ?
  • Variables with interval or ratio scales are sometimes called ________ ?

Measurement

  • Measurement: the process of linking abstract concepts to observable indicators of those concepts (data)\(^1\)
    • When the link is strong, analysis can lead to useful inferences about the relationships among the underlying concepts (theory)
    • When the link is weak, then the empirical tenability of the theory remains unknown
  • Measurement can be complicated because research questions and theories often involve abstract and multidimensional concepts such as democracy, intelligence, religion, socio-economic status, etc.

\(^1\)Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Sage publications.

Measurement

  • The measurement process involves two steps:
    1. Define the concept and the relevant dimensions of the concept; this is called conceptualization
      • What do you mean by democracy and intelligence?
      • What are the relevant dimensions of democracy and intelligence?
    2. Identify observable indicators of the concept and all dimensions of the concept; this is called operationalization
      • How are you going to identify the presence or absence of democracy and intelligence?

Variable

  • A variable is an operationalized representation of a concept (a measure)
    • Observable variables can be directly measured with one or more indicators
      • Examples?
    • Latent variables cannot be directly measured, so they are often measured using multiple manifest or indirect indicators
      • Examples?
      • Note: analysis of latent variables may require latent variable or measurement models

Assessing the Quality of Measures

  • Measures should be valid and reliable
    • A valid measure accurately measures the concept it is intended to measure
    • A reliable measure consistently assigns the same value to the same phenomenon—the value will be the same regardless of who assigns the value and when they assign it

Assessing the Quality of Measures

Assessing Validity

  1. Content validity: extent to which a measure captures all dimensions of a concept
    • Openness, conscientiousness, agreeableness, extraversion & neuroticism \(\rightarrow\) Personality
  2. Concurrent/predictive validity: extent to which a measure is related to other relevant measures
    • SAT score \(\rightarrow\) ACT score
  3. Criterion validity: extent to which a measure is related to a relevant outcome
    • SAT score \(\rightarrow\) Performance in college

Assessing Reliability

  1. Test-retest reliability: assess the degree to which a measure is consistent from one test to the next
  2. Internal consistency: assess the degree to which a measure is consistent with other measures (items) on the same test
    • Cronbach’s alpha (reliability coefficient), factor analysis, item response theory
  3. Inter-rater reliability: assess the degree to which a measure is consistent from one rater (judge) to the next
    • Krippendorff’s alpha, Cohen’s kappa, Scott’s pi