Research Design

Joe Ripberger

Research Process

Design Study

  • Research design: systematic plan for empirically evaluating the validity of a hypothesis or set of hypotheses
  • Specifies the steps that a researcher is going to take to conduct this evaluation:
    • What data are you going to collect?
    • What methods are you going to use to analyze these data?
    • How will you know if your hypothesis or hypotheses is/are supported by the data?

External and Internal Validity

  • Internal validity: the extent to which variation in the dependent variable can be attributed to variation in the independent variable (and not other “spurious” or “confounding” variables)
    • Is there a cause-and-effect relationship between the DV and the IV?
  • External validity: the extent to which a study’s results can be generalized to other people/settings
    • Sometimes there is a trade-off between internal and external validity.

Confounding Variables

  • The basic problem in causal analysis is that of eliminating confounding variables (plausible alternative explanations)

Experimental Research Design

  1. Randomly assign subjects into one of two groups; a “control” group or an “experimental” (or treatment) group
  2. Measure the dependent variable (in both groups, using the same procedure) - pre-test
  3. Expose the experimental group to the treatment or stimulus (the independent variable)
  4. Re-measure the dependent variable (in both groups, using the same procedure) - post-test
    • If the independent variable caused a change in the dependent variable, this result will be evident in the pre-test v. post-test comparison
    • Best way to test causal hypotheses; can have high internal validity
    • Often difficult if not impossible to conduct in the world of public policy and administration—illegal or unethical to administer or withhold a treatment that may harm a particular group

Example of an Experimental Research Design

  • If my theory is correct, then…
    • H: People who live in areas with relatively high false alarm rates will be less likely to respond to tornado warnings than people who live in areas with relatively low false alarm rates
    • H: There will be a negative relationships between false alarm rates and warning responsiveness; the higher the false alarm rate, the less responsive people will be to tornado warnings

Example of an Experimental Research Design

  1. Randomly assign a cities into two groups—an experimental and a control group
  2. Measure warning responsiveness in both groups of cities (pre-test)
  3. Expose the cities assigned to the experimental track to an excessive number of false alarms; do nothing to the cities assigned to the control group
  4. Re-measure warning responsiveness in both groups of cities (post-test)
    • If the false alarms caused a change in warning responsiveness, then the difference between pre-test responsiveness and post-test responsiveness will be negative among cities that received the false alarm treatment and zero among cities that were not exposed to the treatment
    • Problem: Could you imagine what the National Weather Service would say if I asked them to try this?

Example of an Experimental Research Design

Observational Research Design

  1. Find a fairly similar set of subjects that happen to be organized into two or more different groups (no random assignment)
  2. Measure exposure to the treatment/stimulus/independent variable across the different groups (no control over exposure)
  3. Measure the dependent variable across the two groups (no repeated measurement)
    • If differential exposure to the independent variable caused a change in the dependent variable, this result should be evident in group comparisons on the dependent variable
    • Difficult to establish causality—an association between the independent variable does not necessarily mean that X caused Y (low internal validity)
    • Can have higher external validity than experimental designs
    • This is what we usually do in the social sciences

Example of an Observational Research Design

  1. Find a fairly similar set of cities that happen to experience different false alarm rates
  2. Measure false alarm rates in those cities
  3. Measure warning responsiveness in those cities
    • If the false alarms caused a change in warning responsiveness, then people who live in cities that experience a higher number of false alarms will, on average, be less responsive to tornado warnings
    • Problem: How do we know that different levels of warning responsiveness are actually caused by false alarms, not something some other variable (like the most recent warning)?
      • Remember: correlation \(\neq\) causation

Example of an Observational Research Design

Enhancing Confidence in Causal Assertions from Observational Research Designs

  • Theory: explain that there is a compelling reason to believe that variation in the IV causes a variation the DV
  • Association: show that variation in the IV is correlated with variation in the DV
  • Specification: show that the correlation holds when controlling for spurious variables
  • Time Order: show that the change in the IV happened before the change in the DV

Cross-Sectional vs. Longitudinal Design

  • Cross-sectional design: collect data from a population or sample at a single point in time
  • Longitudinal design: collect data from a population or sample at multiple points in time
    • Time-series data: few units, many points in time (EXAMPLE)
    • Panel data: many units, few points in time (same units/participants) throughout
    • Cross-sectional time-series data: ???

Collect Data

  • Types of Data:
    • Qualitative data: words
      • Example sources: interviews, focus groups, archival analysis, observation, etc.
    • Quantitative data: numbers
      • Example sources: surveys, test scores, administrative records, etc.
      • Note: sometimes, qualitative data can be quantified (i.e., content analysis)
    • Statistical analysis requires quantitative data

Levels of Measurement

  • Quantitative measures are often distinguished by the relative precision (level) of the their scale
  • There are three levels of measurement:
    1. Nominal
    2. Ordinal
    3. Interval/ratio

Levels of Measurement

  • Nominal scales exhibit no order among the categories.

    • Example: gender has a nominal scale, because there is no ordering among the attributes male and female
      • This is true, regardless of any coding scheme that might be used such as male=1 and female=2

Levels of Measurement

  • Ordinal scales exhibit order among the categories, but distance is indeterminable

    • Example: anger can be measured on the following scale: irritated, aggravated, and raging mad
      • We can say that raging mad is more angry than aggravated, but we cannot say how much more because the scale lacks determinable distance among the attributes

Levels of Measurement

  • Interval and ratio scales have order and determinable distance

    • Example: height can be measured in inches, cost can be measured in dollars, and test scores can be measured in percentages, all of which are standardized units that are a determinable distance from each other

Levels of Measurement

  • Variables with ordinal or nominal scales are sometimes called categorical variables
  • Variables with interval or ratio scales are sometimes called continuous variables