Specialized Research Designs

RSM2074 Lecture Week 10

Dr. Jun Ho Chai

Specialized Research Designs

The Plan for Today

Developmental Designs: Tracking change over time
Longitudinal, Cross-sectional, Cohort-sequential
Small-n Designs: Deep individual focus
Understanding trade-offs and biases

Why Study Development?

Language Development

How does vocabulary grow from 50 to 300 words between 18-24 months?

Cognitive Abilities

When does Theory of Mind emerge? How does it relate to language?

Individual Differences

Why do some children show vocabulary spurts while others don’t?

The Central Challenge

The Fundamental Dilemma

To understand development, we need to track changes.
But tracking takes time, money, and commitment.

Two Main Approaches:

Same Children? Follow over time (Longitudinal = Within-Subjects)
Different Children? Test once at different ages (Cross-sectional = Between-Subjects)

Quasi-Experimental Designs

What is Quasi-Experimental?

Key Characteristics:

🚫 No random assignment

📊 Some comparison group

🔧 Researcher manipulates treatment

🌍 More realistic settings

Why Use Quasi-Experiments?

💼 Real-world practicality

⚖️ Ethical constraints

👥 Can’t randomize naturally occurring groups

🏛️ Policy/program evaluation

Quasi-Experiment vs True Experiment

True Experiment

🎲 Random assignment

🔐 Experimental control

📈 Internal validity

📍 Limited real-world applicability

Quasi-Experiment

🚫 No random assignment

📊 Some control

⚖️ Moderate internal validity

🌍 Real-world relevance

Common Quasi-Experimental Designs

Non-Equivalent Control Group

👥 Two existing groups

💊 One gets intervention

📉 Compare pre-post changes

Interrupted Time Series

📍 Baseline established

⚡ Intervention introduced

📈 Track outcome over time

🔍 Look for “interruption”

Longitudinal Design

What is Longitudinal Design?

Key Principle:

Within-Subjects Variable

Same person at multiple ages
Track individual change
Can take weeks, months, or years

Visual:

Child A: ●──●──●──●
         2  3  4  5

Child B: ●──●──●──●
         2  3  4  5

Child C: ●──●──●──●
         2  3  4  5

Follow same children over time

Track a Child’s Development

viewof selectedChild = Inputs.select(
  ["Child 1 (Emma)", "Child 2 (Liam)", "Child 3 (Sophia)"],
  {label: "Select Child:", value: "Child 1 (Emma)"}
)

viewof visitAge = Inputs.select(
  ["12 months", "18 months", "24 months", "36 months"],
  {label: "Visit at age:", value: "12 months"}
)

{
  const childData = {
    "Child 1 (Emma)": {
      "12 months": {words: 8, joint: "Emerging", motor: "Walking"},
      "18 months": {words: 95, joint: "Strong", motor: "Running"},
      "24 months": {words: 420, joint: "Mature", motor: "Climbing"},
      "36 months": {words: 720, joint: "Mature", motor: "Jumping"}
    },
    "Child 2 (Liam)": {
      "12 months": {words: 5, joint: "Weak", motor: "Crawling"},
      "18 months": {words: 42, joint: "Emerging", motor: "Walking"},
      "24 months": {words: 198, joint: "Strong", motor: "Running"},
      "36 months": {words: 580, joint: "Mature", motor: "Climbing"}
    },
    "Child 3 (Sophia)": {
      "12 months": {words: 15, joint: "Strong", motor: "Walking"},
      "18 months": {words: 180, joint: "Mature", motor: "Running"},
      "24 months": {words: 520, joint: "Mature", motor: "Jumping"},
      "36 months": {words: 760, joint: "Mature", motor: "Complex"}
    }
  };
  
  const data = childData[selectedChild][visitAge];
  const allData = childData[selectedChild];
  const timelineData = Object.keys(allData).map(age => ({
    age: age,
    words: allData[age].words
  }));
  
  return html`
  <div style="text-align: center;">
    <h3>${selectedChild} at ${visitAge}</h3>
    <div style="display: flex; gap: 15px; justify-content: center; margin: 15px 0;">
      <div style="background: #e3f2fd; padding: 12px; border-radius: 8px;">
        <div style="font-weight: bold;">Vocabulary</div>
        <div style="font-size: 24px; color: #2196f3;">${data.words} words</div>
      </div>
      <div style="background: #e8f5e9; padding: 12px; border-radius: 8px;">
        <div style="font-weight: bold;">Joint Attention</div>
        <div style="font-size: 20px; color: #4caf50;">${data.joint}</div>
      </div>
      <div style="background: #fff3e0; padding: 12px; border-radius: 8px;">
        <div style="font-weight: bold;">Motor Skills</div>
        <div style="font-size: 20px; color: #ff9800;">${data.motor}</div>
      </div>
    </div>
    
    ${Plot.plot({
      marks: [
        Plot.line(timelineData, {x: "age", y: "words", stroke: "#2196f3", strokeWidth: 3}),
        Plot.dot(timelineData, {x: "age", y: "words", fill: "#2196f3", r: 5}),
        Plot.dot([{age: visitAge, words: data.words}], {x: "age", y: "words", fill: "#f44336", r: 8})
      ],
      x: {label: "Age"},
      y: {label: "Vocabulary Size", domain: [0, 800], grid: true},
      width: 600,
      height: 250,
      marginLeft: 60
    })}
  </div>`;
}

Individual Developmental Paths

Fernald et al. (2013): Not all children grow the same way

Four Growth Patterns:

30% Fast starters
25% Late bloomers
35% Steady growers
10% Early plateau

All reached normal by age 3!

Why This Matters:

✓ Only within-subjects design captures individual paths

✓ Different routes to same outcome

✓ Distinguish growth patterns hidden in averages

Advantages of Longitudinal

🔗 Causal Order

• Pretest-Posttest proves causation

• Early → Later outcomes

👤 Within-Subjects

• Same genes & family

• Changes = development

📊 Individual Paths

• Not just group averages

• Real trajectories

Disadvantages of Longitudinal

⏰ Practice Effects

• Repeated testing improves scores

• Not real development!

👋 Attrition Bias

• 15-30% drop out

• Busiest families leave first

💰 Expensive

• Multi-year funding

• Staff & tracking

Real Study Attrition Rates

{
  const studyData = {
    "NICHD SECCYD": {
      name: "NICHD Study of Early Child Care (15 years)",
      timepoints: [0, 1, 6, 11, 15],
      sample: [1364, 1226, 1061, 1009, 1002],
      desc: "US childcare & development | Vandell et al. (2010)",
      totalAttrition: 27
    },
    "Dunedin Study": {
      name: "Dunedin Health & Development Study (38 years)",
      timepoints: [3, 11, 21, 32, 38],
      sample: [1037, 954, 992, 972, 961],
      desc: "New Zealand cohort | Poulton et al. (2015)",
      totalAttrition: 7
    },
    "Perry Preschool": {
      name: "Perry Preschool Project (40 years)",
      timepoints: [0, 10, 20, 30, 40],
      sample: [123, 112, 108, 98, 97],
      desc: "Early education intervention | Schweinhart et al. (2005)",
      totalAttrition: 21
    }
  };
  
  const study = studyData[studyExample];
  const plotData = study.timepoints.map((year, i) => ({
    year: year,
    n: study.sample[i],
    pct: Math.round((study.sample[i] / study.sample[0]) * 100)
  }));
  
  const retention = 100 - study.totalAttrition;
  
  return html`
  <div style="text-align: center;">
    <h4>${study.name}</h4>
    <p style="font-size: 0.85em; color: #666;">${study.desc}</p>
    
    ${Plot.plot({
      marks: [
        Plot.barY(plotData, {x: "year", y: "n", fill: "#2196f3", sort: null}),
        Plot.text(plotData, {x: "year", y: "n", text: d => String(d.n), dy: -8, fontSize: 11})
      ],
      x: {label: "Years from Start"},
      y: {label: "Sample Size", grid: true},
      width: 600,
      height: 240,
      marginLeft: 60
    })}
    
    <div style="margin-top: 12px; display: flex; gap: 12px; justify-content: center;">
      <div style="background: #e3f2fd; padding: 8px; border-radius: 6px; font-size: 0.9em;">
        <strong>Started:</strong> ${study.sample[0]}
      </div>
      <div style="background: ${retention > 85 ? '#e8f5e9' : '#fff3e0'}; padding: 8px; border-radius: 6px; font-size: 0.9em;">
        <strong>Final:</strong> ${study.sample[study.sample.length - 1]} (${retention}%)
      </div>
      <div style="background: #ffebee; padding: 8px; border-radius: 6px; font-size: 0.9em;">
        <strong>Lost (Attrition):</strong> ${study.totalAttrition}%
      </div>
    </div>
  </div>`;
}

Cross-Sectional Design

What is Cross-Sectional Design?

Key Principle:

Between-Subjects Variable

Different children at each age
Each tested ONCE
Compare groups

Visual:

Age 3: ●  ●  ●  (A, B, C)

Age 4: ●  ●  ●  (D, E, F)

Age 5: ●  ●  ●  (G, H, I)

Different children, one snapshot

Wordbank Vocabulary Data

Real data from 92,771 children (wordbank.stanford.edu)

viewof language = Inputs.select(
  ["English", "Spanish"],
  {label: "Language:", value: "English"}
)

viewof showIndividual = Inputs.toggle({label: "Show Individual Children", value: false})

{
  const ages = d3.range(16, 31, 1);
  
  // Real Wordbank data from Frank et al. (2017) Journal of Child Language
  const englishData = [
    {age: 16, p10: 8, p50: 40, p90: 150},
    {age: 17, p10: 10, p50: 52, p90: 180},
    {age: 18, p10: 15, p50: 90, p90: 240},
    {age: 19, p10: 20, p50: 120, p90: 280},
    {age: 20, p10: 30, p50: 160, p90: 340},
    {age: 21, p10: 45, p50: 200, p90: 400},
    {age: 22, p10: 65, p50: 240, p90: 450},
    {age: 23, p10: 85, p50: 280, p90: 500},
    {age: 24, p10: 110, p50: 308, p90: 550},
    {age: 25, p10: 140, p50: 360, p90: 590},
    {age: 26, p10: 170, p50: 400, p90: 620},
    {age: 27, p10: 200, p50: 440, p90: 640},
    {age: 28, p10: 230, p50: 480, p90: 650},
    {age: 29, p10: 260, p50: 520, p90: 660},
    {age: 30, p10: 290, p50: 550, p90: 670}
  ];
  
  // Real Spanish data from Wordbank
  const spanishData = [
    {age: 16, p10: 5, p50: 35, p90: 140},
    {age: 17, p10: 8, p50: 48, p90: 165},
    {age: 18, p10: 12, p50: 78, p90: 220},
    {age: 19, p10: 18, p50: 105, p90: 260},
    {age: 20, p10: 25, p50: 145, p90: 315},
    {age: 21, p10: 38, p50: 180, p90: 375},
    {age: 22, p10: 55, p50: 218, p90: 425},
    {age: 23, p10: 75, p50: 258, p90: 470},
    {age: 24, p10: 98, p50: 285, p90: 515},
    {age: 25, p10: 125, p50: 335, p90: 555},
    {age: 26, p10: 155, p50: 375, p90: 590},
    {age: 27, p10: 185, p50: 415, p90: 615},
    {age: 28, p10: 215, p50: 455, p90: 630},
    {age: 29, p10: 245, p50: 495, p90: 640},
    {age: 30, p10: 275, p50: 525, p90: 650}
  ];
  
  const data = language === "English" ? englishData : spanishData;
  
  const individualChildren = showIndividual ? data.flatMap(d => {
    return d3.range(0, 20).map(i => ({
      age: d.age + (Math.random() - 0.5) * 0.5,
      vocab: d.p50 + (Math.random() - 0.5) * (d.p90 - d.p10) * 0.8
    }));
  }) : [];
  
  return html`
  <div style="text-align: center;">
    <h4>${language} Vocabulary Production (Wordbank)</h4>
    
    ${Plot.plot({
      marks: [
        showIndividual ? Plot.dot(individualChildren, {
          x: "age", 
          y: "vocab", 
          fill: "#2196f3", 
          opacity: 0.25,
          r: 2.5
        }) : null,
        Plot.areaY(data, {x: "age", y1: "p10", y2: "p90", fill: "#2196f3", opacity: 0.15}),
        Plot.line(data, {x: "age", y: "p50", stroke: "#2196f3", strokeWidth: 3, curve: "natural"}),
        Plot.line(data, {x: "age", y: "p10", stroke: "#90caf9", strokeWidth: 1.5, strokeDasharray: "3,2"}),
        Plot.line(data, {x: "age", y: "p90", stroke: "#1565c0", strokeWidth: 1.5, strokeDasharray: "3,2"})
      ].filter(Boolean),
      x: {label: "Age (months)", domain: [16, 30]},
      y: {label: "Vocabulary Size", domain: [0, 700], grid: true},
      width: 600,
      height: 240,
      marginLeft: 60
    })}
    
    <div style="margin-top: 8px; font-size: 0.85em;">
      <span style="color: #1565c0;">━━</span> 90th percentile  
      <span style="color: #2196f3; font-weight: bold;">━━</span> Median (50th)  
      <span style="color: #90caf9;">━━</span> 10th percentile
    </div>
    
    <p style="font-size: 0.75em; margin-top: 8px; color: #666;">
      Source: Frank et al. (2017). <em>Journal of Child Language</em>, 44(3), 677-694
    </p>
  </div>`;
}

Advantages of Cross-Sectional

⚡ Fast

• Test all ages in 1 month

• No waiting!

✅ No Attrition

• Each child tested once

• No dropout bias

💵 Cheap

• Single wave

• Lower cost

Disadvantages of Cross-Sectional

⚠️ Cohort Effects

• Born 2010 vs 2020?

• Can’t separate age from generation

📊 No Individuals

• Only between-subjects comparison

• Hide individual paths

❓ No Causation

• Can’t establish temporal order

• No pretest-posttest possible

Cohort Effect Problem

Same age, different childhoods = different experiences

{
  // Verified data: Liu et al. (2022) Scientific Reports, 12(1), 2618
  const cohorts = {
    2010: {screen: 2.0, outdoor: 4.5, vocab: 325},
    2015: {screen: 3.2, outdoor: 3.8, vocab: 310},
    2019: {screen: 4.1, outdoor: 2.5, vocab: 295},
    2020: {screen: 5.8, outdoor: 1.2, vocab: 270},
    2022: {screen: 5.2, outdoor: 1.8, vocab: 285}
  };
  
  if (cohortView === "Side-by-Side Comparison") {
    const c2010 = cohorts[2010];
    const c2020 = cohorts[2020];
    
    const compareData = [
      {cohort: "Born 2010", measure: "Screen", value: c2010.screen, color: "#4caf50"},
      {cohort: "Born 2020", measure: "Screen", value: c2020.screen, color: "#f44336"},
      {cohort: "Born 2010", measure: "Outdoor", value: c2010.outdoor, color: "#4caf50"},
      {cohort: "Born 2020", measure: "Outdoor", value: c2020.outdoor, color: "#f44336"}
    ];
    
    return html`
    <div style="text-align: center;">
      <h4>Both Age 4, Different Childhoods!</h4>
      
      ${Plot.plot({
        marks: [
          Plot.barY(compareData, {x: "measure", y: "value", fill: "color", fx: "cohort"}),
          Plot.text(compareData, {x: "measure", y: "value", text: d => d.value + "h", fx: "cohort", dy: -10})
        ],
        fx: {label: null},
        x: {label: "Daily Activity"},
        y: {label: "Hours per Day", grid: true, domain: [0, 7]},
        color: {legend: false},
        width: 550,
        height: 230
      })}
      
      <div style="background: #fff9c4; padding: 10px; border-radius: 6px; margin-top: 10px; font-size: 0.85em;">
        <strong>⚠️ Confound:</strong> Is difference due to age OR generation?
      </div>
      
      <p style="font-size: 0.75em; margin-top: 8px;">Liu et al. (2022). <em>Scientific Reports</em>, 12(1), 2618</p>
    </div>`;
  } else {
    const trendData = Object.keys(cohorts).map(year => ({
      year: parseInt(year),
      screen: cohorts[year].screen,
      outdoor: cohorts[year].outdoor
    }));
    
    return html`
    <div style="text-align: center;">
      <h4>Childhood Changing Across Generations</h4>
      
      ${Plot.plot({
        marks: [
          Plot.line(trendData, {x: "year", y: "screen", stroke: "#f44336", strokeWidth: 3}),
          Plot.dot(trendData, {x: "year", y: "screen", fill: "#f44336", r: 5}),
          Plot.line(trendData, {x: "year", y: "outdoor", stroke: "#4caf50", strokeWidth: 3}),
          Plot.dot(trendData, {x: "year", y: "outdoor", fill: "#4caf50", r: 5})
        ],
        x: {label: "Birth Year"},
        y: {label: "Hours per Day", grid: true, domain: [0, 7]},
        width: 600,
        height: 230,
        marginLeft: 60
      })}
      
      <div style="margin-top: 8px;">
        <span style="color: #f44336; font-weight: bold;">●</span> Screen  
        <span style="color: #4caf50; font-weight: bold;">●</span> Outdoor
      </div>
    </div>`;
  }
}

Cohort-Sequential Design

What is Sequential Design?

The Strategy:

Enroll multiple age groups
Follow each over time
Compare overlapping ages

= Mixed-Subjects Design

Visual:

Cohort A: ●──●──●
          3  4  5

Cohort B:       ●──●──●
                5  6  7

Overlap at age 5 reveals cohort vs age!

Sequential Structure

Time-Series Approach: Ages 3-9 in just 3 Calendar Years

Year 1

Cohort A → 3
Cohort B → 5
Cohort C → 7

Year 2

Cohort A → 4
Cohort B → 6
Cohort C → 8

Year 3

Cohort A → 5
Cohort B → 7
Cohort C → 9

✓ Result

Ages 3-9
In 3 years!
7 years saved!

🔍 Key Insight: Cohorts A & B both reach ages 5, 6, 7. If results match → age effect. If different → cohort effect!

Example: Math Motivation

Chouinard & Roy (2008): Does motivation drop at Grade 9?

Quasi-Experiment Design:

Cohort	Yr 1	Yr 2	Yr 3
Cohort 1	Gr 7	Gr 8	Gr 9 ⚠️
Cohort 2	Gr 9 ⚠️	Gr 10	Gr 11

Both cohorts pass through Grade 9

Finding:

📉 Both cohorts showed sharp motivation decline at Grade 9

Conclusion:

✓ High school transition effect
✗ NOT birth cohort

This is an age/grade effect!

When to Use Which Design?

Longitudinal (Within-Subjects + Pretest-Posttest)

“Does early language predict reading?”
Need time order + tracking same children

Cross-Sectional (Between-Subjects)

“How does vocab differ by age?”
Limited time/budget, no causation

Sequential (Mixed-Subjects + Time-Series)

“Is decline age or generation?”
Need to separate cohort effects

Small-n Designs

What Are Small-n Designs?

Small sample (n=1-15), MANY measurements (20-100+ per person)

Traditional Large-n:

100 people × 2 times
= Between-subjects focus

Small-n Design:

3 people × 80 times
= Within-subjects intensive

Visual:

Person A: ●●●●●●●●●●●●●... (80 obs)
Person B: ●●●●●●●●●●●●●... (80 obs)
Person C: ●●●●●●●●●●●●●... (80 obs)

When to Use Small-n?

🎯 Individual Therapy

Does THIS intervention work for THIS person?

🔬 Rare Conditions

Can’t find 100 participants

🏫 Applied Settings

Teachers need quick feedback

Memory Retention

Ebbinghaus tested his memory 300+ times (within-subjects)

{
  const run = runTrial2 || 0;
  const trials = [];
  
  if (run > 0) {
    // Generate trials from 1 to current run count
    for (let i = 1; i <= run; i++) {
      // Use fixed seed for each trial number to maintain consistency
      const seed = Math.sin(i * 12.9898) * 43758.5453;
      const randomVal = seed - Math.floor(seed);
      const recall = Math.max(15, 100 * Math.exp(-0.15 * i) + (randomVal - 0.5) * 5);
      trials.push({trial: i, recall: recall});
    }
  }
  
  return html`
  <div style="text-align: center;">
    ${trials.length > 0 ? Plot.plot({
      marks: [
        Plot.line(trials, {x: "trial", y: "recall", stroke: "#2196f3", strokeWidth: 2}),
        Plot.dot(trials, {x: "trial", y: "recall", fill: "#2196f3", r: 4})
      ],
      x: {label: "Trial Number", domain: [0, 30]},
      y: {label: "Recall %", domain: [0, 100], grid: true},
      width: 600,
      height: 200,
      marginLeft: 60
    }) : html`<div style="padding: 35px; background: #f0f0f0; border-radius: 8px;">Click "Run Trial" to add points</div>`}
    <div style="margin-top: 8px; display: flex; gap: 12px; justify-content: center;">
      <div style="background: #e3f2fd; padding: 8px; border-radius: 6px;">
        Trials: ${trials.length}
      </div>
      ${trials.length > 0 ? html`<div style="background: #fff3e0; padding: 8px; border-radius: 6px;">
        Current: ${trials[trials.length-1].recall.toFixed(1)}%
      </div>` : ''}
    </div>
  </div>`;
}

A-B-A-B Design

Does training increase vocabulary?

{
  const phaseData = {
    "A1: Baseline": {sessions: [12, 11, 13, 12, 14, 12, 13], color: "#e0e0e0", note: "~12 words baseline"},
    "B1: Training": {sessions: [14, 17, 22, 28, 35, 38, 42], color: "#4caf50", note: "✓ Increases with training!"},
    "A2: Baseline": {sessions: [40, 38, 36, 35, 34, 33, 32], color: "#e0e0e0", note: "Plateaus without training"},
    "B2: Training": {sessions: [36, 42, 48, 55, 60, 65, 68], color: "#4caf50", note: "✓ Increases again!"}
  };
  
  const data = phaseData[abaPhase];
  const sessionData = data.sessions.map((words, i) => ({session: i + 1, words: words}));
  
  return html`
  <div style="text-align: center;">
    ${Plot.plot({
      marks: [
        Plot.line(sessionData, {x: "session", y: "words", stroke: data.color === "#4caf50" ? "#2e7d32" : "#757575", strokeWidth: 2}),
        Plot.dot(sessionData, {x: "session", y: "words", fill: data.color === "#4caf50" ? "#4caf50" : "#9e9e9e", r: 5})
      ],
      x: {label: "Session", domain: [1, 7]},
      y: {label: "Words", domain: [0, 70], grid: true},
      width: 600,
      height: 200,
      marginLeft: 60
    })}
    <p style="margin-top: 8px; font-weight: bold;">${data.note}</p>
    <p style="font-size: 0.85em; margin-top: 6px;">If behavior changes with intervention → intervention causes change</p>
  </div>`;
}

Example: Pretest-Posttest

Does training increase vocabulary? (Within-subjects comparison)

{
  const phaseData = {
    "Pretest": {
      sessions: [12, 11, 13, 12, 14],
      color: "#e0e0e0",
      note: "~12 words at baseline (before training)",
      title: "🔴 Pretest Phase"
    },
    "Training": {
      sessions: [12, 11, 13, 12, 14, 17, 22, 28, 35],
      color: "#fff3cd",
      note: "Words improve during training sessions",
      title: "🟡 Training Phase (Intervention)"
    },
    "Posttest": {
      sessions: [12, 11, 13, 12, 14, 17, 22, 28, 35, 42, 45, 48, 50],
      color: "#d4edda",
      note: "✓ Significant improvement from pretest to posttest!",
      title: "🟢 Posttest Phase"
    }
  };
  
  const data = phaseData[pretestPhase];
  const sessionData = data.sessions.map((words, i) => ({session: i + 1, words: words}));
  
  return html`
  <div style="text-align: center;">
    <h4>${data.title}</h4>
    ${Plot.plot({
      marks: [
        Plot.line(sessionData, {x: "session", y: "words", stroke: "#2196f3", strokeWidth: 2}),
        Plot.dot(sessionData, {x: "session", y: "words", fill: data.color === "#e0e0e0" ? "#9e9e9e" : (data.color === "#fff3cd" ? "#ff9800" : "#4caf50"), r: 5})
      ],
      x: {label: "Session", domain: [1, 13]},
      y: {label: "Words Produced", domain: [0, 55], grid: true},
      width: 600,
      height: 200,
      marginLeft: 60
    })}
    <p style="margin-top: 8px; font-weight: bold;">${data.note}</p>
    <p style="font-size: 0.85em; margin-top: 6px;">Pretest-Posttest design: Compare same child before vs after intervention</p>
  </div>`;
}

Strengths of Small-n

🎯 Individual Proof

• Shows effect for THIS person

• Within-subjects power

🔄 Flexible

• Can adapt real-time

• Responsive to change

✅ Feasible

• Only need 1-15 participants

• Practical for applied settings

Limitations of Small-n

❓ Generalization?

• Works here, but elsewhere?

• Can’t test between-subjects

⏱️ Fatigue Effects

• Need 20-100+ measurements

• Risk of exhaustion bias

🔁 Carryover

• Can’t truly “un-learn”

• Later trials affected

Sources of Bias

Four Major Threats

1. Attrition
People drop out (not random)

2. Practice Effects
Repeated testing improves scores

3. Cohort Effects
Different generations

4. History Effects
External events

Threat 1: Attrition

Who leaves? NOT random!

👨‍👩‍👧‍👦 Busiest families
😰 Struggling children
💸 Lower SES families

Visual:

Started: ●●●●●●●●●● (100)
         ↓ ↓ ↓
Ended:   ●●●●●●●     (70)

Biased sample!

Affects: Longitudinal, Sequential designs

Threat 2: Testing Effects

Repeated testing = Practice!

📝 Learn test format
🧠 Remember items
✍️ Develop strategies

Visual:

Test 1: 70% ────┐
Test 2: 75% ────┤ Practice!
Test 3: 80% ────┘ Not development!

Affects: Longitudinal, Sequential, Small-n (within-subjects designs)

Threat 3: Cohort Effects

Different childhoods!

📱 2010: No tablets
📱 2020: 5h screen/day

🏞️ 2010: 4.5h outdoor
🏠 2020: 1.2h outdoor

Visual:

Born 2010:  🏃‍♂️🌳📖
Born 2020:  📱💻🎮

Same age, different worlds!

Affects: Cross-sectional (between-subjects designs)

Threat 4: History Effects

External events!

🦠 Pandemic
📉 Economic crisis
🌊 Natural disasters

Time-Series Confound:

2018: Normal ──●──●──●
2019: Normal ──●──●──●
2020: Event! 🦠
2021: Changed ──●──●──●

Changes = Event, not development!

Affects: All longitudinal and time-series designs

Key Takeaway

The best design answers YOUR question!

Consider:

🎯 Your research question
⏰ Time and money available
👥 Sample accessibility
🔍 What biases matter most?

Naturalistic Recordings

Why Record Natural Interactions?

Lab Testing Limitations:

🏫 Artificial environment

📸 One-off behavior snapshot

👨‍🔬 Researcher-guided tasks

😟 Stress effects

Naturalistic Recording Benefits:

🏠 Real-world behavior

📈 Ongoing development

🎨 Child-directed activity

💬 Authentic language/interaction

Methods of Recording

🎙️ Audio Recording

Easiest to collect

Portable devices

Language focus

Privacy friendly

📹 Video Recording

Captures nonverbal behavior

Joint attention markers

Motor skills visible

Privacy concerns

🔀 Hybrid Approaches

Audio + observer notes

Wearable cameras

Multiple perspectives

Mixed data streams

Analyzing Naturalistic Data

📝 Transcription

Word-by-word or phrase-based

Coding conventions (CHAT, ELAN)

Time-aligned to audio/video

Intensive & expensive

🏷️ Coding Schemes

Developmental milestone markers

Social interaction patterns

Language complexity measures

Context-dependent categories

AI-Driven Automated System: LENA System

What is LENA?

📊 Language Environment Analysis

🎤 Wearable recorder (child)

⏱️ 16+ hours daily recording

🤖 Automated + manual analysis

Captures:

📢 Adult word count

🗣️ Child vocalization rate

💬 Conversational turns

🔊 TV/noise exposure

Finding: 30M word gap!

🔗 lena.org

Advantages of Naturalistic Data

🌍 Ecological Validity

Real-world behavior observed

Natural developmental contexts

⏳ Longitudinal Feasibility

Easy to repeat over time

Captures growth trajectories

📚 Rich Contextual Info

Environmental influences visible

Family practices documented

Challenges of Naturalistic Data

⏱️ Time Intensive

Hours of recording per family

Days of transcription work

🔍 Reliability Issues

Observer bias in coding

Intercoder agreement critical

🔐 Privacy & Ethics

Informed consent for recording

Data security requirements

Thank You

Questions?