Non-Cognitive Outcomes in 2015 PISA Data: Analysis of School Science Resources

Alex Lishinski
4/25/17

Non-Cognitive Outcomes

Non-Cognitive outcomes are widely recognized as important:

  • Schooling, employment, work experience, and choice of occupation are affected by noncognitive skills. (Heckman et al., 2006)
  • Skills and behaviors in high school explain a substantial portion of the socioecononomic, sex, and racial and ethnic gaps in educational attainment and earnings. (Lleras, 2008)
  • Non-cognitive skills and college premiums among women account for nearly 90 percent of the gender gap in higher education after controlling for high school achievement. (Jacob, 2002)

PISA Non-Cognitive Questions

2015 PISA included non-cognitive items related to science, including:

  • Science Self-efficacy
  • Broad interest in learning science
  • Enjoyment of science
  • Instrumental motivation in science
  • Environmental Awareness

These 5 outcomes cover a broad range of positive non-cognitive outcomes from science

School Policies and Resources for Science

2015 PISA also collected school level data on a number of resources relating to science:

  • Whether or not schools had science competitions or clubs
  • Whether the science department was well equipped
  • Whether the science department has extra laboratory equipment and spends extra money on it
  • Whether science teachers are the best educated teachers at the school

Goal of the analysis: Isolate effects of science specific resources on non-cognitive outcomes in science

Method

The PISA data comes from 17911 schools in 73 countries

To deal with the clustered data, HLM was used to estimate the effects of science specific resources

3-level model: students nested in schools nested in countries

Models

Unconditional model tells us the proportions of variance at each level

                    Pct Sci Eff Pct Sci Enj Pct Sci Int Pct Sci Mot
Students (level 1)         0.74        0.67        0.69        0.70
Schools (level 2)          0.14        0.15        0.14        0.12
Countries (level 3)        0.12        0.18        0.17        0.18
                    Pct Env Awr
Students (level 1)         0.68
Schools (level 2)          0.20
Countries (level 3)        0.13

There is significant variance at all 3 levels

The most variance is at the student level, so subsequent models will add student-level covariates

Models: Adding level 1 covariates

Estimating school effects generally involves controlling for student characteristics such as gender, ethnicity, and SES (Konstantopoulous and Hedges 2008)

PISA data also contains a number of other student characteristics specific to science, including:

  • Student's previous experiences in science
  • Student's parents' view of science
  • Whether student encouraged in science
  • Amount of time spent per week studying science
  • Whether student studied science outside of their normal class

Models: Adding level 1 covariates

          L1 Var Exp
Sci Eff  0.068540644
Sci Enj  0.038805427
Sci Int  0.071558973
Sci Mot -0.002851335
Env Awr  0.080668136

The BIC and AIC show that the fit of the model has improved with the addition of the covariates.

Models: Adding level 2 covariates and predictors

  • PISA data contains common school characteristic variables like class size
  • PISA data also includes science specific variables like proportion of teachers with advanced certification
  • Adding the school variables of interest to the model, we can see whether the have any effect, compared to teaching characteristics
  • Also adding a random coefficient for the science resources variable

Model Fixed Effects

Science Efficacy

  (Intercept)     HOME_POSS   HOME_ED_RES   STUD_Gender        PAR_ED 
        -0.17          0.12          0.08         -0.05          0.01 
 PAST_SCI_ACT  PAR_VIEW_SCI  PAR_VIEW_ENV      MINS_SCI        IMMIGR 
         0.16          0.07         -0.01          0.00          0.01 
      SCI_POL SCI_RESOURCES     TEACH_SUP      IB_TEACH      TD_TEACH 
         0.01          0.00          0.05          0.15          0.05

Science Enjoyment

  (Intercept)     HOME_POSS   HOME_ED_RES   STUD_Gender        PAR_ED 
         0.02          0.02          0.09         -0.04          0.00 
 PAST_SCI_ACT  PAR_VIEW_SCI  PAR_VIEW_ENV      MINS_SCI        IMMIGR 
         0.17          0.10         -0.01          0.00          0.02 
      SCI_POL SCI_RESOURCES     TEACH_SUP      IB_TEACH      TD_TEACH 
         0.01          0.01          0.11          0.07          0.14

Model Fixed Effects

Science Interest

  (Intercept)     HOME_POSS   HOME_ED_RES   STUD_Gender        PAR_ED 
         0.13          0.02          0.07         -0.17          0.00 
 PAST_SCI_ACT  PAR_VIEW_SCI  PAR_VIEW_ENV      MINS_SCI        IMMIGR 
         0.12          0.07          0.00          0.00          0.01 
      SCI_POL SCI_RESOURCES     TEACH_SUP      IB_TEACH      TD_TEACH 
         0.01          0.00          0.04          0.07          0.10

and Science Motivation

  (Intercept)     HOME_POSS   HOME_ED_RES   STUD_Gender        PAR_ED 
         0.05         -0.01          0.07          0.02          0.00 
 PAST_SCI_ACT  PAR_VIEW_SCI  PAR_VIEW_ENV      MINS_SCI        IMMIGR 
         0.08          0.09         -0.01          0.00          0.02 
      SCI_POL SCI_RESOURCES     TEACH_SUP      IB_TEACH      TD_TEACH 
         0.00          0.01          0.11          0.10          0.05

Model Fixed Effects

Environmental Awareness

  (Intercept)     HOME_POSS   HOME_ED_RES   STUD_Gender        PAR_ED 
        -0.16          0.15          0.05         -0.01          0.01 
 PAST_SCI_ACT  PAR_VIEW_SCI  PAR_VIEW_ENV      MINS_SCI        IMMIGR 
         0.13          0.06          0.01          0.00          0.00 
      SCI_POL SCI_RESOURCES     TEACH_SUP      IB_TEACH      TD_TEACH 
         0.01          0.01          0.06          0.05          0.14

Conclusions

The results of these models show that the non-cognitive outcomes in science classes are not effected by school science resources.

Controlling for student background factors both general and related to science, as well as school factors related to science makes resources have no effect, and there is almost no differential effect between countries.