Introduction

In a previous article 1 and in a monograph2, we have discussed a methodological problem in comparing performance of a PLTL group of students with a non-PLTL group. Often, average test scores between the two groups are compared. On first glance, this appears to be a reasonable and straightforward procedure. However, this approach is very likely to produce misleading and incorrect conclusions because PLTL groups have overwhelmingly been found to have a higher retention rate than non-PLTL groups, confounding the grade comparson.

Let us explore the situation qualitatively with a reductio ad absurdum argument.

Assume two classes of students, each with one hundred initial members. At the end of the semester ninety students drop out of class one, while the remaining 10 students all obtain a grad of “B”, or 3. In the second class,only ten students drop out, while the remaining students have fifteen A’s, fifteen B’s and sixty C’s, resulting in an average grade of 2.5. According to the direct comparison of average grade, class one had the superior performance, a patently absurd result. Nevertheless, the method of comparison is found in many studies of PLTL, and persists in the literature.

The resolution of this problem lies in establishing some measure of the initial ability of the students so as to make a fair comparison: SAT scores for instance as a measure of prior ability for students taking General Chemistry.

Comparative Linear Regression after Zar

However, there is no necessity for establishing equivalnce for groups. What we would rather look for is the impact of PLTL for a specific SAT score, PLTL vs non-PLTL. Zar3 has described a method for comparing Y values of two different fitted lines at a given X value. Below, we provide a sample analysis of smulated data wth R that we hope could be used as a template for PLTL studes.

#  Comparative linear regression following Zar

#   PLTL vs no-PLTL,  exam score versus prior gpa

#   Compare two predicted y points at same x value  p value

#   Graph of data and two lines


# clear data 
rm(list = ls())        # clear data
par(mfrow=c(1,1))

#  initialize variables
sde  <-12      #   standard deviation 
nt1 <- 100 
nt2 <- 100     #   number of simulated points 

#  no pltl

#  vectorized calculation, simulated exam vs gpa from 1.5 to 3.5

rnd1 <- rnorm(nt1,0,sde)    # random noise
rnd2 <- rnorm(nt2,0,sde)
i1 <- seq(1,nt1,1)
i2 <- seq(1,nt2,1)
x1 <- 0.02 * i1 +  1.5     # sequence of gpa values
x2 <- 0.02 * i2 + 1.5

y1 <- -5.75 + (0.02 * i1 + 1.5) * 22.5 + rnd1   # non-pltl exam scores

y2 <- 6.82 + (0.02 * i2 + 1.5) * 20.7 + rnd2   # pltl exam scores


LF1 <- lm(y1 ~ x1)      #  regressions
LF2 <- lm(y2 ~ x2)

summary(LF1)  # regression summary
## 
## Call:
## lm(formula = y1 ~ x1)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -39.440  -8.241   0.795   7.820  32.325 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    1.508      5.681   0.265    0.791    
## x1            18.890      2.206   8.563 1.57e-13 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 12.73 on 98 degrees of freedom
## Multiple R-squared:  0.428,  Adjusted R-squared:  0.4222 
## F-statistic: 73.33 on 1 and 98 DF,  p-value: 1.57e-13
summary(LF2)
## 
## Call:
## lm(formula = y2 ~ x2)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -30.482  -7.708   2.482   8.109  19.862 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)   10.980      4.875   2.252   0.0265 *  
## x2            19.182      1.893  10.135   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 10.93 on 98 degrees of freedom
## Multiple R-squared:  0.5117, Adjusted R-squared:  0.5068 
## F-statistic: 102.7 on 1 and 98 DF,  p-value: < 2.2e-16
cf1 <- coef(LF1)
cf1
## (Intercept)          x1 
##    1.507956   18.889595
int1 <- cf1[1]       #  use later with plotting lines 

sl1 <- cf1[2]

cf2 <- coef(LF2)
cf2
## (Intercept)          x2 
##    10.98015    19.18188
int2 <- cf2[1]       #  use later with plotting lines 

sl2 <- cf2[2]


#   Zar test comparing two lines

#  t = Y1-Y2/(Sy1-Sy2)

#   sum of the square residuals 

ss1 <- sum(resid(LF1)^2)
ss2 <- sum(resid(LF2)^2)

DF1 <-  nt1 - 2 
DF2 <-  nt2 -2

# pooled degrees of freedom 300-4 = 296

pDF <- DF1 + DF2

# sy1-sy2 = (s2y*x) (1/n1 + 1/n2 +  (X-Xa)^2 / sum x^2 )

# pooled residual mean square zahr

#  (s2Y*X)p =  (ss1 + ss2)/(2DF)

tryx1 <- 2.5       #  for  gpa 2.5

diff1 <- (tryx1-mean(x1))^2/(sum(x1^2)) + (tryx1-mean(x2))^2/(sum(x2^2))

First1  <- ((ss1 + ss2)/(DF1 + DF2)) * (1/nt1 + 1/nt2 + diff1)  

Second <- sqrt(First1)

#  predicted difference y1 and y2

Pr <-predict(LF1)
Pr2 <-  predict(LF2)

Pr[100] 
##      100 
## 67.62154
Pr2[100]    
##      100 
## 78.11673
dy100 <- Pr2[100]-Pr[100]
dy100
##      100 
## 10.49519
#  difference in predicted  values


tval100 <- dy100/Second    #  t value at  gpa = 2

#   
probt <- pt(tval100,(DF1+DF2))

twotail <- 2*(1-probt)

twotail      #  this is the statistic for comparing y t common x
##          100 
## 2.451169e-09
ttest <- t.test(y1,y2)

ttest$p.value    
## [1] 1.353772e-05
# two scatterplots and two regressions 


plot(x1,y1,col="red",cex=0.5,xlab="",ylab="",xaxt="no",yaxt="no")

par (new = TRUE)

plot(x2,y2, col = "blue", xlab = "Prior GPA", ylab = "Exam Score",ylim=c(0,100), cex = 0.5)

abline(a = int1, b = sl1, col = "red",xlim = c(1,4), ylim = c(0,100))



abline(a =  int2, b = sl2, col = "blue", xlim = c(1,4),ylim = c(0,100))

legend("topleft", c("non-PLTL", "PLTL"),
       lty = c(1,1),
       col = c("red", "blue"))

Discussion

  1. Gosser,David (2011). The PLTL Boost. Progressions↩︎

  2. Gosser,David (2011).PLTL: Origns, Research, and Practice. Linus Publcatons↩︎

  3. Zar (1987). Bioanalytcal Statistcs↩︎