Definitions

Ceiling/floor effect occur

• with tests that are relatively easy/difficult,
• when a substantial proportion of individuals obtain either maximum/minimum or near-maximum/minimum scores
  
• when individuals cannot demonstrate the true extent of their abilities
• resulting in score distributions that are compressed at the upper/lower end.

Current Practices in Empirical Research

• Ignore the present ceiling/floor effects and treat the ceiling/floor data as if they were the true values
• Treat the ceiling/floor data as if they were the true values but report ceiling/floor proportion or do a normality check
• Adjust the experimental procedure in order to lessen or to avoid ceiling/floor effects
• Discard the ceiling/floor data
• Transform the data to make them more normal

Current State in Methodological Discussion: Consequences of Ceiling/floor Effects

• underestimated means and variances
• skewed distributions
• attenuated correlation and reliability
• biased linear model results
• biased longitudinal data analysis results

(Jennings & Cribbie, 2016; Wang et al., 2008; uttl, 2005)

Potential Statistical Methods in Handling Ceiling/Floor Effects

1. removing ceiling/floor values
2. ignore ceiling/floor effects
3. median regression
4. permutation regression
5. censored regression
6. Bayesian censored regression estimating censored values
7. Bayesian censored regression with likelihood integration

Maths on Method 1 and 2

• Notations:
  • \(\phi\): normal probability density function
  • \(\psi\): normal cumulative density function
  • \(a\) and \(\alpha\): unstandardized and standardized floor thresholds
  • \(b\) and \(\beta\): unstandardized and standardized ceiling thresholds
• Under Method 1: Truncation
  • \(E(Y|a<Y<b)=\mu+\sigma\frac{\phi(\alpha)-\phi(\beta)}{\psi(\beta)-\psi(\alpha)}\)
  • \(Var(Y|a<Y<b)=\sigma^2[1+\frac{\alpha\phi(\alpha)-\beta\phi(\beta)}{\psi(\beta)-\psi(\alpha)}-(\frac{\phi(\alpha)-\phi(\beta)}{\psi(\beta)-\psi(\alpha)})^2]\)
• Under Method 2, e.g., Right Censoring, let \(\lambda=\frac{\phi(\alpha)}{1-\psi(\alpha)}\) and \(\delta=\lambda^2-\lambda \alpha\)
  • \(E(Y_r)=\psi(a)a+(1-\psi(a)a)(\mu+\sigma\lambda)\)
  • \(Var(Y_r)=\sigma^2(1-\Psi(\alpha))[(1-\delta)+(\alpha-\lambda)^2\Psi(\alpha)]\)

Method 3 and 4: Nonparametric Regressions

• Median Regression:
  • minimizes \(\sum_i |e_i|\)
  • robust to nonnormal errors and outliers
  • requires larger data for both shape and parameter
  • computationally intensive
• Permutation Regression:
  • permutating \(y\) values and obtain \(SS_{b_i}=\hat{\beta}_{b_i}^TV\hat{\beta}_{b_i}\)
  • we have \(p=\sum (\frac{SS_{b_i}}{SS_{r_i}}>\frac{SS_{b_0}}{SS_{r_0}})/N\)
  • robust to nonnormal errors and outliers
  • computationally intensive as \(N\) increases

Method 5 and 6 and 7: Censored Regression

• Under frequentist estimation: maximizing \(\text{log} L=\sum^N_{i=1} [I^a_i \text{log} \psi(\frac{a-X^T_i B}{\sigma_{\epsilon}})+I^b_i \text{log} \psi (\frac{x_i^TB-b}{\sigma_{\epsilon}})+(1-I_i^a-I_i^b)(\text{log}\phi(\frac{y_i^*-x_i^TB}{\sigma_{\epsilon}})-\text{log}\sigma_{\epsilon}]\)
• Bayesian Estimation: Posterior probability \(\propto\) Likelihood \(\times\) Priors \(p(\theta|x)=\frac{p(x|\theta)}{p(x)}p(\theta)\)
  • Parameters:
    • \(\sigma \sim \text{Uniform}[0,\infty)\)
    • \(\alpha, B[p] \sim \text{Uniform}(-\infty,\infty)\)
  • Estimating censored values:
    • additional parameters: \(y_u[N_u] \sim \text{Uniform}[u,\infty)\) and \(y_l[N_l] \sim \text{Uniform}(-\infty,l]\)
    • Only using normal likelihood and given drawn \(\alpha, B[p]\), we evaluate the likelihood for uncensored data and for left- and right-censored data separately
  • Integrating likelihood function: Using the same joint likelihood function as the frequentist approach

Data Generation and Simulation Design

• Data generation:
  • fixed parameters: \(\beta_1=.1, \beta_2=.3, \beta_3=.5\) corresponding to Cohen’s small, medium, large effects
  • random variables: \(X \sim N(0,1)\), \(\epsilon \sim N(0,1)\) and \(Y=\beta_1 X_1+\beta_2 X_2 +\beta_3 X_3 +\epsilon\)
  • inducing ceiling/floor effects: \(y_c = \begin{cases} a, & \mbox{if } y>a \\ y, & \mbox{if } a<y < b \\ b, & \mbox{if } y \geq b \end{cases}\)
• Conditions:
  • sample size \(N \in \{50, 200\}\)
  • floor/ceiling proportions:
    • only ceiling: .1, .2, .3
    • symmetric (.15,.15) and asymetric (.2,.1) floor/ceiling effects
• 1000 replications

Results: Type I Error Rates and Power

Results: Parameter Estimates

Conclusion

• Median regression and naive methods:
  • as low as 1/10 power
  • as much as -60% relative bias
  • extremely conservative (median regression)
• Permutation regression:
  • acceptable power and type I error rates
  • as much as -40% relative bias
• Censored regression:
  • good power and type I error rates
  • frequentist approach: highest power; liberal at n=50
  • two bayesian approaches performed similarly
• Recommendation: mainly drive by computational considerations
  • Use frequentist approach if n>50 for computational reason
  • Use bayesian approach with integration