Estimating Percentage of Non-STEM Students Achieving Competencies/Domain Across Semesters
Lorenzo Pla Serrano, Master of Science in Biostatistics, University of Colorado Anschutz
2025-11-05
Context
A student is considered to have achieved a competency if they score a 5 or higher on that competency. Students could have scored 1 to 8, and were scored by their instructor using a common rubric. In this way, students either achieve or do not achieve a competency, a binary outcome. Therefore, we would be interested in estimating the proportion of students achieving each competency in each semester, regardless of the course or instructor.
Students belong to specific semester and instructor-course group.
- Semesters (time): Fall 2022, Spring 2023, Fall 2023, Spring 2024, Fall 2024, Spring 2025
- Instructor-Course Groups (sampling): 3016-WCP, 3026-RBD, 3026-MPA, 3027-MPA, 3028-RBD, 3016-MPA
Domains and Their Competencies
| Dominio | Componentes |
|---|---|
| Tecnología | gráfica/excel, línea de tendencia |
| Investigación científica | hipótesis, justificación de experimento, identificación del diseño experimental, crítica del experimento, conclusiones |
| Información | evaluación de fuentes, citas APA |
| Comunicación | Comunicación escrita |
How domain scores were calculated: Domain scores were calculated by first counting how many competencies within each domain achieved a score of 5 or more, then applying domain-specific thresholds to determine achievement. Technology, Information, and Communication domains required students to score 5+ on all competencies within that domain (2 of 2, 2 of 2, and 1 of 1, respectively) to be classified as “Achieved.” In contrast, the Scientific Research domain used a more lenient threshold, requiring students to score 5+ on at least 4 out of 5 competencies (80%) to achieve the domain score, allowing for one competency to fall below the threshold.
Boostrap CI
Statistical Note: Resampling at the instructor–course (cluster) level within each semester make the SEs and CIs robust to within-instructor correlation and heteroskedasticity. This ensures that the CIs better reflect the uncertainty in the estimates of the percentage of students achieving the given competency/domain.
Approach: cluster bootstrap percentile CI of the student-weighted proportion per semester
Competency I: Graphing
Competency II: Plotting Linear Trend
Competency III: Formulating Hypothesis
Competency IV: Justifying Experiment
Competency V: Identifying Experimental Design
Competency VI: Critiquing Experiment
Competency VII: Reaching Conclusions
Competency VIII: Evaluating Sources
Competency IX: Ethical Citing
Competency X: Written Communication
Competency IX: Ethical Citing
Competency X: Written Communication
Domain: Technology
Domain: Scientific Research
Domain: Information
Domain: Communication
Questions/Preguntas
- ¿El % que aparece en cada barra corresponde a la media aritmética /
promedio?
- El porcentaje corresponde a la proporción observada de estudiantes que lograron la competencia en ese semestre. Matemáticamente, cuando trabajamos con resultados binarios (logró = 1, no logró = 0), la proporción es equivalente a la media aritmética de esos valores binarios. Sin embargo, no es un promedio de las puntuaciones originales (1-8). No estamos sumando todas las puntuaciones y dividiendo. Solo estamos contando cuántos estudiantes llegaron al punto de corte (≥5) y expresándolo como porcentaje. Esta proporción representa la tasa de logro típica para cada semestre.