STUDENT PERFORMANCE EVALUATION

Using subgroup analysis to inform course structure and pedagogy.

OVERVIEW

Understanding why students perform differently helps inform decisions about course structure and pedagogy. Many variables contribute to differences in student performance; unfortunately, most of these are not readily available to instructors.

For example, I know that aptitude test scores and high school rank account for a significant amount of performance variance at the university level, but I don’t have access to this information for my students. Mental health also significantly influences academic performance, but I can’t reliably evaluate levels of stress or grit. Add in a myriad of additional influences… motivation, mindset, sleep quality, study time, etc., and you quickly recognize that the process of understanding and predicting student performance presents many challenges.

However, we can often make use of limited data to conduct a meaningful analysis and produce actionable results. In this report I demonstrate how using a single demographic variable (student major) to evaluate student performance can provide helpful insights about the populations of students completing my course.

SETUP

The dataset in this example has a simple structure. It includes a single measured variable and two manipulated variables¹, all of which can be readily ascertained by an instructor.

• The measured variable reflects the course performance of each individual. This is a continuous variable, which essentially means that it has a wide range of possible values.
• Each manipulated variable reflects the group membership of each individual. These are categorical variables, which means that they have a limited number of possible values.

The data were collected across multiple semesters of my undergraduate psychopharmacology course.

• The measured variable (Percent) is the final course grade, expressed as percentage of points earned.
• The primary manipulated variable (BIOL) is based on a student’s major. Students in a major with relatively low biology emphasis (mostly psychology, social work, and substance abuse counseling) were categorized as Low, whereas students in a major with relatively high biology emphasis (mostly nursing, neuroscience, and biology) were categorized as High.
• The secondary manipulated variable (Grade) is based on Percent, and ranges from A - F. For example, when Percent is equal to or greater than 94, Grade equals “A”; when Percent is greater than or equal to 88, and less than 94, Grade equals “AB”, etc. There are 8 levels of Grade, equally binned in 6-percent increments.

ANALYSIS

I want to know if students perform differently depending on major category; that is, does Percent differ based on BIOL? To answer this question I need descriptive statistics, a visualization of the data, and an inferential analysis. Since I am comparing two independent groups of a single categorical variable on a single continuous variable, a simple t-test will suffice.

BIOL	n	median	mean	sd	min	max
High	83	87.75	86.93	6.04	71.4	98.6
Low	109	79.00	77.50	10.49	30.5	98.9

Interpretation

Students in the High category of BIOL have mean and median Percent scores about 9 points greater than students in the Low category. The t-test results indicate that this difference is statistically significant, with a large effect size.

In other words, the data indicate that one group of students tends to outperform a different group of students by about 1 standard deviation, and that this difference is unlikely to occur due to chance. As such, it is reasonable to conclude that the groups represent different populations of students (as opposed to different samples from the same population).

However, it is worth pointing out that some overlap exists between the groups. They are not completely distinct.

Informing Decisions

The results suggest that a stronger background in biology facilitates success in psychopharmacology. In an attempt to help students with weaker backgrounds in biology achieve success, I have used the results of this analysis (along with qualitative data including personal observations and course evaluation comments) to alter the course structure and instructional approach in several ways:

1. The course now begins with a week of background material related to behavioral neuroscience. The goal of this change is to level the playing field somewhat, so that all students have some exposure to relevant fundamental concepts before we delve into drugs.
2. I refined the activities and assignments to streamline the course and focus attention on the most important content. For example, I removed group projects and presentations because these required spending a disproportionate amount of time and effort relative to the benefits, and they did not provide an easy way to assess individual comprehension of the material.
3. I pared back the content coverage from 12 sections to 10. This allows for some buffer weeks in the semester so we can spend extra time on material that students find particularly challenging.
4. Students can use a single-sided sheet of handwritten notes on every quiz, and a double-sided sheet of handwritten notes on the final exam. This encourages students to engage with the material outside of class.
5. I discuss and explain test items in class after each quiz. Extra class time is devoted to reviewing concepts that the class found difficult before moving forward with new content.

Limitations

Based on the available data, I cannot fully conclude that the reason for the performance difference is simply due to increased exposure to biology coursework prior to completion of psychopharmacology.

Several additional plausible explanations include:

• Students in High BIOL majors are better academically prepared, and perform better in psychopharmacology simply because they are stronger students in general. I happen to know that, at my institution, nursing and neuroscience majors generally have higher ACT scores and high school ranks than psychology and social work majors.
• Students in Low BIOL majors are less interested in biology, and therefore less motivated to fully engage with the material. They may be just as capable of success in this course as other students, but choose to spend their study time and cognitive resources with other coursework.
• Students in High BIOL majors are more motivated to earn a high GPA. Nursing majors, for example, need to maintain a certain GPA to remain in the program. Many biology and neuroscience majors plan to attend graduate school and need competitive grades.
• Students in Low BIOL majors are more likely to be non-traditional students. These students are older (and therefore more temporally distant from biology coursework), more likely to have full-time jobs, and more likely to have family caretaking obligations (typically children and/or parents). These responsibilities limit the amount of time available to study psychopharmacology as compared to traditional students.

These explanations are non-mutually exclusive and would require additional sources of data to evaluate. This is also not an exhaustive list of plausible explanations.

CONCLUSIONS

Instructing multiple populations of students in the same course is a difficult situation. The course must be at least somewhat challenging for every student while simultaneously providing every student with opportunities for success. While ultimately it is impossible to account for all of the variables that influence student performance, analyzing limited available data can still provide results that inform decisions about course structure and pedagogy in an attempt to enhance instructional effectiveness.

CONTACT INFORMATION

David J. Bauer, Ph.D.
Professor of Psychology
Viterbo University
djbauer@viterbo.edu

Footnotes

1. One of the challenges in understanding data analysis across domains is that different terms are often used to describe the same thing. A measured variable is also referred to as a predicted, criterion, outcome, response, or dependent variable. A manipulated variable is also referred to as a covariate, or as a predictor, feature, or independent variable.