Making Sense of Course Evaluation Data
Maksim Kokushkin, Assistant Professor of Sociology, Member of the Faculty Teaching And Learning Committee
April 8, 2016
Executive Summary
This report explores course evaluation data from the first two terms of the academic 2015-2016. The analysis below starts with the tidy dataset and applies correlation, cluster, and factor nalysis consecutively. The main findings are that: - the survey instrument is generally solid; - focusing on individual items for assessment purposes may misrepresent the data; - the item Effort stands needs to be investigated further, as it is generally weakly associated with the other substantive items. The results from the exploratory analysis presented here can be used to rebalance the survey instrument and to create working assessment indices based on the data. The report concludes with a recommendation for composite measures to replace the currently used summary measures “Instructor Rating” and “Course Rating”. The recommendation results from the need for a more complex and evidence-based assessment of course evaluation data.
Overview and Preprocessing
The raw data are restructured and transformed into a single csv data file with cases as rows and variables as columns. The raw data are in Excel format where each sheet corresponds to a course/instructor id (SecSyn). In order to create a functional dataset, the two Excel files (one from Fall 2015-2016 and one from Winter 2015-2016) are converted to cvs files, each corresponding to a sheet from the original file. The conversion of the data from each term is done using an html application created by Chris West. After that, due to inconsistencies in the Winter 2016 data, the variable SecSyn is populated using the filenames of the individual csv files, which correspond to the orignal tab names, i.e. course/instructor ids. The resulting dataset contains 27 variables and 6,106 observations. The variables are as follows and value labels are listed in Appendix 1.
## Warning: package 'xtable' was built under R version 3.2.3Correlation Analysis
The correlation analysis shows that all bivariate correlations between substantive items are statistically significant. The strongest correlation is between the two legacy summary measures (Course and Instructor) with r = 0.8, the only coefficient equal or above 0.8 (See Appendix 2). There are four bivariate correlations with coefficients between 0.7 and 0.8, three of which are between Techniques and the summary measures (Course, Instructor and Value). The fourth correlation with a exceeding 0.7 but below 0.8 is between Thinking Critically and Deep Understanding of Subject, r = 0.73. Therefore, three preliminary conclusions can be drawn:
- Statistically, the instrument is internally valid. That makes it a reliable tool for surveying students.
- The summary measures are somewhat disconnected from the other substantive measures, except Techniques. That justifies the development of composite substantive measures that highlight different dimensions of student experiences.
- Deep learning and critical thinking are highly associated with one another. This highlights that students connect learning and critical skills.
At the other end, Effort accounts for all of the weekest correlations among substantive measures. Its r values are between 0.2 and 0.3 for the following items: Goals, Student Ideas, Fair Evaluation, Timely Feedback and Availability. Effort’s correlations do not exceed r = 0.4 with any other item which singles it out almost in a category of its own.
The exploratory analysis continues with cluster analysis to create potential groupings of items to help determine the number of factors for the factor analysis section and the clusters that can serve as a foundation for putting together of composite measures in the section after that. The sections that follow focus on the 23 substantive items.
Cluster Analysis
The purpose of exploratory hierarchical cluster analysis is to establish how close or far apart individual variables are. The routine produces a dendogram where distances between items are reflected in the length of the connecting lines. Short connections indicate items that are fairly close together and may measure similar things, while longer lines indicate items that are farther apart and may measure different things. Chauvent, Kuentz, Liquet, and Saracco’s (2015) ClustOfVar R package is used here. The same authors (2012) also provide a statistical description of the package and its benefits.
The two dendograms are based on the same procedure, but visualize the analysis in two different ways. The first dendogram is a standard representation of item groupings, where lines can be “cut” at any point vertically thus forming individual clusters of connected items. The second dendogram is visually more intuitive, because it “unroots” the standard dendogram and shows the groupings in a more direct way. Both dendograms confirm the observations from the correlation analysis. Namely, that Course and Instructor are closest to one another and Techniques, and that Effort is in a category of its own. For the purposes of exploring relationships between the variables, two types of groupings are considered:
Two large clusters (lines in Figure 1 indicate distances between items; color coding for two-cluster membership):
a) The first one includes: ExpressIdeas, ThinkIndCreat, Skills, ConsiderPerspAppr, ThinkCollab, Value, NewInterst, ThinkCrit, DeepUndSubj, Effort. b) The second one includes: Challenge, Goals, ProjAssn, CrsTime, CrsMaterials, Course, Instructor, Techniques, HelpfulSuggest, TimelyFeedback, Available, FairEval, and StuIdeas.
These clusters suggest that the substantive items can be divided in two groups based on their learning value. The first grouping includes mostly items that relate to the logistical side of things and the formal interactions between instructor and students. The second grouping, on the other hand, includes items that deal with critical thinking, deep learning, and interactions with peers.
Four medium clusters plus Effort (line in Figure 2 indicate distances between items; color coding for five-cluster membership):
a) Cluster 1: DeepUndSubj, ThinkCrit, NewInterest, Value. b) Cluster 2: ThinkCollab, ConsiderPerspAppr, Skills, ThinkIndCreat, ExpressIdeas. c) Cluster 3: StuIdeas, FairEval, Available, TimelyFeedback, HelpfulSuggest. d) Cluster 4: Techniques, Instructor, Course, CrsMaterials, CrsTime, ProjAssn, Goals, Challenge. e) Cluster 5: Effort
Conceptually, these four clusters can be seen as Critical Thinking (Cluster 1), Interacting across difference (Cluster 2), Instructor-student interactions (Cluster 3), and Evaluation of Instruction (Cluster 4). The four-cluster grouping points to the following issues and tentative solutions:
Only two clusters are represented in the summary ratings and one of these clusters includes two summary ratings. That means that focusing only on Course, Istructor, and Value, is likely to misrepresent the data. This is particularly problematic because “students are not qualified to evaluate faculty,” according to IDEA, a leading scientific evaluation and development non-profit organization. On the other hand, systematically selecting several items from each cluster to represent a rating for that cluster can produce four summary measures that would represent the data in a balanced way
None of the clusters includes Effort. It may be useful to explore why that is the case. The efforts a students put in their learning matter a lot in how much and how deeply they learn. Moreover, students are uniquely qualified to reflect on that since they know first-hand what motivates them.
The report proceeds with the selection of items from each cluster using statistical factor analysis.
Factor Analysis
The purpose of exploratory factor analysis is to establish associations between items and underlying factors tha can justify including several items in one composite measure. The principal component analysis (PCA) suggests that one single factor accounts for most of the variance (see Figure 3). That is not surprising given the statistically significant correlations betweeen all substantive items. PCA treats individual factors as independent (i.e. not correlated) and holds little exploratory value here.
Still, it is worth pointing out that the dominant factor helps explain only a little over half (51.21%) of the variance in the data (see the table below). Each consecutive factor (component) contributes relatively less, but Components 2 through 5 all help explain over 3% of the variance each.
## Importance of components:
## PC1 PC2 PC3 PC4 PC5 PC6
## Standard deviation 3.4321 1.14215 1.0003 0.9238 0.84362 0.80419
## Proportion of Variance 0.5121 0.05672 0.0435 0.0371 0.03094 0.02812
## Cumulative Proportion 0.5121 0.56887 0.6124 0.6495 0.68041 0.70853
## PC7 PC8 PC9 PC10 PC11 PC12
## Standard deviation 0.77300 0.76258 0.73190 0.72446 0.6883 0.66262
## Proportion of Variance 0.02598 0.02528 0.02329 0.02282 0.0206 0.01909
## Cumulative Proportion 0.73451 0.75980 0.78309 0.80591 0.8265 0.84559
## PC13 PC14 PC15 PC16 PC17 PC18
## Standard deviation 0.65986 0.62836 0.62176 0.6141 0.5932 0.55872
## Proportion of Variance 0.01893 0.01717 0.01681 0.0164 0.0153 0.01357
## Cumulative Proportion 0.86452 0.88169 0.89850 0.9149 0.9302 0.94377
## PC19 PC20 PC21 PC22 PC23
## Standard deviation 0.55758 0.52917 0.52230 0.50323 0.42003
## Proportion of Variance 0.01352 0.01218 0.01186 0.01101 0.00767
## Cumulative Proportion 0.95728 0.96946 0.98132 0.99233 1.00000If the underlying factors are correlated, factor analysis that does not assume independent factors may offer a solution closer to the data. Figure 4 summarizes the results from the oblique factor rotation Promax within the Psych R package. Note that Figure 4 below does not include items that contribute to multiple factors. In that sense, it the factor analysis solution is more compact than the cluster analysis solution.
Conlcusion And Next Steps
This report demonstrated the need for assessing course evaluation data using systematic scientific means. While the internal validity of the survey instrument is confirmed by the statistically significant correlations among all substantive items, two strategies are laid out for a balanced assessment of course evaluation data: 1. Expansive strategy: Advance the cluster analysis results to put together composite measures that can be used to represent groups of items from the survey. Four composite measures would be used while Effort is explored further through additional surveys, focus groups or interviews. 2. Reductive strategy: Advance the factor analysis results to create composite measures for each of four related factors. The correlations between the factors can be used to assign weights to each factor and create one meta-score.
Appendix 1: List of Course Evaluation Survey Variables
ID Variables
SecSyn - A unique number that corresponds to a course section and instructor.
ReqElec - Required (1) or Elective (2)
Cohort - First-year (1), Sophomore (2), Junior (3), Senior (4)
Survey questions measured in terms of Strongly Disagree (1), Disagree (2), Neutral (3), Agree (4), Strongly Agree (5):
DeepUndSubj - Gained a deeper understanding of the subject
ThinkCrit - Gained ability to think critically about course subject matter
NewInterest - Gained a new or increased interest in this subject
ConsiderPerspAppr - Improved ability to consider varying perspectives or approaches
Skills - Improved ability to apply skills required for the course
ThinkIndCreat - Improved ability to think independently and creatively
ThinkCollab - Improved ability to think collaboratively
ExpressIdeas - Improved ability to express my ideas effectively
Goals - Course goals and requirements were clearly explained
Challenge - The course was appropriately challenging
CrsMaterials - Course materials were effective
CrsTime - Class time was organized and used effectively
ProjAssn - Projects and assignments contributed significantly to learning
StuIdeas - Students’ ideas and contributions were encouraged
FairEval - My work was evaluated fairly
TimelyFeedback - The instructor gave me timely feedback on my work
HelpfulSuggest - The instructor made helpful suggestions for improvement
Available - The instructor was available during office hours and for appointments
Techniques - The teaching techniques were effective in helping me learn
Effort - Overall, I put considerable effort into this course
Value - Overall, this course was valuable to my academic/personal growth
Legacy summary measures (Instructor and Course):
Instructor - Overall, this instructor’s teaching was: Poor (1), Fair (2), Good (3), Very Good (4), Excellent (5)
Course - Overall, this instructor’s teaching was: Poor (1), Fair (2), Good (3), Very Good (4), Excellent (5)
Term
- Term - Fall (1), Winter (2)
Appendix 2: Correlations
Pearson correlations assume a normal distribution, linearity, and homoscedasticity. The coefficients are included below with notations for levels of statistical significance: *** p < .001, ** p < .01, * p < .05.corstarsl(evals.1516)## SecSyn ReqElec Cohort DeepUndSubj ThinkCrit
## SecSyn
## ReqElec 0.02
## Cohort 0.01 0.12***
## DeepUndSubj -0.02 0.01 0.04**
## ThinkCrit -0.01 -0.01 0.05*** 0.73***
## NewInterest 0.01 0.01 0.07*** 0.60*** 0.62***
## ConsidPerspAppr -0.01 0.03* 0.05*** 0.57*** 0.63***
## Skills -0.01 -0.01 0.01 0.61*** 0.63***
## ThinkIndCreat -0.01 0.00 0.04** 0.55*** 0.62***
## ThinkCollab 0.06*** -0.05*** 0.03* 0.43*** 0.48***
## ExpressIdeas 0.01 0.00 0.04** 0.53*** 0.59***
## Goals -0.04*** 0.00 0.00 0.45*** 0.45***
## Challenge -0.05*** -0.02 0.01 0.48*** 0.49***
## CrsMaterials -0.03* 0.02 0.03* 0.50*** 0.51***
## CrsTime -0.03* 0.00 -0.02 0.49*** 0.48***
## ProjAssn -0.02 0.00 0.03* 0.53*** 0.53***
## StuIdeas -0.03* 0.02 0.05*** 0.48*** 0.49***
## FairEval -0.03** 0.01 0.05*** 0.48*** 0.48***
## TimelyFeedback -0.02 0.01 0.02 0.41*** 0.42***
## HelpfulSuggest -0.04** 0.01 0.01 0.47*** 0.49***
## Available -0.03* -0.02 0.03* 0.44*** 0.45***
## Techniques -0.02 0.01 0.01 0.57*** 0.57***
## Effort -0.01 -0.08*** 0.00 0.39*** 0.39***
## Value -0.03* -0.03* 0.01 0.61*** 0.59***
## Instructor -0.04*** 0.02 0.02 0.57*** 0.58***
## Course -0.02 0.01 0.03* 0.61*** 0.61***
## Term 0.02 0.08*** 0.04** -0.01 -0.01
## NewInterest ConsidPerspAppr Skills ThinkIndCreat
## SecSyn
## ReqElec
## Cohort
## DeepUndSubj
## ThinkCrit
## NewInterest
## ConsidPerspAppr 0.54***
## Skills 0.52*** 0.61***
## ThinkIndCreat 0.57*** 0.64*** 0.65***
## ThinkCollab 0.44*** 0.49*** 0.52*** 0.60***
## ExpressIdeas 0.54*** 0.58*** 0.62*** 0.68***
## Goals 0.36*** 0.40*** 0.47*** 0.41***
## Challenge 0.44*** 0.44*** 0.49*** 0.46***
## CrsMaterials 0.48*** 0.47*** 0.49*** 0.49***
## CrsTime 0.44*** 0.46*** 0.50*** 0.47***
## ProjAssn 0.51*** 0.51*** 0.56*** 0.55***
## StuIdeas 0.42*** 0.49*** 0.48*** 0.52***
## FairEval 0.45*** 0.45*** 0.49*** 0.48***
## TimelyFeedback 0.35*** 0.38*** 0.44*** 0.42***
## HelpfulSuggest 0.44*** 0.49*** 0.50*** 0.51***
## Available 0.37*** 0.44*** 0.46*** 0.43***
## Techniques 0.57*** 0.53*** 0.57*** 0.57***
## Effort 0.33*** 0.34*** 0.38*** 0.35***
## Value 0.62*** 0.53*** 0.57*** 0.54***
## Instructor 0.55*** 0.51*** 0.55*** 0.54***
## Course 0.66*** 0.54*** 0.59*** 0.58***
## Term 0.00 -0.01 -0.01 0.00
## ThinkCollab ExpressIdeas Goals Challenge CrsMaterials
## SecSyn
## ReqElec
## Cohort
## DeepUndSubj
## ThinkCrit
## NewInterest
## ConsidPerspAppr
## Skills
## ThinkIndCreat
## ThinkCollab
## ExpressIdeas 0.62***
## Goals 0.37*** 0.45***
## Challenge 0.39*** 0.46*** 0.50***
## CrsMaterials 0.39*** 0.47*** 0.49*** 0.53***
## CrsTime 0.41*** 0.47*** 0.54*** 0.52*** 0.58***
## ProjAssn 0.46*** 0.54*** 0.49*** 0.52*** 0.54***
## StuIdeas 0.43*** 0.52*** 0.46*** 0.44*** 0.47***
## FairEval 0.38*** 0.49*** 0.56*** 0.51*** 0.50***
## TimelyFeedback 0.35*** 0.42*** 0.48*** 0.40*** 0.44***
## HelpfulSuggest 0.40*** 0.53*** 0.51*** 0.46*** 0.48***
## Available 0.38*** 0.44*** 0.49*** 0.43*** 0.44***
## Techniques 0.47*** 0.57*** 0.52*** 0.54*** 0.60***
## Effort 0.30*** 0.35*** 0.28*** 0.35*** 0.31***
## Value 0.43*** 0.54*** 0.44*** 0.52*** 0.51***
## Instructor 0.43*** 0.55*** 0.54*** 0.53*** 0.56***
## Course 0.48*** 0.59*** 0.51*** 0.57*** 0.58***
## Term 0.02 0.00 0.00 0.00 0.00
## CrsTime ProjAssn StuIdeas FairEval TimelyFeedback
## SecSyn
## ReqElec
## Cohort
## DeepUndSubj
## ThinkCrit
## NewInterest
## ConsidPerspAppr
## Skills
## ThinkIndCreat
## ThinkCollab
## ExpressIdeas
## Goals
## Challenge
## CrsMaterials
## CrsTime
## ProjAssn 0.60***
## StuIdeas 0.49*** 0.55***
## FairEval 0.52*** 0.55*** 0.59***
## TimelyFeedback 0.50*** 0.47*** 0.44*** 0.56***
## HelpfulSuggest 0.52*** 0.55*** 0.53*** 0.59*** 0.61***
## Available 0.48*** 0.47*** 0.50*** 0.53*** 0.49***
## Techniques 0.66*** 0.63*** 0.53*** 0.56*** 0.49***
## Effort 0.33*** 0.35*** 0.26*** 0.26*** 0.25***
## Value 0.50*** 0.56*** 0.47*** 0.50*** 0.42***
## Instructor 0.64*** 0.58*** 0.54*** 0.59*** 0.52***
## Course 0.60*** 0.62*** 0.53*** 0.57*** 0.48***
## Term -0.03* 0.00 0.01 0.00 0.02
## HelpfulSuggest Available Techniques Effort Value
## SecSyn
## ReqElec
## Cohort
## DeepUndSubj
## ThinkCrit
## NewInterest
## ConsidPerspAppr
## Skills
## ThinkIndCreat
## ThinkCollab
## ExpressIdeas
## Goals
## Challenge
## CrsMaterials
## CrsTime
## ProjAssn
## StuIdeas
## FairEval
## TimelyFeedback
## HelpfulSuggest
## Available 0.60***
## Techniques 0.59*** 0.51***
## Effort 0.31*** 0.28*** 0.34***
## Value 0.51*** 0.43*** 0.61*** 0.50***
## Instructor 0.60*** 0.48*** 0.72*** 0.35*** 0.63***
## Course 0.57*** 0.47*** 0.70*** 0.39*** 0.71***
## Term -0.02 0.01 -0.02 0.02 -0.01
## Instructor Course
## SecSyn
## ReqElec
## Cohort
## DeepUndSubj
## ThinkCrit
## NewInterest
## ConsidPerspAppr
## Skills
## ThinkIndCreat
## ThinkCollab
## ExpressIdeas
## Goals
## Challenge
## CrsMaterials
## CrsTime
## ProjAssn
## StuIdeas
## FairEval
## TimelyFeedback
## HelpfulSuggest
## Available
## Techniques
## Effort
## Value
## Instructor
## Course 0.80***
## Term 0.00 -0.01