Executive Summary

This report explores course evaluation data from the first two terms of the academic 2015-2016. The analysis below starts with the tidy dataset and applies correlation, cluster, and factor nalysis consecutively. The main findings are that: - the survey instrument is generally solid; - focusing on individual items for assessment purposes may misrepresent the data; - the item Effort stands needs to be investigated further, as it is generally weakly associated with the other substantive items. The results from the exploratory analysis presented here can be used to rebalance the survey instrument and to create working assessment indices based on the data. The report concludes with a recommendation for composite measures to replace the currently used summary measures “Instructor Rating” and “Course Rating”. The recommendation results from the need for a more complex and evidence-based assessment of course evaluation data.

Overview and Preprocessing

The raw data are restructured and transformed into a single csv data file with cases as rows and variables as columns. The raw data are in Excel format where each sheet corresponds to a course/instructor id (SecSyn). In order to create a functional dataset, the two Excel files (one from Fall 2015-2016 and one from Winter 2015-2016) are converted to cvs files, each corresponding to a sheet from the original file. The conversion of the data from each term is done using an html application created by Chris West. After that, due to inconsistencies in the Winter 2016 data, the variable SecSyn is populated using the filenames of the individual csv files, which correspond to the orignal tab names, i.e. course/instructor ids. The resulting dataset contains 27 variables and 6,106 observations. The variables are as follows and value labels are listed in Appendix 1.

## Warning: package 'xtable' was built under R version 3.2.3

Correlation Analysis

The correlation analysis shows that all bivariate correlations between substantive items are statistically significant. The strongest correlation is between the two legacy summary measures (Course and Instructor) with r = 0.8, the only coefficient equal or above 0.8 (See Appendix 2). There are four bivariate correlations with coefficients between 0.7 and 0.8, three of which are between Techniques and the summary measures (Course, Instructor and Value). The fourth correlation with a exceeding 0.7 but below 0.8 is between Thinking Critically and Deep Understanding of Subject, r = 0.73. Therefore, three preliminary conclusions can be drawn:

  1. Statistically, the instrument is internally valid. That makes it a reliable tool for surveying students.
  2. The summary measures are somewhat disconnected from the other substantive measures, except Techniques. That justifies the development of composite substantive measures that highlight different dimensions of student experiences.
  3. Deep learning and critical thinking are highly associated with one another. This highlights that students connect learning and critical skills.

At the other end, Effort accounts for all of the weekest correlations among substantive measures. Its r values are between 0.2 and 0.3 for the following items: Goals, Student Ideas, Fair Evaluation, Timely Feedback and Availability. Effort’s correlations do not exceed r = 0.4 with any other item which singles it out almost in a category of its own.

The exploratory analysis continues with cluster analysis to create potential groupings of items to help determine the number of factors for the factor analysis section and the clusters that can serve as a foundation for putting together of composite measures in the section after that. The sections that follow focus on the 23 substantive items.

Cluster Analysis

The purpose of exploratory hierarchical cluster analysis is to establish how close or far apart individual variables are. The routine produces a dendogram where distances between items are reflected in the length of the connecting lines. Short connections indicate items that are fairly close together and may measure similar things, while longer lines indicate items that are farther apart and may measure different things. Chauvent, Kuentz, Liquet, and Saracco’s (2015) ClustOfVar R package is used here. The same authors (2012) also provide a statistical description of the package and its benefits.

The two dendograms are based on the same procedure, but visualize the analysis in two different ways. The first dendogram is a standard representation of item groupings, where lines can be “cut” at any point vertically thus forming individual clusters of connected items. The second dendogram is visually more intuitive, because it “unroots” the standard dendogram and shows the groupings in a more direct way. Both dendograms confirm the observations from the correlation analysis. Namely, that Course and Instructor are closest to one another and Techniques, and that Effort is in a category of its own. For the purposes of exploring relationships between the variables, two types of groupings are considered:

  1. Two large clusters (lines in Figure 1 indicate distances between items; color coding for two-cluster membership):

    a) The first one includes: ExpressIdeas, ThinkIndCreat, Skills, ConsiderPerspAppr, ThinkCollab, Value, NewInterst, ThinkCrit, DeepUndSubj, Effort.

b) The second one includes: Challenge, Goals, ProjAssn, CrsTime, CrsMaterials, Course, Instructor, Techniques, HelpfulSuggest, TimelyFeedback, Available, FairEval, and StuIdeas.

These clusters suggest that the substantive items can be divided in two groups based on their learning value. The first grouping includes mostly items that relate to the logistical side of things and the formal interactions between instructor and students. The second grouping, on the other hand, includes items that deal with critical thinking, deep learning, and interactions with peers.

  1. Four medium clusters plus Effort (line in Figure 2 indicate distances between items; color coding for five-cluster membership):

    a) Cluster 1: DeepUndSubj, ThinkCrit, NewInterest, Value.

b) Cluster 2: ThinkCollab, ConsiderPerspAppr, Skills, ThinkIndCreat, ExpressIdeas.

c) Cluster 3: StuIdeas, FairEval, Available, TimelyFeedback, HelpfulSuggest.

d) Cluster 4: Techniques, Instructor, Course, CrsMaterials, CrsTime, ProjAssn, Goals, Challenge.

e) Cluster 5: Effort