Motivation

We have a client today who is interested in examining student evaluation scores of professors in college courses. The use of these student evaluations as an indicator of course quality and teaching effectiveness is criticized for a variety of reasons, including the concern that these measures may reflect the influence of non-teaching related characteristics, such as the physical appearance of the instructor.

One article that explores this concern is titled “Beauty in the classroom: instructors’ pulchritude and putative pedagogical productivity” (Hamermesh and Parker, 2005). The article concluded that instructors who are viewed to be better looking receive higher instructional ratings.

In this lab we will analyze the data from this study for ourselves. I put the article link down below just for reference, but I will provide a link to the data itself in the next section of the lab.

• Daniel S. Hamermesh, Amy Parker, Beauty in the classroom: instructors’ pulchritude and putative pedagogical productivity, Economics of Education Review, Volume 24, Issue 4, August 2005, Pages 369-376, ISSN 0272-7757, 10.1016/j.econedurev.2004.07.013.[http://www.sciencedirect.com/science/article/pii/S0272775704001165]

The Data

The data for this study were gathered from end of semester student evaluations for a large sample of professors from the University of Texas at Austin. A beauty score was created for each of the sampled professors by having six students rate the physical appearance of each professor. The beauty score in the data for each professor is the average of the six student scores.

The data set we will be working with is a slightly modified version of the original data set that was released as part of the replication data for Data Analysis Using Regression and Multilevel/Hierarchical Models (Gelman and Hill, 2007). Our data set is a data frame where each row contains a different course and columns represent variables about the courses and professors.

To load the data, copy and paste the following line of code into a code chunk and press play:

load(url("http://www.openintro.org/stat/data/evals.RData"))

Model 1: One Numeric X

The fundamental phenomenon suggested by the study is that better looking teachers are evaluated more favorably. Our client asks us to use our data set to see if we find the same result. In other words, they ask us to build a model to explore the relationship between the beauty score (X = bty_avg) and the evaluation score (Y = score) of faculty being evaluated.

We know that the first step in any model building is EDA, so we will being by creating a plot to explore this relationship.

Now that we have plotted the data, our client asks us to use a regression model to explore the relationship. The goal of building the model is to allow us to determine if what we saw in the plot is something more than natural variation.

At this point, we have a model that we can use to describe the relationships in our data set. However, we want to be able to assess whether these data provide strong evidence of a population relationship between beauty and evaluation score. For this, we need inference procedures. You may assume all assumptions needed for inference are reasonable.

There are two things we consider when examining a confidence interval. The first is statistical significance, which for this scenario means checking to see if 0 is included in the CI. If it is, then 0 is a plausible value for \(\beta_1\), suggesting there may be no population relationship between X and Y.

The second thing we check for is practical significance. This means checking to see whether or not the change suggested by the interval is large enough to make a real world impact in the context of the data.

For instance, if we are 95% confident that for every hour you study, your final exam grade on average increases between .005 and .007 points out of 100, does that motivate you to study?? Probably not, as you would need to study more than 142 hours to gain 1% on your final. The interval does not contain 0, so it is statistically significant, but it is not practically significant as the change suggested by either end point of the CI is not large enough to make a real world impact on your exam score.

One other tool we use to assess the strength of the relationship between X and Y is the \(R^2\).

Model 2: One Categorical X

Our client now asks us to build a second model using the evaluation data. They are interested in the relationship between X = the class level of the course and Y = the evaluation score. Class level (cls_level) is a recorded as “lower” if the course is a 100 level or 200 level course and “upper” if the course is a 300 level course.

Why is this an interesting variable to look at? Well, class level can be used as a way to reflect the difficulty level of the course. For instance, it may be reasonable to assume that upper level classes may be more difficult than lower level classes.

To build a regression model for Y = evaluation score and X = class level, we can use the following:

model_Level <- lm(score~ cls_level, data = evals)
summary(model_Level)

If you run summary(model_ClassLevel), you will notice that the coefficient for class level is now called cls_levelupper. The reason is that R re-codes class level from having the values of upper and lower to being an indicator variable called cls_levelupper that takes a value of 0 for lower level classes and a value of 1 for upper level classes.

Okay, so we can see how this model works now. However, how did we decide that we were going to indicate the upper level rather than the lower level classes?? It turns out that the decision to call the indicator variable cls_levelupper instead of cls_levellower has no deeper meaning. R simply picks on level to be 0 and one level to be 1.

Typically, R makes the choice alphabetically, but for any variable, you can use the summary command to check. Whatever value shows up first in the summary table is the baseline, i.e., is coded as 0. So, the indicator variables indicates when the value of the variable is different from the baseline. For class level, the baseline is lower. The indicator variable asks “Is this class an upper level class?” If so, we record a 1, if not a 0. The 1 indicates a value of upper level class, i.e., a value different from the baseline.

You can change the baseline level of a categorical variable (the level that is coded as a 0) using the relevel function. Use ?relevel or ask me to learn more.

Model 3: Putting Them Together

We have built a model with one numeric X and a model with one categorical. However, there is no reason why we have to pick only one! We can build multiple linear regression (MR) models which include multiple X variables.

Why might we want to include multiple X variables? One reason is that that putting two or more X variables together in the model allows us to build models that account for more sources of variation. This can allow us to build models that are stronger fits to the data, explain more variability in Y, and provide more accurate interpretations of the relationships between our X variables and Y.

Our client believes that in addition to beauty score, the variance in evaluation score might be related to whether or not a faculty member is tenure track, tenured, or not tenure track. This information is recorded in the variable rank in the data set. Based on this, let’s try building a model for \(Y\) = evaluation score using \(X_1\) = beauty score and \(X_2\) = professor rank.

Instead of one indicator variable, you will notice that this model uses two, and we can see what these are from the R summary. We have one variable called ranktenuretrack. This variable asks “Is this professor on the tenure track?”, and recording a 1 one means yes. The second variable in the summary output asks “Is this professor tenured?”, where again a 1 means yes and 0 means no.

What is left? Well, a teaching track professor is neither tenured nor tenure track, so such professors will have zeroes for both indicator variables. This means that teaching professors are the baseline we compare to, and the average scores for teaching professors (with beauty score 0) is the intercept of the model.

Now that we have the model built, let’s interpret it. The interpretation of the coefficients in multiple regression is slightly different from that of LSLR. The estimate for beauty average reflects how much higher a group of professors is expected to score if they have a beauty rating that is one point higher keeping all other variables fixed. In this case, that translates into considering only professors of the same rank with beauty average scores that are one point apart.

When we write our interpretations, we reflect this with the phrase ``After controlling for rank”. This indicates that we are interpreting the coefficient as: For professors of the same rank, what happens when we increase the beauty score by 1 point?

Model 4:

Now that we can build multiple regression models, we need to carefully consider the idea of controlling for sources of variation when we build a model. If we know or suspect that something might be related to Y, it is important to build that information into the model. This allows us to build models that are stronger reflections of the patterns in the data, as well as providing a more accurate understanding of the relationships we are interested in.

Our client suspects that professor rank, ethnicity, sex (which is recorded under the name gender in the data set), age, class level, and the number of students in the class should be controlled for before assessing the relationship between beauty and evaluation score. In other words, our client asks us to build a model that contains all of these variables, including beauty average.

When we evaluate multiple regression models, we use something called the adjusted \(R^2\) to reflect the amount of variability in \(Y\) that is captured by the model. We know that \[R^2 = 1 - \frac{RSS}{TSS}\]. The adjusted \(R^2\) is \[R^2_{adj} = 1 - \left( \frac{RSS}{TSS} \times \frac{n-1}{n-p} \right),\] where \(p\) is the number of \(\beta\) terms in the model including the intercept.

Next Steps

As we move forward in the course, we will start to learn how we can use this idea of using multiple explanatory variables to build stronger models. This means we want to build models that are a better reflection of the variability in \(Y\) and the relationships between \(Y\) and other variables. We also need to learn how we can compare models, and how we might decide whether including more explanatory variables in the model is effective. All of that is coming up!