What do we already know?

• test association between two categorical variables (chi-square)
• find relationships between categorical and continuous variables (ANOVA, t-test)
• test relationships between two continuous variables (correlations)

Regression

Definition: Linear regression is a statistical technique where a continuous outcome is regressed on predictors, assuming the relationships among them are linear.

Depending on the outcome type and the ‘link function’ (the shape of relationship), regressions can have various names.

Compare: https://stats.idre.ucla.edu/other/dae/

This course will speak about linear regression only.

Regression and other methods that we have covered

Regression modelling

• describes relationships between variables with the following formula: \[y_i = a + b*x_i + e_i\]
  • a - intercept. This is the value of Y when X = 0.
  • b - slope. The is a regression coefficient showing the change in Y when X increases by 1.

image source: http://www.math.com/school/subject2/lessons/S2U4L2GL.html#sm1

Ordinary least squares (OLS)

Statistical inference in regression

Linear Regression serves to predict a continuous (metric) dependent variable (= ‘the outcome’)

Predictor variables can be both categorical and continuous

Multiple linear regression can absorb several predictors

The results generalise to the population, if the sample is representative.

The relationship between the Y (outcome) and Xs (predictors) is described with an equation of linear regression: y = ax + b

Why do we need regression?

• it is an umbrella method for all of the above
• more than one predictor variable is allowed
• relationship can be expressed as a number (e.g., every extra year of work experience increases the expected salary by £50)

Omitted variable effect (1)

Omitted variable effect (2)

Toy data example

There are variables x, x1 and y, n = 50, normally distributed

##        y               x               x1      
##  Min.   :110.9   Min.   :3.002   Min.   : 1.0  
##  1st Qu.:117.7   1st Qu.:4.365   1st Qu.: 3.0  
##  Median :120.0   Median :4.988   Median : 5.5  
##  Mean   :120.6   Mean   :5.061   Mean   : 5.5  
##  3rd Qu.:123.5   3rd Qu.:5.567   3rd Qu.: 8.0  
##  Max.   :138.1   Max.   :9.026   Max.   :10.0

Visualise their relationship

plot(x, y, pch = 16)

Use correlation to estimate the strength of their relationship

cor.test(x, y, method = "pearson")

## 
##  Pearson's product-moment correlation
## 
## data:  x and y
## t = 8.7521, df = 48, p-value = 1.647e-11
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  0.6469066 0.8720898
## sample estimates:
##       cor 
## 0.7840706

Estimate the model

model1 <- lm(y ~ x, data = data)
summary(model1)

## 
## Call:
## lm(formula = y ~ x, data = data)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -4.8575 -2.4754  0.0283  2.2571  5.2295 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 103.2367     2.0289  50.882  < 2e-16 ***
## x             3.4357     0.3926   8.752 1.65e-11 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.91 on 48 degrees of freedom
## Multiple R-squared:  0.6148, Adjusted R-squared:  0.6067 
## F-statistic:  76.6 on 1 and 48 DF,  p-value: 1.647e-11

What are the conclusions?

• The value of Y when X = 0, e.g. expected salary when work experience is 0 years.
• How much will Y change when X increases by 1?

Plot

Any questions so far?

Residuals

• predicted value of y is \(\hat{y}\)
• residuals are the differences between predicted and observed values of y

Coefficient of determination R²

• shows the relation of variance explained by the model to total variance of the outcome variable \[R^2 = \frac{SS_{reg}}{SS_{tot}}\]
• varies from 0 to 1 (or between 0-100 per cent)
• R² above 0.1 demonstrates a satisfactory predictive power (but that is not a hard fact)
• as the number of predictors grows, R² grows as well. Therefore,
• adjusted R²: \[R^2_{adj} = \frac{SS_{reg} / (n – k))}{(SS_{tot} / (n – 1))}\]
• where n is the sample size, and k is the number of predictors

Several predictors

• The main advantage of regression modelling is that it can digest several predictors in one model

Several predictors

Let’s add another predictor to the model

model2 <- lm(y ~ x + x1, data = data)
summary(model2)

## 
## Call:
## lm(formula = y ~ x + x1, data = data)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -1.80025 -0.56990  0.02968  0.54171  2.30021 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 99.60458    0.63377  157.16   <2e-16 ***
## x            3.06165    0.11958   25.60   <2e-16 ***
## x1           1.00462    0.04581   21.93   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.8774 on 47 degrees of freedom
## Multiple R-squared:  0.9657, Adjusted R-squared:  0.9642 
## F-statistic: 661.8 on 2 and 47 DF,  p-value: < 2.2e-16

sjPlot::tab_model(model1, model2)

# run the following code in R if you'd like to see how a regression with 2 numeric predictors looks like
#library(car)
#scatter3d(x = x, y = y, z = x1)

How many predictors can we add to one regression model?

• Depends on the theoretical model and the dataset
• Rule of thumb: one predictor per 20 observations (e.g., no more than 5 for 100 observations)
• Avoid garbage in, garbage out models (too many predictors with noise, unclear relationships)
• Formulas exist (Tabachnick & Fidell): 108 + k for the overall model fit; 50 + 8*k for individual predictor coefficients.

Questions?

What should I do if I’d like to add categorical predictor?

Dummy-variables

• dichotomised

• show the differences in intercepts between groups (there is a ‘reference group’)

• if class(var) = factor in R, it is recoded into dummies by default

Example: salary of males and females

200 males and females

##       age            salary       sex    
##  Min.   :15.00   Min.   : 46.52   F:100  
##  1st Qu.:26.00   1st Qu.: 83.59   M:100  
##  Median :30.00   Median : 96.21          
##  Mean   :30.01   Mean   : 94.96          
##  3rd Qu.:33.25   3rd Qu.:105.22          
##  Max.   :46.00   Max.   :138.11

Example: exploratory plots

Example: the model

model2 <- lm(salary ~ age + sex, data = genderdata)

tab_model(model2, show.ci = F)

Example: plot

What if there are more than 2 categories? (e.g. educational attainment)

• create n-1 variables where n is the original number of categories
• the first category is now the ‘reference’
• What does the coding look like?

##        educ dummy1 dummy2
## 1  tertiary      0      0
## 2 secondary      1      0
## 3   primary      0      1

Any questions on dummy variables?

Another cool example

Artwork by @allison_horst

Linear Regression Assumptions

• linearity
• normally distributed residuals of the outcome
• independent observations (no correlation between their residuals)
• independent predictors (no multicollinearity)
• no outliers
• homoskedasticity (residual variances are the same along the values of Y)

Homoskedasticity/Heteroskedasticity

x <- 1:100
y1 <- rnorm(n = 100, mean = x, sd = 10)
y2 <- rnorm(n = 100, mean = x, sd = 0.4*x)
par(mfrow = c(1, 2))
plot(x, y1, pch = 16); abline(lm(y1 ~ x), col = "red")
plot(x, y2, pch = 16); abline(lm(y2 ~ x), col = "red")

Pane on the right-hand side with data points ‘fanning out’ shows there is a third variable which comes into play at high values of X

The general idea of control variables

In experiments we can randomize and control conditions.

But it is not true for observational studies. -> Therefore we need to control for a set of variables (usually socio-demographics) in order to take those uncontrolled differences into account and be able to tell whether the predictor of interest is indeed related to the outcome.

Example: happiness and Internet use

Why do we need control variables? Example

Source: Blavatskyy, P. (2021). Obesity of politicians and corruption in post‐Soviet countries. Economics of Transition and Institutional Change, 29(2), 343-356.

How to report regression modelling?

Tables

tab_model(model2, show.ci = F)

How to report regression modelling?

Tables: Examples

Source: Valenzuela, S., Park, N., & Kee, K. F. (2009). Is there social capital in a social network site?: Facebook use and college students’ life satisfaction, trust, and participation. Journal of computer-mediated communication, 14(4), 875-901.

How to report regression modelling?

Tables: Examples

Source: Goidel, K., Gaddie, K., & Ehrl, M. (2017). Watching the news and support for democracy: Why media systems matter. Social Science Quarterly, 98(3), 836-855.

How to report regression modelling?

Tables: Examples

Baker, L. A., Cahalin, L. P., Gerst, K., & Burr, J. A. (2005). Productive activities and subjective well-being among older adults: The influence of number of activities and time commitment. Social Indicators Research, 73(3), 431-458.

What is important when you are reading regression table

• What is dependent variable?
• What type of regression is applied?
• How the predictors are measured?
• Are the coefficients standardized or not?
• What is the difference between the models reported in one table?

How to report regression modelling?

Equations

For example,

Source: Evans, P., & Rauch, J. E. (1999). Bureaucracy and growth: A cross-national analysis of the effects of” Weberian” state structures on economic growth. American sociological review, 748-765.

Summary: nuts and bolts of regression modelling

• Input
  • outcome - 1 pc.
  • predictors - >= 1 pc.
• Output
  • regression coefficients
  • R²

What’s next?

• standardized coefficients
• model selection
• interaction effects
• model diagnostics (next year)
• other tastes and shapes of regression, for example, logistic regression (after that)

Useful links

1. Multiple linear regression in R (Sheffield U): https://www.sheffield.ac.uk/polopoly_fs/1.536483!/file/MASH_multiple_regression_R.pdf

2. Correlation and Regression course (DataCamp): https://learn.datacamp.com/courses/correlation-and-regression

3. Intermediate Regression in R (Chapters 1-3, DataCamp): https://learn.datacamp.com/courses/intermediate-regression-in-r

4. Linear Regression and Modeling (Coursera, Duke U): https://www.coursera.org/learn/linear-regression-model

5. Data Science: Linear Regression (Edx, Harvard U): https://www.edx.org/course/data-science-linear-regression

All models are wrong, but some are useful

Any questions now?

Image source: http://www.ninandrews.com/blog/2018/1/11/where-am-