An Introduction to Classification

class: center, middle, inverse, title-slide

# An Introduction to Classification
### Dr Craig Alexander
### 16th June 2020

---

# Teaching Experience

<center>
<div id="htmlwidget-eccd169e35043e9bb397" style="width:504px;height:504px;" class="grViz html-widget"></div>
<script type="application/json" data-for="htmlwidget-eccd169e35043e9bb397">{"x":{"diagram":"\n      digraph {\n      layout = neato\n      \n      node [shape = rectangle, fontsize = 24]\n      A [label = \"Experience\", fillcolor = \"#169294\", style = \"filled, solid\", fontcolor = \"white\", fixedsize = TRUE, width = 1.9, height = 1, fontname=\"helvetica\", penwidth = 3.0, color = \"black\"]\n      \n      node [shape = circle, fontsize = 22, fixedsize = TRUE, width = 1.5, fontname=\"helvetica\", penwidth = 3.0]\n      B [label = \"Lecturing\"]\n      C [label = \"Tutoring\"]\n      D [label = \"Other\"]\n\n      node [shape = rectangle, fontsize = 22, width = .4, fixedsize = T, fontname=\"helvetica\", penwidth = 1.0]\n      F [label = \"Sci. Fund. Statistics (1)\"]\n      G [label = \"Flexible Regression (4)\"]\n      G2 [label = \"Professional Skills (3/M)\"]\n      H [label = \"Regression (2)\"]\n      I [label = \"Statistical Methods (2)\"]\n      J [label = \"Data Analysis (3)\"]\n      K [label = \"Adv. Data Analysis (4)\"]\n      L [label = \"Statistics courses (PG)\"]\n      M [label = \"Online Quizzes\"]\n      N [label = \"Marking\"]\n      O [label = \"Course Development\"]\n      P [label = \"Supervision\"]\n      \n      \n      edge[dir = none, len = 2]\n      A -> {B};\n      A -> {C};\n      A -> {D};\n      \n      edge[dir = none, len = 2.5]\n      B -> {F};\n      B -> {G};\n      B -> {G2};\n      C -> {H};\n      C -> {I};\n      C -> {J};\n      C -> {K}[len = 2.7];\n      C -> {L};\n      D -> {M};\n      D -> {N}[len = 2.4];\n      D -> {O};\n      D -> {P}\n      \n      }","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>
</center>

---

# Teaching Experience

<center>
<div id="htmlwidget-0b809859693376e80b5e" style="width:504px;height:504px;" class="grViz html-widget"></div>
<script type="application/json" data-for="htmlwidget-0b809859693376e80b5e">{"x":{"diagram":"\n      digraph {\n      layout = neato\n      \n      node [shape = rectangle, fontsize = 24]\n      A [label = \"Experience\", fillcolor = \"#169294\", style = \"filled, solid\", fontcolor = \"white\", fixedsize = TRUE, width = 1.9, height = 1, fontname=\"helvetica\", penwidth = 3.0, color = \"black\"]\n      \n      node [shape = circle, fontsize = 22, fixedsize = TRUE, width = 1.5, fontname=\"helvetica\", penwidth = 3.0]\n      B [label = \"Lecturing\", fillcolor = \"#169294\", style = filled, fontcolor = \"white\"]\n      C [label = \"Tutoring\"]\n      D [label = \"Other\"]\n\n      node [shape = rectangle, fontsize = 22, width = .4, fixedsize = T, fontname=\"helvetica\", penwidth = 1.0]\n      F [label = \"Sci. Fund. Statistics (1)\", fillcolor = \"#d5bbed\", style = filled]\n      G [label = \"Flexible Regression (4)\", fillcolor = \"#d5bbed\", style = filled]\n      G2 [label = \"Professional Skills (3/M)\", fillcolor = \"#d5bbed\", style = filled]\n      H [label = \"Regression (2)\"]\n      I [label = \"Statistical Methods (2)\"]\n      J [label = \"Data Analysis (3)\"]\n      K [label = \"Adv. Data Analysis (4)\"]\n      L [label = \"Statistics courses (PG)\"]\n      M [label = \"Online Quizzes\"]\n      N [label = \"Marking\"]\n      O [label = \"Course Development\"]\n      P [label = \"Supervision\"]\n      \n      \n      edge[dir = none, len = 2]\n      A -> {B};\n      A -> {C};\n      A -> {D};\n      \n      edge[dir = none, len = 2.5]\n      B -> {F};\n      B -> {G};\n      B -> {G2};\n      C -> {H};\n      C -> {I};\n      C -> {J};\n      C -> {K}[len = 2.7];\n      C -> {L};\n      D -> {M};\n      D -> {N}[len = 2.4];\n      D -> {O};\n      D -> {P}\n      \n      }","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>
</center>

---

# Teaching Experience

<center>
<div id="htmlwidget-1c08ef30b23cd74b2420" style="width:504px;height:504px;" class="grViz html-widget"></div>
<script type="application/json" data-for="htmlwidget-1c08ef30b23cd74b2420">{"x":{"diagram":"\n      digraph {\n      layout = neato\n      \n      node [shape = rectangle, fontsize = 24]\n      A [label = \"Experience\", fillcolor = \"#169294\", style = \"filled, solid\", fontcolor = \"white\", fixedsize = TRUE, width = 1.9, height = 1, fontname=\"helvetica\", penwidth = 3.0, color = \"black\"]\n      \n      node [shape = circle, fontsize = 22, fixedsize = TRUE, width = 1.5, fontname=\"helvetica\", penwidth = 3.0]\n      B [label = \"Lecturing\"]\n      C [label = \"Tutoring\", fillcolor = \"#169294\", style = filled, fontcolor = \"white\"]\n      D [label = \"Other\"]\n\n      node [shape = rectangle, fontsize = 22, width = .4, fixedsize = T, fontname=\"helvetica\", penwidth = 1.0]\n      F [label = \"Sci. Fund. Statistics (1)\"]\n      G [label = \"Flexible Regression (4)\"]\n      G2 [label = \"Professional Skills (3/M)\"]\n      H [label = \"Regression (2)\", fillcolor = \"#d5bbed\", style = filled]\n      I [label = \"Statistical Methods (2)\", fillcolor = \"#d5bbed\", style = filled]\n      J [label = \"Data Analysis (3)\", fillcolor = \"#d5bbed\", style = filled]\n      K [label = \"Adv. Data Analysis (4)\", fillcolor = \"#d5bbed\", style = filled]\n      L [label = \"Statistics courses (PG)\", fillcolor = \"#d5bbed\", style = filled]\n      M [label = \"Online Quizzes\"]\n      N [label = \"Marking\"]\n      O [label = \"Course Development\"]\n      P [label = \"Supervision\"]\n      \n      \n      edge[dir = none, len = 2]\n      A -> {B};\n      A -> {C};\n      A -> {D};\n      \n      edge[dir = none, len = 2.5]\n      B -> {F};\n      B -> {G};\n      B -> {G2};\n      C -> {H};\n      C -> {I};\n      C -> {J};\n      C -> {K}[len=2.7];\n      C -> {L};\n      D -> {M};\n      D -> {N}[len = 2.4];\n      D -> {O};\n      D -> {P}\n      \n      }","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>
</center>

---

# Teaching Experience

<center>
<div id="htmlwidget-2002926674f67e1b3f80" style="width:504px;height:504px;" class="grViz html-widget"></div>
<script type="application/json" data-for="htmlwidget-2002926674f67e1b3f80">{"x":{"diagram":"\n      digraph {\n      layout = neato\n      \n      node [shape = rectangle, fontsize = 24]\n      A [label = \"Experience\", fillcolor = \"#169294\", style = \"filled, solid\", fontcolor = \"white\", fixedsize = TRUE, width = 1.9, height = 1, fontname=\"helvetica\", penwidth = 3.0, color = \"black\"]\n      \n      node [shape = circle, fontsize = 22, fixedsize = TRUE, width = 1.5, fontname=\"helvetica\", penwidth = 3.0]\n      B [label = \"Lecturing\"]\n      C [label = \"Tutoring\"]\n      D [label = \"Other\", fillcolor = \"#169294\", style = filled, fontcolor = \"white\"]\n\n      node [shape = rectangle, fontsize = 22, width = .4, fixedsize = T, fontname=\"helvetica\", penwidth = 1.0]\n      F [label = \"Sci. Fund. Statistics (1)\"]\n      G [label = \"Flexible Regression (4)\"]\n      G2 [label = \"Professional Skills (3/M)\"]\n      H [label = \"Regression (2)\"]\n      I [label = \"Statistical Methods (2)\"]\n      J [label = \"Data Analysis (3)\"]\n      K [label = \"Adv. Data Analysis (4)\"]\n      L [label = \"Statistics courses (PG)\"]\n      M [label = \"Online Quizzes\", fillcolor = \"#d5bbed\", style = filled]\n      N [label = \"Marking\", fillcolor = \"#d5bbed\", style = filled]\n      O [label = \"Course Development\", fillcolor = \"#d5bbed\", style = filled]\n      P [label = \"Supervision\", fillcolor = \"#d5bbed\", style = filled]\n      \n      \n      edge[dir = none, len = 2]\n      A -> {B};\n      A -> {C};\n      A -> {D};\n      \n      edge[dir = none, len = 2.5]\n      B -> {F};\n      B -> {G};\n      B -> {G2};\n      C -> {H};\n      C -> {I};\n      C -> {J};\n      C -> {K}[len=2.7];\n      C -> {L};\n      D -> {M};\n      D -> {N}[len = 2.4];\n      D -> {O};\n      D -> {P}\n      \n      }","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>
</center>

---
 
# Teaching Philosophy
--
<br/>
- Be organised, enthusiastic and caring.

--
<br/>
<br/>

- Encourage understanding of why we are learning and applying statistical methods and techniques.

--
<br/>
<br/>

- Teach through relatable interesting examples and show how statistics relates to real life.

--
<br/>
<br/>

- Where possible use visualisations and interactive tools to aid learning.
  
--
<br/>
<br/>

- Be approachable and encourage asking for help both from myself and other students.

<br/>
<center>
.red[Everybody learns differently, by utilising this philosophy all types of learners will be able to learn statistics to their best potential.]
</center>

---
class: inverse, center, middle

# Multiple Regression: Model Selection

---
# Multiple Regression

- Multiple regression is a natural extension to simple linear regression where we are now building a model with a set of (independent) predictor variables.

--
<br/>
<br/>

- Life expectancy (`Life.Exp`)
  - Murder rate per 100 000 people (`Murder`)
  - Percentage of high school graduates (`HS.Grad`)
  - Income per capita (`Income`)
  - Percentage of the population that are illiterate (`Illiteracy`)

<br/>
- This dataset is already contained within `library(faraway)` in `R`.  <a href="https://glasgow.summon.serialssolutions.com/?s.q=Linear+models+in+R&s.cmd=addFacetValueFilters%28ContentType%2CNewspaper+Article%2Ct%7CContentType%2CBook+Review%2Ct%29#!/search?ho=t&fvf=ContentType,Newspaper%20Article,t%7CContentType,Book%20Review,t&l=en-UK&q=Linear%20models%20with%20R"><b class="fa fa-info-circle"></b>
</a>

--
<br/>
<br/>

- We fitted the multiple regression model with life expectancy as the response and the other variables as predictors.

---
# Multiple Regression - Model selection

.red[$$\widehat{\text{Life.Exp}}_i = 69.48 -0.262\cdot\text{Murder}_i + 0.046\cdot\text{HS.Grad}_i + 0.0001\cdot\text{Income}_i + 0.276\cdot\text{Illiteracy}_i $$]
--

- This is the **full model** which contains all of the predictor variables.

- However, there is potentially a model with less predictors that fits the data almost as well as the full model.

- **Model selection** - finding the simplest model that adequately explains the data.

--
<br/>
<br/>

<center>
 <h3> Why not just use all of the available predictors? </h3>
 </center>

-  More precise estimates and predictions may be achieved with a simpler model.

- A model with less predictor variables might also be desirable because collecting data can be costly.

---
# Model selection - Essentials

**Testing based procedures** - are based on determining the significance of the relationship between each predictor and the response variable i.e.

.red[$$\text{H}_0: \beta_j = 0 \hskip6ex \text{vs.} \hskip6ex \text{H}_1: \beta_j \neq 0$$]

`$j=1,\dotsc,p$` where `$p$` is the number of predictors.

- The p-values and test statistics can be obtained through `lm()` in `R`, they can also be manually calculated.

-  .red[Fail to reject H<sub>0</sub>] - `$j^{th}$` parameter **could be 0**, the corresponding predictor is referred to as an **insignificant predictor** and is generally removed from the model.

-  .red[Reject H<sub>0</sub>] -  `$j^{th}$` parameter is **significantly different from 0**, the corresponding predictor is referred to as an **significant predictors** and is kept in the model.

--
<br/>
<br/>

**Criterion based methods** - seek to find, for all possible models, the model which optimises a chosen information criterion, e.g. AIC or BIC.

---
# Testing based procedures

<h4>Backward Elimination</h4>

1. Start with the **full model** and a chosen significance level `$(\alpha)$`.

2. Amongst the predictors whose p-values are above `$\alpha$`, remove the predictor with the largest p-value.

3. Refit the model without your removed predictor.

4. Iterate steps 2 & 3 until all remaining predictors are **significant predictors**, this is your **final model**.

--
<br/>
<br/>

<h4>Forward Selection </h4>

The reverse of backward elimination where we start with a model with no predictors and iteratively add them.

---
# US Life Expectancy Data

![](MR_files/figure-html/unnamed-chunk-7-1.png)
---
# US Life Expectancy Data - Backward Elimination

**Step 1:**

```r
# Full Model
model1 <- lm(Life.Exp ~ Murder + HS.Grad + Income + Illiteracy, 
             data = state.dat)
```

<table class="table table-striped table-hover" style="margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:right;"> Estimate </th>
   <th style="text-align:right;"> Std. Error </th>
   <th style="text-align:right;"> t value </th>
   <th style="text-align:right;"> Pr(&gt;|t|) </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-weight: bold;"> (Intercept) </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 69.4833 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 1.3253 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 52.4275 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0000 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-weight: bold;"> Murder </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> -0.2619 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0445 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> -5.8908 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0000 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-weight: bold;"> HS.Grad </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0461 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0218 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 2.1120 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0403 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: gray85 !important;font-weight: bold;"> Income </td>
   <td style="text-align:right;color: black !important;background-color: gray85 !important;"> 0.0001 </td>
   <td style="text-align:right;color: black !important;background-color: gray85 !important;"> 0.0002 </td>
   <td style="text-align:right;color: black !important;background-color: gray85 !important;"> 0.5160 </td>
   <td style="text-align:right;color: black !important;background-color: gray85 !important;"> 0.6084 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-weight: bold;"> Illiteracy </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.2761 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.3105 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.8891 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.3787 </td>
  </tr>
</tbody>
</table>

---

**Steps 2 & 3:**

```r
# Model - Income rm
model2 <- lm(Life.Exp ~ Murder + HS.Grad + Illiteracy, 
             data = state.dat)
```

<table class="table table-striped table-hover" style="margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:right;"> Estimate </th>
   <th style="text-align:right;"> Std. Error </th>
   <th style="text-align:right;"> t value </th>
   <th style="text-align:right;"> Pr(&gt;|t|) </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-weight: bold;"> (Intercept) </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 69.7354 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 1.2221 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 57.0628 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0000 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-weight: bold;"> Murder </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> -0.2581 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0435 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> -5.9343 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0000 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: white !important;font-weight: bold;"> HS.Grad </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0518 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0188 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 2.7608 </td>
   <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0083 </td>
  </tr>
  <tr>
   <td style="text-align:left;color: black !important;background-color: gray85 !important;font-weight: bold;"> Illiteracy </td>
   <td style="text-align:right;color: black !important;background-color: gray85 !important;"> 0.2540 </td>
   <td style="text-align:right;color: black !important;background-color: gray85 !important;"> 0.3051 </td>
   <td style="text-align:right;color: black !important;background-color: gray85 !important;"> 0.8325 </td>
   <td style="text-align:right;color: black !important;background-color: gray85 !important;"> 0.4094 </td>
  </tr>
</tbody>
</table>

**Step 4:**

```r
# Final Model - Illiteracy rm
model3 <- lm(Life.Exp ~ Murder + HS.Grad, data = state.dat)
```

---
# Summary

<br/>

.red[$$\widehat{\text{Life.Exp}}_i = 69.48 -0.237\cdot\text{Murder}_i + 0.044\cdot\text{HS.Grad}_i $$]

<br/>

- Today we have looked at model selection methods for multiple regression, with a focus on **testing based procedures** and **criterion based methods**.

--
<br/>
<br/>

- Sometimes there might not be a simpler model that explains the data as well as the full model.

--
<br/>
<br/>

- **CAUTION** - Different model selection methods may choose different final models.

--
<br/>
<br/>

- Sometimes there is scientific or expert reasoning for statistically insignificant predictors to be retained in a model.
---

class: inverse, center, middle

# Thank you! Any questions?