Question 1

This problem involves hyperplanes in two dimensions.

  1. Sketch the hyperplane \(1 + 3X_1 - X_2 = 0\). Indicate the set of points for which \(1 + 3X_1 - X_2 > 0\), as well as the set of points for which \(1 + 3X_1 - X_2 < 0\).
library(ggplot2)
xlim <- c(-10, 10)
ylim <- c(-30, 30)
points <- expand.grid(
  X1 = seq(xlim[1], xlim[2], length.out = 50),
  X2 = seq(ylim[1], ylim[2], length.out = 50)
)
p <- ggplot(points, aes(x = X1, y = X2)) +
  geom_abline(intercept = 1, slope = 3) + # X2 = 1 + 3X1
  theme_bw()
p + geom_point(aes(color = 1 + 3 * X1 - X2 > 0), size = 0.1) +
  scale_color_discrete(name = "1 + 3X1 - X2 > 0")

  1. On the same plot, sketch the hyperplane \(-2 + X_1 + 2X_2 = 0\). Indicate the set of points for which \(-2 + X_1 + 2X_2 > 0\), as well as the set of points for which \(-2 + X_1 + 2X_2 < 0\).
p + geom_abline(intercept = 1, slope = -1 / 2) + # X2 = 1 - X1/2
  geom_point(
    aes(color = interaction(1 + 3 * X1 - X2 > 0, -2 + X1 + 2 * X2 > 0)),
    size = 0.5
  ) +
  scale_color_discrete(name = "(1 + 3X1 - X2 > 0).(-2 + X1 + 2X2 > 0)")

Question 2

We have seen that in \(p = 2\) dimensions, a linear decision boundary takes the form \(\beta_0 + \beta_1X_1 + \beta_2X_2 = 0\). We now investigate a non-linear decision boundary.

  1. Sketch the curve \[(1+X_1)^2 +(2-X_2)^2 = 4\].
points <- expand.grid(
  X1 = seq(-4, 2, length.out = 100),
  X2 = seq(-1, 5, length.out = 100)
)
p <- ggplot(points, aes(x = X1, y = X2, z = (1 + X1)^2 + (2 - X2)^2 - 4)) +
  geom_contour(breaks = 0, colour = "black") +
  theme_bw()
p

  1. On your sketch, indicate the set of points for which \[(1 + X_1)^2 + (2 - X_2)^2 > 4,\] as well as the set of points for which \[(1 + X_1)^2 + (2 - X_2)^2 \leq 4.\]
p + geom_point(aes(color = (1 + X1)^2 + (2 - X2)^2 - 4 > 0), size = 0.1)

  1. Suppose that a classifier assigns an observation to the blue class if \[(1 + X_1)^2 + (2 - X_2)^2 > 4,\] and to the red class otherwise. To what class is the observation \((0, 0)\) classified? \((-1, 1)\)? \((2, 2)\)? \((3, 8)\)?
points <- data.frame(
  X1 = c(0, -1, 2, 3),
  X2 = c(0, 1, 2, 8)
)
ifelse((1 + points$X1)^2 + (2 - points$X2)^2 > 4, "blue", "red")
## [1] "blue" "red"  "blue" "blue"
  1. Argue that while the decision boundary in (c) is not linear in terms of \(X_1\) and \(X_2\), it is linear in terms of \(X_1\), \(X_1^2\), \(X_2\), and \(X_2^2\).

The decision boundary is \[(1 + X_1)^2 + (2 - X_2)^2 -4 = 0\] which we can expand to: \[1 + 2X_1 + X_1^2 + 4 - 4X_2 + X_2^2 - 4 = 0\] which is linear in terms of \(X_1\), \(X_1^2\), \(X_2\), \(X_2^2\).