1.(a) This problem involves hyperplanes in two dimensions.

Analyze the hyperplane defined by the equation:

\[ 1 + 3X_1 - X_2 = 0 \]

This equation represents a decision boundary in the two-dimensional space formed by variables \(X_1\) and \(X_2\).

Sketch the Hyperplane

To sketch this hyperplane, we first rearrange the equation to express \(X_2\) in terms of \(X_1\):

\[ X_2 = 3X_1 + 1 \]

This is a straight line with: - Slope: 3 - Y-intercept: 1

Interpretation:

The line divides the 2D plane into two regions: - One where the inequality \(1 + 3X_1 - X_2 > 0\) holds. - The other where \(1 + 3X_1 - X_2 < 0\) holds.

Region Where \(1 + 3X_1 - X_2 > 0\)

\[ 1 + 3X_1 - X_2 > 0 \Rightarrow X_2 < 3X_1 + 1 \]

Interpretation:

All points below the hyperplane lie in this region. This region may correspond to one class or group in a classification setting, depending on context.

Region Where \(1 + 3X_1 - X_2 < 0\)

\[ 1 + 3X_1 - X_2 < 0 \Rightarrow X_2 > 3X_1 + 1 \]

Interpretation:

All points above the hyperplane lie in this region. This could represent a different group or classification region.

Visualization in R using ggplot2

The following R code visualizes: - The hyperplane (as a solid black line), - The region where \(1 + 3X_1 - X_2 > 0\) (light blue), - The region where \(1 + 3X_1 - X_2 < 0\) (pink).

library(ggplot2)


x_vals <- seq(-3, 3, length.out = 100)
line_df <- data.frame(
  X1 = x_vals,
  X2 = 3 * x_vals + 1
)


grid_df <- expand.grid(X1 = seq(-3, 3, length.out = 100),
                       X2 = seq(-10, 10, length.out = 100))


grid_df$region <- with(grid_df, ifelse(1 + 3 * X1 - X2 > 0,
                                       "1 + 3X₁ − X₂ > 0",
                                       "1 + 3X₁ − X₂ < 0"))


ggplot() +
  geom_tile(data = grid_df, aes(x = X1, y = X2, fill = region), alpha = 0.3) +
  geom_line(data = line_df, aes(x = X1, y = X2), color = "black", size = 1) +
  scale_fill_manual(values = c("1 + 3X₁ − X₂ > 0" = "lightblue",
                               "1 + 3X₁ − X₂ < 0" = "pink")) +
  labs(title = "Hyperplane: 1 + 3X₁ − X₂ = 0",
       x = "X₁",
       y = "X₂",
       fill = "Region") +
  theme_minimal()
## Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
## ℹ Please use `linewidth` instead.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.

The black line represents the hyperplane boundary \(1 + 3X_1 - X_2 = 0\).

The light blue region corresponds to \(1 + 3X_1 - X_2 > 0\) — below the line.

The pink region corresponds to \(1 + 3X_1 - X_2 < 0\) — above the line.

(b) Sketch the hyperplane \(-2 + X_1 + 2X_2 = 0\)

Start by rewriting the equation to solve for \(X_2\):

\[ -2 + X_1 + 2X_2 = 0 \Rightarrow 2X_2 = -X_1 + 2 \Rightarrow X_2 = -\frac{1}{2}X_1 + 1 \]

This is a straight line with: - Slope = -0.5
- Y-intercept = 1

Interpretation of Inequalities:

  • If \(-2 + X_1 + 2X_2 > 0 \Rightarrow X_2 > -\frac{1}{2}X_1 + 1\), the point lies above the line.
  • If \(-2 + X_1 + 2X_2 < 0 \Rightarrow X_2 < -\frac{1}{2}X_1 + 1\), the point lies below the line.

Combined Plot: Both Hyperplanes and Their Regions

The following plot shows: - Hyperplane 1 + 3X₁ − X₂ = 0 (black line) - Hyperplane −2 + X₁ + 2X₂ = 0 (blue line) - Shaded regions showing where inequalities for both hyperplanes are greater than or less than 0

line_df2 <- data.frame(
  X1 = x_vals,
  X2 = -0.5 * x_vals + 1
)


grid_df$region2 <- with(grid_df, ifelse(-2 + X1 + 2 * X2 > 0,
                                        "−2 + X₁ + 2X₂ > 0",
                                        "−2 + X₁ + 2X₂ < 0"))

ggplot() +
  geom_tile(data = grid_df, aes(x = X1, y = X2, fill = region2), alpha = 0.2) +
  geom_tile(data = grid_df, aes(x = X1, y = X2, fill = region), alpha = 0.3) +
  geom_line(data = line_df, aes(x = X1, y = X2), color = "black", size = 1) +
  geom_line(data = line_df2, aes(x = X1, y = X2), color = "blue", size = 1, linetype = "dashed") +
  scale_fill_manual(values = c("1 + 3X₁ − X₂ > 0" = "lightblue",
                               "1 + 3X₁ − X₂ < 0" = "pink",
                               "−2 + X₁ + 2X₂ > 0" = "lightgreen",
                               "−2 + X₁ + 2X₂ < 0" = "orange")) +
  labs(title = "Sketch of Two Hyperplanes with Inequality Regions",
       x = "X₁", y = "X₂", fill = "Region") +
  theme_minimal()

Black Line: \(1 + 3X_1 - X_2 = 0\).

Dashed Blue Line: \(-2 + X_1 + 2X_2 = 0\)

Light Blue / Pink: Region above/below the first hyperplane

Light Green / Orange: Region above/below the second hyperplane

Problem -2.a

Investigate a non-linear decision boundary given by:

\[(1 + X_1)^2 + (2 - X_2)^2 = 4\]

Step-by-Step Breakdown

This is the equation of a circle in the \(X_1, X_2\) plane.

1. General Form of a Circle

The general equation is:

\[(x - h)^2 + (y - k)^2 = r^2\]

  • Center: \((h, k)\)
  • Radius: \(r\)

Rewriting the Equation

Rewriting the given equation:

\[(X_1 + 1)^2 + (X_2 - 2)^2 = 4\]

Comparing with the general form, can identify: - Center = \((-1, 2)\)
- Radius = \(\sqrt{4} = 2\)

Final Answer

The curve is a circle centered at (-1, 2) with a radius of 2.

Sketch in R

library(ggplot2)


h <- -1
k <- 2
r <- 2


theta <- seq(0, 2*pi, length.out = 300)
x <- h + r * cos(theta)
y <- k + r * sin(theta)


circle_df <- data.frame(X1 = x, X2 = y)

ggplot(circle_df, aes(x = X1, y = X2)) +
  geom_path(color = "blue", size = 1) +
  geom_point(aes(x = h, y = k), color = "red", size = 3) +
  labs(title = "Circle: (X1 + 1)^2 + (X2 - 2)^2 = 4",
       x = "X1", y = "X2") +
  coord_equal() +
  theme_minimal()
## Warning in geom_point(aes(x = h, y = k), color = "red", size = 3): All aesthetics have length 1, but the data has 300 rows.
## ℹ Please consider using `annotate()` or provide this layer with data containing
##   a single row.

This code plots the circle centered at \((-1, 2)\) with radius 2.

2.(b) Region Indication on the Sketch

now indicate two regions based on the equation:

Will shade the interior region and mark the exterior using point clouds.

x_grid <- seq(-4, 2, length.out = 200)
y_grid <- seq(-1, 5, length.out = 200)
grid <- expand.grid(X1 = x_grid, X2 = y_grid)

grid$inside <- (1 + grid$X1)^2 + (2 - grid$X2)^2 <= 4


circle_df <- data.frame(
  X1 = h + r * cos(theta),
  X2 = k + r * sin(theta)
)

ggplot() +
  geom_point(data = grid, aes(x = X1, y = X2, color = inside), alpha = 0.2) +
  geom_path(data = circle_df, aes(x = X1, y = X2), color = "black", size = 1) +
  geom_point(aes(x = h, y = k), color = "red", size = 3) +
  scale_color_manual(values = c("TRUE" = "skyblue", "FALSE" = "gray"), 
                     labels = c("Inside (≤ 4)", "Outside (> 4)"),
                     name = "Region") +
  labs(title = "Region Plot Based on (1 + X1)^2 + (2 - X2)^2",
       x = "X1", y = "X2") +
  coord_equal() +
  theme_minimal()

This plot: - Shades blue the region inside or on the circle (≤ 4), - Shades gray the region outside the circle (> 4), - Shows the boundary as a solid black circle, - Highlights the center of the circle in red.

2.(c) Classification of Given Points

The classifier rule is:

Assign to blue class if \[(1 + X_1)^2 + (2 - X_2)^2 > 4\]
Otherwise, assign to red class

Now evaluate this for four observations:

classify_point <- function(x1, x2) {
  value <- (1 + x1)^2 + (2 - x2)^2
  if (value > 4) return("Blue") else return("Red")
}


obs <- data.frame(
  X1 = c(0, -1, 2, 3),
  X2 = c(0, 1, 2, 8)
)

obs$Class <- mapply(classify_point, obs$X1, obs$X2)


knitr::kable(obs, caption = "Classification of Observations")
Classification of Observations
X1 X2 Class
0 0 Blue
-1 1 Red
2 2 Blue
3 8 Blue

Interpretation

Observation Computation Classification
(0, 0) \(1^2 + 2^2 = 5\) Blue
(-1, 1) \(0^2 + 1^2 = 1\) Red
(2, 2) \(3^2 + 0^2 = 9\) Blue
(3, 8) \(4^2 + (-6)^2 = 52\) Blue

Ploting Observations on the Decision Boundary

ggplot() +
  geom_point(data = grid, aes(x = X1, y = X2, color = inside), alpha = 0.2) +
  geom_path(data = circle_df, aes(x = X1, y = X2), color = "black", size = 1) +
  geom_point(aes(x = h, y = k), color = "red", size = 3) +
  geom_point(data = obs, aes(x = X1, y = X2, color = Class), size = 3, shape = 17) +
  scale_color_manual(values = c("Red" = "red", "Blue" = "blue", "TRUE" = "skyblue", "FALSE" = "gray"),
                     name = "Classification") +
  labs(title = "Classifier Decision Boundary with Observations",
       x = "X1", y = "X2") +
  coord_equal() +
  theme_minimal()

This plot: - Shows the circular boundary - Classifies the given points as red or blue - Uses triangle shapes to highlight the observation points

2.(d) Linear in Transformed Space

The decision boundary from part (c) is:

\[(1 + X_1)^2 + (2 - X_2)^2 = 4\]

Expanding both terms:

\[(1 + X_1)^2 = 1 + 2X_1 + X_1^2\]
\[(2 - X_2)^2 = 4 - 4X_2 + X_2^2\]

Adding:

\[ 1 + 2X_1 + X_1^2 + 4 - 4X_2 + X_2^2 = 4 \\ \Rightarrow X_1^2 + X_2^2 + 2X_1 - 4X_2 + 1 = 0 \]

This is not linear in \(X_1\) and \(X_2\), due to the squared terms.

But It Is Linear in Transformed Features

Let’s define new features:

  • \(Z_1 = X_1\)
  • \(Z_2 = X_2\)
  • \(Z_3 = X_1^2\)
  • \(Z_4 = X_2^2\)

Then the decision boundary becomes:

\[Z_3 + Z_4 + 2Z_1 - 4Z_2 + 1 = 0\]

This is a linear equation in terms of \(Z_1, Z_2, Z_3, Z_4\) —
i.e., the classifier has a linear decision boundary in the transformed (nonlinear) feature space.

This is the core idea behind polynomial kernels in machine learning!