P10 — Distance-Based and Margin/Kernel-Based Learning

A. Basic Concepts of Distance-Based and Support Vector Learning

In supervised learning, the relationship between predictors and the response can be learned through different approaches. Some algorithms generate predictions based on the proximity of an observation to other observations, whereas others construct a decision function through an optimization process.

K-Nearest Neighbors (KNN) is a distance-based learning method. This algorithm uses a number of training observations closest to a new observation to determine its prediction. In contrast, Support Vector Machine (SVM) applies optimization principles to construct a decision function based on the characteristics of the training data.

In classification, SVM identifies a decision boundary that separates classes while considering the margin. The extension of SVM to numerical responses is known as Support Vector Regression (SVR), which constructs a regression function with a specified tolerance for prediction errors.

The fundamental differences between these approaches are summarized in the following table.

Aspect K-Nearest Neighbors Support Vector Machine
Learning principle Proximity between observations Optimization of a decision function
Basis of prediction Nearest observations Optimized decision function
Classification Voting among neighboring classes Class determination using a decision boundary
Regression Mean response of neighboring observations Regression function with error tolerance
Model characteristics Nonparametric and observation-based Margin optimization and regularization
Main parameters Number of neighbors and distance measure Cost, kernel, and kernel parameters

Both KNN and SVM can be used for classification and regression. However, their mathematical formulations and prediction mechanisms differ.

Data Representation

Suppose a dataset contains \(n\) observations and \(p\) predictors, represented as:

\[\mathcal{D} = \left\{ (\mathbf{x}_i,y_i) \right\}_{i=1}^{n}\]

where:

  • \(\mathbf{x}_i=(x_{i1},x_{i2},\ldots,x_{ip})^\top\) is the predictor vector for the \(i\)-th observation;
  • \(y_i\) is the response value for the \(i\)-th observation;
  • \(n\) is the number of observations;
  • \(p\) is the number of predictors.

For classification, \(y_i\) represents a class category. For regression, \(y_i\) represents a numerical response.

The objective of modeling is to construct a prediction rule for a new observation \(\mathbf{x}_0\) based on the relationship learned from the training data.

B. K-Nearest Neighbors (KNN)

Basic Concept of KNN

K-Nearest Neighbors is a nonparametric supervised learning method that uses proximity between observations as the basis for prediction. This algorithm does not assume a particular functional relationship between predictors and the response, such as a linear or quadratic relationship.

KNN works by calculating the distance between a new observation and each observation in the training data. After all distances have been calculated, the algorithm selects the \(K\) observations with the smallest distances. The responses of these nearest observations are then used to generate a prediction.

In general, the KNN procedure consists of the following steps:

  1. Determine the number of neighbors, \(K\).
  2. Calculate the distance between the new observation and each training observation.
  3. Sort the observations according to their distances.
  4. Select the \(K\) nearest observations.
  5. Generate a prediction based on the responses of the selected neighbors.
NoteFigure Placeholder: KNN Classification

Figure reference: James et al. (2021), An Introduction to Statistical Learning, Figure 2.14.

Description: Illustration of selecting the three nearest observations to determine the class of a new observation.

Consider the black cross in the left panel.

This point represents a new observation whose class is unknown. The circle indicates the region containing its three nearest observations.

Inside the circle, there are:

  • Two blue observations.
  • One orange observation.

Because the majority of its nearest neighbors belong to the blue class, KNN assigns the new observation to the blue class.

The right panel shows the decision boundary obtained when the same procedure is applied to different possible locations of new observations.

Important point: The KNN decision boundary is formed by applying the nearest-neighbor rule across different locations in the predictor space. Unlike a linear classification model, KNN does not first estimate a single equation for a separating line.

Measuring Distances Between Observations

Neighbor selection is the main component of KNN. Therefore, the distance measure determines which observations are considered closest to a new observation.

Euclidean Distance

The Euclidean distance between observations \(\mathbf{x}_i\) and \(\mathbf{x}_j\) is defined as:

\[d(\mathbf{x}_i,\mathbf{x}_j) = \sqrt{ \sum_{r=1}^{p} (x_{ir}-x_{jr})^2 }\]

For two predictors, the formula becomes:

\[d(\mathbf{x}_i,\mathbf{x}_j) = \sqrt{ (x_{i1}-x_{j1})^2+ (x_{i2}-x_{j2})^2 }\]

For example, suppose:

\[\mathbf{x}_1=(2,3)\]

and:

\[\mathbf{x}_2=(5,7)\]

The distance between these two observations is:

\[\begin{aligned} d(\mathbf{x}_1,\mathbf{x}_2) &= \sqrt{(2-5)^2+(3-7)^2}\\ &= \sqrt{9+16}\\ &=5 \end{aligned}\]

A smaller distance indicates that the two observations are closer to each other in the predictor space.

Manhattan Distance

Manhattan distance calculates the sum of the absolute differences across all predictors:

\[d(\mathbf{x}_i,\mathbf{x}_j) = \sum_{r=1}^{p} |x_{ir}-x_{jr}|\]

Using the same example:

\[\begin{aligned} d(\mathbf{x}_1,\mathbf{x}_2) &= |2-5|+|3-7|\\ &=3+4\\ &=7 \end{aligned}\]

Minkowski Distance

Euclidean distance and Manhattan distance are special cases of Minkowski distance:

\[d_q(\mathbf{x}_i,\mathbf{x}_j) = \left( \sum_{r=1}^{p} |x_{ir}-x_{jr}|^q \right)^{1/q}\]

where \(q\geq1\).

When \(q=1\), the formula corresponds to Manhattan distance. When \(q=2\), it corresponds to Euclidean distance.

Different distance measures may produce different rankings of nearest neighbors and, consequently, affect KNN predictions.

KNN for Classification

In classification, KNN uses the most frequently occurring class among the \(K\) nearest neighbors.

Let \(\mathcal{N}_K(\mathbf{x}_0)\) denote the set of indices corresponding to the \(K\) nearest training observations to a new observation \(\mathbf{x}_0\).

The proportion of neighboring observations belonging to class \(g\) is defined as:

\[\widehat{P}(Y=g\mid\mathbf{X}=\mathbf{x}_0) = \frac{1}{K} \sum_{i\in\mathcal{N}_K(\mathbf{x}_0)} I(y_i=g)\]

where the indicator function is:

\[I(y_i=g) = \begin{cases} 1, & \text{if } y_i=g\\ 0, & \text{if } y_i\neq g \end{cases}\]

The predicted class is determined by the largest class proportion:

\[\boxed{ \hat y_0 = \arg\max_g \widehat{P}(Y=g\mid\mathbf{X}=\mathbf{x}_0) }\]

For example, suppose \(K=5\), with three neighbors belonging to class A and two neighbors belonging to class B.

Then:

\[\widehat P(Y=A\mid\mathbf{x}_0) = \frac{3}{5} = 0.6\]

\[\widehat P(Y=B\mid\mathbf{x}_0) = \frac{2}{5} = 0.4\]

Therefore, the new observation is classified as class A.

These proportions represent estimated class probabilities based on the composition of the selected neighbors. However, they are not automatically calibrated probabilities.

KNN for Regression

In regression, KNN uses the mean response value of the \(K\) nearest neighbors.

The prediction formula is:

\[\boxed{ \hat y_0 = \frac{1}{K} \sum_{i\in\mathcal{N}_K(\mathbf{x}_0)} y_i }\]

Suppose the three nearest observations have the following response values:

\[y_1=120,\qquad y_2=150,\qquad y_3=180\]

With \(K=3\), the predicted response is:

\[\begin{aligned} \hat y_0 &= \frac{120+150+180}{3}\\ &=150 \end{aligned}\]

The main difference between KNN classification and regression lies in how the neighboring responses are combined. Classification uses class voting, whereas regression uses the mean of numerical responses.

Weighted KNN

In standard KNN, every selected neighbor contributes equally to the prediction. However, observations that are closer to the new observation can be assigned larger weights.

For regression, the weighted prediction is defined as:

\[\hat y_0 = \frac{ \sum_{i\in\mathcal{N}_K(\mathbf{x}_0)} w_i(\mathbf{x}_0)y_i }{ \sum_{i\in\mathcal{N}_K(\mathbf{x}_0)} w_i(\mathbf{x}_0) }\]

where \(w_i(\mathbf{x}_0)\geq0\) is the weight assigned to the \(i\)-th neighbor, with a positive total weight.

For weighted classification, class contributions are calculated using the sum of the corresponding weights:

\[\hat y_0 = \arg\max_g \sum_{i\in\mathcal{N}_K(\mathbf{x}_0)} w_i(\mathbf{x}_0)I(y_i=g)\]

Weighting allows observations closer to the new observation to have a greater influence on the prediction.

Effect of the Number of Neighbors

The number of neighbors, \(K\), determines the flexibility of the KNN model.

NoteFigure Placeholder: Effect of K on the Decision Boundary

Figure reference: James et al. (2021), An Introduction to Statistical Learning, Figure 2.16.

Description: Comparison of the flexibility of KNN decision boundaries for different numbers of neighbors.

In the left panel, \(K=1\) produces a highly irregular decision boundary. Each prediction depends on only one nearest observation, so a small change in the location of a new observation may change its predicted class.

In the right panel, \(K=100\) produces a smoother and simpler decision boundary. Predictions are influenced by more observations, so smaller local patterns may be overlooked.

A small value of \(K\) produces a more flexible model because predictions are strongly influenced by local observations. However, this also increases the model’s sensitivity to variations in the data.

Conversely, a large value of \(K\) produces smoother predictions because more observations are considered.

In general:

  • A small \(K\) tends to produce lower bias and higher variance.
  • A large \(K\) tends to produce higher bias and lower variance.

This relationship illustrates the bias–variance trade-off.

NoteFigure Placeholder: KNN Flexibility and Prediction Error

Figure reference: James et al. (2021), An Introduction to Statistical Learning, Figure 2.17.

Description: Relationship between KNN flexibility and prediction errors in training and test data.

In Figure 2.17, the horizontal axis uses \(1/K\). Therefore, moving to the right corresponds to a smaller number of neighbors and a more flexible model.

Notice that training error generally decreases as model flexibility increases. However, test error does not necessarily follow the same pattern.

When a model becomes too flexible, training error may become very small, but its predictive performance on new observations may not improve. This situation is associated with the risk of overfitting.

The key lesson: The value of \(K\) should not be selected solely based on how well the model fits the training data. It should be selected using a validation procedure.

Limitations of KNN

There are four important limitations.

First, the curse of dimensionality. As the number of dimensions increases, observations tend to become more dispersed in the predictor space. The relatively nearest observation may no longer be particularly close in geometric terms.

Second, sensitivity to predictor scales. Predictors with larger numerical ranges may dominate distance calculations.

Third, sensitivity to class distributions. In classification with imbalanced classes, observations from the majority class may appear more frequently among the selected neighbors.

Fourth, computational cost during prediction. KNN must identify nearest neighbors for each new observation. For large datasets, this search may require substantial computation, although more efficient search strategies are available.

C. Support Vector Machine for Classification

Basic Concept of SVM

Support Vector Machine is a supervised learning method that constructs a decision function through an optimization process.

In binary classification, SVM seeks to construct a hyperplane that separates two classes with the largest possible margin. The margin describes the distance between the separating hyperplane and the nearest observations that determine the separation.

Observations that play a role in determining the margin are called support vectors.

NoteFigure Placeholder: Maximum-Margin Hyperplane

Figure reference: James et al. (2021), An Introduction to Statistical Learning, Figure 9.3.

Description: Optimal separating hyperplane, margin boundaries, and observations acting as support vectors.

In the illustration:

  • The solid line represents the separating hyperplane.
  • The two dashed lines represent the margin boundaries.
  • Points located exactly on the margin boundaries are support vectors.
  • Regions with different colors indicate the classes predicted by the model.

Support vectors are important because they determine the position of the optimal decision boundary. In the maximum-margin setting, changing the positions of support vectors may change the hyperplane, whereas small changes to other observations that remain outside the margin do not necessarily affect it.

Hyperplane Formulation

For \(p\) predictors, the linear decision function is defined as:

\[f(\mathbf{x}) = \mathbf{w}^{\top}\mathbf{x}+b\]

where:

  • \(\mathbf{w}=(w_1,w_2,\ldots,w_p)^\top\) is the weight vector;
  • \(\mathbf{x}\) is the predictor vector;
  • \(b\) is the intercept.

The separating hyperplane is defined as:

\[\boxed{ \mathbf{w}^{\top}\mathbf{x}+b=0 }\]

For two predictors, the equation becomes:

\[w_1x_1+w_2x_2+b=0\]

In binary classification, response classes are represented as:

\[y_i\in\{-1,+1\}\]

The prediction rule is:

\[\boxed{ \hat y = \operatorname{sign} \left( \mathbf{w}^{\top}\mathbf{x}+b \right) }\]

A positive decision-function value corresponds to class \(+1\), whereas a negative value corresponds to class \(-1\).

Hard-Margin SVM

Hard-margin SVM is used when the two classes can be linearly separated without classification errors.

The canonical margin boundaries are defined as:

\[\mathbf{w}^{\top}\mathbf{x}+b=1\]

and:

\[\mathbf{w}^{\top}\mathbf{x}+b=-1\]

The distance between these two boundaries is:

\[M = \frac{2}{\|\mathbf{w}\|}\]

where:

\[\|\mathbf{w}\| = \sqrt{ \sum_{j=1}^{p}w_j^2 }\]

Maximizing the margin is equivalent to minimizing the squared norm of the weight vector.

The hard-margin SVM optimization problem is:

\[\boxed{ \min_{\mathbf{w},b} \frac{1}{2}\|\mathbf{w}\|^2 }\]

subject to:

\[y_i \left( \mathbf{w}^{\top}\mathbf{x}_i+b \right) \geq1, \qquad i=1,\ldots,n\]

These constraints ensure that every observation lies on the appropriate side of the separating hyperplane and does not violate the margin boundaries.

Soft-Margin SVM

In empirical data, classes often overlap, making perfect linear separation impossible.

Soft-margin SVM introduces slack variables \(\xi_i\) to allow violations of the margin.

The optimization problem becomes:

\[\boxed{ \min_{\mathbf{w},b,\boldsymbol{\xi}} \left[ \frac{1}{2}\|\mathbf{w}\|^2 + C\sum_{i=1}^{n}\xi_i \right] }\]

subject to:

\[y_i \left( \mathbf{w}^{\top}\mathbf{x}_i+b \right) \geq1-\xi_i\]

and:

\[\xi_i\geq0\]

The parameter \(C>0\) controls the penalty imposed on margin violations.

  • A large value of \(C\) imposes a higher penalty on margin violations.
  • A small value of \(C\) allows greater tolerance for margin violations.

Thus, the choice of \(C\) affects the balance between the complexity of the decision function and classification errors.

Kernels in SVM

A linear hyperplane cannot always represent the separation between classes when the underlying pattern is nonlinear.

SVM can use kernel functions to construct nonlinear decision boundaries. Conceptually, predictors are mapped into a feature space using a function \(\phi(\cdot)\).

A kernel represents the inner product in that feature space:

\[K(\mathbf{x}_i,\mathbf{x}_j) = \phi(\mathbf{x}_i)^\top \phi(\mathbf{x}_j)\]

This approach makes it possible to calculate relationships between observations in the feature space without explicitly constructing all coordinates resulting from the mapping.

Several commonly used kernels are described below.

Linear Kernel

\[K(\mathbf{x}_i,\mathbf{x}_j) = \mathbf{x}_i^\top\mathbf{x}_j\]

Polynomial Kernel

\[K(\mathbf{x}_i,\mathbf{x}_j) = \left( \gamma\mathbf{x}_i^\top\mathbf{x}_j+r \right)^d\]

Here, \(\gamma\) controls the scaling of the inner product, \(r\) is an offset parameter, and \(d\) is the polynomial degree.

Radial Basis Function Kernel

\[K(\mathbf{x}_i,\mathbf{x}_j) = \exp \left( -\gamma \|\mathbf{x}_i-\mathbf{x}_j\|^2 \right)\]

where \(\gamma>0\) controls the characteristics of the RBF kernel.

For kernel-based SVM, the binary classification decision function can be written as:

\[f(\mathbf{x}) = \sum_{i\in SV} \alpha_i y_i K(\mathbf{x}_i,\mathbf{x}) +b\]

where \(SV\) denotes the set of support-vector indices and \(\alpha_i\) represents the nonzero dual coefficients associated with the support vectors.

The predicted class is obtained using:

\[\hat y = \operatorname{sign} \left( f(\mathbf{x}) \right)\]

Thus, kernels allow nonlinear decision boundaries to be constructed, while the underlying principle of selecting a decision function remains based on margin optimization.

The basic SVM formulation addresses binary classification. For responses containing more than two classes, multiple binary classifiers can be combined using strategies such as one-versus-one or one-versus-rest.

D. Support Vector Regression (SVR)

Basic Concept of SVR

Support Vector Regression extends support vector principles to numerical responses.

Unlike SVM classification, which constructs a boundary between classes, SVR seeks a regression function while controlling model complexity and prediction errors.

SVR introduces a parameter \(\varepsilon\) that defines a tolerance region around the regression function.

Suppose a linear regression function is defined as:

\[f(\mathbf{x}) = \mathbf{w}^{\top}\mathbf{x}+b\]

The tolerance region is defined by:

\[f(\mathbf{x})-\varepsilon \leq y \leq f(\mathbf{x})+\varepsilon\]

This region is known as the epsilon-insensitive tube, or \(\varepsilon\)-tube.

NoteFigure Placeholder: Epsilon-Insensitive Tube

Figure reference: Smola and Schölkopf (2004), A Tutorial on Support Vector Regression, Figure 1.

Description: Tolerance region around a regression function, showing observations located inside and outside the epsilon-tube.

In the illustration, the regression function lies at the center of a tolerance region bounded by:

\[f(\mathbf{x})+\varepsilon\]

and:

\[f(\mathbf{x})-\varepsilon\]

Observations located inside or exactly on the boundaries of the tube receive no penalty under the epsilon-insensitive loss function. In contrast, observations outside the tube incur a penalty according to the magnitude of their violations.

Epsilon-Insensitive Loss

The SVR loss function is defined as:

\[\boxed{ L_{\varepsilon} \left( y_i,f(\mathbf{x}_i) \right) = \max \left( 0, |y_i-f(\mathbf{x}_i)|-\varepsilon \right) }\]

Suppose the predicted value is 100 and the parameter is \(\varepsilon=5\).

Actual Value Absolute Error Epsilon-Insensitive Loss
102 2 0
105 5 0
108 8 3
112 12 7

This example shows that SVR does not penalize every prediction error. A penalty is incurred only when the absolute error exceeds \(\varepsilon\).

SVR Optimization Formulation

To accommodate observations located outside the epsilon-tube, SVR introduces slack variables \(\xi_i\) and \(\xi_i^*\).

The epsilon-SVR optimization problem is defined as:

\[\boxed{ \min_{ \mathbf{w},b, \boldsymbol{\xi}, \boldsymbol{\xi}^{*} } \left[ \frac{1}{2}\|\mathbf{w}\|^2 + C\sum_{i=1}^{n} (\xi_i+\xi_i^*) \right] }\]

subject to:

\[y_i-f(\mathbf{x}_i) \leq \varepsilon+\xi_i\]

\[f(\mathbf{x}_i)-y_i \leq \varepsilon+\xi_i^*\]

\[\xi_i,\xi_i^*\geq0\]

In this formulation:

  • \(\frac{1}{2}\|\mathbf{w}\|^2\) controls the complexity of the regression function;
  • \(C\) determines the penalty for errors outside the epsilon-tube;
  • \(\varepsilon\) determines the width of the tolerance region on either side of the regression function;
  • \(\xi_i\) and \(\xi_i^*\) represent violations on the two sides of the tube.

Observations located outside the tube may act as support vectors. Observations on the tube boundaries may also become support vectors, whereas observations strictly inside the tube generally have no dual-coefficient contribution to the prediction function.

Kernels in SVR

SVR can use kernel functions to represent nonlinear relationships.

Its prediction function is defined as:

\[\boxed{ f(\mathbf{x}) = \sum_{i\in SV} (\alpha_i-\alpha_i^*) K(\mathbf{x}_i,\mathbf{x}) +b }\]

where \(\alpha_i\) and \(\alpha_i^*\) are dual coefficients from the SVR optimization formulation.

The kernel can be linear, polynomial, or RBF.

Therefore, SVM classification and SVR share related optimization principles and kernel-based representations. However, they differ in their loss functions and prediction objectives.

E. Predictor Representation and Methodological Limitations

Effect of Predictor Scales

Both KNN and SVM can be affected by predictor scales.

In KNN, distance measures depend on differences between predictor values. Predictors with larger numerical ranges may dominate Euclidean distance calculations.

In SVM, differences in scale affect the geometry of the feature space, margin optimization, and the calculation of certain kernel functions.

Standardization is one approach used to adjust predictor scales:

\[z_{ij} = \frac{x_{ij}-\bar{x}_j}{s_j}\]

where \(\bar{x}_j\) and \(s_j\) are the mean and standard deviation of the \(j\)-th predictor, respectively, calculated from the training data.

Categorical Predictors

KNN and SVM generally require numerical representations to calculate distances or kernel functions.

Nominal categorical predictors can be represented using indicator encoding, for example:

\[D_k = I(X=c_k)\]

where \(c_k\) denotes the \(k\)-th category.

However, categorical encoding also affects the geometric representation of the data. In KNN, changing the encoding may alter distances between observations. In SVM, it may affect the resulting decision function.

Therefore, predictor representations should be selected according to the characteristics of the data and the principles of the algorithm.

Curse of Dimensionality

In distance-based methods, increasing the number of predictors may cause observations to become relatively farther apart in the feature space.

This phenomenon is known as the curse of dimensionality.

As the number of predictors increases, the difference between distances to the nearest and farthest observations may become less informative. Consequently, the local proximity concept used in KNN may lose some of its usefulness.

In SVM, adding features can also affect model complexity, although the effects depend on the kernel and regularization being used.

Learning Validity

All transformations requiring parameter estimation must be based on the training data. Information from the test data must not be used to determine transformation parameters or construct the model.

This principle is necessary to ensure that predictive evaluation reflects the model’s performance on observations that were not used during learning.

F. Summary of Methods

KNN and SVM are two supervised learning approaches with different mechanisms.

KNN uses local information based on distances between observations. In classification, predictions are obtained through voting among neighboring classes, whereas regression uses the mean response of the nearest neighbors.

SVM constructs a decision function based on optimization principles. For classification, this process involves class separation and margin optimization. For regression, SVR uses an epsilon-tolerance region to determine prediction-error penalties.

Method Response Type Prediction Principle Main Parameters
KNN Classification Categorical Voting among nearest neighbors \(K\), distance, weights
KNN Regression Numerical Mean response of nearest neighbors \(K\), distance, weights
Support Vector Classification Categorical Margin-based decision function \(C\), kernel, kernel parameters
Support Vector Regression Numerical Regression function with epsilon-insensitive loss \(C\), \(\varepsilon\), kernel, kernel parameters

These differences in learning mechanisms mean that method selection should consider the response type, predictor characteristics, the form of the relationship to be learned, and model complexity.

G. Package Preparation

This practical session uses several R packages for data processing, the application of K-Nearest Neighbors (KNN) and Support Vector Machine (SVM) algorithms, and the evaluation of prediction results.

The packages are selected according to the requirements of each analytical stage, from data preparation to model performance assessment.

Required Packages

Package Purpose in This Practical Session R Functions Used
tidyverse Processes data, selects and transforms variables, produces summaries, and creates visualizations of data and modeling results. select(), mutate(), filter(), ggplot(), geom_point(), geom_line()
tidymodels Splits training and test data, prepares predictors, performs standardization, and evaluates classification and regression results. initial_split(), training(), testing(), recipe(), step_normalize(), accuracy(), roc_auc(), rmse(), mae()
kknn Applies KNN algorithms for classification and regression. This package allows the specification of the number of neighbors, distance measure, and neighbor weighting. kknn(), predict()
e1071 Applies SVM for classification and Support Vector Regression (SVR) for regression, including linear, polynomial, and radial kernels. svm(), predict()
knitr Displays analysis results as well-formatted tables in a Quarto document. kable()

The tidymodels package is a collection of packages for statistical modeling and machine learning. It includes several packages used in this practical session:

  • rsample for data splitting and resampling.
  • recipes for predictor preparation and transformation.
  • yardstick for calculating evaluation metrics.
  • parsnip for defining model specifications.
  • workflows for combining preprocessing and modeling stages.

Therefore, these packages do not need to be installed separately when tidymodels has been successfully installed.

Package Installation

Packages only need to be installed once on the computer being used. If the required packages are already available, the installation step can be skipped.

Show Code
# Install the required packages
install.packages(c(
  "tidyverse",
  "tidymodels",
  "kknn",
  "e1071",
  "knitr"
))

The installation code above is not executed automatically when the Quarto document is rendered. If necessary, run the installation commands directly in the R Console.

Loading Packages

After installation, the packages are loaded using the library() function so that the required functions are available throughout the practical session.

Show Code
# Packages for data processing and visualization
library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.2.1     ✔ readr     2.2.0
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.3     ✔ tibble    3.3.1
✔ lubridate 1.9.5     ✔ tidyr     1.3.2
✔ purrr     1.2.2     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
Show Code
# Packages for data preparation and model evaluation
library(tidymodels)
── Attaching packages ────────────────────────────────────── tidymodels 1.5.0 ──
✔ broom        1.0.13     ✔ rsample      1.3.2 
✔ dials        1.4.4      ✔ tailor       0.1.0 
✔ infer        1.1.0      ✔ tune         2.1.0 
✔ modeldata    1.6.0      ✔ workflows    1.3.0 
✔ parsnip      1.6.1      ✔ workflowsets 1.1.1 
✔ recipes      1.4.0      ✔ yardstick    1.4.0 
── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ──
✖ scales::discard() masks purrr::discard()
✖ dplyr::filter()   masks stats::filter()
✖ recipes::fixed()  masks stringr::fixed()
✖ dplyr::lag()      masks stats::lag()
✖ yardstick::spec() masks readr::spec()
✖ recipes::step()   masks stats::step()
Show Code
# Package for KNN modeling
library(kknn)

# Package for SVM and SVR modeling
library(e1071)

Attaching package: 'e1071'

The following object is masked from 'package:tune':

    tune

The following object is masked from 'package:rsample':

    permutations

The following object is masked from 'package:parsnip':

    tune

The following object is masked from 'package:ggplot2':

    element
Show Code
# Package for presenting tables
library(knitr)

Package Usage by Method

Package usage depends on the method being applied. KNN and SVM can be used for both classification and regression, but their response types and prediction outputs differ.

Method Modeling Task Package Main Function Prediction Output
KNN Classification Classification kknn kknn() Class category
KNN Regression Regression kknn kknn() Numerical value
Support Vector Classification Classification e1071 svm() Class category
Support Vector Regression Regression e1071 svm() Numerical value

In KNN, the predicted class or numerical value is determined using training observations closest to the new observation.

Meanwhile, SVM classification constructs a decision boundary based on maximum-margin principles, whereas SVR uses a loss function that tolerates a specified difference between actual and predicted values.

Although both SVM classification and regression can be performed using the svm() function, their optimization formulations are different. These differences have been explained in the corresponding method sections.

Notes on Package Usage

All preprocessing procedures that require parameter estimation, such as standardization using means and standard deviations, must be performed based on the training data.

The resulting parameters are then applied to the test data. # H. Implementation of KNN and SVM

Data Description

The dataset used in this practical session is Wine Quality, which contains physicochemical characteristics and quality assessments of Portuguese Vinho Verde wines. The dataset was introduced by Cortez et al. (2009) to investigate the relationship between wine physicochemical properties and quality ratings obtained through sensory evaluation.

The dataset includes two types of wine: red wine and white wine. Physicochemical measurements serve as predictor variables, while the quality score (quality) serves as the response variable. These characteristics allow the dataset to be examined using both classification and regression approaches.

Data source: UCI Machine Learning Repository – Wine Quality.

Identifying the Data Structure

Before fitting the models, the data structure must be examined to determine the number of observations, number of variables, data types, and the values stored in each column.

The initial inspection begins by importing the CSV file and displaying the first few observations.

Show Code
# Import the dataset
wine_data <- read_csv("wine-quality-white-and-red.csv",
                      show_col_types = FALSE)

# Display the first six observations
head(wine_data)

Next, the dataset dimensions, variable names, and structure are examined using dim(), names(), and glimpse().

Show Code
# Check the dataset dimensions
dim(wine_data)
[1] 6497   13
Show Code
# Display the variable names
names(wine_data)
 [1] "type"                 "fixed acidity"        "volatile acidity"    
 [4] "citric acid"          "residual sugar"       "chlorides"           
 [7] "free sulfur dioxide"  "total sulfur dioxide" "density"             
[10] "pH"                   "sulphates"            "alcohol"             
[13] "quality"             
Show Code
# Examine the data structure
glimpse(wine_data)
Rows: 6,497
Columns: 13
$ type                   <chr> "white", "white", "white", "white", "white", "w…
$ `fixed acidity`        <dbl> 7.0, 6.3, 8.1, 7.2, 7.2, 8.1, 6.2, 7.0, 6.3, 8.…
$ `volatile acidity`     <dbl> 0.27, 0.30, 0.28, 0.23, 0.23, 0.28, 0.32, 0.27,…
$ `citric acid`          <dbl> 0.36, 0.34, 0.40, 0.32, 0.32, 0.40, 0.16, 0.36,…
$ `residual sugar`       <dbl> 20.70, 1.60, 6.90, 8.50, 8.50, 6.90, 7.00, 20.7…
$ chlorides              <dbl> 0.045, 0.049, 0.050, 0.058, 0.058, 0.050, 0.045…
$ `free sulfur dioxide`  <dbl> 45, 14, 30, 47, 47, 30, 30, 45, 14, 28, 11, 17,…
$ `total sulfur dioxide` <dbl> 170, 132, 97, 186, 186, 97, 136, 170, 132, 129,…
$ density                <dbl> 1.0010, 0.9940, 0.9951, 0.9956, 0.9956, 0.9951,…
$ pH                     <dbl> 3.00, 3.30, 3.26, 3.19, 3.19, 3.26, 3.18, 3.00,…
$ sulphates              <dbl> 0.45, 0.49, 0.44, 0.40, 0.40, 0.44, 0.47, 0.45,…
$ alcohol                <dbl> 8.8, 9.5, 10.1, 9.9, 9.9, 10.1, 9.6, 8.8, 9.5, …
$ quality                <dbl> 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 5, 5, 5, 7, 5, 7,…

The dim() function provides the number of rows and columns, while names() displays the names of all variables. The glimpse() function shows the data types and sample values for each variable, allowing numerical and categorical variables to be identified.

The data structure inspection shows that the dataset contains 6,497 observations and 13 variables. The type variable is stored as character data (chr) and identifies the wine type, while the eleven physicochemical variables are stored as numerical data (dbl). The quality variable is also stored numerically, but its values represent ordered quality ratings and are therefore ordinal in nature.

Several variable names still contain spaces, including fixed acidity, volatile acidity, and residual sugar. This naming convention needs to be considered when referencing variables in R. The variable names will be adjusted during data preparation to simplify their use in modeling.

Descriptive Statistics

Descriptive statistics provide an initial overview of the numerical predictors. The examination includes the minimum, first quartile, median, mean, standard deviation, third quartile, and maximum.

These measures of central tendency and dispersion help identify differences in numerical ranges across variables and provide indications of asymmetric distributions. This information is relevant because predictor representation and scaling can influence the modeling process in KNN and SVM.

Descriptive statistics are calculated for the eleven physicochemical variables. The quality variable will be examined separately through the distribution of quality scores.

Show Code
# Descriptive statistics for numerical predictors
ringkasan_numerik <- wine_data %>%
  select(-type, -quality) %>%
  pivot_longer(
    cols = everything(),
    names_to = "variabel",
    values_to = "nilai"
  ) %>%
  group_by(variabel) %>%
  summarise(
    minimum = min(nilai, na.rm = TRUE),
    kuartil_1 = quantile(nilai, 0.25, na.rm = TRUE),
    median = median(nilai, na.rm = TRUE),
    rata_rata = mean(nilai, na.rm = TRUE),
    simpangan_baku = sd(nilai, na.rm = TRUE),
    kuartil_3 = quantile(nilai, 0.75, na.rm = TRUE),
    maksimum = max(nilai, na.rm = TRUE),
    .groups = "drop"
  )

knitr::kable(
  ringkasan_numerik,
  digits = 4,
  caption = "Descriptive statistics of physicochemical variables"
)
Descriptive statistics of physicochemical variables
variabel minimum kuartil_1 median rata_rata simpangan_baku kuartil_3 maksimum
alcohol 8.0000 9.5000 10.3000 10.4918 1.1927 11.300 14.900
chlorides 0.0090 0.0380 0.0470 0.0560 0.0350 0.065 0.611
citric acid 0.0000 0.2500 0.3100 0.3186 0.1453 0.390 1.660
density 0.9871 0.9923 0.9949 0.9947 0.0030 0.997 1.039
fixed acidity 3.8000 6.4000 7.0000 7.2153 1.2964 7.700 15.900
free sulfur dioxide 1.0000 17.0000 29.0000 30.5253 17.7494 41.000 289.000
pH 2.7200 3.1100 3.2100 3.2185 0.1608 3.320 4.010
residual sugar 0.6000 1.8000 3.0000 5.4432 4.7578 8.100 65.800
sulphates 0.2200 0.4300 0.5100 0.5313 0.1488 0.600 2.000
total sulfur dioxide 6.0000 77.0000 118.0000 115.7446 56.5219 156.000 440.000
volatile acidity 0.0800 0.2300 0.2900 0.3397 0.1646 0.400 1.580

The descriptive statistics reveal substantial differences in scale across the physicochemical variables. For example, total sulfur dioxide ranges from 6 to 440, whereas density ranges only from 0.9871 to 1.0390. These differences are important because predictors with larger scales may dominate distance calculations in KNN and distance-based kernels in SVM. Predictor standardization is therefore needed to prevent measurement units from disproportionately influencing the calculated distances.

Several variables also show indications of right-skewed distributions. For residual sugar, the mean of 5.4432 exceeds the median of 3.0000, while the maximum value of 65.8000 is substantially higher than the third quartile of 8.1000. Relatively high maximum values are also observed for chlorides and free sulfur dioxide. These patterns indicate the need to examine extreme observations, although such values should not automatically be considered measurement errors.

Distribution of the Response Variable

The quality variable represents an ordered wine quality rating, ranging from lower to higher quality levels. Its distribution is examined to determine the frequency of each score and assess whether observations are evenly distributed across quality levels.

Show Code
# Calculate the distribution of quality scores
distribusi_quality <- wine_data %>%
  count(quality) %>%
  mutate(persentase = round(n / sum(n) * 100, 2))

knitr::kable(
  distribusi_quality,
  col.names = c("Quality Score", "Frequency", "Percentage (%)"),
  caption = "Distribution of wine quality scores"
)
Distribution of wine quality scores
Quality Score Frequency Percentage (%)
3 30 0.46
4 216 3.32
5 2138 32.91
6 2836 43.65
7 1079 16.61
8 193 2.97
9 5 0.08

A bar chart is used to illustrate differences in frequency across quality levels.

Show Code
# Visualize the distribution of quality scores
ggplot(distribusi_quality, aes(x = factor(quality), y = n)) +
  geom_col(fill = "#27689B", width = 0.7) +
  labs(
    title = "Distribution of Wine Quality Scores",
    x = "Quality Score",
    y = "Number of Observations"
  ) +
  theme_minimal()

The distribution of quality scores is concentrated at scores of 5 (32.91%) and 6 (43.65%), which together account for 76.56% of observations. In contrast, scores of 3 and 9 represent only 0.54% of the dataset. This concentration indicates that moderate quality levels are more strongly represented than very low or high quality levels. In regression, the limited number of observations at both ends of the score range should be considered because predictive accuracy for rare scores may differ from that for more common scores.

When the scores are grouped into Good (score ≥ 6) and Not Good (score < 6), the class proportions are approximately 63.31% and 36.69%, respectively. This difference should be considered when evaluating classification performance. Model performance should therefore be assessed using metrics such as balanced accuracy and F1-score in addition to overall accuracy.

Data Quality Assessment

Data quality assessment is performed to identify missing values, identical rows, and numerical observations that differ substantially from most of the data. The findings provide a basis for determining the required preprocessing steps before modeling.

Show Code
# Count missing values in each variable
missing_wine <- wine_data %>%
  summarise(across(everything(), ~ sum(is.na(.x)))) %>%
  pivot_longer(
    cols = everything(),
    names_to = "variabel",
    values_to = "jumlah_missing"
  )

knitr::kable(
  missing_wine,
  caption = "Number of missing values by variable"
)
Number of missing values by variable
variabel jumlah_missing
type 0
fixed acidity 0
volatile acidity 0
citric acid 0
residual sugar 0
chlorides 0
free sulfur dioxide 0
total sulfur dioxide 0
density 0
pH 0
sulphates 0
alcohol 0
quality 0

In addition to missing values, rows containing identical values across all variables are examined.

Show Code
# Check for identical rows
duplikat_wine <- tibble(
  jumlah_observasi = nrow(wine_data),
  baris_identik = sum(duplicated(wine_data)),
  baris_unik = nrow(distinct(wine_data))
)

knitr::kable(
  duplikat_wine,
  caption = "Assessment of identical rows"
)
Assessment of identical rows
jumlah_observasi baris_identik baris_unik
6497 1177 5320

Extreme values in the numerical predictors are examined using the IQR rule. Observations are flagged when their values fall below \(Q_1 - 1.5 \times IQR\) or above \(Q_3 + 1.5 \times IQR\).

Show Code
# Identify extreme values using the IQR rule
ekstrem_wine <- wine_data %>%
  select(where(is.numeric), -quality) %>%
  summarise(across(everything(), ~ {
    q1 <- quantile(.x, 0.25, na.rm = TRUE)
    q3 <- quantile(.x, 0.75, na.rm = TRUE)
    iqr <- q3 - q1

    batas_bawah <- q1 - 1.5 * iqr
    batas_atas <- q3 + 1.5 * iqr

    sum(.x < batas_bawah | .x > batas_atas, na.rm = TRUE)
  })) %>%
  pivot_longer(
    cols = everything(),
    names_to = "variabel",
    values_to = "jumlah_ekstrem"
  )

knitr::kable(
  ekstrem_wine,
  caption = "Number of extreme values based on the IQR rule"
)
Number of extreme values based on the IQR rule
variabel jumlah_ekstrem
fixed acidity 357
volatile acidity 377
citric acid 509
residual sugar 118
chlorides 286
free sulfur dioxide 62
total sulfur dioxide 10
density 3
pH 73
sulphates 191
alcohol 3

Boxplots are used to examine the distribution of physicochemical variables and identify potential extreme values. Points beyond the whiskers represent observations identified as extreme values according to the 1.5 × IQR rule.

Show Code
wine_data %>%
  select(where(is.numeric), -quality) %>%
  pivot_longer(
    cols = everything(),
    names_to = "variabel",
    values_to = "nilai"
  ) %>%
  ggplot(aes(x = "", y = nilai)) +
  geom_boxplot(
    fill = "#2C6E9B",
    color = "#253746",
    outlier.colour = "#D94F4F",
    outlier.size = 1,
    outlier.alpha = 0.7,
    width = 0.5
  ) +
  facet_wrap(~ variabel, scales = "free_y", ncol = 4) +
  labs(
    x = NULL,
    y = "Observed value"
  ) +
  theme_minimal(base_size = 11) +
  theme(
    axis.text.x = element_blank(),
    axis.ticks.x = element_blank(),
    strip.text = element_text(face = "bold")
  )

Boxplots of wine physicochemical variables

Data Preparation

Defining Response Variables

KNN and SVM models are implemented for two types of prediction tasks: classification and regression. For classification, the quality score is grouped into two categories: Not Good for scores below 6 and Good for scores of 6 or higher. For regression, the response variable remains the original numerical quality score.

Show Code
# Create the response variable for classification
wine_data <- wine_data %>%
  mutate(
    quality_class = factor(
      if_else(quality >= 6, "Good", "Not Good"),
      levels = c("Not Good", "Good")
    )
  )

# Examine the distribution of the resulting classes
class_distribution <- wine_data %>%
  count(quality_class, name = "frequency") %>%
  mutate(
    percentage = round(100 * frequency / sum(frequency), 2)
  )

knitr::kable(
  class_distribution,
  caption = "Distribution of wine quality categories"
)
Distribution of wine quality categories
quality_class frequency percentage
Not Good 2384 36.69
Good 4113 63.31
Show Code
# Confirm that the regression response remains numeric
class(wine_data$quality)
[1] "numeric"

Handling Identical Rows

Observations with identical values across all variables are removed using the distinct() function. This step is performed before splitting the dataset to prevent identical rows from appearing in both the training and testing sets.

Show Code
# Remove identical rows across all variables
wine_clean <- wine_data %>%
  distinct()

# Compare the number of observations
duplicate_summary <- tibble(
  stage = c("Before removal", "After removal"),
  observations = c(
    nrow(wine_data),
    nrow(wine_clean)
  )
)

knitr::kable(
  duplicate_summary,
  caption = "Number of observations before and after removing identical rows"
)
Number of observations before and after removing identical rows
stage observations
Before removal 6497
After removal 5320
Show Code
# Examine class distribution after removing identical rows
wine_clean %>%
  count(quality_class, name = "frequency") %>%
  mutate(
    percentage = round(100 * frequency / sum(frequency), 2)
  ) %>%
  knitr::kable(
    caption = "Distribution of wine quality categories after removing identical rows"
  )
Distribution of wine quality categories after removing identical rows
quality_class frequency percentage
Not Good 1988 37.37
Good 3332 62.63

Removing identical rows reduced the dataset from 6,497 to 5,320 observations, representing 1,177 removed rows (18.12%). The proportion of Good observations decreased from 63.31% to 62.63%, while Not Good increased from 36.69% to 37.37%. These relatively small changes indicate that removing identical rows did not substantially alter the class distribution. The cleaned dataset is then used for the training and testing split.

Training and Testing Data Split

The dataset is divided into 80% training data and 20% testing data using stratified random sampling based on quality_class. Stratification helps preserve the relative proportions of both quality categories across the training and testing sets.

Show Code
# Perform stratified data splitting
set.seed(123)

split_wine <- rsample::initial_split(
  wine_clean,
  prop = 0.80,
  strata = quality_class
)

# Create training and testing datasets
wine_train <- rsample::training(split_wine)
wine_test  <- rsample::testing(split_wine)

# Examine class counts and proportions
split_summary <- bind_rows(
  wine_train %>% mutate(dataset = "Training"),
  wine_test %>% mutate(dataset = "Testing")
) %>%
  count(dataset, quality_class, name = "frequency") %>%
  group_by(dataset) %>%
  mutate(
    percentage = round(100 * frequency / sum(frequency), 2)
  ) %>%
  ungroup()

knitr::kable(
  split_summary,
  caption = "Distribution of wine quality categories in training and testing data"
)
Distribution of wine quality categories in training and testing data
dataset quality_class frequency percentage
Testing Not Good 398 37.37
Testing Good 667 62.63
Training Not Good 1590 37.37
Training Good 2665 62.63

Predictor Sourcing and Processing

Next, we need to prepare the predictors for KNN and SVM.

There are two main processes:

  • Categorical encoding: The variable type is converted into a numerical variable using dummy encoding.

  • Standardization: All predictors are standardized using the mean and standard deviation of the training data.

Standardization is necessary because KNN relies on distance metrics, whereas SVM is influenced by the scale of the predictors when forming decision boundaries. Variables with larger ranges can exert a disproportionate influence if the scale of the predictors is not unified.

Standardization uses the formula:

\[z_{ij} = \frac{x_{ij} - \bar{x}_{j,\text{train}}}{s_{j,\text{train}}}\]

where \(\bar{x}_{j,\text{train}}\) and \(s_{j,\text{train}}\) are the mean and standard deviation of the \(j\)-th predictor in the training data, respectively. The standardization parameters are calculated solely from the training data and applied to the test data to prevent data leakage.

Show Code
# Prepare predictors without response variables
train_predictors <- wine_train %>%
  select(-quality, -quality_class) %>%
  mutate(type = factor(type, levels = c("red", "white")))

test_predictors <- wine_test %>%
  select(-quality, -quality_class) %>%
  mutate(type = factor(type, levels = c("red", "white")))

# Convert the categorical variable into a dummy variable
x_train <- model.matrix(~ ., data = train_predictors)[, -1, drop = FALSE]
x_test  <- model.matrix(~ ., data = test_predictors)[, -1, drop = FALSE]

# Calculate standardization parameters from training data
train_mean <- colMeans(x_train)
train_sd <- apply(x_train, 2, sd)

# Check for predictors with zero standard deviation
stopifnot(all(train_sd > 0))

# Standardize training and testing predictors
x_train_scaled <- scale(
  x_train,
  center = train_mean,
  scale = train_sd
)

x_test_scaled <- scale(
  x_test,
  center = train_mean,
  scale = train_sd
)

# Examine the standardization results
scaling_summary <- tibble(
  variable = colnames(x_train),
  original_mean = round(colMeans(x_train), 3),
  original_sd = round(apply(x_train, 2, sd), 3),
  standardized_mean = round(colMeans(x_train_scaled), 3),
  standardized_sd = round(apply(x_train_scaled, 2, sd), 3)
)

knitr::kable(
  scaling_summary,
  caption = "Comparison of predictor scales before and after standardization in training data"
)
Comparison of predictor scales before and after standardization in training data
variable original_mean original_sd standardized_mean standardized_sd
typewhite 0.748 0.434 0 1
fixed acidity 7.218 1.324 0 1
volatile acidity 0.344 0.168 0 1
citric acid 0.319 0.148 0 1
residual sugar 5.074 4.541 0 1
chlorides 0.056 0.036 0 1
free sulfur dioxide 30.181 17.988 0 1
total sulfur dioxide 114.371 56.846 0 1
density 0.995 0.003 0 1
pH 3.223 0.161 0 1
sulphates 0.533 0.149 0 1
alcohol 10.548 1.179 0 1

The standardization results show that all training predictors have a mean of 0 and a standard deviation of 1. Before standardization, predictor scales differed considerably. For example, total sulfur dioxide had a standard deviation of 56.846, whereas density had a standard deviation of only 0.003. Standardization places predictors on comparable scales, preventing variables with larger numerical scales from disproportionately influencing distance calculations in KNN and decision boundaries in SVM. However, standardization does not remove previously identified extreme values.

KNN Classification

K-Nearest Neighbors (KNN) classification is used to predict wine quality categories (Good or Not Good) based on physicochemical characteristics. KNN assigns an observation to a class based on the majority vote among its (k) nearest neighbors.

Selecting an appropriate value of (k) is an important step in model development. A small (k) tends to be sensitive to local variations and extreme values, while a large (k) may reduce the model’s ability to capture local patterns.

In this practical session, the optimal value of (k) is selected using the following procedure:

  1. Evaluate odd values of (k) from 3 to 25 to reduce the possibility of voting ties in binary classification.
  2. Apply 5-fold cross-validation to the training data to assess model performance.
  3. Select the value of (k) that produces the highest mean balanced accuracy.

Balanced accuracy is used because the two wine quality categories have different numbers of observations. This metric calculates the average of sensitivity and specificity, ensuring that the model’s ability to identify both classes is considered.

\[\text{Balanced Accuracy} = \frac{\text{Sensitivity}+\text{Specificity}}{2}\]

Next, KNN models are fitted using each candidate value of (k), and their cross-validation results are compared to determine the optimal number of neighbors.

Show Code
# Define balanced accuracy as an evaluation metric
knn_summary <- function(data, lev = NULL, model = NULL) {
  cm <- caret::confusionMatrix(
    data = data$pred,
    reference = data$obs,
    positive = "Good"
  )
  
  c(
    BalancedAccuracy = unname(cm$byClass["Balanced Accuracy"]),
    Accuracy = unname(cm$overall["Accuracy"])
  )
}

# Set up cross-validation
knn_control <- caret::trainControl(
  method = "cv",
  number = 5,
  summaryFunction = knn_summary
)

# Select the optimal k
set.seed(123)

knn_model <- caret::train(
  x = x_train,
  y = wine_train$quality_class,
  method = "knn",
  tuneGrid = data.frame(k = seq(3, 25, 2)),
  trControl = knn_control,
  preProcess = c("center", "scale"),
  metric = "BalancedAccuracy"
)
Loading required package: lattice

Attaching package: 'caret'
The following object is masked from 'package:kknn':

    contr.dummy
The following objects are masked from 'package:yardstick':

    precision, recall, sensitivity, specificity
The following object is masked from 'package:rsample':

    calibration
The following object is masked from 'package:purrr':

    lift
Show Code
# Display cross-validation results
knn_results <- knn_model$results %>%
  select(k, BalancedAccuracy, BalancedAccuracySD, Accuracy)

knitr::kable(
  knn_results,
  digits = 4,
  caption = "KNN cross-validation results for different k values"
)
KNN cross-validation results for different k values
k BalancedAccuracy BalancedAccuracySD Accuracy
3 0.6867 0.0068 0.7159
5 0.6981 0.0085 0.7286
7 0.7027 0.0113 0.7349
9 0.7056 0.0117 0.7387
11 0.7050 0.0097 0.7373
13 0.7042 0.0134 0.7380
15 0.7057 0.0138 0.7398
17 0.7099 0.0148 0.7438
19 0.7178 0.0159 0.7523
21 0.7113 0.0209 0.7478
23 0.7096 0.0154 0.7462
25 0.7068 0.0162 0.7436

KNN balanced accuracy across different k values

Show Code
# Visualize cross-validation performance
ggplot(knn_results, aes(x = k, y = BalancedAccuracy)) +
  geom_line(color = "#2C6E9B", linewidth = 1) +
  geom_point(color = "#D94F4F", size = 2.5) +
  scale_x_continuous(breaks = seq(3, 25, 2)) +
  labs(
    x = "Number of neighbors (k)",
    y = "Balanced accuracy",
    title = "Selecting k for KNN Classification"
  ) +
  theme_minimal(base_size = 12)

KNN balanced accuracy across different k values
Show Code
# Display the optimal k
knn_model$bestTune

KNN balanced accuracy across different k values

Cross-validation results show that the balanced accuracy of KNN increases from 0.6867 at \(k=3\) until reaching its highest value of 0.7178 at \(k=19\). After that, performance tends to decline to 0.7068 at \(k=25\). This pattern indicates that using more neighbors initially helps produce better classifications, but adding more neighbors does not always improve performance. Based on the tested candidates, \(k=19\) is selected as the best value, with an accuracy of 0.7523 and a standard deviation of balanced accuracy across folds of 0.0159. The performance difference among several \(k\) values around the optimal point is relatively small, suggesting that these results do not imply \(k=19\) is definitely superior for every data sample.

Prediction and Evaluation on the Test Data

The KNN model with the selected \(k\) value is used to predict the wine quality categories on the test data. The predicted results are compared with the actual categories through a confusion matrix. The classification performance is then summarized using accuracy, balanced accuracy, sensitivity, specificity, precision, and F1-score.

  • Accuracy: The proportion of all correct predictions.

  • Balanced accuracy: The average ability of the model to recognize both classes.

  • Sensitivity: The proportion of “Good” wine successfully recognized.

  • Specificity: The proportion of “Not Good” wine successfully recognized.

  • Precision: The proportion of “Good” predictions that actually belong to the “Good” category.

  • F1-score: The harmonic mean of precision and sensitivity.

Show Code
# Predict wine quality categories on the test set
knn_predictions <- predict(
  knn_model,
  newdata = x_test
)

# Construct the confusion matrix
knn_evaluation <- caret::confusionMatrix(
  data = knn_predictions,
  reference = wine_test$quality_class,
  positive = "Good",
  mode = "everything"
)

knitr::kable(
  as.data.frame.matrix(knn_evaluation$table),
  caption = "KNN confusion matrix on the test set"
)
KNN confusion matrix on the test set
Not Good Good
Not Good 225 97
Good 173 570
Show Code
# Summarize classification metrics
knn_metrics <- tibble(
  metric = c(
    "Accuracy",
    "Balanced Accuracy",
    "Sensitivity",
    "Specificity",
    "Precision",
    "F1-score"
  ),
  value = round(c(
    knn_evaluation$overall["Accuracy"],
    knn_evaluation$byClass["Balanced Accuracy"],
    knn_evaluation$byClass["Sensitivity"],
    knn_evaluation$byClass["Specificity"],
    knn_evaluation$byClass["Precision"],
    knn_evaluation$byClass["F1"]
  ), 4)
)

knitr::kable(
  knn_metrics,
  caption = "KNN classification performance on the test set"
)
KNN classification performance on the test set
metric value
Accuracy 0.7465
Balanced Accuracy 0.7099
Sensitivity 0.8546
Specificity 0.5653
Precision 0.7672
F1-score 0.8085
ImportantImportant

Use x_test, not x_test_scaled, when running predict(). Models built via caret::train() already store the standardization process, so the transformation is automatically applied to new predictors. Using data that has already been standardized will cause the standardization to be performed twice.

The KNN model achieved an accuracy of 0.7465 and a balanced accuracy of 0.7099 on the test set. The confusion matrix shows that 570 Good observations and 225 Not Good observations were correctly classified. However, 173 Not Good observations were incorrectly predicted as Good, while 97 Good observations were predicted as Not Good.

The sensitivity of 0.8546 indicates that the model correctly identified 85.46% of Good observations, whereas the specificity of 0.5653 indicates that only 56.53% of Not Good observations were correctly identified. Thus, the model performed better at recognizing Good wines than Not Good wines. The precision of 0.7672 and F1-score of 0.8085 further describe its performance for the Good class. This difference in class-specific performance remains important despite the relatively higher overall accuracy.

KNN Regression

KNN Regression is used to predict numerical wine quality scores (quality) based on physicochemical characteristics. Unlike classification, which relies on majority voting, KNN Regression produces numerical predictions by averaging the response values of the (k) nearest neighbors.

Mathematically, KNN Regression predictions are expressed as:

\[\hat{y}(\mathbf{x}) = \frac{1}{k} \sum_{i \in N_k(\mathbf{x})} y_i\]

where:

  • (()): predicted quality score for observation ().
  • (N_k()): set of the (k) nearest neighbors of ().
  • (y_i): observed quality score of neighbor (i).
  • (k): number of neighbors used for prediction.

The value of (k) is selected using 5-fold cross-validation on the training data. Unlike binary classification, KNN Regression does not require odd values of (k) because predictions are obtained through averaging rather than majority voting.

In this practical session, even values of (k) from 2 to 30 are evaluated using Root Mean Squared Error (RMSE). RMSE measures prediction errors in the same units as the response variable.

\[RMSE = \sqrt{ \frac{1}{n} \sum_{i=1}^{n} (y_i-\hat{y}_i)^2 }\]

A lower RMSE indicates that predictions are closer to the observed values. The value of (k) with the lowest mean cross-validation RMSE is selected as the optimal parameter.

Show Code
# Set up cross-validation
knn_reg_control <- caret::trainControl(
  method = "cv",
  number = 5
)

# Select the optimal k
set.seed(123)

knn_reg_model <- caret::train(
  x = x_train,
  y = wine_train$quality,
  method = "knn",
  tuneGrid = data.frame(k = seq(2, 30, 2)),
  trControl = knn_reg_control,
  preProcess = c("center", "scale"),
  metric = "RMSE"
)

# Display cross-validation results
knn_reg_results <- knn_reg_model$results %>%
  select(k, RMSE, RMSESD, Rsquared, MAE)

knitr::kable(
  knn_reg_results,
  digits = 4,
  caption = "KNN Regression cross-validation results"
)
KNN Regression cross-validation results
k RMSE RMSESD Rsquared MAE
2 0.8133 0.0361 0.2460 0.5978
4 0.7561 0.0369 0.2886 0.5787
6 0.7367 0.0338 0.3091 0.5655
8 0.7302 0.0346 0.3158 0.5650
10 0.7281 0.0350 0.3176 0.5659
12 0.7231 0.0362 0.3257 0.5633
14 0.7229 0.0347 0.3259 0.5633
16 0.7204 0.0361 0.3308 0.5627
18 0.7192 0.0368 0.3333 0.5619
20 0.7192 0.0379 0.3338 0.5621
22 0.7197 0.0375 0.3334 0.5629
24 0.7193 0.0382 0.3346 0.5632
26 0.7206 0.0385 0.3328 0.5647
28 0.7196 0.0389 0.3353 0.5646
30 0.7195 0.0389 0.3357 0.5648

KNN Regression RMSE across different k values

Show Code
# Visualize RMSE across candidate k values
ggplot(knn_reg_results, aes(x = k, y = RMSE)) +
  geom_line(color = "#2C6E9B", linewidth = 1) +
  geom_point(color = "#D94F4F", size = 2.5) +
  scale_x_continuous(breaks = seq(2, 30, 2)) +
  labs(
    x = "Number of neighbors (k)",
    y = "RMSE",
    title = "Selecting k for KNN Regression"
  ) +
  theme_minimal(base_size = 12)

KNN Regression RMSE across different k values
Show Code
# Display the optimal k
knn_reg_model$bestTune

KNN Regression RMSE across different k values

Cross-validation results show that the RMSE of KNN Regression decreases from 0.8133 at \(k=2\) to approximately 0.7192 at \(k=18\)–\(20\). The substantial decline at small \(k\) values indicates that using more neighbors helps reduce prediction errors. However, starting around \(k=18\), the RMSE becomes relatively stable, meaning that adding more neighbors no longer yields meaningful improvements. Based on the parameter selection results, \(k=20\) is selected with an RMSE of 0.7192, MAE of 0.5621, and R-squared of 0.3338. The RMSE values for \(k=18\) and \(k=20\) are identical after rounding to four decimal places; therefore, the advantage of \(k=20\) should be understood as a selection result based on the pre-rounded RMSE value, rather than a major difference in performance.

Prediction and Evaluation on the Test Set

The KNN Regression model with the selected value of (k) is used to predict wine quality scores on the test set. Model performance is evaluated using RMSE, MAE, and Rsquared. The relationship between actual and predicted scores is also visualized using a scatter plot.

The diagonal line represents perfect predictions, where predicted values equal actual values. Observations closer to this line have smaller prediction errors.

Show Code
# Predict quality scores on the test set
knn_reg_predictions <- predict(
  knn_reg_model,
  newdata = x_test
)

# Calculate evaluation metrics
knn_reg_evaluation <- caret::postResample(
  pred = knn_reg_predictions,
  obs = wine_test$quality
)

knn_reg_metrics <- tibble(
  metric = names(knn_reg_evaluation),
  value = round(as.numeric(knn_reg_evaluation), 4)
)

knitr::kable(
  knn_reg_metrics,
  caption = "KNN Regression performance on the test set"
)
KNN Regression performance on the test set
metric value
RMSE 0.7145
Rsquared 0.3521
MAE 0.5598

Comparison of actual and predicted wine quality scores using KNN Regression

Show Code
# Prepare data for visualization
knn_reg_comparison <- tibble(
  actual = wine_test$quality,
  predicted = as.numeric(knn_reg_predictions)
)

# Visualize actual versus predicted values
ggplot(knn_reg_comparison, aes(x = actual, y = predicted)) +
  geom_jitter(
    width = 0.12,
    height = 0,
    color = "#2C6E9B",
    alpha = 0.35,
    size = 1.5
  ) +
  geom_abline(
    slope = 1,
    intercept = 0,
    linetype = "dashed",
    color = "#D94F4F",
    linewidth = 1
  ) +
  scale_x_continuous(breaks = 3:9) +
  labs(
    x = "Actual quality score",
    y = "Predicted quality score",
    title = "Actual vs Predicted Values: KNN Regression"
  ) +
  theme_minimal(base_size = 12)

Comparison of actual and predicted wine quality scores using KNN Regression
NoteMethodological Note

In caret::postResample(), \(R^2\) is calculated as the squared correlation between the actual and predicted values. Therefore, this metric is not always identical to the coefficient of determination based on \(1-\mathrm{SSE}/\mathrm{SST}\).

On the test set, KNN Regression achieved an RMSE of 0.7145 and an MAE of 0.5598. This indicates an average absolute prediction error of approximately 0.56 points on the wine quality scale, while the higher RMSE suggests that some observations have relatively large prediction errors. The Rsquared value of 0.3521 indicates a limited relationship between actual and predicted values.

The scatter plot shows that predictions are concentrated around quality scores of 5 to 7. Observations with low actual scores, particularly 3 and 4, tend to be overestimated, while observations with scores of 7 to 9 tend to be underestimated. This pattern reflects a tendency toward middle-range predictions because KNN Regression averages the response values of nearby observations. Thus, the model has not fully captured variations in wine quality scores, particularly for observations with extreme values.

SVM Classification

Support Vector Machine (SVM) is a classification method that identifies a decision boundary (hyperplane) to separate observations from different classes. SVM aims to maximize the margin, defined as the distance between the decision boundary and the closest observations from each class. The observations that determine this boundary are called support vectors.

In this practical session, SVM with a Radial Basis Function (RBF) kernel is used to allow nonlinear decision boundaries. The RBF kernel is defined as:

\[K(\mathbf{x}_i,\mathbf{x}_j) = \exp\left( -\sigma\|\mathbf{x}_i-\mathbf{x}_j\|^2 \right)\]

where:

  • (K(_i,_j)): similarity between two observations measured using the RBF kernel.
  • (_i) and (_j): predictor vectors of two observations.
  • (): a parameter controlling how rapidly similarity decreases as the distance between observations increases.

SVM performance is influenced by two main parameters, (C) and (). The parameter (C) controls the penalty for classification errors. A larger (C) imposes a stronger penalty, while a smaller (C) allows greater tolerance for classification errors. Meanwhile, a larger () makes the influence of individual observations more localized, potentially producing more complex decision boundaries.

Hyperparameters are selected through grid search using 5-fold cross-validation. Each combination of (C) and () is evaluated using balanced accuracy, and the combination with the highest mean value is selected for the final model.

Show Code
# Define balanced accuracy as an evaluation metric
svm_summary <- function(data, lev = NULL, model = NULL) {
  cm <- caret::confusionMatrix(
    data = data$pred,
    reference = data$obs,
    positive = "Good"
  )

  c(
    BalancedAccuracy = unname(cm$byClass["Balanced Accuracy"]),
    Accuracy = unname(cm$overall["Accuracy"])
  )
}

# Set up cross-validation
svm_control <- caret::trainControl(
  method = "cv",
  number = 5,
  summaryFunction = svm_summary
)

# Define hyperparameter combinations
svm_grid <- expand.grid(
  sigma = c(0.02, 0.08, 0.20),
  C = c(0.1, 1, 10)
)

# Train and select the optimal model
set.seed(123)

svm_model <- caret::train(
  x = x_train,
  y = wine_train$quality_class,
  method = "svmRadial",
  tuneGrid = svm_grid,
  trControl = svm_control,
  preProcess = c("center", "scale"),
  metric = "BalancedAccuracy"
)

# Display cross-validation results
svm_results <- svm_model$results %>%
  select(sigma, C, BalancedAccuracy, BalancedAccuracySD, Accuracy)

knitr::kable(
  svm_results,
  digits = 4,
  caption = "SVM Classification cross-validation results"
)
SVM Classification cross-validation results
sigma C BalancedAccuracy BalancedAccuracySD Accuracy
0.02 0.1 0.6878 0.0152 0.7311
0.02 1.0 0.7203 0.0199 0.7506
0.02 10.0 0.7286 0.0244 0.7572
0.08 0.1 0.7111 0.0149 0.7459
0.08 1.0 0.7418 0.0206 0.7669
0.08 10.0 0.7415 0.0125 0.7645
0.20 0.1 0.6983 0.0153 0.7429
0.20 1.0 0.7451 0.0131 0.7687
0.20 10.0 0.7226 0.0063 0.7481

SVM balanced accuracy across hyperparameter combinations

Show Code
# Visualize hyperparameter combinations
ggplot(
  svm_results,
  aes(
    x = factor(C),
    y = factor(sigma),
    fill = BalancedAccuracy
  )
) +
  geom_tile(color = "white", linewidth = 1) +
  geom_text(
    aes(label = sprintf("%.3f", BalancedAccuracy)),
    size = 4
  ) +
  scale_fill_gradient(
    low = "#D8EAF5",
    high = "#2C6E9B"
  ) +
  labs(
    x = "Cost parameter (C)",
    y = "Sigma parameter",
    fill = "Balanced accuracy",
    title = "Selecting SVM Classification Hyperparameters"
  ) +
  theme_minimal(base_size = 12)

SVM balanced accuracy across hyperparameter combinations
Show Code
# Display the optimal hyperparameters
svm_model$bestTune

SVM balanced accuracy across hyperparameter combinations

Cross-validation results show that the combination of (C=1) and () achieved the highest balanced accuracy of 0.7451, with an accuracy of 0.7687. At (C=0.1), model performance was generally lower across all evaluated () values. Increasing (C) to 1 improved classification performance, but a further increase to 10 did not consistently produce better results.

The effect of () also depended on the value of (C). For example, at (C=1), balanced accuracy increased from 0.7203 to 0.7451 as () increased from 0.02 to 0.20. However, at (C=10), using () resulted in lower performance. This highlights the importance of selecting both parameters jointly. Nevertheless, performance differences among several leading combinations were relatively small compared with variation across folds. Therefore, the selected combination is the best among those evaluated, rather than a guarantee of global optimality.

Prediction and Evaluation on the Test Set

The SVM model with (C=1) and () is used to predict wine quality categories on the test set. Predicted categories are compared with actual categories using a confusion matrix. Classification performance is then evaluated using accuracy, balanced accuracy, sensitivity, specificity, precision, and F1-score.

Show Code
# Predict wine quality categories on the test set
svm_predictions <- predict(
  svm_model,
  newdata = x_test
)

# Construct the confusion matrix
svm_evaluation <- caret::confusionMatrix(
  data = svm_predictions,
  reference = wine_test$quality_class,
  positive = "Good",
  mode = "everything"
)

knitr::kable(
  as.data.frame.matrix(svm_evaluation$table),
  caption = "SVM confusion matrix on the test set"
)
SVM confusion matrix on the test set
Not Good Good
Not Good 257 108
Good 141 559
Show Code
# Summarize classification metrics
svm_metrics <- tibble(
  metric = c(
    "Accuracy",
    "Balanced Accuracy",
    "Sensitivity",
    "Specificity",
    "Precision",
    "F1-score"
  ),
  value = round(c(
    svm_evaluation$overall["Accuracy"],
    svm_evaluation$byClass["Balanced Accuracy"],
    svm_evaluation$byClass["Sensitivity"],
    svm_evaluation$byClass["Specificity"],
    svm_evaluation$byClass["Precision"],
    svm_evaluation$byClass["F1"]
  ), 4)
)

knitr::kable(
  svm_metrics,
  caption = "SVM Classification performance on the test set"
)
SVM Classification performance on the test set
metric value
Accuracy 0.7662
Balanced Accuracy 0.7419
Sensitivity 0.8381
Specificity 0.6457
Precision 0.7986
F1-score 0.8178

On the test set, SVM achieved an accuracy of 0.7662 and a balanced accuracy of 0.7419. The confusion matrix shows that 559 Good observations and 257 Not Good observations were correctly classified. Meanwhile, 141 Not Good observations were incorrectly predicted as Good, and 108 Good observations were predicted as Not Good.

The sensitivity of 0.8381 indicates that the model correctly identified 83.81% of Good observations, while the specificity of 0.6457 indicates that 64.57% of Not Good observations were correctly identified. Thus, SVM still performed better at recognizing the Good class, but the difference in class-specific performance was smaller than that observed for KNN Classification. The precision of 0.7986 and F1-score of 0.8178 also indicate reasonably good predictive performance for the Good class.

Support Vector Regression

Support Vector Regression (SVR) extends SVM to predict numerical response variables. In this practical session, SVR is used to predict wine quality scores (quality) based on physicochemical characteristics.

Unlike regression approaches that directly penalize all prediction errors, SVR uses the concept of epsilon-insensitive loss. Under this approach, prediction errors within a tolerance threshold () are not penalized.

The loss function is expressed as:

\[L_{\epsilon}(y_i,\hat{y}_i) = \max\left(0,\lvert y_i-\hat{y}_i\rvert-\epsilon\right)\]

where:

  • (y_i): actual quality score of observation (i).
  • (_i): predicted quality score.
  • (): prediction error tolerance.
  • (L_{}): the magnitude of the penalized prediction error.

Thus, SVR aims to construct a prediction function that balances model complexity and errors exceeding the specified tolerance.

In this practical session, SVR with a Radial Basis Function (RBF) kernel is implemented. Model performance is influenced by (C), which controls the penalty for errors outside the tolerance threshold, and (), which determines the characteristics of the RBF kernel.

Hyperparameters are selected through grid search using 5-fold cross-validation. Each parameter combination is evaluated using RMSE, and the combination with the lowest mean RMSE is selected. The parameter () is fixed at 0.1 throughout model training.

Show Code
# Set up cross-validation
svr_control <- caret::trainControl(
  method = "cv",
  number = 5
)

# Define hyperparameter combinations
svr_grid <- expand.grid(
  sigma = c(0.02, 0.08, 0.20),
  C = c(0.1, 1, 10)
)

# Train and select the optimal SVR model
set.seed(123)

svr_model <- caret::train(
  x = x_train,
  y = wine_train$quality,
  method = "svmRadial",
  tuneGrid = svr_grid,
  trControl = svr_control,
  preProcess = c("center", "scale"),
  metric = "RMSE",
  epsilon = 0.1
)

# Display cross-validation results
svr_results <- svr_model$results %>%
  select(sigma, C, RMSE, RMSESD, Rsquared, MAE)

knitr::kable(
  svr_results,
  digits = 4,
  caption = "Support Vector Regression cross-validation results"
)
Support Vector Regression cross-validation results
sigma C RMSE RMSESD Rsquared MAE
0.02 0.1 0.7252 0.0440 0.3247 0.5600
0.02 1.0 0.7080 0.0447 0.3524 0.5451
0.02 10.0 0.7022 0.0459 0.3645 0.5378
0.08 0.1 0.7125 0.0424 0.3455 0.5465
0.08 1.0 0.6969 0.0402 0.3728 0.5293
0.08 10.0 0.7125 0.0314 0.3550 0.5415
0.20 0.1 0.7193 0.0373 0.3359 0.5475
0.20 1.0 0.7052 0.0335 0.3588 0.5332
0.20 10.0 0.7676 0.0303 0.2946 0.5888

SVR RMSE across hyperparameter combinations

Show Code
# Visualize hyperparameter combinations
ggplot(
  svr_results,
  aes(
    x = factor(C),
    y = factor(sigma),
    fill = RMSE
  )
) +
  geom_tile(color = "white", linewidth = 1) +
  geom_text(
    aes(label = sprintf("%.3f", RMSE)),
    size = 4
  ) +
  scale_fill_gradient(
    low = "#BFDDEB",
    high = "#F19C79"
  ) +
  labs(
    x = "Cost parameter (C)",
    y = "Sigma parameter",
    fill = "RMSE",
    title = "Selecting Support Vector Regression Hyperparameters"
  ) +
  theme_minimal(base_size = 12)

SVR RMSE across hyperparameter combinations
Show Code
# Display the optimal hyperparameters
svr_model$bestTune

SVR RMSE across hyperparameter combinations

Cross-validation results show that the combination of (C=1) and () achieved the lowest RMSE of 0.6969, with an MAE of 0.5293 and Rsquared of 0.3728. At (C=1), RMSE decreased as () increased from 0.02 to 0.08, but increased again at (). Meanwhile, the combination of (C=10) and () produced the highest RMSE of 0.7676.

This pattern indicates that the effects of (C) and () are interrelated. Stronger penalties for prediction errors and more localized kernel effects do not necessarily improve predictive performance. Among the evaluated combinations, (C=1) and () were selected for the final SVR model, which is subsequently evaluated using the test set.

Prediction and Evaluation on the Test Set

The SVR model with (C=1) and () is used to predict wine quality scores on the test set. Prediction performance is evaluated using RMSE, MAE, and Rsquared.

The relationship between actual and predicted values is also visualized using a scatter plot. The diagonal line represents perfect predictions, so the distance between observations and this line provides a visual indication of prediction errors.

Show Code
# Predict quality scores on the test set
svr_predictions <- predict(
  svr_model,
  newdata = x_test
)

# Calculate evaluation metrics
svr_evaluation <- caret::postResample(
  pred = svr_predictions,
  obs = wine_test$quality
)

svr_metrics <- tibble(
  metric = names(svr_evaluation),
  value = round(as.numeric(svr_evaluation), 4)
)

knitr::kable(
  svr_metrics,
  caption = "Support Vector Regression performance on the test set"
)
Support Vector Regression performance on the test set
metric value
RMSE 0.6864
Rsquared 0.3974
MAE 0.5221

Comparison of actual and predicted wine quality scores using SVR

Show Code
# Prepare data for visualization
svr_comparison <- tibble(
  actual = wine_test$quality,
  predicted = as.numeric(svr_predictions)
)

# Visualize actual versus predicted values
ggplot(svr_comparison, aes(x = actual, y = predicted)) +
  geom_jitter(
    width = 0.12,
    height = 0,
    color = "#2C6E9B",
    alpha = 0.35,
    size = 1.5
  ) +
  geom_abline(
    slope = 1,
    intercept = 0,
    linetype = "dashed",
    color = "#D94F4F",
    linewidth = 1
  ) +
  scale_x_continuous(breaks = 3:9) +
  labs(
    x = "Actual quality score",
    y = "Predicted quality score",
    title = "Actual vs Predicted Values: SVR"
  ) +
  theme_minimal(base_size = 12)

Comparison of actual and predicted wine quality scores using SVR

On the test set, SVR achieved an RMSE of 0.6864 and an MAE of 0.5221. The MAE indicates an average absolute prediction error of approximately 0.52 wine quality points. Meanwhile, the Rsquared value of 0.3974 indicates a relationship between actual and predicted scores, although the model’s ability to represent variations in quality scores remains limited.

The scatter plot shows that low quality scores, particularly 3 and 4, tend to be overestimated, while scores of 8 and 9 tend to be underestimated. Most predictions remain concentrated around scores of 5 to 7. Thus, SVR has not fully captured extreme quality scores accurately, but it produces smaller prediction errors than KNN Regression on the same test set.

Model Evaluation and Comparison

Classification Model Comparison

KNN Classification and SVM Classification are compared using evaluation results from the same test set. The metrics include accuracy, balanced accuracy, sensitivity, specificity, precision, and F1-score.

Balanced accuracy is the primary comparison metric because it considers the model’s ability to recognize both quality categories. Sensitivity and specificity are also compared to examine differences in identifying Good and Not Good wines.

Show Code
# Combine evaluation metrics
classification_comparison <- bind_rows(
  knn_metrics %>% mutate(model = "KNN"),
  svm_metrics %>% mutate(model = "SVM")
)

# Display the comparison table
classification_comparison %>%
  pivot_wider(
    names_from = metric,
    values_from = value
  ) %>%
  knitr::kable(
    digits = 4,
    caption = "Classification model performance comparison on the test set"
  )
Classification model performance comparison on the test set
model Accuracy Balanced Accuracy Sensitivity Specificity Precision F1-score
KNN 0.7465 0.7099 0.8546 0.5653 0.7672 0.8085
SVM 0.7662 0.7419 0.8381 0.6457 0.7986 0.8178

Performance comparison of KNN and SVM Classification

Show Code
# Select key metrics for visualization
classification_plot <- classification_comparison %>%
  filter(
    metric %in% c(
      "Balanced Accuracy",
      "Sensitivity",
      "Specificity"
    )
  )

# Visualize model performance
ggplot(
  classification_plot,
  aes(x = metric, y = value, fill = model)
) +
  geom_col(
    position = position_dodge(width = 0.75),
    width = 0.7
  ) +
  scale_fill_manual(
    values = c(
      "KNN" = "#2C6E9B",
      "SVM" = "#D94F4F"
    )
  ) +
  scale_y_continuous(
    limits = c(0, 1),
    breaks = seq(0, 1, 0.2)
  ) +
  labs(
    x = "Evaluation metric",
    y = "Score",
    fill = "Model",
    title = "KNN vs SVM Classification"
  ) +
  theme_minimal(base_size = 12)

Performance comparison of KNN and SVM Classification

The test-set comparison shows that SVM achieved a balanced accuracy of 0.7419, exceeding KNN’s 0.7099. SVM also produced higher accuracy, precision, and F1-score. Its main advantage was observed in specificity, which increased from 0.5653 for KNN to 0.6457 for SVM. This indicates that SVM was more effective at identifying Not Good wines.

Conversely, KNN achieved slightly higher sensitivity, at 0.8546 compared with 0.8381 for SVM, meaning that it correctly identified more Good observations. Considering performance across both classes, SVM demonstrated better classification performance on this test set. However, this comparison is descriptive and based on a single data split, so it does not establish that SVM would consistently outperform KNN across other splits.

Regression Model Comparison

KNN Regression and Support Vector Regression (SVR) are compared using predictions from the same test set. Model performance is evaluated based on RMSE, MAE, and Rsquared.

RMSE and MAE measure the magnitude of prediction errors, while Rsquared represents the strength of the relationship between actual and predicted values. The visualization focuses on RMSE and MAE because both metrics are expressed in wine quality score units and can be directly compared.

Show Code
# Combine regression evaluation results
regression_comparison <- bind_rows(
  knn_reg_metrics %>% mutate(model = "KNN Regression"),
  svr_metrics %>% mutate(model = "SVR")
)

# Display the comparison table
regression_comparison %>%
  pivot_wider(
    names_from = metric,
    values_from = value
  ) %>%
  knitr::kable(
    digits = 4,
    caption = "Regression model performance comparison on the test set"
  )
Regression model performance comparison on the test set
model RMSE Rsquared MAE
KNN Regression 0.7145 0.3521 0.5598
SVR 0.6864 0.3974 0.5221

Prediction error comparison between KNN Regression and SVR

Show Code
# Select error metrics for visualization
regression_plot <- regression_comparison %>%
  filter(metric %in% c("RMSE", "MAE"))

# Visualize RMSE and MAE comparison
ggplot(
  regression_plot,
  aes(x = metric, y = value, fill = model)
) +
  geom_col(
    position = position_dodge(width = 0.75),
    width = 0.7
  ) +
  geom_text(
    aes(label = sprintf("%.4f", value)),
    position = position_dodge(width = 0.75),
    vjust = -0.5,
    size = 4
  ) +
  scale_fill_manual(
    values = c(
      "KNN Regression" = "#2C6E9B",
      "SVR" = "#D94F4F"
    )
  ) +
  scale_y_continuous(
    limits = c(0, 0.85),
    breaks = seq(0, 0.8, 0.2)
  ) +
  labs(
    x = "Evaluation metric",
    y = "Prediction error",
    fill = "Model",
    title = "KNN Regression vs SVR"
  ) +
  theme_minimal(base_size = 12)

Prediction error comparison between KNN Regression and SVR

The comparison shows that SVR achieved an RMSE of 0.6864, lower than KNN Regression’s 0.7145. SVR also produced a lower MAE of 0.5221 compared with 0.5598 for KNN Regression. Meanwhile, SVR achieved a higher Rsquared value of 0.3974 compared with 0.3521 for KNN Regression. These three metrics indicate that SVR provided better predictions on the test set.

Nevertheless, both models still tended to produce predictions concentrated around middle-range quality scores. Observations with very low or high quality scores were not predicted accurately. Thus, SVR was the better regression model among the two evaluated methods, although it remained limited in capturing variations in wine quality, particularly at extreme values.

Implementation Summary

The application of KNN and SVM to the wine quality dataset demonstrates that model selection should be aligned with the prediction task. In classification, SVM achieved higher balanced accuracy than KNN and performed better at identifying Not Good wines, although KNN showed slightly higher sensitivity for the Good category. In regression, SVR achieved lower RMSE and MAE than KNN Regression, but both approaches remained limited in predicting extreme quality scores.

These findings also highlight the importance of predictor standardization, hyperparameter selection through cross-validation, and evaluation on a separate test set. Model performance should not be judged using a single metric, but should account for the characteristics of the response variable and the types of prediction errors. Overall, SVM with an RBF kernel performed better on both tasks in this evaluation, although these results should not be generalized without further assessment across different data splits or datasets.

References

Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018

Cortez, P., Cerdeira, A., Almeida, F., Matos, T., & Reis, J. (2009a). Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems, 47(4), 547–553. https://doi.org/10.1016/j.dss.2009.05.016

Cortez, P., Cerdeira, A., Almeida, F., Matos, T., & Reis, J. (2009b). Wine Quality [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C56S3T

Cover, T. M., & Hart, P. E. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21–27. https://doi.org/10.1109/TIT.1967.1053964

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning: With applications in R (2nd ed.). Springer. https://doi.org/10.1007/978-1-0716-1418-1

Kuhn, M. (2008). Building predictive models in R using the caret package. Journal of Statistical Software, 28(5), 1–26. https://doi.org/10.18637/jss.v028.i05

Smola, A. J., & Schölkopf, B. (2004). A tutorial on support vector regression. Statistics and Computing, 14(3), 199–222. https://doi.org/10.1023/B:STCO.0000035301.49549.88

Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. Springer. https://doi.org/10.1007/978-3-319-24277-4