Show Code
# Install the required packages
install.packages(c(
"tidyverse",
"tidymodels",
"kknn",
"e1071",
"knitr"
))In supervised learning, the relationship between predictors and the response can be learned through different approaches. Some algorithms generate predictions based on the proximity of an observation to other observations, whereas others construct a decision function through an optimization process.
K-Nearest Neighbors (KNN) is a distance-based learning method. This algorithm uses a number of training observations closest to a new observation to determine its prediction. In contrast, Support Vector Machine (SVM) applies optimization principles to construct a decision function based on the characteristics of the training data.
In classification, SVM identifies a decision boundary that separates classes while considering the margin. The extension of SVM to numerical responses is known as Support Vector Regression (SVR), which constructs a regression function with a specified tolerance for prediction errors.
The fundamental differences between these approaches are summarized in the following table.
| Aspect | K-Nearest Neighbors | Support Vector Machine |
|---|---|---|
| Learning principle | Proximity between observations | Optimization of a decision function |
| Basis of prediction | Nearest observations | Optimized decision function |
| Classification | Voting among neighboring classes | Class determination using a decision boundary |
| Regression | Mean response of neighboring observations | Regression function with error tolerance |
| Model characteristics | Nonparametric and observation-based | Margin optimization and regularization |
| Main parameters | Number of neighbors and distance measure | Cost, kernel, and kernel parameters |
Both KNN and SVM can be used for classification and regression. However, their mathematical formulations and prediction mechanisms differ.
Suppose a dataset contains \(n\) observations and \(p\) predictors, represented as:
\[\mathcal{D} = \left\{ (\mathbf{x}_i,y_i) \right\}_{i=1}^{n}\]
where:
For classification, \(y_i\) represents a class category. For regression, \(y_i\) represents a numerical response.
The objective of modeling is to construct a prediction rule for a new observation \(\mathbf{x}_0\) based on the relationship learned from the training data.
K-Nearest Neighbors is a nonparametric supervised learning method that uses proximity between observations as the basis for prediction. This algorithm does not assume a particular functional relationship between predictors and the response, such as a linear or quadratic relationship.
KNN works by calculating the distance between a new observation and each observation in the training data. After all distances have been calculated, the algorithm selects the \(K\) observations with the smallest distances. The responses of these nearest observations are then used to generate a prediction.
In general, the KNN procedure consists of the following steps:
Figure reference: James et al. (2021), An Introduction to Statistical Learning, Figure 2.14.
Description: Illustration of selecting the three nearest observations to determine the class of a new observation.
Consider the black cross in the left panel.
This point represents a new observation whose class is unknown. The circle indicates the region containing its three nearest observations.
Inside the circle, there are:
Because the majority of its nearest neighbors belong to the blue class, KNN assigns the new observation to the blue class.
The right panel shows the decision boundary obtained when the same procedure is applied to different possible locations of new observations.
Important point: The KNN decision boundary is formed by applying the nearest-neighbor rule across different locations in the predictor space. Unlike a linear classification model, KNN does not first estimate a single equation for a separating line.
Neighbor selection is the main component of KNN. Therefore, the distance measure determines which observations are considered closest to a new observation.
The Euclidean distance between observations \(\mathbf{x}_i\) and \(\mathbf{x}_j\) is defined as:
\[d(\mathbf{x}_i,\mathbf{x}_j) = \sqrt{ \sum_{r=1}^{p} (x_{ir}-x_{jr})^2 }\]
For two predictors, the formula becomes:
\[d(\mathbf{x}_i,\mathbf{x}_j) = \sqrt{ (x_{i1}-x_{j1})^2+ (x_{i2}-x_{j2})^2 }\]
For example, suppose:
\[\mathbf{x}_1=(2,3)\]
and:
\[\mathbf{x}_2=(5,7)\]
The distance between these two observations is:
\[\begin{aligned} d(\mathbf{x}_1,\mathbf{x}_2) &= \sqrt{(2-5)^2+(3-7)^2}\\ &= \sqrt{9+16}\\ &=5 \end{aligned}\]
A smaller distance indicates that the two observations are closer to each other in the predictor space.
Manhattan distance calculates the sum of the absolute differences across all predictors:
\[d(\mathbf{x}_i,\mathbf{x}_j) = \sum_{r=1}^{p} |x_{ir}-x_{jr}|\]
Using the same example:
\[\begin{aligned} d(\mathbf{x}_1,\mathbf{x}_2) &= |2-5|+|3-7|\\ &=3+4\\ &=7 \end{aligned}\]
Euclidean distance and Manhattan distance are special cases of Minkowski distance:
\[d_q(\mathbf{x}_i,\mathbf{x}_j) = \left( \sum_{r=1}^{p} |x_{ir}-x_{jr}|^q \right)^{1/q}\]
where \(q\geq1\).
When \(q=1\), the formula corresponds to Manhattan distance. When \(q=2\), it corresponds to Euclidean distance.
Different distance measures may produce different rankings of nearest neighbors and, consequently, affect KNN predictions.
In classification, KNN uses the most frequently occurring class among the \(K\) nearest neighbors.
Let \(\mathcal{N}_K(\mathbf{x}_0)\) denote the set of indices corresponding to the \(K\) nearest training observations to a new observation \(\mathbf{x}_0\).
The proportion of neighboring observations belonging to class \(g\) is defined as:
\[\widehat{P}(Y=g\mid\mathbf{X}=\mathbf{x}_0) = \frac{1}{K} \sum_{i\in\mathcal{N}_K(\mathbf{x}_0)} I(y_i=g)\]
where the indicator function is:
\[I(y_i=g) = \begin{cases} 1, & \text{if } y_i=g\\ 0, & \text{if } y_i\neq g \end{cases}\]
The predicted class is determined by the largest class proportion:
\[\boxed{ \hat y_0 = \arg\max_g \widehat{P}(Y=g\mid\mathbf{X}=\mathbf{x}_0) }\]
For example, suppose \(K=5\), with three neighbors belonging to class A and two neighbors belonging to class B.
Then:
\[\widehat P(Y=A\mid\mathbf{x}_0) = \frac{3}{5} = 0.6\]
\[\widehat P(Y=B\mid\mathbf{x}_0) = \frac{2}{5} = 0.4\]
Therefore, the new observation is classified as class A.
These proportions represent estimated class probabilities based on the composition of the selected neighbors. However, they are not automatically calibrated probabilities.
In regression, KNN uses the mean response value of the \(K\) nearest neighbors.
The prediction formula is:
\[\boxed{ \hat y_0 = \frac{1}{K} \sum_{i\in\mathcal{N}_K(\mathbf{x}_0)} y_i }\]
Suppose the three nearest observations have the following response values:
\[y_1=120,\qquad y_2=150,\qquad y_3=180\]
With \(K=3\), the predicted response is:
\[\begin{aligned} \hat y_0 &= \frac{120+150+180}{3}\\ &=150 \end{aligned}\]
The main difference between KNN classification and regression lies in how the neighboring responses are combined. Classification uses class voting, whereas regression uses the mean of numerical responses.
In standard KNN, every selected neighbor contributes equally to the prediction. However, observations that are closer to the new observation can be assigned larger weights.
For regression, the weighted prediction is defined as:
\[\hat y_0 = \frac{ \sum_{i\in\mathcal{N}_K(\mathbf{x}_0)} w_i(\mathbf{x}_0)y_i }{ \sum_{i\in\mathcal{N}_K(\mathbf{x}_0)} w_i(\mathbf{x}_0) }\]
where \(w_i(\mathbf{x}_0)\geq0\) is the weight assigned to the \(i\)-th neighbor, with a positive total weight.
For weighted classification, class contributions are calculated using the sum of the corresponding weights:
\[\hat y_0 = \arg\max_g \sum_{i\in\mathcal{N}_K(\mathbf{x}_0)} w_i(\mathbf{x}_0)I(y_i=g)\]
Weighting allows observations closer to the new observation to have a greater influence on the prediction.
The number of neighbors, \(K\), determines the flexibility of the KNN model.
Figure reference: James et al. (2021), An Introduction to Statistical Learning, Figure 2.16.
Description: Comparison of the flexibility of KNN decision boundaries for different numbers of neighbors.
In the left panel, \(K=1\) produces a highly irregular decision boundary. Each prediction depends on only one nearest observation, so a small change in the location of a new observation may change its predicted class.
In the right panel, \(K=100\) produces a smoother and simpler decision boundary. Predictions are influenced by more observations, so smaller local patterns may be overlooked.
A small value of \(K\) produces a more flexible model because predictions are strongly influenced by local observations. However, this also increases the model’s sensitivity to variations in the data.
Conversely, a large value of \(K\) produces smoother predictions because more observations are considered.
In general:
This relationship illustrates the bias–variance trade-off.
Figure reference: James et al. (2021), An Introduction to Statistical Learning, Figure 2.17.
Description: Relationship between KNN flexibility and prediction errors in training and test data.
In Figure 2.17, the horizontal axis uses \(1/K\). Therefore, moving to the right corresponds to a smaller number of neighbors and a more flexible model.
Notice that training error generally decreases as model flexibility increases. However, test error does not necessarily follow the same pattern.
When a model becomes too flexible, training error may become very small, but its predictive performance on new observations may not improve. This situation is associated with the risk of overfitting.
The key lesson: The value of \(K\) should not be selected solely based on how well the model fits the training data. It should be selected using a validation procedure.
There are four important limitations.
First, the curse of dimensionality. As the number of dimensions increases, observations tend to become more dispersed in the predictor space. The relatively nearest observation may no longer be particularly close in geometric terms.
Second, sensitivity to predictor scales. Predictors with larger numerical ranges may dominate distance calculations.
Third, sensitivity to class distributions. In classification with imbalanced classes, observations from the majority class may appear more frequently among the selected neighbors.
Fourth, computational cost during prediction. KNN must identify nearest neighbors for each new observation. For large datasets, this search may require substantial computation, although more efficient search strategies are available.
Support Vector Machine is a supervised learning method that constructs a decision function through an optimization process.
In binary classification, SVM seeks to construct a hyperplane that separates two classes with the largest possible margin. The margin describes the distance between the separating hyperplane and the nearest observations that determine the separation.
Observations that play a role in determining the margin are called support vectors.
Figure reference: James et al. (2021), An Introduction to Statistical Learning, Figure 9.3.
Description: Optimal separating hyperplane, margin boundaries, and observations acting as support vectors.
In the illustration:
Support vectors are important because they determine the position of the optimal decision boundary. In the maximum-margin setting, changing the positions of support vectors may change the hyperplane, whereas small changes to other observations that remain outside the margin do not necessarily affect it.
For \(p\) predictors, the linear decision function is defined as:
\[f(\mathbf{x}) = \mathbf{w}^{\top}\mathbf{x}+b\]
where:
The separating hyperplane is defined as:
\[\boxed{ \mathbf{w}^{\top}\mathbf{x}+b=0 }\]
For two predictors, the equation becomes:
\[w_1x_1+w_2x_2+b=0\]
In binary classification, response classes are represented as:
\[y_i\in\{-1,+1\}\]
The prediction rule is:
\[\boxed{ \hat y = \operatorname{sign} \left( \mathbf{w}^{\top}\mathbf{x}+b \right) }\]
A positive decision-function value corresponds to class \(+1\), whereas a negative value corresponds to class \(-1\).
Hard-margin SVM is used when the two classes can be linearly separated without classification errors.
The canonical margin boundaries are defined as:
\[\mathbf{w}^{\top}\mathbf{x}+b=1\]
and:
\[\mathbf{w}^{\top}\mathbf{x}+b=-1\]
The distance between these two boundaries is:
\[M = \frac{2}{\|\mathbf{w}\|}\]
where:
\[\|\mathbf{w}\| = \sqrt{ \sum_{j=1}^{p}w_j^2 }\]
Maximizing the margin is equivalent to minimizing the squared norm of the weight vector.
The hard-margin SVM optimization problem is:
\[\boxed{ \min_{\mathbf{w},b} \frac{1}{2}\|\mathbf{w}\|^2 }\]
subject to:
\[y_i \left( \mathbf{w}^{\top}\mathbf{x}_i+b \right) \geq1, \qquad i=1,\ldots,n\]
These constraints ensure that every observation lies on the appropriate side of the separating hyperplane and does not violate the margin boundaries.
In empirical data, classes often overlap, making perfect linear separation impossible.
Soft-margin SVM introduces slack variables \(\xi_i\) to allow violations of the margin.
The optimization problem becomes:
\[\boxed{ \min_{\mathbf{w},b,\boldsymbol{\xi}} \left[ \frac{1}{2}\|\mathbf{w}\|^2 + C\sum_{i=1}^{n}\xi_i \right] }\]
subject to:
\[y_i \left( \mathbf{w}^{\top}\mathbf{x}_i+b \right) \geq1-\xi_i\]
and:
\[\xi_i\geq0\]
The parameter \(C>0\) controls the penalty imposed on margin violations.
Thus, the choice of \(C\) affects the balance between the complexity of the decision function and classification errors.
A linear hyperplane cannot always represent the separation between classes when the underlying pattern is nonlinear.
SVM can use kernel functions to construct nonlinear decision boundaries. Conceptually, predictors are mapped into a feature space using a function \(\phi(\cdot)\).
A kernel represents the inner product in that feature space:
\[K(\mathbf{x}_i,\mathbf{x}_j) = \phi(\mathbf{x}_i)^\top \phi(\mathbf{x}_j)\]
This approach makes it possible to calculate relationships between observations in the feature space without explicitly constructing all coordinates resulting from the mapping.
Several commonly used kernels are described below.
\[K(\mathbf{x}_i,\mathbf{x}_j) = \mathbf{x}_i^\top\mathbf{x}_j\]
\[K(\mathbf{x}_i,\mathbf{x}_j) = \left( \gamma\mathbf{x}_i^\top\mathbf{x}_j+r \right)^d\]
Here, \(\gamma\) controls the scaling of the inner product, \(r\) is an offset parameter, and \(d\) is the polynomial degree.
\[K(\mathbf{x}_i,\mathbf{x}_j) = \exp \left( -\gamma \|\mathbf{x}_i-\mathbf{x}_j\|^2 \right)\]
where \(\gamma>0\) controls the characteristics of the RBF kernel.
For kernel-based SVM, the binary classification decision function can be written as:
\[f(\mathbf{x}) = \sum_{i\in SV} \alpha_i y_i K(\mathbf{x}_i,\mathbf{x}) +b\]
where \(SV\) denotes the set of support-vector indices and \(\alpha_i\) represents the nonzero dual coefficients associated with the support vectors.
The predicted class is obtained using:
\[\hat y = \operatorname{sign} \left( f(\mathbf{x}) \right)\]
Thus, kernels allow nonlinear decision boundaries to be constructed, while the underlying principle of selecting a decision function remains based on margin optimization.
The basic SVM formulation addresses binary classification. For responses containing more than two classes, multiple binary classifiers can be combined using strategies such as one-versus-one or one-versus-rest.
Support Vector Regression extends support vector principles to numerical responses.
Unlike SVM classification, which constructs a boundary between classes, SVR seeks a regression function while controlling model complexity and prediction errors.
SVR introduces a parameter \(\varepsilon\) that defines a tolerance region around the regression function.
Suppose a linear regression function is defined as:
\[f(\mathbf{x}) = \mathbf{w}^{\top}\mathbf{x}+b\]
The tolerance region is defined by:
\[f(\mathbf{x})-\varepsilon \leq y \leq f(\mathbf{x})+\varepsilon\]
This region is known as the epsilon-insensitive tube, or \(\varepsilon\)-tube.
Figure reference: Smola and Schölkopf (2004), A Tutorial on Support Vector Regression, Figure 1.
Description: Tolerance region around a regression function, showing observations located inside and outside the epsilon-tube.
In the illustration, the regression function lies at the center of a tolerance region bounded by:
\[f(\mathbf{x})+\varepsilon\]
and:
\[f(\mathbf{x})-\varepsilon\]
Observations located inside or exactly on the boundaries of the tube receive no penalty under the epsilon-insensitive loss function. In contrast, observations outside the tube incur a penalty according to the magnitude of their violations.
The SVR loss function is defined as:
\[\boxed{ L_{\varepsilon} \left( y_i,f(\mathbf{x}_i) \right) = \max \left( 0, |y_i-f(\mathbf{x}_i)|-\varepsilon \right) }\]
Suppose the predicted value is 100 and the parameter is \(\varepsilon=5\).
| Actual Value | Absolute Error | Epsilon-Insensitive Loss |
|---|---|---|
| 102 | 2 | 0 |
| 105 | 5 | 0 |
| 108 | 8 | 3 |
| 112 | 12 | 7 |
This example shows that SVR does not penalize every prediction error. A penalty is incurred only when the absolute error exceeds \(\varepsilon\).
To accommodate observations located outside the epsilon-tube, SVR introduces slack variables \(\xi_i\) and \(\xi_i^*\).
The epsilon-SVR optimization problem is defined as:
\[\boxed{ \min_{ \mathbf{w},b, \boldsymbol{\xi}, \boldsymbol{\xi}^{*} } \left[ \frac{1}{2}\|\mathbf{w}\|^2 + C\sum_{i=1}^{n} (\xi_i+\xi_i^*) \right] }\]
subject to:
\[y_i-f(\mathbf{x}_i) \leq \varepsilon+\xi_i\]
\[f(\mathbf{x}_i)-y_i \leq \varepsilon+\xi_i^*\]
\[\xi_i,\xi_i^*\geq0\]
In this formulation:
Observations located outside the tube may act as support vectors. Observations on the tube boundaries may also become support vectors, whereas observations strictly inside the tube generally have no dual-coefficient contribution to the prediction function.
SVR can use kernel functions to represent nonlinear relationships.
Its prediction function is defined as:
\[\boxed{ f(\mathbf{x}) = \sum_{i\in SV} (\alpha_i-\alpha_i^*) K(\mathbf{x}_i,\mathbf{x}) +b }\]
where \(\alpha_i\) and \(\alpha_i^*\) are dual coefficients from the SVR optimization formulation.
The kernel can be linear, polynomial, or RBF.
Therefore, SVM classification and SVR share related optimization principles and kernel-based representations. However, they differ in their loss functions and prediction objectives.
Both KNN and SVM can be affected by predictor scales.
In KNN, distance measures depend on differences between predictor values. Predictors with larger numerical ranges may dominate Euclidean distance calculations.
In SVM, differences in scale affect the geometry of the feature space, margin optimization, and the calculation of certain kernel functions.
Standardization is one approach used to adjust predictor scales:
\[z_{ij} = \frac{x_{ij}-\bar{x}_j}{s_j}\]
where \(\bar{x}_j\) and \(s_j\) are the mean and standard deviation of the \(j\)-th predictor, respectively, calculated from the training data.
KNN and SVM generally require numerical representations to calculate distances or kernel functions.
Nominal categorical predictors can be represented using indicator encoding, for example:
\[D_k = I(X=c_k)\]
where \(c_k\) denotes the \(k\)-th category.
However, categorical encoding also affects the geometric representation of the data. In KNN, changing the encoding may alter distances between observations. In SVM, it may affect the resulting decision function.
Therefore, predictor representations should be selected according to the characteristics of the data and the principles of the algorithm.
In distance-based methods, increasing the number of predictors may cause observations to become relatively farther apart in the feature space.
This phenomenon is known as the curse of dimensionality.
As the number of predictors increases, the difference between distances to the nearest and farthest observations may become less informative. Consequently, the local proximity concept used in KNN may lose some of its usefulness.
In SVM, adding features can also affect model complexity, although the effects depend on the kernel and regularization being used.
All transformations requiring parameter estimation must be based on the training data. Information from the test data must not be used to determine transformation parameters or construct the model.
This principle is necessary to ensure that predictive evaluation reflects the model’s performance on observations that were not used during learning.
KNN and SVM are two supervised learning approaches with different mechanisms.
KNN uses local information based on distances between observations. In classification, predictions are obtained through voting among neighboring classes, whereas regression uses the mean response of the nearest neighbors.
SVM constructs a decision function based on optimization principles. For classification, this process involves class separation and margin optimization. For regression, SVR uses an epsilon-tolerance region to determine prediction-error penalties.
| Method | Response Type | Prediction Principle | Main Parameters |
|---|---|---|---|
| KNN Classification | Categorical | Voting among nearest neighbors | \(K\), distance, weights |
| KNN Regression | Numerical | Mean response of nearest neighbors | \(K\), distance, weights |
| Support Vector Classification | Categorical | Margin-based decision function | \(C\), kernel, kernel parameters |
| Support Vector Regression | Numerical | Regression function with epsilon-insensitive loss | \(C\), \(\varepsilon\), kernel, kernel parameters |
These differences in learning mechanisms mean that method selection should consider the response type, predictor characteristics, the form of the relationship to be learned, and model complexity.
This practical session uses several R packages for data processing, the application of K-Nearest Neighbors (KNN) and Support Vector Machine (SVM) algorithms, and the evaluation of prediction results.
The packages are selected according to the requirements of each analytical stage, from data preparation to model performance assessment.
| Package | Purpose in This Practical Session | R Functions Used |
|---|---|---|
tidyverse |
Processes data, selects and transforms variables, produces summaries, and creates visualizations of data and modeling results. | select(), mutate(), filter(), ggplot(), geom_point(), geom_line() |
tidymodels |
Splits training and test data, prepares predictors, performs standardization, and evaluates classification and regression results. | initial_split(), training(), testing(), recipe(), step_normalize(), accuracy(), roc_auc(), rmse(), mae() |
kknn |
Applies KNN algorithms for classification and regression. This package allows the specification of the number of neighbors, distance measure, and neighbor weighting. | kknn(), predict() |
e1071 |
Applies SVM for classification and Support Vector Regression (SVR) for regression, including linear, polynomial, and radial kernels. | svm(), predict() |
knitr |
Displays analysis results as well-formatted tables in a Quarto document. | kable() |
The tidymodels package is a collection of packages for statistical modeling and machine learning. It includes several packages used in this practical session:
rsample for data splitting and resampling.recipes for predictor preparation and transformation.yardstick for calculating evaluation metrics.parsnip for defining model specifications.workflows for combining preprocessing and modeling stages.Therefore, these packages do not need to be installed separately when tidymodels has been successfully installed.
Packages only need to be installed once on the computer being used. If the required packages are already available, the installation step can be skipped.
# Install the required packages
install.packages(c(
"tidyverse",
"tidymodels",
"kknn",
"e1071",
"knitr"
))The installation code above is not executed automatically when the Quarto document is rendered. If necessary, run the installation commands directly in the R Console.
After installation, the packages are loaded using the library() function so that the required functions are available throughout the practical session.
# Packages for data processing and visualization
library(tidyverse)── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.2.1 ✔ readr 2.2.0
✔ forcats 1.0.1 ✔ stringr 1.6.0
✔ ggplot2 4.0.3 ✔ tibble 3.3.1
✔ lubridate 1.9.5 ✔ tidyr 1.3.2
✔ purrr 1.2.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
# Packages for data preparation and model evaluation
library(tidymodels)── Attaching packages ────────────────────────────────────── tidymodels 1.5.0 ──
✔ broom 1.0.13 ✔ rsample 1.3.2
✔ dials 1.4.4 ✔ tailor 0.1.0
✔ infer 1.1.0 ✔ tune 2.1.0
✔ modeldata 1.6.0 ✔ workflows 1.3.0
✔ parsnip 1.6.1 ✔ workflowsets 1.1.1
✔ recipes 1.4.0 ✔ yardstick 1.4.0
── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ──
✖ scales::discard() masks purrr::discard()
✖ dplyr::filter() masks stats::filter()
✖ recipes::fixed() masks stringr::fixed()
✖ dplyr::lag() masks stats::lag()
✖ yardstick::spec() masks readr::spec()
✖ recipes::step() masks stats::step()
# Package for KNN modeling
library(kknn)
# Package for SVM and SVR modeling
library(e1071)
Attaching package: 'e1071'
The following object is masked from 'package:tune':
tune
The following object is masked from 'package:rsample':
permutations
The following object is masked from 'package:parsnip':
tune
The following object is masked from 'package:ggplot2':
element
# Package for presenting tables
library(knitr)Package usage depends on the method being applied. KNN and SVM can be used for both classification and regression, but their response types and prediction outputs differ.
| Method | Modeling Task | Package | Main Function | Prediction Output |
|---|---|---|---|---|
| KNN Classification | Classification | kknn |
kknn() |
Class category |
| KNN Regression | Regression | kknn |
kknn() |
Numerical value |
| Support Vector Classification | Classification | e1071 |
svm() |
Class category |
| Support Vector Regression | Regression | e1071 |
svm() |
Numerical value |
In KNN, the predicted class or numerical value is determined using training observations closest to the new observation.
Meanwhile, SVM classification constructs a decision boundary based on maximum-margin principles, whereas SVR uses a loss function that tolerates a specified difference between actual and predicted values.
Although both SVM classification and regression can be performed using the svm() function, their optimization formulations are different. These differences have been explained in the corresponding method sections.
All preprocessing procedures that require parameter estimation, such as standardization using means and standard deviations, must be performed based on the training data.
The resulting parameters are then applied to the test data. # H. Implementation of KNN and SVM
The dataset used in this practical session is Wine Quality, which contains physicochemical characteristics and quality assessments of Portuguese Vinho Verde wines. The dataset was introduced by Cortez et al. (2009) to investigate the relationship between wine physicochemical properties and quality ratings obtained through sensory evaluation.
The dataset includes two types of wine: red wine and white wine. Physicochemical measurements serve as predictor variables, while the quality score (quality) serves as the response variable. These characteristics allow the dataset to be examined using both classification and regression approaches.
Data source: UCI Machine Learning Repository – Wine Quality.
Before fitting the models, the data structure must be examined to determine the number of observations, number of variables, data types, and the values stored in each column.
The initial inspection begins by importing the CSV file and displaying the first few observations.
# Import the dataset
wine_data <- read_csv("wine-quality-white-and-red.csv",
show_col_types = FALSE)
# Display the first six observations
head(wine_data)Next, the dataset dimensions, variable names, and structure are examined using dim(), names(), and glimpse().
# Check the dataset dimensions
dim(wine_data)[1] 6497 13
# Display the variable names
names(wine_data) [1] "type" "fixed acidity" "volatile acidity"
[4] "citric acid" "residual sugar" "chlorides"
[7] "free sulfur dioxide" "total sulfur dioxide" "density"
[10] "pH" "sulphates" "alcohol"
[13] "quality"
# Examine the data structure
glimpse(wine_data)Rows: 6,497
Columns: 13
$ type <chr> "white", "white", "white", "white", "white", "w…
$ `fixed acidity` <dbl> 7.0, 6.3, 8.1, 7.2, 7.2, 8.1, 6.2, 7.0, 6.3, 8.…
$ `volatile acidity` <dbl> 0.27, 0.30, 0.28, 0.23, 0.23, 0.28, 0.32, 0.27,…
$ `citric acid` <dbl> 0.36, 0.34, 0.40, 0.32, 0.32, 0.40, 0.16, 0.36,…
$ `residual sugar` <dbl> 20.70, 1.60, 6.90, 8.50, 8.50, 6.90, 7.00, 20.7…
$ chlorides <dbl> 0.045, 0.049, 0.050, 0.058, 0.058, 0.050, 0.045…
$ `free sulfur dioxide` <dbl> 45, 14, 30, 47, 47, 30, 30, 45, 14, 28, 11, 17,…
$ `total sulfur dioxide` <dbl> 170, 132, 97, 186, 186, 97, 136, 170, 132, 129,…
$ density <dbl> 1.0010, 0.9940, 0.9951, 0.9956, 0.9956, 0.9951,…
$ pH <dbl> 3.00, 3.30, 3.26, 3.19, 3.19, 3.26, 3.18, 3.00,…
$ sulphates <dbl> 0.45, 0.49, 0.44, 0.40, 0.40, 0.44, 0.47, 0.45,…
$ alcohol <dbl> 8.8, 9.5, 10.1, 9.9, 9.9, 10.1, 9.6, 8.8, 9.5, …
$ quality <dbl> 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 5, 5, 5, 7, 5, 7,…
The dim() function provides the number of rows and columns, while names() displays the names of all variables. The glimpse() function shows the data types and sample values for each variable, allowing numerical and categorical variables to be identified.
The data structure inspection shows that the dataset contains 6,497 observations and 13 variables. The type variable is stored as character data (chr) and identifies the wine type, while the eleven physicochemical variables are stored as numerical data (dbl). The quality variable is also stored numerically, but its values represent ordered quality ratings and are therefore ordinal in nature.
Several variable names still contain spaces, including fixed acidity, volatile acidity, and residual sugar. This naming convention needs to be considered when referencing variables in R. The variable names will be adjusted during data preparation to simplify their use in modeling.
Descriptive statistics provide an initial overview of the numerical predictors. The examination includes the minimum, first quartile, median, mean, standard deviation, third quartile, and maximum.
These measures of central tendency and dispersion help identify differences in numerical ranges across variables and provide indications of asymmetric distributions. This information is relevant because predictor representation and scaling can influence the modeling process in KNN and SVM.
Descriptive statistics are calculated for the eleven physicochemical variables. The quality variable will be examined separately through the distribution of quality scores.
# Descriptive statistics for numerical predictors
ringkasan_numerik <- wine_data %>%
select(-type, -quality) %>%
pivot_longer(
cols = everything(),
names_to = "variabel",
values_to = "nilai"
) %>%
group_by(variabel) %>%
summarise(
minimum = min(nilai, na.rm = TRUE),
kuartil_1 = quantile(nilai, 0.25, na.rm = TRUE),
median = median(nilai, na.rm = TRUE),
rata_rata = mean(nilai, na.rm = TRUE),
simpangan_baku = sd(nilai, na.rm = TRUE),
kuartil_3 = quantile(nilai, 0.75, na.rm = TRUE),
maksimum = max(nilai, na.rm = TRUE),
.groups = "drop"
)
knitr::kable(
ringkasan_numerik,
digits = 4,
caption = "Descriptive statistics of physicochemical variables"
)| variabel | minimum | kuartil_1 | median | rata_rata | simpangan_baku | kuartil_3 | maksimum |
|---|---|---|---|---|---|---|---|
| alcohol | 8.0000 | 9.5000 | 10.3000 | 10.4918 | 1.1927 | 11.300 | 14.900 |
| chlorides | 0.0090 | 0.0380 | 0.0470 | 0.0560 | 0.0350 | 0.065 | 0.611 |
| citric acid | 0.0000 | 0.2500 | 0.3100 | 0.3186 | 0.1453 | 0.390 | 1.660 |
| density | 0.9871 | 0.9923 | 0.9949 | 0.9947 | 0.0030 | 0.997 | 1.039 |
| fixed acidity | 3.8000 | 6.4000 | 7.0000 | 7.2153 | 1.2964 | 7.700 | 15.900 |
| free sulfur dioxide | 1.0000 | 17.0000 | 29.0000 | 30.5253 | 17.7494 | 41.000 | 289.000 |
| pH | 2.7200 | 3.1100 | 3.2100 | 3.2185 | 0.1608 | 3.320 | 4.010 |
| residual sugar | 0.6000 | 1.8000 | 3.0000 | 5.4432 | 4.7578 | 8.100 | 65.800 |
| sulphates | 0.2200 | 0.4300 | 0.5100 | 0.5313 | 0.1488 | 0.600 | 2.000 |
| total sulfur dioxide | 6.0000 | 77.0000 | 118.0000 | 115.7446 | 56.5219 | 156.000 | 440.000 |
| volatile acidity | 0.0800 | 0.2300 | 0.2900 | 0.3397 | 0.1646 | 0.400 | 1.580 |
The descriptive statistics reveal substantial differences in scale across the physicochemical variables. For example, total sulfur dioxide ranges from 6 to 440, whereas density ranges only from 0.9871 to 1.0390. These differences are important because predictors with larger scales may dominate distance calculations in KNN and distance-based kernels in SVM. Predictor standardization is therefore needed to prevent measurement units from disproportionately influencing the calculated distances.
Several variables also show indications of right-skewed distributions. For residual sugar, the mean of 5.4432 exceeds the median of 3.0000, while the maximum value of 65.8000 is substantially higher than the third quartile of 8.1000. Relatively high maximum values are also observed for chlorides and free sulfur dioxide. These patterns indicate the need to examine extreme observations, although such values should not automatically be considered measurement errors.
The quality variable represents an ordered wine quality rating, ranging from lower to higher quality levels. Its distribution is examined to determine the frequency of each score and assess whether observations are evenly distributed across quality levels.
# Calculate the distribution of quality scores
distribusi_quality <- wine_data %>%
count(quality) %>%
mutate(persentase = round(n / sum(n) * 100, 2))
knitr::kable(
distribusi_quality,
col.names = c("Quality Score", "Frequency", "Percentage (%)"),
caption = "Distribution of wine quality scores"
)| Quality Score | Frequency | Percentage (%) |
|---|---|---|
| 3 | 30 | 0.46 |
| 4 | 216 | 3.32 |
| 5 | 2138 | 32.91 |
| 6 | 2836 | 43.65 |
| 7 | 1079 | 16.61 |
| 8 | 193 | 2.97 |
| 9 | 5 | 0.08 |
A bar chart is used to illustrate differences in frequency across quality levels.
# Visualize the distribution of quality scores
ggplot(distribusi_quality, aes(x = factor(quality), y = n)) +
geom_col(fill = "#27689B", width = 0.7) +
labs(
title = "Distribution of Wine Quality Scores",
x = "Quality Score",
y = "Number of Observations"
) +
theme_minimal()The distribution of quality scores is concentrated at scores of 5 (32.91%) and 6 (43.65%), which together account for 76.56% of observations. In contrast, scores of 3 and 9 represent only 0.54% of the dataset. This concentration indicates that moderate quality levels are more strongly represented than very low or high quality levels. In regression, the limited number of observations at both ends of the score range should be considered because predictive accuracy for rare scores may differ from that for more common scores.
When the scores are grouped into Good (score ≥ 6) and Not Good (score < 6), the class proportions are approximately 63.31% and 36.69%, respectively. This difference should be considered when evaluating classification performance. Model performance should therefore be assessed using metrics such as balanced accuracy and F1-score in addition to overall accuracy.
Data quality assessment is performed to identify missing values, identical rows, and numerical observations that differ substantially from most of the data. The findings provide a basis for determining the required preprocessing steps before modeling.
# Count missing values in each variable
missing_wine <- wine_data %>%
summarise(across(everything(), ~ sum(is.na(.x)))) %>%
pivot_longer(
cols = everything(),
names_to = "variabel",
values_to = "jumlah_missing"
)
knitr::kable(
missing_wine,
caption = "Number of missing values by variable"
)| variabel | jumlah_missing |
|---|---|
| type | 0 |
| fixed acidity | 0 |
| volatile acidity | 0 |
| citric acid | 0 |
| residual sugar | 0 |
| chlorides | 0 |
| free sulfur dioxide | 0 |
| total sulfur dioxide | 0 |
| density | 0 |
| pH | 0 |
| sulphates | 0 |
| alcohol | 0 |
| quality | 0 |
In addition to missing values, rows containing identical values across all variables are examined.
# Check for identical rows
duplikat_wine <- tibble(
jumlah_observasi = nrow(wine_data),
baris_identik = sum(duplicated(wine_data)),
baris_unik = nrow(distinct(wine_data))
)
knitr::kable(
duplikat_wine,
caption = "Assessment of identical rows"
)| jumlah_observasi | baris_identik | baris_unik |
|---|---|---|
| 6497 | 1177 | 5320 |
Extreme values in the numerical predictors are examined using the IQR rule. Observations are flagged when their values fall below \(Q_1 - 1.5 \times IQR\) or above \(Q_3 + 1.5 \times IQR\).
# Identify extreme values using the IQR rule
ekstrem_wine <- wine_data %>%
select(where(is.numeric), -quality) %>%
summarise(across(everything(), ~ {
q1 <- quantile(.x, 0.25, na.rm = TRUE)
q3 <- quantile(.x, 0.75, na.rm = TRUE)
iqr <- q3 - q1
batas_bawah <- q1 - 1.5 * iqr
batas_atas <- q3 + 1.5 * iqr
sum(.x < batas_bawah | .x > batas_atas, na.rm = TRUE)
})) %>%
pivot_longer(
cols = everything(),
names_to = "variabel",
values_to = "jumlah_ekstrem"
)
knitr::kable(
ekstrem_wine,
caption = "Number of extreme values based on the IQR rule"
)| variabel | jumlah_ekstrem |
|---|---|
| fixed acidity | 357 |
| volatile acidity | 377 |
| citric acid | 509 |
| residual sugar | 118 |
| chlorides | 286 |
| free sulfur dioxide | 62 |
| total sulfur dioxide | 10 |
| density | 3 |
| pH | 73 |
| sulphates | 191 |
| alcohol | 3 |
Boxplots are used to examine the distribution of physicochemical variables and identify potential extreme values. Points beyond the whiskers represent observations identified as extreme values according to the 1.5 × IQR rule.
wine_data %>%
select(where(is.numeric), -quality) %>%
pivot_longer(
cols = everything(),
names_to = "variabel",
values_to = "nilai"
) %>%
ggplot(aes(x = "", y = nilai)) +
geom_boxplot(
fill = "#2C6E9B",
color = "#253746",
outlier.colour = "#D94F4F",
outlier.size = 1,
outlier.alpha = 0.7,
width = 0.5
) +
facet_wrap(~ variabel, scales = "free_y", ncol = 4) +
labs(
x = NULL,
y = "Observed value"
) +
theme_minimal(base_size = 11) +
theme(
axis.text.x = element_blank(),
axis.ticks.x = element_blank(),
strip.text = element_text(face = "bold")
)KNN and SVM models are implemented for two types of prediction tasks: classification and regression. For classification, the quality score is grouped into two categories: Not Good for scores below 6 and Good for scores of 6 or higher. For regression, the response variable remains the original numerical quality score.
# Create the response variable for classification
wine_data <- wine_data %>%
mutate(
quality_class = factor(
if_else(quality >= 6, "Good", "Not Good"),
levels = c("Not Good", "Good")
)
)
# Examine the distribution of the resulting classes
class_distribution <- wine_data %>%
count(quality_class, name = "frequency") %>%
mutate(
percentage = round(100 * frequency / sum(frequency), 2)
)
knitr::kable(
class_distribution,
caption = "Distribution of wine quality categories"
)| quality_class | frequency | percentage |
|---|---|---|
| Not Good | 2384 | 36.69 |
| Good | 4113 | 63.31 |
# Confirm that the regression response remains numeric
class(wine_data$quality)[1] "numeric"
Observations with identical values across all variables are removed using the distinct() function. This step is performed before splitting the dataset to prevent identical rows from appearing in both the training and testing sets.
# Remove identical rows across all variables
wine_clean <- wine_data %>%
distinct()
# Compare the number of observations
duplicate_summary <- tibble(
stage = c("Before removal", "After removal"),
observations = c(
nrow(wine_data),
nrow(wine_clean)
)
)
knitr::kable(
duplicate_summary,
caption = "Number of observations before and after removing identical rows"
)| stage | observations |
|---|---|
| Before removal | 6497 |
| After removal | 5320 |
# Examine class distribution after removing identical rows
wine_clean %>%
count(quality_class, name = "frequency") %>%
mutate(
percentage = round(100 * frequency / sum(frequency), 2)
) %>%
knitr::kable(
caption = "Distribution of wine quality categories after removing identical rows"
)| quality_class | frequency | percentage |
|---|---|---|
| Not Good | 1988 | 37.37 |
| Good | 3332 | 62.63 |
Removing identical rows reduced the dataset from 6,497 to 5,320 observations, representing 1,177 removed rows (18.12%). The proportion of Good observations decreased from 63.31% to 62.63%, while Not Good increased from 36.69% to 37.37%. These relatively small changes indicate that removing identical rows did not substantially alter the class distribution. The cleaned dataset is then used for the training and testing split.
The dataset is divided into 80% training data and 20% testing data using stratified random sampling based on quality_class. Stratification helps preserve the relative proportions of both quality categories across the training and testing sets.
# Perform stratified data splitting
set.seed(123)
split_wine <- rsample::initial_split(
wine_clean,
prop = 0.80,
strata = quality_class
)
# Create training and testing datasets
wine_train <- rsample::training(split_wine)
wine_test <- rsample::testing(split_wine)
# Examine class counts and proportions
split_summary <- bind_rows(
wine_train %>% mutate(dataset = "Training"),
wine_test %>% mutate(dataset = "Testing")
) %>%
count(dataset, quality_class, name = "frequency") %>%
group_by(dataset) %>%
mutate(
percentage = round(100 * frequency / sum(frequency), 2)
) %>%
ungroup()
knitr::kable(
split_summary,
caption = "Distribution of wine quality categories in training and testing data"
)| dataset | quality_class | frequency | percentage |
|---|---|---|---|
| Testing | Not Good | 398 | 37.37 |
| Testing | Good | 667 | 62.63 |
| Training | Not Good | 1590 | 37.37 |
| Training | Good | 2665 | 62.63 |
Next, we need to prepare the predictors for KNN and SVM.
There are two main processes:
Categorical encoding: The variable type is converted into a numerical variable using dummy encoding.
Standardization: All predictors are standardized using the mean and standard deviation of the training data.
Standardization is necessary because KNN relies on distance metrics, whereas SVM is influenced by the scale of the predictors when forming decision boundaries. Variables with larger ranges can exert a disproportionate influence if the scale of the predictors is not unified.
Standardization uses the formula:
\[z_{ij} = \frac{x_{ij} - \bar{x}_{j,\text{train}}}{s_{j,\text{train}}}\]
where \(\bar{x}_{j,\text{train}}\) and \(s_{j,\text{train}}\) are the mean and standard deviation of the \(j\)-th predictor in the training data, respectively. The standardization parameters are calculated solely from the training data and applied to the test data to prevent data leakage.
# Prepare predictors without response variables
train_predictors <- wine_train %>%
select(-quality, -quality_class) %>%
mutate(type = factor(type, levels = c("red", "white")))
test_predictors <- wine_test %>%
select(-quality, -quality_class) %>%
mutate(type = factor(type, levels = c("red", "white")))
# Convert the categorical variable into a dummy variable
x_train <- model.matrix(~ ., data = train_predictors)[, -1, drop = FALSE]
x_test <- model.matrix(~ ., data = test_predictors)[, -1, drop = FALSE]
# Calculate standardization parameters from training data
train_mean <- colMeans(x_train)
train_sd <- apply(x_train, 2, sd)
# Check for predictors with zero standard deviation
stopifnot(all(train_sd > 0))
# Standardize training and testing predictors
x_train_scaled <- scale(
x_train,
center = train_mean,
scale = train_sd
)
x_test_scaled <- scale(
x_test,
center = train_mean,
scale = train_sd
)
# Examine the standardization results
scaling_summary <- tibble(
variable = colnames(x_train),
original_mean = round(colMeans(x_train), 3),
original_sd = round(apply(x_train, 2, sd), 3),
standardized_mean = round(colMeans(x_train_scaled), 3),
standardized_sd = round(apply(x_train_scaled, 2, sd), 3)
)
knitr::kable(
scaling_summary,
caption = "Comparison of predictor scales before and after standardization in training data"
)| variable | original_mean | original_sd | standardized_mean | standardized_sd |
|---|---|---|---|---|
| typewhite | 0.748 | 0.434 | 0 | 1 |
fixed acidity |
7.218 | 1.324 | 0 | 1 |
volatile acidity |
0.344 | 0.168 | 0 | 1 |
citric acid |
0.319 | 0.148 | 0 | 1 |
residual sugar |
5.074 | 4.541 | 0 | 1 |
| chlorides | 0.056 | 0.036 | 0 | 1 |
free sulfur dioxide |
30.181 | 17.988 | 0 | 1 |
total sulfur dioxide |
114.371 | 56.846 | 0 | 1 |
| density | 0.995 | 0.003 | 0 | 1 |
| pH | 3.223 | 0.161 | 0 | 1 |
| sulphates | 0.533 | 0.149 | 0 | 1 |
| alcohol | 10.548 | 1.179 | 0 | 1 |
The standardization results show that all training predictors have a mean of 0 and a standard deviation of 1. Before standardization, predictor scales differed considerably. For example, total sulfur dioxide had a standard deviation of 56.846, whereas density had a standard deviation of only 0.003. Standardization places predictors on comparable scales, preventing variables with larger numerical scales from disproportionately influencing distance calculations in KNN and decision boundaries in SVM. However, standardization does not remove previously identified extreme values.
K-Nearest Neighbors (KNN) classification is used to predict wine quality categories (Good or Not Good) based on physicochemical characteristics. KNN assigns an observation to a class based on the majority vote among its (k) nearest neighbors.
Selecting an appropriate value of (k) is an important step in model development. A small (k) tends to be sensitive to local variations and extreme values, while a large (k) may reduce the model’s ability to capture local patterns.
In this practical session, the optimal value of (k) is selected using the following procedure:
Balanced accuracy is used because the two wine quality categories have different numbers of observations. This metric calculates the average of sensitivity and specificity, ensuring that the model’s ability to identify both classes is considered.
\[\text{Balanced Accuracy} = \frac{\text{Sensitivity}+\text{Specificity}}{2}\]
Next, KNN models are fitted using each candidate value of (k), and their cross-validation results are compared to determine the optimal number of neighbors.
# Define balanced accuracy as an evaluation metric
knn_summary <- function(data, lev = NULL, model = NULL) {
cm <- caret::confusionMatrix(
data = data$pred,
reference = data$obs,
positive = "Good"
)
c(
BalancedAccuracy = unname(cm$byClass["Balanced Accuracy"]),
Accuracy = unname(cm$overall["Accuracy"])
)
}
# Set up cross-validation
knn_control <- caret::trainControl(
method = "cv",
number = 5,
summaryFunction = knn_summary
)
# Select the optimal k
set.seed(123)
knn_model <- caret::train(
x = x_train,
y = wine_train$quality_class,
method = "knn",
tuneGrid = data.frame(k = seq(3, 25, 2)),
trControl = knn_control,
preProcess = c("center", "scale"),
metric = "BalancedAccuracy"
)Loading required package: lattice
Attaching package: 'caret'
The following object is masked from 'package:kknn':
contr.dummy
The following objects are masked from 'package:yardstick':
precision, recall, sensitivity, specificity
The following object is masked from 'package:rsample':
calibration
The following object is masked from 'package:purrr':
lift
# Display cross-validation results
knn_results <- knn_model$results %>%
select(k, BalancedAccuracy, BalancedAccuracySD, Accuracy)
knitr::kable(
knn_results,
digits = 4,
caption = "KNN cross-validation results for different k values"
)| k | BalancedAccuracy | BalancedAccuracySD | Accuracy |
|---|---|---|---|
| 3 | 0.6867 | 0.0068 | 0.7159 |
| 5 | 0.6981 | 0.0085 | 0.7286 |
| 7 | 0.7027 | 0.0113 | 0.7349 |
| 9 | 0.7056 | 0.0117 | 0.7387 |
| 11 | 0.7050 | 0.0097 | 0.7373 |
| 13 | 0.7042 | 0.0134 | 0.7380 |
| 15 | 0.7057 | 0.0138 | 0.7398 |
| 17 | 0.7099 | 0.0148 | 0.7438 |
| 19 | 0.7178 | 0.0159 | 0.7523 |
| 21 | 0.7113 | 0.0209 | 0.7478 |
| 23 | 0.7096 | 0.0154 | 0.7462 |
| 25 | 0.7068 | 0.0162 | 0.7436 |
KNN balanced accuracy across different k values
# Visualize cross-validation performance
ggplot(knn_results, aes(x = k, y = BalancedAccuracy)) +
geom_line(color = "#2C6E9B", linewidth = 1) +
geom_point(color = "#D94F4F", size = 2.5) +
scale_x_continuous(breaks = seq(3, 25, 2)) +
labs(
x = "Number of neighbors (k)",
y = "Balanced accuracy",
title = "Selecting k for KNN Classification"
) +
theme_minimal(base_size = 12)# Display the optimal k
knn_model$bestTuneKNN balanced accuracy across different k values
Cross-validation results show that the balanced accuracy of KNN increases from 0.6867 at \(k=3\) until reaching its highest value of 0.7178 at \(k=19\). After that, performance tends to decline to 0.7068 at \(k=25\). This pattern indicates that using more neighbors initially helps produce better classifications, but adding more neighbors does not always improve performance. Based on the tested candidates, \(k=19\) is selected as the best value, with an accuracy of 0.7523 and a standard deviation of balanced accuracy across folds of 0.0159. The performance difference among several \(k\) values around the optimal point is relatively small, suggesting that these results do not imply \(k=19\) is definitely superior for every data sample.
The KNN model with the selected \(k\) value is used to predict the wine quality categories on the test data. The predicted results are compared with the actual categories through a confusion matrix. The classification performance is then summarized using accuracy, balanced accuracy, sensitivity, specificity, precision, and F1-score.
Accuracy: The proportion of all correct predictions.
Balanced accuracy: The average ability of the model to recognize both classes.
Sensitivity: The proportion of “Good” wine successfully recognized.
Specificity: The proportion of “Not Good” wine successfully recognized.
Precision: The proportion of “Good” predictions that actually belong to the “Good” category.
F1-score: The harmonic mean of precision and sensitivity.
# Predict wine quality categories on the test set
knn_predictions <- predict(
knn_model,
newdata = x_test
)
# Construct the confusion matrix
knn_evaluation <- caret::confusionMatrix(
data = knn_predictions,
reference = wine_test$quality_class,
positive = "Good",
mode = "everything"
)
knitr::kable(
as.data.frame.matrix(knn_evaluation$table),
caption = "KNN confusion matrix on the test set"
)| Not Good | Good | |
|---|---|---|
| Not Good | 225 | 97 |
| Good | 173 | 570 |
# Summarize classification metrics
knn_metrics <- tibble(
metric = c(
"Accuracy",
"Balanced Accuracy",
"Sensitivity",
"Specificity",
"Precision",
"F1-score"
),
value = round(c(
knn_evaluation$overall["Accuracy"],
knn_evaluation$byClass["Balanced Accuracy"],
knn_evaluation$byClass["Sensitivity"],
knn_evaluation$byClass["Specificity"],
knn_evaluation$byClass["Precision"],
knn_evaluation$byClass["F1"]
), 4)
)
knitr::kable(
knn_metrics,
caption = "KNN classification performance on the test set"
)| metric | value |
|---|---|
| Accuracy | 0.7465 |
| Balanced Accuracy | 0.7099 |
| Sensitivity | 0.8546 |
| Specificity | 0.5653 |
| Precision | 0.7672 |
| F1-score | 0.8085 |
Use x_test, not x_test_scaled, when running predict(). Models built via caret::train() already store the standardization process, so the transformation is automatically applied to new predictors. Using data that has already been standardized will cause the standardization to be performed twice.
The KNN model achieved an accuracy of 0.7465 and a balanced accuracy of 0.7099 on the test set. The confusion matrix shows that 570 Good observations and 225 Not Good observations were correctly classified. However, 173 Not Good observations were incorrectly predicted as Good, while 97 Good observations were predicted as Not Good.
The sensitivity of 0.8546 indicates that the model correctly identified 85.46% of Good observations, whereas the specificity of 0.5653 indicates that only 56.53% of Not Good observations were correctly identified. Thus, the model performed better at recognizing Good wines than Not Good wines. The precision of 0.7672 and F1-score of 0.8085 further describe its performance for the Good class. This difference in class-specific performance remains important despite the relatively higher overall accuracy.
KNN Regression is used to predict numerical wine quality scores (quality) based on physicochemical characteristics. Unlike classification, which relies on majority voting, KNN Regression produces numerical predictions by averaging the response values of the (k) nearest neighbors.
Mathematically, KNN Regression predictions are expressed as:
\[\hat{y}(\mathbf{x}) = \frac{1}{k} \sum_{i \in N_k(\mathbf{x})} y_i\]
where:
The value of (k) is selected using 5-fold cross-validation on the training data. Unlike binary classification, KNN Regression does not require odd values of (k) because predictions are obtained through averaging rather than majority voting.
In this practical session, even values of (k) from 2 to 30 are evaluated using Root Mean Squared Error (RMSE). RMSE measures prediction errors in the same units as the response variable.
\[RMSE = \sqrt{ \frac{1}{n} \sum_{i=1}^{n} (y_i-\hat{y}_i)^2 }\]
A lower RMSE indicates that predictions are closer to the observed values. The value of (k) with the lowest mean cross-validation RMSE is selected as the optimal parameter.
# Set up cross-validation
knn_reg_control <- caret::trainControl(
method = "cv",
number = 5
)
# Select the optimal k
set.seed(123)
knn_reg_model <- caret::train(
x = x_train,
y = wine_train$quality,
method = "knn",
tuneGrid = data.frame(k = seq(2, 30, 2)),
trControl = knn_reg_control,
preProcess = c("center", "scale"),
metric = "RMSE"
)
# Display cross-validation results
knn_reg_results <- knn_reg_model$results %>%
select(k, RMSE, RMSESD, Rsquared, MAE)
knitr::kable(
knn_reg_results,
digits = 4,
caption = "KNN Regression cross-validation results"
)| k | RMSE | RMSESD | Rsquared | MAE |
|---|---|---|---|---|
| 2 | 0.8133 | 0.0361 | 0.2460 | 0.5978 |
| 4 | 0.7561 | 0.0369 | 0.2886 | 0.5787 |
| 6 | 0.7367 | 0.0338 | 0.3091 | 0.5655 |
| 8 | 0.7302 | 0.0346 | 0.3158 | 0.5650 |
| 10 | 0.7281 | 0.0350 | 0.3176 | 0.5659 |
| 12 | 0.7231 | 0.0362 | 0.3257 | 0.5633 |
| 14 | 0.7229 | 0.0347 | 0.3259 | 0.5633 |
| 16 | 0.7204 | 0.0361 | 0.3308 | 0.5627 |
| 18 | 0.7192 | 0.0368 | 0.3333 | 0.5619 |
| 20 | 0.7192 | 0.0379 | 0.3338 | 0.5621 |
| 22 | 0.7197 | 0.0375 | 0.3334 | 0.5629 |
| 24 | 0.7193 | 0.0382 | 0.3346 | 0.5632 |
| 26 | 0.7206 | 0.0385 | 0.3328 | 0.5647 |
| 28 | 0.7196 | 0.0389 | 0.3353 | 0.5646 |
| 30 | 0.7195 | 0.0389 | 0.3357 | 0.5648 |
KNN Regression RMSE across different k values
# Visualize RMSE across candidate k values
ggplot(knn_reg_results, aes(x = k, y = RMSE)) +
geom_line(color = "#2C6E9B", linewidth = 1) +
geom_point(color = "#D94F4F", size = 2.5) +
scale_x_continuous(breaks = seq(2, 30, 2)) +
labs(
x = "Number of neighbors (k)",
y = "RMSE",
title = "Selecting k for KNN Regression"
) +
theme_minimal(base_size = 12)# Display the optimal k
knn_reg_model$bestTuneKNN Regression RMSE across different k values
Cross-validation results show that the RMSE of KNN Regression decreases from 0.8133 at \(k=2\) to approximately 0.7192 at \(k=18\)–\(20\). The substantial decline at small \(k\) values indicates that using more neighbors helps reduce prediction errors. However, starting around \(k=18\), the RMSE becomes relatively stable, meaning that adding more neighbors no longer yields meaningful improvements. Based on the parameter selection results, \(k=20\) is selected with an RMSE of 0.7192, MAE of 0.5621, and R-squared of 0.3338. The RMSE values for \(k=18\) and \(k=20\) are identical after rounding to four decimal places; therefore, the advantage of \(k=20\) should be understood as a selection result based on the pre-rounded RMSE value, rather than a major difference in performance.
The KNN Regression model with the selected value of (k) is used to predict wine quality scores on the test set. Model performance is evaluated using RMSE, MAE, and Rsquared. The relationship between actual and predicted scores is also visualized using a scatter plot.
The diagonal line represents perfect predictions, where predicted values equal actual values. Observations closer to this line have smaller prediction errors.
# Predict quality scores on the test set
knn_reg_predictions <- predict(
knn_reg_model,
newdata = x_test
)
# Calculate evaluation metrics
knn_reg_evaluation <- caret::postResample(
pred = knn_reg_predictions,
obs = wine_test$quality
)
knn_reg_metrics <- tibble(
metric = names(knn_reg_evaluation),
value = round(as.numeric(knn_reg_evaluation), 4)
)
knitr::kable(
knn_reg_metrics,
caption = "KNN Regression performance on the test set"
)| metric | value |
|---|---|
| RMSE | 0.7145 |
| Rsquared | 0.3521 |
| MAE | 0.5598 |
Comparison of actual and predicted wine quality scores using KNN Regression
# Prepare data for visualization
knn_reg_comparison <- tibble(
actual = wine_test$quality,
predicted = as.numeric(knn_reg_predictions)
)
# Visualize actual versus predicted values
ggplot(knn_reg_comparison, aes(x = actual, y = predicted)) +
geom_jitter(
width = 0.12,
height = 0,
color = "#2C6E9B",
alpha = 0.35,
size = 1.5
) +
geom_abline(
slope = 1,
intercept = 0,
linetype = "dashed",
color = "#D94F4F",
linewidth = 1
) +
scale_x_continuous(breaks = 3:9) +
labs(
x = "Actual quality score",
y = "Predicted quality score",
title = "Actual vs Predicted Values: KNN Regression"
) +
theme_minimal(base_size = 12)In caret::postResample(), \(R^2\) is calculated as the squared correlation between the actual and predicted values. Therefore, this metric is not always identical to the coefficient of determination based on \(1-\mathrm{SSE}/\mathrm{SST}\).
On the test set, KNN Regression achieved an RMSE of 0.7145 and an MAE of 0.5598. This indicates an average absolute prediction error of approximately 0.56 points on the wine quality scale, while the higher RMSE suggests that some observations have relatively large prediction errors. The Rsquared value of 0.3521 indicates a limited relationship between actual and predicted values.
The scatter plot shows that predictions are concentrated around quality scores of 5 to 7. Observations with low actual scores, particularly 3 and 4, tend to be overestimated, while observations with scores of 7 to 9 tend to be underestimated. This pattern reflects a tendency toward middle-range predictions because KNN Regression averages the response values of nearby observations. Thus, the model has not fully captured variations in wine quality scores, particularly for observations with extreme values.
Support Vector Machine (SVM) is a classification method that identifies a decision boundary (hyperplane) to separate observations from different classes. SVM aims to maximize the margin, defined as the distance between the decision boundary and the closest observations from each class. The observations that determine this boundary are called support vectors.
In this practical session, SVM with a Radial Basis Function (RBF) kernel is used to allow nonlinear decision boundaries. The RBF kernel is defined as:
\[K(\mathbf{x}_i,\mathbf{x}_j) = \exp\left( -\sigma\|\mathbf{x}_i-\mathbf{x}_j\|^2 \right)\]
where:
SVM performance is influenced by two main parameters, (C) and (). The parameter (C) controls the penalty for classification errors. A larger (C) imposes a stronger penalty, while a smaller (C) allows greater tolerance for classification errors. Meanwhile, a larger () makes the influence of individual observations more localized, potentially producing more complex decision boundaries.
Hyperparameters are selected through grid search using 5-fold cross-validation. Each combination of (C) and () is evaluated using balanced accuracy, and the combination with the highest mean value is selected for the final model.
# Define balanced accuracy as an evaluation metric
svm_summary <- function(data, lev = NULL, model = NULL) {
cm <- caret::confusionMatrix(
data = data$pred,
reference = data$obs,
positive = "Good"
)
c(
BalancedAccuracy = unname(cm$byClass["Balanced Accuracy"]),
Accuracy = unname(cm$overall["Accuracy"])
)
}
# Set up cross-validation
svm_control <- caret::trainControl(
method = "cv",
number = 5,
summaryFunction = svm_summary
)
# Define hyperparameter combinations
svm_grid <- expand.grid(
sigma = c(0.02, 0.08, 0.20),
C = c(0.1, 1, 10)
)
# Train and select the optimal model
set.seed(123)
svm_model <- caret::train(
x = x_train,
y = wine_train$quality_class,
method = "svmRadial",
tuneGrid = svm_grid,
trControl = svm_control,
preProcess = c("center", "scale"),
metric = "BalancedAccuracy"
)
# Display cross-validation results
svm_results <- svm_model$results %>%
select(sigma, C, BalancedAccuracy, BalancedAccuracySD, Accuracy)
knitr::kable(
svm_results,
digits = 4,
caption = "SVM Classification cross-validation results"
)| sigma | C | BalancedAccuracy | BalancedAccuracySD | Accuracy |
|---|---|---|---|---|
| 0.02 | 0.1 | 0.6878 | 0.0152 | 0.7311 |
| 0.02 | 1.0 | 0.7203 | 0.0199 | 0.7506 |
| 0.02 | 10.0 | 0.7286 | 0.0244 | 0.7572 |
| 0.08 | 0.1 | 0.7111 | 0.0149 | 0.7459 |
| 0.08 | 1.0 | 0.7418 | 0.0206 | 0.7669 |
| 0.08 | 10.0 | 0.7415 | 0.0125 | 0.7645 |
| 0.20 | 0.1 | 0.6983 | 0.0153 | 0.7429 |
| 0.20 | 1.0 | 0.7451 | 0.0131 | 0.7687 |
| 0.20 | 10.0 | 0.7226 | 0.0063 | 0.7481 |
SVM balanced accuracy across hyperparameter combinations
# Visualize hyperparameter combinations
ggplot(
svm_results,
aes(
x = factor(C),
y = factor(sigma),
fill = BalancedAccuracy
)
) +
geom_tile(color = "white", linewidth = 1) +
geom_text(
aes(label = sprintf("%.3f", BalancedAccuracy)),
size = 4
) +
scale_fill_gradient(
low = "#D8EAF5",
high = "#2C6E9B"
) +
labs(
x = "Cost parameter (C)",
y = "Sigma parameter",
fill = "Balanced accuracy",
title = "Selecting SVM Classification Hyperparameters"
) +
theme_minimal(base_size = 12)# Display the optimal hyperparameters
svm_model$bestTuneSVM balanced accuracy across hyperparameter combinations
Cross-validation results show that the combination of (C=1) and () achieved the highest balanced accuracy of 0.7451, with an accuracy of 0.7687. At (C=0.1), model performance was generally lower across all evaluated () values. Increasing (C) to 1 improved classification performance, but a further increase to 10 did not consistently produce better results.
The effect of () also depended on the value of (C). For example, at (C=1), balanced accuracy increased from 0.7203 to 0.7451 as () increased from 0.02 to 0.20. However, at (C=10), using () resulted in lower performance. This highlights the importance of selecting both parameters jointly. Nevertheless, performance differences among several leading combinations were relatively small compared with variation across folds. Therefore, the selected combination is the best among those evaluated, rather than a guarantee of global optimality.
The SVM model with (C=1) and () is used to predict wine quality categories on the test set. Predicted categories are compared with actual categories using a confusion matrix. Classification performance is then evaluated using accuracy, balanced accuracy, sensitivity, specificity, precision, and F1-score.
# Predict wine quality categories on the test set
svm_predictions <- predict(
svm_model,
newdata = x_test
)
# Construct the confusion matrix
svm_evaluation <- caret::confusionMatrix(
data = svm_predictions,
reference = wine_test$quality_class,
positive = "Good",
mode = "everything"
)
knitr::kable(
as.data.frame.matrix(svm_evaluation$table),
caption = "SVM confusion matrix on the test set"
)| Not Good | Good | |
|---|---|---|
| Not Good | 257 | 108 |
| Good | 141 | 559 |
# Summarize classification metrics
svm_metrics <- tibble(
metric = c(
"Accuracy",
"Balanced Accuracy",
"Sensitivity",
"Specificity",
"Precision",
"F1-score"
),
value = round(c(
svm_evaluation$overall["Accuracy"],
svm_evaluation$byClass["Balanced Accuracy"],
svm_evaluation$byClass["Sensitivity"],
svm_evaluation$byClass["Specificity"],
svm_evaluation$byClass["Precision"],
svm_evaluation$byClass["F1"]
), 4)
)
knitr::kable(
svm_metrics,
caption = "SVM Classification performance on the test set"
)| metric | value |
|---|---|
| Accuracy | 0.7662 |
| Balanced Accuracy | 0.7419 |
| Sensitivity | 0.8381 |
| Specificity | 0.6457 |
| Precision | 0.7986 |
| F1-score | 0.8178 |
On the test set, SVM achieved an accuracy of 0.7662 and a balanced accuracy of 0.7419. The confusion matrix shows that 559 Good observations and 257 Not Good observations were correctly classified. Meanwhile, 141 Not Good observations were incorrectly predicted as Good, and 108 Good observations were predicted as Not Good.
The sensitivity of 0.8381 indicates that the model correctly identified 83.81% of Good observations, while the specificity of 0.6457 indicates that 64.57% of Not Good observations were correctly identified. Thus, SVM still performed better at recognizing the Good class, but the difference in class-specific performance was smaller than that observed for KNN Classification. The precision of 0.7986 and F1-score of 0.8178 also indicate reasonably good predictive performance for the Good class.
Support Vector Regression (SVR) extends SVM to predict numerical response variables. In this practical session, SVR is used to predict wine quality scores (quality) based on physicochemical characteristics.
Unlike regression approaches that directly penalize all prediction errors, SVR uses the concept of epsilon-insensitive loss. Under this approach, prediction errors within a tolerance threshold () are not penalized.
The loss function is expressed as:
\[L_{\epsilon}(y_i,\hat{y}_i) = \max\left(0,\lvert y_i-\hat{y}_i\rvert-\epsilon\right)\]
where:
Thus, SVR aims to construct a prediction function that balances model complexity and errors exceeding the specified tolerance.
In this practical session, SVR with a Radial Basis Function (RBF) kernel is implemented. Model performance is influenced by (C), which controls the penalty for errors outside the tolerance threshold, and (), which determines the characteristics of the RBF kernel.
Hyperparameters are selected through grid search using 5-fold cross-validation. Each parameter combination is evaluated using RMSE, and the combination with the lowest mean RMSE is selected. The parameter () is fixed at 0.1 throughout model training.
# Set up cross-validation
svr_control <- caret::trainControl(
method = "cv",
number = 5
)
# Define hyperparameter combinations
svr_grid <- expand.grid(
sigma = c(0.02, 0.08, 0.20),
C = c(0.1, 1, 10)
)
# Train and select the optimal SVR model
set.seed(123)
svr_model <- caret::train(
x = x_train,
y = wine_train$quality,
method = "svmRadial",
tuneGrid = svr_grid,
trControl = svr_control,
preProcess = c("center", "scale"),
metric = "RMSE",
epsilon = 0.1
)
# Display cross-validation results
svr_results <- svr_model$results %>%
select(sigma, C, RMSE, RMSESD, Rsquared, MAE)
knitr::kable(
svr_results,
digits = 4,
caption = "Support Vector Regression cross-validation results"
)| sigma | C | RMSE | RMSESD | Rsquared | MAE |
|---|---|---|---|---|---|
| 0.02 | 0.1 | 0.7252 | 0.0440 | 0.3247 | 0.5600 |
| 0.02 | 1.0 | 0.7080 | 0.0447 | 0.3524 | 0.5451 |
| 0.02 | 10.0 | 0.7022 | 0.0459 | 0.3645 | 0.5378 |
| 0.08 | 0.1 | 0.7125 | 0.0424 | 0.3455 | 0.5465 |
| 0.08 | 1.0 | 0.6969 | 0.0402 | 0.3728 | 0.5293 |
| 0.08 | 10.0 | 0.7125 | 0.0314 | 0.3550 | 0.5415 |
| 0.20 | 0.1 | 0.7193 | 0.0373 | 0.3359 | 0.5475 |
| 0.20 | 1.0 | 0.7052 | 0.0335 | 0.3588 | 0.5332 |
| 0.20 | 10.0 | 0.7676 | 0.0303 | 0.2946 | 0.5888 |
SVR RMSE across hyperparameter combinations
# Visualize hyperparameter combinations
ggplot(
svr_results,
aes(
x = factor(C),
y = factor(sigma),
fill = RMSE
)
) +
geom_tile(color = "white", linewidth = 1) +
geom_text(
aes(label = sprintf("%.3f", RMSE)),
size = 4
) +
scale_fill_gradient(
low = "#BFDDEB",
high = "#F19C79"
) +
labs(
x = "Cost parameter (C)",
y = "Sigma parameter",
fill = "RMSE",
title = "Selecting Support Vector Regression Hyperparameters"
) +
theme_minimal(base_size = 12)# Display the optimal hyperparameters
svr_model$bestTuneSVR RMSE across hyperparameter combinations
Cross-validation results show that the combination of (C=1) and () achieved the lowest RMSE of 0.6969, with an MAE of 0.5293 and Rsquared of 0.3728. At (C=1), RMSE decreased as () increased from 0.02 to 0.08, but increased again at (). Meanwhile, the combination of (C=10) and () produced the highest RMSE of 0.7676.
This pattern indicates that the effects of (C) and () are interrelated. Stronger penalties for prediction errors and more localized kernel effects do not necessarily improve predictive performance. Among the evaluated combinations, (C=1) and () were selected for the final SVR model, which is subsequently evaluated using the test set.
The SVR model with (C=1) and () is used to predict wine quality scores on the test set. Prediction performance is evaluated using RMSE, MAE, and Rsquared.
The relationship between actual and predicted values is also visualized using a scatter plot. The diagonal line represents perfect predictions, so the distance between observations and this line provides a visual indication of prediction errors.
# Predict quality scores on the test set
svr_predictions <- predict(
svr_model,
newdata = x_test
)
# Calculate evaluation metrics
svr_evaluation <- caret::postResample(
pred = svr_predictions,
obs = wine_test$quality
)
svr_metrics <- tibble(
metric = names(svr_evaluation),
value = round(as.numeric(svr_evaluation), 4)
)
knitr::kable(
svr_metrics,
caption = "Support Vector Regression performance on the test set"
)| metric | value |
|---|---|
| RMSE | 0.6864 |
| Rsquared | 0.3974 |
| MAE | 0.5221 |
Comparison of actual and predicted wine quality scores using SVR
# Prepare data for visualization
svr_comparison <- tibble(
actual = wine_test$quality,
predicted = as.numeric(svr_predictions)
)
# Visualize actual versus predicted values
ggplot(svr_comparison, aes(x = actual, y = predicted)) +
geom_jitter(
width = 0.12,
height = 0,
color = "#2C6E9B",
alpha = 0.35,
size = 1.5
) +
geom_abline(
slope = 1,
intercept = 0,
linetype = "dashed",
color = "#D94F4F",
linewidth = 1
) +
scale_x_continuous(breaks = 3:9) +
labs(
x = "Actual quality score",
y = "Predicted quality score",
title = "Actual vs Predicted Values: SVR"
) +
theme_minimal(base_size = 12)On the test set, SVR achieved an RMSE of 0.6864 and an MAE of 0.5221. The MAE indicates an average absolute prediction error of approximately 0.52 wine quality points. Meanwhile, the Rsquared value of 0.3974 indicates a relationship between actual and predicted scores, although the model’s ability to represent variations in quality scores remains limited.
The scatter plot shows that low quality scores, particularly 3 and 4, tend to be overestimated, while scores of 8 and 9 tend to be underestimated. Most predictions remain concentrated around scores of 5 to 7. Thus, SVR has not fully captured extreme quality scores accurately, but it produces smaller prediction errors than KNN Regression on the same test set.
KNN Classification and SVM Classification are compared using evaluation results from the same test set. The metrics include accuracy, balanced accuracy, sensitivity, specificity, precision, and F1-score.
Balanced accuracy is the primary comparison metric because it considers the model’s ability to recognize both quality categories. Sensitivity and specificity are also compared to examine differences in identifying Good and Not Good wines.
# Combine evaluation metrics
classification_comparison <- bind_rows(
knn_metrics %>% mutate(model = "KNN"),
svm_metrics %>% mutate(model = "SVM")
)
# Display the comparison table
classification_comparison %>%
pivot_wider(
names_from = metric,
values_from = value
) %>%
knitr::kable(
digits = 4,
caption = "Classification model performance comparison on the test set"
)| model | Accuracy | Balanced Accuracy | Sensitivity | Specificity | Precision | F1-score |
|---|---|---|---|---|---|---|
| KNN | 0.7465 | 0.7099 | 0.8546 | 0.5653 | 0.7672 | 0.8085 |
| SVM | 0.7662 | 0.7419 | 0.8381 | 0.6457 | 0.7986 | 0.8178 |
Performance comparison of KNN and SVM Classification
# Select key metrics for visualization
classification_plot <- classification_comparison %>%
filter(
metric %in% c(
"Balanced Accuracy",
"Sensitivity",
"Specificity"
)
)
# Visualize model performance
ggplot(
classification_plot,
aes(x = metric, y = value, fill = model)
) +
geom_col(
position = position_dodge(width = 0.75),
width = 0.7
) +
scale_fill_manual(
values = c(
"KNN" = "#2C6E9B",
"SVM" = "#D94F4F"
)
) +
scale_y_continuous(
limits = c(0, 1),
breaks = seq(0, 1, 0.2)
) +
labs(
x = "Evaluation metric",
y = "Score",
fill = "Model",
title = "KNN vs SVM Classification"
) +
theme_minimal(base_size = 12)The test-set comparison shows that SVM achieved a balanced accuracy of 0.7419, exceeding KNN’s 0.7099. SVM also produced higher accuracy, precision, and F1-score. Its main advantage was observed in specificity, which increased from 0.5653 for KNN to 0.6457 for SVM. This indicates that SVM was more effective at identifying Not Good wines.
Conversely, KNN achieved slightly higher sensitivity, at 0.8546 compared with 0.8381 for SVM, meaning that it correctly identified more Good observations. Considering performance across both classes, SVM demonstrated better classification performance on this test set. However, this comparison is descriptive and based on a single data split, so it does not establish that SVM would consistently outperform KNN across other splits.
KNN Regression and Support Vector Regression (SVR) are compared using predictions from the same test set. Model performance is evaluated based on RMSE, MAE, and Rsquared.
RMSE and MAE measure the magnitude of prediction errors, while Rsquared represents the strength of the relationship between actual and predicted values. The visualization focuses on RMSE and MAE because both metrics are expressed in wine quality score units and can be directly compared.
# Combine regression evaluation results
regression_comparison <- bind_rows(
knn_reg_metrics %>% mutate(model = "KNN Regression"),
svr_metrics %>% mutate(model = "SVR")
)
# Display the comparison table
regression_comparison %>%
pivot_wider(
names_from = metric,
values_from = value
) %>%
knitr::kable(
digits = 4,
caption = "Regression model performance comparison on the test set"
)| model | RMSE | Rsquared | MAE |
|---|---|---|---|
| KNN Regression | 0.7145 | 0.3521 | 0.5598 |
| SVR | 0.6864 | 0.3974 | 0.5221 |
Prediction error comparison between KNN Regression and SVR
# Select error metrics for visualization
regression_plot <- regression_comparison %>%
filter(metric %in% c("RMSE", "MAE"))
# Visualize RMSE and MAE comparison
ggplot(
regression_plot,
aes(x = metric, y = value, fill = model)
) +
geom_col(
position = position_dodge(width = 0.75),
width = 0.7
) +
geom_text(
aes(label = sprintf("%.4f", value)),
position = position_dodge(width = 0.75),
vjust = -0.5,
size = 4
) +
scale_fill_manual(
values = c(
"KNN Regression" = "#2C6E9B",
"SVR" = "#D94F4F"
)
) +
scale_y_continuous(
limits = c(0, 0.85),
breaks = seq(0, 0.8, 0.2)
) +
labs(
x = "Evaluation metric",
y = "Prediction error",
fill = "Model",
title = "KNN Regression vs SVR"
) +
theme_minimal(base_size = 12)The comparison shows that SVR achieved an RMSE of 0.6864, lower than KNN Regression’s 0.7145. SVR also produced a lower MAE of 0.5221 compared with 0.5598 for KNN Regression. Meanwhile, SVR achieved a higher Rsquared value of 0.3974 compared with 0.3521 for KNN Regression. These three metrics indicate that SVR provided better predictions on the test set.
Nevertheless, both models still tended to produce predictions concentrated around middle-range quality scores. Observations with very low or high quality scores were not predicted accurately. Thus, SVR was the better regression model among the two evaluated methods, although it remained limited in capturing variations in wine quality, particularly at extreme values.
The application of KNN and SVM to the wine quality dataset demonstrates that model selection should be aligned with the prediction task. In classification, SVM achieved higher balanced accuracy than KNN and performed better at identifying Not Good wines, although KNN showed slightly higher sensitivity for the Good category. In regression, SVR achieved lower RMSE and MAE than KNN Regression, but both approaches remained limited in predicting extreme quality scores.
These findings also highlight the importance of predictor standardization, hyperparameter selection through cross-validation, and evaluation on a separate test set. Model performance should not be judged using a single metric, but should account for the characteristics of the response variable and the types of prediction errors. Overall, SVM with an RBF kernel performed better on both tasks in this evaluation, although these results should not be generalized without further assessment across different data splits or datasets.
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018
Cortez, P., Cerdeira, A., Almeida, F., Matos, T., & Reis, J. (2009a). Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems, 47(4), 547–553. https://doi.org/10.1016/j.dss.2009.05.016
Cortez, P., Cerdeira, A., Almeida, F., Matos, T., & Reis, J. (2009b). Wine Quality [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C56S3T
Cover, T. M., & Hart, P. E. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21–27. https://doi.org/10.1109/TIT.1967.1053964
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning: With applications in R (2nd ed.). Springer. https://doi.org/10.1007/978-1-0716-1418-1
Kuhn, M. (2008). Building predictive models in R using the caret package. Journal of Statistical Software, 28(5), 1–26. https://doi.org/10.18637/jss.v028.i05
Smola, A. J., & Schölkopf, B. (2004). A tutorial on support vector regression. Statistics and Computing, 14(3), 199–222. https://doi.org/10.1023/B:STCO.0000035301.49549.88
Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. Springer. https://doi.org/10.1007/978-3-319-24277-4