QUESTION 1

We have been tasked with analyzing a dataset from a farm to understand how various factors affect mealie quality.

To prepare for this analysis, we first checked the dataset for any missing values to ensure data integrity. We then created a copy of the dataset for manipulation, preserving the original data. In our preprocessing steps, we transformed the ‘pests’ column into a binary variable for ease of analysis, subsequently removing the original column to streamline our dataset.

Additionally, we converted the ‘quality’ column into an ordered factor to accurately reflect the hierarchical nature of mealie quality levels.

Finally, we inspected the structure of our modified dataset to confirm these adjustments were successfully applied, setting a solid foundation for our analysis. The structure of the data is found below.

## 'data.frame':    1250 obs. of  6 variables:
##  $ longitude   : num  26.3 26.3 26.3 26.3 26.3 ...
##  $ latitude    : num  -28.8 -28.8 -28.8 -28.8 -28.8 ...
##  $ count       : int  8 5 8 6 7 4 4 4 5 8 ...
##  $ height      : num  1.8 1.89 1.46 1.93 1.8 ...
##  $ quality     : Ord.factor w/ 4 levels "A"<"B"<"C"<"D": 2 4 4 4 2 3 3 4 4 4 ...
##  $ pests_binary: num  1 1 1 1 1 1 1 1 1 1 ...

To delve deeper into the factors influencing mealie quality on the farm, we divided our dataset into training and testing sets, adhering to an 80/20 split to balance the depth of learning with the need for unbiased validation.

We then constructed a classification tree, intentionally overfitting it by allowing it to grow until all terminal nodes were homogenous (observations that are as pure as possible regarding the target variable, mielie quality).

This extreme level of detail was achieved by not setting a limit on the tree’s depth, ensuring that each leaf node represents a pure subset of the data.

The purpose behind creating such an overfitted tree was to illustrate the potential pitfalls of over-complex models, including their tendency to learn the noise in the training data rather than the underlying patterns.

Finally, we evaluated the tree’s performance on the unseen test data, calculating the misclassification rate to gauge how well our model generalizes to new data. This rate reflects the proportion of mealie quality predictions that were incorrect, offering a quantitative measure of the model’s predictive accuracy outside the training dataset.

Despite the low misclassification rate on the test set, the approach taken with this tree is typically not advisable for creating predictive models meant for general application due to the risks of overfitting and poor generalization to new data.

## [1] "Testing misclassification rate: 0.065"

In finding an optimal tree size, we can use cross-validation (CV) to find the most appropriate level of model complexity. The optimal complexity parameter, determined via 10-fold CV, emerged as 0.004, suggesting a balanced model that avoids the extremes of underfitting and overfitting.

The graph, shown below, plots the CV accuracy against the complexity parameter. The chosen cp value corresponds to the peak of the graph before the accuracy starts to decline, indicating the point of diminishing returns for increasing complexity.

Using this CV-determined tree size to our test data, we observed a misclassification rate of 0.081. This rate is slightly higher than the 0.065 from the overfitted model described in the overfitted model. This is not unexpected; the overfitted model, which perfectly classified the training data, is likely too tailored to the specifics of the training set, hence its better performance there but poorer generalisability.

When comparing the CV error rate with the training error rate, the latter is usually lower due to the model being assessed on the same data it was trained on. The CV error rate offers a more robust estimate of out-of-sample performance and is typically higher. This is aligned with our observations, as the training error rate is expectedly lower than the CV error rate. The test error rate being closer to the CV error rate than the training error rate validates the efficacy of CV in predicting the model’s performance on unseen data.

## [1] "Optimal tree complexity parameter (cp): 0.004"
## [1] "Test misclassification rate for the best tree: 0.081"
## [1] "Training misclassification rate for the best tree: 0.051"

The plot below shows the geographical distribution of mealie quality.

By plotting longitude and latitude against mealie quality, we aim to understand the spatial relationship and identify any visible patterns or clusters that indicate regional effects on mealie quality.

This initial exploration helps justify the need for further analysis, specifically the rotation of the feature space to potentially uncover clearer patterns or decision boundaries.

Next we align decision boundaries with the axes for enhanced model interpretability.

We apply a systematic rotation to the geographical features (longitude and latitude) across a series of angles (theta values). Using 10-fold cross-validation, we assess model performance at each rotation angle to identify the optimal theta.

This optimal angle suggests the best rotation for separating mealie quality regions with orthogonal decision boundaries. The process involves rotating the data, fitting a classification model to the rotated data, and evaluating its accuracy through cross-validation.

The best theta that improves the model’s ability to predict mealie quality based on geographical location, taking into account the rotated feature space is found to be 0.565 radians.

## [1] "Best theta: 0.565"

The classification tree structure, shown below, pruned to 4 terminal nodes and fitted on the rotated geographical features, reveals significant insights into the factors affecting mealie quality.

The tree’s splitting variables and conditions highlight the critical features influencing mealie quality post-rotation. Specifically, the rotated geographical features (rotated longitude and latitude) likely emerged as key predictors, indicating that the spatial distribution of mealie quality can be captured after adjusting the feature space.

This suggests that the rotation helped in aligning the mealie quality regions more closely with the model’s decision boundaries, thereby enhancing the interpretability and predictive power of the model.

## [1] "Misclassification rate for the pruned tree: 0.109"

The pruned tree model, with a misclassification rate of 0.109, performs differently compared to the earlier models.

The initial overfitted model had a lower misclassification rate of 0.065, which, despite its seemingly superior performance, suffered from poor generalization due to overfitting. The CV-optimized model, designed to balance model complexity and predictive accuracy, yielded a misclassification rate of 0.081. This rate was an improvement in terms of model generalization compared to the overfitted model. The pruned model’s higher misclassification rate, in comparison, might initially seem like a step back. However, it’s essential to consider the model’s simplicity, interpretability, and the robustness of its predictions.

With only 4 terminal nodes, the pruned model is far simpler and likely to generalise better to unseen data despite the higher misclassification rate observed in the test data. The rotation of features, as evidenced by the selection of the best theta, was instrumental in revealing this balance, showing that a model could achieve reasonable accuracy with a significantly simplified structure, underscoring the value of feature engineering and model complexity optimisation in predictive modeling.

QUESTION 2

In this question, we have been tasked with developing a predictive model using data from 743 completed matches of the Indian Premier League (IPL). The dataset encompasses various features, including the names of the defending and chasing teams, the venue (home, away, or neutral), the time of the match (day or day/night), the toss outcome, as well as quantitative measures of both teams’ batting and bowling strengths. Additionally, it includes the score being defended, the venue’s median scores for the first and second innings, and its altitude and dimensions. Our objective is to predict the outcome of a match (win or loss for the defending team) after the first innings, based on these features.

In tackling the challenge of predicting outcomes in the Indian Premier League (IPL), we approached using logistic regression with elastic-net regularization. This method helps us deal with a mix of different types of data and picks out the most relevant information for making predictions.

We chose a balanced setting for our model, with an alpha value of 0.5. This means we’re using a mix of two techniques to make sure our model is just right—not too complex but still able to pick up on the important patterns in the data. We found the best lambda value of 0.028 determined through cross-validation, specifically using the cv.glmnet function. By selecting lambda.min, we aimed for a model that boasts the lowest possible error on unseen data, thus ensuring a robust prediction capability.

After setting up our model, we looked at which factors it found important for predicting if the defending team would win. This included factors such as which team was playing, where the match was held, and the match’s time.

We then tested how well our model worked by looking at its accuracy and F1-score, which are two ways to measure how good a model’s predictions are.

## [1] "Accuracy: 0.658"
## [1] "F1 Score: 0.73"

Our model achieved an accuracy of about 65.77% and an F1-score of about 73.02%. These results show that our model does a decent job at predicting match outcomes, though there’s still room for improvement.

We then conducted an ROC Curve Analysis, the ROC Curve and resluts are plotted below.

## AUC: 0.6799
## Minimum τ for recall ≥ 0.75: 0.3834

We calculated the AUC and identified the critical decision threshold, τ, that achieves a minimum recall of 0.75. The curve and calculated AUC of 0.6799 confirm the model’s discriminatory capacity. Meanwhile, the τ of 0.3834 marks the probability cutoff point, ensuring a recall of at least 0.75, thus balancing the model’s sensitivity and specificity. This τ value is visually represented on the ROC curve.

QUESTION 3

In Question 3, we have been tasked with analyzing a dataset that contains various speech signal measurements from patients with early-stage Parkinson’s disease. Our primary objective is to predict the values of the Unified Parkinson’s Disease Rating Scale (UPDRS).

The dataset comprises several extracted features, including the age and biological sex of the subjects, alongside a range of voice signal measurements such as jitter, shimmer, noise-to-harmonic ratios, and other non-linear dynamical complexity measures. These voice signal attributes are indicative of variations in frequency and amplitude that are potential biomarkers of Parkinson’s disease severity.

We approach the problem by comparing multiple predictive modeling techniques to average predicted UPDRS score for each model. These models include:

-A regularised linear model,

-K-Nearest Neighbors (KNN) with a k value of 5, chosen after ten different values of k were evaluated,

-A Random Forest model with the hyperparamter mtry (The number of variables randomly sampled as candidates at each split) being tuned along with a report of Variable Importance, and

-A boosted tree model using XGBoost, with hyperparameter selection and an exploration of the most influential features.

## Average Predicted UPDRS Score for Linear Model: 28.8883
## Average Predicted UPDRS Score for KNN: 29.35373
## Average Predicted UPDRS Score for Random Forest: 29.28632
##                %IncMSE IncNodePurity
## age           91.68856     134968.38
## sex           41.46497      12628.79
## Jitter...     25.38539      11936.40
## Jitter.Abs.   42.03772      24408.48
## Jitter.RAP    26.33001      10726.11
## Jitter.PPQ5   28.42388      12694.50
## Jitter.DDP    25.60124      10926.72
## Shimmer       21.32539      10775.34
## Shimmer.dB.   21.87228      10736.35
## Shimmer.APQ3  24.38269      12227.94
## Shimmer.APQ5  19.31926      12058.53
## Shimmer.APQ11 22.78965      13329.69
## Shimmer.DDA   24.02753      12189.60
## NHR           28.76326      17025.90
## HNR           33.03047      24497.00
## RPDE          38.34933      24332.76
## DFA           75.92546      47318.98
## PPE           31.88739      20628.07

Age has the highest %IncMSE and IncNodePurity values, indicating it’s the most important feature in predicting UPDRS scores. This suggests that patient age has a significant impact on the disease progression, making it a critical factor for prediction.”)

## Average Predicted UPDRS Score for XGBoost: 29.0977

The variable importance plot essentially confirms that both demographic (age, sex) and speech signal attributes (DFA, Jitter.Abs, HNR, Shimmer.dB) are significant predictors of the UPDRS score, which is aligned with clinical findings that Parkinson’s disease affects motor control, which in turn affects speech production.

Next, we will compare the different model’s RMSE scores and show a visual illustration of the models out of sample errors. After this we will select a model that we will use on unseen data.

## RMSE for Linear Model:  9.882
## RMSE for KNN:  6.352
## RMSE for Random Forest:  5.378426
## RMSE for XGBoost:  3.406736

Based on the RMSE values for the four models, the XGBoost model has the lowest RMSE of 3.406736, which suggests that it performs the best in predicting the UPDRS score from the given features. This model would be my choice for use on unseen data, given that a lower RMSE indicates better fit to the data and likely better predictions on new data.

The XGBoost model’s lower error rate implies that it is capable of capturing the underlying patterns in the dataset more effectively than the other models. It benefits from gradient boosting, which iteratively improves upon the predictions by focusing on the residuals of prior iterations, enabling the model to learn complex relationships.