Forest Fires
Samantha Gouveia
12/28/2023
Case Study
The data set we are using for the case study, College Data, can be found on Kaggle. The data was collected from many different colleges in the United states. The colleges are both a mix of private and public.
Data and Variable Description
There are 18 variables in the data set.
Private: Factor with yes or no indicating private or public university
Apps: Number of applications the school received
Accept: Number of students accepted
Enroll: Number of new students who enrolled
Top10perc: New students in top 10% of their high school class
Top25perc: New students in top 25% of their high school class
F.undergrad: Number of full time undergraduates
P.undergrad: Number of part time undergraduates
Outstate: out of state tuition
Room.Board: Cost of room and board
Books: Estimated cost of books
Personal: Estimates personal spending
PhD: Percent of faculty with a PhD
Terminal: Percent of faculty with terminal degree
S.F.Ratio: Student faculty ratio
Perc.alumni: Percent of alumni who donate
Expend: Instructional expenditure per student
Grad.Rate: Graduation rate
Research Question
The objective of the case study is to identify the criteria necessary to get into a private college.
The report aims to explore the correlation between variables.
The report also aims to explore which model, with different variables and number of variables, will preform the best predictions.
Data cleaning
I begin by removing all the colleges with missing values and keeping only those with complete records for the case study. The final analytic data has 777 colleges.
We are going to begin by looking at the association between private College and acceptance
The Acceptance histogram is highly skewed to the right. We see that the acceptance is extremely high between 0-2500 and drastically drops after that. The acceptance continues to decrease until it reaches zero around 15000.
Exploratory Analysis
A pairwise scatter plot is used to identify any potential issues with the predictor variables. The final analytic model has 777 observations and 19 variables. Showing none of the predictors had any issues.
The pairwise scatter plot shows many patterns in the predictor variables.
All Predictor variables are unimodal. We see that Apps and Accept are significantly skewed to the right. Therefore we take a closer look at the two variables.
After Exploratory analysis, we decided to regroup Accept and App, we then defined dummy variables for the associated variables. The new variables will be used in the model during the search process.
A moderate correlations is observed between: Apps vs Accept, Top10perc vs Top25perc, Apps vs Enroll,Enroll Vs F.Undergrad, and Accept vs Enroll. None of the variables will be dropped as an automatic variable selection process will occur. The variable selection will remove potential redundant variables since some of them will be forced into the final model.
Our goal of the report is an association analysis between a set of criteria for a private colleges. Therefore we do not preform variable transformations yet.
Looking at the data set we know in a real world situation many of the variables are important. However when doing a statistical analysis those variables may not be statistically significant. Regardless of the statistical significance the variables will be included. In the college study Acceptance, Enroll,Top10perc, and Apps are considered significant risk factors. These variables will be included in the final model.
Standizing numerical Predictor variables
When standardizing a numerical predictor variable we are subtracting from the sample mean and dividing it by the standard deviation. Once that is complete we get a zero mean and standard deviation.
We use standardizing on regression models to reduce multicollinearity and the associates problems that are caused by having higher order terms.
Standardizing the variables also ensures that the coefficients are on the same scale and can help make interpretations of the results straightforward.
Once standardizing of the predictor variables is complete we remove variables from the dataset. We keep only our response variable, the standardized variables, and the original version of the standardized variables. The combination of these variables will create the new data set used for the remainder of the report.
Data splitting
We split the data into two subsets without replacement. 80% of the data is used as training data and will search candidate model, validate them, and identify the final model using the cross-validation method. The 20% will be used to assess the performance of the final model.
Best model identification
To find the best model we are going to be using a full, reduced, and final model. The models will be based on the step-wise variable selection as the three candidate models.
We will use both the predictive error values and size of the model to choose the best model.
Cross-validation model identificantion
A cross validation model identification can provide several advantages these include: detecting over fitting and estimating a models performance more accurately.
The cross validation model will split our data into multiple training and test data sets. The different splits constitute different cross validation methods.
We will use k-fold cross validation as it helps mitigate risks of over fitting and will provide a more reliable assessment of how well the model is expected to preform on unseen data. We will use 5-fold cross validation as the training data is relatively small and to ensure the validation data set has enough colleges.
We will then create three candidate models which will allow us to find the PE(prediction errors) of the candidate models.
| PE1 | PE2 | PE3 |
|---|---|---|
| 0.9355 | 0.9355 | 0.9355 |
We see in the full model that l and p are highly correlated with one another. l and p are also highly correlated with variables in the full model. Due to the high correlation we remove a and l from the full and reduced model.
In candidate 2 we use a forward step wise as it help to improve model performance by selecting features based on individual and combined predicted power. Forward selection also helps building models that have a higher accuracy,precision,recall, other performance metrics.
To find the pre.outcomes we use the ifelse function. The function takes the conditions yes,no,and missing into account when determining the output.
Once completing the cross validation model identification we obtain the PE values for PE1,2,and 3. We find that each of the PE values are the same at PE = .9387. Due to them being the same we will use model 2 as the predictive model because it is simpler than model 1. The prediction error is also very good as the closer the value is to one the better it is at predicting errors. # Final model reporting
The cross validation procedure was used to find the best model with the pre-selected cut off of 0.5. The final model will show the actual accuracy report based on the withheld test data.
| x |
|---|
| 0.5935 |
After running the code we find the actual accuracy for the final model is x = 0.6.
When using a random split method for training and testing data when the code is rerun the actual accuracy value will be slightly different.
ROC analysis Global Performance
An ROC curve is a graph that shows the performance of classification models at all classification thresholds. The ROC curve is used to compare the global performance of binary predictive models.
We are going to construct the curve based on the training data and calculate the corresponding AUC values.
The first thing we are doing is estimating the TPR(true positive rate, sensitivity) and the FPR(false positive rate, 1- specificity) at each cut off probability for each of the three candidate models.
The three candidate models are being created using the same response variable and predictor variables as used before. The only change in the data we are using, the new data set is called train.
Once the candidate models are created the predictions will be fun for each using the new data. The ROC curve will be created using the prediction variables we created.
The AUC(area under the curve) is a way of measuring the performance of a model. The AUC values will be created using the data set and the response variable. An AUC value will be created for each of the three models.
When looking at the ROC curves we can see that the full model and step wise model are better than the reduced model. The full model and step wise model also have the same the same ROC curve and AUC value. The final model we will use for the report is the step wise model. This is due to the step wise model being simpler than the full model. The AUC value for both full and step wise model = .9787 and for the reduced model = .9744. We conclude from these values that the model is very capable of distinguishing between classes as the value is close to one.
Conclusion
The case study focused on the criteria necessary to get into a private college. We used three candidate model and both cross validation and ROC curves to select the final model. We found that both the ROC curve and the cross validation selected the same final model.
The high ROC and PE values show that in a real world scenario the step wise model would have a highly accurate prediction and therefore could be used in a work place. However when looking at the actual accuracy of the final model we get only a 60% correct prediction rate. Although a 60-to-40 ratio makes for a good prediction rate the usefulness of the model decreases. The ROC AUC is generally seen as more a more important measure of how good a model will be as it provides a more comprehensive evaluation of the models performance. The actual accuracy of .6 is something to be aware of however it is not something that should be taken with great affect.
The report also found there is a high correlation between the accept, apps, enroll, and top10perc variables. This allows for us to conclude that these factors will be the most important predictors of getting into a private college.
References
Gupta, Y. (2019, October 28). US College Data. Kaggle. https://www.kaggle.com/datasets/yashgpt/us-college-data