Survey Analysis - Validation and PCA

1 Introduction

The purpose of this study is to investigate various aspects of the learning experience that impacts students’ satisfaction. The study population is undergraduate students in two business schools at two regional universities in the US. The goal is to figure out which aspects lead best to satisfaction whether academic-related (engagement in classes or learning styles) to extracurricular resources (how students pay, if they utilize campus resources, self-reported growth and development).

2 Data Management

The survey instrument consists of 121 questions that ask about demographics, performance in class and 9 sections that contain questions that seek to explain performance and satisfaction.

The 9 sections that ask explanatory questions are:

• Students’ Engagement in learning
• Student Learning Styles
• Writing and Reading Load
• Remedial Experience
• Encouragement and Support
• Growth and Development
• Campus Resource Utilization
• Retention
• How Students Pay For College

The 3 sections that ask response questions are:

• Academic Standing
• Satisfaction/Loyalty
• Demographics

2.1 Missing Values

When reading the .csv file in, every other row was empty. As a result, we collapsed the data set down by removing all rows with only NA values. As a result, there were no missing values present after this removal.

2.2 Grouping related questions

Almost every question asked on the survey was classified into a specific section. For these survey questions, we grouped these into their individual sections. However, questions 1, 2, 3, 7, and 14 were standalone questions that were not automatically classified into sections. As a result, I classified these questions based on the information they would generate.

Question 1 asked ‘Are you a: 1 – Freshman, 2 – Sophomore, 3 – Junior, 4 – Senior’.

Question 2 asked ‘Did you begin college at our school or elsewhere? 1 - started at our school, 2 - started elsewhere’.

Since these questions are about demographics, I grouped these under the ‘Demographics’ section.

Question 3 asked ‘How many credit hours are you currently taking during this semester? 1 - less than 12 credits, 2 - 12 credits or more’.

Question 7 asked ’ Mark the response that best represents the extent to which your examinations during the current school year have challenged you: 1 - Extremely easy, 2 - Easy, 3 - Slightly easy, 4 - Neither easy nor challenging, 5 - Slightly challenging, 6 - Challenging, 7 - Extremely challenging, 8 - NA.’.

This is a question related to the student’s academic standing, so it was put into the ‘Academic Standing’ section.

Question 14 asked ‘When do you plan to take classes at our school again? 1 - I will accomplish my goal during this term and will not be returning, 2 - I have no current plan to return, 3 - Within the next 12 months, 4 - Uncertain’.

This is a question about students’ satisfaction or loyalty, so it was grouped into the ‘Satisfaction/Loyalty’ section.

3 Validity and Reliability Analyses

Now we will check for validity and reliability by each section. It’s important to test the validity as we want to ensure that the conclusions drawn accurately reflect what we’re attempting to assess. Reliability is equally important to check as we need to confirm that the data isn’t being influenced solely by random variability.

We will look at each subscale’s correlation matrix and Cronbach’s Alpha to determine reliability.

3.1 Student Engagement in Learning

First, we look at the correlation matrix. The correlation matrix is displaying anywhere from a neutral to strong positive correlation. As a result, we now look at the Cronbach’s alpha to examine internal reliability.

The Cronbach’s Alpha demonstrates strong internal reliability as the 95% confidence interval for alpha is [0.858, 0.896].

Confidence Interval of Cronbach Alpha

	LCI	alpha	UCI
	0.858	0.878	0.896

3.2 Student Learning Styles

First, we look at the correlation matrix. The correlation matrix is displaying anywhere from a moderate to strong positive correlation. As a result, we now look at the Cronbach’s alpha to examine internal reliability.

The Cronbach’s Alpha demonstrates strong internal reliability as the 95% confidence interval for alpha is [0.821, 0.872].

Confidence Interval of Cronbach Alpha

	LCI	alpha	UCI
	0.821	0.847	0.872

3.3 Writing and Reading Load

First, we look at the correlation matrix. The correlation matrix is displaying anywhere from a moderate to strong positive correlation. As a result, we now look at the Cronbach’s alpha to examine internal reliability.

The Cronbach’s Alpha demonstrates weak internal reliability as the 95% confidence interval for alpha is [0.389, 0.579], all of which represent that the subscale exhibits weak internal consistency. As a result, we decide to not incorporate this in our study.

Confidence Interval of Cronbach Alpha

	LCI	alpha	UCI
	0.389	0.491	0.579

3.4 Remedial Experience

First, we look at the correlation matrix. The correlation matrix is displaying anywhere from a moderate to strong positive correlation. As a result, we now look at the Cronbach’s alpha to examine internal reliability.

The Cronbach’s Alpha demonstrates an acceptable amount internal reliability as the 95% confidence interval for alpha is [0.775, 0.837], all of which represent that the subscale exhibits moderate internal consistency.

Confidence Interval of Cronbach Alpha

	LCI	alpha	UCI
	0.775	0.807	0.837

3.5 Encouragement and Support

First, we look at the correlation matrix. The correlation matrix is displaying anywhere from a moderate to strong positive correlation. As a result, we now look at the Cronbach’s alpha to examine internal reliability.

The Cronbach’s Alpha demonstrates strong internal reliability as the 95% confidence interval for alpha is [0.804, 0.859].

Confidence Interval of Cronbach Alpha

	LCI	alpha	UCI
	0.804	0.833	0.859

3.6 Growth and Development

First, we look at the correlation matrix. The correlation matrix is displaying anywhere from a moderate to strong positive correlation. As a result, we now look at the Cronbach’s alpha to examine internal reliability.

The Cronbach’s Alpha demonstrates strong internal reliability as the 95% confidence interval for alpha is [0.938, 0.955].

Confidence Interval of Cronbach Alpha

	LCI	alpha	UCI
	0.938	0.947	0.955

3.7 Campus Resource Utilization

First, we look at the correlation matrix. The correlation matrix is displaying anywhere from a moderate to strong positive correlation. As a result, we now look at the Cronbach’s alpha to examine internal reliability.

The Cronbach’s Alpha demonstrates strong internal reliability as the 95% confidence interval for alpha is [0.896, 0.923].

Confidence Interval of Cronbach Alpha

	LCI	alpha	UCI
	0.896	0.91	0.923

3.8 Retention

First, we look at the correlation matrix. The correlation matrix is displaying anywhere from a moderate to strong positive correlation. As a result, we now look at the Cronbach’s alpha to examine internal reliability.

The Cronbach’s Alpha demonstrates an acceptable internal reliability as the 95% confidence interval for alpha is [0.732, 0.810].

Confidence Interval of Cronbach Alpha

	LCI	alpha	UCI
	0.732	0.773	0.81

3.9 How Students Pay for College

First, we look at the correlation matrix. The correlation matrix is displaying anywhere moderate to weak positive and negative correlations. As a result, we now look at the Cronbach’s alpha to examine internal reliability, keeping in mind to reverse variables to maintain positive correlations across the board.

Even after reversing negatively correlated variables, the Cronbach’s alpha displays no internal reliability with a 95% confidence interval of [0.409, 0.577]. Since there are no obvious ways to incorporate the questisons in this subscale, a decision is made to get rid of the subscale; this has obvious implications as it substantially reduces a key aspect of college - how to pay for it.

Confidence Interval of Cronbach Alpha

	LCI	alpha	UCI
	0.409	0.497	0.577

4 PCA to Aggregate Information

4.1 Research Question

We’re going to focus on how student engagement and writing/reading loads affect the response variable of academic standing.

4.2 Preparing functions for PCA

First, we create Scree plots to assess the number of components we’ll choose to keep in each subscale as our component(s).

We then find our factor loadings and proportion variance explained by each factor.

4.3 PCA Extraction

First, we analyze our Scree plots to determine the number of principal components to maintain in our aggregation.

The Scree plot demonstrates that 5 components should be retained for exploratory analysis. The first 4 eigenvalues are higher than the rest, which explains a significant amount of the variance. The fifth component has an eigenvalue where the line begins to flatten, but it should be included to provide additional meaningful variance.

Next, we look at the factor loadings to determine which variables will be extracted specifically.

Factor loadings of the first few PCAs and the cumulative proportion of variation explained by the corresponding PCAs in the Engagement subscale.

	PC1	PC2
q41	0.196	0.116
q42	0.253	0.192
q43	0.211	0.184
q44	0.231	0.209
q45	0.093	-0.391
q46	0.223	0.193
q47	0.239	0.108
q48	0.150	-0.246
q49	0.210	-0.319
q410	0.174	0.154
q411	0.232	0.226
q412	0.266	0.105
q413	0.270	-0.027
q414	0.272	-0.189
q415	0.266	0.174
q416	0.220	0.094
q417	0.237	-0.300
q418	0.257	-0.070
q419	0.185	-0.273
q420	0.199	-0.257
q421	0.035	-0.335

Cumulative and proportion of variances explained by each the principal component in the Engagement subscale.

	PC1	PC2	PC3	PC4	PC5
Standard deviation	2.533	1.366	1.244	1.211	1.071
Proportion of Variance	0.306	0.089	0.074	0.070	0.055
Cumulative Proportion	0.306	0.394	0.468	0.538	0.593

We see that after the 6th component, the proportion of variation drops to under 7% of total variation. The cumulative proportion of variation from the first 4 components gets 53.8% of total variation.

The distributions of the 4 indices are relatively normal.

There are concerns with the 2nd, 3rd, and 4th components as there are correlations that are moderate to strongly negatively correlated; however, if we only relied on the very first index, this would pose issues with the total variation they may explain as the first component only explains 30.6% of the total variation. Depending on the threshold that the client is willing to lose in terms of information, this may or may not be an acceptable amount of information loss.