Questionnaire Validation

Descriptive Stat

Perceived performance questions

Data summary

Name	Perform[[“01-data”]]
Number of rows	100
Number of columns	15
_______________________
Column type frequency:
numeric	15
________________________
Group variables	None

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
Q1	0	1	3.83	1.01	1	3	4	5.00	5	▁▁▃▇▅
Q2	0	1	3.82	0.97	1	3	4	5.00	5	▁▂▅▇▅
Q3	0	1	3.90	0.97	2	3	4	5.00	5	▂▅▁▇▇
Q4	0	1	3.80	0.97	1	3	4	5.00	5	▁▂▅▇▅
Q5	0	1	3.98	1.00	1	3	4	5.00	5	▁▂▅▇▇
E1	0	1	3.78	1.01	1	3	4	5.00	5	▁▂▅▇▆
E2	0	1	3.82	0.97	1	3	4	5.00	5	▁▂▅▇▆
E3	0	1	3.84	1.02	1	3	4	5.00	5	▁▂▅▇▇
E4	0	1	3.74	0.93	2	3	4	4.00	5	▂▅▁▇▃
E5	0	1	3.81	0.97	2	3	4	5.00	5	▂▇▁▇▇
T1	0	1	3.83	0.99	1	3	4	5.00	5	▁▁▅▇▆
T2	0	1	3.79	0.97	1	3	4	5.00	5	▁▂▆▇▆
T3	0	1	3.72	0.94	1	3	4	4.00	5	▁▁▇▇▅
T4	0	1	3.79	0.96	1	3	4	4.25	5	▁▂▅▇▅
T5	0	1	3.64	0.94	1	3	4	4.00	5	▁▂▇▇▃

Job satisfaction questions

Data summary

Name	Satisfy[[“01-data”]]
Number of rows	100
Number of columns	34
_______________________
Column type frequency:
numeric	34
________________________
Group variables	None

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
CF1	0	1	3.70	1.23	1	2	4	5.00	5	▁▆▁▇▇
CF2	0	1	3.63	1.29	1	2	4	5.00	5	▁▇▁▇▇
CF3	0	1	3.67	1.24	1	2	4	5.00	5	▁▆▁▇▇
CF4	0	1	3.63	1.26	1	2	4	5.00	5	▁▆▁▇▇
L1	0	1	3.47	1.32	1	2	4	5.00	5	▂▆▂▇▆
L2	0	1	3.31	1.38	1	2	4	4.25	5	▂▇▁▇▆
L3	0	1	3.26	1.42	1	2	4	4.25	5	▃▇▁▇▆
L4	0	1	3.54	1.39	1	2	4	5.00	5	▂▅▁▇▇
L5	0	1	3.43	1.37	1	2	4	5.00	5	▂▇▁▇▇
E2	0	1	3.48	1.30	1	2	4	5.00	5	▁▇▁▇▆
E1	0	1	3.61	1.20	1	2	4	5.00	5	▁▆▁▇▅
E3	0	1	3.55	1.21	1	2	4	5.00	5	▁▇▁▇▆
E4	0	1	3.58	1.22	1	2	4	5.00	5	▁▆▁▇▅
E5	0	1	3.89	1.12	1	4	4	5.00	5	▁▃▁▇▆
PA1	0	1	3.53	1.26	1	2	4	5.00	5	▁▇▁▇▆
PA2	0	1	3.49	1.29	1	2	4	5.00	5	▁▇▁▇▆
PA3	0	1	3.58	1.26	1	2	4	5.00	5	▁▇▁▇▇
PA4	0	1	3.44	1.31	1	2	4	5.00	5	▁▇▁▇▆
PA5	0	1	3.62	1.23	1	2	4	5.00	5	▁▅▁▇▅
I1	0	1	3.50	1.28	1	2	4	5.00	5	▁▇▁▇▆
I2	0	1	3.44	1.27	1	2	4	4.00	5	▁▇▁▇▅
I3	0	1	3.52	1.26	1	2	4	5.00	5	▁▇▁▇▆
I4	0	1	3.00	0.89	1	2	3	4.00	4	▁▇▁▆▇
I5	0	1	3.32	1.32	1	2	4	4.25	5	▁▇▁▆▅
ED1	0	1	3.53	1.24	1	2	4	5.00	5	▁▇▁▇▆
ED2	0	1	3.48	1.27	1	2	4	5.00	5	▁▇▁▇▆
ED3	0	1	3.47	1.23	1	2	4	4.00	5	▁▇▁▇▅
ED4	0	1	3.48	1.27	1	2	4	5.00	5	▁▇▁▇▆
ED5	0	1	3.39	1.22	1	2	4	4.00	5	▁▇▁▇▅
RM1	0	1	3.62	1.21	1	2	4	5.00	5	▁▆▁▇▆
RM2	0	1	3.60	1.17	2	2	4	4.25	5	▆▁▁▇▅
RM3	0	1	3.61	1.19	1	2	4	4.25	5	▁▅▁▇▅
RM4	0	1	3.65	1.18	2	2	4	5.00	5	▆▁▁▇▆
RM5	0	1	3.62	1.20	1	2	4	5.00	5	▁▆▁▇▆

Interpretation:

Descriptive Stat shows summary statistics on the collected data.

Construct Validity

Construct validity evaluates whether a survey or test measures the theoretical construct it intends to measure.
Factor Analysis is a common statistical method used to check construct validity by identifying underlying structures in the data.

Exploratory Factor Analysis

EFA will be used to test whether the set of questions are measuring the same construct

Step 1: Checking Sampling Adequacy

Factor Analysis requires:

1️⃣ Kaiser-Meyer-Olkin (KMO) test → Checks sample adequacy (should be ≥ 0.60).

[1] "Perceived performance"

Kaiser-Meyer-Olkin factor adequacy
Call: KMO(r = Perform[["01-data"]])
Overall MSA =  0.95
MSA for each item = 
  Q1   Q2   Q3   Q4   Q5   E1   E2   E3   E4   E5   T1   T2   T3   T4   T5 
0.93 0.93 0.97 0.91 0.95 0.96 0.95 0.96 0.96 0.93 0.98 0.96 0.95 0.94 0.95 

Perceived performance

	MSA
Q1	0.9256642
Q2	0.9315976
Q3	0.9687490
Q4	0.9084615
Q5	0.9510895
E1	0.9626467
E2	0.9463807
E3	0.9578144
E4	0.9587813
E5	0.9334827
T1	0.9751672
T2	0.9622190
T3	0.9534971
T4	0.9375298
T5	0.9529314
Overall	0.9480763

Job Satisfaction

	MSA
CF1	0.9657551
CF2	0.9335115
CF3	0.9314618
CF4	0.9588260
L1	0.8895295
L2	0.9237876
L3	0.8844444
L4	0.9548260
L5	0.9216280
E2	0.9443449
E1	0.9309878
E3	0.9537231
E4	0.9221422
E5	0.9538823
PA1	0.9668021
PA2	0.9559752
PA3	0.9086324
PA4	0.9683483
PA5	0.9664113
I1	0.9270253
I2	0.8873321
I3	0.9223565
I4	0.9556711
I5	0.9616464
ED1	0.8940627
ED2	0.9139832
ED3	0.9135873
ED4	0.9381815
ED5	0.9606524
RM1	0.9447022
RM2	0.9466630
RM3	0.9490725
RM4	0.9150753
RM5	0.9109256
Overall	0.9334225

Interpretation:

KMO test on sampling adequacy shows that the overall MSA of questions on Perceived performance and Job Satisfaction, which are 0.9334225 and 0.9480763, respectively, are both >= 0.60. This means that we can perform EFA on the survey data on Perceived Performance and Job Satisfaction and that that the sample of 100 are considered adequate for Exploratory Factor Analysis.

2️⃣ Bartlett’s test → Checks if variables are correlated (p-value should be < 0.05).

Perceived performance

Chi_sq	p.value	df
2264.652	0	105

Job Satisfaction

Chi_sq	p.value	df
5940.36	0	561

Interpretation:

Bartlett’s test show that both the data on Perceived Performance (p-value = 0 ) and Job Satisfaction (p-value = 0 ) have p-value of less than 0.05. This means that questions are in fact, correlated, which may also imply that they measure their respective major construct.

Step 2: Check optimal number of factors (constructs)

Parallel analysis suggests that the number of factors =  1  and the number of components =  1 

Parallel analysis suggests that the number of factors =  1  and the number of components =  1 

Interpretation:

Scree plot shows that the ideal number of significant factor/s for Perceived Performance and Job Satisfaction are both 1.

Step 3: Perform EFA

Perceived performance

Factor loadings

item	PA1	PA2	PA3
Q1	0.834	0.295	0.048
Q2	0.841	0.247	0.077
Q3	0.833	0.238	0.111
Q4	0.864	0.062	0.231
Q5	0.85	0.083	0.26
E1	0.927	-0.139	0.115
E2	0.964	-0.134	0.034
E3	0.932	-0.108	0.008
E4	0.939	-0.095	-0.04
E5	0.928	-0.095	0.014
T1	0.945	-0.049	-0.1
T2	0.935	-0.092	-0.122
T3	0.894	0.008	-0.134
T4	0.889	0.109	-0.246
T5	0.9	0.066	-0.092

Factor summary statistcs

	PA1	PA2	PA3
SS loadings	12.206	0.381	0.292
Proportion Var	0.814	0.025	0.019
Cumulative Var	0.814	0.839	0.859
Proportion Explained	0.948	0.030	0.023
Cumulative Proportion	0.948	0.977	1.000

Job Satisfcation

Factor loadings

item	PA4	PA6	PA2	PA5	PA1	PA3	PA7
CF1	0.081	0.13	0.261	0.467	-0.006	0.187	-0.042
CF2	-0.05	0.118	0.174	0.637	0.118	0.142	-0.003
CF3	0.082	0.013	0.025	0.739	0.057	0.028	0.069
CF4	0.206	0.171	0.341	0.367	-0.185	-0.023	0.192
L1	0.191	0.067	0.437	0.215	-0.056	0.062	0.173
L2	0.096	0.146	0.623	0.088	0.051	-0.025	0.111
L3	0.046	0.049	0.681	0.124	0.159	0.065	0.016
L4	0.159	-0.004	0.459	0.128	0.007	0.331	0.036
L5	0.142	0.121	0.441	0.156	0.128	0.02	0.181
E2	0.008	0.178	0.276	-0.042	0.033	0.31	0.427
E1	0.036	0.261	0.081	0.085	0.159	0.297	0.196
E3	-0.143	0.178	0.177	0.068	0.248	0.443	0.205
E4	0.139	0.111	0.074	-0.002	0.028	0.715	-0.043
E5	0.116	0.019	-0.164	0.281	-0.096	0.6	0.144
PA1	0.185	0.252	-0.007	0.141	0.264	0.024	0.33
PA2	0.164	0.174	0.042	0.222	0.309	0.047	0.22
PA3	0.284	0.124	-0.025	0.181	0.322	0.062	0.248
PA4	0.316	0.048	0.063	0.18	0.311	0.078	0.156
PA5	0.547	-0.049	0.099	-0.031	0.133	0.232	0.165
I1	0.32	0.187	0.092	0.188	0.229	-0.078	0.192
I2	0.023	0.098	0.183	0.165	0.569	0.029	0.126
I3	0.179	0.123	0.147	0.163	0.445	-0.047	0.147
I4	0.198	0.128	0.067	-0.024	0.44	0.284	-0.029
I5	0.255	0.285	0.209	0	0.324	0.023	0.017
ED1	0.255	0.357	-0.005	0.172	0.323	0.093	-0.086
ED2	0.19	0.563	0.03	0.177	0.137	0.029	-0.049
ED3	-0.005	0.89	0.072	-0.043	0.005	0.073	0.015
ED4	-0.009	1.008	-0.038	0.029	-0.025	-0.006	0.033
ED5	0.234	0.393	0.238	0.014	0.164	0.029	0.028
RM1	0.68	0.067	0.032	0.035	0.043	0.019	0.226
RM2	0.526	0.093	0.012	0.217	0.166	0.037	0.031
RM3	0.755	0.091	0.063	0.021	0.01	0.087	-0.021
RM4	0.7	0.142	-0.002	0.139	0.106	0.053	-0.087
RM5	0.643	0.029	0.247	0.027	-0.052	0.122	0.083

Factor summary statistcs

	PA4	PA6	PA2	PA5	PA1	PA3	PA7
SS loadings	6.384	5.689	4.330	4.218	3.735	3.129	2.332
Proportion Var	0.188	0.167	0.127	0.124	0.110	0.092	0.069
Cumulative Var	0.188	0.355	0.482	0.607	0.716	0.808	0.877
Proportion Explained	0.214	0.191	0.145	0.141	0.125	0.105	0.078
Cumulative Proportion	0.214	0.405	0.550	0.692	0.817	0.922	1.000

Interpretation:

Using EFA, results showed that for Perceived performance questions, only one factor explained 80% of the total variance. This suggests that all fifteen questions are measuring just one construct. Additional factors/constructs may not be as meaningful ( SS Loadings< 1 ). It can also be seen that only the first construct have high proportion of variance ( around 81% ) while other factors have very low share of variance ( <1% ). This may suggest revising the factor structure.

For questions on Job Satisfaction, variance is too spread across 7 factors, which may suggest overfactoring. Consider decrease the number of factors when some of the grouped questions does not uniquely identify to one construct. Factors may be poorly defined and items are not well-grouped into 7 factors.

Confirmatory Factor Analysis

CFA for Perceived performance

CFA Factor loadings

Factor	Item	Estimate	SE	p_value
Quality	Q1	1	0.000	NA
Quality	Q2	0.963	0.044	0
Quality	Q3	0.961	0.054	0
Quality	Q4	0.95	0.053	0
Quality	Q5	0.985	0.054	0
Efficiency	E1	1	0.000	NA
Efficiency	E2	0.997	0.059	0
Efficiency	E3	1.026	0.061	0
Efficiency	E4	0.934	0.052	0
Efficiency	E5	0.968	0.053	0
Timeliness	T1	1	0.000	NA
Timeliness	T2	0.962	0.049	0
Timeliness	T3	0.919	0.067	0
Timeliness	T4	0.935	0.051	0
Timeliness	T5	0.926	0.056	0

CFA Fit indices

	Index	Value
chisq	chisq	173.170
df	df	87.000
pvalue	pvalue	0.000
cfi	cfi	0.963
tli	tli	0.955
rmsea	rmsea	0.100
srmr	srmr	0.019

Interpretation

Estimates of CFA factor loadings shows that each question in the Perceived performance strongly correlates to their respective latent factors ( factor loadings of >0.6 ).

CFA for Job Satisfaction

CFA Factor loadings

Factor	Item	Estimate	SE	p_value
Customer Focus	CF1	1	0.000	NA
Customer Focus	CF2	1.09	0.050	0
Customer Focus	CF3	0.939	0.080	0
Customer Focus	CF4	0.996	0.070	0
Leadership	L1	1	0.000	NA
Leadership	L2	1.084	0.054	0
Leadership	L3	1.134	0.067	0
Leadership	L4	1.04	0.076	0
Leadership	L5	1.115	0.062	0
Engagement of the People	E1	1	0.000	NA
Engagement of the People	E2	1.15	0.073	0
Engagement of the People	E3	1.042	0.084	0
Engagement of the People	E4	0.933	0.079	0
Engagement of the People	E5	0.749	0.091	0
Process Approach	PA1	1	0.000	NA
Process Approach	PA2	1.025	0.037	0
Process Approach	PA3	1.016	0.036	0
Process Approach	PA4	1.002	0.042	0
Process Approach	PA5	0.874	0.063	0
Evidence based Decision Making	ED1	1	0.000	NA
Evidence based Decision Making	ED2	1.045	0.044	0
Evidence based Decision Making	ED3	0.97	0.069	0
Evidence based Decision Making	ED4	1.023	0.062	0
Evidence based Decision Making	ED5	0.953	0.060	0
Improvement	I1	1	0.000	NA
Improvement	I2	1.013	0.034	0
Improvement	I3	0.998	0.042	0
Improvement	I4	0.609	0.043	0
Improvement	I5	1.011	0.043	0
Relationship Management	RM1	1	0.000	NA
Relationship Management	RM2	0.965	0.035	0
Relationship Management	RM3	0.964	0.033	0
Relationship Management	RM4	0.988	0.030	0
Relationship Management	RM5	0.972	0.036	0

CFA Fit indices

	Index	Value
chisq	chisq	1427.016
df	df	506.000
pvalue	pvalue	0.000
cfi	cfi	0.853
tli	tli	0.837
rmsea	rmsea	0.135
srmr	srmr	0.034

Interpretation

Estimates of CFA factor loadings shows that each question in the Job Satisfaciton strongly correlates to their respective latent factors ( factor loadings of >0.6 ).

Convergence and Discriminant Validity

Average Variance Extracted (AVE)

AVE measures how much variance in indicators is explained by the latent factor

Convergent validity of Perceived performance questions

	x
Quality	0.8729198
Efficiency	0.8526864
Timeliness	0.8295374

Convergent validity of Job Satisfaction questions

	x
Customer Focus	0.8387971
Leadership	0.8595479
Engagement of the People	0.7362619
Process Approach	0.8663970
Evidence based Decision Making	0.8866793
Improvement	0.8654847
Relationship Management	0.8771183

Interpretation

Estimated AVE for each factor shows that items under each factor have high convergent validity. This means that items are strongly related and that participants interpret the items consistently

Composite Reliability

CR measures construct reliability and internal consistency

Convergent validity of Perceived performance questions

	x
Quality	0.9718841
Efficiency	0.9667543
Timeliness	0.9600002

Convergent validity of Job Satisfaction questions

	x
Customer Focus	0.9557580
Leadership	0.9691124
Engagement of the People	0.9215045
Process Approach	0.9722438
Evidence based Decision Making	0.9753546
Improvement	0.9714854
Relationship Management	0.9723574

Interpretation:

Results show factors have high Composite reliability which implies that all items consistently measure their respective construct.

Fornell-Lacker Criterion (AVE > squared correlation)

A construct has good discriminant validity if the AVE is greater than the squared correlations.

Correlation Matrix for Perceived performance questions

	Quality	Efficiency	Timeliness
Quality	1.00	0.870	0.860
Efficiency	0.87	1.000	0.948
Timeliness	0.86	0.948	1.000

Correlation Matrix for Job Satisfaction questions

	Customer Focus	Leadership	Engagement of the People	Process Approach	Evidence based Decision Making	Improvement	Relationship Management
Customer Focus	1.000	0.871	0.784	0.825	0.801	0.785	0.787
Leadership	0.871	1.000	0.837	0.818	0.781	0.829	0.781
Engagement of the People	0.784	0.837	1.000	0.828	0.782	0.772	0.733
Process Approach	0.825	0.818	0.828	1.000	0.880	0.939	0.879
Evidence based Decision Making	0.801	0.781	0.782	0.880	1.000	0.892	0.831
Improvement	0.785	0.829	0.772	0.939	0.892	1.000	0.836
Relationship Management	0.787	0.781	0.733	0.879	0.831	0.836	1.000

Interpretation

Looking at the squared correlation matrix, if the computed AVE is higher than the squared correlation, then the construct have good discriminant validity with other constructs. If the AVE is lower, then the constructs or factors are not distinct.

Reliablity testing

Reliability testing evaluates how consistently a set of items measures a construct

Cronbach’s Alpha - Internal Consistency

Reliablity analysis for Perceived performance questions

	Overall	Quality	Efficiency	Timeliness
raw_alpha	0.9853	0.9716	0.9663	0.9604
std.alpha	0.9854	0.9716	0.9667	0.9604
G6(smc)	0.9895	0.9691	0.9633	0.9538
average_r	0.8181	0.8726	0.8529	0.8292
S/N	67.4514	34.2605	28.9933	24.2699
ase	0.0022	0.0045	0.0054	0.0063
mean	3.8060	3.8660	3.7980	3.7540
sd	0.8877	0.9329	0.9205	0.8903
median_r	0.8140	0.8807	0.8481	0.8318

Reliablity analysis for Job Satisfaction questions

	Overall	Customer.focus	Leadership	Engagement.of.the.people	Process.Approach	Evidence.based.decision.making	Improvement	Relationship.Management
raw_alpha	0.9914	0.9524	0.9675	0.9065	0.9686	0.9747	0.9611	0.9728
std.alpha	0.9915	0.9524	0.9676	0.9063	0.9685	0.9746	0.9637	0.9728
G6(smc)	0.9972	0.9477	0.9682	0.8840	0.9649	0.9755	0.9627	0.9682
average_r	0.7744	0.8333	0.8564	0.7073	0.8602	0.8849	0.8415	0.8775
S/N	116.7064	19.9888	29.8272	9.6669	30.7592	38.4344	26.5555	35.8299
ase	0.0012	0.0079	0.0052	0.0152	0.0051	0.0041	0.0054	0.0043
mean	3.5188	3.6575	3.4020	3.6575	3.5320	3.4700	3.3560	3.6200
sd	1.1065	1.1740	1.2957	1.0497	1.1970	1.1866	1.1316	1.1290
median_r	0.7804	0.8274	0.8619	0.7234	0.8653	0.8728	0.8526	0.8695

Results showed the computed Cronbach’s Alpha for each set of questions. Consider the following criteria for interpretation:

Alpha value	Interpretation
>0.9	Excellent reliability
0.80 to 0.89	Good reliability
0.70 to 0.79	Acceptable reliability
0.60 to 0.69	Questionable reliability
<0.60	Poor reliability

McDonald Omega

Omega values for Perceived performance questions

measure	Quality	Efficiency	Timeliness
omega	0.9717	0.9666	0.9605
omega2	0.9717	0.9666	0.9605
omega3	0.9719	0.9668	0.9600

Omega values for Job Satisfaction questions

measure	Customer Focus	Leadership	Engagement of the People	Process Approach	Evidence based Decision Making	Improvement	Relationship Management
omega	0.9540	0.9683	0.9320	0.9700	0.9750	0.9690	0.9727
omega2	0.9540	0.9683	0.9320	0.9700	0.9750	0.9690	0.9727
omega3	0.9558	0.9691	0.9215	0.9722	0.9754	0.9715	0.9724

Results showed the computed McDonald’s Omega for each set of questions. Consider the following criteria for interpretation:

Omega value	Interpretation
>0.9	Excellent reliability
0.80 to 0.89	Good reliability
0.70 to 0.79	Acceptable reliability
0.60 to 0.69	Poor reliability

Test Retest Reliablity using Intraclass Correlation Coefficient (ICC)

 Single Score Intraclass Correlation

   Model: twoway 
   Type : consistency 

   Subjects = 100 
     Raters = 15 
   ICC(C,1) = 0.818

 F-Test, H0: r0 = 0 ; H1: r0 > 0 
 F(99,1386) = 68.2 , p = 0 

 95%-Confidence Interval for ICC Population Values:
  0.773 < ICC < 0.86

 Single Score Intraclass Correlation

   Model: twoway 
   Type : consistency 

   Subjects = 100 
     Raters = 34 
   ICC(C,1) = 0.773

 F-Test, H0: r0 = 0 ; H1: r0 > 0 
 F(99,3267) = 117 , p = 0 

 95%-Confidence Interval for ICC Population Values:
  0.723 < ICC < 0.822

Results showed the computed Test Retest reliablity using ICC for each set of questions. Consider the following criteria for interpretation:

ICC value	Interpretation
>0.9	Excellent reliability
0.75 to 0.89	Good reliability
0.50 to 0.74	Acceptable reliability
0.60 to 0.69	Poor reliability