LLMs and psychometrics - Luis & Louise

1 Data processing

1.1 Packages

pacman::p_load(tidyverse, readxl, janitor,  dplyr, tidyr, ggplot2, openxlsx,  psych, corrplot, factoextra,lavaan)

1.2 Dataframe

df <- read_excel("C:/Users/louis/Downloads/four_countries_data.xlsx")

1.3 Clean data

df = clean_names(df)
df = remove_empty(df)

value for "which" not specified, defaulting to c("rows", "cols")

df = df %>% mutate_all(~ tolower(.) %>% stringi::stri_trans_general(.,"Latin-ASCII"))

1.4 Remove duplicated lines

df <- df[-c(1, 2), ]

1.5 Create a new df only with the items

#important items: Perception of AI - q1_p_ai_1, q2_p_ai_1 and q3_p_ai_1; Adoption of AI - q4_ado_ai_1, q5_ado_ai_1, q6_ado_ai_1. 

df_items <- subset(df, select = c(finished, q1_p_ai_1, q2_p_ai_1, q3_p_ai_1, q4_ado_ai_1, q5_ado_ai_1, q6_ado_ai_1))

1.6 Check - everyone finished the questionnaire?

unique(df_items$finished) #yes, everyone completed the questionnaire

[1] "true"

1.7 Check NA’s

sum(is.na(df_items$q1_p_ai_1))

[1] 55

sum(is.na(df_items$q2_p_ai_1))

[1] 55

sum(is.na(df_items$q3_p_ai_1))

[1] 55

sum(is.na(df_items$q4_ado_ai_1))

[1] 55

sum(is.na(df_items$q5_ado_ai_1))

[1] 55

sum(is.na(df_items$q6_ado_ai_1))

[1] 55

1.8 Rename variables and transform into numeric

df_items <- df_items %>%
  rename(perception_1 = q1_p_ai_1,
         perception_2 = q2_p_ai_1,
         perception_3 = q3_p_ai_1,
         adoption_1 = q4_ado_ai_1,
         adoption_2 = q5_ado_ai_1,
         adoption_3 = q6_ado_ai_1) %>%
  mutate(across(c(perception_1:perception_3, adoption_1:adoption_3), 
                as.numeric)) %>%
  mutate(across(c(perception_1:perception_3, adoption_1:adoption_3), 
                ~ round(., 2)))

1.9 Check variables

df_items %>% names

[1] "finished"     "perception_1" "perception_2" "perception_3" "adoption_1"   "adoption_2"   "adoption_3"  

2 Data analysis

2.1 Data imputation

df_items <- df_items %>%
  mutate(across(perception_1:adoption_3, ~ ifelse(is.na(.), mean(., na.rm = TRUE), .))) %>%
  mutate(across(c(perception_1:perception_3, adoption_1:adoption_3), 
                ~ round(., 2)))

2.2 Internal consistency - AI Perception

df_items %>%
  select(perception_1:perception_3) %>%
  alpha ()

Number of categories should be increased  in order to count frequencies. 

Reliability analysis   
Call: alpha(x = .)

 

    95% confidence boundaries 

 Reliability if an item is dropped:

 Item statistics 

2.3 Internal consistency - AI Adoption

df_items %>%
  select(adoption_1:adoption_3) %>%
  alpha ()

Number of categories should be increased  in order to count frequencies. 

Reliability analysis   
Call: alpha(x = .)

 

    95% confidence boundaries 

 Reliability if an item is dropped:

 Item statistics 

correlation_matrix <- cor(df_items[, c("perception_1", "perception_2", "perception_3")])

corrplot(correlation_matrix, method = "circle", type = "upper", tl.col = "black", tl.srt = 45)

correlation_total <- cor(df_items[, c("perception_1", "perception_2", "perception_3", "adoption_1", "adoption_2", "adoption_3")])

correlation_total

             perception_1 perception_2 perception_3 adoption_1 adoption_2 adoption_3
perception_1    1.0000000    0.1393240    0.1422010  0.1431429  0.1615556  0.1603307
perception_2    0.1393240    1.0000000    0.8191796  0.6211067  0.5757151  0.4947517
perception_3    0.1422010    0.8191796    1.0000000  0.6550311  0.5989819  0.5229324
adoption_1      0.1431429    0.6211067    0.6550311  1.0000000  0.6380226  0.5054160
adoption_2      0.1615556    0.5757151    0.5989819  0.6380226  1.0000000  0.4980452
adoption_3      0.1603307    0.4947517    0.5229324  0.5054160  0.4980452  1.0000000

df_items <- df_items %>% select(-finished) # I removed the questionnaire completion variable

2.4 Exploratory Factor Analysis

fa.parallel(df_items, fa = "fa")

Parallel analysis suggests that the number of factors =  2  and the number of components =  NA 

fa_result <- fa(df_items, nfactors = 2, rotate = "varimax")

print(fa_result)

Factor Analysis using method =  minres
Call: fa(r = df_items, nfactors = 2, rotate = "varimax")
Standardized loadings (pattern matrix) based upon correlation matrix

                       MR1  MR2
SS loadings           1.83 1.55
Proportion Var        0.30 0.26
Cumulative Var        0.30 0.56
Proportion Explained  0.54 0.46
Cumulative Proportion 0.54 1.00

Mean item complexity =  1.6
Test of the hypothesis that 2 factors are sufficient.

df null model =  15  with the objective function =  2.81 with Chi Square =  3039.18
df of  the model are 4  and the objective function was  0 

The root mean square of the residuals (RMSR) is  0.01 
The df corrected root mean square of the residuals is  0.02 

The harmonic n.obs is  1084 with the empirical chi square  3.08  with prob <  0.54 
The total n.obs was  1084  with Likelihood Chi Square =  3.51  with prob <  0.48 

Tucker Lewis Index of factoring reliability =  1.001
RMSEA index =  0  and the 90 % confidence intervals are  0 0.043
BIC =  -24.44
Fit based upon off diagonal values = 1
Measures of factor score adequacy             
                                                   MR1  MR2
Correlation of (regression) scores with factors   0.90 0.79
Multiple R square of scores with factors          0.81 0.63
Minimum correlation of possible factor scores     0.63 0.26

fa.diagram(fa_result)

2.5 Confirmatory Factor Analysis

df_items %>% names

[1] "perception_1" "perception_2" "perception_3" "adoption_1"   "adoption_2"   "adoption_3"  

model <- '
  Factor1 =~ perception_1 + perception_2 + perception_3
  Factor2 =~ adoption_1 + adoption_2 + adoption_3
'

fit <- cfa(model, data = df_items)

summary(fit, fit.measures = TRUE, standardized = TRUE)

lavaan 0.6.17 ended normally after 51 iterations

  Estimator                                         ML
  Optimization method                           NLMINB
  Number of model parameters                        13

  Number of observations                          1084

Model Test User Model:
                                                      
  Test statistic                                17.165
  Degrees of freedom                                 8
  P-value (Chi-square)                           0.028

Model Test Baseline Model:

  Test statistic                              3049.966
  Degrees of freedom                                15
  P-value                                        0.000

User Model versus Baseline Model:

  Comparative Fit Index (CFI)                    0.997
  Tucker-Lewis Index (TLI)                       0.994

Loglikelihood and Information Criteria:

  Loglikelihood user model (H0)             -13870.105
  Loglikelihood unrestricted model (H1)     -13861.523
                                                      
  Akaike (AIC)                               27766.210
  Bayesian (BIC)                             27831.059
  Sample-size adjusted Bayesian (SABIC)      27789.768

Root Mean Square Error of Approximation:

  RMSEA                                          0.033
  90 Percent confidence interval - lower         0.010
  90 Percent confidence interval - upper         0.054
  P-value H_0: RMSEA <= 0.050                    0.906
  P-value H_0: RMSEA >= 0.080                    0.000

Standardized Root Mean Square Residual:

  SRMR                                           0.021

Parameter Estimates:

  Standard errors                             Standard
  Information                                 Expected
  Information saturated (h1) model          Structured

Latent Variables:
                   Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
  Factor1 =~                                                            
    perception_1      1.000                               0.513    0.165
    perception_2      3.859    0.734    5.261    0.000    1.979    0.884
    perception_3      4.156    0.789    5.265    0.000    2.131    0.926
  Factor2 =~                                                            
    adoption_1        1.000                               2.098    0.819
    adoption_2        0.924    0.036   25.926    0.000    1.939    0.769
    adoption_3        0.866    0.041   21.049    0.000    1.817    0.639

Covariances:
                   Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
  Factor1 ~~                                                            
    Factor2           0.924    0.182    5.085    0.000    0.859    0.859

Variances:
                   Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
   .perception_1      9.359    0.403   23.222    0.000    9.359    0.973
   .perception_2      1.099    0.084   13.053    0.000    1.099    0.219
   .perception_3      0.751    0.087    8.635    0.000    0.751    0.142
   .adoption_1        2.156    0.143   15.043    0.000    2.156    0.329
   .adoption_2        2.597    0.148   17.519    0.000    2.597    0.409
   .adoption_3        4.770    0.231   20.614    0.000    4.770    0.591
    Factor1           0.263    0.100    2.626    0.009    1.000    1.000
    Factor2           4.402    0.288   15.304    0.000    1.000    1.000

3 Final conclusion

Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) are both heavily used in psychology and psychometrics. Despite both analyses sharing some common background, we tend to use EFA when we have no prior knowledge about the internal structure of a tool. Therefore, even if we think the tool consists of two constructs (perception and adoption), we believe we don’t have a strong theoretical background to run a CFA on this data. However, if we ask the computer to run the analysis, it will run the analysis. The interpretation of the results, on the other hand, tends to be fragile and can be severely criticized. In summary, we carried out four analyses: (1) we calculated the Cronbach’s alpha of each proposed factor (perception and adoption) and their respective items; (2) we checked the correlations between these items; (3) we carried out an exploratory analysis to explore the nature and number of hidden factors; and (4) we carried out the confirmatory analysis based not on the results of the EFA but on the proposed relationship between all items and their respective constructs. Despite the differences in naming and rationale behind all analyses, these models are very similar and share the same mathematical background, which is based on correlation analysis. Due to computational settings, we used the maximum likelihood estimator. Other estimators could be used to better handle categorical data. Minor issues just occurred. When calculating Cronbach’s alpha, we noticed a discrepancy between the proposed factors of AI Perception and AI Adoption. The perception factor had a lower alpha (0.63) compared to the adoption factor (0.78). This greater variability in the perception scale can be justified by several reasons, which is why we implemented other analyses such as exploratory analysis. In this analysis, we found that item 1 of perception is not related to either the perception or AI adoption factor. Therefore, we do not recommend using this item. Regarding the correlational results, we observed that the correlations between the perception items (perception_1, perception_2, perception_3) and the adoption items (adoption_1, adoption_2, adoption_3) indicate that there is a moderate to strong relationship between items within the same categories (perception or adoption). The correlations are higher among perception items (e.g., 0.819 between perception_2 and perception_3) and among adoption items (e.g., 0.638 between adoption_1 and adoption_2). The correlations between perception items and adoption items are lower, suggesting that although there is some relationship between perception and adoption, they are distinct enough to be considered separate factors.