Nosotros queremos juntar dos conjuntos de datos. Esto significa:
Hay que juntar (merge) ambos conjuntos de datos. Para ello necesitamos que ambas datas tengan las mismas unidades de análisis (en una o más columnas), esas columnas se llaman las claves (keys).
El merge puede perder filas, pues una misma unidad de análisis puede haberse escrito de manera diferente en ambas datas. Eso requiere limpieza de data.
Las datas viene de diferentes fuentes, y tipos de archivos, lo cual requiere subsetear y limpiar hasta que cada data solo tenga las filas apropiadas.
El IDH está disponible como un archivo en Excel que descargaremos desde el link respectivo:
rm(list = ls())
library(rio)
linkToFileIDH="https://hdr.undp.org/sites/default/files/2023-24_HDR/HDR23-24_Statistical_Annex_HDI_Table.xlsx"
# queda en objeto 'idh'
idh=import(linkToFileIDH)El IDE está en una tabla en Wikipedia, lo que requiere scrapping:
library(rvest)
linkToWebIDE="https://en.wikipedia.org/wiki/The_Economist_Democracy_Index"
xpathToTableIDE='//*[@id="mw-content-text"]/div[1]/table[5]'
ide <- read_html(linkToWebIDE)%>%html_nodes(xpath = xpathToTableIDE)%>%html_table()%>% .[[1]]Cada data (dataframe) debe tener las filas de países y los nombres de columnas. Veamos el caso del IDH:
library(magrittr) # needed for pipe %>%
head(idh,10)%>%
rmarkdown::paged_table()El Excel está desordenado. Vemos que los nombres de columnas están en las filas 4 y 5.
idh[4:10,]%>%
rmarkdown::paged_table()Entonces:
names(idh)=idh[4,]
head(idh,10)%>%
rmarkdown::paged_table()Note que la fila cinco indica fecha, pero solo necesitamos los datos desde la fila 8:
idh=idh[-c(1:7),]
head(idh,10)%>%
rmarkdown::paged_table()Notemos que hay nombres de columnas sin valores:
names(idh)## [1] NA
## [2] NA
## [3] "Human Development Index (HDI)"
## [4] "NA"
## [5] "Life expectancy at birth"
## [6] NA
## [7] "Expected years of schooling"
## [8] NA
## [9] "Mean years of schooling"
## [10] NA
## [11] "Gross national income (GNI) per capita"
## [12] NA
## [13] "GNI per capita rank minus HDI rank"
## [14] NA
## [15] "HDI rank"Estos faltan:
names(idh)[1]='hdiRanking'
names(idh)[2]='country'Veamos de quedarnos con los no perdidos:
names(idh)[complete.cases(names(idh))]## [1] "hdiRanking"
## [2] "country"
## [3] "Human Development Index (HDI)"
## [4] "NA"
## [5] "Life expectancy at birth"
## [6] "Expected years of schooling"
## [7] "Mean years of schooling"
## [8] "Gross national income (GNI) per capita"
## [9] "GNI per capita rank minus HDI rank"
## [10] "HDI rank"Estos quedan:
columnasOK=names(idh)[complete.cases(names(idh))]
idh=idh[,columnasOK]
head(idh)Aun hay que borrar otros que no necesitamos:
#quedemonos con lo necesario
idh=idh[,-c(4,9,10)]
#now
head(idh,10)%>%
rmarkdown::paged_table()Veamos los tipos de Datos
str(idh)## 'data.frame': 269 obs. of 7 variables:
## $ hdiRanking : chr "1" "2" "3" "4" ...
## $ country : chr "Switzerland" "Norway" "Iceland" "Hong Kong, China (SAR)" ...
## $ Human Development Index (HDI) : chr "0.96699999999999997" "0.96599999999999997" "0.95899999999999996" "0.95599999999999996" ...
## $ Life expectancy at birth : chr "84.254999999999995" "83.393000000000001" "82.814999999999998" "84.314999999999998" ...
## $ Expected years of schooling : chr "16.5837307" "18.638460160000001" "19.106729510000001" "17.849590299999999" ...
## $ Mean years of schooling : chr "13.904066220000001" "13.062342790000001" "13.76716995" "12.34776974" ...
## $ Gross national income (GNI) per capita: chr "69432.786689999994" "69189.76165" "54688.379209999999" "62485.505160000001" ...Formateando:
idh[,-2]=lapply(idh[,-2],as.numeric)## Warning in lapply(idh[, -2], as.numeric): NAs introducidos por coerción
## Warning in lapply(idh[, -2], as.numeric): NAs introducidos por coerción
## Warning in lapply(idh[, -2], as.numeric): NAs introducidos por coerción
## Warning in lapply(idh[, -2], as.numeric): NAs introducidos por coerciónEl paso anterior produjo valores perdidos:
idh[!complete.cases(idh),]%>%
rmarkdown::paged_table()No hay nada relevante, entonces:
idh=idh[complete.cases(idh),]Ahora pidamos descriptivos:
summary(idh)## hdiRanking country Human Development Index (HDI)
## Min. : 1.00 Length:193 Min. :0.3800
## 1st Qu.: 49.00 Class :character 1st Qu.:0.6020
## Median : 97.00 Mode :character Median :0.7400
## Mean : 96.85 Mean :0.7237
## 3rd Qu.:145.00 3rd Qu.:0.8470
## Max. :193.00 Max. :0.9670
## Life expectancy at birth Expected years of schooling Mean years of schooling
## Min. :53.00 Min. : 5.635 Min. : 1.341
## 1st Qu.:65.91 1st Qu.:11.586 1st Qu.: 6.518
## Median :72.23 Median :13.328 Median : 9.424
## Mean :71.84 Mean :13.535 Mean : 9.017
## 3rd Qu.:77.91 3rd Qu.:15.662 3rd Qu.:11.657
## Max. :84.82 Max. :21.080 Max. :14.256
## Gross national income (GNI) per capita
## Min. : 690.7
## 1st Qu.: 4781.7
## Median : 12360.8
## Mean : 21057.3
## 3rd Qu.: 32171.2
## Max. :146673.2La tabla escrapeada normalmente viene con buena estructura. Veamos los nombres de columnas
names(ide)## [1] "Rank"
## [2] ".mw-parser-output .tooltip-dotted{border-bottom:1px dotted;cursor:help}Δ Rank"
## [3] "Country"
## [4] "Regime type"
## [5] "Overall score"
## [6] "Δ Score"
## [7] "Electoral processand pluralism"
## [8] "Functioningof government"
## [9] "Politicalparticipation"
## [10] "Politicalculture"
## [11] "Civilliberties"Quedémonos con las necesarias:
ide=ide[,-c(2,6)]Verifiquemos tipos de datos:
str(ide)## tibble [172 × 9] (S3: tbl_df/tbl/data.frame)
## $ Rank : chr [1:172] "" "Full democracies" "1" "2" ...
## $ Country : chr [1:172] "" "Full democracies" "Norway" "New Zealand" ...
## $ Regime type : chr [1:172] "" "Full democracies" "Full democracy" "Full democracy" ...
## $ Overall score : chr [1:172] "" "Full democracies" "9.81" "9.61" ...
## $ Electoral processand pluralism: chr [1:172] "" "Full democracies" "10.00" "10.00" ...
## $ Functioningof government : chr [1:172] "" "Full democracies" "9.64" "9.29" ...
## $ Politicalparticipation : chr [1:172] "" "Full democracies" "10.00" "10.00" ...
## $ Politicalculture : chr [1:172] "" "Full democracies" "10.00" "8.75" ...
## $ Civilliberties : chr [1:172] "" "Full democracies" "9.41" "10.00" ...Corrigiendo:
ide[,-c(2,3)]=lapply(ide[,-c(2,3)], as.numeric)## Warning in lapply(ide[, -c(2, 3)], as.numeric): NAs introducidos por coerción
## Warning in lapply(ide[, -c(2, 3)], as.numeric): NAs introducidos por coerción
## Warning in lapply(ide[, -c(2, 3)], as.numeric): NAs introducidos por coerción
## Warning in lapply(ide[, -c(2, 3)], as.numeric): NAs introducidos por coerción
## Warning in lapply(ide[, -c(2, 3)], as.numeric): NAs introducidos por coerción
## Warning in lapply(ide[, -c(2, 3)], as.numeric): NAs introducidos por coerción
## Warning in lapply(ide[, -c(2, 3)], as.numeric): NAs introducidos por coerciónDe nuevo, vemos datos perdidos generados:
ide[!complete.cases(ide),]%>%
rmarkdown::paged_table()No hay nada que perder si eliminamos esas files, entonces:
ide=ide[complete.cases(ide),]El merge requiere columnas en ambos lados que tengan la misma información: escrita de manera exacta. Preventivamente, eliminemos espacios antes y despues de cada valor de celda de las columnas key:
idh$country=trimws(idh$country,whitespace = "[\\h\\v]")
ide$Country=trimws(ide$Country,whitespace = "[\\h\\v]")Verificando intersecciones: El merge usa columnas comunes. Antes del merge definitivo hay que verificar si hay correcciones posible para que el merge no pierda tantas filas.
Una manera práctica para darnos cuenta que NO está coincidiendo en dos conjuntos es usar diferencia de conjuntos 1:
# en ide$Country pero no en idh$country
sort(setdiff(ide$Country,idh$country))## [1] "Bolivia" "Cape Verde"
## [3] "Czech Republic" "Democratic Republic of the Congo"
## [5] "Eswatini" "Hong Kong"
## [7] "Iran" "Ivory Coast"
## [9] "Laos" "Moldova"
## [11] "North Korea" "Palestine"
## [13] "Republic of the Congo" "Russia"
## [15] "South Korea" "Syria"
## [17] "Taiwan" "Tanzania"
## [19] "Turkey" "Venezuela"
## [21] "Vietnam"De igual manera:
# en idh$country pero no en ide$Country
sort(setdiff(idh$country,ide$Country))## [1] "Andorra" "Antigua and Barbuda"
## [3] "Bahamas" "Barbados"
## [5] "Belize" "Bolivia (Plurinational State of)"
## [7] "Brunei Darussalam" "Cabo Verde"
## [9] "Congo" "Congo (Democratic Republic of the)"
## [11] "Côte d'Ivoire" "Czechia"
## [13] "Dominica" "Eswatini (Kingdom of)"
## [15] "Grenada" "Hong Kong, China (SAR)"
## [17] "Iran (Islamic Republic of)" "Kiribati"
## [19] "Korea (Republic of)" "Lao People's Democratic Republic"
## [21] "Liechtenstein" "Maldives"
## [23] "Marshall Islands" "Micronesia (Federated States of)"
## [25] "Moldova (Republic of)" "Nauru"
## [27] "Palau" "Palestine, State of"
## [29] "Russian Federation" "Saint Kitts and Nevis"
## [31] "Saint Lucia" "Saint Vincent and the Grenadines"
## [33] "Samoa" "San Marino"
## [35] "Sao Tome and Principe" "Seychelles"
## [37] "Solomon Islands" "Somalia"
## [39] "South Sudan" "Syrian Arab Republic"
## [41] "Tanzania (United Republic of)" "Tonga"
## [43] "Türkiye" "Tuvalu"
## [45] "Vanuatu" "Venezuela (Bolivarian Republic of)"
## [47] "Viet Nam"Se puede corroborar que sí hay valores que pueden ser corregidos. Hagamos los cambios en idh:
idh[idh$country=='Bolivia (Plurinational State of)','country']='Bolivia'
idh[idh$country=='Cabo Verde','country']='Cape Verde'
idh[idh$country=='Czechia','country']='Czech Republic'
idh[idh$country=='Congo (Democratic Republic of the)','country']='Democratic Republic of the Congo'
idh[idh$country=='Eswatini (Kingdom of)','country']='Eswatini'
idh[idh$country=='Hong Kong, China (SAR)','country']='Hong Kong'
idh[idh$country=='Iran (Islamic Republic of)','country']='Iran'
idh[idh$country=="Côte d'Ivoire",'country']='Ivory Coast'
idh[idh$country=="Lao People's Democratic Republic",'country']='Laos'
idh[idh$country=="Moldova (Republic of)",'country']='Moldova'
idh[idh$country=="Palestine, State of",'country']='Palestine'
idh[idh$country=='Congo','country']='Republic of the Congo'
idh[idh$country=='Russian Federation','country']='Russia'
idh[idh$country=='Korea (Republic of)','country']='South Korea'
idh[idh$country=='Syrian Arab Republic','country']='Syria'
idh[idh$country=='Tanzania (United Republic of)','country']='Tanzania'
idh[idh$country=='Türkiye','country']='Turkey'
idh[idh$country=='Venezuela (Bolivarian Republic of)','country']='Venezuela'
idh[idh$country=='Viet Nam','country']='Vietnam'Ahora si hay más comodidad para hacer el merge:
idhdemo=merge(idh,ide, by.x = "country", by.y = 'Country')
idhdemo%>%
rmarkdown::paged_table()Ajustemos sus nombres:
# actual
names(idhdemo)## [1] "country"
## [2] "hdiRanking"
## [3] "Human Development Index (HDI)"
## [4] "Life expectancy at birth"
## [5] "Expected years of schooling"
## [6] "Mean years of schooling"
## [7] "Gross national income (GNI) per capita"
## [8] "Rank"
## [9] "Regime type"
## [10] "Overall score"
## [11] "Electoral processand pluralism"
## [12] "Functioningof government"
## [13] "Politicalparticipation"
## [14] "Politicalculture"
## [15] "Civilliberties"# nuevos nombres
newNames=c("hdi","hdiLife","hdiSchoolExpec","hdiMeanEduc","hdiGni",
'ideRanking','ideRegime','ide','ideElectoral','ideFunctioning','ideParticipation','ideCulture','ideLiberties')
names(idhdemo)[3:15]=newNamesTenemos ya un data frame que integra diversas variables que quieren medir conceptos complejos (latentes). Guardemos los datos mergeados:
write.csv(idhdemo,"idhdemo.csv",row.names = F)…y veamos cómo se les usará en el análisis factorial.
Sí A y B son conjuntos, A−B serán los elementos que están en A pero que NO están en B, por ejemplo si A={1,2,3} y B={3,5}, entonces A−B={1,2}.↩︎