Prompt: Pick one of the above 3 surveys and pick 2 of the countries (they can be 2 poor countries or 1 poor country and 1 rich country). Analyze the data from the 2 countries you picked (do them separately, don’t try to analyze both countries at once), checking for duplicates as discussed in the above post. Then write up what you found.
For this assignment I chose the 2013 Pew Global Attitudes & Trends (GAT) survey downloaded from “http://www.pewglobal.org/category/datasets/?download=31111”. The two countries I chose to analyze were Israel and Malaysia. For reference, the International Monetary Fund reported in 2015 that the GDP per capita was 35,702 (USD) for Israel and 10,073 (USD) for Malaysia.
The dataset for Israel included 922 observations while that for Malaysia included 822 observations. Both had 761 variables.
#read in 2013 Pew Global Attitudes & Trends survey
worldpew13 <- read.spss("worldpew13.sav", to.data.frame = TRUE)
israeldf <- worldpew13 %>% filter(COUNTRY == "Israel") #my rich country
malaysiadf <- worldpew13 %>% filter(COUNTRY == "Malaysia") #my poor country
#save as RData
save(israeldf, file = "israeldf.RData")
save(malaysiadf, file = "malaysiadf.RData")
load("israeldf.RData")
load("malaysiadf.RData")
The following function, percentMatch, is my attempt to replicate the percentmatch function created for use in STATA by Kuriakose and Robbins (2015). My function is explained via comments at each line of the code. It essentially finds the highest percentage of values that are the same after comparing each row with every other row in the dataset. Note that as this function was computationally extensive, I saved the results of applying it to the GAT datasets for Israel and Malaysia to load them later for the purposes of this report. To provide an idea of the function’s cost, it took about 24 hours for it to complete with the GAT for Israel as the input.
percentMatch <- function(df) {
calculated <- list() #create an empty vector with length nrows(df)
for (baserow in 1:nrow(df)) { #stop baserow before the last row
highest <- 0 #set var to store the highest percentage of value matches per row comparison
for (nextrow in 1:nrow(df)) { #start comparing at the row after baserow
if (nextrow == baserow) next #don't compare baserow with itself
matchsum <- 0 #set var to store all of the match values
for (col in 1:ncol(df)) { #loop through all of the columns
match <- is.element(df[baserow, col], df[nextrow, col]) #compare the same columns
matchsum <- sum(matchsum, match) #add each column match value to the running total
}
if (matchsum > highest) highest <- matchsum #store only the highest match value
}
calculated[baserow] <- ((highest / ncol(df)) * 100) #storing the highest percent match in the calculated vector
}
return(calculated)
}
#test data
df <- read.table(header = TRUE, text = '
label value x
A 4 3
B 3 4
C 6 5
B 9 6
')
#applying percentMatch to datasets and saving to wd
pctmatch_israel <- percentMatch(israeldf)
pctmatch_malaysia <- percentMatch(malaysiadf)
save(pctmatch_malaysia, file = "pctmatch_malaysia.RData")
save(pctmatch_israel, file = "pctmatch_israel.RData")
The histograms above show the distributions of highest percentage matches (with values rounded to the nearest integer) from applying percentMatch to the 2013 GAT surveys for Israel and Malaysia. I’ve made the x and y scales the same in each histogram for ease of comparison. While Kuriakose and Robbins (2015) use 85% as a maximum highest match percentage (where anything above that indicates duplicate rows), I did not alter the datasets in any way here (e.g., removing columns with NAs or calculated columns like sampling weights). I suspect that this is why the median values are about 96% and 95% for the Israel and Malaysia datasets, respectively. With that said, there are a considerable amount of observations that resulted in highest match percentages above the median values. In fact, it looks like for both of the surveys, the majority of the rows had highest match values above the median values.
#Crosstab of percentMatch results for 2013 GAT Israel
table(pctmtch_IL)
## pctmtch_IL
## 94 95 96 97 98 99
## 76 345 367 115 15 4
#same for Malaysia
table(pctmtch_MY)
## pctmtch_MY
## 90 91 92 93 94 95 96 97 98
## 1 2 16 100 224 183 186 72 38
The contingency tables above show that the dataset for Israel had a maximum value of 99% with 4 observations resulting in that highest match percentage. While the dataset for Malaysia had a maximum value of 98%, 38 observations resulted in that highest match percentage. There were no exact matches in either of the datasets. If the cutoff for highest match percentage to indicate duplicates were at 98%, there would be (15+4 / 922obs) 2% of the observations in the dataset for Israel being duplicates and (38 / 822obs) 5% of those in the dataset for Malaysia being duplicates.
The histograms above show the distributions of highest percent matches after removing columns that had NA values for every observation. In addition to removing the columns indicating the country being surveyed as those values were the same for every observation. The final datasets here therefore were only those that could have exhibited variation across the observations.
#Crosstab of percentMatch results for 2013 GAT Israel with NA vars removed
table(clean_IL)
## clean_IL
## 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78
## 1 4 2 7 14 9 22 17 22 40 86 59 60 78 66 60 63 49 32 37 29 31 20 28 22
## 79 80 81 83 84 85 86 87 88 90 96
## 30 11 4 2 2 2 4 4 1 2 2
#duplicate observations with 85% cutoff
nrow(subset(clean_IL, clean_IL[[1]] >= 85))
## [1] 15
#same for Malaysia
table(clean_MY)
## clean_MY
## 42 47 49 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72
## 1 2 1 1 3 4 7 5 18 7 11 17 42 29 25 52 27 28 34 59 28 25 28 43 29
## 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 89 90
## 27 23 47 23 13 34 42 10 4 8 19 8 16 6 9 4 3
nrow(subset(clean_MY, clean_MY[[1]] >= 85))
## [1] 38
The median values are now 68 for the Israel dataset and 69 for the Malaysia dataset. The number of observations with a highest percentage match of 85% or higher was 15 for the Israel dataset and 38 for the Malaysia dataset. Those translate to about 2% of the observations in the Israel dataset and about 5% in the Malaysia dataset. Interestingly, these were the same percentage of duplicates using the 98% cutoff with the unaltered data. However, for logical and computational reasons, I recommend removing columns with all missing values prior to performing the percentMatch function.
In both cases, these findings provide evidence in favor of the claims made by Kuriakose and Robbins (2015) that many international surveys involved about 5% of the dataset being near duplicate observations. To be sure, these conclusions are contingent on the definition of duplicates being a value match of 85% or higher and no perfect matches were found in either dataset.