GOOGLE IMAGES
1 Introduction
In the age of information, as you might know the amount of data available is expansive and growing exponentially. This leads to the need of data services like Microsoft Azure, Amazon AWS, and Google Cloud. Services that let any business store and run analysis on the data to see what works best for their company by tracking views, purchases, and even to test advertising. I personally have not explored any of these products until the start of this project. I wanted to get a more aqquainted with these services and choose Google’s Cloud Services to start with because they had some datasets that were already available to do projects on in Google’s Big Query. This is great because I don’t have insanely large data sets just laying around to test on! Yet. If you want to follow along and dip your toes into a major cloud service I will go step-by-step below. To get started, you will need to setup a free account with Google’s Big Query.
Click [HERE] to start an account with Google Cloud and Big Query!
2 Goal of Project
The goal of this project is to demonstrate how anyone can utilize a cloud service and analyze the information from it. I will take a particular dataset already on the cloud and run a simple query on it to reduce the dimensions of the dataset to something simple. I will then visualize the data, run some machine learning models against the data for predictive analytics analysis on the data with an added visual for easy interpretation.
3 Data
The data I will use for the project is the Census_adult_income, which is located in the ml_datasets.
3.1 Data Definitions
From the project we are only going to utilize three variables to make things simple. The three variables used are income_bracket, sex, and education_num.
- income_bracket: Either “>50K” or “<=50K” based on income.
- sex: Gender of the observation.
- education_num: Estimated years of education completed based on the value of the education field.
List of all of the data variables, types, and definitions of the set.
3.2 Data Query Process
Big Query supports two dialects of the coding language SQL, both standard and legacy. This is the query used to select the variables we will be working with. After running the query you can download the data to run analysis on.
3.3 Data Table
Below is a sample view of the data that was imported.
Now that we have data, we need to analyze it. To do this we need to make sure it is complete and of the proper type. Just looking at the table we see that two of the variables are of the character type. Since these columns only have two responses, the data will work better as binary.
Check data for missing observations. 
There isn’t any missing observations in the entire data frame. Next step will be to alter the data types so we can work with it a little easier. This can be done by converting the categorical variables from character into binary format to make the observations numeric.
# if-else statement to make observations more than 50K equal to 1, less than 50K to 0, and convert to integer.
df$income_bracket_bin <- as.integer(ifelse(df$income_bracket == " >50K", "1", "0"))
# if-else statement to make observations male equal to 1, female to 0, and convert to integer.
df$sex_bin <- as.integer(ifelse(df$sex == " Male", "1", "0"))The data now seems to be in order. Now, to visualize the data.
3.4 Visual Representation of Data
3.4.1 Violin Plot
3.4.2 Stacked Barplot
4 Analysis
For fun, we can try to predict the income bracket people will fall into based on sex and education using a few different machine learning models. To do so, we will need to split the data into training and testing. I organized the visuals for the different machine learning models in decreasing order by accuracy.
4.1 Machine learning Visuals
4.1.1 Neural Network Plot
## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 4465 884
## 1 479 684
##
## Accuracy : 0.7907
## 95% CI : (0.7806, 0.8005)
## No Information Rate : 0.7592
## P-Value [Acc > NIR] : 8.787e-10
##
## Kappa : 0.3722
##
## Mcnemar's Test P-Value : < 2.2e-16
##
## Precision : 0.5881
## Recall : 0.4362
## F1 : 0.5009
## Prevalence : 0.2408
## Detection Rate : 0.1050
## Detection Prevalence : 0.1786
## Balanced Accuracy : 0.6697
##
## 'Positive' Class : 1
## 4.1.2 Decision Tree
## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 4471 894
## 1 473 674
##
## Accuracy : 0.7901
## 95% CI : (0.78, 0.7999)
## No Information Rate : 0.7592
## P-Value [Acc > NIR] : 1.828e-09
##
## Kappa : 0.3679
##
## Mcnemar's Test P-Value : < 2.2e-16
##
## Sensitivity : 0.9043
## Specificity : 0.4298
## Pos Pred Value : 0.8334
## Neg Pred Value : 0.5876
## Prevalence : 0.7592
## Detection Rate : 0.6866
## Detection Prevalence : 0.8239
## Balanced Accuracy : 0.6671
##
## 'Positive' Class : 0
## 4.1.3 Naive Bayes Plot
## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 4908 1438
## 1 36 130
##
## Accuracy : 0.7736
## 95% CI : (0.7633, 0.7838)
## No Information Rate : 0.7592
## P-Value [Acc > NIR] : 0.003202
##
## Kappa : 0.1089
##
## Mcnemar's Test P-Value : < 2.2e-16
##
## Precision : 0.78313
## Recall : 0.08291
## F1 : 0.14994
## Prevalence : 0.24079
## Detection Rate : 0.01996
## Detection Prevalence : 0.02549
## Balanced Accuracy : 0.53781
##
## 'Positive' Class : 1
## We can somewhat accurately predict income bracket based on gender and education level. This means that there is some correlation between the variables. Normally you could just use the pairs function to view the correlation between the variables, but since the variables are mostly binary it will not have much readability. The question I am most curious about with this data, that has not been answered yet, would be the proportions of the two variables that are in a income bracket above fifty-thousand. This may be best observed with the use of tables.
4.2 Proportion Tables
4.2.1 Gender of Income Above 50K
## [1] "Counts"##
## Female Male
## FALSE 9592 15128
## TRUE 1179 6662## [1] "Percentages"##
## Female Male
## FALSE 29.458555 46.460490
## TRUE 3.620896 20.460060Conclusion
Nearly half of the observations are male above 50K income level. There is some bias here as the proportion of men to women has a ratio of almost 2:1. The ratios of sex in the higher income differ quite a bit. For female, it is 1:10 of >50K to <=50K. For Male, it is 1:2. This means that for the genders, only a 10th of the female population make a great wage while nearly a third of the male population is above 50K of income.
4.2.2 Education Level of Income Above 50K
## [1] "Counts"##
## 1 2 3 4 5 6 7 8 9 10 11 12 13 14
## FALSE 51 162 317 606 487 871 1115 400 8826 5904 1021 802 3134 764
## TRUE 0 6 16 40 27 62 60 33 1675 1387 361 265 2221 959
##
## 15 16
## FALSE 153 107
## TRUE 423 306## [1] "Percentages"##
## 1 2 3 4 5 6
## FALSE 0.15662910 0.49752772 0.97355732 1.86112220 1.49565431 2.67497927
## TRUE 0.00000000 0.01842695 0.04913854 0.12284635 0.08292129 0.19041184
##
## 7 8 9 10 11 12
## FALSE 3.42434200 1.22846350 27.10604711 18.13212125 3.13565308 2.46306932
## TRUE 0.18426952 0.10134824 5.14419090 4.25969718 1.10868831 0.81385707
##
## 13 14 15 16
## FALSE 9.62501152 2.34636528 0.46988729 0.32861399
## TRUE 6.82104358 2.94524124 1.29910015 0.93977458Conclusion
Most of the population has a education level between 9 and 14. For the most part, the ratio for the same education numbers are 1:4, the 1 being those in the income bracket above 50K.
4.2.3 Education Level of Income Above 50K and Male
## [1] "Counts"##
## 1 2 3 4 5 6 7 8 9 10 11 12 13 14
## FALSE 51 162 319 607 492 874 1123 404 9052 6101 1088 858 3473 943
## TRUE 0 6 14 39 22 59 52 29 1449 1190 294 209 1882 780
##
## 15 16
## FALSE 195 157
## TRUE 381 256## [1] "Percentages"##
## 1 2 3 4 5 6
## FALSE 0.15662910 0.49752772 0.97969964 1.86419336 1.51101010 2.68419275
## TRUE 0.00000000 0.01842695 0.04299622 0.11977519 0.06756549 0.18119837
##
## 7 8 9 10 11 12
## FALSE 3.44891127 1.24074813 27.80012899 18.73713952 3.34142072 2.63505421
## TRUE 0.15970025 0.08906360 4.45010903 3.65467891 0.90292067 0.64187218
##
## 13 14 15 16
## FALSE 10.66613433 2.89610270 0.59887596 0.48217192
## TRUE 5.77992076 2.39550382 1.17011148 0.78621664Conclusion
For the male population above 50K, the majority fall on education numbers of 9 and 13.
4.2.4 Education Level of Income Above 50K and Female
## [1] "Counts"##
## 1 2 3 4 5 6 7 8 9 10 11 12
## FALSE 51 168 331 645 509 930 1167 429 10275 7094 1315 1011
## TRUE 0 0 2 1 5 3 8 4 226 197 67 56
##
## 13 14 15 16
## FALSE 5016 1544 534 363
## TRUE 339 179 42 50## [1] "Percentages"##
## 1 2 3 4 5
## FALSE 0.156629096 0.515954670 1.016553546 1.980897393 1.563219803
## TRUE 0.000000000 0.000000000 0.006142317 0.003071159 0.015355794
##
## 6 7 8 9 10
## FALSE 2.856177636 3.584042259 1.317527103 31.556156138 21.786800160
## TRUE 0.009213476 0.024569270 0.012284635 0.694081877 0.605018273
##
## 11 12 13 14 15
## FALSE 4.038573754 3.104941494 15.404932281 4.741869107 1.639998772
## TRUE 0.205767636 0.171984890 1.041122816 0.549737416 0.128988667
##
## 16
## FALSE 1.114830626
## TRUE 0.153557937Conclusion
For the female population above 50K, the majority fall on education numbers of 9 and 13 similar to the males. The percentages are much smaller, but that was expected from the earlier table conclusions.
5 Closing
I hope you enjoyed this write-up and learned something from it. If you wish to see some of my other projects you can check out my portfolio website, github, or Linkedin.
[Github]
[Linkedin]
6 Session Info
sessionInfo()## R version 4.1.2 (2021-11-01)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 10 x64 (build 19042)
##
## Matrix products: default
##
## locale:
## [1] LC_COLLATE=English_United States.1252
## [2] LC_CTYPE=English_United States.1252
## [3] LC_MONETARY=English_United States.1252
## [4] LC_NUMERIC=C
## [5] LC_TIME=English_United States.1252
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] naivebayes_0.9.7 rpart.plot_3.1.0 rpart_4.1-15 dplyr_1.0.7
## [5] caret_6.0-90 lattice_0.20-45 neuralnet_1.44.2 viridis_0.6.2
## [9] viridisLite_0.4.0 ggplot2_3.3.5 Amelia_1.8.0 Rcpp_1.0.7
##
## loaded via a namespace (and not attached):
## [1] sass_0.4.0 jsonlite_1.7.2 splines_4.1.2
## [4] foreach_1.5.1 prodlim_2019.11.13 bslib_0.3.1
## [7] assertthat_0.2.1 stats4_4.1.2 highr_0.9
## [10] yaml_2.2.1 globals_0.14.0 ipred_0.9-12
## [13] pillar_1.6.4 glue_1.5.1 pROC_1.18.0
## [16] digest_0.6.28 colorspace_2.0-2 recipes_0.1.17
## [19] htmltools_0.5.2 Matrix_1.3-4 plyr_1.8.6
## [22] timeDate_3043.102 pkgconfig_2.0.3 listenv_0.8.0
## [25] bookdown_0.24 purrr_0.3.4 scales_1.1.1
## [28] gower_0.2.2 lava_1.6.10 proxy_0.4-26
## [31] tibble_3.1.6 generics_0.1.1 farver_2.1.0
## [34] ellipsis_0.3.2 withr_2.4.2 nnet_7.3-16
## [37] survival_3.2-13 magrittr_2.0.1 crayon_1.4.2
## [40] evaluate_0.14 fansi_0.5.0 future_1.23.0
## [43] parallelly_1.29.0 nlme_3.1-153 MASS_7.3-54
## [46] foreign_0.8-81 class_7.3-19 data.table_1.14.2
## [49] tools_4.1.2 lifecycle_1.0.1 stringr_1.4.0
## [52] munsell_0.5.0 e1071_1.7-9 compiler_4.1.2
## [55] jquerylib_0.1.4 rlang_0.4.12 grid_4.1.2
## [58] iterators_1.0.13 labeling_0.4.2 rmarkdown_2.11
## [61] ModelMetrics_1.2.2.2 gtable_0.3.0 codetools_0.2-18
## [64] DBI_1.1.1 reshape2_1.4.4 R6_2.5.1
## [67] gridExtra_2.3 lubridate_1.8.0 knitr_1.36
## [70] fastmap_1.1.0 future.apply_1.8.1 utf8_1.2.2
## [73] stringi_1.7.5 parallel_4.1.2 rmdformats_1.0.3
## [76] vctrs_0.3.8 tidyselect_1.1.1 xfun_0.28