Week5_Assignment
Abhiram
2024-02-10
library(readr)
library(dplyr)##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionoptions(repos = c(CRAN = "https://cran.rstudio.com"))
if (!requireNamespace("DT", quietly = TRUE)) {
install.packages("DT", dependencies = TRUE)
}mydata <- read_delim("C:\\Users\\abhir\\OneDrive\\Documents\\luddy\\H510-Stats,R\\adult.csv", delim = ",")## Rows: 32561 Columns: 15
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (9): Work-Class, Education, Marital Status, Occupation, Relationship, Ra...
## dbl (6): Age, Fnl_Wgt, Education_Level, Capital_Gain, Capital_Loss, hrs_week
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.mydata## # A tibble: 32,561 × 15
## Age `Work-Class` Fnl_Wgt Education Education_Level `Marital Status`
## <dbl> <chr> <dbl> <chr> <dbl> <chr>
## 1 39 " State-gov" 77516 " Bachelo… 13 " Never-married"
## 2 50 " Self-emp-not-inc" 83311 " Bachelo… 13 " Married-civ-s…
## 3 38 " Private" 215646 " HS-grad" 9 " Divorced"
## 4 53 " Private" 234721 " 11th" 7 " Married-civ-s…
## 5 28 " Private" 338409 " Bachelo… 13 " Married-civ-s…
## 6 37 " Private" 284582 " Masters" 14 " Married-civ-s…
## 7 49 " Private" 160187 " 9th" 5 " Married-spous…
## 8 52 " Self-emp-not-inc" 209642 " HS-grad" 9 " Married-civ-s…
## 9 31 " Private" 45781 " Masters" 14 " Never-married"
## 10 42 " Private" 159449 " Bachelo… 13 " Married-civ-s…
## # ℹ 32,551 more rows
## # ℹ 9 more variables: Occupation <chr>, Relationship <chr>, Race <chr>,
## # Sex <chr>, Capital_Gain <dbl>, Capital_Loss <dbl>, hrs_week <dbl>,
## # Country <chr>, Annual_pay <chr>spec(mydata)## cols(
## Age = col_double(),
## `Work-Class` = col_character(),
## Fnl_Wgt = col_double(),
## Education = col_character(),
## Education_Level = col_double(),
## `Marital Status` = col_character(),
## Occupation = col_character(),
## Relationship = col_character(),
## Race = col_character(),
## Sex = col_character(),
## Capital_Gain = col_double(),
## Capital_Loss = col_double(),
## hrs_week = col_double(),
## Country = col_character(),
## Annual_pay = col_character()
## )head(mydata,20)## # A tibble: 20 × 15
## Age `Work-Class` Fnl_Wgt Education Education_Level `Marital Status`
## <dbl> <chr> <dbl> <chr> <dbl> <chr>
## 1 39 " State-gov" 77516 " Bachelo… 13 " Never-married"
## 2 50 " Self-emp-not-inc" 83311 " Bachelo… 13 " Married-civ-s…
## 3 38 " Private" 215646 " HS-grad" 9 " Divorced"
## 4 53 " Private" 234721 " 11th" 7 " Married-civ-s…
## 5 28 " Private" 338409 " Bachelo… 13 " Married-civ-s…
## 6 37 " Private" 284582 " Masters" 14 " Married-civ-s…
## 7 49 " Private" 160187 " 9th" 5 " Married-spous…
## 8 52 " Self-emp-not-inc" 209642 " HS-grad" 9 " Married-civ-s…
## 9 31 " Private" 45781 " Masters" 14 " Never-married"
## 10 42 " Private" 159449 " Bachelo… 13 " Married-civ-s…
## 11 37 " Private" 280464 " Some-co… 10 " Married-civ-s…
## 12 30 " State-gov" 141297 " Bachelo… 13 " Married-civ-s…
## 13 23 " Private" 122272 " Bachelo… 13 " Never-married"
## 14 32 " Private" 205019 " Assoc-a… 12 " Never-married"
## 15 40 " Private" 121772 " Assoc-v… 11 " Married-civ-s…
## 16 34 " Private" 245487 " 7th-8th" 4 " Married-civ-s…
## 17 25 " Self-emp-not-inc" 176756 " HS-grad" 9 " Never-married"
## 18 32 " Private" 186824 " HS-grad" 9 " Never-married"
## 19 38 " Private" 28887 " 11th" 7 " Married-civ-s…
## 20 43 " Self-emp-not-inc" 292175 " Masters" 14 " Divorced"
## # ℹ 9 more variables: Occupation <chr>, Relationship <chr>, Race <chr>,
## # Sex <chr>, Capital_Gain <dbl>, Capital_Loss <dbl>, hrs_week <dbl>,
## # Country <chr>, Annual_pay <chr>#List of 3 columns which are unclear From the above sample data we can conclude that a)Education Level-As Education is in integer form one needs to read the documentation to understand which level interprets the exact education level b)Occupation-There are short forms for the title of the occupation which makes it unclear. c)Capital_Loss-Most of the capitol_loss values are missing or null.
mydata[472,]## # A tibble: 1 × 15
## Age `Work-Class` Fnl_Wgt Education Education_Level `Marital Status`
## <dbl> <chr> <dbl> <chr> <dbl> <chr>
## 1 22 " ?" 139883 " Some-college" 10 " Never-married"
## # ℹ 9 more variables: Occupation <chr>, Relationship <chr>, Race <chr>,
## # Sex <chr>, Capital_Gain <dbl>, Capital_Loss <dbl>, hrs_week <dbl>,
## # Country <chr>, Annual_pay <chr>#Unclear After reading the documentation The Documentation only explains the type of column and doesn’t explain the missing values in a row. 1) for the sex and marital status it only contains male and female category and marital status cannot be categorized in a particular facet. 2) The Annual pay is only categorized as 50k above or below but the denomination and currency changes from country to country . 50K in USA,Mexico,Jamaica and India cannot be same . The mean Annual pay changes from country to country and currency varies . This effects the capital gain and loss variables. 3)
library(ggplot2)
ggplot(data = mydata, aes(x = Occupation, y = Age, fill = Occupation)) +
geom_boxplot() +
labs(x = "Occupation", y = "Age", title = "Distribution of Age by Occupation") +
theme(axis.text.x = element_text(angle = 45, hjust = 1))
#Discrepancies and risks without reading documentation Data Quality Issues: Data discrepancies and null values can compromise the overall quality of the dataset. Inaccurate or inconsistent data can lead to erroneous conclusions and unreliable analyses.
Biased Analysis: If null values are not handled properly, they can introduce bias into the analysis. For example, if null values are systematically related to certain characteristics or outcomes, excluding or imputing them without careful consideration can bias the results.
Reduced Statistical Power: Null values reduce the sample size available for analysis, potentially reducing the statistical power of the study. This can make it more challenging to detect significant effects or relationships in the data.
Misleading Insights: Data discrepancies and null values can lead to misleading insights and interpretations. For example, if null values are ignored or improperly handled, the analysis may overlook important patterns or trends in the data.
Increased Uncertainty: Null values introduce uncertainty into the analysis, as their presence may indicate missing or unknown information. Without proper handling, this uncertainty can propagate throughout the analysis and decision-making process.
Impact on Models: In predictive modeling and machine learning, null values can affect the performance of models. Many algorithms cannot handle null values directly, so they must be imputed or removed. Improper handling of null values can degrade model performance and predictive accuracy.