- Basic Steps
- Sampling from a Database
- Dummy Variables
- Missing Value
week4 section1
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ── ## ✔ dplyr 1.1.4 ✔ readr 2.1.5 ## ✔ forcats 1.0.0 ✔ stringr 1.5.1 ## ✔ ggplot2 3.5.0 ✔ tibble 3.2.1 ## ✔ lubridate 1.9.3 ✔ tidyr 1.3.1 ## ✔ purrr 1.0.2 ## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ── ## ✖ ggplot2::%+%() masks psych::%+%() ## ✖ ggplot2::alpha() masks psych::alpha() ## ✖ dplyr::filter() masks stats::filter() ## ✖ dplyr::lag() masks stats::lag() ## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors ## Thank you for using fastDummies! ## ## To acknowledge our work, please cite the package: ## ## Kaplan, J. & Schlegel, B. (2023). fastDummies: Fast Creation of Dummy (Binary) Columns and Rows from Categorical Variables. Version 1.7.1. URL: https://github.com/jacobkap/fastDummies, https://jacobkap.github.io/fastDummies/. ## ## Registered S3 method overwritten by 'quantmod': ## method from ## as.zoo.data.frame zoo ## ## ## 载入程辑包:'gridExtra' ## ## ## The following object is masked from 'package:dplyr': ## ## combine ## ## ## ## 载入程辑包:'kableExtra' ## ## ## The following object is masked from 'package:dplyr': ## ## group_rows
classification
Supervised Learning - Goal: Predict a single “target” or “outcome” variable - Training data, where target value is known - Score to data where value is not known - Methods: Classication and Prediction
Unsupervised Learning - Goal: Segment data into meaningful segments; detect patterns - There is no target (outcome) variable to predict or classify - Methods: Association rules, collaborative fi lters, data reduction & exploration, visualization
- Dimension of data - viewing all the data - displaying only selected rows and columns - computing summary statistics for variables of interest
Dimension of housing.df: 5802 rows and 14 columns
| TOTAL.VALUE | TAX | LOT.SQFT | YR.BUILT | GROSS.AREA | LIVING.AREA | FLOORS | ROOMS | BEDROOMS | FULL.BATH | HALF.BATH | KITCHEN | FIREPLACE | REMODEL |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 344.2 | 4330 | 9965 | 1880 | 2436 | 1352 | 2 | 6 | 3 | 1 | 1 | 1 | 0 | None |
| 412.6 | 5190 | 6590 | 1945 | 3108 | 1976 | 2 | 10 | 4 | 2 | 1 | 1 | 0 | Recent |
| 330.1 | 4152 | 7500 | 1890 | 2294 | 1371 | 2 | 8 | 4 | 1 | 1 | 1 | 0 | None |
| 498.6 | 6272 | 13773 | 1957 | 5032 | 2608 | 1 | 9 | 5 | 1 | 1 | 1 | 1 | None |
| 331.5 | 4170 | 5000 | 1910 | 2370 | 1438 | 2 | 7 | 3 | 2 | 0 | 1 | 0 | None |
| 337.4 | 4244 | 5142 | 1950 | 2124 | 1060 | 1 | 6 | 3 | 1 | 0 | 1 | 1 | Old |
| TOTAL.VALUE | TAX | LOT.SQFT | YR.BUILT | GROSS.AREA | LIVING.AREA | FLOORS | ROOMS | BEDROOMS | FULL.BATH | HALF.BATH | KITCHEN | FIREPLACE | REMODEL |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 447.6 | 5630 | 7406 | 1950 | 2510 | 1600 | 2 | 7 | 3 | 1 | 1 | 1 | 1 | None |
| TOTAL.VALUE | TAX | LOT.SQFT | YR.BUILT | GROSS.AREA | LIVING.AREA | FLOORS | ROOMS | BEDROOMS | FULL.BATH | HALF.BATH | KITCHEN | FIREPLACE | REMODEL | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Min. : 105.0 | Min. : 1320 | Min. : 997 | Min. : 0 | Min. : 821 | Min. : 504 | Min. :1.000 | Min. : 3.000 | Min. :1.00 | Min. :1.000 | Min. :0.0000 | Min. :1.000 | Min. :0.0000 | Length:5802 | |
| 1st Qu.: 325.1 | 1st Qu.: 4090 | 1st Qu.: 4772 | 1st Qu.:1920 | 1st Qu.:2347 | 1st Qu.:1308 | 1st Qu.:1.000 | 1st Qu.: 6.000 | 1st Qu.:3.00 | 1st Qu.:1.000 | 1st Qu.:0.0000 | 1st Qu.:1.000 | 1st Qu.:0.0000 | Class :character | |
| Median : 375.9 | Median : 4728 | Median : 5683 | Median :1935 | Median :2700 | Median :1548 | Median :2.000 | Median : 7.000 | Median :3.00 | Median :1.000 | Median :1.0000 | Median :1.000 | Median :1.0000 | Mode :character | |
| Mean : 392.7 | Mean : 4939 | Mean : 6278 | Mean :1937 | Mean :2925 | Mean :1657 | Mean :1.684 | Mean : 6.995 | Mean :3.23 | Mean :1.297 | Mean :0.6139 | Mean :1.015 | Mean :0.7399 | NA | |
| 3rd Qu.: 438.8 | 3rd Qu.: 5520 | 3rd Qu.: 7022 | 3rd Qu.:1955 | 3rd Qu.:3239 | 3rd Qu.:1874 | 3rd Qu.:2.000 | 3rd Qu.: 8.000 | 3rd Qu.:4.00 | 3rd Qu.:2.000 | 3rd Qu.:1.0000 | 3rd Qu.:1.000 | 3rd Qu.:1.0000 | NA | |
| Max. :1217.8 | Max. :15319 | Max. :46411 | Max. :2011 | Max. :8154 | Max. :5289 | Max. :3.000 | Max. :14.000 | Max. :9.00 | Max. :5.000 | Max. :3.0000 | Max. :2.000 | Max. :4.0000 | NA |
Typically, we perform data mining on less than the complete database
Data Mining algorithms will have varying limitations on what they can handle on the term of the numbers and variables
Accurate models can often built with as few as several thousand records.
Dummy variables are a coding method used to represent categorical data. They transform categorical variables into binary variables, where each variable represents a level (category) of the original categorical variable.
Nominal categorical variables
In many cases, only three of the dummy variables need to be used; if the values of three are known, the fourth is also known.
| .data | .data_None | .data_Old | .data_Recent |
|---|---|---|---|
| None | 1 | 0 | 0 |
| Recent | 0 | 0 | 1 |
| None | 1 | 0 | 0 |
| None | 1 | 0 | 0 |
| None | 1 | 0 | 0 |
| Old | 0 | 1 | 0 |
Typically, some records will contain missing values. If the number of records with missing values is small, those records might be omitted. However, if we have a large number of variables, even a small proportion of missing values can affect a large number of variables
An alternative to omitting records with missing values is to replace the missing value with an imputed value, based on the other values for that variable across all records.
| Min | X1st.Qu. | Median | Mean | X3rd.Qu. | Max |
|---|---|---|---|---|---|
| 1 | 3 | 3 | 3.230093 | 4 | 9 |
| Min. | X1st.Qu. | Median | Mean | X3rd.Qu. | Max. |
|---|---|---|---|---|---|
| 1 | 3 | 3 | 3.229282 | 4 | 9 |
Overfitting
Creation and Use of Data Partitions
Training Partition
Validation Partition
Test Partition
The more variables we include in a model, the greater the risk of overfi tting the particular data used for modeling.
Overfitting is a concept in data science, which occurs when a statistical model fi ts exactly against its training data. When this happens, the algorithm unfortunately cannot perform accurately against unseen data, defeating its purpose.
We could connect up these points with a smooth but complicated function, one that interpolates all these data points perfectly and leaves no error(residuals).This can be seen if Figure2.3.However, we can see that such a curve is unlikely to be accurate, or even useful, in predicting future sales on the basis of advertising expenditures. For instance, it is hard to believe that increasing expenditures from 400$ to 500$ will actually decrease revenue.
we choose the model that works best with the data, this model’s superior performance comes from two sources:
A superior model
Chance aspects of the data that happen to match the chosen model better than they match other models
Training, Validation, and Test Partitions
This is done by first drawing a random sample of records into the training set, then assigning the remaining records as validation. In the case of three partitions, the validation records are chosen randomly from the data after excluding the records already sampled into the training set.
When the number of records in our sample is small, data partitioning might not be advisable as each partition will contain too few records for model building and performance evaluation.
An alternative to data partitioning is cross-validation, which is especially useful with small samples.
Cross-Validation is a procedure that starts with partitioning the data into “folds,” or non-overlapping sub samples.
## .data .data_None .data_Old .data_Recent ## 1 None 1 0 0 ## 2 Recent 0 0 1 ## 3 None 1 0 0 ## 4 None 1 0 0 ## 5 None 1 0 0 ## 6 Old 0 1 0
## ## Call: ## lm(formula = TOTAL.VALUE ~ ., data = housing.df) ## ## Residuals: ## Min 1Q Median 3Q Max ## -0.041717 -0.019783 0.000174 0.019725 0.041288 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 3.188e-02 1.806e-02 1.765 0.0776 . ## TAX 7.949e-02 5.542e-07 143429.651 <2e-16 *** ## LOT.SQFT 2.113e-07 1.400e-07 1.510 0.1312 ## YR.BUILT 1.230e-06 9.003e-06 0.137 0.8913 ## GROSS.AREA 1.835e-06 8.875e-07 2.068 0.0387 * ## LIVING.AREA -3.015e-06 1.636e-06 -1.843 0.0654 . ## FLOORS 1.254e-03 9.311e-04 1.346 0.1782 ## ROOMS -4.400e-04 3.453e-04 -1.274 0.2026 ## BEDROOMS 8.617e-05 5.260e-04 0.164 0.8699 ## FULL.BATH 7.072e-04 7.189e-04 0.984 0.3253 ## HALF.BATH 9.051e-04 6.607e-04 1.370 0.1707 ## KITCHEN 4.093e-03 2.553e-03 1.603 0.1090 ## FIREPLACE 1.875e-04 5.753e-04 0.326 0.7445 ## REMODELOld -1.002e-03 1.012e-03 -0.990 0.3224 ## REMODELRecent 3.273e-05 8.909e-04 0.037 0.9707 ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 0.02271 on 5777 degrees of freedom ## (因为不存在,10个观察量被删除了) ## Multiple R-squared: 1, Adjusted R-squared: 1 ## F-statistic: 7.863e+09 on 14 and 5777 DF, p-value: < 2.2e-16
## ## Call: ## lm(formula = TOTAL.VALUE ~ ., data = housing.df, subset = train.row) ## ## Residuals: ## Min 1Q Median 3Q Max ## -0.045310 -0.019767 0.000656 0.019440 0.042434 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) -1.277e-02 3.534e-02 -0.361 0.7179 ## TAX 7.949e-02 7.125e-07 111568.262 <2e-16 *** ## LOT.SQFT 1.436e-07 1.793e-07 0.801 0.4234 ## YR.BUILT 2.212e-05 1.780e-05 1.243 0.2140 ## GROSS.AREA 2.095e-06 1.189e-06 1.763 0.0781 . ## LIVING.AREA -3.044e-06 2.153e-06 -1.414 0.1574 ## FLOORS 1.522e-03 1.246e-03 1.221 0.2220 ## ROOMS -7.009e-04 4.429e-04 -1.582 0.1137 ## BEDROOMS 2.696e-04 6.799e-04 0.397 0.6917 ## FULL.BATH -1.221e-04 9.376e-04 -0.130 0.8964 ## HALF.BATH 1.288e-03 8.665e-04 1.486 0.1373 ## KITCHEN 8.093e-03 3.365e-03 2.405 0.0162 * ## FIREPLACE -9.028e-04 7.453e-04 -1.211 0.2259 ## REMODELOld -1.103e-03 1.302e-03 -0.847 0.3969 ## REMODELRecent 6.641e-05 1.147e-03 0.058 0.9538 ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 0.02274 on 3462 degrees of freedom ## (因为不存在,4个观察量被删除了) ## Multiple R-squared: 1, Adjusted R-squared: 1 ## F-statistic: 4.924e+09 on 14 and 3462 DF, p-value: < 2.2e-16
## ## Call: ## lm(formula = TOTAL.VALUE ~ ., data = housing.df, subset = validation.row) ## ## Residuals: ## Min 1Q Median 3Q Max ## -0.042487 -0.019700 -0.000585 0.019775 0.040487 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 5.123e-02 2.127e-02 2.409 0.0161 * ## TAX 7.949e-02 8.871e-07 89602.787 <2e-16 *** ## LOT.SQFT 3.517e-07 2.261e-07 1.555 0.1200 ## YR.BUILT -6.529e-06 1.044e-05 -0.625 0.5319 ## GROSS.AREA 1.817e-06 1.359e-06 1.337 0.1814 ## LIVING.AREA -3.320e-06 2.551e-06 -1.301 0.1933 ## FLOORS 1.344e-03 1.439e-03 0.934 0.3507 ## ROOMS 5.187e-05 5.570e-04 0.093 0.9258 ## BEDROOMS -2.241e-04 8.335e-04 -0.269 0.7880 ## FULL.BATH 1.696e-03 1.161e-03 1.461 0.1442 ## HALF.BATH 3.140e-06 1.046e-03 0.003 0.9976 ## KITCHEN -1.264e-03 3.926e-03 -0.322 0.7475 ## FIREPLACE 1.675e-03 9.134e-04 1.834 0.0668 . ## REMODELOld -6.575e-04 1.618e-03 -0.406 0.6846 ## REMODELRecent -5.030e-06 1.427e-03 -0.004 0.9972 ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 0.02267 on 2300 degrees of freedom ## (因为不存在,6个观察量被删除了) ## Multiple R-squared: 1, Adjusted R-squared: 1 ## F-statistic: 2.922e+09 on 14 and 2300 DF, p-value: < 2.2e-16
## 6 9 12 13 16 19 ## 337.3981 333.5062 344.4753 315.4601 298.2103 330.7221
## validation.data.TOTAL.VALUE pred residuals ## 6 337.4 337.3981 0.001921083 ## 9 333.5 333.5062 -0.006151054 ## 12 344.5 344.4753 0.024739939 ## 13 315.5 315.4601 0.039948696 ## 16 298.2 298.2103 -0.010348967 ## 19 330.7 330.7221 -0.022067038
## [1] 0.0005407983
## ME RMSE MAE MPE MAPE ## Test set 19.52708 142.5065 107.5641 -0.713923 26.94098
## ME RMSE MAE MPE MAPE ## Test set 0.0005407983 0.02274081 0.01964867 0.0001561046 0.005349903
-Head(first 6 observation of the data)
## TOTAL.VALUE TAX LOT.SQFT YR.BUILT GROSS.AREA LIVING.AREA FLOORS ROOMS ## 1 344.2 4330 9965 1880 2436 1352 2 6 ## 2 412.6 5190 6590 1945 3108 1976 2 10 ## 3 330.1 4152 7500 1890 2294 1371 2 8 ## 4 498.6 6272 13773 1957 5032 2608 1 9 ## 5 331.5 4170 5000 1910 2370 1438 2 7 ## 6 337.4 4244 5142 1950 2124 1060 1 6 ## BEDROOMS FULL.BATH HALF.BATH KITCHEN FIREPLACE REMODEL ## 1 3 1 1 1 0 None ## 2 4 2 1 1 0 Recent ## 3 4 1 1 1 0 None ## 4 5 1 1 1 1 None ## 5 3 2 0 1 0 None ## 6 3 1 0 1 1 Old
-Last 6 of the data
## TOTAL.VALUE TAX LOT.SQFT YR.BUILT GROSS.AREA LIVING.AREA FLOORS ROOMS ## 5797 413.414 5200 9150 1950 2324 1326 2 7 ## 5798 404.800 5092 6762 1938 2594 1714 2 9 ## 5799 407.900 5131 9408 1950 2414 1333 2 6 ## 5800 406.500 5113 7198 1987 2480 1674 2 7 ## 5801 308.700 3883 6890 1946 2000 1000 1 5 ## 5802 447.600 5630 7406 1950 2510 1600 2 7 ## BEDROOMS FULL.BATH HALF.BATH KITCHEN FIREPLACE REMODEL ## 5797 3 1 1 1 1 None ## 5798 NA 2 1 1 1 Recent ## 5799 3 1 1 1 1 None ## 5800 3 1 1 1 1 None ## 5801 2 1 0 1 0 None ## 5802 3 1 1 1 1 None