week4 section1

## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr     1.1.4     ✔ readr     2.1.5
## ✔ forcats   1.0.0     ✔ stringr   1.5.1
## ✔ ggplot2   3.5.0     ✔ tibble    3.2.1
## ✔ lubridate 1.9.3     ✔ tidyr     1.3.1
## ✔ purrr     1.0.2     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ ggplot2::%+%()   masks psych::%+%()
## ✖ ggplot2::alpha() masks psych::alpha()
## ✖ dplyr::filter()  masks stats::filter()
## ✖ dplyr::lag()     masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
## Thank you for using fastDummies!
## 
## To acknowledge our work, please cite the package:
## 
## Kaplan, J. & Schlegel, B. (2023). fastDummies: Fast Creation of Dummy (Binary) Columns and Rows from Categorical Variables. Version 1.7.1. URL: https://github.com/jacobkap/fastDummies, https://jacobkap.github.io/fastDummies/.
## 
## Registered S3 method overwritten by 'quantmod':
##   method            from
##   as.zoo.data.frame zoo 
## 
## 
## 载入程辑包:'gridExtra'
## 
## 
## The following object is masked from 'package:dplyr':
## 
##     combine
## 
## 
## 
## 载入程辑包:'kableExtra'
## 
## 
## The following object is masked from 'package:dplyr':
## 
##     group_rows

2.1 Introduction

2.2 Core Idea in Data Mining

Classification

classification

  • Classification is perhaps the most basic form of data analysis.
  • The recipient of an offer can respond or not respond.
  • An applicant for a loan can repay on time, repay late, or declare bankruptcy.
    • A credit card transaction can be normal or fraudulent. A packet of data traveling on a network can be benign or threatening.
    • A bus in a fleet can be available for service or unavailable. The victim of an illness can be recovered, still be ill, or be deceased.

Prediction

  • trying to predict the value of a numerical variable
  • the prediction of the value of a continuous variable
  • prediction may be used for both continuous and categorical data

Association Rules and Recommendation Systems

  • Large databases of customer transactions lend themselves naturally to the analysis of associations among items purchased
  • “what goes with what.”

Predictive Analytics

  • Classification, prediction, and to some extent, association rules and collaborative filtering constitute the analytical methods employed in predictive analytics

Data Reduction and Dimension Reduction

  • This process of consolidating a large number of records (or cases) into a smaller set is termed data reduction.

Data Exploration and Visualization

  • Exploration is aimed at understanding the global landscape of the data, and detecting unusual values

Supervised and Unsupervised Learning

Supervised Learning - Goal: Predict a single “target” or “outcome” variable - Training data, where target value is known - Score to data where value is not known - Methods: Classication and Prediction

Unsupervised Learning - Goal: Segment data into meaningful segments; detect patterns - There is no target (outcome) variable to predict or classify - Methods: Association rules, collaborative fi lters, data reduction & exploration, visualization

2.3 The Steps in Data MIning

Detailed steps

    1. Develop an understanding of the purpose of the data mining project.
    1. Obtain the data set to be used in the analysis.
    1. Explore, clean, and preprocess the data.
    1. Reduce the data dimension, if necessary.
    1. Determine the data mining task.
    1. Partition the data (for supervised tasks).
    1. Choose the data mining techniques to be used.
    1. Use algorithms to perform the task.
    1. Interpret the results of the algorithms.
    1. Deploy the model.

2.4 Preliminary Steps

Content

  • Basic Steps
  • Sampling from a Database
  • Dummy Variables
  • Missing Value

Some basic steps you need to do before doing any analysis

-   Dimension of data


-   viewing all the data


-   displaying only selected rows and columns


-   computing summary statistics for variables of interest

Take the WEST ROXBURY HOME VALUES DATASET as example

  • Find the dimension of the dataset
  • Dimension of housing.df: 5802 rows and 14 columns

    • Show the first six rows
    TOTAL.VALUE TAX LOT.SQFT YR.BUILT GROSS.AREA LIVING.AREA FLOORS ROOMS BEDROOMS FULL.BATH HALF.BATH KITCHEN FIREPLACE REMODEL
    344.2 4330 9965 1880 2436 1352 2 6 3 1 1 1 0 None
    412.6 5190 6590 1945 3108 1976 2 10 4 2 1 1 0 Recent
    330.1 4152 7500 1890 2294 1371 2 8 4 1 1 1 0 None
    498.6 6272 13773 1957 5032 2608 1 9 5 1 1 1 1 None
    331.5 4170 5000 1910 2370 1438 2 7 3 2 0 1 0 None
    337.4 4244 5142 1950 2124 1060 1 6 3 1 0 1 1 Old

    • Show the last row
    TOTAL.VALUE TAX LOT.SQFT YR.BUILT GROSS.AREA LIVING.AREA FLOORS ROOMS BEDROOMS FULL.BATH HALF.BATH KITCHEN FIREPLACE REMODEL
    447.6 5630 7406 1950 2510 1600 2 7 3 1 1 1 1 None
    • Find Summary statistic for each column
    TOTAL.VALUE TAX LOT.SQFT YR.BUILT GROSS.AREA LIVING.AREA FLOORS ROOMS BEDROOMS FULL.BATH HALF.BATH KITCHEN FIREPLACE REMODEL
    Min. : 105.0 Min. : 1320 Min. : 997 Min. : 0 Min. : 821 Min. : 504 Min. :1.000 Min. : 3.000 Min. :1.00 Min. :1.000 Min. :0.0000 Min. :1.000 Min. :0.0000 Length:5802
    1st Qu.: 325.1 1st Qu.: 4090 1st Qu.: 4772 1st Qu.:1920 1st Qu.:2347 1st Qu.:1308 1st Qu.:1.000 1st Qu.: 6.000 1st Qu.:3.00 1st Qu.:1.000 1st Qu.:0.0000 1st Qu.:1.000 1st Qu.:0.0000 Class :character
    Median : 375.9 Median : 4728 Median : 5683 Median :1935 Median :2700 Median :1548 Median :2.000 Median : 7.000 Median :3.00 Median :1.000 Median :1.0000 Median :1.000 Median :1.0000 Mode :character
    Mean : 392.7 Mean : 4939 Mean : 6278 Mean :1937 Mean :2925 Mean :1657 Mean :1.684 Mean : 6.995 Mean :3.23 Mean :1.297 Mean :0.6139 Mean :1.015 Mean :0.7399 NA
    3rd Qu.: 438.8 3rd Qu.: 5520 3rd Qu.: 7022 3rd Qu.:1955 3rd Qu.:3239 3rd Qu.:1874 3rd Qu.:2.000 3rd Qu.: 8.000 3rd Qu.:4.00 3rd Qu.:2.000 3rd Qu.:1.0000 3rd Qu.:1.000 3rd Qu.:1.0000 NA
    Max. :1217.8 Max. :15319 Max. :46411 Max. :2011 Max. :8154 Max. :5289 Max. :3.000 Max. :14.000 Max. :9.00 Max. :5.000 Max. :3.0000 Max. :2.000 Max. :4.0000 NA

    Sampling from a Database

    • Typically, we perform data mining on less than the complete database

      • Data Mining algorithms will have varying limitations on what they can handle on the term of the numbers and variables

      • Accurate models can often built with as few as several thousand records.

    Dummy Variable

    • Dummy variables are a coding method used to represent categorical data. They transform categorical variables into binary variables, where each variable represents a level (category) of the original categorical variable.

    • Nominal categorical variables

    • In many cases, only three of the dummy variables need to be used; if the values of three are known, the fourth is also known.

    .data .data_None .data_Old .data_Recent
    None 1 0 0
    Recent 0 0 1
    None 1 0 0
    None 1 0 0
    None 1 0 0
    Old 0 1 0

    Missing Value

    • Typically, some records will contain missing values. If the number of records with missing values is small, those records might be omitted. However, if we have a large number of variables, even a small proportion of missing values can affect a large number of variables

    • An alternative to omitting records with missing values is to replace the missing value with an imputed value, based on the other values for that variable across all records.

    NA not removed

    Min X1st.Qu. Median Mean X3rd.Qu. Max
    1 3 3 3.230093 4 9

    NA has been removed

    Min. X1st.Qu. Median Mean X3rd.Qu. Max.
    1 3 3 3.229282 4 9

    2.5 Predictive Power and Overfitting

    Content

    • Overfitting

    • Creation and Use of Data Partitions

      • Training Partition

      • Validation Partition

      • Test Partition

    Overfitting

    • The more variables we include in a model, the greater the risk of overfi tting the particular data used for modeling.

    • Overfitting is a concept in data science, which occurs when a statistical model fi ts exactly against its training data. When this happens, the algorithm unfortunately cannot perform accurately against unseen data, defeating its purpose.

    Overfitting Example

    We could connect up these points with a smooth but complicated function, one that interpolates all these data points perfectly and leaves no error(residuals).This can be seen if Figure2.3.However, we can see that such a curve is unlikely to be accurate, or even useful, in predicting future sales on the basis of advertising expenditures. For instance, it is hard to believe that increasing expenditures from 400$ to 500$ will actually decrease revenue.

    Creation and Use of Data Partitions

    • we choose the model that works best with the data, this model’s superior performance comes from two sources:

      • A superior model

      • Chance aspects of the data that happen to match the chosen model better than they match other models

    Three Data Partitions And Their Role in the Data Mining Process

    Training, Validation, and Test Partitions

    Partition in R and Visualization

    Explanation to the R code

    This is done by first drawing a random sample of records into the training set, then assigning the remaining records as validation. In the case of three partitions, the validation records are chosen randomly from the data after excluding the records already sampled into the training set.

    Additional - Cross-Validation

    • When the number of records in our sample is small, data partitioning might not be advisable as each partition will contain too few records for model building and performance evaluation.

    • An alternative to data partitioning is cross-validation, which is especially useful with small samples.

    • Cross-Validation is a procedure that starts with partitioning the data into “folds,” or non-overlapping sub samples.

    2.6 Building a Predivetive Model

    Modeling Process

    • fitting a regression model to training data (WestRoxbury)
    ##    .data .data_None .data_Old .data_Recent
    ## 1   None          1         0            0
    ## 2 Recent          0         0            1
    ## 3   None          1         0            0
    ## 4   None          1         0            0
    ## 5   None          1         0            0
    ## 6    Old          0         1            0
    ## 
    ## Call:
    ## lm(formula = TOTAL.VALUE ~ ., data = housing.df)
    ## 
    ## Residuals:
    ##       Min        1Q    Median        3Q       Max 
    ## -0.041717 -0.019783  0.000174  0.019725  0.041288 
    ## 
    ## Coefficients:
    ##                 Estimate Std. Error    t value Pr(>|t|)    
    ## (Intercept)    3.188e-02  1.806e-02      1.765   0.0776 .  
    ## TAX            7.949e-02  5.542e-07 143429.651   <2e-16 ***
    ## LOT.SQFT       2.113e-07  1.400e-07      1.510   0.1312    
    ## YR.BUILT       1.230e-06  9.003e-06      0.137   0.8913    
    ## GROSS.AREA     1.835e-06  8.875e-07      2.068   0.0387 *  
    ## LIVING.AREA   -3.015e-06  1.636e-06     -1.843   0.0654 .  
    ## FLOORS         1.254e-03  9.311e-04      1.346   0.1782    
    ## ROOMS         -4.400e-04  3.453e-04     -1.274   0.2026    
    ## BEDROOMS       8.617e-05  5.260e-04      0.164   0.8699    
    ## FULL.BATH      7.072e-04  7.189e-04      0.984   0.3253    
    ## HALF.BATH      9.051e-04  6.607e-04      1.370   0.1707    
    ## KITCHEN        4.093e-03  2.553e-03      1.603   0.1090    
    ## FIREPLACE      1.875e-04  5.753e-04      0.326   0.7445    
    ## REMODELOld    -1.002e-03  1.012e-03     -0.990   0.3224    
    ## REMODELRecent  3.273e-05  8.909e-04      0.037   0.9707    
    ## ---
    ## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
    ## 
    ## Residual standard error: 0.02271 on 5777 degrees of freedom
    ##   (因为不存在,10个观察量被删除了)
    ## Multiple R-squared:      1,  Adjusted R-squared:      1 
    ## F-statistic: 7.863e+09 on 14 and 5777 DF,  p-value: < 2.2e-16
    ## 
    ## Call:
    ## lm(formula = TOTAL.VALUE ~ ., data = housing.df, subset = train.row)
    ## 
    ## Residuals:
    ##       Min        1Q    Median        3Q       Max 
    ## -0.045310 -0.019767  0.000656  0.019440  0.042434 
    ## 
    ## Coefficients:
    ##                 Estimate Std. Error    t value Pr(>|t|)    
    ## (Intercept)   -1.277e-02  3.534e-02     -0.361   0.7179    
    ## TAX            7.949e-02  7.125e-07 111568.262   <2e-16 ***
    ## LOT.SQFT       1.436e-07  1.793e-07      0.801   0.4234    
    ## YR.BUILT       2.212e-05  1.780e-05      1.243   0.2140    
    ## GROSS.AREA     2.095e-06  1.189e-06      1.763   0.0781 .  
    ## LIVING.AREA   -3.044e-06  2.153e-06     -1.414   0.1574    
    ## FLOORS         1.522e-03  1.246e-03      1.221   0.2220    
    ## ROOMS         -7.009e-04  4.429e-04     -1.582   0.1137    
    ## BEDROOMS       2.696e-04  6.799e-04      0.397   0.6917    
    ## FULL.BATH     -1.221e-04  9.376e-04     -0.130   0.8964    
    ## HALF.BATH      1.288e-03  8.665e-04      1.486   0.1373    
    ## KITCHEN        8.093e-03  3.365e-03      2.405   0.0162 *  
    ## FIREPLACE     -9.028e-04  7.453e-04     -1.211   0.2259    
    ## REMODELOld    -1.103e-03  1.302e-03     -0.847   0.3969    
    ## REMODELRecent  6.641e-05  1.147e-03      0.058   0.9538    
    ## ---
    ## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
    ## 
    ## Residual standard error: 0.02274 on 3462 degrees of freedom
    ##   (因为不存在,4个观察量被删除了)
    ## Multiple R-squared:      1,  Adjusted R-squared:      1 
    ## F-statistic: 4.924e+09 on 14 and 3462 DF,  p-value: < 2.2e-16
    ## 
    ## Call:
    ## lm(formula = TOTAL.VALUE ~ ., data = housing.df, subset = validation.row)
    ## 
    ## Residuals:
    ##       Min        1Q    Median        3Q       Max 
    ## -0.042487 -0.019700 -0.000585  0.019775  0.040487 
    ## 
    ## Coefficients:
    ##                 Estimate Std. Error   t value Pr(>|t|)    
    ## (Intercept)    5.123e-02  2.127e-02     2.409   0.0161 *  
    ## TAX            7.949e-02  8.871e-07 89602.787   <2e-16 ***
    ## LOT.SQFT       3.517e-07  2.261e-07     1.555   0.1200    
    ## YR.BUILT      -6.529e-06  1.044e-05    -0.625   0.5319    
    ## GROSS.AREA     1.817e-06  1.359e-06     1.337   0.1814    
    ## LIVING.AREA   -3.320e-06  2.551e-06    -1.301   0.1933    
    ## FLOORS         1.344e-03  1.439e-03     0.934   0.3507    
    ## ROOMS          5.187e-05  5.570e-04     0.093   0.9258    
    ## BEDROOMS      -2.241e-04  8.335e-04    -0.269   0.7880    
    ## FULL.BATH      1.696e-03  1.161e-03     1.461   0.1442    
    ## HALF.BATH      3.140e-06  1.046e-03     0.003   0.9976    
    ## KITCHEN       -1.264e-03  3.926e-03    -0.322   0.7475    
    ## FIREPLACE      1.675e-03  9.134e-04     1.834   0.0668 .  
    ## REMODELOld    -6.575e-04  1.618e-03    -0.406   0.6846    
    ## REMODELRecent -5.030e-06  1.427e-03    -0.004   0.9972    
    ## ---
    ## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
    ## 
    ## Residual standard error: 0.02267 on 2300 degrees of freedom
    ##   (因为不存在,6个观察量被删除了)
    ## Multiple R-squared:      1,  Adjusted R-squared:      1 
    ## F-statistic: 2.922e+09 on 14 and 2300 DF,  p-value: < 2.2e-16

    Applying the regression model to predict validation set (WestRoxbury)

    ##        6        9       12       13       16       19 
    ## 337.3981 333.5062 344.4753 315.4601 298.2103 330.7221
    ##    validation.data.TOTAL.VALUE     pred    residuals
    ## 6                        337.4 337.3981  0.001921083
    ## 9                        333.5 333.5062 -0.006151054
    ## 12                       344.5 344.4753  0.024739939
    ## 13                       315.5 315.4601  0.039948696
    ## 16                       298.2 298.2103 -0.010348967
    ## 19                       330.7 330.7221 -0.022067038
    ## [1] 0.0005407983

    computing model evaluation metrics

    ##                ME     RMSE      MAE       MPE     MAPE
    ## Test set 19.52708 142.5065 107.5641 -0.713923 26.94098
    ##                    ME       RMSE        MAE          MPE        MAPE
    ## Test set 0.0005407983 0.02274081 0.01964867 0.0001561046 0.005349903

    2.7 Using R for Data Mining on a Local Machine

    • The data mining process require a huge number of records.
    • The use of a sample of 20,000 is likely to yield as accurate
    • The number of records required in each partition (training, validation, and test) can be accommodated within the memory limit allowed in R (memory.limit())

    2.8 Automating Data Mining Solutions

    Automating Data Mining Solutions

    • once the computational environment is set and functioning, the data miner’s work is not done.
    • The environment in which a model operates is typically dynamic
    • Predictive models often have a short shelf life—one leading consultant fi nds they rarely continue to function effectively for more than a year.
    • So, even in a fully deployed state, models must be periodically checked and re-evaluated.
    • Once performance fl ags, it is time to return to prototype mode and see if a new model can be developed.

    -Head(first 6 observation of the data)

    ##   TOTAL.VALUE  TAX LOT.SQFT YR.BUILT GROSS.AREA LIVING.AREA FLOORS ROOMS
    ## 1       344.2 4330     9965     1880       2436        1352      2     6
    ## 2       412.6 5190     6590     1945       3108        1976      2    10
    ## 3       330.1 4152     7500     1890       2294        1371      2     8
    ## 4       498.6 6272    13773     1957       5032        2608      1     9
    ## 5       331.5 4170     5000     1910       2370        1438      2     7
    ## 6       337.4 4244     5142     1950       2124        1060      1     6
    ##   BEDROOMS FULL.BATH HALF.BATH KITCHEN FIREPLACE REMODEL
    ## 1        3         1         1       1         0    None
    ## 2        4         2         1       1         0  Recent
    ## 3        4         1         1       1         0    None
    ## 4        5         1         1       1         1    None
    ## 5        3         2         0       1         0    None
    ## 6        3         1         0       1         1     Old

    -Last 6 of the data

    ##      TOTAL.VALUE  TAX LOT.SQFT YR.BUILT GROSS.AREA LIVING.AREA FLOORS ROOMS
    ## 5797     413.414 5200     9150     1950       2324        1326      2     7
    ## 5798     404.800 5092     6762     1938       2594        1714      2     9
    ## 5799     407.900 5131     9408     1950       2414        1333      2     6
    ## 5800     406.500 5113     7198     1987       2480        1674      2     7
    ## 5801     308.700 3883     6890     1946       2000        1000      1     5
    ## 5802     447.600 5630     7406     1950       2510        1600      2     7
    ##      BEDROOMS FULL.BATH HALF.BATH KITCHEN FIREPLACE REMODEL
    ## 5797        3         1         1       1         1    None
    ## 5798       NA         2         1       1         1  Recent
    ## 5799        3         1         1       1         1    None
    ## 5800        3         1         1       1         1    None
    ## 5801        2         1         0       1         0    None
    ## 5802        3         1         1       1         1    None