2024-03-14

Introduction

Overview of the steps involved in data mining

The general steps are

-data collection,

-cleaning,

-preprocess,

-choose the data mining techniques

-perform the task

Core Idea in Data Mining

General definitions of data mining

  • what has come to be called predictive analytics, the tasks of classification and prediction as well as pattern discovery

  • which have become key elements of a“business analytics”function in most large firms.

Data level summary

The following is the structure of the data

## 'data.frame':    5802 obs. of  14 variables:
##  $ TOTAL.VALUE: num  344 413 330 499 332 ...
##  $ TAX        : int  4330 5190 4152 6272 4170 4244 4521 4030 4195 5150 ...
##  $ LOT.SQFT   : int  9965 6590 7500 13773 5000 5142 5000 10000 6835 5093 ...
##  $ YR.BUILT   : int  1880 1945 1890 1957 1910 1950 1954 1950 1958 1900 ...
##  $ GROSS.AREA : int  2436 3108 2294 5032 2370 2124 3220 2208 2582 4818 ...
##  $ LIVING.AREA: int  1352 1976 1371 2608 1438 1060 1916 1200 1092 2992 ...
##  $ FLOORS     : num  2 2 2 1 2 1 2 1 1 2 ...
##  $ ROOMS      : int  6 10 8 9 7 6 7 6 5 8 ...
##  $ BEDROOMS   : int  3 4 4 5 3 3 3 3 3 4 ...
##  $ FULL.BATH  : int  1 2 1 1 2 1 1 1 1 2 ...
##  $ HALF.BATH  : int  1 1 1 1 0 0 1 0 0 0 ...
##  $ KITCHEN    : int  1 1 1 1 1 1 1 1 1 1 ...
##  $ FIREPLACE  : int  0 0 0 1 0 1 0 0 1 0 ...
##  $ REMODEL    : chr  "None" "Recent" "None" "None" ...

This data contain 5802, 1 observation and 5802, 1 variables

## [1] 5802   14
##   TOTAL.VALUE  TAX LOT.SQFT YR.BUILT GROSS.AREA LIVING.AREA FLOORS ROOMS
## 1       344.2 4330     9965     1880       2436        1352      2     6
## 2       412.6 5190     6590     1945       3108        1976      2    10
## 3       330.1 4152     7500     1890       2294        1371      2     8
## 4       498.6 6272    13773     1957       5032        2608      1     9
## 5       331.5 4170     5000     1910       2370        1438      2     7
## 6       337.4 4244     5142     1950       2124        1060      1     6
##   BEDROOMS FULL.BATH HALF.BATH KITCHEN FIREPLACE REMODEL
## 1        3         1         1       1         0    None
## 2        4         2         1       1         0  Recent
## 3        4         1         1       1         0    None
## 4        5         1         1       1         1    None
## 5        3         2         0       1         0    None
## 6        3         1         0       1         1     Old
##      TOTAL.VALUE  TAX LOT.SQFT YR.BUILT GROSS.AREA LIVING.AREA FLOORS ROOMS
## 5797     413.414 5200     9150     1950       2324        1326      2     7
## 5798     404.800 5092     6762     1938       2594        1714      2     9
## 5799     407.900 5131     9408     1950       2414        1333      2     6
## 5800     406.500 5113     7198     1987       2480        1674      2     7
## 5801     308.700 3883     6890     1946       2000        1000      1     5
## 5802     447.600 5630     7406     1950       2510        1600      2     7
##      BEDROOMS FULL.BATH HALF.BATH KITCHEN FIREPLACE REMODEL
## 5797        3         1         1       1         1    None
## 5798        3         2         1       1         1  Recent
## 5799        3         1         1       1         1    None
## 5800        3         1         1       1         1    None
## 5801        2         1         0       1         0    None
## 5802        3         1         1       1         1    None

The Steps in Data Mining

-Develop an understanding of the purpose of the data mining project

-Obtain the dataset to be used in the analysis

-Explore, clean, and preprocess the data

-Reduce the data dimension, if necessary.

-Determine the data mining task

-Partition the data (for supervised tasks)

-Choose the data mining techniques to be used

-Use algorithms to perform the task

-Interpret the results of the algorithms

-Deploy the model

Preliminary Steps

Organization of Datasets

-Datasets are nearly always constructed and displayed so that variables are in columns and records are in rows.

Predicting Home Values in the West Roxbury Neighborhood

Loading and Looking at the Data in R

Sampling from a Database

Oversampling Rare Events in Classification Tasks

Preprocessing and Cleaning the Data

-Types of Variables

-There are several ways of classifying variables. Vari- ables can be numerical or text (character/string). They can be continuous, integer, categorical, or date. Categorical variables can be either coded as numerical or text. Categorical variables can be unordered (called nominal variables) with categories or they can be ordered with categories such as high value, low value, and nil value. 

-Handling Categorical Variables

-Categorical variables can also be han- dled by most data mining routines, but often require special handling. If the categorical variable is ordered, we can sometimes code the categories numerically and treat the vari- able as if it were a continuous variable. The smaller the number of categories, and the less they represent equal increments of value, the more problematic this approach becomes, but it often works well enough.
-Nominal categorical variables, however, often cannot be used as is. In many cases, they must be decomposed into a series of binary variables, called dummy variables.

-Variable Selection

-How Many Variables and How Much Data?

-Outliers

-Missing Values

-Normalizing (Standardizing) and Rescaling Data

Predictive Power and Overfitting

Overfitting

Creation and Use of Data Partitions

-Training Partition

-The training partition, typically the largest partition, contains the data used to build the various models we are examining. The same training partition is generally used to develop multiple models.

-Validation Partition

-The validation partition is used to assess the predictive performance of each model so that you can compare models and choose the best one. In some algorithms, the validation partition may be used in an automated fashion to tune and improve the model.

-Test Partition

-The test partition is used to assess the performance of the chosen model with new data.

-Cross-Validation

Building a Predictive Medel

Modeling Process

Using R for Data Mining on a Local Machine

An important aspect of the data mining process is that the heavy-duty analysis does not necessarily require a huge number of records.

Automation Data Mining Solutions

In most supervised data mining applications, the goal is not a static, one-time analysis of a particular dataset. Rather, we want to develop a model that can be used on an ongoing basis to predict or classify new records. Our initial analysis will be in prototype mode, while we explore and define the problem and test different models. We will follow all the steps outlined earlier in this chapter.