Overview of the steps involved in data mining
The general steps are
-data collection,
-cleaning,
-preprocess,
-choose the data mining techniques
-perform the task
2024-03-14
-data collection,
-cleaning,
-preprocess,
-choose the data mining techniques
-perform the task
what has come to be called predictive analytics, the tasks of classification and prediction as well as pattern discovery
which have become key elements of a“business analytics”function in most large firms.
The following is the structure of the data
## 'data.frame': 5802 obs. of 14 variables: ## $ TOTAL.VALUE: num 344 413 330 499 332 ... ## $ TAX : int 4330 5190 4152 6272 4170 4244 4521 4030 4195 5150 ... ## $ LOT.SQFT : int 9965 6590 7500 13773 5000 5142 5000 10000 6835 5093 ... ## $ YR.BUILT : int 1880 1945 1890 1957 1910 1950 1954 1950 1958 1900 ... ## $ GROSS.AREA : int 2436 3108 2294 5032 2370 2124 3220 2208 2582 4818 ... ## $ LIVING.AREA: int 1352 1976 1371 2608 1438 1060 1916 1200 1092 2992 ... ## $ FLOORS : num 2 2 2 1 2 1 2 1 1 2 ... ## $ ROOMS : int 6 10 8 9 7 6 7 6 5 8 ... ## $ BEDROOMS : int 3 4 4 5 3 3 3 3 3 4 ... ## $ FULL.BATH : int 1 2 1 1 2 1 1 1 1 2 ... ## $ HALF.BATH : int 1 1 1 1 0 0 1 0 0 0 ... ## $ KITCHEN : int 1 1 1 1 1 1 1 1 1 1 ... ## $ FIREPLACE : int 0 0 0 1 0 1 0 0 1 0 ... ## $ REMODEL : chr "None" "Recent" "None" "None" ...
This data contain 5802, 1 observation and 5802, 1 variables
## [1] 5802 14
## TOTAL.VALUE TAX LOT.SQFT YR.BUILT GROSS.AREA LIVING.AREA FLOORS ROOMS ## 1 344.2 4330 9965 1880 2436 1352 2 6 ## 2 412.6 5190 6590 1945 3108 1976 2 10 ## 3 330.1 4152 7500 1890 2294 1371 2 8 ## 4 498.6 6272 13773 1957 5032 2608 1 9 ## 5 331.5 4170 5000 1910 2370 1438 2 7 ## 6 337.4 4244 5142 1950 2124 1060 1 6 ## BEDROOMS FULL.BATH HALF.BATH KITCHEN FIREPLACE REMODEL ## 1 3 1 1 1 0 None ## 2 4 2 1 1 0 Recent ## 3 4 1 1 1 0 None ## 4 5 1 1 1 1 None ## 5 3 2 0 1 0 None ## 6 3 1 0 1 1 Old
## TOTAL.VALUE TAX LOT.SQFT YR.BUILT GROSS.AREA LIVING.AREA FLOORS ROOMS ## 5797 413.414 5200 9150 1950 2324 1326 2 7 ## 5798 404.800 5092 6762 1938 2594 1714 2 9 ## 5799 407.900 5131 9408 1950 2414 1333 2 6 ## 5800 406.500 5113 7198 1987 2480 1674 2 7 ## 5801 308.700 3883 6890 1946 2000 1000 1 5 ## 5802 447.600 5630 7406 1950 2510 1600 2 7 ## BEDROOMS FULL.BATH HALF.BATH KITCHEN FIREPLACE REMODEL ## 5797 3 1 1 1 1 None ## 5798 3 2 1 1 1 Recent ## 5799 3 1 1 1 1 None ## 5800 3 1 1 1 1 None ## 5801 2 1 0 1 0 None ## 5802 3 1 1 1 1 None
-Develop an understanding of the purpose of the data mining project
-Obtain the dataset to be used in the analysis
-Explore, clean, and preprocess the data
-Reduce the data dimension, if necessary.
-Determine the data mining task
-Partition the data (for supervised tasks)
-Choose the data mining techniques to be used
-Use algorithms to perform the task
-Interpret the results of the algorithms
-Deploy the model
-Datasets are nearly always constructed and displayed so that variables are in columns and records are in rows.
-Types of Variables
-There are several ways of classifying variables. Vari- ables can be numerical or text (character/string). They can be continuous, integer, categorical, or date. Categorical variables can be either coded as numerical or text. Categorical variables can be unordered (called nominal variables) with categories or they can be ordered with categories such as high value, low value, and nil value.
-Handling Categorical Variables
-Categorical variables can also be han- dled by most data mining routines, but often require special handling. If the categorical variable is ordered, we can sometimes code the categories numerically and treat the vari- able as if it were a continuous variable. The smaller the number of categories, and the less they represent equal increments of value, the more problematic this approach becomes, but it often works well enough. -Nominal categorical variables, however, often cannot be used as is. In many cases, they must be decomposed into a series of binary variables, called dummy variables.
-Variable Selection
-How Many Variables and How Much Data?
-Outliers
-Missing Values
-Normalizing (Standardizing) and Rescaling Data
-Training Partition
-The training partition, typically the largest partition, contains the data used to build the various models we are examining. The same training partition is generally used to develop multiple models.
-Validation Partition
-The validation partition is used to assess the predictive performance of each model so that you can compare models and choose the best one. In some algorithms, the validation partition may be used in an automated fashion to tune and improve the model.
-Test Partition
-The test partition is used to assess the performance of the chosen model with new data.
-Cross-Validation
Modeling Process
An important aspect of the data mining process is that the heavy-duty analysis does not necessarily require a huge number of records.
In most supervised data mining applications, the goal is not a static, one-time analysis of a particular dataset. Rather, we want to develop a model that can be used on an ongoing basis to predict or classify new records. Our initial analysis will be in prototype mode, while we explore and define the problem and test different models. We will follow all the steps outlined earlier in this chapter.