Introduction

classification

• Classification is perhaps the most basic form of data analysis.
• The recipient of an offer can respond or not respond.
• An applicant for a loan can repay on time, repay late, or declare bankruptcy.
• A credit card transaction can be normal or fraudulent. A packet of data traveling on a network can be benign or threatening.
• A bus in a fleet can be available for service or unavailable. The victim of an illness can be recovered, still be ill, or be deceased.

Prediction

• trying to predict the value of a numerical variable
• the prediction of the value of a continuous variable
• prediction may be used for both continuous and categorical data

Association Rules and Recommendation Systems

• Large databases of customer transactions lend themselves naturally to the analysis of associations among items purchased
• “what goes with what.”

Predictive Analytics

• Classification, prediction, and to some extent, association rules and collaborative filtering constitute the analytical methods employed in predictive analytics

Data Reduction and Dimension Reduction

• This process of consolidating a large number of records (or cases) into a smaller set is termed data reduction.

Data Exploration and Visualization

• Exploration is aimed at understanding the global landscape of the data, and detecting unusual values

Supervised and Unsupervised Learning

Supervised Learning - Goal: Predict a single “target” or “outcome” variable - Training data, where target value is known - Score to data where value is not known - Methods: Classication and Prediction

Unsupervised Learning - Goal: Segment data into meaningful segments; detect patterns - There is no target (outcome) variable to predict or classify - Methods: Association rules, collaborative fi lters, data reduction & exploration, visualization

• 1.Develop an understanding of the purpose of the data mining project.
• 2.Obtain the data set to be used in the analysis.
• 3.Explore, clean, and preprocess the data.
• 4.Reduce the data dimension, if necessary.
• 5.Determine the data mining task.
• 6.Partition the data (for supervised tasks).
• 7.Choose the data mining techniques to be used.
• 8.Use algorithms to perform the task.
• 9.Interpret the results of the algorithms.
• 10.Deploy the model.

• Basic Steps
• Sampling from a Database
• Dummy Variables
• Missing Value

-   Dimension of data


-   viewing all the data


-   displaying only selected rows and columns


-   computing summary statistics for variables of interest

• Take the WEST ROXBURY HOME VALUES DATASET as example

• Find the dimension of the dataset

Dimension of housing.df: 5802 rows and 14 columns

• Show the first six rows

• Show the last row

• Find Summary statistic for each column

• Typically, we perform data mining on less than the complete database

  • Data Mining algorithms will have varying limitations on what they can handle on the term of the numbers and variables

  • Accurate models can often built with as few as several thousand records.

• Dummy variables are a coding method used to represent categorical data. They transform categorical variables into binary variables, where each variable represents a level (category) of the original categorical variable.

• Nominal categorical variables

• In many cases, only three of the dummy variables need to be used; if the values of three are known, the fourth is also known.

• Typically, some records will contain missing values. If the number of records with missing values is small, those records might be omitted. However, if we have a large number of variables, even a small proportion of missing values can affect a large number of variables

• An alternative to omitting records with missing values is to replace the missing value with an imputed value, based on the other values for that variable across all records.

NA not removed

NA has been removed

• Overfitting

• Creation and Use of Data Partitions

  • Training Partition

  • Validation Partition

  • Test Partition

• The more variable we include in a model, the greater the risk of overfitting the particular data used for modeling.

• Somewhat surprisingly, even if we know for a fact that a higher-degree curve is the appropriate model, if the model-fitting dataset is not large enough, a lower-degree function (that is not as likely to fit the noise) is likely to perform better in terms of predicting new values.

• Overfitting can also result from the application of many different models, from which the best performing model is selected.

We could connect up these points with a smooth but complicated function, one that interpolates all these data points perfectly and leaves no error(residuals).This can be seen if Figure2.3.However, we can see that such a curve is unlikely to be accurate, or even useful, in predicting future sales on the basis of advertising expenditures. For instance, it is hard to believe that increasing expenditures from 400$ to 500$ will actually decrease revenue.

• When we use the same data both to develop the model and to assess its performance, we introduce an “optimism” basis. This is because when we choose the model that works best with the data, this model’s superior performance comes from two sources:

  • A superior Model

  • Chance aspects of the data that happen to match the chosen model better than they match their model

• To address the overfitting problem, we simply divide(partition) our data and develop our model using only one of the partitions. After we have a model, we try it out on another partition and see how if performs, which we can measure in several ways.

Training, Validation, and Test Partitions

This is done by first drawing a random sample of records into the training set, then assigning the remaining records as validation. In the case of three partitions, the validation records are chosen randomly from the data after excluding the records already sampled into the training set.

• When the number of records in our sample is small, data partitioning might not be advisable as each partition will contain too few records for model building and performance evaluation.

• An alternative to data partitioning is cross-validation, which is especially useful with small samples.

• Cross-Validation is a procedure that starts with partitioning the data into “folds,” or non-overlapping subsamples.

• fitting a regression model to training data (WestRoxbury)

• The data mining process require a huge number of records.

• The use of a sample of 20,000 is likely to yield as accurate

• The number of records required in each partition (training, validation, and test) can be accommodated within the memory limit allowed in R (memory.limit())

• once the computational environment is set and functioning, the data miner’s work is not done.

• The environment in which a model operates is typically dynamic

• Predictive models often have a short shelf life—one leading consultant fi nds they rarely continue to function effectively for more than a year.

• So, even in a fully deployed state, models must be periodically checked and re-evaluated.

• Once performance fl ags, it is time to return to prototype mode and see if a new model can be developed.