7-Steps Predictive Modeling Process

7-Steps Predictive Modeling Process

• Step 1: Understand Business Objective
• Step 2: Define Modeling Goals
• Step 3: Select/Get Data
• Step 4: Prepare Data
• Step 5: Analyze and Transform Variables. Random Sampling
• Step 6: Model Selection and Develop Models (Training)
• Step 7: Validate Models (Testing), Optimize and Profitability
• Document Methodology and Models
• Implement Models and UAT
• Monitoring and Performance Tracking

Why Standard Process?

Organizations today are increasing their use of advance analytics and predictive modeling. The processes of generating predictive models involve data preparation, checking of data quality, reduction, modelling, prediction, and analysis of results. Generating high-quality predictive models is a time consuming activity because of the tuning process in finding optimum model parameters and often required to redevelop, tweak or reuse the models in the future. Thus it is important to

• Follow standard methodologies and industry best practices
• Establish Governance in Modeling building
• Maintain Quality standards across the board
• Ensure Re-usability of codes
• Save Time and Cost for Future Development
• Best Practices
• Comply with Internal and External Reviews
• Audit Requirements

For Whom?

• Advanced Analytics Professionals
• Predictive Modelers
• Data Scientists
• Business Analysts
• Change Managers
• CxOs

Key Stake Holders

• Business Managers, Product Managers, Risk Managers, Marketing Managers
• Operation Managers, IT Managers, Data Managers, Change Managers
• Business Analysts, Data Scientist, Consultants
• IT Developers, Information Security Risk Managers
• Independent Reviewers
• Internal and External Auditors

Step 1: Business Objective(s)

• Target Marketing
• Risk & Fraud Management
• Strategy Implementation and Change Management
• Operational Efficiency
• Increase Customer Experience
• Manage Marketing Campaigns
• Forecast Revenue or Loss
• Workforce Management
• Financial Modeling
• Churn Management
• Social Media Influencers

Step 1.1: Business Objectives - Asking Right Questions!

It is very important to clearly define the goals based on business objective.

• Do you want to understand the characteristics of customers?
• Do you want to make unprofitable customers profitable?
• Do you want to understand what driving sales?
• Do you want to win-back lost customers?
• Do you want to increase sales?
• Do you want to reduce customer churn?
• Do you want to reduce cost of production or operation?
• Do you want to target new customers?
• Do you want to identify probable credit default customers?
• Do you want to know X-sell/Upsell opportunities?

Businesses want to find answers all such important questions and make decisions based on data…

Step 1.2: Business Objective(s) - Target Modeling Opportunities

Industry	Response	Risk Mitigation	Attrition	Cross-Sell/Upsell	Net Present Value	Life Time Value
Retail	X		X	X	X	X
Banking	X	X	X	X	X	X
Insurance	X	X	X	X	X	X
Telecom	X	X	X	X	X	X
Utilities	X	X	X	X	X	X
Hospitality	X		X	X	X	X
Catalog	X			X	X	X
Publishing	X		X	X	X	X

Step 2: Define Goals - translate business objective into analytics goal

Based on the business questions we want to answer, translate the business objective into Analytic terms

• Profile Analysis
• Segmentations
• Response Modeling
• Risk Modeling
• Activation
• Cross-Sell and Upsell
• Attrition/Churn Modeling
• Net Present Value(NPV)
• Customer Life Time Value (CLTV)
• etc.

Step 3: Selecting Data for Modeling

Selecting best data for target modeling requires thorough understanding of the market, business and the objective. The model is only as good and relevant as the underlying data:

Data Types

Data Type	Predictive Power	Stability	Cost
Demographic	Medium	High	Low
Behavioural	High	Low	High
Psychographic	Medium	Medium	High

Sources of Data

Internal Sources	External Sources
Customer Data, Transaction Data	Survey Data, Research Data, Suppliers, Ratings
Other History	Credit Bureau Data, Third Party data, Sellers, Compilers

Step3.1: A Case Study - Target Marketing

Typical data required for Target Marketing

Demographic Data	Behaviour Data	External Data
Customer Demographic	Transaction	Customer Survey
Income	Loyalty	Market Research
Purchasing Power	-	Macro Economic Factors
-	-	Competitions

Step 4: Prepare Data

1. In this step we need prepare data into right format for analysis and the tool you may want use.

2. Do initial cleaning up

3. Define Variables and Create Data Dictionary

4. Joining/Appending multiple datasets

5. Validate for correctness

6. Produce Basic Summary Reports

Step 5: Analyze and Transform Variables

Once data is in right shape and perform

• univariate analysis: to check the distribution of each of the variables and features

• multivariate analyses: to check relationships with other variables and with dependent variables

Based on type of model you are going to use, you may need to transform the variables using one of the approaches

1. Bining approach: create distinct groups

2. Transformation:

• Logarithmic, Polynomial

• Square Root, Inverse, Square, boxCox

3. Extreme value (outlier) treatments

4. Missing Value Treatment

5. Dimension Reduction - Information Value(IV) and Weight of Evidence(WoE), Variable Clustering, PCA, Factor Analysis, etc.

Step 5.1: Random Sampling (Train and Test)

• Training Sample: Model will be developed on this sample. Typically 50%, 60%, 70% or 80% of the data goes here.

• Test Sample: Model performances will be validated on this sample. Typically 50%, 40%, 30% or 20% of the data goes here

Step 6.1: Model Selection

Based on the defined goal(s) (supervised or unsupervised) we have to select one of or combinations of modeling techniques. Such as

• General linear model
• Non-Linear Regression
• Linear Regression
• Lasso Regression
• Ridge Regression
• Non-Negative Garrotte Regression
• Percentage Regression
• Quantile Regression
• Non-parametric regression
• Logistic Regression
• Tobit Regression
• Probit Regression
• Classification/Decision Trees
• Random Forest
• Support Vector Machine (SVM)
• Distance metric learning
• Bayesian methods
• Graphical Models
• Neural Networks
• Genetic Algorithm
• The Hazard and Survival Functions
• Time Series Models
• Signal Processing
• Clustering Techniques
• Market Basket Analysis
• Frequent Itemset Mining
• Association Rule Mining etc.

There are wide variety of choices available outside this list.

Step 6.2: Build/Develop/Train Models

• Validate the assumptions of the chosen algorithm
• Check for Multicollinearity and Redundancies of Independent Variables (Features). Sometime in Machine Learning, we are keen on accuracies of the models and hence we may not perform these checks!
• Develop/Train Model on Training Sample, which is 80%/70%/60%/50% of the available data(Population)
• Check Model performance - Error, Accuracy, ROC, KS, Gini

Step 7: Validate/Test Models

• Score and Predict using Test Sample
• Check for the robustness and stability of the model
• Check Model Performance: Accuracy, ROC, AUC, KS, GINI etc.

• Perform Cross Validation to increase accuracy/performance of the models

Document Methodology and Models

It is good practice to document the entire process and findings for audit requirements and future work. Ideally we should document the followings

• Business Objectives and Goals

• Data Sources and Data used

• Type of Analysis performed: what, why, findings

• Exclusions

• Variable transformations

• Business inputs

• Methodology used, background, benefits

• Alternative methodologies tried- if any

• Model performance and Validation Results

• Cut-Off Analysis

• Recommendations

• Pseudo codes for implementation

• Key issues/challenges

• Future Plans

Implement Models

Once model is developed and validated, we need to implement the model within in a system or run off-line on real business data. After implementation, need to perform User Acceptance Test(UAT) to ensure model is implemented correctly and outcomes are sync with desired outcomes. Typical systems for model implementation include

• Acqusition/Account Opening Systems
• Account Management Systems
• Campaign Management Systems
• Decision Making Systems
• Collection Systems
• Customer Relationship Management (CRM) systems
• Remote Monitoring systems
• Data Products
• Analytics Platforms

Model Management: Monitoring and Performance Tracking

• Monitor performance of the models on a periodic basis

• Frequency: Monthly, Quarterly, Half-Yearly etc.

• Check for Population Stability

• Check for Characteristic Stability

R Markdown

This is an R Markdown presentation. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.

When you click the Knit button a document will be generated that includes both content as well as the output of any embedded R code chunks within the document.