IT408 / IT408 SC:
Data Mining

Course Textbook:

Hadley Wickham,Mine Cetinkaya-Rundel and Garrett Grolemund, R for data science: import, tidy, transform, visualize, and model data. 2nd Edition, O’Reilly Press

Tidyverse

A framework to support Data Science work flow.

Data Anomalies

Handling Missing Values

• Imputation: Filling in missing values using statistical metrics like the mean, median, or mode (for numerical data) or a placeholder like “Unknown” (for categorical data).

• Deletion: Dropping rows or columns with missing data entirely, though this is usually done as a last resort if a significant chunk of data is missing.

• Prediction: Using machine learning models to predict and fill in the missing values based on other available data.

Removing Duplicates

Duplicate entries skew analysis by artificially inflating the weight of certain data points.

• Exact Duplicates: Identifying and removing rows that are 100% identical.

• Partial Duplicates: Catching entries that represent the same entity but have slight variations (e.g., the same customer signing up twice with different emails, or two entries for John Smith at the same address).

Structural and Format Standardization

• Data Type Conversion: Ensuring numbers are stored as floats/integers, dates as datetime objects, and text as strings. (e.g., converting “100” from text to the number 100).

• String Uniformity: Standardizing text casing (e.g., converting all entries to lowercase), removing accidental leading or trailing whitespaces, and unifying date formats (e.g., converting everything to YYYY-MM-DD).

• Categorical Unification: Mapping synonymous labels to a single standard value (e.g., collapsing N/A, Not Applicable, and n/a into a single NA label).

Detecting and Correcting Outliers

• Investigation: Determining if the outlier is a legitimate, extreme real-world anomaly or simply a data entry error (e.g., a person’s age entered as 320 instead of 32).

• Cap/Floor Adjustment: Resetting extreme values to a designated maximum or minimum threshold if they skew statistical models too aggressively.

Validating and Verifying Constraints

This operation checks that the data makes logical sense according to real-world rules.

• Cross-field Validation: Making sure fields don’t contradict each other (e.g., a Delivery Date cannot occur before a Purchase Date).

• Range Checks: Ensuring values fall within expected bounds (e.g., a monthly satisfaction score must be between 1 and 10).

Dropping Irrelevant Data

Not all collected data is useful for the specific problem you are trying to solve.

• Feature Selection: Strip away columns that don’t add value to your specific analysis (e.g., dropping IP addresses if you are analyzing overall sales trends) to reduce processing time and noise.

Why It Matters

• Skipping these steps leads to the classic “Garbage In, Garbage Out” dilemma.

• Dirty data will produce flawed products even from the most sophisticated machine learning algorithms or statistical models.

Processing Pipe Characters

Cleansing processes can be chained by adding pipe characters between function calls

• |>
• %>%

The chain will be executed sequentually from left to right.

Another bad dataset

# A tibble: 5 × 15
     id category date      jan     feb   mar   apr   may   jun   jul   aug   sep
  <dbl> <chr>    <chr>     <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1     1 " A "    2024/01/… "10"     NA    12    11     9    10    11    10    12
2     1 "A"      2024-01-… "10"     NA    12    11     9    10    11    10    12
3     2 "b"      15-02-20… " 5"      6     7    NA     6     7     8     7     6
4     3 "B "     2024 Mar… "3"       4  -999     4     4     5     6     5     4
5     4 " c"     2024.04.… "8"       9    10     9     8     9    10     9     8
# ℹ 3 more variables: oct <chr>, nov <dbl>, dec <dbl>

Cleanup pipe

Goals:

• Trim whitespace and standardize category values (e.g., ” A ” / “A” / “A” -> “A”, “b” / “B” -> “B”, ” c” -> “C”)
• Fix messy date into a proper Date
• Remove duplicates
• Pivot the 12 month columns into long format (one row per month)
• Ensure month values are numeric
• Handle missing values (example: replace with NA-kept; or impute—here we just keep them)
• Detect/remove/mark outliers (example: set to NA if outside reasonable bounds)

Tidycode

# A tibble: 48 × 7
      id category date       month value month_num month_date
   <dbl> <chr>    <date>     <chr> <dbl>     <int> <date>    
 1     1 A        2024-01-15 jan      10         1 2024-01-01
 2     1 A        2024-01-15 feb      NA         2 2024-02-01
 3     1 A        2024-01-15 mar      12         3 2024-03-01
 4     1 A        2024-01-15 apr      11         4 2024-04-01
 5     1 A        2024-01-15 may       9         5 2024-05-01
 6     1 A        2024-01-15 jun      10         6 2024-06-01
 7     1 A        2024-01-15 jul      11         7 2024-07-01
 8     1 A        2024-01-15 aug      10         8 2024-08-01
 9     1 A        2024-01-15 sep      12         9 2024-09-01
10     1 A        2024-01-15 oct      NA        10 2024-10-01
# ℹ 38 more rows

An exponential pulse

\[y(x, t) = A e^{-\frac{(x - vt)^2}{2\sigma^2}}\]