Response
library(knitr)
library(tidyverse)
library(kableExtra)
library(cowplot)
library(skimr)
library(arules)
library(arulesViz)We will load the GroceryDataSet.csv file into an arules transaction object.
grocery_data_raw = read.csv("GroceryDataSet.csv")
dim(grocery_data_raw)## [1] 9834 32# Convert the data frame of groceries to transactions.
grocery_transactions = read.transactions("GroceryDataSet.csv", sep = ",")
summary(grocery_transactions)## transactions as itemMatrix in sparse format with
## 9835 rows (elements/itemsets/transactions) and
## 169 columns (items) and a density of 0.02609146
##
## most frequent items:
## whole milk other vegetables rolls/buns soda
## 2513 1903 1809 1715
## yogurt (Other)
## 1372 34055
##
## element (itemset/transaction) length distribution:
## sizes
## 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
## 2159 1643 1299 1005 855 645 545 438 350 246 182 117 78 77 55 46
## 17 18 19 20 21 22 23 24 26 27 28 29 32
## 29 14 14 9 11 4 6 1 1 1 1 3 1
##
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 1.000 2.000 3.000 4.409 6.000 32.000
##
## includes extended item information - examples:
## labels
## 1 abrasive cleaner
## 2 artif. sweetener
## 3 baby cosmeticsThere are 9834 transactions with 169 distinct items purchased. We print the most occurring items below. Whole milk, vegetables, rolls/buns, soda, yogurt lead the top 5 items as seen when we use itemFrequencyPlot to visualize the 20 most frequent items.
plot_item_frequencies_ranked = itemFrequencyPlot(grocery_transactions, topN=20, type = "absolute", main="Frequency Ranked")
We now display rules computed by the apriori algorithm from the groceries in which the support \(P(A \bigcap B) \geq 0.001\) and confidence \(P(A \bigcap B)/P(A) \geq 0.3\). By choosing a relatively high confidence level and support of 0.1%, we require at least 9-10 relevant transactions involving both \(A\) and \(B\). This reduces the problem that the discovered associations are statistically spurious.
rules <- apriori( grocery_transactions, parameter = list(supp = 0.001, conf = 0.3))## Apriori
##
## Parameter specification:
## confidence minval smax arem aval originalSupport maxtime support minlen
## 0.3 0.1 1 none FALSE TRUE 5 0.001 1
## maxlen target ext
## 10 rules TRUE
##
## Algorithmic control:
## filter tree heap memopt load sort verbose
## 0.1 TRUE TRUE FALSE TRUE 2 TRUE
##
## Absolute minimum support count: 9
##
## set item appearances ...[0 item(s)] done [0.00s].
## set transactions ...[169 item(s), 9835 transaction(s)] done [0.00s].
## sorting and recoding items ... [157 item(s)] done [0.00s].
## creating transaction tree ... done [0.00s].
## checking subsets of size 1 2 3 4 5 6 done [0.01s].
## writing ... [13770 rule(s)] done [0.01s].
## creating S4 object ... done [0.01s].Nonetheless, we still obtain over 13000 rules. The rules are then ranked by lift and the top 10 rules by lift are displayed below.
rules_lift= sort( rules, by ="lift", decreasing = TRUE ) # Display by lift
top_rules_by_lift = DATAFRAME(head(rules_lift, n = 10) )
top_rules_by_lift %>% kable(digits = 3) %>% kable_styling(bootstrap_options = c("hover", "striped"), position = "left")| LHS | RHS | support | confidence | coverage | lift | count | |
|---|---|---|---|---|---|---|---|
| 252 | {bottled beer,red/blush wine} | {liquor} | 0.002 | 0.396 | 0.005 | 35.716 | 19 |
| 906 | {ham,white bread} | {processed cheese} | 0.002 | 0.380 | 0.005 | 22.928 | 19 |
| 251 | {bottled beer,liquor} | {red/blush wine} | 0.002 | 0.413 | 0.005 | 21.494 | 19 |
| 311 | {Instant food products,soda} | {hamburger meat} | 0.001 | 0.632 | 0.002 | 18.996 | 12 |
| 1269 | {curd,sugar} | {flour} | 0.001 | 0.324 | 0.003 | 18.608 | 11 |
| 1193 | {baking powder,sugar} | {flour} | 0.001 | 0.312 | 0.003 | 17.973 | 10 |
| 905 | {processed cheese,white bread} | {ham} | 0.002 | 0.463 | 0.004 | 17.803 | 19 |
| 281 | {popcorn,soda} | {salty snack} | 0.001 | 0.632 | 0.002 | 16.698 | 12 |
| 1192 | {baking powder,flour} | {sugar} | 0.001 | 0.556 | 0.002 | 16.408 | 10 |
| 904 | {ham,processed cheese} | {white bread} | 0.002 | 0.633 | 0.003 | 15.045 | 19 |
The top rule, for example, associates the purchase of beer and wine with an additional purchase of liquor. The support \(P(A \bigca B)\) is 0.2%. The confidence of 0.396 tells us that 39.6% of purchases of the former also include purchases of the latter (liquor) and this association is extremely unlikely to be due to chance due to a lift of 35.7. We also notice a rule in the opposite direction. The top 3rd rule is effectively as association of beer and liquor to induce the purchase of red wine. As these rules are simply correlations of triples of items, we cannot really infer causality.
Rather, we are left to infer causality or reasonableness of the observed association by using common sense or other quantitative evidence.
We are also left to apply business knowledge and commonsense to distinguish the good and bad associations. For example, the top 4th rule (instant food products, soda) \(\implies\) hamburger meat does not quite make sense to me. While soda and hamburger meat might be associated with a BBQ, it is unclear what instant food products would mean.
Lastly, we should be cautious of inferring product placement from association rules.
While some associations may suggest moving associated products closer to increase revenues, the causality may actually go in the other direction. Some products are traditionally placed together. For example, baking powder, sugar and flour are usually in the same aisle.
Relocating some products closer to strengthen proximity (and increase revenues) may cause other product associations to weaken (and decrease revenues). It may be helpful to search other associations from the rule set and inspect the net impact.
Lastly, we visualize the rules by showing lift on the y-axis and support on the x-axis. The color intensity is related to confidence of the rule. We can use this to see if any rules with lower lift still have offsetting greater support. More support means more revenues could be impacted by a good marketing decision.
plot(rules, method = "scatterplot", measure = c("support", "lift"), shading = "confidence" )## To reduce overplotting, jitter is added! Use jitter = 0 to prevent jitter.
