Big Data Tools and Analytics Group Project

  Market Basket Analysis

Alessandro Consiglio | Marc Behse | Emil Henriksson Ene


1. MBA - Questions

2. Transactions & Item Frequencies

2.1. Example of Transactions

##     items                                                    transactionID
## [1] {All Natural No Stir Creamy Almond Butter,                            
##      Carrots,                                                             
##      Classic Blend Cole Slaw,                                             
##      Coconut Butter,                                                      
##      Garlic Powder,                                                       
##      Michigan Organic Kale,                                               
##      Natural Sweetener,                                                   
##      Organic Egg Whites,                                                  
##      Original Unflavored Gelatine Mix}                                   2
## [2] {Air Chilled Organic Boneless Skinless Chicken Breasts,               
##      Lemons,                                                              
##      Organic Baby Spinach,                                                
##      Organic Ezekiel 49 Bread Cinnamon Raisin,                            
##      Organic Ginger Root,                                                 
##      Total 2% with Strawberry Lowfat Greek Strained Yogurt,               
##      Unsweetened Almondmilk,                                              
##      Unsweetened Chocolate Almond Breeze Almond Milk}                    3

2.2. Example of item Frequency

##       0% Fat Peach Greek Yogurt  0% Fat Strawberry Greek Yogurt 
##                    1.007455e-05                    7.052186e-05 
## 0% Fat Superfruits Greek Yogurt     0% Fat Vanilla Greek Yogurt 
##                    1.208946e-04                    3.022366e-05

2.3. Top 20 most frequent items

3. Apriori algorithm

## Apriori
## 
## Parameter specification:
##  confidence minval smax arem  aval originalSupport maxtime support minlen
##        0.25    0.1    1 none FALSE            TRUE       5   0.006      2
##  maxlen target  ext
##      10  rules TRUE
## 
## Algorithmic control:
##  filter tree heap memopt load sort verbose
##     0.1 TRUE TRUE  FALSE TRUE    2    TRUE
## 
## Absolute minimum support count: 595 
## 
## set item appearances ...[0 item(s)] done [0.00s].
## set transactions ...[35098 item(s), 99260 transaction(s)] done [0.48s].
## sorting and recoding items ... [199 item(s)] done [0.01s].
## creating transaction tree ... done [0.04s].
## checking subsets of size 1 2 3 done [0.01s].
## writing ... [16 rule(s)] done [0.00s].
## creating S4 object  ... done [0.01s].

3.1. Apriori output

# look at the first three rules
inspect(groceryrules[1:3])
##     lhs                                       rhs                          support confidence   coverage     lift count
## [1] {Yellow Onions}                        => {Banana}                 0.006488011  0.2898290 0.02238565 1.949212   644
## [2] {Organic Large Extra Fancy Fuji Apple} => {Bag of Organic Bananas} 0.007263752  0.3140244 0.02313117 2.626617   721
## [3] {Organic Gala Apples}                  => {Bag of Organic Bananas} 0.006588757  0.2793678 0.02358453 2.336736   654
# sorting grocery rules by lift
inspect(sort(groceryrules, by = "lift")[1:5])
##     lhs                                       rhs                          support confidence   coverage     lift count
## [1] {Organic Lemon}                        => {Organic Hass Avocado}   0.006981664  0.2533821 0.02755390 3.839216   693
## [2] {Organic Large Extra Fancy Fuji Apple} => {Bag of Organic Bananas} 0.007263752  0.3140244 0.02313117 2.626617   721
## [3] {Organic Fuji Apple}                   => {Banana}                 0.010326415  0.3742242 0.02759420 2.516803  1025
## [4] {Organic Raspberries}                  => {Bag of Organic Bananas} 0.012794681  0.2996697 0.04269595 2.506548  1270
## [5] {Organic Hass Avocado}                 => {Bag of Organic Bananas} 0.019101350  0.2894215 0.06599839 2.420829  1896

4. Network of rules

plot(groceryrules, method = "graph",  engine = "htmlwidget")

5. Spark vs. R

5.1 R - 32.000.000 ordered items

 

Processing the dataframe with 32.000.000 observations exhausted the on-premise memory

5.2 Spark - 32.000.000 ordered items

 

Using the Spark framework on Databricks enabled us to run the command in 59.70 seconds

The apriori defined 381 rules