Big Data Tools and Analytics Group Project
Market Basket Analysis
Alessandro Consiglio | Marc Behse | Emil Henriksson Ene
## items transactionID
## [1] {All Natural No Stir Creamy Almond Butter,
## Carrots,
## Classic Blend Cole Slaw,
## Coconut Butter,
## Garlic Powder,
## Michigan Organic Kale,
## Natural Sweetener,
## Organic Egg Whites,
## Original Unflavored Gelatine Mix} 2
## [2] {Air Chilled Organic Boneless Skinless Chicken Breasts,
## Lemons,
## Organic Baby Spinach,
## Organic Ezekiel 49 Bread Cinnamon Raisin,
## Organic Ginger Root,
## Total 2% with Strawberry Lowfat Greek Strained Yogurt,
## Unsweetened Almondmilk,
## Unsweetened Chocolate Almond Breeze Almond Milk} 3
## 0% Fat Peach Greek Yogurt 0% Fat Strawberry Greek Yogurt
## 1.007455e-05 7.052186e-05
## 0% Fat Superfruits Greek Yogurt 0% Fat Vanilla Greek Yogurt
## 1.208946e-04 3.022366e-05
## Apriori
##
## Parameter specification:
## confidence minval smax arem aval originalSupport maxtime support minlen
## 0.25 0.1 1 none FALSE TRUE 5 0.006 2
## maxlen target ext
## 10 rules TRUE
##
## Algorithmic control:
## filter tree heap memopt load sort verbose
## 0.1 TRUE TRUE FALSE TRUE 2 TRUE
##
## Absolute minimum support count: 595
##
## set item appearances ...[0 item(s)] done [0.00s].
## set transactions ...[35098 item(s), 99260 transaction(s)] done [0.48s].
## sorting and recoding items ... [199 item(s)] done [0.01s].
## creating transaction tree ... done [0.04s].
## checking subsets of size 1 2 3 done [0.01s].
## writing ... [16 rule(s)] done [0.00s].
## creating S4 object ... done [0.01s].
# look at the first three rules
inspect(groceryrules[1:3])
## lhs rhs support confidence coverage lift count
## [1] {Yellow Onions} => {Banana} 0.006488011 0.2898290 0.02238565 1.949212 644
## [2] {Organic Large Extra Fancy Fuji Apple} => {Bag of Organic Bananas} 0.007263752 0.3140244 0.02313117 2.626617 721
## [3] {Organic Gala Apples} => {Bag of Organic Bananas} 0.006588757 0.2793678 0.02358453 2.336736 654
# sorting grocery rules by lift
inspect(sort(groceryrules, by = "lift")[1:5])
## lhs rhs support confidence coverage lift count
## [1] {Organic Lemon} => {Organic Hass Avocado} 0.006981664 0.2533821 0.02755390 3.839216 693
## [2] {Organic Large Extra Fancy Fuji Apple} => {Bag of Organic Bananas} 0.007263752 0.3140244 0.02313117 2.626617 721
## [3] {Organic Fuji Apple} => {Banana} 0.010326415 0.3742242 0.02759420 2.516803 1025
## [4] {Organic Raspberries} => {Bag of Organic Bananas} 0.012794681 0.2996697 0.04269595 2.506548 1270
## [5] {Organic Hass Avocado} => {Bag of Organic Bananas} 0.019101350 0.2894215 0.06599839 2.420829 1896
plot(groceryrules, method = "graph", engine = "htmlwidget")
Processing the dataframe with 32.000.000 observations exhausted the on-premise memory
Using the Spark framework on Databricks enabled us to run the command in 59.70 seconds
The apriori defined 381 rules