It was nice to hear about a real word implementation of recommender system and challenges faced in implementing it.He discussed different ways of implementing recommender systems and pros and cons.
For example, Pandora manually tagging 200+ attributes and scalability issues related to the manually process.

  1. Manual Curation
  2. Manually Tag Attributes
  3. Audio Content, Metadata, Text Analysis
  4. Collaborative Filtering

Different ways to factorize matrices like implicit and explicit matrix factorization and solving it using ALS.

Implicit Matrix Factorization

Implicit Matrix Factorization

Implicit Matrix Factorization

Explicit Matrix Factorization

Explicit Matrix Factorization

Explicit Matrix Factorization

ALS

ALS

ALS

Books and articles rarely discuss actual production implementation and challenges implementing solutions. He discussed about the problems they faced with Hadoop and using Hadoop in high traffic production environment. He talked about the incredible growth Spotify experienced and the challenges of implementing solutions that scales and why they had to move to spark. This remined me some challenges I faced with implementing/designing scalable solutions. He also discussed some of the problems they had with using APIs, this remined me of the API issues I faced when building a system to extract meta data from MS office documents using third party APIs. Implementing scalable solutions for millions of users and for big data is not an easy task. This video highlighted some of the challenges involved in that.

Hadoop I/O overhead

Hadoop

Hadoop

Spark over Hadoop

Spark

Spark