FB Prep 2

Problem → H₀ → DoE → Assess → Conclusion & Recommendation

Examples of current work

1. Experimentation Platform
   
2. Modelling

Stategic and Leadership

Ask what?
Why? Why we do what we do? and motive?

Hypothesis Driven

Key is not just coming up with hypotheses, rather frame the problem.

Hypothesis is the statement of solution to an specific problem, that can be tested, hence approved or disapproved by evidence (data).

Fallacies
- mix shift aka simpson paradox - selection bias
- regression to the mean
- recency, sesonal effect? … experiment just on one day… could it be breaks? off season? peak? experiment more. - information bias… more information is not always better.. determine key measure(s) then ride with it

• data shiftt (covariate shift in predictors, prior probability shift, concept shift) … more for modelling
• avoid label leakage

First treatment, what is the treatment? why? Distinguish from control?

Power and Randomize is import, but not at the top of list…

SQL

• joins
  
• nested query
  
• analytic functions (aggregate, window)

kaggle adv. SQL

Question to ask

1. What’s your day like ? Followed by favorite part ? Top challenges ?
2. How would you describe the facebook culture?
3. Ax adoption … how does it compare to classical experimentation ? Bayesian, anyone?