- This is quite important as it dictates ultimately who has the final say in certain decision making. I’ve worked in settings where:
- consensus decision making is used heavily agnostic to attendees familiarity of the implications
- top down approach where attendees sometimes may feel unfair.
- Netflix runs A/B testing to build trust in the results. This is almost a requirement.
- Across the organisation, there is data-driven mindset. This makes Data & Insights team at the center of the company.
Below: Netflix’s A/B test Tooling back in 2001
- There is always multiple A/B tests running at any given time. It would be rare to have just 1 test running.
- When the results comes back, significant decisions are ultimately decided by “Informed Captains”, not consensus decision making.
- Informed captains are experts in the area, they are responsible for hearing all views before making the final call.
- Some decisions farms dissent, ensure all points are reviewed and then disagree and commit.
- Not all decisions are good ones either. Move on and reflect on the learnings along the way.
- Unfortunately there’s low win rate even with A/B testing. Even experts have low win rate, this is both humbling and motivating.
- Built internal tooling to support
- data retrieval,
- design evaluations
- analysis
- reproducible reporting
- Continued heavy investments in research tooling – something also not a lot of companies know how to or willing to implement.
- Experimentation Platform is a real team that has owners like product manager with roadmaps.
- If the road maps doesn’t get covered, there is a way to contribute democratically so other teams are not blocked based on Experimentation Platform team’s velocity.
- This is very interesting.
Takeways:
- Netflix has strong culture of experimentation
- Although Netflix’s approach is very academic and sounding, they may be one of the few companies that has the Engineering foundation and time to do all this research before making the changes.
- Netflix’s famous recommendation system worked out and drives user engagement, this shows the length of the company is willing to go through to make informed decisions.
Ref:
Coffee Break 
Leave a comment