Coffee Break

Topics:

• How Teams and projects operate
• Documentation
• Testing
• Codebase and Toolsets
• Team Compositions

How Teams and projects operate

• As big as Facebook(meta) is, very surprised on the mindset of how they handle work still similar to small companies
• Engineers wears a variety of different hats, not just coding. They lead many projects individually in terms of executions.
• Not Scrum focused, sounds more like Kanban. Less bureaucracy more efficiency
• Basically Engineers have a lot to say on the project after given the basic directions by PMs
• However also because of this, meetings are frequent and heavy

Releasing

• Record incidents when outages happens. Similar to Postmortems.
• Severity levels are as follow:
• SEV0: One or more critical sites are down. The 2021 Facebook outage, when all of Facebook, Whatsapp, Messenger, Instagram and Oculus went down for 7 hours, was a SEV0.
• SEV1: “There’s a large fire!” Urgent and very critical.
• SEV2: urgent, and critical.
• SEV3: urgent, but not critical.
• SEV4: not urgent, not critical.

Documentation

• Follows real engineer approach. The source of truth is in the code. No real convention of documenting
• Teams generally can decide themselves how much documentations are required.
• Usually is a Top-Down Approach when solving problems. EG: during code reviews, a piece of code is given the exact proposed change instead of going over back and forth what the change should be.

Testing

• Surprisingly for such a big company, it is a company not popular on unit tests.
• Kent Beck, creator of Extreme Programming and advocate of TDD joined FB when he was 50.

“I see this place, it’s crazy. It looks like a clown show and yet, things are working really well. They weren’t doing the things in my books. (…)

In the back of my mind there is the mystery of this bumblebee. In theory, this process should be a disaster. In practice, it’s working extremely well at doing two things at the same time: at scaling, and at exploration.”

• Unit tests was in the way of moving fast and is the major reason of not being adopted.
• If a test failed but the site was up, they just delete the test.
  • A test producing noise and not producing signal is considered bad test. It is a waste of time to figure out what broke the test
  • Majority of the issues in production was only found in production environment
• To compensate for the lack of tests, FB has a very extensive staged rollout plans that is more impressive than companies of similar size.
• Code goes through about 4 layers of deployments before going into production.All of this was automated.
• Still today this canary rollout is highly being relied upon as the user base is very big. Tests is more common today on products that are important. For non important products, tests are optional.
• Rarely have dedicated QAs across teams.

The above approach worked for FB due to the nature of the product.

• There’s large amount of users that can be used as canary for testing
• Small frequent bugs does not harm businesses but moving faster than competition is essential to gaining market share
• If the same approach was applied on Meta’s VR Horizon World, that explains why the VR App is so shitty. The user base on Horizon World is small and therefore making every bug much bigger impact to the overall userbase and company’s reputation.

Codebase and Toolsets

• FB is an advocate of monorepo. It houses everything:
  • www: core of FB
  • ios, android and others in in same repo just different folder names.
  • Whatsapp, Instagram, Messengers are in different folders.
• Trunk based approach
• Builds every tool in house due to early days these tools were not available or could not scale to what FB wants.
  • Eg. Built pagerduty instead of paying pagerduty and then later finding out that pagerduty cannot support some edge case that they need.
  • Facebook Workplace – Slack + wiki + documentation + social.
  • Workplace has plugins for many other Facebook tools. For example, people can type “#fileabug {add details}” and this would automatically create a bug ticket.

Team Compositions

• As a product first company, project ideas that have higher potential impact are paid more attention.
• Engineers may choose to join a team at will. Usually this will be based on the perceived impact of the project.
• Head counts exists, but not fixed allocated to specific teams. It is quite fluid. As long as a team still has “Position Ids” they can either hire more or get allocated with new team members.

Ref: https://newsletter.pragmaticengineer.com/p/facebook-2#%C2%A7performance-reviews-bonuses-and-equity-refreshers