How Facebook develops code

I am fascinated by the workings of Facebook. This is a very unique society, not easily reconstituted (and their method would not work for all companies, even if they tried). It notes, collected from conversations with many friends from Facebook about how the company develops and markets software products.

It has been over six months since I collected these observation, and I’m sure that even now, Facebook is constantly improving their methods of software development. So these notes may be a bit outdated. And also, it seems that the culture of Facebook, managed by the developers, is gaining public attention. So now I feel more comfortable letting the notes … Many thanks to many people who helped put together a presentation about Facebook from within! Also express gratitude to people and epries fryfrog, who made ​​corrections and editing.

Notes:

  • As of June 2010 the company was nearly 2000 employees, with 1100 employees in 10 months before. Almost doubling the state less than a year!
  • The largest team – the specialists and the operational department, about 400-500 people in each. Both teams nearly 50% statewide. The ratio of managers to the number of specialists around 1:7 or 1:10.
  • All experts are in 4-6 week training at the training center, where they learn the system through the Facebook bug fixes (bug fixing) and listening to lectures senior / professional staff.
  • Approximately 10% of people in each training class does not go ahead and retire with the recommendations from the company. After the training center all specialists have access to the actual database (accompanied by a standard lecture, “With great power comes great responsibility” and a clear list of the “crime-to-dismissal» (fire-able offenses), for example, the disclosure of personal data). [Ed. thx fryfrog] «There is also a very good security measures to protect anyone in the company from doing terrible things that might come to mind.
  • Surrounding have the opportunity to become familiar with the case and try to solve the problem. But if you really “get” those who need help, this fact is recorded along with cause and carefully considered.
  • To stray from the true path is not allowed, of course. “Any person can edit any part of the code Facebook and Komichi it as desired.Culture, driven by developers. “Product Manager, essentially, useless here” – a quotation specialist. Specialists may change the specifications of the design process, change the order of business projects and implement new ideas at any time.
  • During the monthly inter-team meetings, the experts – the ones who provide a progress report.
  • Marketing departments and management participate in these meetings, but if they are too honest, this is reported to senior management as “the product too much talking at the last meeting.” They actually want to open the specialists were the owners of their development and is the main point of contact for the projects that they are also developed.
  • Resource allocation to projects is completely voluntary.
    • PM gathers a group of specialists, trying to give them the opportunity to enter into excitement when discussing their own ideas.
    • Specialists decide which of the ideas sound more interesting to start work on it.
    • Specialists communicate with their managers and say: “I would like to work here over these five things during the week. “Tech. Director usually leaves the preferences of experts at their discretion, may occasionally be asked to do certain tasks in the first place.
    • Professionals to manage the entire development of themselves – JavaScript on frontend code database on the backend and everything in between.
    • If they want to help the designer (specialized staff of designers is limited), they have sufficiently strong interest to the designer, so he took it for their project.
    • The same applies to the architects. But it is expected that in most cases, experts will handle all your needs.
  • Is it worth a damn idea usually becomes clear within a week of its implementation and further testing of selected users, such as 1% of users of Nevada.
  • In general, specialists prefer to work on infrastructure, scalability, and “difficult problems” – the most prestigious area. Can be difficult to observe professionals enthusiastically working on the project’s front-and user interfaces.
  • This is the opposite of what you can see in other consumer markets, where everyone wants to develop things that users directly touched, and you can poke a finger at a particular part and say “I did it.”
  • In Facebook backend, such as algorithms, newswires, algorithms-targeted advertising, optimization memcache, etc. – first-class projects that professionals want to work.Comite, which affect the certain high-priority functional (for example, news ticker), pass code review before merge (note first. «Merge»).
  • Newsline is very important, so that he Zuckerberg view any of the changes, but it is – an exceptional case.[Correction – thx epriest] «There is a mandatory code reviews of all changes (one or several experts).
  • I think the point of simply explains that the CMC does not look at each change personally. “[Correction thx fryfrog] «All changes are reviewed by at least one person, and the system is such that any other can take and view your code, even if you have not asked.
  • Otherwise, it could lead to the deliberate introduction of malicious code in the untrusted code. “Specialists are responsible for testing, bug fixes and support for their work after the launch.
  • There are several unit-testing and integration-testing frameworks, but they are only used from time to time.[Correction thx fryfrog] «I would also like to add that we have, of course, is QA, not just the official group.
  • Every employee at the office or connected via VPN, uses a version of the site, including all changes in the queue for the next computation. This version is updated constantly, and usually for 1-12 hours before the whole world would see her.
  • All employees are urged to report any bugs, and it all works very well. “re: surprised by the lack of QA or automated unit tests – the majority of specialists are able to write error-free code. This is something that they see no point to do in most companies: when there is a QA-department, it is easy to just throw them all to find mistakes. “[Please note that this was a subjective opinion, I wrote this because of the striking contrast which is seen as standard practices for other companies]. [Correction thx epriest] «We have an automated testing, including” push-blocking “tests that must pass before laying out the release.
  • We absolutely do not believe in the phrase “most experts are able to write error-free code, we no longer believe that this is reasonable as one of the basic design principles.”re: surprised at the lack of influence / control PM – managers have a lot of independence and freedom.
  • The key to independence – to build a really good relationship with technical directors. Need to be fairly technically savvy, not to offer a silly idea.
  • In addition, there is no need to request permission or to pass any test roadmepov / beklogov. “My director of product does not even know all the things that are in my roadmepe.
  • Accordingly, there are some PM, but they all feel great responsibility for a really important area for a company with a personal interest.
  • By default, all packed into Comite code releases weekly (Tuesdays).With extra effort changes may be posted on the same day.Press releases on Tuesdays require the presence of all professionals who Komichi code last week for the release candidate, which should be posted.
  • Before the beginning of release specialists should be present at a special IRC-channel for the “Call to the calculation, otherwise will be punished in public” shame “.The team runs the tellers release, gradually rolling it on the server.
    • Facebook has about 60,000 servers.
    • There are 9 concentric levels for rolling out the new release. [Correction thx epriest] «Nine stages of the calculations are not concentric.
    • There are 3 concentric phase (p1 = internal release, p2 = small external release, p3 = external full release).
    • The remaining six stages – such as levels of domestic support tools, server, video downloads, etc. “The smallest level – 6 servers. For example, each vtornichny release rolls out of 6 servers (level 1), then the team oversees the tellers of these 6-servers and makes sure that they work properly before rolling out to the next level.
    • If the release there are any problems (eg, falling errors, etc.), then the calculation will be canceled. Specialist who did the faulty Comite, called to correct the error.
    • Then the calculation starts from the beginning.Thus, the release may pass through multiple levels: 1-2-3-fixes. Return to 1. 1-2-3-4-5-fixes. Return to 1. 1-2-3-4-5-6-7-8-9.
  • Team tellers really well prepared, united and very concerned about his case. Their server figures represent more than just error reports, performance load and memory usage – they also include custom indicators. For example, if a new release changes the percentage of people using Facebook, the team tellers see it in their terms and therefore can stop the release to clarify the issue.
  • In the calculations of release team tellers uses paging based on the IRC, which can to throw the information engineers via Facebook, e-mail, IRC, IM or SMS, if you want their attention. Ignoring messages tellers leads to publicly “shame”.
  • Once the code is racked at level 9 and it is stable, weekly vakladka is completed.If the functionality was not developed in time to the day of week calculations, it is not as critical (if it does not contain a rigid external dependencies) – functional only be fully implemented when it is incomplete.
  • Receiving svn-complaint (svn-blammed), public shaming, or too frequent project delays could affect a specialist dismissal. “This is a very high performance culture”. People who are not productive or not super-gifted, really put themselves in jeopardy.
  • Managers will take literally, will be allocated toward underachieving within 6 months after hiring, and say, “Just did not work, you are not suited for this culture.” Generally, it is applicable to any level of the company even hired a C-level and VP-level were quickly dismissed, if they were not super-productive.[Correction, thx epriest] «People are not called to familiarize themselves with mistakes. They are called only if they asked for the change included in the release, but not to support the change, when something went wrong (and if you have not found anyone to replace).[Correction, thx epriest] «Because of the complaints you will not be fired (Anon: meaning svn-blame).
  • We are very lenient in this regard, and most of the major specialists laid out at least one terrible thing, including myself. As far as I know, no one was ever fired for committing such mistakes. “[Correction, thx fryfrog] «I also do not know anyone who would be fired for mistakes in the article. I know people who accidentally dropped his site. They work hard on correcting what caused the problem, and everyone learns from this. Public shaming is much more effective than the fear of being fired, in my opinion. “

Will be very interesting to see how the culture of development in Facebook evolves with time, and especially to see how this culture will continue to expand with the expansion of the company to thousands of employees.

How Facebook develops code

How Facebook develops code

What do you think? Will the “culture-driven developer, working in your company?