Release Engineering Lessons from Google and Facebook

I had the good fortune to be a release engineer at Google and Facebook from 2006–2015, and participated in the development and trial of deployment patterns which are now common throughout the tech industry.

First: consumer-facing apps are, at best, dynamically stable. They grow more stable the more frequently you are able to deploy small changes to production. This can be counterintuitive — because folks usually recall “oh, that’s the last change that broke something.”

However, an overall development and deployment cycle that can’t deploy quickly isn’t robust, it’s brittle and unresponsive. That makes sense if, say, you’re deploying code to a VAX cluster in a submarine, but not for a consumer web property or the backend APIs for a mobile app.

Remember that since Accelerate by Dr. Forsgren et al. was published in 2018 (at least), we’ve known that bureaucratic oversight before deployment from uninvolved staff does not improve code quality. Sorry, SVP of WTF, your sign-off does not make a deployment more stable or a release bug free. The same goes for managers who are not part of a relevant on-call.

Much like lifting weights, the way to get better is to do it. I’d trust a team that routinely deploys new code and reverts errors over one that does a six-week QA cycle and then ships. (Worth noting: teams describing things as “NASA-style” should read a bit about software and the space program; there were hotfixes in live instructions and ferrite-core memory, too.)

This is a complex topic, but some of the precepts I use are:

1. Your team is not prepared for what it never practices.
2. A production environment is never 100% duplicated outside of production.
3. Your safety nets have degraded silently. Tests are now disabled or ignored. The metrics are incomplete.
4. Heroic effort and hypervigilance are exhaustible, and you’re not able to screen or hire for those traits anyway.
5. Your first instinct should be to revert a flag or deployment. If there is any doubt, there is no doubt.