Failure Injection with Kolton Andrus

Servers in a data center fail. Sometimes entire data centers have a power outage. Bugs in an application make it into production. Human operators make mistakes and cause data to be deleted.

Failure is unavoidable. We make backups and replicate our servers so that when a failure occurs, we can quickly respond to it without making the user feel much pain. But how can we test that our response will work before an actual catastrophe occurs?

Kolton Andrus is CEO of Gremlin, a company that works on failure injection as a service. Gremlin is based on ideas around planned failure that Kolton learned from his years at Amazon and Netflix.

We ended up talking as much about the culture of Netflix and Amazon as we did about how and why to build failure injection. It’s always nice to share war stories with other people who have worked at Amazon because the culture is so distinct. If you want to know more about Amazon’s culture, check out the episode tomorrow with Brad Stone, author of The Everything Store.

Transcript

Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view or download the transcript for this show.