Huge scale deployments

Deploying to 10 or 20 servers can be complicated. But what are some of the tools, tips and patterns for successfully deploying to 1,000s of servers around the world? Last month I participated in an online panel on the subject of Huge Scale Deployments, as part of Continuous Discussions (#c9d9), a series of community panels about Agile, Continuous Delivery and DevOps hosted by Electric Cloud. You can watch a recording of the panel:

The discussion progressed through a short deck of slides with big ticket titles. Plenty of interesting things were said (mostly by other people), but the main insight I had was that the practices discussed are applicable at any scale. Meanwhile, here are a few edited extracts from my musings on the video.

How do you practice huge scale deployments? How did the teams at Amazon make it work?

“I was a software developer at a little-known Amazon development center in Edinburgh, Scotland. … at Amazon, teams are responsible for provisioning environments, continuous deployment, everything on their own. Because we knew that we were picking up the pieces if it went wrong, we were going in making sure our deployment was good in early stages of the pipeline, because we had responsibility. When people have responsibility they make sure that things don’t break.”

“With regard to incentives – at Amazon there’s no need to dangle carrots, the teams know that everything that happens, designing, provisioning for expected loads, all the way to fixing problems in production, is their responsibility. So everyone practices ‘if it hurts do it more often’. It’s a natural human tendency to back off if there’s pain, but in software development, if something causes you pain you need to really work at that, you’re just not good enough at it […]

By |September 1st, 2015|Continuous Delivery, ContinuousDelivery, Musings, Practices|0 Comments

Branching and Continuous Delivery video discussion

Following on from my previous post and discussions online, we arranged a Google Hangout to discuss things in more detail. I was joined by Dave Farley, Lars Kruse, Olve Maudal and Mike Long and you can watch the unedited video here:

I don’t think we reached any agreement and after the video ended Olve suggested that we meet again to focus on specific topics, in several sessions. Olve’s suggestions were (paraphrased by me):

pre/post-commit checks and their effect on feedback speed
benevolent dictator/trusted serf as a general model
even more about branching

If you’d like to see more, please Tweet with the hashtag #cdBranching indicating which (if any) you’d like to hear more about.

By |May 20th, 2015|Agile, Continuous Delivery, ContinuousDelivery|1 Comment