Continuous Delivery

/Continuous Delivery

Huge scale deployments

Deploying to 10 or 20 servers can be complicated. But what are some of the tools, tips and patterns for successfully deploying to 1,000s of servers around the world? Last month I participated in an online panel on the subject of Huge Scale Deployments, as part of Continuous Discussions (#c9d9), a series of community panels about Agile, Continuous Delivery and DevOps hosted by Electric Cloud. You can watch a recording of the panel:

The discussion progressed through a short deck of slides with big ticket titles. Plenty of interesting things were said (mostly by other people), but the main insight I had was that the practices discussed are applicable at any scale. Meanwhile, here are a few edited extracts from my musings on the video.

How do you practice huge scale deployments? How did the teams at Amazon make it work?

“I was a software developer at a little-known Amazon development center in Edinburgh, Scotland. … at Amazon, teams are responsible for provisioning environments, continuous deployment, everything on their own. Because we knew that we were picking up the pieces if it went wrong, we were going in making sure our deployment was good in early stages of the pipeline, because we had responsibility. When people have responsibility they make sure that things don’t break.”

“With regard to incentives – at Amazon there’s no need to dangle carrots, the teams know that everything that happens, designing, provisioning for expected loads, all the way to fixing problems in production, is their responsibility. So everyone practices ‘if it hurts do it more often’. It’s a natural human tendency to back off if there’s pain, but in software development, if something causes you pain you need to really work at that, you’re just not good enough at it […]

By |September 1st, 2015|Continuous Delivery, ContinuousDelivery, Musings, Practices|0 Comments

Branching and Continuous Delivery video discussion

Following on from my previous post and discussions online, we arranged a Google Hangout to discuss things in more detail. I was joined by Dave Farley, Lars Kruse, Olve Maudal and Mike Long and you can watch the unedited video here:

I don’t think we reached any agreement and after the video ended Olve suggested that we meet again to focus on specific topics, in several sessions. Olve’s suggestions were (paraphrased by me):

pre/post-commit checks and their effect on feedback speed
benevolent dictator/trusted serf as a general model
even more about branching

If you’d like to see more, please Tweet with the hashtag #cdBranching indicating which (if any) you’d like to hear more about.

By |May 20th, 2015|Agile, Continuous Delivery, ContinuousDelivery|1 Comment

Continuous delivery conversations

Last week I was at the CoDeOSL conference in Oslo. It was an interesting day, with some very good sessions, but as usual it was conversations had in the breaks that were of the most interest. I’d like to describe two discussions that I had that seem, on the face of it, to be in conflict.
Conversation #1
The night before the conference there was a meetup in a bar just around the corner from the conference hotel. I arrived a bit late and tired (because I was still recovering from the ACCU conference which had finished the previous Saturday) and struck up a conversation with Mike Long (who was also recovering from the ACCU conference). We were talking about patterns of interacting with source control that support continuous delivery – specifically the Automated Git Branching Flow that has been proposed by JOSRA.

Mike had an A5 leaflet that described the flow, including the diagram above, but I found that I needed to read the full article on the web before I could start to make sense of it. My distillation of their words is that:

continuous delivery depends on trunk always being potentially shippable
merging is always dangerous, so don’t allow complex merges onto trunk
automate the process of performing trivial, fast-forward merges onto trunk

Adopting this process (using the Open Source Jenkins plugins provided) leads to a flow that requires developers to merge from trunk before pushing their changes. As long as the automated merge to trunk that results from this push is a fast-forward merge, then everything is good. If it isn’t, then the push fails and the developer has to fix the problem locally, before attempting to push again.

Steve Smith voiced some scepticism on Twitter:

Conversation #2
At lunchtime I headed into Oslo to get an anniversary […]

By |May 2nd, 2015|Continuous Delivery, Practices|3 Comments

Rolling Rocks Downhill

It’s almost a year since I posted a glowing review of “The Phoenix Project” – a business novel, following in the footsteps of Goldratt’s “The Goal”, about continuous delivery. If you haven’t yet read it, then I’m going to recommend that you hold fire, and read “Rolling Rocks Downhill” by Clarke Ching instead.

I should point out that Clarke is a personal friend of mine and a fellow resident of Edinburgh, so I may be a little biased. But I don’t think that this is why I feel this way – it’s just that Clarke’s book feels more real. Both “Rolling rocks….” and “Phoenix…” have a common ancestor – “The Goal”; both apply the Theory of Constraints to an IT environment; and both lead eventually to triumph for the team and the lead protagonist.

There’s something decidedly european about Clarke’s book, which makes me feel more comfortable. The reactions of the characters were less cheesy and I really felt like people I know would behave in the manner described. (Maybe that’s because these characters really are based upon people I know ;)) There really are cultural differences between Europe and North America, and this book throws them into sharp relief. The inclusion of the familiar problems caused by repeated acquisitions and a distributed organisation only added to the feeling that this story actually described the real world, rather than an imaginary universe powered by wishful thinking. The only time that my patience wore thin was around the interventions of the “flowmaster”, Rob Lally, who is ironically a real person.

A truly useful addition in Clarke’s novel is the description of the Evaporating Cloud diagram, which (I now know) is one of the 6 thinking processes from the theory of constraints. For […]

By |February 12th, 2015|Agile, Continuous Delivery, Practices|1 Comment

5 rules of continuous delivery

Inspired by Sandi Metz’s BaRuCo 2013 presentation “Rules” (which you should watch if you haven’t yet) I started thinking about whether there were some  rules that might be useful in the continuous delivery domain to “screen for cooperative types”.

I came up with these as a starting point:

Check in everything – we’re used to putting source code in version control, but we’re often less good at configuration management. Do you control your test data? Your database scripts? Your operating system patch level? If not how can you be sure that the environment you provision tomorrow will be identical to the one you provisioned last week?
Automate everything – “to err is human”, as the saying goes. Any manual process is susceptible to failures, so get rid of as many as you can. Some continuous delivery pipelines still have manual approval steps in them as part of the process, but the presence of the human is not functionally essential.
Continuous != occasionally – the more we do something, the easier it gets. One of the reasons to do something continuously is to decrease the cost and remove the fear. If it “costs too much” to deploy often, then work on reducing the cost not reducing the frequency.
Collaborate – people are not plug compatible. To get the most from the different people in the organisation we need to work together. For me, this was one of the major “innovations” of devops – no more ‘us’ and ‘them’, just ‘we’.
One step at a time – it’s hard to do everything all at once, which is why we iterate in software development. Continuous delivery is no different – don’t expect everything to “just happen”.

Do these make sense to you? What essential ingredients have […]

By |February 28th, 2014|Agile, Continuous Delivery, Practices|3 Comments