Musings

/Musings

Huge scale deployments

Deploying to 10 or 20 servers can be complicated. But what are some of the tools, tips and patterns for successfully deploying to 1,000s of servers around the world? Last month I participated in an online panel on the subject of Huge Scale Deployments, as part of Continuous Discussions (#c9d9), a series of community panels about Agile, Continuous Delivery and DevOps hosted by Electric Cloud. You can watch a recording of the panel:

The discussion progressed through a short deck of slides with big ticket titles. Plenty of interesting things were said (mostly by other people), but the main insight I had was that the practices discussed are applicable at any scale. Meanwhile, here are a few edited extracts from my musings on the video.

How do you practice huge scale deployments? How did the teams at Amazon make it work?

“I was a software developer at a little-known Amazon development center in Edinburgh, Scotland. … at Amazon, teams are responsible for provisioning environments, continuous deployment, everything on their own. Because we knew that we were picking up the pieces if it went wrong, we were going in making sure our deployment was good in early stages of the pipeline, because we had responsibility. When people have responsibility they make sure that things don’t break.”

“With regard to incentives – at Amazon there’s no need to dangle carrots, the teams know that everything that happens, designing, provisioning for expected loads, all the way to fixing problems in production, is their responsibility. So everyone practices ‘if it hurts do it more often’. It’s a natural human tendency to back off if there’s pain, but in software development, if something causes you pain you need to really work at that, you’re just not good enough at it […]

By |September 1st, 2015|Continuous Delivery, ContinuousDelivery, Musings, Practices|0 Comments

Flatulent agile

Recipes
I’ve been working with a number of larger, older organisations recently and it has really brought home to me the difference between the promise of a nimble, responsive teams and the reality of a sluggish, bureaucratic behemoth. Then, looking back over years of writings, posts, promises and dreams I see frequent repetitions of the phrase “Agile says that …” What? What exactly does agile ‘say’?

I’m not going to invoke the manifesto. It’s there if anyone wants to look at it and it is a valuable historical document. Its contents are as relevant today as they were when they were written (tinkering aside), but it is declarative not imperative. It is aspirational not procedural. We can’t ‘implement’ it or ‘transition’ to it.

Instead we have the plethora of methods that existed at the time of its original drafting and a few more besides. Many of them include recipes, processes, state diagrams – but they don’t seem to reliably help organisations improve their ability to deliver. These organisations need to change, but not according to some checklist brought down from the agile mountain chiselled in stone. The agile values and principles help us suggest ways of change, but that’s as far from a recipe as the contents of a box of oatcakes are from the serving suggestion in the picture.
Wind
There are plenty of winds in the world. There are the four winds. There are the trade winds. There are the winds of change. And there is… how shall I put this delicately?… bottom wind.

One of the definitions of flatulent is “generating excessive gas in the alimentary canal”, but it’s not that relevant to what we’re discussing. A more appropriate definition for my purposes is:

“having unsupported pretensions; inflated and empty; pompous; turgid”

I’ve heard […]

By |August 27th, 2015|Agile, doa, Musings|0 Comments

Is the customer always right?

Today I’m in London for the first day of SPA 2015. I was early to the venue and I went to the registration desk to sign in.

“What’s your name?” asked the receptionist.

“Seb Rose” I replied.

“No. You’re not registered” she said.

After a few minutes of spelling out my name, and searching around on the database she did find me. She looked up at me and said: “You are Rose, Sebastian”.

It’s true she was not a native english speaker, and that I have a slight accent, but even so I think you’ve got to assume that I know my own name. The reason I bring this up is that it’s something that happens a lot in all walks of life – people assume that the system is right, its data canonical. There’s no reason to believe this – it’s a form of institutional hubris.

Start from a more humble position. Accept the possibility of ignorance, the likelihood that the problem is rooted in the system not the customer. And even when it turns out that the customer is the cause of the error remember that it’s probably a failing of the system that allowed them to screw up anyway.

The customer is often right, and when they’re not it’s not good business to rub their nose in it.

By |June 28th, 2015|Musings|0 Comments

Entanglement (or there’s nothing new under the sun)

I’ve just read The Age of Entanglement : When Quantum Physics was Reborn by Louisa Gilder. It’s a tremendous book, looking at the interplay between great physicists over the whole of the 20th century. If you want to learn about quantum physics itself, this is probably not the book for you, but if a mix of science and history is your thing, then I can’t recommend this book enough. But that’s not why I’m writing this post.

As I read the book, there were two passages that jumped out at me because they were so relevant to experiences I have regularly. One was about testing and the other was about collaboration. Maybe I shouldn’t have been surprised, but there you have it – I was.

Experimental physicists understand that testing saves time. They sound a lot like developers who:
“want to slap it all together, turn it on, and see what happens.”
Inevitably:
“you can almost guarantee it’s not going to work right.”
Doesn’t this sound familiar? Their conclusion might sound familiar too:
“People always think you don’t have the time to test everything. The truth is you don’t not have the time. It’s actually a time-saving way of doing it.”

And then I found the description of a conference that sounded like a pre-cursor to the modern, open space ‘unconferences’ that have been springing up. An explicit acknowledgement that:
“the best part of any conference is always the conversation over coffee or beer, the chance meeting in the hall, the argument over dinner.”
This led directly to their decision to:
“organise their conference to be nothing but these events. No prepared talks, no schedule, no proceedings.”
I’ve heard of regular, private get-togethers like this that go on in the software community, where a selected group of invitees hole up […]

By |February 19th, 2015|Musings, Practices, Systems, Unit testing|0 Comments

Always Be Coding

Last night I finally got around to watching Erik Meijer’s keynote from last year’s Reaktor conference. It was called “One Hacker Way” and, while it contains much that is apocryphal – or at least wildly inaccurate – it scores over the older, more pedestrian type of keynote in two important ways: first it is highly contentious, and second, Erik sports a bright, tie-dyed T-shirt.

Some of the highlights of this personal rant were:

“Agile is a cancer”
“TDD is for pussies”
Stand-ups eat into valuable coding time
Coders should be treated (and paid) like professional athletes
Hierarchical structures work for the army and the church, so they must be good

There were times that I found myself nodding in agreement, though – when he compared Scrum certification to a pyramid selling scheme, for instance – but mostly it was just opinion, conspicuously lacking the support of empirical data (which was one of his criticisms of agile).

I could go on, but if you’re still interested then I recommend you just go watch the video.

My final thought is that whole experience was vaguely reminiscent of a scene from the excellent movie “Glengarry Glen Ross”, where Alec Baldwin’s character harangues the dejected real-estate salesmen:
F*** YOU, that’s my name!! You know why, mister? ‘Cause you drove a Hyundai to get here tonight, I drove an $80,000 BMW. That’s my name!! And your name is “you’re wanting”. And you can’t play in a hacker’s game. You can’t write code. Then go home home and tell your wife your troubles. Because only one thing counts in this life! Checking in code! You hear me, you f***in’ faggots? A-B-C. A-Always, B-Be, C-Coding. Always be coding! Always be coding!
And, if you’re over 32 (when Erik […]

By |January 16th, 2015|Agile, doa, Musings, Practices, TDD, Unit testing|0 Comments

Reduce, Reuse, Recycle

From http://kids.niehs.nih.gov/explore/reduce/:
Three great ways YOU can eliminate waste and protect your environment!

Waste, and how we choose to handle it, affects our world’s environment—that’s YOUR environment. The environment is everything around you including the air, water, land, plants, and man-made things. And since by now you probably know that you need a healthy environment for your own health and happiness, you can understand why effective waste management is so important to YOU and everyone else. The waste we create has to be carefully controlled to be sure that it does not harm your environment and your health.
What exactly is “waste?”
Here’s a list (not exhaustive, by any means):

Code that doesn’t get used
Log entries that give no information
Comments that echo the code
Tests that you don’t understand
Continuous integration builds that aren’t consistently green
Estimates that you aren’t confident in

How can you help?
You can help by learning about and practicing the three R’s of waste management: Reduce, reuse, and recycle!

Reduce – do less. Don’t do something because that’s what you’ve always done. Do it because it adds value; because it helps you or a colleague or a customer; because it’s worth it.
Reuse – repetition kills. Maybe not today or tomorrow, but eventually. Death by a thousand cuts. Eliminate pointless repetition,  and go DRY (Don’t Repeat Yourself) for January. You’ll enjoy it so much, you’ll never go back.
Recycle – is there nothing to salvage? Really? When you find yourself building that same widget from scratch again, ask yourself “why didn’t I recycle the last version?” Was it too many implicit dependencies? Or was the granularity of your components not fine enough? Try building cohesive, decoupled components. Practice doing it – it gets easier.

By |January 12th, 2015|Agile, Cyber-Dojo, Musings, Practices|0 Comments

Do you practice BDSA?

I get to visit a lot of teams. I talk to my peers about their experiences too. For every agile success story there seem to be a load of less happy outcomes.

There are three common ways that I’ve seen agile ‘adoption’ go wrong, and they are:

Bondage – “You can never change the business. That’s just the way things work here.”
Domination – “That estimate is too big – make it smaller. These tasks are assigned to you.”
Sadism – “Work longer hours. Honour your commitments. Of course you will work on multiple projects”

If these sound familiar, then maybe your organisation uses BDSA – Bondage, domination and sado-agile.

By |December 18th, 2014|Agile, Musings, Practices|0 Comments

Half a glass

Is this glass half empty or half full?

There’s normally more than one way to interpret a situation, but we often forget that the situation itself may be under our control.

I often find my clients have backed themselves into a corner by accepting an overly restrictive understanding of what they’re trying to achieve. They will tell me, for example, that this story is too big BUT that it can’t be split, because then it wouldn’t deliver value. Or, they might be saying that there’s no point writing unit tests BECAUSE the architecture doesn’t support it. Or even, it’s not worth fixing this flickering test, because it ALWAYS works when they run it locally.

I encourage them to rethink the situation. A slice through that story may not deliver the final solution the customer needs, but it will act as a small increment towards the solution. Minor refactorings are usually enough to make a section of the code independently testable. That flickering test is undermining any confidence there might be in the test suite – fix it or delete it.

Don’t try to put a spin on a half-full glass – rethink the glass.

Instead of asking if the glass is half-full or half-empty, use half a glass.

(It turns out that wittier minds than mine have already applied themselves to this proverb – here’s a bumper collection.)

By |December 15th, 2014|Agile, Musings, Practices|0 Comments