DevOps War Stories
“So, we have a problem. Being a roaming engineer I’ve had the distinct pleasure of working with lots of companies here in the states. Concerning DevOps and the execution of lean concepts applied to operations: in almost every circumstance I’ve found myself in I’ve discovered the same striking observation:
You keep using that word. I don’t think it means what you think it means…
Of course, with DevOps being the hot new concept off the presses it’s likely to be expected that some degree of confusion and self definition will exist. After all, in computing, unlike the formal engineering disciplines, we simply make things up as we go and deal with the consequences later. It’s a beautiful yet terrifying dichotomy that enables us to simultaneously become gloriously innovative and yet dismally repetitive.
As I have found myself in many interesting situations I thought it prudent to provide two common misconceptions I have observed in my travels. As a disclaimer, please understand that I sit on the engineering side of the DevOps world, so well designed code gets me way more excited than administrating bare metal server farms or writing iptables rules (which are actually a thing of nightmares), so my observations will come from that point of view.
Common misconception number one: DevOps is (insert tool). This one comes with statements like: our code is being compiled when our engineers check in so we’re doing DevOps right? We’re using Jenkins pushing containers to artifactory and we use Gulp and Mocha and with phantom.js for our front end testing and task orchestration. We just bought a selenium license and we’re looking into SOAPUI for automated API testing–we were thinking about installing a local Sonar instance for code metrics, what do you think?
I think all of that is great, and you have a lot of money to spend and we should talk about a support contract for all that. Never mind the fact that your engineers are dealing with four lower lifecycle environments; all of which are completley different, along with sinking 30 grand or so for three vendors to provide “test beds” for the engineers to develop against, and the time it takes to provision a new developer is a day or so. Clearly automation is too much work and places such a drag on the engineering process that it’s not worth the up front cost. Instead we’ll buy the first silver bullet someone pitches us.
Devops isn’t a tool or a grouping of tools. Often times people become attached to the hype and the marketing produced by sales machines–remember big data? Similar to fine craftsman engineers who understand DevOps concepts in depth solve problems with tools, not create problems to justify the existance (or licensing cost) of a tool. And as an addendum, most of the “ooo shiney’s” I mentioned a moment ago are part of a healthy, functioning engineering pipeline and have absolutely nothing to do with DevOps.
Common misconception number two: DevOps is a person. This one people will argue with me on, but I’m sticking to my story here. I’m not out to get any of you fired if you hold a DevOps title, but I believe the DevOps concept to be a horizontal concern understood and executed by many people, not a vertical role executed by one. Puppet labs has an interesting article defining the roles of a DevOps engineer:
“The DevOps engineer encapsulates depth of knowledge and years of hands-on experience, You’re battle tested. This person blends the skills of the business analyst with the technical chops to build the solution – plus they know the business well, and can look at how any issue affects the entire company.”
If you’ve ever worked in a truely lean engineering shop you know that usually the engineers know and understand the business. If only there were a ceremony in Agile that when a team has finished up a sprint they can demo what they built, talk about how they built it, and then describe how it benefits the business…
It’s very clear that lean concepts are pervasive and deeply influence DevOps ideaology on both sides. Just read Continuous Delivery and then The Phoenix project cover to cover in sequence. Jez and Gene are clearly describing lean concepts as applied to both sides of the equation, and these ideologies go hand in hand.
To be clear, however this I do not think this means that operations teams disappear or are folded into engineering groups. It simply means that just like business and engineering have to work together with fast feedback loops to ideate and create the glorious product, operations and engineering must work together with fast feedback loops in order to “pave the glorious golden road to production.” Business gets software, engineers get to go home, and everyone’s happy. In a DevOps world, everyone’s a hero and no one hates to go to prod. It’s a cultural shift and a change of mindset that couldn’t come any sooner.”
– Jeremy Duvall, ImagineX Practice Director, Lean Engineering & DevOp as presented during DevOps Days ATL Ignite Talk 2016