Rundeck Blog

Getting to Self-Service: Break Up the Monolithic Thinking About Ops Procedures

 The Takeaways:

  • Ops procedures are not as monolithic as commonly believed
  • Separate each element and move to where it will do the the most to improve the flow of work or labor utilization
  • Implement Operations as a Service capabilities and move your organization towards a Self-Service Operations model 

"But... how?" is an immediate response we often hear when presenting the idea of moving to a Self-Service Operations model that allows the standardization and delegation of operations tasks to people outside of traditional ops roles.  This is usually followed by a statement like "the benefits make sense and the end-state is appealing, but I'm having difficulty seeing how we get from here to there."

We've learned that this uncertainty is usually an artifact of the historical belief that automated operations procedures are a monolithic task. As people separate their thinking about automated operations procedures into distinct elements, the belief dissipates, and the path to Self-Service Operations becomes clearer. These elements are: the ability to define a procedure, the ability to execute that procedure, and the governance around that procedure. Once separated, you can start to see how each element can be distributed to the teams and roles in your organization where it most improves the flow of work or utilization of labor. 


The Elements of an Automated Operations Procedure




If you break down what goes into an automated operations procedure, there are actually three core elements:

Define - Who creates the definition of the automated procedure and how do they do it 
Execute - Who can execute the automated procedure and how are they provided the ability to do it
Govern - Who has governance over the procedure and how do they manage security, management oversight, and compliance

Let's look at the different patterns that emerge as you decouple these elements and start to move towards Self-Service Operations.


Traditional Siloed Operations

Let's start by looking at the traditional siloed operating model where all operations activity is executed by a centralized operations team. All three elements are the sole responsibility of that centralized operations team. If you need an operations task completed, you'll find yourself filling out tickets and waiting in request queues.  



History has shown that this is the model where the organization experiences the most pain. Operations bottlenecks and labor shortages are common. Handoffs from outside of operations (or even within operations) are long and error-prone. Most of the DevOps conversations of the past seven years have warned against this operating model so I won't rehash all of that here. In short, when faced with today's high-velocity lifecycles and dynamic infrastructures, operations organizations struggle under this model.


Rigid Self-Service

This is usually the first step towards Self-Service Operations. Operations will setup a "button" (e.g. GUI, CLI, API) for people to "push" to do a specific repetitive task. We call it "Rigid Self-Service" because everything is still defined and managed by Operations. Others can call the procedure but are very limited in being able to modify it (maybe they are presented with a few options). Enabling Developers to deploy (e.g. CI loop or push button deploy the next version of a .war file) is a common DevOps-inspired move that falls into this rigid self-service model. 



In reality, due to the rigid nature of most of these procedures, only a limited amount of self-service is actually possible. Operations still ends up performing most of the operations activity themselves. This is really "limited, rigid self-service". The rigid self-service pattern is a good first step as it does help improve the flow of work for teams who need  operational tasks performed and it does lighten the load on Operations -- it's just not a game-changing level of improvement for either party.




High-Velocity Handoffs

Most of systems currently running in your environments were created or brought into your company by a team outside of traditional operations. So it stands to reason that the people who know the most about how those component systems are built and how to fix them are outside of Operations. Operations is always in a position of playing catch-up.

The traditional pattern has been for teams outside of Operations to attempt to perform a knowledge handoff to Operations in the form of a documentation dump (readme files, "do-this-then-do-that" word documents on sharepoint servers, ticket comments, etc.) and perhaps some "it worked in my environment" scripts. From that, Operations was supposed to fully digest that handoff and then go and build the procedures and the automation necessary for the management of systems in the higher pre-production (e.g. UAT, STAGE, etc.) and production environments.

In addition to the error-prone nature of human-to-human knowledge transfer, these handoffs are time consuming and resource intensive, often requiring the full attention of your most skilled engineers. As lifecycles speed up and environments becomes more dynamic and complex, these handoffs quickly become a major bottleneck and their error-prone nature is weak point for quality and security. 



A key step in transitioning to Self-Service Operations is pushing the responsibility for defining and creating automated operations procedures to the creators of the systems and software running in your environment. A simple example would be the mandate that developers have to write and maintain automated operations procedures for all things that they create.

Of course, this means that the automated procedures have to be seamlessly reusable by both the creating team and the Operations team.  It must also work seamlessly with automation for other components. One would think that this implies that a single automation tool would have to be used across all teams. However, forcing teams to standardize on one language or automation framework just isn't realistic given the heterogeneous and dynamic nature of enterprises. Teams need to be able to use the automation languages and tools that they want, while allowing for other tools, accessible by all, to orchestrate procedures across those underlying frameworks and languages.

If you get this correct, you will be able to achieve high-velocity handoffs that improve the flow of work, improve quality, and relieve Operations of significant pressure.


Self-Service Operations

When you combine self-service capabilities for both defining and executing automated procedures , you have enabled Operations as a Service. Like any modern on-demand "_aaS" , you are putting as much control as possible into the hands of the requesting party. This lets the requester do for themselves what they need to get done, when they need to do it. This is true Self-Service Operations. 



Obviously, security is a significant requirement. This pattern only works in an enterprise setting if Operations can maintain full security controls, enforce compliance, and have management oversight (and do so in a low-friction manner!). If done correctly, everybody wins. Operations stays out of the way of teams who need operations tasks performed while simultaneously being able to lock things down to a greater degree than is possible today.

As an organization advances with the Operations as a Service model and more self-service is enabled, a common step is to also push some of the governance capabilities to other teams. This is popular with organizations that have multiple lines of business with different risk profiles. While Operations must retain ultimate control, there are some access control and compliance decisions that can be pushed closer to the owners of the risk in those other lines of business. 



The power of the Operations as a Service pattern is that an organization can move the different elements of their operations procedures to where it makes the most sense for their specific business to maximize flow and decrease costs. This ability to shift work and collaborate on the fly is critical as no enterprise is a "one size fits all" and no enterprise stands still. 



If you are interested in building your Operations as a Service capabilities and enabling Self-Service Operations, we'd like to talk to you. Feel free to contact us to setup a discussion.


Damon Edwards

Damon Edwards

Damon Edwards is a co-founder of Rundeck, Inc. and cares a lot about improving IT operations and how enterprises operate.

Have new posts sent to your inbox

Get new blog posts sent to your email