13,000+ manual reviews of production operations activities avoided
Time to Repair reduced by 25 minutes on average
200 self-inflicted operations incidents avoided
Let’s put this transformation story into perspective. First, this effort was driven by the bank’s mainline technology organization that was responsible for the more than 1,000 applications that accounted for virtually all of Standard Chartered’s revenue.
Second, Shaun appreciated the daunting scale and history of Standard Chartered. The bank’s charter was signed by Queen Victoria before the invention of the lightbulb. They have 86,000 employees operating in more than 60 countries (each with its own regulatory requirements ). Finally, this wasn’t a transformation of a special project or an internal startup, but rather an entire institution.
While compliance and security should be the highest priorities of any bank, speed and agility are essential to compete in today’s global financial marketplace. Shaun and his colleagues embarked on a journey to optimize Standard Chartered’s processes for both speed and compliance.
Their transformation featured several significant initiatives including streamlining infrastructure delivery, private and public cloud adoption (stressing that “how is more important than where”), the creation of an SRE capability, and the adoption of a standard, modern DevOps toolchain.
Perhaps most importantly, their leadership developed a standard set of tenets and principles (Figure 1) to be adopted across the technology organization. This gave the organization alignment around expectations and clear direction on improvement efforts.
Rundeck expands from solving local pain to delivering global impact. Like many other Rundeck community members, Rundeck initially entered Standard Chartered through a “grassroots” effort to solve a set of acute problems in a specific group. In this case, it was a middleware and web support team within Standard Chartered that deployed Rundeck to automate tasks like service restarts and failover actions.
A single engineer discovered Rundeck and undertook the not insignificant task of managing the approval process. Once in-house and running, other engineers saw the results and started scripting all of the their manual tasks and created standardized and shareable Rundeck jobs.
When Shaun came in to run Standard Chartered’s retail banking technology group, he noticed the early success of Rundeck and saw an opportunity to spread that success to the rest of the bank. By 2018 Rundeck was used for incident remediation in over 400 applications across the bank.
In addition to replicating the speed and quality benefits, Shaun found another important way to exploit Rundeck across the bank.
Before Rundeck, all production changes required special access tokens and all terminal sessions had to be recorded for immediate review by the manager of the person making changes. Yes, each and every change, no exceptions. This was no minor feat. Thousands of changes had to be reviewed annually (13,000+ by Shaun’s estimate). Imagine the employee time required for this effort.
Rundeck dramatically reduced that burden by providing a safe way to distribute production access. With Rundeck, each team member has restricted access to the jobs that they have been approved to run. Rundeck logs all execution so that auditors can see who ran what, when, where, and what the outcome was. This includes traceability of job definitions to ensure the correct job (and underlying scripts) are run.