For organizations looking to improve their delivery process, the continuous delivery (CD) approach promises significant benefits. However, for CD to have any meaningful impact, organizations need to also implement internal platforms, training, and metrics to ensure those promised benefits are being fulfilled.
As an early pioneer of CD during his time at Walmart, Bryan Finster knows firsthand the trials and triumphs this software engineering approach can bring to organizations - as long as they’re willing to provide holistic support and learn from their mistakes.
Introducing CD at Walmart
Bryan began piloting CD in 2014 in Walmart’s supply chain, and would later move to the platform team to bring CD to every team in the company. While many organizations fall into the trap of thinking that their tooling facilitates easy CD flows, Bryan’s time on the platform team demonstrated that it takes more than tooling to get code into production daily.
One of the first things he learned was that working to improve delivery exposes all of the problems that prevent organizations from delivering. Bryan explains:
“What we discovered early on when we were piloting CD was that the inability to get working, tested, non-broken code deployed to the trunk every single day exposed so many problems.”
In particular, continuous integration (CI) outlined a lot of issues with testing, silos, and vague requirements within the engineering process. However, many organizations focus too much on utilizing agile coaches who often don’t understand what CI is, or how agile processes are designed to facilitate CI/CD.
If organizations focus on how well they run standups or other agile ceremonies, then they’re not diving into their processes deep enough to find the larger issues. As Bryan puts it, “focusing on how you get things done shows you all the things that stop you from getting things done.”
This led Bryan to focus on two core areas to define CD at Walmart - minimum CD and metrics.
By using the definition of minimum CD from MinimumCD.org - which Bryan contributes to - organizations can understand what processes they need to put into place to start seeing the benefits of CD. This will also help them to focus on delivery instead of processes.
Metrics also help to measure the benefits of CD and drive further improvement. However, organizations should focus on metrics that are indicators of behavior. Bryan says:
“For us, a leading indicator was code integration frequency. So, if you had a team of five, how often was the team on average integrating code with the trunk? Even if that was once a month, it gave us a good understanding of that team.”
It’s important to look at metrics, and particularly DORA metrics, as health indicators rather than goals. That’s because in many cases, metrics can be contextual to the environment engineers are working in or deploying to. Similarly, metrics need to drive quality, not just speed, and this needs to be understood on the executive level as well as in engineering.
Building the DevOps Dojo
With that being said, defining CD, minimum requirements, and related metrics was only the first part of the strategy. To introduce CD in a meaningful way, it needs to be supported holistically. For Bryan, this meant founding the DevOps Dojo at Walmart.
The DevOps Dojo ran 6-week challenges where teams would run 2 ½ day sprints with minimal ceremonies. Here’s how Bryan explains the process:
“We’d go through the process of defining the application, understanding its goal, defining stakeholders, and defining who was responsible on the business side in case we needed to step in to clear blockages. We’d have some basic housekeeping things so we could all work cleanly together. Then we’d have short sprint planning and a demo and retro every 2 ½ days.”
By stripping sprints back to 2 ½ days, teams would only have meetings if they added value. Demos existed to either demonstrate what the team achieved or what they were aiming to achieve, but they missed because they were still in the learning process. It was expected that for the first half of the course, teams would fail regularly because they were uncovering problems in the process.
This disciplined approach built sustainable change by solving problems with the team rather than prescribing a solution. By using the tools and resources available to a team, the Dojo team could learn new solutions and introduce new ones - such as Walmart’s internal platform - to help them to improve.
In addition to this, the DevOps Dojo would take what they’d learned from these 6-week challenges and feed that information into the platform team. Bryan says:
“One of our jobs at the Dojo was to inform the platform team of challenges that people were having so we could find ways to make the platform better. We were the eyes and ears of the platform. We were also generally the first people who would teach people how to use the platform.”
Finally, the DevOps Dojo would also serve as developer advocates. If teams ran into blockages, the Dojo would speak to directors or executives about those issues, particularly if those executives were the ones causing problems for their teams. By having an open line of communication and advocacy, the Dojo could continuously work to improve the developer experience and ensure CD was supported at higher levels of the business.
Executive buy-in and cultural support
Bryan focused on the concept of Behavior-Driven Development, or BDD, to deliver continuous development at Walmart. As part of this, both the engineering communities and executives within the organization needed to be encouraged to participate.
It was vital for the success of CD and the DevOps Dojo for the CTO to be actively involved, particularly as he set the goal of every team having the ability to deploy to production daily. Not only did the CTO participate in the quarterly all-hands meetings held by Bryan’s team, but the CTO would help the team to open communication with other executives. Bryan also adds:
“We had a community meeting practice around CD called Continuous Chai, and if you presented one you’d get a t-shirt. The CTO presented one, so my wife and I did a presentation to give him his t-shirt. Even though he paid for everything, the rules were he couldn’t have a t-shirt unless he presented at one of these community events. It really showed the active involvement of the CTO. He was supporting behind the scenes and funding the effort, and actively encouraging people to participate too.”
Supporting communities and opening lines of communication was a significant part of the process. They marketed the DevOps Dojo internally and, even if teams were too big to see any benefit from the immersive training process, Bryan and his team would still offer tailored advice.
On top of that, Bryan’s team would run internal DevOps days with external speakers talking about their experiences. They would also host lunch-and-learns to garner interest for the CD initiative and DevOps Dojo.
Executive buy-in and cultural support were vital to shifting teams from outcome thinking to process thinking. Bryan explains:
“When it comes to output vs. outcome, that comes down to incentives from executives. If they’re focused on output, everyone focuses on output. If they want better business outcomes, that’s what everyone focuses on too. So you educate them on how CD can deliver better outcomes because you’re delivering smaller things more quickly. You can also help them to see that if the requirements are wrong or they need to change before you deliver, CD allows you to fail small.”
Most importantly, community support needs to be widespread for it to be effective. Teams like Bryan’s can’t rely on grassroots support only, because those people don’t have the power to advocate for change. However, they can’t rely on executive support only, because executives change frequently. It’s vital to build bridges and maintain constant communication across the whole business to drive meaningful change.
Enhancing CD with an internal platform
Platforms serve to give engineers the flexibility to implement CD while keeping them safe and removing friction from their daily work. This gives them the ability to use CD to solve business problems. However, for this to work effectively, the platform needs to be treated as any other product. Bryan says:
“I consider the internal platform to be the highest goal of any good developer, but when you work on a platform, you need to consider what you’d want as a customer if you had a platform inflicted upon you. But how do you know if the platform’s performing well? Well, no one talks to you. That’s the goal because people only talk to you as a platform engineer if they’re unhappy.”
Thanks to Bryan Finster for this eye-opening AMA session. There’s a lot more content we didn’t have time to cover here, so make sure you watch the session recording and check Bryan’s blog on Medium for even more inspiration on scaling continuous development in your organization.
.jpg)
.jpg)
