.avif)
For the last decade or so my focus has been on improving developer productivity and enabling developers to use continuous integration and continuous delivery (CI/CD) pipelines, providing them with tools to improve delivery quality. In this article, I'll talk about how to invest in developer experience (DevEx) to reduce barriers to continuous delivery.
DevEx is not DevOps
DevEx enables DevOps.
DevOps is a set of practices and principles. It provides a set of rules that allow you to push value forward to your customer as soon as possible, in the safest way possible. It's not the name of a team or an organization that deploys for somebody else. It’s about using CI/CD for its true purpose: continuous integration and continuous deployment into production. This means developer teams not only own the build and test, but also the deployment and monitoring in production. So you develop, test, and release, which means the value is now flowing to the customer. Then you repeat the entire process. In a way, DevOps is about flow in the system and the value stream, how you enable that for a given product, and how the development team enables themselves to do that.
So what is DevEx? It’s all about allowing the developer to feel joy as they contribute their code to this process. It's about removing friction by giving them tools to check in code quickly, build quickly, and have rapid pipelines that get to production. All while providing them with feedback they can use to better help their customer. DevEx should include the use of automation tools that are simple and easy to use for any developer, and that provide the highest levels of self-service possible. This removes the need to file tickets or wait for someone else to help them. It allows developers to make changes with confidence and if they run into issues, to debug with greater ease.
The evolution of software delivery
So before I jump into what DevEx looks like today, let's talk a little bit about the historical evolution of software delivery from the 2000s to now.
Delivery in the 2000s was a longer release cadence. When I started working for my first company the release cadence was about six to 12 months. Sometimes we would ship software on CDs such as tar balls for which there would be a two-month QA and certification cycle. These approaches were over-the-wall, siloed, and very waterfall with few agile adoptions. Towards the end of the decade, many companies started adopting more agile methodologies and by the time AWS launched in 2006 people began realizing, "Hey, we can actually use cloud." But at this time, there was barely any large-scale cloud adoption.
So in the 2000s, DevEx was also limited. There would be a team called Core Services that would provide common libraries (such as Eclipse and Visual Studio) for any teams that needed to use them. Then there were builds (such as Maven) and test systems that were written and used to run as part of the continuous integration service. Hudson, which is now known as Jenkins, was a continuous integration server. And this was DevEx. Delivery was nowhere near as fast as we needed it to be for the 2020s.
Today, delivery is demanded as often as possible and at high quality. That means daily and hourly release clearances. The expectation is for the development team to be doing full-cycle development which means they write the code, contribute the tests, and ensure validity in terms of the acceptance criteria provided by the product managers. They release it, and then monitor the traffic for any issues in partnership with SREs. Any learnings will be included, and the code cycle continues. It's all about optimizing the value stream for the product, with heavy emphasis on cloud-native development.
Investments in DevEx for improved delivery
So what kind of investments in DevEx would enable us to improve delivery? I'm going to focus on three different aspects of the dev lifecycle: onboarding, developers’ day-to-day, and refactoring.
Automated onboarding
Currently, most companies have a process that goes something like this: developers are set up on a computer, trained on the code base, and given access to the right kind of cloud solutions. They then write code, check out the code, write code, compile it, and eventually check in something that’s a contribution to the company's code base. This process can sometimes take longer than three months. In some companies, it can take up to six months depending on training and access. Many tickets go back and forth in the process. So what investments can solve this?
An automated workstation setup along with sensible defaults and rapid prototyping tools baked into the laptop can make things easier. Imagine a developer could open their laptop on day one, and find all the software they need pre-installed with instructions and streamlined self-training modules - which are considered an “onboarding buddy”. Investing in an onboarding buddy would mean that at the end of their first week, developers can release small changes to production. These don’t don't have to be huge, sensitive changes. It could be something as small as a bug fix or something that does not affect consumer flow. The key point is that this investment means we could potentially reduce this timescale down to one week.
Day-to-day developer productivity
So now the developer is onboarded, they are contributing day-to-day, building and refreshing their setup. They want to make sure pipelines are running optimally, so a good DevEx investment would be to ensure high CI/CD tooling availability. If at any point the CI/CD pipeline isn't working or is not well-resourced, you cannot push quickly to production, resulting in customer delivery delays. Also, developers may be able to spot and resolve issues as quickly, so therefore unable to code to test to deploy within the ideal 60 minutes.
Performance testing and tuning within the pipeline, and observability across every single stage of the pipeline from test to stage to production and beyond, are all the investments we need. We plan, along with the product owners, and the team then codes it. There's verification, profiling (which is performance testing), maybe resiliency testing, chaos engineering, deployment, to like blue-green, red-black, or whatever the environments are. Production monitoring then finds issues that are coded back into the fixes, coded back in, and the cycle continues.
Refactoring and modernization
Let's next focus on refactoring. Refactoring is crucial when there is either a large piece of the code base that is sensitive and needs to be broken into smaller pieces; needs to be rewritten for different factors; or needs to be updated to reflect the newest and best technologies. This can be a very hard job for any developer. They will depend on the setup that's provided to make sure they can trust their changes, which means DevEx investments have to focus on quality gating and guardrails. Developers need to have top-notch debugging tools.
Great security and compliance checks are essential because we don’t want to introduce new vulnerabilities while we’re trying to fix things. You need to have solid code coverage thresholds so you can stop if there’s any issue with unit testing. And you need to have code visibility to understand the complexity, map it out visually so you can learn more, and then contribute back. These are investments in pre-factory.
Return on investment (ROI)
So if we made investments in these three areas, would there be a return? If so, what would the return on investment be? To understand, let's take a theoretical scenario of a company with around 50 developers.
We discussed how we could cut onboarding from three months to around a week. That's about ten weeks per developer upon voting savings. A daily savings because of shortened pipelines, and reduced time to debug. So let's imagine your pipeline goes down 50% from 30 minutes to 15 minutes. Your debugging time goes from half an hour to five minutes. Your changes no longer require multiple JIRAs and approvals. Let's say it saves an hour per day for about 200 productive days in a year. That's about 200 hours of savings annually.
If we assume we can onboard a team of 50 developers within a week compared to three months, we can save $2MN as a one-time saving. If 50 developers are saving an hour every single day debugging and fixing things, that means we’re saving a million dollars annually. And if we assume this, it's an assumption that the developer rate is about $100 per hour, so about $4K a week. These numbers are not marked so they could vary for different scenarios, as could the assumptions.
Key takeaways
The big takeaways from this discussion are that DevOps by itself cannot be improved unless good DevEx investments are made. Investing in developer joy is essential which means providing great tooling and top-notch automation, and removing any kind of friction within the developer process. Enablement is what ensures the effectiveness of engineering. If developers are not enabled, they’ll find another way that will not be repeatable or benefit others.
Finally, we talked about three stages of the developer lifecycle, specifically onboarding, day-to-day, and then refactoring. But in reality, all stages of that lifecycle need focus from onboarding to off-boarding, which requires us to understand what it means and what the return on investment is.
If you have any other questions, please reach out to me via social media, or via the platform engineering Slack.
