A common misconception about successful Internal Developer Platforms (IDP) is that they’re just a well-connected suite of tools and abstractions behind a shiny front end. This couldn’t be further from the truth.
Building a successful IDP goes far beyond the technical aspect of the platform. You need to consider how your engineers want to work, how much cognitive load they can handle, and what cultural changes you need to introduce in order to ensure a successful rollout.
To discuss how to build a successful IDP, Alan Barr, Platform Product Owner at Veterans United Home Loans, joined us for a meetup in January 2022. Keep on reading to get his learnings from building an IDP from scratch.
How Internal Developer Platforms provide value
Veterans United Home Loans had a significant problem with their legacy environment. Because they were using virtual machines to support their development pipeline, engineers were spending most of their time waiting for operations to provide infrastructure.
“If we wanted to go from a new idea to experimenting with it, we wasted a ton of time between third parties and waiting for tickets to be resolved” Alan explains.
However, with an IDP, organizations can drastically reduce the amount of steps engineers need to take throughout the development and deployment pipeline. The internal platform allowed engineers to have a greater degree of self-service when provisioning their own infrastructure. The amount of time it took from concept to implementation and monitoring was more than halved.
“As a developer, you’re not waiting so much and you can be autonomous” adds Alan. “You can help the business as much as possible because you’re not waiting to add value”.
Dev.lab, Alan’s IDP, allowed for a streamlined, consistent experience with a vastly reduced need to wait for third parties to support engineers. By allowing engineers to spin up new environments and provision infrastructure as needed, they can focus entirely on the task at hand.
Similarly, the abstraction of tasks that would normally distract engineers and require extra cognitive load, like navigating infrastructure changes and security, allows engineers to focus all of their energy on the product.
Early missteps
Despite all of the early planning, there were a lot of issues in the initial stages of Dev.lab.
Before Dev.lab, the organization very rarely built products from scratch, and it was a lot more common for the engineers to be adding features to existing products. The team wasn’t comfortable building a new product from scratch.
At the start, the vision the team had for the platform wasn’t entirely clear. “We’d ask ourselves, are we building a container as a service platform? Are we building a platform as a service? What are we building?” Alan explains. So, as the project progressed, the initial vision of what the team wanted from the platform drifted over time.
As the vision drifted, so did the plan for future iterations of the platform. The team started to draft features years in advance. “But, planning features more than six months into the future is sketchy,” explains Alan. Because the team fell back on old habits of designing everything upfront, it would have left little initial room for feedback and improvement.
Too many cultural changes were happening at the same time, too, and the implementation of the IDP revealed further areas where the organization’s culture needed to change. The Dev.lab project highlighted weaknesses in the team’s interactions and communications. It seemed that the coordination they initially had with the organization wasn’t going to sustain itself and they needed to develop a new plan.
Alan says “You’re going to go through a lot of these changes while you’re trying to use a tool, and you might not be aware you’re going through this”. With that in mind, it’s important to pay attention to how the organization is adopting this change throughout the process.
The Dev.lab timeline
Building an IDP is an extensive process, particularly for organizations that opt to build one from scratch on their own. The initial projected timeline of the project spanned 18 - 24 months.
“It’s a very common industry timeline for people to start changing their work processes and get used to something new. Despite how fast we could have built it, this timeline is reasonable for a very large organization dealing with a lot of people,” explains Alan. “But maybe someone will do it in six months, who knows!”
It’s important to understand that building an IDP on your own isn’t a quick project. Not only will the platform team have to work with the technical aspects of building it, but they’ll also have to deal with a lot of different teams and stakeholders across the wider organization to ensure they have the support and resources they need for the project.
“At the beginning of 2020, I had a boss that rallied the executives to pare down the 100 different projects we could have done into two or three. The IDP was one of the three projects”, explains Alan.
While it can be difficult to explain technical projects to non-technical people, it’s vital to get executive support because the project needs a lot of resources for it to be successful. Building an IDP from scratch can easily cost millions of dollars.
“Then, once we got that alignment, we pitched the research. We then completed the research, and showed them the value with a rough prototype. From there, we started to develop the proof of concept and the minimum viable product (MVP).”
In hindsight, however, Alan wouldn’t use the MVP methodology with an IDP project. “Platforms are not minimal. They’re thick and need to do a lot of stuff because you’re looking to replicate what already works. It also takes a lot of work to get there”.
Alan goes on to explain that two months after the MVP was delivered, the team released the alpha version of the platform and focused on getting engineers on board. “There were moments of time where it wasn’t very stable, so people were hesitant to start using it because they were used to a rock-solid experience”, continues Alan. But, a few months later, they stabilized the platform and made it generally available.
It was also important for the team to name their IDP so it felt more concrete: Dev.lab. Specifically, they wanted a short, concise name that was easy to use in conversations. Marketing the platform in a similar way as any other product produced by the company helped engineers to talk about it both between themselves and with the platform team.
Dev.lab’s overall goal was to be a hospitality-focused platform that made it as easy as possible for developers, and in particular junior developers, to fall into the pit of success by following a strict path. Similarly, the value it provided would come through a streamlined, consistent experience because engineers wouldn’t have to wait on third-party support.
The choice of tools
By involving all relevant stakeholders as early as possible, platform teams can get a good understanding of what tools they need to connect to their IDP. However, this can quickly become an overwhelming task, as every company’s cloud native toolchain is constantly and rapidly expanding.
To address this, Alan worked with a “trail map” provided by the CNCF, which he describes as a recipe for success. Along this route, platform teams note down each type of tool they need - such as for CI/CD, service meshes, security, and so on - and decide which ones they want to integrate into their IDP for each part of the toolchain.
“Once you settle on a specific technology, you can move on with your time as long as you’re satisfied it’s accomplishing the goal. So you can focus on building the platform”, Alan explains. In the end, the team decided on a suite of 15 tools.
Reducing cognitive load
One of the main ways that IDPs bring value is by reducing cognitive load for engineering teams, allowing them to focus on the task at hand.
Standardization is a crucial element in delivering a seamless developer experience. “Software engineers have so many choices for tools, and you might be swapping between different projects that need different tools,” explains Alan. “It’s really hard to keep all of that in your memory. The more we can keep systems consistent, though, the more we can move people around, and the more we can educate people.”
By standardizing things like security practices and commit formatting, the platform team ensured they could reduce cognitive load as much as possible.
What we learned
It’s vital to work with an experienced trainer who can help engineers understand the platform and learn how to best use it as part of their workflow. Specifically, enforcing standards like the 12 Factor methodology to ensure developers follow best practices is crucial to helping engineers understand how the IDP is different from previous systems.
“12factor.net is a wonderful resource to communicate to development teams why cloud-native is different. Without it, developers at Veterans United Home Loans were applying old mental models to the new system. With 12 Factor and a great trainer, we developed a curriculum for our engineers to communicate the why behind the platform”, Alan explains.
Platform teams also need to realize that having to balance stakeholder expectations, they can’t make everyone happy. It’s important to communicate this with stakeholders to help them understand what their role is during the transition to using the IDP and beyond. “It’s a constant maintenance effort”, Alan says.
Understanding that an IDP can’t support every single use case is the key to success, so that platform teams don’t stretch themselves too thin between feature requests. Every organization will have unique situations that can’t be covered by an IDP, so there’s no harm in pointing teams towards the cloud, virtual machines, or other tools to cover those use cases.
Wrapping up: The IDP that doesn’t suck
As Alan made clear, building an IDP from scratch on your own can be really hard. There are many prerequisites for this to succeed. You have to come up with a strong product vision that is shared by all stakeholders. To manage a roadmap for such a robust and key product, you need the right people, who are ideally experienced in building new products from scratch. You also need to make the right choices out of the overwhelming amount of tools you can use to build an IDP and glue them so your IDP can provide a great developer experience to your dev team. To enable developer self-service, you have to provide the right abstraction layers, for which standardization across the whole engineering organization is key. This needs to be trained and will not happen on its own, so do not underestimate the cultural changes it might take to build and adopt an IDP. Last but not least, you need to ensure executive support for this long journey, which can take 18 to 24 months and a significant budget, if you build an IDP on your own from scratch.
Thanks again to Alan Barr for hosting this informative and eye-opening meetup. If you haven’t already, don’t forget to check out the recording or bookmark it for later.
.webp)
.avif)
