A couple of weeks back I was planning the rollout of a platform engineering initiative with an enterprise team. As I normally do in these cases, I insisted they get a good sense of what their current status quo is in terms of developer experience, degree of automation, standardization, and security best practices.
Among other interesting things, it emerged that just to get a simple resource like a new database, developers had to wait on average 3 weeks and 17 manual steps with several handoffs between engineers, the I&O team, the security team (review, policy checks) and engineering managers were required almost every time. Crazy. Except it’s a very common story from the frontlines of DevOps setups at scale that I am sure many of you are familiar with.
And the main reason for the huge lack of automation and standardization at the root of these terribly inefficient workflows is the fact that most engineering orgs today still build software like small artisan shops.
Software engineering is still in the craftsmanship phase
What do we mean by craftsmanship phase? Well, master craftsmen and craftswomen are experts at everything required to produce the finished product they specialize in. For example, a master carpenter will be able to go from a raw tree to a finished chair or table on his or her own. In addition, they’ll normally have a deep knowledge of how the tree grows, and what conditions favor different properties of the wood, and they’ll also be able to use and maintain a vast range of hand and power tools.
Now compare this to our senior backend engineer, who’s usually expected to understand everything from how registers on a CPU work, all the way to how to configure and maintain a VPC. They’re expected to be able to tackle challenges such as optimizing low-level algorithms, building containers, defining CI/CD pipelines, writing Terraform, and configuring and working with monitoring and observability tools. This is a large range of things and is pretty much a description of the kind of space that a master craftsman would be working in.
That’s really what makes the development of software today stand in stark contrast with most industries (which software ironically enables). While virtually every other sector since the Industrial Revolution has seen an accelerating trend toward specialization and commodification, software engineering has somewhat lagged behind and is still stuck in this craftsman paradigm. Other industries have progressively added layers and layers of platforms stacked on top of each other, driving incredible levels of standardization.
Think about simple examples like the screws holding together the table on top of which your computer might be sitting right now (unless it’s on your lap, causing potential neck and back pain in a few minutes). Screws are an example of both a commodity and a standard. Technically it’s a secondary commodity, as the primary is the metal they are made of. With metal, you can mint screws and if I have an M5 screw that’s 30 millimeters long, I can go and source that M5 screw from multiple people in multiple places, practically all over the world. Those will all be broadly interchangeable, and unless I’m doing something very sophisticated (e.g. I need very high tolerances or I am using the screws in very unique environments), most screws will probably be fine for the job I’m trying to do.
We can also go up to tertiary commodities or finished products and consider those as commodities. A great example here is hard drives. Hard drives are very standardized: they fit into your machine in the same way, and give or take some specs, they will pretty much do the job, irrelevant of which brand you end up buying. Google famously built its first data centers using commodity hardware. They weren’t too worried about the exact makes or the exact specs of the hard drives. As long as they did 'hard-drivey' type things, that was good enough for them.
You get the point. Commodification and standardization are the keys to scalability and efficiencies at scale. And the way most organizations build and ship software today is still very far from the way we make and use screws.
From craftsmanship to industrialized assembly lines
If you really think what platform engineering is all about, it’s this. It’s enabling engineering organizations, particularly in the enterprise, to grow out of the craftsmanship phase and into becoming industrialized assembly lines. This is what will drive 10x efficiency and dev productivity gains across the entire organization.
Let’s look at some concrete examples of commodification in the software engineering world that can be enabled by platform engineering initiatives.
Developers normally have to run their apps somewhere, and almost every single cloud provider has, for example, their own Kubernetes (K8s) service. Each K8s service has its own special features and quirks, but at the end of the day, they’re all pretty much just Kubernetes. As a developer, I’m probably not too interested in all of the details of actually how these K8s services function and what their special features are. To me, a K8s API is a K8s API, and that’s all I care about, not which particular flavor of K8s I am deploying onto. In most cases, I don’t even care if it’s Kubernetes at all!
Let’s take another example and say that I am a developer trying to use a Postgres database. I could use a service like Aiven, which is a database service provider. I could use the database service from my cloud provider, e.g. Google Cloud SQL if I am running on GCP. Or I could host it in my own VM or in Kubernetes or something like that. All of these things will give me a Postgres. They all have different tradeoffs, different costs, different values, and features that they support. But, at the end of the day, as far as my particular application is concerned, it is just Postgres, right? My Postgres client is not going to care if I’m talking to Aiven, Google, or an in-cluster database.
So once again, we have kind of commodified this into just a Postgres service, and this is actually a larger example of something that has been becoming more important recently, which is the idea of commodification of the cloud. The basic idea about this is that as we commodify these things, we make things easier for us to build. So as a software engineer, I don’t need to worry too much about the specifics of each cloud, I can just take commodity components and build my software solution. That makes me, as a software developer or a software architect, incredibly more productive.
Finally, platforms are an excellent way to deliver these commodities. And that’s the crucial hint of where we need to be in the platform engineering space. We have seen how commodification in other industries can tackle the same broad challenges that we
face in software development: cognitive load, dependence on individuals, and low levels of standardization. Delivering commodities should be the goal of platform engineering, and it should just be the case that platforms are how we do this, rather than the goal of what we’re trying to achieve.
This framing, by the way, also gives us an opportunity to understand what makes platform engineering different from DevOps. DevOps is about integrating development and operations teams through automation and shared ownership. It is focused on elevating and optimizing existing development practices. Now, compare that to platform engineering, where we are seeking to commodify a lot of these areas of shared ownership and then build on top of what DevOps already provides. This hopefully gives you yet another way of thinking about the differences between DevOps and platform engineering (and might make those conversations a bit easier).
Conclusion
Commodification is really the whole point of the end product of your platform engineering initiative, your Internal Developer Platform (IDP). It’s to eliminate all the manual steps and hand-offs (ticket ops, manual reviews) I mentioned at the beginning. But also to relieve developers from having to be craftspeople, i.e. experts at everything across the entire software delivery life cycle, which leads to high cognitive load, long onboarding times, and a huge amount of friction in the system.
Our goal as an industry should be to enable developer self-service and standardize the way how infrastructure and applications are provisioned and managed. Just like a commodity. Your Internal Developer Platform is the vending machine delivering standardized components to all key stakeholders in an automated way.
