What is Infrastructure Platform Engineering?

In the rapidly evolving landscape of modern software development, platform engineering has emerged as a pivotal discipline for organizations seeking to optimize their delivery processes. However, within the broader concept of platform engineering, a critical sub-discipline often requires a more focused approach: Infrastructure Platform Engineering (IPE). Gartner predicts, that by 2027, more than 75% of Fortune 1000 companies will have formal infrastructure platform organizations - up from less than 20% in 2023.

IPE is not merely about streamlining developer workflows; it's about fundamentally transforming how infrastructure is managed, provisioned, and consumed within an organization. This transformation requires a shift in mindset, treating infrastructure as a product rather than an operational overhead.

The imperative for Infrastructure Platform Engineering (IPE)

The conventional approach to infrastructure management frequently results in a disconnect between development teams and the operations teams responsible for maintaining the underlying infrastructure. 

On one end of the spectrum, in many organizations, you still find a ticket-ops approach, where developers are expected to create tickets if they need new infrastructure or resources  (for example a new database for their application) and then wait ages until the ops team delivers. This is not only a bad developer experience, but it also results in low developer productivity due to long waiting times.

On the other hand, developers, with increased access to cloud consoles, developer portals on top of non-standardized setups, and infrastructure-as-code tools, may create resources in an ad-hoc, inconsistent manner. This can lead to what Kaspar von Grünberg termed "Infrastructure Tetris" in his talk at PlatformCon 2024, where I&O teams struggle to manage a patchwork of resources that are not standardized or well-governed.

This situation often results in:

• Overwhelmed Infrastructure and Operations (I&O) teams: Teams are inundated with manual tasks, ticket requests, and the burden of managing a sprawling, inconsistent infrastructure landscape.
• Lack of standardization: The absence of consistent resource creation practices makes it extremely challenging to manage, maintain, and secure the overall environment.
• Increased security risks: Inconsistent and unmanaged resources become potential vulnerabilities that can be exploited, leading to security breaches and compliance issues.
• Uncontrolled cost escalation: Inconsistent practices often result in uncontrolled cloud spending, with resources being provisioned without proper governance or cost optimization.

This scenario is often the result of a misinterpretation of the "shift left" philosophy. Rather than truly empowering developers, it can devolve into a chaotic, free-for-all where developers are given more responsibility without clear guidance or constraints. Instead, Infrastructure Platform Engineering should give developers more control within well-defined and standardized boundaries, supported by self-service tools and automation.

Defining Infrastructure Platform Engineering

Infrastructure Platform Engineering (IPE) is the discipline of building internal infrastructure products that present infrastructure to users or other parts of the platform like portals in an easily consumable way as a service. This can be related to compute, networking, databases at the lowest level, but also message brokers or anything else that is infrastructure-related and workloads depend on.

For being able to provide infrastructure as a service, IPE also focuses on infrastructure standardization and offering an integration into a platform backend and its abstraction layers. Interfaces on top of these enable non-expert users - primarily application developers - to provision, deploy, and manage infrastructure resources themselves. Simultaneously, it ensures that I&O teams can retain governance, security, and compliance. 

IPE can be defined by the following core principles:

• Productizing infrastructure: Treating infrastructure as a product that needs to be carefully designed, developed, maintained, and continuously improved based on user feedback. 
• Standardization: Ensuring consistent provisioning, management, and configuration across different environments to maintain order and reduce the risk of configuration drift. Standardized infrastructure enables better security and compliance posture.
• Automation: Automating repetitive and error-prone infrastructure tasks is central to IPE, so teams can move faster without breaking things. This includes automating infrastructure provisioning, configuration, deployment, and scaling.
• Self-service: IPE provides the foundation for developer self-service tools, eliminating the need for manual intervention from I&O teams. This reduces wait times and allows developers to be more self-sufficient, thus improving their overall experience.

Following the principle of Pareto Efficiency, IPE improves the situation for developers and the overall engineering organization. By building a unified abstraction layer, it enables I&O teams that are managing data centers to work at eye level with hyperscalers that already provide self-service offerings via consoles.

Building a vending machine

The vision here is to create a “vending machine” of infrastructure “products”. It represents a self-service approach where platform and infrastructure engineers make these "products" available for developers and potentially other users to consume by following golden paths. Platform and infrastructure engineers can add their products, which could be anything from databases to environments, into the platform - the vending machine - for others to select. The users are then able to choose what they need, complete their "transaction," and receive a consistently delivered resource.

This model promotes a clear separation of concerns between developers, platform engineers, and infrastructure engineers. Infrastructure engineers are able to offer their resources in a standardized, productized way, with platform engineers maintaining the products that are available and how they are made accessible along self-service golden paths to developers. 

When shaping the vending machine, it’s essential to start with a solid platform backend, which provides clear business logic and allows for flexibility in the front-end interfaces, rather than the other way around. The entire system (including infrastructure, the platform itself, and all application configurations) should be managed through Infrastructure as Code (IaC) best practices. Finally, cross-cutting concerns, such as those related to FinOps, security, and compliance must be integrated across the platform to ensure these essential aspects are included from the start.

Conclusion

IPE is essential for bridging the gap between infrastructure and development, and organizations that adopt an infrastructure-as-a-product approach can achieve significant improvements. These benefits include faster software delivery due to streamlined processes, improved efficiency through automation and a reduction in manual errors, reduced costs by optimizing resource usage and management, enhanced security via standardized practices and policies, and empowered developers who are given the tools and autonomy to manage their own infrastructure needs.

In summary, Infrastructure Platform Engineering is essential for any organization that wants to optimize its software development processes. By productizing infrastructure, automating tasks, and creating self-service options, engineering organizations can achieve greater efficiency, security, and developer satisfaction, which are all key elements of modern software development.

If you want to learn more about best practices around infrastructure platform engineering for your own organization, take a look at the Platform Engineering Fundamentals course. Or upskill your whole team in one go with Trainings.