Balancing simplicity and flexibility: the platform engineer’s dilemma

Platform Engineers face a challenging dual mandate: they must provide a streamlined, simple-to-manage solution that minimizes bugs and technical debt while also offering a flexible environment that caters to the diverse needs of developers. This balancing act can be daunting, but technologies like Kubernetes offer a way forward, helping Platform Engineering teams navigate this tricky situation.

The platform engineer’s dilemma

Much like application developers, Platform Engineers are responsible for creating resilient, easy-to-manage code that is free from technical debt. Just as developers strive to write reusable, standardized code, Platform Engineers must build infrastructure that can be consistently and efficiently deployed across various teams and applications.

Using repeatable scripts, deployment manifests, and modules helps manage different deployment scenarios. However, new demands from development teams for additional functionality often force Platform Engineers to make difficult decisions:

• Add new capabilities to the Internal Developer Platform (IDP) to accommodate development teams, risking increased technical debt.
• Deny requests for additional capabilities to keep the platform manageable, potentially stifling innovation and experimentation.

This balancing act is why the concept of "Golden Paths" has emerged. To a Platform Engineer, a Golden Path represents a streamlined way to deploy code with all necessary security, compliance, availability, and data protection requirements. It's not just about deploying code but ensuring that the deployment environment meets business standards without overburdening developers with additional cognitive load.

Even with a Golden Path for specific applications on a particular cloud environment, requests to use a different cloud can introduce significant technical debt. Automation routines are often tailored to specific APIs, and changing the cloud provider might require a complete rewrite of the IDP's code, along with new scrutiny from security or compliance teams to ensure apps are secured.

The foundation for simplicity and flexibility

How can Platform Engineers offer more benefits to their customers without increasing the complexity of their solution, technical debt in their IDP, or workload for their team? The key lies in using fewer environments more effectively. Kubernetes provides a compelling solution.

Kubernetes, now over a decade old, offers an open-source solution available for on-prem or cloud environments, and the major public cloud providers offer managed solutions as well. Kubernetes provides consistent functionality regardless of the environment and a singular API to work with. This consistency allows Platform Engineers to code against Kubernetes as a unified application platform, enabling applications to move between environments with minimal code changes.

Kubernetes offers essential capabilities from the inception of a cluster, such as service discovery, high availability, and scaling. These built-in features mean Platform Engineers don't need to write complex deployment code for each application iteration on different cloud APIs. By using a standard API, fewer code changes are required, reducing the maintenance burden of Golden Paths for development teams. This enables Platform Engineers to say "yes" to more development requests.

Integrating data management within kubernetes

To maintain a streamlined and consistent management approach, ancillary services such as databases and storage solutions also need to be integrated within the Kubernetes ecosystem. After all, if you’re using a cloud’s API for a managed service, you’re again straying off the golden path for a feature like a database, when you might need to re-code this for the same service to move it to another environment.

1. Unified data management

A comprehensive data management platform should seamlessly integrate with Kubernetes, ensuring that all data-related services can be managed using Kubernetes-native tools and practices.

• Declarative data management: Just like application deployments, data services should be managed declaratively to ensure consistency and ease of automation.
• Kubernetes-native storage: Using solutions based on the Container Storage Interface (CSI) enables dynamic provisioning and management of storage resources.

2. High availability and resilience

Kubernetes clusters inherently provide high availability and resilience features, which should extend to data services as well.

• Data Replication and Failover: Implementing data replication and automated failover ensures data availability even during failures.
• Backup and Disaster Recovery: Regular automated backups and robust disaster recovery plans ensure data protection and quick recovery from incidents.

3. Security and compliance

Security and compliance must be consistently enforced across all components of the infrastructure, including data services.

• Encryption and access controls: Data should be encrypted both at rest and in transit, with access controls managed through Kubernetes' role-based access control (RBAC).
• Compliance automation: Integrating compliance checks into the CI/CD pipeline ensures that applications adhere to regulatory standards before deployment.

4. Monitoring and logging

Effective monitoring and logging are crucial for maintaining the health and performance of data services.

• Unified monitoring tools: Kubernetes-native tools like Prometheus and the EFK (Elasticsearch, Fluentd, and Kibana) stack provide a consistent approach to monitoring the entire stack.
• Centralized dashboards: Aggregating data from all services offers a comprehensive view of the system’s health, facilitating quick issue resolution.

Expanding the golden paths

Adopting Kubernetes as a standardized platform allows Platform Engineers to streamline the number of solutions they manage across environments. This simplification opens up new opportunities to enhance the Internal Development Platform (IDP) with additional features without significantly increasing complexity or technical debt.

By integrating all workloads, including databases and virtual machines, into Kubernetes, Platform Engineers can create a more unified and manageable infrastructure. KubeVirt plays a critical role in this by enabling the management of VMs alongside containers within the same Kubernetes cluster.

1. Unified workload management with KubeVirt

KubeVirt extends Kubernetes' capabilities, allowing you to run VMs within your Kubernetes clusters. This integration means that both containerized applications and traditional VM-based workloads can be managed using the same tools and practices, including declarative management. With KubeVirt, Platform Engineers get:

• Consistent deployment: Deploy and manage VMs declaratively, just like containers, ensuring consistency across all environments.
• Simplified operations: Use the same Kubernetes APIs and tooling for both VMs and containers, reducing the learning curve and operational overhead.

2. Enhanced flexibility

With a unified platform, Platform Engineers can more easily introduce new features and capabilities into the IDP. This flexibility allows for:

• Faster innovation: By reducing the complexity of managing different solutions, engineers can focus on developing and integrating new features that drive innovation.
• Scalable solutions: Streamlined management enables scalable solutions that can grow with the organization’s needs without introducing additional complexity.

3. Streamlined infrastructure

By consolidating workloads into Kubernetes, Platform Engineers can streamline infrastructure management:

• Fewer APIs: Managing a small set of standardized APIs simplifies development and maintenance, making it easier to integrate new services and capabilities.
• Unified monitoring and logging: Centralizing monitoring and logging for both VMs and containers within Kubernetes tools provides comprehensive visibility and simplifies troubleshooting.

4. Improved resource utilization

Running VMs and containers on the same Kubernetes cluster improves resource utilization:

• Dynamic scaling: Kubernetes' dynamic scaling capabilities ensure that resources are used efficiently, with VMs and containers scaling as needed.
• Optimized costs: Using a unified platform optimizes costs associated with managing different virtual machine and container platforms.

Summary

Platform Engineers can balance simplicity and flexibility by leveraging a common application platform such as Kubernetes and its consistent API. This approach reduces technical debt and complexity while providing the necessary tools for developers to innovate. By embracing Kubernetes and integrating data management within this ecosystem, Platform Engineers can streamline infrastructure management and offer a robust, flexible environment that meets the diverse needs of their teams without burdening them with the increased complexity of managing different environments with different requirements.

‍

This article was sponsored by Portworx for PlatformCon 2024