Shared staging environments are one of the most persistent bottlenecks in modern software delivery - slowing developers down while keeping platform engineers in a constant state of defensive maintenance. In this talk, Francis Halt of ClearRoute walks through how virtual clusters built with vCluster can give every developer or team a disposable, production-like Kubernetes environment on demand. The approach also opens up a safe sandbox for AI agents to operate in before anything touches a shared environment.

TL;DR: Main insights

• The shared staging environment is the wrong unit of isolation. Kubernetes ephemeral environments resolve the tension between developer velocity and platform stability.
• vCluster runs lightweight, logically isolated Kubernetes clusters inside a host cluster, delivering a real API experience without provisioning new infrastructure.
• A GitOps pipeline using Argo CD and a resource template makes provisioning a pull request - platform engineers keep their guardrails.
• Environment fidelity is the most critical design decision you will make. It determines how much value teams actually get from these environments.

Francis Holt is a Staff Engineer at ClearRoute, an IT consultancy focused on platform engineering, route to live, and, more recently, AI. Francis shares how the team built an ephemeral environment capability on their internal development platform, Constellation, which runs on AWS EKS.

You can watch the full discussion here if you missed it.

Why shared environments hold teams back

Francis opens with the core problem ephemeral environments exist to solve. A path-to-production environment - such as system test - contains databases, applications, microservices, policies, and integrations. Different stakeholders want fundamentally different things from it:

• Developers want to push changes and see features and bug fixes work immediately.
• Platform engineers want environment stability and a protected path to production.
• Project leads want velocity above all else.

This is the classic DevOps conflict, and Francis speaks to it from direct experience: "I know how much I wanted to protect my path-to-production environments." The pain compounds with scale. Imagine five development teams pushing changes to ten applications on the same day. When system test breaks, you spend hours working out which of the ten changes caused it. Context switching adds further cost - a developer finishes feature X, moves on to feature Y, and two days later learns that feature X broke system test.

What ephemeral environments are

Francis defines an ephemeral environment as a small replica of a real path-to-production environment, scoped to a developer or development team. The key properties are:

• Short-lived: A developer can provision one at the press of a button, without a Jira ticket or a message to a platform engineer.
• Developer-owned: The owner can deploy pods, delete pods, add secrets, mount disks, and tear the whole thing down.
• Isolated: Each environment is isolated from the path to production and from every other ephemeral environment, so breaking one carries almost no cost.
• Representative: It contains the applications, platform services, and potentially the external integrations a developer needs to validate changes before submitting them upstream.

Benefits across the organization

For developers: Isolated experimentation without harming the path to production, less cross-team noise, immediate feedback loops, and far less context switching because problems surface before changes are submitted.

For platform engineers: No SLA burden on ephemeral environments - as Francis puts it, "as long as they are functional and performant, I think that's absolutely perfect," because these environments are experimental by nature. Fewer integrations to maintain, since the virtual cluster reuses what the host cluster already provides - operators, Prometheus monitoring, a service mesh - and no new infrastructure to provision per environment.

For the organization: Testing shifts left, and uptime of the real path-to-production environments improves as a direct result.

How vCluster works

vCluster delivers tenant environments on top of a host Kubernetes cluster. The host cluster - EKS in ClearRoute's case - has its standard namespaces such as kube-system and Argo CD. Each virtual cluster places itself in a tenant namespace and runs a control plane pod. Developers work against the nested cluster and experience what feels like a real Kubernetes cluster, complete with its own kube-system namespace, pods, and standard resources.

This is not simply Kubernetes inside Kubernetes in the way you might run Docker inside a container. The critical detail is how resources sync:

• Compute resources - deployments, stateful sets, pods - deployed inside the virtual cluster actually land on the host cluster. The virtual cluster's control plane pod syncs with the host's API server to make this happen.
• Key-value resources such as secrets and config maps stay in the virtual cluster and are not copied to the host, unless a synced workload references them - in which case they must also exist on the host.
• Host-side pods carry annotations showing vCluster's ownership, identifying the original pod name and namespace inside the virtual cluster.

Because workloads run on the host, they must still follow the host's rules. Network policies, service accounts, roles, container security settings such as running as non-root, and resource quotas all still apply. As Francis notes: "I still have to run as non-root." Platform engineers keep control without having to manage each environment individually.

How ClearRoute provisions environments

The provisioning flow Francis describes is one of the real design wins of this approach:

1. A developer raises a pull request in GitHub requesting a virtual cluster, supplying only a name and a Kubernetes version - CPU and memory are optional.
2. Argo CD syncs the request to the cluster.
3. A Kubernetes Resource Orchestrator controller - which Francis calls Crow - hosts a resource graph definition: a single YAML template that defines every resource a virtual cluster needs, including service accounts, roles, role bindings, config maps, and a network policy.
4. Crow detects the request, populates the template, and deploys the virtual cluster.

The practical payoff is that scaffolding is defined once. If you later want every environment to include a service mesh or a database, you update the template once and every subsequent environment gets it automatically. Karpenter handles node autoscaling underneath, so you are not provisioning infrastructure per request.

Design challenges to plan for

Francis is candid that ephemeral environments are not simple to implement. He highlights three areas of difficulty, using Constellation as a concrete example - a cluster running around 200 pods and approximately 20 applications on its development environment:

• Complexity of the existing environment: You are creating a replica, so a complex source environment means a hard replication problem. You must decide which applications, infrastructure components such as RDS and S3, and platform services such as the service mesh the replica actually needs.
• The deployment mechanism: Environments do not appear by magic. You must design how they deploy - whether through a Tekton pipeline, GitHub Actions, GitOps, Helm, or Kustomize - and consider how Argo CD handles environments spun up at any time by anyone.
• Fidelity: How closely must an ephemeral environment mirror your real path-to-production environment? For some teams, a simple virtual cluster is enough. For others, the environment must contain every application and integration present in the real thing. Francis calls this "a string that's gone through this entire presentation" - it is the decision that shapes everything else.

Safe sandboxes for AI agents

Francis closes with a demo that points to where this capability is heading. He built an agent called Cube Mechanic - a set of containers running as a pod that observes cluster logs and events and autonomously fixes problems it finds, such as a deployment referencing a missing config map or a bad image reference.

Deploying an autonomous agent like this into a shared environment would be a significant risk. In the virtual cluster, Francis deploys a deployment with a misspelled image name - busybow instead of busybox. The pod immediately enters an image pull error, and the agent works through validation, triage, diagnosis, a repair plan, and repair. A Postgres database tracks each stage. If the agent had broken the environment, Francis could raise a pull request or have Argo CD sync again for a clean start. The ephemeral environment becomes the safe proving ground that makes autonomous AI tooling viable.

Q&A highlights

• Cost: Infrastructure scaling is the first cost you see. If you build a good capability, everyone will spin up environments. Licensing for enterprise vCluster and the additional platform engineering effort for logging and monitoring are also worth factoring in. Pre-provisioned headroom speeds creation; Karpenter scaling on pending pods is slower. Francis suggests an environment ready within 15 to 20 minutes is a reasonable target.
• Where to start: He sees ephemeral environments as a useful forcing function - they push you to get pipelines, GitOps, and config as code right from the beginning.
• Middleware: On a previous program, a Tekton pipeline deployed a full replica of system test including middleware, databases, and Kafka. That worked because everything was defined as YAML in Argo CD.

​

If you enjoyed this, check more great insights and events from our Platform Engineering Community.

For more comprehensive guidance, check out the Platform Engineering Certified Architect Course and learn best practices from industry experts.

​

Key takeaways

• Isolate by team, not by environment tier. Give developers their own virtual cluster so they can test changes where breaking things costs almost nothing, keeping the shared path to production stable and reducing cross-team noise at scale.
• Reuse your host cluster's guardrails. Because vCluster workloads run on the host, your network policies, quotas, roles, and container security rules still apply. You can offer developer freedom without surrendering platform control.
• Make provisioning a pull request. Define a single resource template and drive it through GitOps. Developers request an environment with a name and version, and you improve every future environment by updating the template once - no per-request infrastructure work required.
• Decide fidelity early and plan for scale. Be explicit about how closely an ephemeral environment must mirror your real environments, account for infrastructure and licensing costs as adoption grows, and consider these environments as the right safe sandbox for AI agents operating autonomously.