AI is changing the future of platform engineering. And the future is much closer than you might think. Of course, we’re talking about the latest in the AI world 👉 LLMs, or large language models.
I’m sure this isn’t the first article telling you that AI and, more specifically, LLMs will change how we develop and ship software in the next few years. I’ve been chatting with folks in the platform engineering community about this topic for a while, and they’re coming up with some pretty interesting stuff. It’s the sort of stuff you’ll want to know about before anyone else, and that you should start thinking about too. So, let’s dive in.
Understanding large language models (LLMs)
First, let’s look at the most recent headliner in AI. A large language model (LLM) is “a type of artificial intelligence (AI) algorithm that uses deep learning techniques and massively large data sets to understand, summarize, generate, and predict new content.” With the ability to synthesize large volumes of information, LLMs can potentially improve the efficiency, scale, and consistency of business processes that previously required human specialists.
Wharton School business professor Ethan Mollick explained in a recent episode of the Foward Thinking podcast, that usage of LLMs in early controlled experiments net anywhere between 30% and 80% performance improvements for individual tasks like coding, writing, marketing, and business materials. He compared that to 18% to 22% for adding steam power to factories in the early 1800s.
The exciting news is that the power of LLMs is more accessible to businesses than ever, thanks to the emergence of multiple freely available models like OpenAI’s ChatGPT, GitHub’s Copilot, and Google’s Bard. With the price of LLMs dropping and new offerings on the market, organizations of all sizes can now also buy more sophisticated and secure versions of pre-trained LLMs instead of investing the large amount of resources required to train a model from scratch.
Leveraging LLMs for platform engineering
Platform engineering is the discipline of designing and building Internal Developer Platforms (IDPs) to enable developer self-service and standardization by design for software engineering organizations in the cloud-native era. An IDP consists of many different tech and tools glued together in a way that lowers developer cognitive load without abstracting away context and underlying technologies.
LLMs can help platform teams build more compelling IDPs. While individual developers and platform engineers can use ChatGPT without the organization’s support, organizations that intentionally integrate LLMs into their platform could further automate repetitive tasks in a standardized way, improve discoverability, and provide support for developers, platform engineers, and managers.
The table below outlines some potential use cases for LLMs in platform engineering that can change the game.
Some use cases are easy, low-hanging fruit. They can be implemented independently or with a 3rd party solution, are reliable at least 95% of the time within a few weeks, use available data, and require knowledge that fits into the context. On the other end of the spectrum are use cases that are hard to implement. They would require expertise in LLMs and take more months to become acceptably reliable. They would also force the organization to create its own dataset, refine an existing model, or potentially train a new model.
It’s important to consider not only how LLMs can augment platform engineering now but also what’s coming in the future. Longer-term thinking can help organizations better understand if and when LLMs are worth the investment. Based on what we know about LLMs now, I think it would be worth considering leveraging them for these potential benefits:
Reduced repetitive work
LLMs can minimize the burden of boring and repetitive work. Think of them as an assistant you can delegate your most tedious tasks to: creating configuration files, identifying mistakes in configuration files, documentation, resource definitions, reports, etc. Whether you’re a developer, platform engineer, or manager, LLMs can free up more time to focus on creative and high-value tasks. This not only boosts productivity but also has the potential to mitigate the risk of human error that often accompanies monotonous work. By expediting tasks that previously required significant time and effort, LLMs can make the IDP experience more efficient and enjoyable for users.
I found Manuel Odendahl’s description of how LLMs had transformed his work to be a particularly compelling example of why leveraging LLMs is worth considering:
This might be the most subtle shift, but I believe it’s also the most profound change that using LLMs has brought to me. After a day of being focused on getting “the tedious stuff” right (using the right API, checking the right efforts, properly calling API X, implementing the unit tests and mocks for Y, writing an API wrapper for Z), my brain would be entirely gone, consumed by the minutiae and intensity required not to slip up. I would spend the evening playing video games or watching Netflix to recover.
Since extensively using Copilot and ChatGPT, this cognitive exhaustion is pretty much gone. 6pm strikes and I feel like I spent the day chatting with a buddy, yet 5 PRs have been merged, the unit tests have been written, two tools have been improved and the code has shipped.
Unsurprisingly, large volumes of mindless, unengaging work contribute to developer burnout. In addition to helping developers boost their productivity, LLMs might also have the potential to make them happier while doing it.
In the same article, Odendahl wrote that he has been programming with Copilot since the tool was in beta. The benefits he has gained from LLMs are likely, to a significant degree, the result of his desire to learn about and refine his use of the technology. Other developers might lack the experience or the motivation to become proficient in coding with LLMs. Platform teams should not assume that their organization’s developers will use LLMs if they are not meaningfully integrated into the platform’s golden paths. Furthermore, platform teams should avoid relying on models not trained or finetuned to the organization’s specific platform as that increases the risk of LLMs generating faulty answers. Therefore, by weaving an LLM into an IDP, an organization’s platform team can better ensure it improves developer experience while also enforcing a greater degree of standardization.
Greater discoverability
LLMs’ natural language processing capabilities have the potential to enable developers to explore the platform’s features, application examples or sample code, and command syntax by using plain language queries. They can also simplify understanding the platform’s resources by synthesizing extensive documentation into contextually relevant recommendations or guided tutorials.
Better support
If adequately trained on specific to the organization’s IDP, LLMs can act as virtual support assistants for the platform, responding to developer queries, addressing technical issues, and providing real-time guidance. By integrating LLMs into their IDP, platform teams can offer more efficient and responsive support, ensuring developers have the assistance they need to work smoothly and resolve issues promptly. Using LLMs to support users means fewer support tickets for the platform team and more time to focus on complex and meaningful work.
LLMs could also help platform engineers understand and communicate the impact of the IDP to non-engineering stakeholders. Platform engineers could more easily get answers to questions like:
- How often have we deployed this week?
- How many unused environments are running?
- How many hours have we saved this week?
- What is the current NPS of the platform?
This could also minimize executives’ or sponsors’ reliance on the platform team to generate up-to-date and plain-language reports on metrics to measure the IDP’s success. LLMs could, in essence, act as a virtual translator between engineers and executives – stakeholder groups who often speak very different “languages” – making communication less frustrating and more efficient.
Important considerations for incorporating LLMs in platform engineering
Of course, organizations should be careful about how they incorporate LLMs into their platform. As an emerging technology, LLMs are imperfect and can become a major liability if misused. Your organization probably doesn’t want to be the next Samsung and has to start banning employees from using AIs for fear of data leakage. Better to set the right guardrails from the get-go.
Use LLM responses with caution
Like with any other technology, LLMs have potential security risks associated with their use. Research conducted by vulnerability and risk management company Vulcan Cyber revealed that ChatGPT occasionally recommended code libraries that did not currently exist. The researchers warned that threat actors could collect the names of non-existent libraries recommended by ChatGPT and create malicious versions for developers to download.
The Open Worldwide Application Security Project (OWASP), a nonprofit foundation working to improve software security, shared a list of the top ten most critical LLM vulnerabilities developed in collaboration with hundreds of expert contributors. In an interview with Infosecurity, John Sotireopolous, a senior security architect at Kainos and part of the core group creating OWASP’s Top 10 for LLMs explained how the relative danger of each vulnerability was determined: “For example, ‘Prompt injection’ (LLM01), which is the ability to influence the outputs of a language model with specific, hard-to-detect instructions, is the top vulnerability because you will face it regardless of what model you use and how you use it. On the other hand, ‘Insecure Plugin Design’ (LLM07) requires additional plugins, so it’s further down the list.”
Even without a security threat, LLMs are far from 100% reliable for 100% of tasks. LLMs can “hallucinate” incorrect answers to queries. As this TechCrunch article notes, “the mistakes range from strange and innocuous – like claiming that the Golden Gate Bridge was transported across Egypt in 2016 – to highly problematic, even dangerous.”
Even when an LLM generates faulty code, the individual who pushes that code into production is on the hook. Anything generated by an LLM needs to be checked by the user to ensure that the contents of the response are correct and compliant. Ideally, platform teams have also tested the LLM integrated into their IDP to understand better what tasks it can assist with most reliably.
Some organizations might assume that an LLM’s capabilities eliminate the need for skilled human labor. For example, let’s say the thinking is that an LLM can create a config file for an AWS-based setup based on a natural language prompt. The assumption may be that the developer providing the prompt might not need to understand the configurations in detail. However, this is far from the case. Individuals using LLM assistance need the expertise to verify the LLM’s output. Experts can use LLMs to execute simple tasks much faster. On the flip side, unskilled users can use LLMs to generate faulty code much faster. 👎
Platform teams should understand the risks of integrating LLMs into the IDP and decide what policies to establish accordingly. As an example, you want to ensure you build the right checks and balances into your platform setup before your LLM hallucinates a config file that deletes your production database.
Investing in domain-specific LLM prompt engineering or training
Software engineering organizations leveraging an LLM for their IDP might also consider training their own model. Or finetuning a pre-trained one to improve the LLM’s reliability for workflows more specific to the organization. This would have more significant costs associated with it, but it could also improve the usability and reliability of the LLM.
If LLMs are prevalent in an organization’s IDP, there might be a need for dedicated professionals to engineer prompts for users and facilitate further training of the model. Platform teams should treat prompts and training like any other type of feature development for the platform. Namely, they should follow a platform as a product approach to ensure that prompts and training ease developers’ most pressing pain points.
Do LLMs threaten job security?
A lot of folks are afraid that LLMs will eliminate their jobs. For now, the technology isn’t there yet. Human labor is still needed to verify LLMs’ output. Ideally, LLMs can continue eliminating boring and repetitive work without eliminating jobs.
LLM usage can, however, threaten the job security of professionals reluctant to embrace it. David Eastman argued that “Copilot is a tool, and works with or because of the developer’s ability to define small bits of functional code, use convention, and work within the context familiar in any Microsoft solution. What it is effectively doing is removing the step of opening a tab on Stack Overflow.” He suggested that junior developers would use LLMs like Copilot judiciously to gain an advantage over peers who are more resistant to change. If more platform teams integrate LLMs into their IDP, I expect that the gap between developers who can leverage LLMs and those who choose not to will grow. Thankfully, developers can choose how much they will invest to gain that competitive edge.
The same situation also applies to platform engineers. For platform engineers still focused on fielding ticket requests from developers or primarily engaged in repetitive work, LLMs can either present an opportunity to improve or the risk of falling behind the pack.
What’s clear is that early adopters of LLMs could quickly outpace everyone else in terms of quality and quantity of code output. This is true for every field of software engineering, and platform engineering is no exception.
Next-level platform engineering with LLMs
LLMs will change technology as we know it, and the best platform teams will learn how to leverage them to build the next generation of IDPs. As Ethan Mollick shared, “Use this thing. I think the only way out is through, and I have a theory that the only way you know if you’ve really started to get what this thing means is you have three sleepless nights… where you’re like, ‘Oh, my God. This is so exciting. This is so terrifying.’”
Ultimately, there’s no difference between LLMs and previous waves of technological innovation. They are a tool that multiplies the input in either direction: It can make your developers 10x more productive or your IDP 10x messier and less secure. Dive deep, embrace it, and come out victorious on the other end. See you there (maybe).
.webp)
