Fighting compliance trolls killing platform engineering

Because of compliance

• No DevOps because it lacks segregation of duties
• Automatic deployment of code is not allowed
• No developer may have access in production
• No developer may deploy to production
• No testing in production
• All changes MUST be approved by security
• All changes MUST be approved by QA
• All changes MUST be manually tested
• Continuous delivery practices can't be compliant due to legacy systems
• People are generally stupid and must be protected from themselves

All of the above might perfectly well be true in your organization

But it doesn’t have to be!

What is compliance?

In the software industry, compliance typically covers three things:

1. Legislation (GDPR, HIPAA): Complex rules, policies and procedures that regulate business practices in a particular jurisdiction
2. Information security frameworks (ISO27001, SOC2): Policies, guidelines, and best practices designed to manage an organization's information security risks
3. Contractual commitments: Ensuring that all parties involved fulfill their obligations according to the agreement.

The first topic, legislation, is non-negotiable. Legislation is, literally, the law. You need to follow the law or you could find yourself out of business (or even in jail). As for information security frameworks, these are industry standards which get audited by a third party. This gives your organization the advantage that you can somewhat prove that you follow best practices around information security. When catering for regulated industries such as banking, health or government, this might even be table stakes. Finally, customers have expectations on how you operate to safely work on their data. These expectations often also find their way into written contractual agreements.

Expecting everyone in a large organization to have a full overview over all of these requirements, for all the markets you operate in, is not practical. This is why organizations have policies, or an Information Security Management System (ISMS). This is a collection of all the various requirements, put into a (mostly) coherent set of documents for internal governance on how to fulfill them. If you have ever heard, “We cannot do this because of compliance,” it would typically be rooted in your own organization's policies in the ISMS.

Now, here’s the kicker. With the information security frameworks, the ISMS typically gets audited on an yearly basis. If you do not follow your own policy, you risk losing the certification altogether. And if customers audit you, and you don’t deliver on your written agreement, you will be in breach of contract. These things matter!

Here be dragons! Or rather (since I’m Norwegian), this is where the Trolls are hiding.

What is platform engineering?

In this context, what I mean by platform engineering is three things

1. DevOps at scale
2. Platform as a Product
3. Developer Self-Service

DevOps at scale

One of my favorite definitions of platform engineering is that it is what happens when multiple DevOps practitioners start talking together. Unfortunately, the term DevOps has lost almost all meaning by now, but I’m referring to the practice of being on-call for any system you build and ship yourself to production.

If you are a small SaaS startup, you would by necessity practice this way of work. You wouldn't really call it DevOps, because shipping and operating software is just a part of how you deliver customer value. Code only has value when it is in use! Preferably by paying customers. When your organization becomes bigger, there is a natural pull towards specialization. Benevolent dictator system administrators (sysadmins) run the code, while the code monkeys churn out feature requests. Recipe for success?

Once you enter into “us” and “them”, friction arrives. And when there’s friction, velocity slows. Escaping this friction has become crucial in order to remain competitive. If you scale up using the multiple mini-startup way of having code owners fully operate their code in production (DevOps) to avoid this friction, it looks a bit like this:

An analogy for this way of scaling is food trucks. They operate independently, with separate product offerings. This autonomy enables them to remain agile and act quickly to market changes. But you also have a whole lot of duplication!

Platform as a product and developer self-service

Platform engineering to the rescue! There are a number of things which can be standardized, without compromising developer autonomy. Does everyone need to be on separate cloud providers? On separate contracts and prices? Different solutions for vulnerability scanning? Code hosting? Probably not. Standardizing these and exposing them as a platform to the organization allows for reduced cognitive overhead for the teams. You can keep the DevOps way of work, but reduce the operations burden.

This centralized platform is commonly referred to as an Internal Developer Platform (IDP). Efficiency is driven through standardization, providing built-in security and self-service capabilities. It is important that this platform is offered as a product. This means it must be an attractive option, and more importantly: it needs to be opt-in. If the platform is not providing value to the code owners, they should not be forced into it (or, you’ll end up back with benevolent dictators). An analogy for this pattern is the shopping mall.

Once the platform is up and running, you can leverage economies of scale to reduce operational cost. This gives you the bandwidth to create an even more delightful software delivery experience to attract more users (internal customers), which in turn drives standardization feeding into the economies of scale. This type of self-enforcing feedback loop is also known as the flywheel effect. The flywheel takes a lot of effort to get going, but when it turns, it builds momentum almost by itself!

Compliance trolls

Trolls in compliance are just like Trolls in Norwegian folklore. When the sun shines on them, they turn to stone. Not so dangerous anymore. These might still be big boulders, and hard to move. But at least there’s now a known path on how to remove boulders. Even if you need to chip away at them with a hammer and chisel, it is possible to move forward.

Trolls in legislation

These trolls are lurking in complex laws and regulations that seem to put up barriers to innovation. But with the right strategies, you can work with, not against, them. Legislation often leaves room for interpretation and evaluation. Legal departments generally have a tendency to lean towards the side of caution. Examples:

• Any data must be stored, processed, supported and accessed from within the European Union only. Any, and all vendor, processing data, must be European companies, and all vendor support personnel must be within the European Union only
• Credit card information should never touch our systems
• We can not, under any circumstances, store any health related data

While it is possible to remove GDPR compliance risks when using vendors with strict restrictions, it’s also extremely limiting on what you can do. Is it possible to be compliant with GDPR, and use services from the United States? Absolutely. But it requires additional considerations, like risk evaluation. For any company fully subject to GDPR law directly, there is no risk. If you’re told that you cannot use some SaaS service outside the EU because of GDPR, it often means that nobody has done the necessary work to ensure standard contractual clauses (SCC), a data transfer impact assessment (DTIA), or to verify if the company is part of the data privacy framework in the case of US-based services. If other organizations are successfully using such services, chances are you could use them in your organization as well, if you do the necessary work.

Hint: “You cannot do this” sometimes means, “It’s cumbersome to do this”.

Trolls in contracts

Trolls in contracts come in many shapes and sizes. Some “fun” examples may include:

• Customer to validate all changes before going into production
• Hosting vendor and location specified in contract
• Only named people may have customer data access
• Commitment to glue shut all USB ports for company computers

If you have contracts like this, good luck doing platform engineering! It’s natural to be customer obsessed, and to try to please the customer as much as possible. Especially when trying to get contracts signed. But don’t get possessed! Having your customer take over the control presents a very dangerous situation to your organization's health and sanity, as you may well end up hurting yourself. Not all customer terms should be accepted by default. Contractual commitments can be a major source of friction, but they’re also negotiable. By pushing back on overly restrictive clauses, you can unlock faster development.

Customer obsessed is okay, customer possessed is dangerous.

– Jon Skarpeteig

Hint: Contracts are negotiated!

Trolls in security frameworks

These trolls sometimes come across like a game of whispers. Security frameworks such as SOC2 and ISO27001 outline a set of security controls which must be implemented. Interpretations can manifest themselves in fun ways:

• Separation between dev and ops (separation of concerns)
• Any and all access needs tickets (access controls)
• Only the CISO can approve any risk (risk controls)

The “separation of concern” or “segregations of duties” in particular are recurring platform engineering complications. Since a separation between developers and operators (dev and ops) used to be the way of the world, old practices, old guidelines and reference implementations still advocate for this! When this set up is proposed by an auditor or your compliance department, the risk is that this implementation becomes the truth. As in, the only way to operate in a compliant way! But if you look at the specification of two of the most common security frameworks, they do not state that you need to separate dev and ops:

If you go to the actual control description, instead of company policy or previous practices you will discover that there are many different ways to meet the control requirements.

Hint: Information security frameworks don’t tell you how to be compliant!

Shift left compliance

Doing things earlier in the development cycle means exposing more decisions to the development phase. This concept is well established for testing. Creating more elaborate test suites around unit tests and integration tests compared with manual testing will reduce costs, and in many ways improve quality, because a big part of the feedback loop gets shorter.

Shift left security is also a hot trend in the market today. It is moving the focus into securing the exposed application, supply chain security and code, more than on the perimeter defence around networking and firewalls. The traditional security is still there, but represents a very small portion of web application security risks overall.

For compliance, this pattern seems less established. However, if you ever had to retrofit GDPR into a legacy application, you quickly discover these things aren’t so easy to bolt on after. And if you are serious about succeeding with platform engineering, enabling the developer community to reason, and make their own decisions, is the only way to provide team autonomy. 

    The majority of decisions should be taken by those who have the most context.

                                                                                                            – Jon Skarpeteig

Bridging the Dev-Sec-Compliance-Ops golden path

A well designed Information Security Management Systems (ISMS) can be made to fit perfectly with a well designed Internal Developer Platform (IDP). By introducing design and product thinking consistently, it will reduce cognitive overhead, which can supercharge your deliveries. It builds on a very simple principle: The majority of decisions should be taken by those who have the most context.

Risk management

The ISMS and IDP symbiosis goes into several layers. From ISO27001 there is this concept of an asset. In platform engineering, there is a concept of product. These are for the most part the same thing! If you own the product, by definition you own the asset. In the information security framework, it means you’re also the risk owner. Embracing this concept allows the development team, which has the most context about a change, to also evaluate and accept risk as needed. This doesn’t mean that any engineering team should be able to introduce and accept risks which puts the survival of your company at risk on their own! But the majority of changes are not in this category and should not be left to indecision.

More is lost by indecision than wrong decision. Indecision is the thief of opportunity. It will steal you blind.

                                                                                                                -Marcus Tullius Cicero

Separation of concerns

The idea about separation is that a single job function (individual) shouldn’t both approve the change and execute the change. An example from finance is to separate the one approving loans, from the one paying out the money. In software, this roughly translates to change management. Separating “dev” and “ops” would fulfil this, but it contradicts modern development practices with platform engineering.

For many organizations this requirement is actually already met by a single mechanism: Mandatory merge request approvals, by someone else than the code author.

Done, and done. Concerns separated. Duties segregated.

You may choose to implement other mechanisms as well:

• Continuous integration
• Infrastructure as Code (IaC)
• Continuous deployment automation
  
  • GitOps workflow

But these would be because you want to, not because you need to because of compliance.

Access management

Assets require an access policy. But what you write in the access policy is up to you!

Ensuring only the people which must have access to perform their duties, and no one else, is a key aspect of access management. This is commonly referred to as the Least Access Principle. As in, all access requires a justified need. A common pattern here is to require a ticket for requesting access, providing justified need, and get explicit approval from the asset owner, or someone the asset owner delegates this to. This can pose a significant friction for new joiners, and can be labour intensive.

To reduce this friction, you can create a predefined access list in the access policy itself. That way the justified need is already supplied! Example: Developer working in the signature product team has pre-approved for access to:

• Source code
• Create merge requests
• Production namespace
• Observability tooling
• Product databases

With pre-approved access defined in your access policy, it’s also an excellent area to automate, eliminating the onboarding friction.

Fight your compliance trolls

Can you afford not to leverage modern development practices? Moving fast is a competitive advantage! From the Accelerate State of DevOps Report 2021, Elite performers have:

• 973x more frequent code deployments
• 6570x faster lead time from commit to deploy
• 3x lower change failure rate
• 6570x faster time to recover from incidents.

At the core of elite performance: Short feedback cycles. Getting code into production as soon as possible after writing it.

Adopting platform engineering into compliance

Are your internal policies conflicting with modern development practices? Change the policy! Remember that contracts are subject to negotiation, everything comes with a price. Make sure your ability to move fast is not undervalued! And finally, partner technology with security, compliance and legal groups. Their day job involves interacting with auditors and reviewing contracts. They are excellent at interpreting the intent of requirements. By honoring the intent, many clever solutions become possible! And they are uniquely positioned to help push back on incompatible customer requests.

Conclusion

In the sun, trolls turn to stone and are no longer scary. Security frameworks don’t tell you how to be compliant and contractual commitments are a negotiation. The majority of decisions should be taken by those who have the most context. You build it, you own it: software, risks, security, and compliance. Security and compliance departments are our friends!