How to evaluate open source software? A DevOps guide

Hila Fish
Senior DevOps Engineer @

Open source software (OSS) is a vital part of any DevOps toolchain or platform setup. However, choosing the right OSS project for your environment is challenging. It’s often difficult to determine whether it’ll be the right fit and if you’ll waste your time trying to make it work alongside your other tools. 

In June 2022, Hila Fish, Senior DevOps Engineer at Wix, led an eye-opening talk on a DevOps perspective on open-source software. Carry on reading to learn more about her recommended best practices on how to evaluate OSS projects before adding them to your DevOps toolchain. 

Open source, open choice

(Source -

A lot of the tools that are used in the DevOps toolchain are open source and publicly available. Tools like Python, React, Java, and more are commonplace, but it hasn’t always been this way. In the late 90s, proprietary software was the standard, but the concept of open source is now moving into the mainstream. 

OSS projects have significant advantages over proprietary software in that publicly available and modifiable source code helps to build a rich developer community around the project. It’s a community that’s based on sharing knowledge, communication, and collaboration as the foundation for successful software development. 

On top of that, OSS projects boost adaptivity. The more users a project has, then the more use cases it’s typically able to support, so the code is typically more robust. In addition, when OSS projects are adaptive to various use cases, bugs that typically only surface in edge cases are found much more rapidly than through traditional QA processes. 

Finally, a lot of OSS projects follow modern software development processes, which leads to improved agility through rapid development cycles and more frequent releases. There’s also less bureaucracy whenever contributors open a PR because there typically isn’t a set release schedule, so new updates can be pushed without any sacrifice in quality. 

Hila recommends checking out a great article by Ben Balter from GitHub that covers more aspects about OSS adoption. 

The DevOps perspective on OSS projects

(Source -

When developers are looking at adding tools to the development environment, they have functionality to consider. Budget, compatibility, security, and maintenance are all key factors both parties (DevOps and Developers) need to consider, and it’s a focus for DevOps specifically because the environment's health and stability is the DevOps’ first priority. All of these factors are influenced by the biggest uniting factor between DevOps and OSS - collaboration. Hila quotes a reference from Ben Balter’s article:

“When it comes to challenges, collaboration leads to better conclusions and solutions. The diversity of ideas that naturally emerges from a community forum around shared challenges leads to better solutions.”

And adds:

“Of course, we all want better solutions because we care about our engineering environment and want to make it stable, safe, and secure.” 

Environmental stability is a key concern, but DevOps operates in a state of trade-offs and maintaining a balance between integrating new tools and stability. Every new tool creates added complexity, so developers need to carefully consider whether the OSS project they’re looking to integrate into their environment is worth the time it takes to fix bugs and maintain a stable environment. 

With thousands of OSS projects on the market, developers need to do their research to make sure they know what the tool they want to integrate into their environment will do. So, it’s important to read blog posts, watch YouTube reviews, and understand the tool fully before committing to the project. 

The great thing about OSS projects is that developers can read the documentation and planned improvements to determine when it’s best to consider integrating a project. In some cases, a project that shows promise may not be suitable for a developer’s use case until its second version has been released, which introduces a much needed bug fix, or a new feature that is more suitable for the desired use case. 

Understanding when to integrate OSS projects

Open-source software can confer a lot of benefits to the development environment, but developers need to know when it’s appropriate to integrate OSS projects. 

First of all, OSS projects are a good choice when it comes to uncommon use cases. Hila says: 

“In one of my previous companies, we needed to downscale our Kubernetes pods to zero to save money. At the time, Kubernetes on Google Cloud only allowed us to downscale to one pod, not to zero. So, we searched online and found Keda which allowed us to downscale to zero. Even though our use case was uncommon, we were able to find an OSS solution that worked for us.” 

Developers also need to consider budget limitations. As OSS projects are typically free or have a very low cost, it removes the need for organizations to purchase expensive enterprise-grade software. 

In addition, OSS projects are a good option when developers don’t have the in-house resources to build their own solutions. If there’s an OSS project that fits that use case, then, as Hila puts it, “there’s no point in reinventing the wheel.” 

With that being said, it is in order to consider OSS downsides and complexities while evaluating an OSS project and considering integrating it in our environment.

(Source -

Security is a significant concern with open-source projects. Because the source code can be viewed by anyone, it’s easier for threat actors to find security loopholes and exploit them. By its nature, OSS projects can’t use security by obscurity to protect software. 

OSS projects can also be prone to abuse. While this is rare, it can happen because open-source software doesn’t have a warranty or guarantee in the same way as licensed software. If organizations need to maintain compliance with certain laws or security standards, they should consider whether the OSS software of their choosing will allow them to meet those requirements. 

In addition, OSS projects aren’t always free. If budget is a concern, organizations need to pay close attention to the licences of the OSS projects they want to integrate. 

Projects being discontinued is also a possibility. Since a lot of OSS projects are not company backed, but maintained by individuals, at any point, the developers maintaining the project can decide to discontinue it, which means organizations will need to maintain the project themselves or migrate to another tool. Support isn’t guaranteed as it is with licensed software, and there’s a need to “assume good faith” when it comes to the request to have such support, and Hila shares her own example of that, and receiving support for the Keda project she integrated. 

How to choose the right OSS project

There’s no right or wrong when it comes to adopting an OSS project as it’s a matter of perspective. It all depends on the needs of the organization and development team. So, how can organizations make sure they’re adopting the right project?

(Source -

There are several questions that organizations and developers need to ask and research when they’re narrowing down which tools they want to integrate into their toolchain. These are:

Is it popular? Check its online presence, including the GitHub Stars count, and if the project is part of CNCF / Incubator. 

Is it active? Check the commits rate, number of releases & issues, and how many developers are on the project. 

Is it ready? Look at whether the current features are enough to cover the use case. If those features don’t cover the use case completely, are the developers OK with that?

Is it documented? Hila says “Documentation is the gateway for the project.” as it helps developers to implement and maintain the tool. 

Does it have an ecosystem? See if it has a dedicated community channel (such as Slack or Discord) that can provide support on a day-to-day basis, and know that a large ecosystem means that the project has a lot of users, which you can assume means a lot of use cases and implementations. 

Is it easy to use? Check the Issues on GitHub - are they about features or asking how to use the tool? If so, it might not be easy to use. 

Is there a roadmap? Look at whether the developers plan on keeping it open source forever or want to monetize it, and if there’s a roadmap for future features, which means the project is very well thought of and maintained. 

Contributing to OSS projects

Organizations and developers can contribute to open-source projects and support this ecosystem without having to write code. Opening bug fixes or feature requests will help the project developers make the software more robust. Similarly, contributing as a documentation writer will help other developers understand whether the project is the right choice for their toolchain. 

Developers can also share their use case. Hila says:

“Writing blog posts about the tools you use, helps people learn about tools they may not have come across before. From there, they can take a look, maybe adopt it into their own toolchain, and inspire other developers to share their use cases too. It also spreads the word about the benefits of open-source software.”

Organizations can also sponsor OSS projects that they use within their toolchain to help the developers to maintain the project. Any sponsorship money boosts the open-source ecosystem which, in turn, benefits users of OSS projects.

Finally, having an open-source mindset and seeing OSS projects as a viable alternative to licensed software helps to spread the word on open-source tools and make it a de-facto option for engineering day-to-day solutions. Organizations and developers speaking about the OSS projects they use in conferences, meetups, or webinars boosts the presence of open-source software, which in turn benefits everyone involved. 

Thanks to Hila Fish for this in-depth exploration of open-source software and its role within the DevOps toolchain. If you didn’t get to listen to her webinar in June, you can find the recording on the Platform Engineering YouTube channel