A core area that we track at Work-Bench is the adoption of DevOps and what the transition looks like in practice for large enterprises. As businesses are increasingly tasked with delivering applications and services at higher velocity, DevOps has established itself as the right guiding principles for faster and safer development and releases of software.
Continuous Integration and Continuous Delivery (CI/CD) has emerged as a best practice within the DevOps space — it solves for quicker testing and deployment of applications, and provides a workflow pipeline that integrates and automates different stages of the delivery process and gets releases ready to be shipped. Through this CI/CD pipeline, engineering teams can minimize manual handoffs, release more often with incremental updates, and enhance the feedback loop of their software development lifecycle.
To better deep dive into the CI/CD space and refine where the best opportunities for investment are, I spoke to practitioners in our network spanning from large Fortune 500s to emerging enterprise startups to get their first-hand perspective on the current challenges and opportunities across the CI/CD tooling chain. While CI/CD and DevOps have clear advantages, there are some major barriers to adoption that engineering teams face across the board.
I’m sharing notes from these recent conversations I’ve had with engineering leaders and would love to connect with folks working through similar challenges:
Innovation around Infrastructure-as-a-Code has made it possible for one to create multiple environments for testing and development, so much that developers are having to maintain multiple environments that weren’t available prior to the datacenter inflection point, and need different tools to adequately maintain each environment:
“CI/CD as it exists today is like a Swiss Army knife: you can use a hammer to nail it down or use a screwdriver, or a drill, it offers so much optionality, but there is no opinionated way of doing it.”
In fact, existing tools can’t keep up with the shifting paradigms around setting up repeatable environments:
“Many large enterprises still treat their servers like pets instead of cattle, i.e., instead of tearing down their servers and recreating them in an automated fashion, organizations are too quick to fix existing ones.
If companies cannot repeatably set their infrastructures and don’t have real diligence around that as a practice, then doing CI/CD is challenging, especially when it comes to spinning up similar environments.”
While continuous integration and delivery has become common practice, achieving continuous deployment remains a huge pain-point for many organizations. And not every company automatically releases to production:
“Our deploy process is extremely manual: we deploy maybe once a day, sometimes we don’t deploy at all…To enable continuous deployment, you need to have a measure of safety, i.e., you need to be able to deploy and roll back quickly if there is something wrong, because you don’t want to have an outage for the users for a very long time.”
To address some of the challenges around continuous deployment at a large financial services firm, they built a system that automates as much of the process as possible:
“To deploy continuously, the software has to meet a set of security scans and performance tests and a certain number of required quality gates. In this case, what automates checking off the gate depends on what the gate is. For example, if you were doing a security scan you would first integrate with a security scanning tool which would then check if the scan has passed or failed.”
The bottom line is that
“you need to have an automated testing framework that you believe to be sufficient.”
Automatically running test suites and generating reports is becoming standard practice at large organizations. What’s great is that by completely automating the entire deployment pipeline, product owners are notified in real-time when their builds are ready to be executed to deployment. They can also view the results of their test suites and depending on the situation, if a test has failed but shipping a new feature is mission critical, then the product owner can help bypass a failed test to push the feature to production.
One of the main challenges with testing microservices is that each of the decoupled components in a microservices environment needs to be deployed independently and there might even be different versions of a component which makes integration testing so much more expensive.
To address this, limit the scale of integration testing to the most critical paths:
“Model explainability for example is really challenging for us to test independently which is why it is so critical to just do integration tests on it.”
However, the caveat with integration testing in general is that it can be hard to execute and doesn’t always provide high degrees of reliability:
“The biggest problem in integration in that it is so brittle, there are so many false positives which causes things to break and it just adds more cycles and that’s really the problem with automated CI/CD.”
It’s important to have a well-defined workflow that clearly outlines the steps that one needs to take in order to test successfully:
“Developers need key integration points to be built-in right into their workflows for them to move fast enough…integrations with test suites, code repositories, security scans should all be packaged in developer workflows. Things like Slack integrations that notify you when something is broken are also great because they enable some reporting capability that is critical.”
We continue to track this space and if you’re a startup building in this space or a DevOps practitioner tackling some of the ongoing challenges in CI/CD, I’d love to hear from you.
Many thanks to my colleague Kelley Mak for contributing to this post!
