Given the seemingly unending stream of cyberattacks, most developers don’t need anyone to remind them that securing development pipelines is important. But what eludes many teams is how you design and implement a secure CI/CD pipeline. It’s one thing to talk about CI/CD security and another to put it into practice.
Here’s a look at five practical strategies for securing CI/CD pipelines with that reality in mind. Although not every one of these practices will make sense as a security strategy for every team and pipeline, most organizations can benefit from these processes to bolster CI/CD security.
Why Secure the CI/CD Pipeline?
Before delving into practices for securing the CI/CD pipeline, let’s explain why CI/CD security is essential.
To some developers, the importance of securing CI/CD processes may seem so obvious that it’s not even worth discussing. However, because most CI/CD pipeline stages occur before applications are in production, it can be easy to overlook security there and instead focus on securing production environments, where the most significant risks tend to exist.
It’s true that most threats don’t touch your applications until the deployment stage of the pipeline or later. Yet, because the vulnerabilities that those threats seek to exploit are often introduced in earlier stages of the pipeline, baking security into all stages of CI/CD is critical for delivering applications that are as secure as possible in production.
In addition, certain risks, such as insecure management of secrets within development environments, can creep into the pre-deployment stages of the pipeline. That’s another reason to secure all stages of the pipeline.
Five Best Practices for Securing the CI/CD Pipeline
Now, let’s look at actual practices for keeping CI/CD pipelines secure.
Add Security Tests to Your Testing Routine
You probably (hopefully) already run pre-production tests on your application releases to vet them for reliability and performance. Doing so is a best practice for ensuring quality releases for your end-users.
But it’s equally important to include security tests within your testing. Although you may not be accustomed to thinking of security testing as something that you can do alongside performance testing, the fact is that you can extend frameworks like Selenium to perform security tests, too. See this SQL Injection as an example.
So, if you’re not already running security tests as part of your testing routine, now’s the time to start. After all, performing security testing early in the CI/CD pipeline is what “shift-left” security is all about.
Leverage Feature Flags to Manage Risks
Developers often use feature flags (or feature toggles) to add new features to applications while mitigating the risk that those features will introduce performance or stability issues. By using “flags” to turn the features on and off, developers can easily integrate them into the codebase while retaining the ability to turn them off quickly if necessary.
Although most developers likely don’t think of feature flags as a security tool, they can be used for that purpose, too. After all, new features are prime vectors for security vulnerabilities, especially if developers haven’t thoroughly tested them yet.
By using security flags to disable new features within production environments until they have appropriately vetted them, developers can reduce the risk that those features will cause security issues. In this way, feature flags help developers continue to improve their applications while ensuring the associated risks are always in check.
Use Roll Backs to Manage Production Security Bugs
No one likes a rollback, which means reverting an application to an earlier version. Unfortunately, though, rollbacks are a fact of life. Sometimes, the fastest and smoothest way to solve an issue is to revert the application to a known, stable version.
That’s true not just when performance or stability bugs arise but also in the case of security issues. If you can quickly roll back an insecure release, you can mitigate the amount of harm caused by inadvertently pushing such an application into production.
Designing for fast and reliable rollbacks from the beginning is a best practice for overall CI/CD security. Just keep the binaries for earlier releases on hand so that you can redeploy them if necessary. Make sure your deployment tools can efficiently pull one version of the app out of production and replace it with another one quickly.
Securely Saving Secrets
It can be tempting to cut corners with secrets management in the development and testing stages of the pipeline, even for developers who are dead serious about securely managing passwords, SSH keys, and other secrets in production environments.
After all, only trusted internal users would have access to dev/test environments in theory. It wouldn’t seem like a big deal to do something like letting multiple users share a single account or hard-coding passwords into source code, then removing them before deploying the application.
The reality, of course, is that these are not best practices from a security standpoint. How easy is it to forget to remove a hard-coded secret before deployment? Shared accounts are never a scalable route, even if it’s just for dev/test.
The solution to risks like these is to apply the same security standards to environments used during the dev/test stages of the pipeline as you do to production. To simplify things, use secret managers to store credentials securely, even if it’s just for testing. It’s crucial to require each internal user to have a separate account. And, most critically, do not store sensitive data without enforcing authentication and authorization rules to govern it, even if it is only accessible from a local or private network.
Anomaly Detection Across the Pipeline
You probably already use a SIEM or similar security platform to detect anomalies within production environments that could be signs of a breach. That’s an essential best practice for securing user-facing applications.
An even better best practice, however, is to detect anomalies at all stages of the pipeline. That means analyzing logs from dev/test environments and logs from various CI/CD tools themselves. Any patterns in this data representing a departure from the norm could signify something is off – such as a misconfigured file, failure to adhere to internal governance rules, or a security issue within upstream software (like open source libraries) that you incorporate into your application.
The more data you scan for anomalies and the more representative that information is of the entire pipeline, the greater your ability to detect and address vulnerabilities before the software is in production.
Conclusion: Securing the Entire Pipeline
You can’t secure your application without securing your CI/CD pipeline. Although most real-world threats don’t materialize until the software is already in production, the goal of developers should be to leverage practices that minimize the risk of active vulnerabilities reaching production environments in the first place. The earlier the better. These actions consist of security tests, feature flags, and across-the-pipeline anomaly detection. The more secure your CI/CD pipeline is, the fewer security issues you’ll have to contend with once your code is in production.
