Continuous Insecurity?

Continuous Integration (CI) and Continuous Deployment (CD) systems are critical to today’s software development lifecycle (SDLC). CI systems are a piece of critical infrastructure in the SDLC where a number of controls and checks can be enforced; I will discuss some of these in a later post. In this post I will focus on some anti-patterns I have experienced related to CI systems.

CI systems are often centralized and have access to your source code, potentially spanning business units. If the same system is used for deployment, this system would also have deploy credentials to your infrastructure. These factors make the CI system an extremely attractive target for attackers. At a high level, this means that the CI system must be treated as a production system.

Anti-Patterns

Not treating CI like a production system

This means that at the very least your CI system should be updated in a timely manner and have regular base OS updates. A procedure for CI system updates should be put in place early - before the system becomes too fragile to test incremental updates. Additionally, OS level access limited based on least privilege, remote logging should be enabled, and the system should have regular backups taken and a tested restore procedure.

Unnecessary exposure

Though I believe in zero trust networks, I also believe that CI/CD systems should be behind a firewall that limits the potential for attackers to mount an attack on the system. In my opinion this falls under defense in depth; it raises the bar for attackers who want the intellectual property and credentials contained in your CI system. Making the system accessible on the public internet raises the risk of bugs in the CI system code - a public system is potentially only one 0-day away from granting access to your code. Currently on Shodan there are over 40,000 results for Jenkins and about 1,000 for GitLab. Don’t do this.

Over-reliance on perimeter firewall

As stated above, I do believe on using a perimeter firewall for the CI system, but I believe it is equally important to not rely solely on the perimeter for CI security. Don’t run an outdated and vulnerable version of CI software just because you have a perimeter firewall.

Not using TLS

Developers have to login to the CI, right? That means they are likely submitting a password to the CI’s web interface which in turn makes an LDAP query. What happens if the CI isn’t using TLS or is using LDAP instead of LDAPS? Cedentials might be sniffed - hello lateral movement! If the CI uses TLS, but still uses LDAP instead of LDAPS, the credentials could still be sniffed when the directory is checked.

Are valid TLS certificates hard? No. It’s 2019 and Let’s Encrypt is easy and free so there’s no excuse for this.

Not using least privilege

Should everyone who can login to your CI/CD be able to edit jobs, see automation credentials, and run deploy jobs? Should read access be restricted? Depending on your team structure and business needs, at least some of those actions should probably be restricted. Again, following least privilege for RBAC is a good idea. If there’s a risk that a developer accidentally logs credentials in job output (hint: there is), even read access should be limited.

Many modern CI systems have adopted a container-based approach that enables an easier developer experience. I like GitLab CI’s model of configuring CI as code so that developers can have self-service with the added benefit of having a code review of CI changes. In my past experience, a major shortcoming of Jenkins was the UI-based configuration. This led to a messy attempt to apply a “Jenkins as code” pattern with a templating scheme. In practice, this meant the developers still needed job edit access in the UI to test their templated configuration. Obviously with that access, job configuration could still drift. On the other hand, if developers can configure CI as code, then they do not need permissions to configure other CI job options, meaning that they can be given a more appropriate role. In GitLab this is the “Developer” role while the DevOps team can retain “Maintainer” role, allowing them to configure CI secrets.

Conclusion

These are a few of the anti-patterns that I believe can lead to continuous insecurity of CI systems. I have a few more that I will write up in a follow up post.