You’ve certainly come across one of the terms SecDevOps, DevSecOps or DevOpsSec in the last couple of months – strictly speaking these are all semantically different, but generally refer to the notion of a “secure” version of DevOps. Let’s call it SecDevOps going forward and see where and how to put security into DevOps. For starters, let’s step back and quickly review what the software development practices DevOps and its foundation and companion Agile are.

Quick review: Agile and DevOps

Agile was born roughly 20 years ago driven by the frustration of software developers with waterfall-style project management of software development. Agile promotes iterative and incremental development of software in small batches and prefers working code, human interactions, continuous improvement and frequent feedback loops over Big Design Up Front (BDUF) and documentation. This creates flexibility in the light of ever-changing requirements and new insights along the way. An important enabler of Agile and its fast iterations is automation by moving the build, integration and deployment processes as well as the configuration of the environment and infrastructure into code. That is, the build pipeline and network/infrastructure configuration are managed like regular code and also “live” in repos.

To achieve confidence in the correctness of the frequently and automatically deployed code (“don’t break the build”) testing has to be automated as well and becomes part of the build and deployment pipeline. Clearly, for Agile the old model of “throwing the code over the fence” from the agile devs to the rather stability-preferring ops guys creates misalignments and conflicts. For Agile to work properly end-to-end, dev and ops need to organizationally move together and the joint DevOps-team supports their own applications by taking collaborative responsibility for packaging, provisioning, runtime configuration and deployment as well as monitoring and incident response.

Adopting a Secure Software Development Framework

Traditional risk, security and change control practices that rely on extensive up front design and documentation and pre-scheduled security gates cannot keep up with the speed and iterative nature of Agile and DevOps. New approaches are needed to make sure that DevOps not only runs fast, but also securely and that it creates well-secured products.
The good news: all we know about risks, threat actors, the three tenets of secure systems (CIA) and the security fundamentals and principles (least privilege, defense in depth, separation of duties, complexity is the enemy of security, etc.) all still hold. The new challenge is to integrate all these into the iterative design, coding and deployment loop.  While this was already true before, with DevOps security now definitely cannot be an afterthought, but needs to be an integral part and first class citizen of the Software Development Life Cycle (SDLC). The adoption of a Secure Software Development Framework (SSDF) [1] helps to drive this change in all relevant dimensions (roles, processes and tools). In the following we’ll look at some of the process and tooling aspects of a SSDF.

Security is a dimension of software quality

Like usability or performance, security can be defined as a property or quality of a software product. Security issues are bugs and, therefore, generally all measures that improve code quality (e.g. coding guidelines and best practices) and reduce the number of bugs also have a positive impact on security. Typically, not all developers in a project can be security experts (but some should be), everyone on the team needs to have a basic understanding about trust boundaries, threat modeling, attack trees and awareness of the most common security risks [2,3] and how these are managed. For security to become part of the backlog, miss-use cases by anti-personas and corresponding negative testing should be added to the normal “happy path” requirements and testing by using, for example, the STRIDE-model [4]. This results in security being included in the “definition of done”.
Agile teams are self-organised and autonomous and should not be centrally controlled, not even by the security team. However, existing corporate security teams can put enablers in place to help the agile teams to produce secure software – these might include education about and easy access to (i.e. self-service and automation-ready) security tools, patterns, configurations, hardening templates as well as checklists.

Protect the deployment pipeline

Reliable, consistent and repeatable generation of deployable software whenever there’s a committed change in the source code is the goal of the automated build and deployment pipeline. However, the privileges and power of the deployment pipeline make it a very attractive target for an attacker: insertion of malicious code or deliberate misconfigurations that are automatically deployed and pushed to all servers can wreak havoc. Protection of the build pipeline including artifact repos, continuous integration and delivery servers, configuration management servers and all build, deploy and release tools is essential. The classic security measures like hardening, access control and privilege management (for both humans and machines (tokens, API keys)) as well as audit log generation should be applied to protect from unauthorized access and tampering with code and tools.

Reviews, scanning and tests to reduce the number of vulnerabilities

Vulnerabilities can stem both from design flaws in the system as well as implementation errors (bugs) and can result in data, systems or people being misused, exposed or harmed in some other way. Code reviews and extensive testing at multiple layers are the most common measures to prevent vulnerabilities. While we’d like to automate as much as possible in DevOps with automated code scanning, the human forms of pair programming and peer reviews (“pull requests”) are still of extreme importance in DevOps. The automated reviews and security tests need be lightweight, practical and fast and should be integrated into the overall test pyramid and deployment process. Typically, white-box / black-box and static / dynamic scans are combined and applied at different stages in the process.

Static security testing includes analysers and linters in the Integrated Development Environment (IDE), compiler warnings and dedicated code vulnerability scanners (SAST) that are able to detect insecure coding practices (missing input validation or potential buffer overflows), hard coded credentials or potentially other sensitive data.

Nowadays, 80-90% of a software project is third party code in the form of libraries and open source components. Therefore, code scanning needs to analyse the composition of software (SCA) and check for known vulnerabilities in third party software (including e.g. container images).

In the area of dynamic application testing there are a both well-known and new approaches that can be applied: Dynamic Application Scanners (DAST) run (black-box) tests again the live application to spot application vulnerabilities (XSS, CSRF, SQL injection etc.). Interactive Application Scanners (IAST) analyse the running code for vulnerabilities while the application is being tested. Run-time Application Self-Protection scanners (RASP) are also deployed alongside the code (like IAST), build a model of the running application and inspect the call stack and variables to identify and block vulnerabilities at run-time.
Corporate security teams most often have ample experience in network and host vulnerability scanners and are well-placed to help a DevOps-team to setup, configure and run SAST, DAST or RASP tools.

Protect data assets, configurations and secrets

The classification and sensitivity of customer, business and log data, and hence the necessary security and privacy measures, are highly application- and use-case specific and need to be thoroughly addressed as part of the SSDF. These measures are implemented with methods that are based on keys or other secrets that need to be protected similarly to application secrets like passwords, API tokens, configurations or certificates. None of these should be stored directly in the code base which usually is accessible by a larger audience than intended for the secrets. Keys and secrets should be easy to rotate and access or change should be logged in a tamper-proof way. Dedicated key management systems and vault solutions have evolved to cater the needs of a DevOps-environment.

Protect cloud assets

Although DevOps is basically orthogonal to anything cloud-related, the flexibility, agility, standardization and automation capabilities of the cloud, make DevOps and the cloud good bedfellows. This means that security in SecDevOps most often also implies cloud security [5]. From a DevOps perspective the setup and configuration of all cloud assets (compute, storage and network) is code, which improves security and compliance by means of repeatability, consistency and reliability.  Although in the cloud “everything is virtualized”, cloud-native developers shouldn’t forget that for software-defined infrastructure and networking most of the classic concepts such as network segmentation, DMZ or traffic filtering are still important and necessary.