Accelerating the Secure Software Delivery Lifecycle with GitOps

More than ever, businesses must be able to respond faster to intense competitive pressure, increase operational efficiency, and adapt to constant disruption. One key to accomplishing this is enabling shorter and shorter software delivery lifecycles – that don’t sacrifice reliability, security, or compliance. This is the goal of the integrated DevSecOps strategy. However, today’s DevSecOps implementations require that developers understand and participate in setting up the delivery pipeline and are vigilant when it comes to security issues – otherwise the risks associated with cybercrime and regulatory compliance failures can soar.

GitOps offers a solution. By integrating GitOps into the DevSecOps workflow, businesses can leverage the benefits of Git to dramatically simplify the software delivery workflow, including for production releases, while also automating pipeline setup in the background and handling off security and compliance tasks to the respective teams.

The goal of GitOps is to enable software delivery and infrastructure configuration with Git as the source of truth for state-of-service delivery. GitOps also enables software delivery operations with familiar workflows and tools for developers, enabling DevOps processes with little or no additional friction.

Enabling enterprise security teams to specify security policies that apply to the software delivery process via the GitOps delivery model vastly improves the security of application delivery. For example, the security team could require applications deployed to a specific environment to perform dynamic application security tests (DAST) and ensure the results of the checks conform to expectations.

Specifying such security policies as code acts not only as documentation of security policies but also helps in the automation of compliance checks for these policies. In addition, security policy as code, managed by security teams independent of the software development processes and integrated with the tools that developers use, significantly improves the security of the delivered software.

This separation is critical for developing a zero-trust environment around software releases. It can also accelerate the delivery of new software by surfacing security issues as they arise, eliminating the time and complexity of separate security reviews.

DevSecOps – A solid foundation

Despite the increasing risks associated with the software delivery lifecycle, most organizations struggle to get their operations, product development, and security teams to collaborate in a way that improves security without adding cumbersome process steps that end up slowing down the software delivery lifecycle. DevSecOps builds this collaboration into the software delivery lifecycle based on the following principles:

• Providing visibility into security issues throughout the software delivery workflow
  The security team, developers, and project managers should all have visibility into the results of a comprehensive list of security tests, including static application security test (SAST), dynamic application security test (DAST), fuzzy testing, dependency scanning, binary scanning, license compliance, and secret detection.





• Identifying security issues as early as possible in the software delivery lifecycle
  The sooner risks and vulnerabilities can be surfaced, the faster and more easily they can be remediated. Automation should be used to eliminate the time-consuming manual review of the large number of logs and metrics created by the build, test, deploy, and production stages.





• Enabling the automatic enforcement of security policies throughout the workflow
  Application security includes ensuring that production software complies with evolving regulatory requirements, especially those related to data privacy, and reducing the risk of breaches. It also includes ensuring that a release complies with internal policies, such as all the review steps required before deployment.

The currently popular Pipeline Model for DevSecOps adheres to these principles. Both AWS and Google Cloud have best practices outlined for this approach. Using an orchestrator tool, such as Spinnaker, changes in Git or other source repositories trigger an automated workflow (a “push model”) that goes through the required steps for deploying to a target environment. These are typically represented as directed acyclic graphs with requirements for running a stage/step in the workflow and dependent stages/steps to run based on the status or output of the stage. This model allows for the required visibility into all the steps for delivery by all the stakeholders (dev, ops, security, SRE, etc.). And it encodes the rules/requirements as guardrails or gates – which are also exposed to all stakeholders in the delivery ecosystem.

However, this pipeline model puts a significant burden on the developers to become familiar with the entire process, including how to use and configure the orchestrator. This means it requires a demanding onboarding process for developers to be able to use the system.

GitOps Model for Delivery

The GitOps Model allows developers to simply commit their code in Git and be done. They don’t have to run or track anything using the new tools in the delivery process. They don’t have to understand and use the orchestrator. Instead, SecOps is able to put the necessary visibility, risk identification, and rule enforcement into Git, making Git a single source for all the necessary process controls, which run in the background, invisible to the developers.

The GitOps Model depends on Kubernetes-native capabilities. Kubernetes supports a declarative model for running applications, where the configuration required for the application services can be specified in configuration files as declarative specifications. The specifications can then be applied through the Kubernetes API, and Kubernetes deduces the operations that should be performed to attain the state declared in the configuration files and execute them. This model allows for the detection of changes between the expected state in Git and the actual running state (drift detection), enabling corrective actions.

Kubernetes has an extension capability that extends its base functionality, allowing for additional checks as part of the verification of the deployment requirements. The Kubernetes Admission Control (KAC) mechanism plays a key role in allowing GitOps simplicity for developers and providing the required reliability and security guarantees for the delivery.

KAC also provides the critical ability to separate enterprise policies for process requirements from the development process, allowing policy changes to be made independently of the development process. As an example, to ensure alignment with the SOX compliance requirement of ensuring more than one individual is involved in the verification of changes to production, the policy at admission control can verify that the Git PR review and QA approval are by two different individuals before allowing deployments.

Taking advantage of these capabilities, the GitOps Model allows for:

The huge advantage of this model for developers is that they need to take only one step to participate in the DevSecOps workflow: look for changes and reconcile them in the target environment. There is no need to perform security checks, conduct testing, set policies, or seek approvals as separate steps in their DevOps processes.

Instead, in the GitOps model, the DevOps tools can be required to perform their actions asynchronously in place of sequential orchestration steps and then, at deployment time, verify that the required steps have been performed according to the specified policy.

For example, a code check-in performs continuous integration (CI) unit tests, allowing developers to have faster dev/test cycles. The binary scans, dynamic scans, and other policy checks are performed asynchronously to the developer workflow. Test cases are run automatically, or manually for integration tests, and results are published to the same or another central repository. The results stored in the central repository are associated with a signed binary artifact that can be used for verification. This is typically done using the output of CI – for example, a container that can be signed. For a production deployment, the Git configuration is updated with the new version manifests.

When the configuration for a new version is updated in the Git repository, the deployment will trigger to the Kubernetes namespace. The Configuration of Admission Controller will then trigger the checks with the signed metadata for the container for allowing the deployment. These checks will determine whether the static scans, dynamic scans, compliance requirements like SOX, and test results are acceptable for deployment by approving or denying the deployment and sending notifications to users on the reasons for any failure.

The Open Policy Agent (OPA) Gateway and Kyverno are examples of frameworks for implementing admission control checks for deployments. They can be configured as an extension of Kubernetes and include an open policy agent or Kyverno that allows verification of security policies. When a new application or a Git change is applied, the extension recognizes the need for a new deployment and runs security policy checks to determine if all the specified security and compliance requirements have been met. It can then approve the deployment or stop it.

While these are great frameworks for aiding the GitOps Model, they do not yet support the following:

• The ability to centrally audit the checks required for delivery across various tools working asynchronously.
• An interface for modifying and viewing policies and applying them to different types of applications and Kubernetes clusters.
• Visibility into applications running on various Kubernetes clusters and the status of security, resolution requirements, etc.

What this means is that although the GitOps Model has the potential to make life simpler for developers, there is a need for additional tooling to support real-time visibility and collaboration capabilities.

In enterprise environments, the security team should be able to view the compliance status of all applications including exceptions granted for specific applications. The system should support the ability to inform developer groups of violations and collaborate with the security team on the resolution of an issue, for example, a new vulnerability identified in a deployed application. These capabilities are critical for ensuring security without compromising the speed of software delivery.

Speed ​​in software delivery workflows is essential. But speed without security and compliance is reckless. The GitOps Model for DevSecOps offers the best of all worlds. It can free developers to focus on their projects without having to understand security issues and learn to use an orchestrator tool in the quick dev/test cycles – while also allowing them to continue working with their tool of choice. At the same time, it can enable SecOps to add the required guardrails to the software delivery workflow and surface risks as they emerge, so they can be resolved faster and more easily – ensuring software releases are safer and more reliable. No, we aren’t quite there yet, but over the next year or two, expect to see a crop of solutions hitting the market that will make true GitOps a reality.