This post is part of a series on DevSecOps and CI/CD security. Check out the overview for context and links to the rest of the series.
This post focuses on DevSecOps technical strategy. A healthy DevSecOps culture emphasizes:
- Tight collaboration and continuous feedback between security and engineering teams
- Heavy automation using infrastructure-as-code and security-as-code tools
- Robust testing of applications and infrastructure, including layered security tests
Technical Strategy
Apply these principles to foster a healthy DevSecOps technical culture.
Invest in security engineering: provide engineering teams with secure-by-default platforms and automated “guardrails”. The goal is to make it easier for engineers without security expertise to build secure software. For example:
- Provide hardened defaults with explanatory comments or documentation. This can take the form of infrastructure-as-code, reference architectures, libraries, configuration files, and reusable code snippets. Prioritize building these artifacts based on the popularity of tech stacks in your organization.
- Translate security requirements into executable tests.
- Build base images for virtual machines and containers with hardened operating system defaults. Packer and Ansible help here.
- Ease developer security burden with tools to meet security requirements, such as HashiCorp Vault for secrets management and encryption-as-a-service. Another example is Terraform templates for authenticating or authorizing users with AWS Cognito, Azure AD, or Okta.
Meet timing expectations: slow or buggy tools in the daily flow of development get annoying quickly, and engineers often abandon them. Particular actions need to finish in specific timeframes, so make sure security tools execute quickly enough for these expectations. For example:
- Committing code to a local repository should be near-instantaneous. Commits that take longer than a couple seconds are painful. This discourages frequent, small, closely-related commits commits (an important best practice). Pre-commit hooks provide opportunities for validating code against secure coding standards, quick-running security tests, and keeping secrets out of version control.
- Code merges into a shared branch should take seconds or (max) a few minutes – a good place for deeper static analysis that’s tuned to the codebase, or a small-to-medium-sized security test suite.
- Running a full set of integration or deployment tests in a production-like environment may happen overnight, on a weekend, or before a release. These types of events are opportunities for long-running security activities like penetration testing, DAST, or network vulnerability scanning.
Break the build judiciously: you can break a build or deployment in the CI/CD pipeline when tools discover a security defect. However, breaking the build too often (especially on false positives) will make engineers resent and ignore noisy tools. Break the build for verified critical or high-severity issues, and continuously tune the breaking conditions to suit your risk tolerance and tech stack. Automated security tools are prone to false positives, and low-severity or informational true positives don’t always warrant breaking the build.
Track CWE data: use security tools that categorize each identified issue under one or more CWEs. This provides visibility into recurring issue types such as API/function errors, authorization problems, or secrets management. This data can then be used to guide investment and prioritization for tools, training, project selection, etc.
Use and contribute to open-source: there are many open-source, automation-friendly tools available to improve application and infrastructure security. Make good use of this ecosystem! Where you have a problem that’s not covered by an existing tool, write your own or fork an existing one. Consider open-sourcing and publicizing your security tools to build customer trust in your security program (e.g. Netflix’s Security Monkey or Square’s Keywhiz). Open-sourcing your security tools also builds your organization’s credibility among engineers and “lifts the boat” of the whole industry.
Filter false positives: the security team should act as an informational filter for engineering teams, removing false positives from tool output and prioritizing discovered issues. At best, providing a long list of unverified, unprioritized security issues risks missing the needle in the haystack. More likely, this wastes engineering time and development bypasses security to meet the release schedule.
Expose useful metrics: security teams collect lots of data – server/application/cloud provider API logs, NetFlow data, vulnerability scanner output, etc. Collaborate with engineering and other stakeholders to use this data effectively. For example: aggregate it into dashboards/graphs/charts that track OKR-related metrics, or trigger notifications for time-sensitive issues. This provides security value, increases efficiency across teams, and builds political capital.
Contextualize security issues: provide engineers with contextualized security information, especially through integrations with CI, version control, and issue trackers. Include issue priority, realistic descriptions of exploitation likelihood and impact, and remediation guidance tailored to your tech stack and environment. Make sure they’re assigned to the right people for remediation.
Thanks for reading! If you enjoyed this, check out the next post on SAST and code quality.