Touchpoints: Software Supply Chain

This post is part of a series on DevSecOps and CI/CD security. Check out the overview for context and links to the rest of the series.

Modern software is made by writing some custom code and re-using a large amount of existing code. Taking dependencies on existing code enables rapid, flexible development – it also creates many opportunities for supply chain attacks. Unfortunately, these are quite common, and often are highly impactful: Log4j, MeDoc, and the SolarWinds ORION attack are prime examples.

This post explores securing the software supply chain:

• Software Supply Chain Context
• Dependency Risks
• Patching Best Practices
• Dependency Management Tools
  • Operating System Updates
  • Application Updates

Software Supply Chain Context

Modern software weaves together custom code with existing components such as:

• APIs and SDKs
• Web servers
• Cloud provider services
• Operating systems (containers, virtual machines, bare-metal hosts, …)
• Infrastructure-as-code
• Libraries and frameworks
• Middleware
• Databases and ORMs
• Message queues/brokers

That’s surely an incomplete, high-abstraction list. The volume and variety of third-party software that modern applications rely on is enormous; applications have tens, hundreds, even thousands of dependencies!

The set of software components your application depends on is its software supply chain. Each link in this chain increases your attack surface.

Software composition analysis (SCA) is the area of application security that deals with the software supply chain. Using vulnerable or outdated components is in the OWASP top ten; understanding your software supply chain and keeping it updated with current security patches goes a long ways towards reducing risk.

Dependency Risks

Each application dependency adds risk, such as:

• Dependency vulnerabilities: security issues in the dependency’s code
• Malicious dependency author: a bad actor distributes malicious code by gaining control over a dependency’s source code or community package
• Compromised dependency author: a bad actor attacks the dependency’s legitimate author/maintainer (such as phishing attacks against PyPi package maintainers) and distributes malicious code with their privileges
• Dependency typosquatting: a bad actor publishes a package to a community repository that’s a character or two off from a popular package’s name. This causes developers to accidentally install the wrong version through a typo. This has happened with npm, PyPi, and RubyGems (and probably in other package ecosystems as well).
• Dependency confusion: package management tools can install packages from public or private registries, and typically install the highest version of a package by default. An attacker can trick a package management tool into running malicious code by creating a package namespace collision; guess the names of internal packages (or find them through a leak), and publish an identically-named package to a public registry (NPM, PyPi, RubyGems, etc.). The attacker uses a higher version number than the internal registry, resulting in a compromise the next time that package is installed.

Patching Best Practices

Here are some best practices for keeping your application patched.

Practice dependency minimalism: each dependency increases attack surface and maintenance burden, so apply YAGNI (you aren’t gonna need it) to your dependencies. Be strict about the criteria for taking new dependencies, and regularly trim them to reduce attack surface. It’s better to have a small number of well-understood, well-tested dependencies than a huge tree of dubious-quality dependencies.

Offload patching to third parties: at design time, delegate patching responsibility by using cloud services such as AWS Lambda, RDS or S3. Doing this gives you compute and storage resources without the usual patching burden.

Use immutable infrastructure: for applications with robust deployment automation, apply patches by destroying and re-deploying from infrastructure-as-code rather than patching running systems. Immutable infrastructure makes environments more consistent and testable, improves your ability to reason about your environment, and reduces “configuration drift” from engineers tweaking running systems.

Allocate time for patching: researching, testing, and applying patches takes engineering time. Budget time for security updates.

Automate patching and testing to improve consistency and free up engineers for higher-leverage activities. When applying patches, test them in a CI/CD pipeline to find application and infrastructure breakage automatically. Serverspec is useful here.

Subscribe to patch notifications: keep an eye on notifications from package repositories, GitHub releases, mailing lists, or whatever update channels your dependencies use.

Verify update authenticity using GPG keys, SHA checksums, or other signing mechanisms offered by the update channel.

Download updates over encrypted protocols like HTTPS or SSH/SFTP.

Use an artifact repository: for most package ecosystems you can self-host a package repository rather than relying on public ones. This is a function of the artifact manager (for more on this see the CI/CD background post). Some benefits of using an artifact repository:

• Control dependency mirroring: you get faster software build times, less broken builds when public registries or internet connections go down, and protection against upstream package maintainers deleting or moving code.
• Understand and control your dependencies: you get an inventory for (aspects of) your software supply chain. This enables better decisions about dependencies considering factors like CVEs, code quality, and software licenses.

Dependency Management Tools

Software updates come through many channels, such as:

• Windows Update and WSUS
• Linux distribution packages: apt/yum/apk/etc. repositories
• Package managers: pip for Python, npm for Node.js, bundle or gem for Ruby, …
• Container registries: Docker Hub, Quay, AWS ECR, …

Here are some places in a CI/CD pipeline that patching is especially important:

It’s a complex landscape; use tools to tame the chaos!

Operating System Updates

At the operating system level, rebuild base images regularly with the latest patches from Microsoft or your Linux distribution. Packer and Ansible are excellent tools for this. They can be combined using Packer’s Ansible provisioner (Packer also has provisioners for shell scripts other IaC tools).

Roboxes offers a model in this space, packaging many Linux and BSD-based operating systems for a variety of hypervisors and platforms using Packer templates.

For containers:

• Regularly pull fresh images from upstream registries, and rebuild images from their Dockerfile in a CI/CD pipeline
• Use a scanner such as Clair or Trivy to flag vulnerable or outdated container components
• Execute tests in CI to ensure that new images behave in the ways you expect them to

Application Updates

Developers typically manage application dependencies using paired package manager tools + community repositories. This pattern applies across many programming languages and software platforms, such as:

• docker pull from Docker Hub in the container/Docker ecosystem
• npm install from NPM in the JavaScript ecosystem
• pip install from PyPi in the Python ecosystem
• gem install/bundle install from RubyGems in the Ruby ecosystem

To keep application dependencies up-to-date, consider using:

• Dependabot to monitor your project’s dependencies for updates and automatically create pull requests to patch them.
• OWASP Dependency-Check to discover packages with known vulnerabilities across a range of languages and tech stacks.
• Package manager built-in mechanisms to list and update outdated dependencies. For example:
  • pip list --outdated and pip install package-name --upgrade
  • npm oudated/npm audit and npm update package-name

Thanks for reading! If you like what you read, check out the overview for links to the rest of the series.