Are you thinking about container security? Maybe you are on a security team trying to manage rogue cloud resources. Perhaps you work in the DevOps space and know what container security is, but you want to figure out how to decrease the pain around security triaging your containers for everyone involved.
In this post, we’ll look at security for containers in a scalable environment, how deployment to that environment can affect your rollout of container security, and how Docker can help.
What is container security?
Container security is knowing that a container image you run in your environment includes only the libraries, base image, and any custom bits you declare in your Dockerfile, and not malware or known vulnerabilities. (We’d also love to say no zero days, but such is the nature of the beast.)
You want to know that those libraries used to build your image and any base image behind it come from sources you expect — open source or otherwise — and are free from critical vulnerabilities, malware, and other surprises.
The base image is usually a common image (for example, Alpine Linux, Ubuntu, or BusyBox) that is a building block upon which other companies add their own image layers. Think of an image layer as a step in the install process. Whenever you take a base image and add new libraries or steps to it for creation, you are essentially creating a new image.
We’ve talked about the most immediate piece of container security, the image layers, but how is the image built and what is the source of those image layers?
Container image provenance
Here’s where container security gets tricky: the image build and source tracking process. You want assurances that your images, libraries, and any base images you depend on contain what you expect them to and not anything nefarious. So you should care about image provenance: where an image gets built, who builds it, and where it gets stored.
You should pay attention to any infrastructure or automation used to build your images, which typically means continuous integration (CI) tooling such as GitHub Actions, AWS CodeBuild, or CircleCI. You need to ensure any workloads running your image builds are on build environments with minimal access and potential attack surfaces. You need to consider who has access to your GitHub actions runners, for example. Do you need to create a VPN connection from your runner to your cloud account? If so, what are the security protections on that VPN connection? Consider the confidentiality and integrity of your image pipeline carefully.
To put it more directly: Managing container provenance in cloud workloads can make deployments easier, but it can also make it easier to deploy malware at scale if you aren’t careful. The nature of the cloud is that it adds complexity, not necessarily security.
Software Bill of Materials (SBOM) attestations can also help ensure that only what you want is inside your images. With an SBOM, you can review a list of all the libraries and dependencies used to build your image and ensure the versioning and content matches what you expect by viewing an SBOM attestation. Docker Engine provides this with docker sbom and Docker BuildKit provides it in versions newer than 0.11.
Other considerations with SBOM attestations include attestation provider trust and protection from man-in-the-middle attacks, such as replacing libraries in the image. Docker is working to create signed SBOM attestations for images to create strong assurances around SBOM to help strengthen this part of image security.
You also want to consider software composition analysis (SCA) against your images to ensure open source tooling and licenses are as expected. Docker Official Images, for example, have a certified seal of provenance behind them for your base image, which provides assurance around a base image you might be using.
Vulnerability and malware scanning
And what about potential CVEs and malware? How do you scan your images at scale for those issues?
A number of static scanning tools are available for CVE scanning, and some provide dynamic malware scanning. When researching tools in this space, consider what you use for your image repository, such as Docker Hub, Amazon Elastic Container Registry (ECR), Artifact Registry, or an on-premises/in-colocation option like Nexus. Depending on the dynamics and security controls you have in place on your registry, one tooling option might make more sense than another. For example, AWS ECR offers some static vulnerability scanning out of the box. Some other options bundle software composition analysis (SCA) scanning of images as well.
The trick is to find a tool with the right signal-to-noise mix for your team. For example, you might want static scanning but minimal false positives and the ability to create exclusions.
As with any static vulnerability scanning tool, the Common Vulnerability Scoring System (CVSS) score of a vulnerability is just a starting point. Only you and your team can determine the exploitability, possible risks, and attack surface of a particular vulnerability and whether those factors outweigh the potential effects of upgrading or changing an image deployed at scale in your environment.
In other words, a scanning tool might find some high or critical (per CVSS scoring) vulnerabilities in some of your images. Still, those vulnerabilities might not be exploitable because the affected images are only used internally inside a virtual private cloud (VPC) in your environment with no external access. But you’d want to ensure that the image stays internal and isn’t used for production. So guardrails, monitoring, and gating around the use of that image and it staying in internal workloads only is a must.
Finally, imagine an image that is pervasive and used across all your workloads. The effort to upgrade that image might take several sprint cycles for your engineering teams to safely deploy and require service downtime as you unravel the library dependencies. Regarding vulnerability rating for the two examples — an internal-only image and a pervasive image that is difficult to upgrade — you might want to lower the priority of the vulnerability in the former and slowly track progress toward remediating the latter.
Docker’s Security Team is intimately familiar with two of the biggest blockers security teams face: time and resources. Your team might not be able to triage and remediate all vulnerabilities across production, development, and staging environments, especially if your team is just starting its journey with container security. So start with what you can and must do something about: production images.
Production vs. non-production
Only container images that have gone through appropriate approval and automation workflows should be deployed in production. Like any mature CI/CD workflow, this means thorough testing in non-production environments, scanning before release to production, and monitoring and guardrails around images that are already live in production with things like cloud resource tagging, version control, and appropriate role-based access control around who can approve an image’s deployment to production.
At its root, this means that Security teams that have not previously had their feet in the infrastructure or DevOps team’s ocean of work in your company’s cloud accounts should. Just as DevOps culture has caused a shift for developers in handling infrastructure, scaling, and service decisions in the cloud, the same shift is happening in the security community with DevSecOps culture and Security Engineering. It is in the middle of this intersection where container security resides.
Not only does your tool choice matter in terms of best-fit for your environment’s landscape with container security, your ability to collaborate with your infrastructure, engineering, and DevOps teams matters even more for this work. To reiterate, to get a good handle on gating production deployments and having good automation and monitoring tied to those production deployments and resources, security teams must familiarize themselves with this space and get comfortable in this intersection. Good tooling can help make all the difference in fostering that culture of collaboration, especially for a security team new to this space.
Container security tools: What to look for
Like any well-thought-out tool selection, sometimes what matters most is not the number of bells and whistles a tool offers but the tool’s fit to your organization’s needs and gaps.
Avoid container security tools that promise to be the silver bullet. Instead, think of tools that will help your team conquer small challenges today and work to build on goals for the larger challenges down the road. (Security folks know that any tool on the market promising to be a silver bullet is just selling something and isn’t a reality with the ever-changing threat landscape.)
In short, tools for container security should enable your workflow and build trust and facilitate cross-team collaboration from Engineering to Security to DevOps, not tools that become a landscape of noise and overwhelming visuals for your engineers. And here’s where Docker Scout can help.
Docker Scout
Docker engineers have been working on a new product to help increase container security: Docker Scout. Scout gives you the list of discovered vulnerabilities in your container images and offers guidance for remediation in an iterative small-improvements style. You can compare your scores from one deployment to the next and show improvement to create a sense of accomplishment for your teams, not an overwhelming bombardment of vulnerabilities and risk that seems insurmountable.
Docker Scout lets you set target goals for your images and markers for iterative improvement. You can define different goals for production images versus development or staging images so that each environment gets the level of security it needs.
Conclusion
As with most security problems, there is no silver bullet with container security. The technical, operational, and organizational moving pieces that go into protecting your company’s container images often reside at the boundaries between teams, functions, and responsibilities. This adds complexity to an already complex problem. Rather than further adding to the burdens created by this complexity, you should look for tools that enable your teams to work together and reach a deeper understanding of where goals, risks, and priorities overlap and coexist.
Even more importantly, look for container security solutions that are clear about what they can offer you and extend help in areas where they do not have offerings.
Whether you are a security team member new to the ocean of DevOps and container security or have been in these security waters for a while, Docker is here to help support you and get to more stable waters. We are beside you in this ocean and trying to make the space better for ourselves, our customers, and developers who use Docker all over the world.
Learn more
- Get the latest release of Docker Desktop.
- Try Docker Scout.
- Learn about Docker Security.
- Generate the SBOM for Docker images.
- Learn about SBOM attestations.
- Check out Docker Official Images.
- Visit Docker Hub.