On-Demand Training

Container Security and Docker Scout

 

Transcript

This is Docker Container Security and Scout. In this presentation, we’ll be talking about how to secure images and containers, including production images and containers. Changing image dependencies, configuration, and or run-top options can affect how your application operates. Please thoroughly test your changes before deployment to production.

 

Table of Contents

 

Container Security (0:23)

In this section, we’ll be talking about container security. We’re going to be focusing on Linux containers and how to secure Linux processes as well as how to create the most secure images. The first thing we’ll talk about is root versus non-root users. Root is the default user for a container. You can change that to a non-root user and you should when you’re running a container, especially in a production environment. You can do this several different ways. You can do this in the Dockerfile and the Docker run command or in Docker Compose. You can see the first option here, we’re adding a user – myuser – within the Dockerfile. This means that when the Dockerfile executes, that root user will change to that myuser you’ve specified when it runs that particular container. You can also do the same thing here in the run command. You can see the Docker run -user, and we see the group ID and the user ID specified within it. Or, in our bottom example, we can see the Compose file. Within this Compose file, we can see the user being specified with the user ID from an environment variable being plugged in. Down at the bottom here is a blog that goes into much more detail about the user instruction and the different ways of using it, which we’d highly recommend you take a look at.

To give one more example to show exactly all the types of things you can do with the user command, here’s how Docker init generates the add user command. You can see here there’s a lot of different options that are being added onto this. We’re disabling password. We’re saying there’s no home. We’re saying there’s no login. No shell. No create home. The user ID we want to specify for this, these are all options that you can add into the add user command if you want. They’re not required, but they certainly can help better secure the image and container by doing it this way. The next piece is read only containers and Linux capabilities. By default, when you run a container, you can still write to that container’s file system while it’s running. You can go in and make modifications to it. Your application can make modifications to it while it’s running. In a production environment, you may not want that. You may want your environment to be immutable in production, which would make sense. To do this, what we’re doing is giving you the option to do a read only capability on it. When you make the image read only, that means you cannot write to it. The issue of this, of course, is there are things the application needs to write while it’s operating. So things, for instance, to the temp directory. You might need to be able to write to that. And so the read only flag gives you the capability of doing that through what’s called tempfs. So what tempfs is doing is allowing you to write to memory instead of writing to the file system. What this means is that when that container stops, that tempfs memory area goes away. It’s not saved at a particular point in time. But if you’re just writing things out temporarily, that’s not an issue. So by default, with the read only flag, the temp directory is covered by tempfs. But what if you had another thing that you wanted to have to write to in that particular directory structure as well? Well, you can use tempfs yourself. You can go to the link listed below here for tempfs and you can specify other directories to use tempfs as well. Now please keep in mind, if you’re using memory to be able to store things, you need to keep in mind your memory constraints. If you run out of memory, then you’re going to run out of space to actually write stuff. The other thing on the slide is Linux capabilities. Because the container is a Linux process, it has Linux capabilities just like any Linux process does. If you see over here on the right, these are the capabilities that are assigned by default. Some of these capabilities may not be things we want this process to be able to go execute in a production environment such as kill or set group ID or set user ID, those types of things. So you can go in and decide if you want to remove or add capabilities. So you can see on the left hand side, I have this –cap-drop=ALL. What this is doing is telling you to drop all the Linux capabilities because I don’t want to be able to do any of those in my production environment. There’s also a whole bunch of capabilities beyond these that you can add if you want. And you can choose individual ones to remove or to add as you see fit. The second link in here has the list of all the different capabilities and you can decide which ones you want to add or remove.

 

Secrets and Containers (5:02)

Alright, let’s talk a little bit about secrets and containers. Containers often need access to secrets to be able to run. It could be things like usernames or passwords, it could be things like TLS certificates and keys or SSH keys or just other important data such as the name or address of a database or other internal server. So if we need access to these secrets, where are we going to get those secrets from? Well, we’ve got a number of different options here. Several bad options, one okay option and one good option. We could locate the secrets in the source code, which means anybody with access to the source code can see those. And if you need to change the secret, you’re going to need to rebuild your application. You could build them straight into the image, which means again, it’s going to be visible to anybody with access to the image. And if you need to change the secret, you’re going to have to rebuild the image. You could put it into an execution script to back up to source control. Now you don’t have to rebuild the image if you change it, but anyone with access to source control can still see it. You could put it into an environment variable. This is often shown as a way of doing secrets and that kind of thing. But if you put an environment variable, if you do a log dump, for instance, this is into a vendor, it’s going to show up in there as an environment variable. And it’s available to all processes on that machine. So that’s really not a great option either. You could put it into a file so that the process knows where to go get the secret, but it’s available on that production machine. This is an okay option. So the process is the only one who knows where to go get the secrets. But anybody with access to the machine can go find that secret. And finally, we have a secrets vault. So this would be a secrets management application. And only the process that is asking for the secret can actually access it from that vault. This is the recommended way of handling a secret in a highly secure environment.

All right, so we’ve talked a little bit about how to run that particular process. Now let’s talk a little bit about building that image. What do we need to consider when building an image for a production or secure type environment? Well, the first thing is we want to limit what’s in that image. So, we want to limit the dev tools and production image. We only want to have what’s needed to run the application in the production environment. That means we don’t need any source code. We don’t need an IDE. We don’t need a compiler. We don’t need debuggers. We don’t need non-deployable build artifacts. Basically, we don’t need any of the tooling that developers need in a production environment. We want that production image to only have what’s required to run that application. The same thing goes for OS tools. If we’re saying that the container should only have what’s needed to run the application, that means we don’t need package installers or editors or network tools like Curl or even things like Ping or especially things like Sudo.

We don’t need any of that kind of stuff if we’re just running our application, which is what our production image is supposed to be doing. How far can we go with this? Well, we can keep producing the base image even further. We can go to certain distroless base images that have very little in them or you can go all the way down to what’s called the scratch image. Scratch image has barely anything in it whatsoever and you can start from there. We’ve been talking about removing things from the base image so that the application is smaller. We’re saying this is great, but we need to understand at some point we can start cutting things that we actually might need to run the application. So if you’re going all the way down to something like scratch, you need to understand all the underlying technologies that your application needs. For example, if you go for scratch, you may find that the network library doesn’t support something like DNS over TCP and you might realize that your Kubernetes cluster requires that for your application to be able to run there. These are the types of considerations you have to have the further you try to strip down a base image to the most minimal state it can be.

 

Trusted Base Images (9:11)

All right, so we’ve talked about stripping down the base image. Let’s assume that we’ve gone with Alpine as our base image. The next question is, where do I get that Alpine from? Who built that and how do I know how they did it? So I could go with community images, something like super elite Alpine here, but I don’t know who built it and I don’t know why they built it and I don’t know if it’s secure or not. A better option would be to go with something like Docker official images, like shown at the bottom here, here’s the Alpine with the tag on it to say it’s a Docker official image. That means that either Docker built it itself or Docker worked directly with the community and oversaw the building of it. So we know exactly what’s in it. Docker verified publisher images are ones from our partners like Red Hat, Suse, Grafana, and so on, where they stand behind the provenance of those. And then we have Hardened images. Hardened images are ones that have CVEs removed and have SLAs around them. That’s another option that’s available as well.

 

Docker Scout (10:17)

All right, so let’s move now from the securing individual images and start talking about Docker Scout and what it can provide in this to provide a trusted software supply chain. All right, so let’s look at the development process. So as a developer, I’m going to be the producer in this case. I’m going to try to follow best practices as I’m writing my application. I’ve got my code, my source code, that I’m going to write. I’m going to have different dependencies I’m going to pull in. There’s going to be different libraries and other things that I need for my application to work correctly. I’m going to build those using the Dockerfile. There’s going to be some kind of vulnerability scan at that point. There’s going to be package or image that’s created. And then it’s going to go out to the consumer. Now in this process, I’m going to try to follow best practices. I’m going to try to make sure that the code I’m writing is secure. I’m going to try to make sure that the dependencies that I’m getting are the right ones that seem to be popular and secure. I’m going to try to follow the build best practices. So I as the developer feel pretty good. I’ve tried to do the common sense things to make sure this is a secure code that’s going out. So I feel like I’ve done my job pretty well.

So now if I’m the consumer of this process, and if everything for this is a closed box, and I don’t know what happened, I suddenly have all kinds of questions. Who built this? How do I know that source code wasn’t compromised or manipulated? Or how do I know the dependencies are what you say they are? Or how do I know that the build wasn’t tampered with or altered or the package was compromised? Or if this is just all old and now there’s a bunch of vulnerabilities against it? So these are key questions that consumers of software are asking because they really want to understand what they’re actually getting and make sure it’s going to be secure for their environment. So we need tools to help to define and solve these questions.

All right, so there’s a number of tools available to do this. The first one is the software bill of materials (SBOM). This is going to tell us what’s in this software artifact. So it’s going to tell us all the different packages that are in it. The next one is Provenance. Provenance is an attestation about the history of an artifact, where it came from, who produced it, how they produced it. And finally, we have signing of the package. This is an attestation that verifies the source and the trustworthiness of it, so that the consumer knows what they’re getting is what we say it actually is and not some kind of substitution.

 

Software Bill of Materials (12:50)

All right, so let’s talk about software bill of materials in a little bit more detail. So we have a food nutrition and ingredients list. This is required for all food things and it’s very helpful. It gives us information about the calories and cholesterol and all that kind of stuff. It also gives us the list of ingredients in descending order. So we can see what things are in here and we can see how much of each one relative to the next item is in there. We also at the bottom have things about whether it contains certain allergens, which is important to people who have allergies. This gives us a lot of information about these cookies and what’s in them. So we can make some informed decisions about whether we’re going to eat those cookies or not. So what if we can do the same thing for software, but what if we can actually do better than this? With a software bill of materials, we can not only provide the list of ingredients, which would be the packages, but we can actually give information about each one of those packages, who the supplier is, what the version is, what the dependencies are between them. Who’s the author of this for their trustworthiness. All of that information can be included within the software bill of materials. So it’s not only going to give us the cookie level information, it’s going to give us much more than that, we can make better decisions about how we trust this particular artifact that has the software build materials associated to it.

There are different formats for software bill of materials. There is the software package data exchange, and a cyclone DX. Scout supports both those different formats. There’s also what’s known as a vulnerability exploitability exchange or VEX document that can optionally be added. What the VEX document is doing is it’s allowing you to be able to say this particular package has this particular vulnerability, but this vulnerability cannot be exploited because it’s not actually executing that part of the code for whatever reason. And so that gives you the ability to understand that yes, there is this vulnerability, but it’s not something you need to worry about. So that’s down on the noisiness of vulnerability concerns and that kind of thing.

 

Docker Scout (15:01)

With Docker Scout we’re offering a number of different things. There’s trusted content within Docker Hub for people to pull down base images and other images they can use. We are doing policy evaluation all the way through the SDLC from the source through the build package so everybody can see what’s going on and providing a system of record around this entire process. We all understand what’s going on with images regardless of where they are in this process. Scout provides actionable insights across the entire supply chain. So with Docker Scout, you can ensure the quality of software and make sure it’s trusted by your customers and that complies with the guardrails or policies that you’re implementing.

So what is Scout actually doing? Let’s go into some details here. The first thing it’s doing is actually pulling together a bunch of different sources of vulnerability information. Now, when people talk about vulnerabilities, they often talk about the national vulnerability database. This is the place that many people go to get vulnerabilities. What we found though is that many of these other databases or catalogs of vulnerabilities actually give more detail about the vulnerabilities than the vulnerabilities database. What we’re doing is we’re pulling all these different sources up to three times an hour and we’re then pulling all those together into a single database so that you can have the best information across all these different sources to use to actually understand what vulnerabilities you have in your software.

Now, the next piece of this is finding and fixing vulnerabilities sooner. We’re seeing that people are finding vulnerabilities in their CI process or in their testing process or unfortunately in production. It would be much better to find vulnerabilities earlier in the process where possible. So we want developers to be able to understand what vulnerabilities are in the software they’re working with and to get information about how to remediate those. This is incredibly important to us as Docker to make sure that developers can see this information and do so in such a way that it’s not an entirely different process or entirely different way of seeing it, but it’s a natural part of how they’re working with the software today. We’re not just talking about vulnerabilities though. We’re also talking about policy. We want to be able to set policies and have the developers, the CI, the entire SDLC comply with those policies. Now, it could be things like ‘no critical vulnerabilities that could be fixed,’ but also could be ‘are my base images up to date’ or it could be ‘am I using specific GPL licenses that I don’t want to be using’ or ‘am I running as a root user when I should not be doing it.’ So policies could be on a number of different things that go beyond vulnerabilities.

So the next thing is looking at the CI process itself. So if we have a CI process and we have a different way of evaluating vulnerabilities than what the developers are doing, then we could have issues with trying to understand what vulnerabilities and how they’re being evaluated and how they’re being fixed. We’re providing the ability for Docker Scout to be integrated directly into the CI process. This way you have the same evaluation process going on in both development and your CI so that everybody agrees about what a vulnerability is and whether or not it has a VEX statement or any of those types of things.

All right, so now we want to evaluate the vulnerability status. We need to understand what it is across our SDLC. So we’re going to gather that information looking at what the developers are doing, looking at what’s checked into the different repositories and actually be able to then understand it through the Scout dashboard. So we can then look at all the metadata across all of our images in a real time format and actually understand what’s happening here and be able to then prioritize the action that needs to be taken about. The important part to note here with the SDLC and vulnerabilities is you’re never going to be at zero vulnerabilities all the time. There’s always going to be new vulnerabilities going in, and so having a real time view as to what’s going on with vulnerabilities is going to be important across all your different environments.

We also want to be able to understand what’s going on in production. With Docker Scout you can go in and actually look at the images that have been tagged to production and see what’s happening in your production environment. You can also tie in runtime analysis of your production environment and be able to see what’s happening with that as well. So you can get better understanding of what’s actually executing in your production environment and see what vulnerabilities or policy issues there might be with that. Docker Scout integrates with a number of different tools, whether it be Docker Desktop and the command line for development, whether it be a number of different CI/CD tools, whether it be a number of registries, including artifactory, ECR and ACR or runtime tools like sysdig, we’re constantly adding new capabilities to these integrations over time.

 

Demo (16.18)

With that, it’s time to show some of these capabilities I’ve been talking about. So we’re going to use the Scout demo service. You can see the URL here. You can download that same GitHub repository and do the same demo I’m about to do yourself using that GitHub repo. I’m now going to switch over to VS code. Here I have my Scout demo service and I’m going to start off just by building this particular image. And now it’s going to build this particular image. It’s going to take about 20 seconds to do so. And then once we build this image, we’re going to start looking at what Scout can offer us. All right, so we built this particular image. You’ll notice here we got a tip right away: Hey, you want to see image vulnerability recommendations, do this Docker Scout quick view. I do want to do that. So I’m going to go ahead and copy that, paste it in there.

All right, so let’s see what happened here. We generated a software bill of materials for it. This software bill of materials was cached from a previous example, but it would generate software bill materials for it. It would then evaluate the policies against this particular image. And then from here, it’s going to give us information about vulnerabilities. Wow, it looks like I’ve got two criticals, 20 highs, 8 mediums, 4 lows. I really want to try to take care of these criticals and highs if I can. And then if I go down to policy, I’m going to see my policy status is failed. So it failed for not only fixable critical and high vulnerabilities, which I need to go deal with, but I also have no default non-root user in my Dockerfile. So I need to go fix that and I’m missing some supply chain out attestations, which I can’t really fix as a developer, but it’s important for me to understand what’s going on with that.

So from here, I can do a number of things. Again, it’s giving me tips as to what I can do. I can go look at my policy violations. I can look at my vulnerabilities. I’ll go ahead and do that. So I’m going to go ahead and do that right here. I’m going to get the list of CVEs by package. So I can come in here and actually see for each package what CVEs are in it. And for each CVE, I’m going to get the CVE number. I’m going to get the URL to actually go look at details for that particular CVE. I’m going to get an affected range and a fixed range. I actually know what I need to change to actually fix each one of these. This is giving me all that information that I’m looking for. I can also do a Docker Scout SBOM on Scout demo. And this is going to give me a readable format of the SBOM. This is going to show us what that software bill of material looks like. I can see in here the type, the name, the version of the package, the author of the package, the description, the licenses of it, the URL to go to it, and then the path to actually get to this particular piece of it. So this is the type of information you’re going to see within a software build materials that’s going to help us understand whether or not these packages are things that we want to include in there and we feel comfortable including or not.

Everything I’ve shown so far has all been through the command line and you can do all of this work from the command line just using it the way I’ve shown. But I’m also going to show the GUI and show what we can do with that. Here I am in Docker Desktop GUI. I can actually now go to my Scout demo image and I can either go over here and say view packages and CVEs, or I could just simply click on it and it’s going to give me the same thing, which now it’s going to show me the same vulnerability information as I saw before, but it’s going to show it to me in a layer by layer format. So I can see a bunch of vulnerabilities are in the base image, but if I come down here, I know, oh, wait a minute, there’s some vulnerabilities that I introduced as part of my build that I need to take care of. And so I can look at these and try to see what’s going on with these. So if I go to my express because I know how express in this particular application, I’m going to see there’s multiple vulnerabilities in here. So I’m going to go to the first one, it’s going to give me a description and it’s going to tell me, OK, this was fixed in 4.17.3. OK, so I need to update to at least 4.17.3. Let me go down to the next one and see what it says. This one says 4.19.2. All right, so I’ll do that one. And then there’s another one here. And this one says 4.20.0. Now this one was just for a low, but I know that 4.20.0 is a better version for me to be on. So I’m going to go ahead and just update that to 4.20.0. So let’s do that now. Go back to VS code. I’m going to open up my package list right here. And I’m going to come down and I’m going to see here’s my express. So I’m going to go ahead and change that to 4.20.0. And we’re going to save it. And now I’m going to build V2 of this. Now it’s going to build with the new packages. Again, it’s going to take a few seconds for it to build. And then we’re going to rerun the quick view and see what’s going on. If I do the quick view now, I’m going to see that the number of vulnerabilities has gone down, which is good. That’s what we want is we want to actually see the number of vulnerabilities going down. Now I still have a bunch in the base image. So again, I’m going to pop back over to the GUI just because it provides a nice graphical way of showing this. I’m going to go into my V2 and I’m now going to see recommended fixes. I can do this again in the CLI or I can do it here. I’m going to get recommendations for base images. It says, all right, one option is you could simply refresh the base image and pull the latest. But that’s not really what I want to do. I want to specify my base image. So I’m going to come in here. I’m currently on 3.14. And so now it’s saying, OK, you can go to 3.20. And that’s actually going to reduce your criticals by 2, your highs by 15. You’re still going to have 1 medium. I’m OK with that. It’s giving me a little information about this new package. And it’s saying, hey, here’s the from command. So I’m going to grab that from. I’m going to pop back over to VS code. I’m going to go to my Dockerfile. I’ve got my from statement here. And I’m going to drop in the new one and save it. And now let’s build this one more time.

So now we’re going to build V3. Oh, before I do that, I almost forgot I do need to fix that user. So that I’m not running this as root. So let’s go ahead and do that as well. And now we’ll build v3. I just got to pull the new base image. So it’s going to take a few seconds for it to build again. All right. So now let’s do one more quick view. Go ahead and move this up so we can see. Now we see that the non-root users been solved. The no fixable critical or highs have been solved. And if I scroll up here, I see I’ve got 1 medium and 2 lows left, which I’m OK with. So this shows an example of the developer view of things where the individual developer can understand the vulnerabilities and policy issues and be able to remediate those with guidance. Now let’s go take a look at the Scout dashboard (scout.docker.com) to be able to actually see what’s going on at a higher level.

If I come over here to my browser and I go to scout.docker.com, I’m going to see a dashboard of all of my images that have been indexed from the repositories that I’ve linked. Regardless of whether those are Docker Hub, artifactory, ECR, ACR and be able to see information about those. Here I can see my policies, my default non-root user, my no AGPL V3 licenses, my no fixable critical or high vulnerabilities. I’m seeing information about the compliance, but I’m also seeing trending in the last seven days, have things gotten better or have they gotten worse. I can better understand what’s going on. I can see things like SonarQube quality gates, that kind of thing as well. I can also see realtime what the vulnerabilities are, because there’s always new vulnerabilities being discovered. So if there’s a new vulnerability being discovered, you want to know right away. You don’t have to do a whole rescan of everything. You want to know real time- How am I impacted by that? If the next Log4j comes out, you don’t want to have to spend weeks rescading on your code. You want to know immediately which applications have this version Log4j and how am I going to fix it? And so that’s what we’re providing with Scout is we’re providing the ability to actually be able to see these high and critical vulnerabilities or any vulnerability and be able to see immediately which of the different images are impacted by it. This is what the software bill materials allows us to do to understand what packages are in which images and what the impacts of those are. So here we can see some of the different recent high and critical vulnerabilities and we can see which images are impacted. So I can immediately go in that and look at those particular ones.

Now from here, we’re showing last pushed but we can also if we use tagging to specify environments, we can actually look at the ones that are tagged as production and be able to look at and see how we’re doing on a production basis as well. So this gives us the ability to actually look at how’s our production environment versus our other production environments as well. Now beyond just the overview, we can go look at things like the individual policies and see how we’re doing on a week over week basis. We can go look at the individual images, all the different images that have been indexed, how many vulnerabilities, when’s the last time they were pushed, all of that kind of stuff, whether they’re compliant with policies, all of that information. We can go look at the base images, we can see what base images are being used by all our different images and get information about that. We can look at each individual package, we can go look and say okay which particular packages are being used by which particular images. We can go by vulnerability. So if we’ve got a CVE number, we can just go drop it in here and see every particular image that’s impacted by that. And we can also just sort by criticality. So here the CVS score 10 vulnerabilities and here are the images those are included with. And then we could also go look at our integrations to different types of systems here. We can see the integrations to the container registries, we can see the CI integrations, real-time updates, integrations, all of those. You can get information about all the different integrations here as well.

Alright, I want to thank you all very much for listening to this presentation. Hope you enjoy working with Scout and secure new containers. Thank you.

 

Learn more

Containers often run in secure environments like production. What goes into creating a secure container? Both how the image is built and how it is executed as a container affect the container’s security. The container security and Docker Scout session covers multiple aspects of security – container users, read only containers, linux privileges, what should be included in the image, base images, vulnerabilities, policy enforcement, and remediation.

Our speakers

Kevin Barfield, Director, Solutions Architect, Docker