Scaling Docker’s Business to Serve Millions More Developers: Storage

Update as of October 22, 2020:  The dates in the blog below are no longer accurate. Please read about the updates at: Docker Hub Image Retention Policy Delayed, Subscription Updates

At Docker, our mission is to enable developers worldwide to quickly bring their ideas to life by reducing the complexity of application development. While over 6.5 million registered Docker developers are enjoying the benefits of Docker today, we want to scale Docker’s business to the tens of millions of developers who are just discovering Docker. Offering free tools and services is a cornerstone of our mission, and these are funded by our paid subscription services.

In this blog series, we will deep dive into why and how the recently announced Terms of Service changes were introduced. This blog, Part 1, will explore the inactive image retention policy and how it will impact development teams who use Docker Hub for managing their container images. Part 2 will focus on the new rate limits policies that were introduced for image pulls.

A deeper look at Docker Hub images

Delivering containerized applications in a portable, secure, and resource efficient manner also requires tools and services for securely storing and sharing applications with your development team. Today, Docker is proud to offer the world’s largest container registry, Docker Hub, which is used by over 6.5 million developers around the world. Over 15 PB of container images are stored on Docker Hub today, spanning everything from the world’s most popular in-memory databases to event streaming platforms, to highly curated and trusted Docker Official Images, and to the 150 million images created by the Docker community.

Docker’s internal analytics tools have shown that of the 15PB of images stored in Docker Hub, over 10PB of these images have not been accessed for more than 6 months. Digging a little deeper, we found that over 4.5PB of these inactive images are associated with free accounts. Many of these images were only being used for a short period of time, including images that resulted from CI pipelines configured with Docker Hub where the deletion of these ephemeral images was often overlooked.

With an enormous footprint of inactive data sitting idle in Docker Hub, the team was faced with a difficult question – how do we limit the amount of inactive data that Docker is incurring storage charges for on a monthly basis, while not impacting the overall Docker customer experience?

The guiding principles for addressing this challenge were two-fold:

1. Continue to offer a compelling suite of free-to-use Docker tools and services that developers – including open source projects – can use to build, share, and run applications.
2. Ensure Docker can scale to satisfy the demand of new developers by limiting the currently unbounded storage costs – one of the largest operating expenses for Docker Hub.

Helping developers manage inactive images

To help Docker economically scale its infrastructure to support free services for our growing base of users, several updates were announced. First, a new inactive image retention policy was introduced that will automatically delete images hosted in free accounts that have not been used in 6 months. In addition, Docker will also be providing tooling, in the form of a UI and APIs, that will allow users to more easily manage their images. Together, these changes will allow developers to more easily clean up their inactive images and also ensure Docker can economically scale its infrastructure.

With this new policy, starting on November 1, images stored in free Docker Hub repositories that have not had their manifest pushed or pulled in the last 6 months will be removed. This policy does not apply to images stored by paid Docker Hub subscription accounts, Docker Verified Publishers , or Docker Official Images.

• Example #1: Molly, a free Docker Hub user, pushed a tagged image molly/hello-world:v1 to Docker Hub on January 1, 2019. The image was never pulled since it was pushed. This tagged image will be considered inactive beginning November 1, 2020 when the new policy takes effect. The image and any tag pointing to hit, will be subject to deletion on November 1, 2020.

• Example #2: Molly has another untagged image molly/myapp@sha256:c0ffee that was first pushed on January 1, 2018. This image was last pulled on August 1, 2020. This image will be considered an active image and will not be subject to deletion on November 1, 2020.

Minimizing impact to the developer community

For free accounts, Docker offers free retention of images inactive for six months. For users that need their inactive images to be retained, Docker also offers unlimited image retention as a feature of the Pro and Team plans. Visit www.docker.com/pricing to view the available plans.

In addition, Docker will be offering a set of tools and services to help developers easily view and manage their images, including the following product updates which will be made available on Docker Hub in the coming months:

• Image management dashboard to view and manage images across all repositories within a namespace (roadmap issue #146)
• Email notifications for images that are set to expire (roadmap issue #147)
• Restoration of recently deleted images (roadmap issue #148)

Finally, as part of Docker’s commitment to the open source community, before November 1 we will be announcing availability of new open source plans. To apply for an open source plan, please complete the short form here.

For more information regarding the recent terms of service changes, please refer to the FAQ.

Stay tuned for email communications regarding any upcoming expiring images in your account or sign up for a Pro or Team plan for unlimited inactive image retention.

While we tried to minimize the impact on developers, it’s possible you have questions or use cases that are not covered. As always, we welcome your questions and feedback at [email protected].