This is a guest post from Viktor Petersson, CEO of Screenly.io. Screenly is the most popular digital signage product for the Raspberry Pi. Find Viktor on Twitter @vpetersson.
In the previous blog post, we talked about how we compile Qt for Screenly OSE using Docker’s nifty multi-stage and multi-platform features. In this article, we build on this topic further and zoom in on caching.
Docker does a great job with caching using layers. Each command (e.g., RUN, ADD, etc.) generates a layer, which Docker then reuses in future builds unless something changes. As always, there are exceptions to this process, but this is generally speaking true. Another type of caching is caching for a particular operation, such as compiling source code, inside a container.
At Screenly, we created a Qt build environment inside a Docker container. We created this Qt build to ensure that the build process was reproducible and easy to share among developers. Since the Qt compilation process takes a long time, we leveraged ccache to speed up our Qt compilation. Implementing ccache requires volume mounting a folder from outside of the Docker environment.
The above steps work well if you are the only developer working on the project. What happens if you want to be able to have a shared cache across a team?
There are a few ways to accomplish this style of caching in Docker.
The easiest way to establish a shared cache is by following what we did in the previous article. We used disk cache along with some neat features for speeding up caching in BuildKit. We then compressed the cached files and distributed them to team members. The process is not very elegant, but it gets the job done.
If we want to automate the process a bit further, we can include retrieving the cache as part of the build process. An example of this could looks like this:
RUN curl -o /tmp/build-cache.tgz https://some-domain.com/build-cache.tgz && \
tar xfz /tmp/build-cache.tgz -C /tmp && \
rm /tmp/build-cache.tgzThis process above is neat, but it does mean that someone will need to periodically upload the build cache in order to keep cache files fresh. In addition, you need somewhere to store the files (such as S3).
It would be nice if we could avoid manual tasks and use native Docker technologies to do the same thing, right? As it turns out, we can use Docker to improve the process. We just need to use our imagination.
As we showed in the previous article, we can use multi-stage builds to copy data between different docker images. What if we move the cache to a dedicated Docker image? We can then push this image to Docker Hub and pull it into the build process.
The process is straightforward. Start by creating two different images in Docker Hub. Call them screenly/build-cache and screenly/build-env. Building on the previous article, we use this Dockerfile as the basis for screenly/build-env.
In the Dockerfile, we set the environment variable CCACHE_DIR to /src/ccache. This step tells ccache that the cache resides in /src/ccache. In the previous post, the step was just a volume mount to the system. However, in this case, we want to alter this step so that the cache lives outside of /src, as this is used for volume mounting the code base, such as /usr/ccache.
We can now launch the container with:
$ docker run --rm -t \
-v ~/tmp/qt-src:/src \
-v ~/tmp/qt-build:/build \
-v ~/tmp/ccache:/usr/ccache \
screenly-build-env
After you have run through your compilation, you can now build and push our cache image. The final Dockerfile will look like this:
FROM scratch
COPY ccache /ccache
To build this image, use the following code:
$ cd ~/tmp
$ docker build \
-f /path/to/Dockerfile \
-t screenly/build-cache
$ docker push screenly/build-cache
Finally, you can now include this layer in the Dockerfile for screenly/build-env. Add the line:
COPY --from=screenly/build-cache /ccache /usr/ccache
Next time you rebuild screenly/build-env, Docker will automatically pull down the cache. Also, you only need to add volume mounting when you are refreshing the cache.
About Screenly
Screenly is the most popular digital signage product for the Raspberry Pi. If you want to turn a physical screen into a secure, remote-managed device (over UI or digital signage API), Screenly makes setup a breeze. Users can display beautiful dashboards, images, videos, and web page content.
Screenly is available in two flavors: an open source version and a commercial version. You can try out our commercial version with a 14-day free trial.